Deep learning picture data preprocessing method and system for heterogeneous memory device

Abstract

The application provides a deep learning picture data preprocessing method and system for heterogeneous memory devices. In the deep learning model training process, all training data are divided into important samples and unimportant samples based on the loss importance sampling results of all training data in the last round of iterative training. The important samples are preprocessed by at least two data enhancement methods, and the unimportant samples are preprocessed by one data enhancement method, which are used as the input of the model in the next round of training. By using more complex data enhancement methods for important data and simpler data enhancement methods for unimportant data, the CPU computing burden is greatly relieved, and the training speed is accelerated without losing the model training accuracy.

Description

Technical Field

[0001] This invention relates to the field of deep learning in computer science, and in particular to an efficient preprocessing system for deep learning image data. Background Technology

[0002] In the field of artificial intelligence, deep learning has increasingly become a key research area for scholars in recent years, promoting the development of computer vision, natural language processing, and other related fields. The deep learning model training process can be summarized as continuously adjusting the model parameters to minimize the model's loss value on the training data. The final effect of model training depends on whether the distribution of the training data is consistent with the distribution in the real world.

[0003] To ensure that training data accurately reflects real-world conditions, the common practice is to collect a large-scale dataset. However, this requires enormous manpower, making it impractical to collect a suitable dataset. Therefore, "data augmentation" methods have emerged. These methods perform simple transformations on the training data to create new data, protecting it from noise (e.g., the angle at which an image is taken) and making it more closely resemble the distribution of the real world. For example, in computer vision, images used to train models are often cropped, rotated, or otherwise manipulated before training. The transformations involved in all training data are usually multiple, and the transformations used for a specific data point are selected from these operations according to a certain strategy.

[0004] Unlike model training, where forward propagation and gradient updates are performed on GPUs, data augmentation computations are conducted on CPUs. This is because different images require different data augmentation methods, making it impossible to utilize the parallel computing capabilities of GPUs. However, data augmentation demands significant computational power, which CPUs cannot handle quickly enough to complete. Furthermore, GPUs must wait for the data augmentation results before updating the model's gradients, and since CPU data augmentation takes longer than GPU gradient updates, this waiting time for GPU computation extends model training time.

[0005] In the field of deep learning, there is a technique called importance sampling. This technique evaluates input data based on a calculated metric (e.g., loss value) to determine the importance of all data points. This importance information guides the training process, accelerates model convergence, and improves accuracy to some extent. Previously, each training data point was used to train the model in each epoch, meaning each training data point had an equal probability of being sampled. However, by utilizing this importance information, the sampling probability of important data points is increased, ensuring that important data points appear multiple times in each epoch. This is how importance information has traditionally been used. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by combining importance sampling algorithms with data augmentation, proposing a new way to utilize importance information, thereby reducing the need for data augmentation, decreasing the computational burden on the CPU, and ultimately reducing model training time.

[0007] The technical solution adopted in this invention is:

[0008] A deep learning image data preprocessing method for heterogeneous memory devices, specifically:

[0009] During the training of a deep learning model, based on the loss importance sampling results of all training data in the previous round of training, all training data are divided into important samples and unimportant samples. The important samples are preprocessed using at least two data augmentation methods, and the unimportant samples are preprocessed using one data augmentation method, which are then used as the input of the model in the next round of training.

[0010] Furthermore, it also includes:

[0011] The important and unimportant samples after preprocessing are randomly shuffled, and the randomly shuffled samples are distributed to each batch during iterative training according to the proportion of the loss importance sampling results.

[0012] Furthermore, the unimportant samples undergo final enhancement preprocessing, specifically as follows:

[0013] Enhanced images corresponding to unimportant samples are obtained from the data storage module and preprocessed using final enhancement. The data storage module stores one or more enhanced images of all training data.

[0014] Furthermore, the enhanced image is a compressed quality enhanced image.

[0015] Furthermore, the enhanced image is obtained by performing random augmentation on the training data.

[0016] A deep learning image data preprocessing system for heterogeneous memory devices includes:

[0017] The importance differentiation module is used to divide all training data into important samples and unimportant samples based on the loss importance sampling results of all training data in the previous round of training iteration;

[0018] The data augmentation module is used to preprocess the important samples using at least two data augmentation methods, and to preprocess the unimportant samples using one data augmentation method.

[0019] Furthermore, it also includes:

[0020] The data shuffle module is used to randomly shuffle the important and unimportant samples after preprocessing, and then distribute the randomly shuffled samples to each batch during iterative training according to the proportion of the loss importance sampling results.

[0021] Furthermore, it also includes:

[0022] A data storage module that stores one or more augmented images containing all the training data.

[0023] The beneficial effects of this invention are: This invention provides a deep learning image data preprocessing method for heterogeneous memory devices. By using more complex data augmentation techniques for important data and simpler data augmentation techniques for unimportant data, the CPU computing burden is greatly reduced, and the training speed is accelerated without sacrificing the model training accuracy. Attached Figure Description

[0024] Figure 1 The system architecture diagram shows that ① is the request for the original image for important data, ② is the request for the enhanced image for unimportant data, ③ is the reading of important data and the random and final enhancement, ④ is the reading of unimportant data and the final enhancement, ⑤ is the transmission of the enhanced data to the GPU for model training, and ⑥ is the transmission of importance information to the CPU for importance differentiation.

[0025] Figure 2 A comparison chart of accuracy under different enhancement methods;

[0026] Figure 3 A comparison of image sizes at different quality levels;

[0027] Figure 4 This is a comparison chart showing the accuracy of models trained using images of different qualities. Detailed Implementation

[0028] This invention provides a deep learning image data preprocessing method for heterogeneous memory devices based on a combination of importance sampling and data augmentation. Unlike previous methods that utilize data importance (increasing the frequency of training models on important data), this invention employs more complex data augmentation techniques for important data and simpler techniques for unimportant data, as follows:

[0029] During the training of a deep learning model, based on the loss importance sampling results of all training data in the previous iteration, all training data are divided into important samples and unimportant samples. The important samples are preprocessed using at least two data augmentation methods, while the unimportant samples are preprocessed using one data augmentation method.

[0030] The importance algorithm employed is based on the loss values ​​of images augmented with data from the previous training iteration of the training data. The importance ranking is based on the loss values ​​obtained during forward propagation; that is, the importance information for each epoch is obtained during the forward propagation of the previous epoch, thus introducing no additional overhead. Each epoch ranks all samples according to their loss values; a higher loss value indicates a more important sample. A threshold L is selected, and samples ranked before L are considered important, while the rest are considered unimportant. For important samples, the original image is read in and preprocessed using at least two data augmentation methods; the results are then used as input to the model. For unimportant samples, a single data augmentation method is used for preprocessing, and the result is then used as input to the model.

[0031] The choice of the importance threshold can be determined based on the performance of the CPU and GPU. Since the computational load of the GPU is fixed, its computation time is readily available. To offset the time required for data augmentation on the CPU against the time required for model training on the GPU, the CPU computation time can be adjusted by regulating the importance threshold L. A larger importance threshold L requires more data augmentation, thus increasing the CPU computation. Additionally, a user-adjustable parameter m can be set, representing the user's acceptable additional time commitment for data augmentation. A larger m allows for a greater amount of computation on the CPU per unit time, and the importance threshold L can be chosen accordingly.

[0032] Data augmentation methods generally include random augmentation and final augmentation. To verify the effectiveness of combining importance sampling with data augmentation, we took random augmentation and final augmentation as examples. Data was sorted according to the loss value, with the top 30% considered important and the rest unimportant. Three experimental groups were set up: RandAugment and Final operations were performed on the top 30% of the important data, while only Final operation was performed on the rest; RandAugment and Final operations were performed on the bottom 30% of the important data, while only Final operation was performed on the rest; and RandAugment and Final operations were performed on 30% of the data randomly selected, while only Final operation was performed on the rest. The experimental results are as follows: Figure 2 As shown, this demonstrates that combining importance sampling with data augmentation is effective.

[0033] Typically, data augmentation strategies are consistent, performing RandAugment and Final operations on all data. However, by utilizing importance sampling methods, RandAugment operations on 70% of the data can be reduced, significantly alleviating the CPU's computational burden. In practical applications, the proportion of important data needs to be determined based on the computing performance of the CPU and GPU.

[0034] Furthermore, to reduce the computation time for data augmentation, this invention first pre-stores multiple augmented results for each sample in a data storage module. Augmented images corresponding to unimportant samples are then retrieved from the data storage module and preprocessed using the final augmentation. The pre-stored data augmentation results can be obtained using various data augmentation methods, preferably RandAugment. However, this method consumes a significant amount of space; therefore, as a preferred solution, compression is also employed, where the data augmentation results are compressed before storage, resulting in compressed-quality augmented images. Figure 3 and Figure 4 Experimental results show that the accuracy of the model trained with images of a certain compression quality does not decrease significantly, but the required storage space will decrease significantly.

[0035] The number of pre-stored images is determined by the space provided by the user, with a minimum of 0 and a maximum of all available storage space on the storage device. A larger space allows for the storage of more augmented images, resulting in greater data diversity and higher accuracy of the trained model.

[0036] When training a deep learning model, an epoch, which is the process of training all data once, is often divided into multiple iterations. The data trained in each iteration is called a batch. The training data needs to undergo a shuffle operation, which randomly shuffles the order of the training data. This is to ensure that the distribution of training data in each batch is as consistent as possible with the entire training set.

[0037] Using a completely random shuffle method introduces new problems. Consider this scenario: Batch 1 contains all important data, while the adjacent Batch 2 contains all unimportant data. Since important data requires more augmentation than unimportant data, this leads to a significant difference in CPU computation time between Batch 1 and Batch 2. Consequently, the GPU has to wait for the CPU computation of Batch 1, while the CPU computation of Batch 2 completes before the GPU computation, which greatly extends the overall training time of the model.

[0038] To address this issue, this invention introduces a novel Shuffle mechanism: after preprocessing, important and unimportant samples are separated and randomly shuffled. Based on the proportion of the loss importance sampling results, the randomly shuffled samples are distributed proportionally to each batch during iterative training (the proportion of important to unimportant samples in each batch is consistent). The distribution is performed according to the order of the data index after random shuffling.

[0039] The new Shuffle mechanism will ensure the randomness of the training data order while also distributing important data evenly across each batch, thus ensuring that the data distribution of each batch is as consistent as possible with the entire training set.

[0040] The present invention also provides an embodiment of a deep learning image data preprocessing system for heterogeneous memory devices.

[0041] See Figure 1 A deep learning image data preprocessing system for heterogeneous memory devices mainly includes:

[0042] The importance differentiation module is used to divide all training data into important samples and unimportant samples based on the loss importance sampling results of all training data in the previous round of training iteration;

[0043] The data augmentation module is used to preprocess the important samples using at least two data augmentation methods, and to preprocess the unimportant samples using one data augmentation method.

[0044] The system of this invention is deployed on the CPU, which greatly reduces the CPU's computing burden. The data preprocessed by this invention is finally sent to the model training module of the GPU as input to the model for the next round of training.

[0045] Furthermore, it also includes:

[0046] The data shuffle module is used to randomly shuffle the important and unimportant samples after preprocessing, and then distribute the randomly shuffled samples to each batch during iterative training according to the proportion of the loss importance sampling results.

[0047] And / or a data storage module that stores one or more augmented images containing all the training data.

[0048] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0049] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A deep learning image data preprocessing method for heterogeneous memory devices, characterized in that, Specifically: During the training of a deep learning model, based on the loss importance sampling results of all training data in the previous iteration, all training data are divided into important samples and unimportant samples. The important samples are preprocessed using at least two data augmentation methods, while the unimportant samples are preprocessed using one data augmentation method. The important and unimportant samples after preprocessing are randomly shuffled, and the randomly shuffled samples are distributed to each batch during iterative training according to the proportion of the loss importance sampling results.

2. The method according to claim 1, characterized in that, The unimportant samples undergo final enhancement preprocessing, specifically: Enhanced images corresponding to unimportant samples are obtained from the data storage module and preprocessed using final enhancement. The data storage module stores one or more enhanced images of all training data.

3. The method according to claim 2, characterized in that, The enhanced image is a compressed quality enhanced image.

4. The method according to claim 2, characterized in that, The enhanced images are obtained by performing random augmentation on the training data.

5. A deep learning image data preprocessing system for heterogeneous memory devices, characterized in that, include: The importance differentiation module is used to divide all training data into important samples and unimportant samples based on the loss importance sampling results of all training data in the previous round of training iteration; The data augmentation module is used to preprocess the important samples using at least two data augmentation methods, and to preprocess the unimportant samples using one data augmentation method. The data shuffle module is used to randomly shuffle the important and unimportant samples after preprocessing, and then distribute the randomly shuffled samples to each batch during iterative training according to the proportion of the loss importance sampling results.

6. The system according to claim 5, characterized in that, Also includes: A data storage module that stores one or more augmented images containing all the training data.

Images