Intelligent training acceleration method and system based on data densification characteristics in neural networks

Abstract

The application provides an intelligent training acceleration method based on data dense characteristics in a neural network, which collects relevant data information and performance model information by pre-training a model, and dynamically reduces the data amount required to be transmitted between a GPU and a CPU by using the dense characteristics of a deep learning network in combination with compression and decompression algorithms, so as to improve the overall performance of the system. The application also provides an intelligent training acceleration system based on data dense characteristics in a neural network, which comprises a data acquisition module, a dense data compression rate decision module, a selective compression decision module and a training module; and the neural network training is carried out by using the system, so that the training speed and the overall performance of the system can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer science and artificial intelligence, and in particular to an intelligent training acceleration method and system for data with dense characteristics in a neural network. BACKGROUND

[0002] The innovation of deep learning technology has greatly promoted the development of various fields. In order to meet the high-precision requirements, the training of the model needs more device storage space to support. However, the storage capacity of the accelerator represented by GPU is very limited, which cannot accommodate the training of the model with large parameters, thereby restricting the development of deep learning. In order to solve this problem, the existing method selects to directly exchange data between the storage devices of GPU and CPU, thereby releasing the storage pressure of the GPU device. And in the transfer process, a compression method is introduced to compress the data, so as to alleviate the performance loss brought by the transfer. However, the existing compression method is only effective for sparse data, i.e. data with more 0s. For dense data (data with almost no 0s), the method cannot play a role. SUMMARY

[0003] For deep learning models, especially for deep learning models with dense data, in order to solve the performance problem caused by the long data transfer memory time, the present application proposes an intelligent training acceleration method and system based on the data dense characteristics in a neural network. The method uses the dense characteristics of the deep learning activation layer output data combined with the compression algorithm to realize dynamic decision-making, selects the most suitable compression ratio for each round of model training to compress the data transferred out of the GPU and decompresses it after being transferred back, so as to reduce the data transmission amount between the GPU and the CPU, reduce the transfer time, and thereby accelerate the overall training performance of the deep learning model.

[0004] The technical solution adopted by the present application is as follows:

[0005] An intelligent training acceleration method based on the data dense characteristics in a neural network, which selectively lossy compresses and decompresses the training data transferred between the GPU and the CPU in each round of training after the first round during the training of the neural network model, thereby accelerating the training of the neural network model; wherein the selective lossy compression and decompression comprises:

[0006] Compression rate selection:

[0007] During the first round of training of the neural network model, the size of the data of each layer, the forward propagation and backward propagation training time of each layer without compression, and the transfer bandwidth between the GPU and the CPU are collected;

[0008] Based on the collected data and the performance model of the lossy compression algorithm, the maximum and minimum compression ratios of each layer of data are determined. The maximum P value Top_t for that layer is obtained when the following two formulas are simultaneously satisfied, where t is the data index. The minimum compression ratio is then determined.

[0009] s / bs×(P / MAX) / b>=tc

[0010] s / bs×(P / MAX) / b>=tdc

[0011] In the formula, s is the size of each layer of data, b is the transfer bandwidth between the GPU and the CPU, tc and tdc represent the compression and decompression operation times, respectively; P is a hyperparameter for adjusting the compression ratio, compression ratio C = 1 - (P / MAX), and MAX is the maximum value set by P in the lossy compression algorithm;

[0012] When both of the following formulas are satisfied, the minimum value of P (Bottom_t) of the data in this layer is obtained, and the maximum compression ratio C is obtained. MAX ;

[0013] s×(P / MAX) / b<=h f

[0014] s×(P / MAX) / b<=h b

[0015] In the formula, h f with h b These represent the training time for the forward propagation and backward propagation processes, respectively.

[0016] The compression ratio for each training round is selected when transferring training data between the GPU and CPU based on the maximum and minimum compression ratio values.

[0017] Compression option: Calculate the transfer cost before and after compression based on the collected data. If the cost after compression is less than the transfer cost of the uncompressed data, then compress / decompress the corresponding data during the output data transfer process in the model training process; otherwise, do not compress.

[0018] T'(s)=max(2×(s / b)–h f -h b ,0)

[0019] T(s,r,tc,tdc)=tc+tdc+O f +O b

[0020] O f =max(s×(P / MAX) / b–h) f ,0)

[0021] Ob =max(s×(P / MAX) / b–h) b ,0)

[0022] In the formula, T'(s) represents the transfer cost of the uncompressed data s, and T(s,r,tc,tdc) represents the transfer cost of the compressed data.

[0023] Furthermore, the step of selecting the compression ratio when transferring training data between the GPU and CPU in each round of training based on the maximum and minimum compression ratios specifically involves: selecting a compression ratio of C during the early stages of neural network model training. MIN ~1 / 2(C MAX -C MIN )+C MIN The compression ratio chosen during the later stages of neural network model training is 1 / 2 (C MAX -C MIN )+C MIN ~C MAX The early stage of neural network model training refers to the number of training rounds being less than or equal to 1 / 2 Epoch, while the later stage of neural network model training refers to the number of training rounds being greater than 1 / 2 Epoch. Epoch is the total number of training rounds.

[0024] Furthermore, the method of selecting the compression ratio when transferring training data between the GPU and CPU in each training round based on the maximum and minimum values ​​of the compression ratio specifically means that the compression ratio gradually decreases as the number of training rounds increases.

[0025] Furthermore, the compression ratio C of the training data during each training round when transferred between the GPU and CPU is calculated using the following formula:

[0026] C = 1 - (P / MAX)

[0027] P = max(Top-Epoch) cur ×△P,Bottom)

[0028] △P = [(Top - Bottom) / Epoch + 0.5], where △P is an integer.

[0029] In the formula, Epoch cur This indicates the current epoch number, and Epoch is the total number of training epochs. △P is the difference between P in the previous and next epochs.

[0030] Furthermore, the selective lossy compression and decompression also includes: data selection, selecting dense data for lossy compression and decompression.

[0031] Furthermore, the performance model of the lossy compression algorithm is established using the following method:

[0032] The compression and decompression algorithms were pre-run using different data sizes and compression ratios. A linear regression model was used to quickly collect information and fit the relationship curves between data size, compression ratio and compression / decompression time, thus establishing a compression / decompression performance model for the lossy compression algorithm.

[0033] Furthermore, traditional lossless compression cannot effectively reduce the size of dense data, so this invention considers lossy compression algorithms. Lossy compression algorithms include ZFP compression or cuSZ compression. Moreover, testing has shown that ZFP compression has high compression efficiency, low accuracy loss, and the compression ratio can be set via an interface; therefore, ZFP is a more preferable choice for data compression.

[0034] Furthermore, by selecting the largest value among all data Top_t as the fixed parameter Top, the minimum compression ratio C during the training process of the neural network model is calculated. MIN The smallest value among all data points (Bottom_t) is selected as the fixed parameter Bottom, and the maximum compression ratio C obtained during the training process of the neural network model is calculated. MAX Based on the maximum compression ratio C MAX and minimum compression ratio C MIN Choose the compression rate for training data when transferring between the GPU and CPU in each training round.

[0035] A method and system for accelerating intelligent training based on the dense data characteristics of neural networks, for implementing the above method, comprising:

[0036] The data acquisition module is used to acquire data information during the training of the neural network model, including the size of the data in each layer, the training time of forward propagation and back propagation in each layer without compression, and the transfer bandwidth between the GPU and the CPU.

[0037] The compression ratio decision module is used to determine the maximum and minimum compression ratio of each layer of data based on the collected data information and performance model, and select the compression ratio when the training data is transferred between the GPU and CPU in each round of training based on the maximum and minimum compression ratio.

[0038] The selective compression decision module is used to calculate the transfer cost before and after compression based on the collected data information. If the cost after compression is less than the transfer cost of the uncompressed data, then the corresponding compression / decompression is performed when the output data is transferred during the model training process; otherwise, no compression is performed.

[0039] The training module is used to train the neural network model by executing the decisions of the compression ratio decision module and the selective compression decision module.

[0040] The beneficial effects of this invention are:

[0041] Traditional deep learning memory transfer optimization schemes require waiting for data transfer to complete, which severely limits overall system performance when the amount of data to be transferred is large or the transfer time is too long. Existing solutions can only compress sparse data. Therefore, this invention proposes an intelligent training acceleration method based on the dense data characteristics of neural networks. Unlike existing technologies that perform unprocessed data transfer or sparse data compression, this method dynamically adjusts the compression ratio P to perform lossy compression on the data before data transfer to balance the efficiency and accuracy of model training, and then decompresses the data after transfer. When the sum of the compressed data transfer time and the compression or decompression operation time is less than the uncompressed data transfer time, the system performance will improve. Attached Figure Description

[0042] Figure 1 The system architecture module diagram of the present invention includes a data acquisition module, a dense data compression rate decision module, a selective compression decision module, and a training module.

[0043] Figure 2 Diagram illustrating the case where compressed transfer overhead is greater than non-compressed transfer overhead.

[0044] Figure 3 A diagram illustrating the case where transfer costs equal benefits.

[0045] Figure 4 This is a diagram illustrating the scenario where compressed data is transferred and completely hidden.

[0046] Figure 5 Schematic diagram of selective compression Detailed Implementation

[0047] The method of the present invention will be described in detail below with reference to the accompanying drawings. The present invention is an intelligent training acceleration method based on the dense data characteristics of neural networks. During neural network model training, this method selectively performs lossy compression and decompression on the training data of each subsequent training round when transferring it between the GPU and CPU, thereby accelerating the training of the neural network model. The selective lossy compression and decompression includes: compression ratio selection and whether to compress. Specifically, it includes the following steps:

[0048] Step 1: Establish compression and decompression performance models for the corresponding lossy compression algorithm.

[0049] Taking the ZPF compression algorithm as an example, a large amount of dense data of different sizes is randomly generated and input into the ZPF compression algorithm. The compression and decompression times are obtained under different compression ratios. The data size, compression ratio, and compression and decompression time are used as feature values ​​to train the linear regression model and obtain the converged performance model.

[0050] Step 2: During the first round of training, collect data size, forward propagation time for each layer without compression, backpropagation training time, and transfer bandwidth between GPU and CPU.

[0051] Step 3: Based on the collected data and the performance models of the compression and decompression algorithms, determine the maximum and minimum compression ratios for each layer of data. A hyperparameter P is introduced to adjust the compression ratio, where C = 1 - (P / MAX), and MAX is the maximum value set for P in the lossy compression algorithm. The maximum and minimum compression ratios can be determined by defining the upper and lower bounds of P. Specifically:

[0052] (1) Finding the Upper Bound (Top): Before training, a suitable range of P values ​​will be found for each data to be compressed, i.e., a search will be conducted from P = MAX (the maximum value set in the ZFP lossy compression algorithm) to P = 0. First, the compression and decompression time cost of the data at the compression ratio P will be predicted. Second, the benefit of compressing the data at the current P value will be calculated to determine whether the benefit outweighs the cost. Figure 2 , Figure 3 As shown, if Benefit is less than Cost at this point, the P value is disregarded, and P continues to decrease until Benefit is greater than Cost. This is calculated using a formula. When the following two formulas are simultaneously satisfied (Benefit > Cost), the current P value is recorded as Top_t for this data:

[0053] s / bs×(P / MAX) / b>=tc

[0054] s / bs×(P / MAX) / b>=tdc

[0055] In the formula, s is the size of the uncompressed data in each layer, b is the transfer bandwidth between the GPU and the CPU, tc and tdc represent the compression and decompression operation times, respectively; s×(p / MAX) represents the size of the data after compression using a specific p value.

[0056] (2) Finding the Bottom Lower Bound: When the compression ratio is too small, the transitions will run completely in parallel with the training time of forward and backward propagation. This will not improve performance but will instead increase accuracy loss and affect model precision. Therefore, a lower bound P-value, Bottom_t, needs to be selected for each data point. During the decision-making process, a P-value that completely hides the transitions is found for each data point to be compressed. The formula is shown below, and the illustration is as follows. Figure 4 As shown.

[0057] s×(P / MAX) / b<=h f

[0058] s×(P / MAX) / b<=h b

[0059] h f with h b These represent the training time for the forward propagation and backward propagation processes, respectively.

[0060] A neural network model has N data points to be compressed. For each data point t, we need to find its corresponding Top_t and Bottom_t values. To facilitate calculation, we can take the largest of all Top_t values ​​as the fixed parameter Top and the smallest of all Bottom_t values ​​as the fixed parameter Bottom.

[0061] A larger P-value results in a lower compression ratio, worse data compression, and less precision loss; a smaller P-value results in a higher compression ratio and better compression, but with greater precision loss. P = Top, which yields the minimum compression ratio C. MIN Correspondingly, P = Bottom, thus obtaining the maximum compression ratio C. MAX .

[0062] Therefore, it is necessary to find a suitable hyperparameter P to achieve a good balance between accuracy and performance. That is, the compression ratio during the transfer of training data between the GPU and CPU in each training round is selected based on the maximum and minimum values ​​of the compression ratio.

[0063] Furthermore, the applicant found that deep learning models are more sensitive to accuracy loss in the early stages of training, requiring a higher P-value; as the model converges, the compression ratio can be appropriately increased, and a smaller P-value can be set to further improve training performance.

[0064] As a preferred option, the compression ratio chosen during the early stages of neural network model training is 1 / 2 (C MAX -C MIN )+C MIN ~C MAX The compression ratio chosen during the later stages of neural network model training is 1 / 2 (C MAX -C MIN )+C MIN ~C MAX The early stage of neural network model training refers to the number of training rounds being less than or equal to 1 / 2 Epoch, while the later stage of neural network model training refers to the number of training rounds being greater than 1 / 2 Epoch. Epoch is the total number of training rounds.

[0065] As another preferred approach, the compression ratio gradually decreases as the number of training rounds increases.

[0066] Specifically, based on the obtained range [Bottom, Top], and according to the number of training rounds set by the user, a parameter P is assigned to each training round. Round 0 starts from Top and gradually decreases ΔP until Bottom, as shown in the following formula:

[0067] P = max(Top - Epoch cur ×△P, Bottom)

[0068] △P is calculated according to the number of training epochs Epoch set by the user, and the calculation formula is as follows:

[0069] △P = [(Top - Bottom) / Epoch + 0.5], and △P is an integer

[0070] Step 4: After determining the compression ratio used in each round of training, it is necessary to make a compression selection: perform a selective compression judgment on the training data of each round, and compare the transfer time of the uncompressed data in the forward propagation and backward propagation with the total transfer time after optimization. The schematic diagram of the forward propagation is as shown in Figure 5 the left part, and the backward propagation is as shown in Figure 5 the right part. If the overhead after compression is less than the transfer overhead of the uncompressed data, corresponding compression / decompression is performed when outputting data transfer during model training; otherwise, no compression is performed;

[0071] T’(s) = max(2×(s / b) – h f – h b , 0)

[0072] T(s, r, tc, tdc) = tc + tdc + O f + O b

[0073] O f = max(s×(P / MAX) / b – h f , 0)

[0074] O b = max(s×(P / MAX) / b – h b , 0)

[0075] In the formula, T’(s) represents the transfer overhead of the uncompressed data, and T(s, r, tc, tdc) represents the transfer overhead corresponding to the compressed data. If T < T’, then the lossy compression algorithm is selected to compress the data during this round of training;

[0076] Step 5: During the training of the neural network model, based on the compression ratio selected in each round and the selected compressed data, selective lossy compression and decompression are performed when transferring the training data between the GPU and the CPU in each round of training after the first round, accelerating the training of the neural network model;

[0077] Based on the above method, the present invention also provides an intelligent training acceleration system based on the data density characteristics in the neural network, as shown in Figure 1As shown, it includes a data acquisition module, a dense data compression rate decision module, a selective compression decision module, and a training module; wherein:

[0078] The data acquisition module is used to acquire data information during the training of neural network models, including data size, parallelizable data transfer time, and transfer bandwidth between GPU and CPU;

[0079] The compression ratio decision module is used to determine the maximum and minimum compression ratio of each layer of data based on the collected data information and performance model, and select the compression ratio when the training data is transferred between the GPU and CPU in each round of training based on the maximum and minimum compression ratio.

[0080] The selective compression decision module is used to calculate the transfer cost before and after compression based on the collected data information. If the cost after compression is less than the transfer cost of the uncompressed data, the corresponding compression / decompression is performed when the output data is transferred during model training; otherwise, no compression is performed to further improve the system training performance.

[0081] The training module is used to execute the above decisions and train the model.

[0082] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0083] The following is a specific embodiment to further illustrate the technical effects of the present invention. The specific experiment is as follows:

[0084] Experimental setup:

[0085] (1) Operating system: Ubuntu 18.04.3LTS;

[0086] (2) CPU: Model 8-core Intel(R) Xeon(R) Gold 6126 CPU@2.60GHz, equipped with 32GB DRAM;

[0087] (3) GPU: Tesla V100 with 32GB of video memory;

[0088] (4) Storage devices: 512GB SK hynix SC311 SATA SSD; Western Digital WDCWD40EZRZ-75G HDD;

[0089] Model configuration:

[0090] (1) Model: Vgg16 PyTorch standard code;

[0091] (2) Dataset: Cifar10, containing 60,000 color images, each 32*32 in size, divided into 10 classes, with 6,000 images per class, of which 50,000 images are used for training and 10,000 images are used for testing;

[0092] (3) Batch size: 2560 images / batch;

[0093] (4) Number of training epochs: 50 epochs;

[0094] Solution configuration:

[0095] (1) Compression algorithm: ZFP compression algorithm;

[0096] (2) Activation function: Leaky ReLU (generates dense data)

[0097] Final test results:

[0098] Using the original, uncompressed method, the training throughput is 4.56 samples / second, while using the selective compression method described in this invention, it is 5.4 samples / second. The performance of the system using this invention is 1.2 times that of the original method under the VGG16 model, and the accuracy loss is controlled within 0.1%.

[0099] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An intelligent training acceleration method based on data density characteristics in a neural network, characterized in that, The method selectively compresses and decompresses the training data transferred between the GPU and the CPU during each round of training after the first round of training of the neural network model to accelerate the training of the neural network model; wherein the selective compression and decompression comprises: Compression rate selection: During the first round of training of the neural network model, the size of each layer of data, the forward propagation and backward propagation training time of each layer without compression, and the transfer bandwidth between the GPU and the CPU are collected; According to the collected data information and the performance model of the lossy compression algorithm, the maximum and minimum values of the compression rate of each layer of data are determined, and when the following two formulas are simultaneously satisfied, the maximum value Top_t of the compression rate of the layer of data is obtained, t is the number of data, and the minimum compression rate is obtained: s / b - s×(P / MAX) / b >= tc s / b - s×(P / MAX) / b >= tdc In the formula, s is the size of each layer of data, b is the transfer bandwidth between the GPU and the CPU, tc and tdc respectively represent the operation time of compression and decompression; P is a hyperparameter for adjusting the compression rate, the compression rate C = 1 - (P / MAX), and MAX is the maximum value of P set in the lossy compression algorithm; The minimum P value Bottom_t of the layer data is obtained when the following two formulas are satisfied, and the maximum compression rate C is obtained MAX ; s x (P / MAX) / b <= h f s x (P / MAX) / b <= h b where h f and h b denote the training time for the forward and backward pass, respectively. According to the maximum and minimum values of the compression rate, the compression rate of the training data transferred between the GPU and the CPU during each round of training is selected; Compression selection: according to the collected data information, the transfer overhead before and after compression is calculated, if the transfer overhead after compression is less than the transfer overhead of uncompressed data, the corresponding compression / decompression is performed during the transfer of the output data in the model training process, otherwise, no compression is performed. T'(s) = max(2 x (s / b) - h f - h b , 0) T(s, r, tc, tdc) = tc + tdc + O f + O b O f =max(s×(P / MAX) / b – h f , 0) O b =max(s×(P / MAX) / b – h b , 0) In the formula, T'(s) represents the transfer overhead of uncompressed data s, and T(s, r, tc, tdc) represents the corresponding transfer overhead of compressed data.

2. The method of claim 1, wherein, The compression rate of the training data selected in each round of training according to the maximum and minimum values of the compression rate is transferred between the GPU and the CPU, specifically: the compression rate selected in the early stage of neural network model training is C MIN ~1 / 2(C MAX -C MIN ) + C MIN , and the compression rate selected in the late stage of neural network model training is 1 / 2(C MAX -C MIN ) + C MIN ~C MAX , wherein the early stage of neural network model training refers to the number of neural network model training rounds being less than or equal to 1 / 2Epoch, the late stage of neural network model training refers to the number of neural network model training rounds being greater than 1 / 2Epoch, and Epoch is the total number of training rounds.

3. The method of claim 1, wherein, According to the increase of the training round number, the compression rate gradually decreases.

4. The method of claim 3, wherein, The compression rate C of the training data transferred between the GPU and the CPU during each round of training is obtained by the following formula: C = 1 - (P / MAX) P = max(Top-Epoch cur × △P, Bottom) △P = [(Top-Bottom) / Epoch+0.5], △P is an integer In the formula, Epoch cur represents the current round, and Epoch is the total number of training rounds. △P is the difference between the two P values, Top represents the upper bound of the P value, and Bottom represents the lower bound of the P value.

5. The method of claim 1, wherein, The selective compression and decompression further comprises data selection, which selects dense data for lossy compression and decompression.

6. The method of claim 1, wherein, The performance model of the lossy compression algorithm is established by the following method: Pre-run the compression and decompression algorithm using different data sizes and different compression rates, use a linear regression model to quickly collect information and fit a curve of the relationship between data size, compression rate and compression / decompression time, and establish a compression / decompression performance model of the lossy compression algorithm.

7. The method of claim 1, wherein, The lossy compression algorithm comprises a ZFP compression algorithm or a cuSZ algorithm.

8. The method of claim 1, wherein, Select the largest of all data Top_t as the fixed parameter Top, and calculate the minimum compression rate C in the training process of the neural network model MIN Select the smallest of all data Bottom_t as the fixed parameter Bottom, and calculate the maximum compression rate C obtained in the training process of the neural network model MAX According to the maximum compression rate C MAX And the minimum compression rate C MIN Select the compression rate of the training data transferred between GPU and CPU in each round of training.

9. An intelligent training acceleration method and system based on data dense characteristics in neural networks, for implementing the method of any one of claims 1-8, characterized in that, Comprises: The data collection module is configured to collect data information during the training of the neural network model, including the size of each layer of data, the training time of forward propagation and backward propagation of each layer of data without compression, and the transfer bandwidth between the GPU and the CPU. The compression rate decision module is configured to determine the maximum value and the minimum value of the compression rate of each layer of data according to the collected data information and the performance model, and to select the compression rate of the training data transferred between the GPU and the CPU in each round of training according to the maximum value and the minimum value of the compression rate. The selective compression decision module is configured to calculate the transfer overhead before and after compression according to the collected data information, and if the overhead after compression is less than the transfer overhead of the uncompressed data, corresponding compression / decompression is performed when the training data is output during the model training process, otherwise, no compression is performed. The training module is configured to train the neural network model according to the decisions of the compression rate decision module and the selective compression decision module.

Images