A method to increase the amount of deep learning training data
By adjusting the ratio of batch size and sub-batch size, the near-pool distribution of sample distribution is increased, and the gradient descent method is improved, the problem of training large neural networks with limited memory is solved, and the training stability and effect is improved. It is suitable for different network structures and Optimizer.
Patent Information
- Application Number
- CN201910510397.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-06-13
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-06-13
AI Technical Summary
In existing deep learning training, devices with limited memory or video memory are difficult to effectively train large neural networks, and the training effect is limited by batch size, resulting in the failure to fully realize the training potential.
By adjusting the proportion of batch size and sub-batch size, increasing the batch size to increase the near-popular distribution of the sample distribution, the improved gradient descent method is used to gradually accumulate and average gradient update model parameters, which is suitable for different network structures and optimizers.
Improve training stability and effectiveness, making memory-limited devices available for training large networks, speeding up training processes, and lowering training thresholds.
Smart Images

Figure CN112085179B_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to the field of deep learning technology, and in particular to a method for increasing the amount of deep learning training data. [Background technology]
[0002] In the past few years, the scale of neural networks has continued to expand, and training may require a large amount of data and computing resources. For example, one of the classic CNN models, inception-v3, has 23 million parameters. The larger the model, the more training data is required. Batch stochastic gradient descent is usually used to train neural networks, that is, the samples are divided into m samples (mini-batches), each sample (mini-batch) contains n samples (mini-batches); the gradient of each sample is calculated in each sample (mini-batch), and then the sum and average in this sample (mini-batch) are used as the final gradient to update the parameters.
[0003] Generally speaking, if the samples contained in each sample (mini-batch) are more consistent with the overall distribution, the better the training effect. The larger the batch size of the sample, the more likely it is to be consistent with the overall distribution. Unfortunately, the memory occupied by training (if training on the CPU) or video memory (if training on the GPU) increases proportionally with the batch size. However, memory or video memory is always limited. Therefore, training always fails to realize the theoretical potential. [Summary of the invention]
[0004] The technical problem to be solved by the present invention is to provide a method for increasing the amount of deep learning training data, which can lower the training threshold and allow CPUs with limited memory and GPUs with limited video memory to be used for training large networks; improve the training effect, the increase of batch size batch_size makes the distribution of samples closer to the distribution of the population, making the training more stable and the training effect better; speed up the training process, the increase of batch size batch_size can use a larger learning rate, thereby speeding up the training process; through the improvement of the ordinary gradient descent method, it can be applicable to different network structures, such as fully connected, CNN, RNN and other networks; it can be applicable to different optimizers, such as SGD, ADAM, ADAGRAD, RMSPROP, etc.
[0005] To solve the above technical problems, an embodiment of the present invention provides a method for increasing the amount of deep learning training data, comprising the steps of:
[0006] S1. Determine the batch size for updating the gradient according to the total amount of data, and determine the sub-batch size for calculating the gradient according to the memory or video memory data;
[0007] S2. Calculate the number of sub-batches included in the batch size for updating the gradient, where the number of sub-batches included in the batch size for updating the gradient is the quotient of the batch size for updating the gradient and the sub-batch size for calculating the gradient;
[0008] S3. Run the network forward on the number of sub-batches of data contained in the batch size of the updated gradient, and then perform backpropagation to calculate the gradients with respect to the network parameters.
[0009] S4, executing step S3 to update the batch size of the gradient for the number of sub-batches included, accumulating the number of sub-batches included in the batch size of the updated gradient and taking the average gradient;
[0010] S5. Update the model parameters using the average gradient, i.e. w t+1 =w t –ηgrad, where w t+1 is the model parameter of the next step, w t is the model parameter of the previous step, η is the learning rate, and grad is the average gradient.
[0011] Preferably, the batch size for updating the gradient is 1024-65536.
[0012] Preferably, the batch size includes 16-256 sub-batches.
[0013] Preferably, the learning rate η is 10 -4 ~1.
[0014] Preferably, step S5 of updating the model parameters using the average gradient includes:
[0015] S51, after each epoch of training data is used up, shuffle all training data, and execute step S52 to cyclically update the gradient;
[0016] S52, perform a gradient update, take batch data in sequence, execute batch loop to calculate gradients, accumulate these gradients and calculate the average value, and update the gradient;
[0017] S53: Perform a gradient calculation. In each batch, take the sub-batch data for calculating the gradient in turn and calculate the gradient.
[0018] The network's forward propagation is run on the number of sub-batches of data contained in the batch size of the updated gradient, and then backpropagation is performed to calculate the gradients with respect to the network parameters, and the gradient calculation results with respect to the network parameters are placed in the buffer.
[0019] Preferably, when the loss continues to increase or remains constant, stop updating the model parameters with the average gradient.
[0020] Preferably, the batch size of the updated gradient is 4096.
[0021] Preferably, the batch size includes 32 sub-batches.
[0022] Preferably, the learning rate η is 1.
[0023] Preferably, the difference in average loss between two trainings using all samples in the training set is no more than 1% as a criterion for judging whether the loss continues to increase or remains unchanged.
[0024] Compared with the prior art, the above technical solution has the following advantages: by increasing the batch-size, the distribution of samples is closer to the distribution of the population, making the training more stable and the training effect better; by increasing the batch-size, a larger learning rate can be used, thereby speeding up the training process; CPUs with limited memory and GPUs with limited video memory can be used to train large networks, lowering the training threshold; by improving the ordinary gradient descent method, it can be applied to different network structures, such as fully connected, CNN, RNN and other networks; it can be applied to different optimizers, such as SGD, ADAM, ADAGRAD, RMSPROP, etc.
Brief Description of the Drawings
[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0026] Figure 1 It is a flow chart of the method for increasing the amount of deep learning training data of the present invention.
[0027] Figure 2 yes Figure 1 Flowchart of the mini-batch gradient update algorithm in the embodiment. [Specific implementation method]
[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0029] In the following, batch-size indicates the batch size for updating gradients, mini-batch-size indicates the sub-batch size for calculating gradients, and mini-batch-number indicates the number of sub-batches contained in the batch size for updating gradients.
[0030] Embodiment 1
[0031] Figure 1 This is a flow chart of the method for increasing the amount of deep learning training data in the present invention. Step 1:
[0032] Step 1, parameter setting, includes the following sub-steps.
[0033] Step 1.1: Determine the batch size for updating the gradient according to the total amount of data, etc. For example, the batch size for updating the gradient can be 1024 to 65536, and the value in this embodiment is 4096.
[0034] Step 1.2: According to the memory or video memory limitation, determine the mini-batch-size for calculating the gradient of gradient descent. For example, the mini-batch-size for calculating the gradient can be designed to be 16 to 256, and the value in this embodiment is 32.
[0035] Step 1.3: Calculate the number of sub-batches contained in the batch size of the required gradient update, and calculate the number of sub-batches contained in the batch size of the gradient update, mini-batch-number = batch-size / mini-batch-size. Using the above data, the sub-batch size of the gradient is calculated as mini-batch-number = 4096 / 32 = 128.
[0036] Step 2, training the data, includes the following sub-steps, taking the above numbers as an example.
[0037] Step 2.1: Run the network forward on the mini-batch-size data, and then perform error backpropagation to calculate the gradient with respect to the network parameters. The gradient results are placed in the buffer.
[0038] Step 2.2: Continuously execute step 2.1 mini-batch-number, that is, 128 times. The 128 gradient values in the buffer are accumulated and averaged, that is,
[0039] Where grad represents the average gradient, i represents the i-th sub-batch, and its maximum value is the mini-batch-number. iRepresents the calculated gradient of the i-th sub-batch.
[0040] Step 2.3: Update the model parameters using the average gradient grad. t+1 =w t -η×grad. Where grad represents the average gradient, w t+1 is the model parameter of the next step, w t is the model parameter of the previous step, and the learning rate η is 10 -4 ~1, in this embodiment, η is 1.
[0041] Step 2.4: Repeat the above process until the loss no longer decreases, that is, the loss continues to increase or remains unchanged. In specific implementation, the difference in the average loss of the two trainings using all the samples in the training set is no more than 1% as the criterion for judging whether the loss no longer decreases, that is, the loss continues to increase or remains unchanged.
[0042] Embodiment 2
[0043] Figure 2 yes Figure 1 Flowchart of the mini-batch gradient update algorithm in the embodiment. In the traditional method, the execution unit batch is equivalent to step, and the gradient is updated while calculating the gradient for each batch of data. However, this patent has 3 levels of execution units, forming 3 levels of nested loops:
[0044] One epoch is equal to using all samples in the training set for training once. In layman's terms, the value of epoch is how many times the entire dataset is spun around. One epoch uses up all the training data. In each epoch, the entire training set is shuffled and the step loop is executed to update the gradient.
[0045] The mini-batch gradient update algorithm starts, shuffles the data, takes the batch data, takes the mini-batch data, and then calculates the gradient. Perform a gradient update step by step. In each step, take the batch data in turn, execute the batch loop to calculate the gradient, accumulate and average these gradients, and update the gradient. Perform a gradient calculation for a batch. In each batch, take the mini-batch data in turn and calculate the gradient. Determine whether the sample data has been used up. If so, determine whether the loss has decreased; if the sample data has not been used up, continue to take the mini-bacth value. If the loss has decreased, end the mini-batch gradient update algorithm process; if the loss has not decreased, continue to shuffle the data, and continue in this way.
[0046] From the above description, it can be seen that the beneficial effects of using a method for increasing the amount of deep learning training data according to the present invention are: by increasing the batch-size, the distribution of samples is closer to the distribution of the population, making the training more stable and the training effect better; by increasing the batch-size, a larger learning rate can be used, thereby speeding up the training process; CPUs with limited memory and GPUs with limited video memory can be used to train large networks, lowering the training threshold; through improvements on ordinary gradient descent methods, it can be applicable to different network structures, such as fully connected, CNN, RNN and other networks; it can be applicable to different optimizers, such as SGD, ADAM, ADAGRAD, RMSPROP, etc.
[0047] The embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A method for increasing the amount of deep learning training data, characterized in that: include: S1. Determine the batch size for updating the gradient based on the total amount of data, and determine the sub-batch size for calculating the gradient based on the memory or video memory limit; S2. Calculate the number of sub-batches included in the batch size for updating the gradient, where the number of sub-batches included in the batch size for updating the gradient is the quotient of the batch size for updating the gradient and the sub-batch size for calculating the gradient; S3. Run the network forward on the number of sub-batches of data contained in the batch size of the updated gradient, and then perform backpropagation to calculate the gradients with respect to the network parameters; S4, executing step S3 to update the batch size of the gradient for the number of sub-batches included, accumulating the number of sub-batches included in the batch size of the updated gradient and taking the average gradient; S5. Update the model parameters using the average gradient, i.e. w t+1 =w t –ηgrad, where w t+1 is the model parameter of the next step, w t is the model parameter of the previous step, η is the learning rate, and grad is the average gradient; The step S5, updating the model parameters using the average gradient, comprises: S51, after each epoch of training data is used up, shuffle all training data, and execute step S52 to cyclically update the gradient; S52, perform a gradient update, take batch data in sequence, execute batch loop to calculate gradients, accumulate these gradients and calculate the average value, and update the gradient; S53: Perform a gradient calculation. In each batch, take the sub-batch data for calculating the gradient in turn and calculate the gradient.
2. The method for increasing the amount of deep learning training data according to claim 1, characterized in that: The batch size of the gradient update is 1024 to 65536.
3. The method for increasing the amount of deep learning training data according to claim 1, characterized in that: The batch size includes 16 to 256 sub-batches.
4. The method for increasing the amount of deep learning training data according to claim 1, characterized in that: The learning rate η is 10 -4 ~1.
5. The method for increasing the amount of deep learning training data according to claim 1, characterized in that , run the network's forward propagation on the data of the number of sub-batches contained in the batch size of the updated gradient, and then perform backpropagation to calculate the gradients relative to the network parameters, and the gradient calculation results relative to the network parameters are placed in the buffer.
6. The method for increasing the amount of deep learning training data according to claim 1, characterized in that: When the loss continues to increase or remains unchanged, stop updating the model parameters with the average gradient.
7. The method for increasing the amount of deep learning training data according to claim 2, characterized in that: The batch size for updating the gradient is 4096.
8. The method for increasing the amount of deep learning training data according to claim 3, characterized in that: The batch size contains 32 sub-batches.
9. The method for increasing the amount of deep learning training data according to claim 4, characterized in that: The learning rate η is 1.
10. The method for increasing the amount of deep learning training data according to claim 6, characterized in that: The difference of the average loss between the two trainings using all the samples in the training set is no more than 1%, which is used as the criterion for judging whether the loss continues to increase or remain unchanged.
