A deep neural network optimization method combining AdamW and SGDM

Abstract

The present invention discloses a deep neural network optimization algorithm that combines AdamW and SGDM. Considering that large models and big data currently have high requirements for model convergence speed, all optimization algorithms have the possibility of falling into local optimum. The present invention introduces the idea of separate learning rate, integrates the AdamW and SGDM algorithms, improves the algorithm accuracy, makes the convergence speed faster and more efficient; adopts the method of reusing some calculation results to avoid the increase of algorithm time and space complexity; constructs a parallel structure to avoid local optimum strategy, adopts the strategy of discarding part of the update amount for AdamW, avoids the problem of falling into local optimum and being difficult to jump out of the extreme point due to the small gradient in the later stage, and adopts the method of dynamically adjusting the learning rate for SGDM to jump out of the local optimum; proposes a learning rate adjustment strategy, when the algorithm stops optimizing, by increasing the learning rate, so that the deep neural network jumps out of the local optimum, effectively improving the final accuracy of the algorithm.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep neural networks, and specifically relates to a deep neural network optimization method combining AdamW and SGDM. Background Art

[0002] Deep neural networks are an emerging research area in machine learning. They use complex nonlinear functions to simulate the inherent patterns and representational hierarchies of learning sample data. Deep neural networks include many branches, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). They have achieved results far exceeding previous technologies in areas such as image processing and speech recognition.

[0003] The process by which a deep neural network calculates gradients based on input data and optimizes its parameters to mimic the patterns of the sample data is called training. Optimization algorithms in deep neural networks optimize model parameters based on the gradients of each parameter during training. Different optimization algorithms directly affect the model's convergence speed and ultimate accuracy. A suitable optimization algorithm can conserve computing resources during training while improving the model's final performance. Therefore, designing a suitable optimization algorithm has become a pressing challenge for deep neural network researchers.

[0004] Currently, deep neural network optimization primarily relies on non-adaptive and adaptive algorithms, both of which are gradient-based. Non-adaptive algorithms, such as gradient descent (GD), do not change the direction of the model gradients they obtain. Some, such as stochastic gradient descent (SGD), simply accumulate the gradients themselves. Adaptive algorithms scale each gradient value, thereby changing the direction of the model update. These include Adagrad, rmsprop, Adam, and AdamW, based on their development history. While non-adaptive optimization algorithms are slower during training, they are less likely to fall into local optima and ultimately achieve higher accuracy. Adaptive optimization algorithms converge quickly, which is crucial given the ever-increasing number of model parameters and datasets. However, their final results often fall short of the performance of non-adaptive optimization algorithms.

[0005] In summary, the existing algorithms have the following main problems: First, large models and big data have high requirements for model convergence speed; second, all optimization algorithms have the possibility of falling into local optimality. Summary of the Invention

[0006] In response to the deficiencies or improvement needs of the existing technology, the present invention proposes a deep neural network optimization method that combines the advantages of AdamW and SGDM.

[0007] A deep neural network optimization method combining AdamW and SGDM, comprising the following steps:

[0008] S1: Build a deep neural network model according to the data set requirements, randomly initialize the parameters in the deep neural network model, and divide the data set into two parts: training set and test set. The data set is image data;

[0009] S2: Calculate the gradient of the deep neural network model corresponding to the current training data;

[0010] S3: Calculates the update amount of AdamW and SGDM parts in the deep neural network optimization algorithm based on the gradient of the current training data;

[0011] S4: Discard some of the updates of AdamW in the deep neural network optimization algorithm;

[0012] S5: If the deep neural network stops optimizing, adjust the learning rate of the SGDM part;

[0013] S6: Multiply the processed AdamW and SGDM parts in S3 and S4 by their respective learning rates and add them together to obtain the update amount of each parameter;

[0014] S7: Complete the weight decay part and update the parameters of the deep neural network according to the update amount in S6. The number of iterations increases by 1. If the training of all data in the training set has been completed, the epoch increases by 1.

[0015] S8: When the epoch reaches the set value, output the current model parameters and end this process; otherwise, return to S2;

[0016] The S1 includes the following sub-steps:

[0017] S11: Read the preset hyperparameters and random seeds;

[0018] S12: Randomly initialize the parameters of the deep neural network model according to the random seed according to a normal distribution with a mean of 0 and a variance of 1;

[0019] The S3 includes the following sub-steps:

[0020] S31: Calculate the cumulative gradient and the cumulative square of the gradient;

[0021] S32: Calculate the adaptive learning rate part of AdamW;

[0022] S33: Multiply the accumulated gradient, the inverse of the accumulated squared gradient, and the adaptive learning rate to form the update amount of AdamW. At the same time, the accumulated gradient also constitutes the update amount of SGDM.

[0023] S4 includes the following sub-steps:

[0024] S41: Generate a sequence that conforms to the Bernoulli distribution and has the same total number of parameters as the deep learning model according to the hyperparameters;

[0025] S42: Multiply the update amount of AdamW by the numbers in the sequence in turn to obtain the processed update amount of AdamW;

[0026] S5 includes the following sub-steps:

[0027] S51: After the last training of each epoch ends, record the accuracy of the current training set. If the number of recorded accuracies has reached 5 at this time, enter S52; otherwise, enter S6;

[0028] S52: Delete the earliest recorded accuracy of the training set. If the difference between the maximum value and the minimum value in the record is less than 1% at this time, record the current epoch number as epst, and the last epoch after 5 epochs as eped;

[0029] S53: If epst ≤ epoch < eped, increase the learning rate corresponding to the SGDM part; otherwise, restore the learning rate to the initial value of the hyperparameters.

[0030] Preferably, the weight decay in S7 is to multiply the parameters in the deep neural network model by a number in the interval (0, 1) before updating, to prevent the parameters from diverging and causing the gradient to disappear.

[0031] Preferably, epoch is the number of times the training set is fully trained, and the initial value is 0.

[0032] Preferably, the test set is used to verify the effectiveness of the algorithm after each epoch ends.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] 1. By integrating the AdamW algorithm and the SGDM algorithm, the present invention provides a new solution for optimizing deep neural networks; at the same time, by using the method of separate learning rates, the present invention enables SGDM and AdamW to adopt different learning rate change strategies, improving the convergence speed in the early stage of updating and achieving better results in the late stage of updating.

[0035] 2. By using the method of reusing partial calculation results, while making the algorithm accuracy and convergence speed better, the present invention does not change the computational complexity; compared with AdamW, the time complexity of the present invention increases slightly, and the space complexity does not increase. During the training process, it does not occupy more computing resources and does not extend the training time.

[0036] 3. In view of the different characteristics of AdamW and SGDM, the present invention adopts different methods to avoid local optimality for the two parts of the algorithm, thereby improving the final convergence accuracy of training; for AdamW, which is more likely to fall into local optimality, the method of discarding part of the update amount is adopted to avoid the problem of difficulty in jumping out of the extreme point due to the small gradient in the later stage after entering the local optimal point; for SGDM, which is not prone to falling into local optimality, the method of dynamically adjusting the learning rate is adopted to jump out of the local optimal point.

[0037] 4. The present invention adopts learning rate adjustment instead of the original method of discarding part of the update amount. When the algorithm stops optimizing, the learning rate is increased to make the deep neural network jump out of the local optimum, thereby improving the final accuracy of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flowchart of a deep neural network optimization method that combines AdamW and SGDM provided by the present invention.

[0039] Figure 2 shows the performance of different optimization algorithms for training the MINIST dataset on ResNet-18.

[0040] Figure 3 shows the performance of different optimization algorithms when training the Cifar-10 dataset on ResNet-34.

[0041] Figure 4 shows the performance of different optimization algorithms when training the Cifar100 dataset on ResNet-50. DETAILED DESCRIPTION

[0042] The following is a further detailed description of the embodiments of the present invention in conjunction with the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are intended to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention. Those skilled in the art can combine the technical features in the embodiments to form new embodiments without creative effort, and these new embodiments are also covered by the scope of protection of the present invention.

[0043] The present invention is described in detail below by taking the training of CIfar-10 dataset on ResNet-34 as an example and combining with the accompanying drawings.

[0044] The present invention provides a deep neural network optimization method that combines AdamW and SGDM. The basic idea is to combine AdamW and SGDM, two of the most advantageous deep neural networks currently, through a separable learning rate and a differentiated local optimal avoidance method, thereby improving the convergence speed and final accuracy of model training while ensuring that the required computing resources and training time do not increase significantly.

[0045] The present invention provides a deep neural network optimization method that combines AdamW and SGDM, which specifically includes the following steps:

[0046] Step 1: Process the original dataset, taking the Cifar-10 dataset as an example.

[0047] Step 1.1. Standardize the dataset. The present invention normalizes the three channels of each pixel according to formula (1), obtaining a result between [-1, 1]. x represents the value of a single channel at a single pixel in the image, and y represents the normalized result of this value. The mean and standard deviation of the RGB channels are [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225], respectively.

[0048]

[0049] Step 1.2: Convert the trainable data. Randomly shuffle the training set of 50,000 images and concatenate them into a 32 × 32 × 3 × 50,000 data tensor. Unshuffle the test set of 20,000 images and concatenate them into a 32 × 32 × 3 × 20,000 data tensor.

[0050] Step 2: Initialize the model, including randomly initializing the deep neural network parameters and accepting predefined hyperparameters.

[0051] Step 2.1: Change the input channel of the ResNet-34 model to three channels. The present invention expands the parameters of the deep neural network model into a one-dimensional tensor P = {p0, p1, p2, ... p s}.

[0052] Step 2.2: Accept predefined hyperparameters. The optimization algorithm described in this invention includes the following hyperparameters: the total number of training epochs, epmax; the learning rates lr1 and lr2 for the two components of the algorithm; the parameter dropout, which controls the proportion of updates discarded by AdamW; the parameters β1 and β2, which control the rate at which gradients and gradient squared are accumulated; the weight decay parameter wd; and the random seed rand, which controls model initialization.

[0053] Step 2.3. The present invention adopts a parameter random initialization method as shown in formula (2), where i is the index of the deep neural network parameter p.

[0054]

[0055] Where rand(-1,1) is a random number between the interval [-1,1] generated according to the fixed random seed rand.

[0056] Step 2.4: Initialize the cumulative amount during the update process. Initialize epoch to 0, initialize the total number of optimization times t to 1, and initialize the cumulative amount of gradient and gradient square Initialize the training set accuracy record Acc = {acc0, acc1, acc2, acc3, acc4}, and adjust the learning rate starting and ending points epst = epmax, eped = 0.

[0057] Step 3: Input an untrained data in the dataset into the deep neural network model and obtain the gradient value of the error function as G t ={g t0 ,g t1 ,g t2 …g ts}.

[0058] Step 4: Calculate the cumulative gradient and the square of the gradient.

[0059] Step 4.1: The accumulation of gradients is given by formula (3), where j is the index of the gradient accumulation m in the current training.

[0060]

[0061] Step 4.2: The accumulation of squared gradients is given by equation (4), where k is the index of the squared gradient accumulation v in the current training.

[0062]

[0063] Step 5: Calculate the update amount for the AdamW part. The weight decay part of AdamW and the weight decay part of SGDM are integrated in the subsequent steps, so Step 5 only consists of two steps: calculating the update amount of AdamW and discarding part of the update amount.

[0064] Step 5.1: Calculate the adaptive learning rate, as shown in formula (5), and calculate the adjustment factor lr to adjust the update step size. a .

[0065]

[0066] Step 5.2: According to the total number of optimizations t, the gradient and the accumulation of the square of the gradient are adaptively scaled according to formula (6).

[0067]

[0068] in and Same dimensions as before scaling.

[0069] Step 5.3: Combine the results of step 5.1 and step 5.2 using formula (7) to obtain the basic update amount A of AdamWt ={a t0 ,a t1 ,a t2 …a ts}, where k is the index of parameter a.

[0070]

[0071] Step 5.4: According to the dropout in the hyperparameter, some of the results in step 5.3 are set to 0 through a series of random numbers generated by formula (8), where F represents the probability of the event occurring.

[0072]

[0073] In formula (9), the basic update amount of AdamW is multiplied by the above random number sequence in sequence to obtain the final update amount of AdamW, where l represents the index of the parameter.

[0074]

[0075] Step 6: Calculate the update amount of the SGDM part, including calculating the update amount of SGDM itself and adjusting the learning rate.

[0076] Step 6.1: Construct the update amount of SGDM through formula (10).

[0077] S t =M t (10)

[0078] Step 6.2, if t<6, go to step 7, otherwise if acc max <acc min +1, where acc max is the maximum value in ACC, acc min is the minimum value in ACC, then proceed to step 6.3.

[0079] Step 6.3: Record the current epoch as the starting point epst of the learning rate adjustment, and the ending point is eped = epst + 5. The adjusted learning rate is given by formula (11).

[0080]

[0081] Step 7: Integrate the results obtained in the above steps to obtain the final parameter update amount update, and apply it to parameter update.

[0082] Step 7.1: Combine AdamW and SGDM through their learning rates, as shown in formula (12).

[0083]

[0084] Thus, the update of the deep neural network parameters after weight decay is shown in Equation (13).

[0085]

[0086] Step 8: End the current optimization, and increment t by 1. If a complete dataset training is completed, increment epoch by 1.

[0087] Step 9: If epoch < epmax, execute Step 3; otherwise, end the training.

[0088] To verify the effectiveness of a deep neural network optimization algorithm that combines the advantages of AdamW and SGDM proposed in this invention, the present invention uses the deep learning framework PyTorch to build the model and the optimization algorithm, and compares with the two currently best-performing optimization algorithms, AdamW and SGDM, for three classic image classification datasets, MINIST, Cifar-10, and Cifar-100.

[0089] For the three datasets with different characteristics, ResNet models with different depths are used according to the size of the dataset. In each experiment, the default hyperparameters in PyTorch are used to train for 200 epochs respectively, and the results are shown in Figures 2, 3, and 4.

[0090] As can be seen from Figures 2, 3, and 4, for datasets of different scales and neural networks of different depths, the algorithm described in the present invention has better effects. It has a higher convergence rate in the initial stage of training and a higher accuracy in the final stage of training, and the experiment demonstrates the effectiveness of the algorithm. In summary, the above is only a preferred embodiment of the present invention and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A deep neural network optimization method combining AdamW and SGDM, characterized in that: It includes the following steps: S1: Build a deep neural network model according to the requirements of the dataset, randomly initialize the parameters in the deep neural network model, divide the dataset into two parts, a training set and a test set, and the dataset is image data; S2: Calculate the gradient of the deep neural network model corresponding to the current training data; S3: Calculate the update amounts of the AdamW and SGDM parts in the deep neural network optimization algorithm according to the gradient of the current training data; S4: Discard part of the update amount of AdamW in the deep neural network optimization algorithm; S5: If the deep neural network stops optimizing, adjust the learning rate of the SGDM part; S6: Add the processed AdamW and SGDM parts in S3 and S4 after multiplying them by their respective learning rates to obtain the update amount of each parameter; S7: Complete the weight decay part and update each parameter in the deep neural network according to the update amount in S6, and increment the iteration count by 1; if the training of all data in the training set has been completed, increment the epoch by 1; S8: When the epoch reaches the set value, output the current model parameters and end this process; otherwise, return to S2; The S1 includes the following sub-steps: S11: Read the preset hyperparameters and random seeds; S12: Randomly initialize the parameters of the deep neural network model according to the random seed according to the normal distribution with a mean of 0 and a variance of 1; The S3 includes the following sub-steps: S31: Calculate the cumulative sum of gradients and the cumulative sum of gradient squares; S32: Calculate the adaptive learning rate part of AdamW; S33: Multiply the cumulative sum of gradients, the reciprocal of the cumulative sum of gradient squares, and the adaptive learning rate to form the update amount of AdamW, and at the same time, the cumulative sum of gradients also forms the update amount of SGDM; S4 includes the following sub-steps: S41: Generate a sequence that conforms to the Bernoulli distribution and has the same total number of parameters as the deep learning model according to the hyperparameters; S42: Multiply the update amount of AdamW by the numbers in the sequence in turn to obtain the processed update amount of AdamW; S5 includes the following sub-steps: S51: After the last training of each epoch ends, record the accuracy of the current training set. If the number of recorded accuracies reaches 5 at this time, enter S52, otherwise enter S6; S52: Delete the earliest recorded accuracy of the training set. If the difference between the maximum value and the minimum value in the record is less than 1% at this time, record the current epoch number as epst, and the last epoch after 5 epochs as eped; 2. The deep neural network optimization method combining AdamW and SGDM according to claim 1, characterized in that S53: If epst ≤ epoch < eped, increase the learning rate corresponding to the SGDM part; otherwise, restore the learning rate to the initial value of the hyperparameters.

3. The deep neural network optimization method combining AdamW and SGDM according to claim 1, characterized in that The weight decay in S7 is to multiply the parameters in the deep neural network model by a number in the interval (0, 1) before updating, to prevent the parameters from diverging and causing the gradient to disappear.

4. The deep neural network optimization method combining AdamW and SGDM according to claim 1, characterized in that The epoch is the number of times the training set is fully trained, and the initial value is 0. The test set is used to verify the effectiveness of the algorithm after each epoch ends.

Images