Adaptive optimization training method for convolutional neural network for image classification

Abstract

The application discloses an adaptive optimization training method of a convolutional neural network for image classification, has a faster convergence speed, and a model obtained through training has higher accuracy and stronger generalization capability. A convolutional neural network model M is constructed. A sample B is randomly extracted from a training image set and input to the M, and a loss function gradient of a model parameter is calculated according to an output result; and unbiased second moment estimation of the gradient and a gradient difference is calculated. The unbiased second moment estimation of the gradient and the gradient difference is used to construct a parameter update expression. If the number of convolutional layers in the M is not more than 5, the parameter update expression is directly used to update the parameters of the M, otherwise, the influence of a proportion item of the current gradient is added to the parameter update expression to construct a second parameter update expression, and the parameters of the M are updated. After the parameters are updated, the current training period is simultaneously updated, and the training is repeated until a trained convolutional neural network model is obtained, which is used for image classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an adaptive optimization training method for a convolutional neural network for image classification. BACKGROUND

[0002] With the rapid rise and vigorous development of artificial intelligence technology, its application breadth and depth are also constantly improving. As one of the branches, the deep learning technology based on deep neural network model has become one of the hotspots in the field of artificial intelligence research due to its good performance. Compared with the traditional machine learning method, the deep learning has the ability to automatically extract features from data, so as to better obtain the required key information from high-dimensional and large-volume data, and therefore has a wide application in natural language processing, computer vision, automatic speech recognition and automatic driving, etc. and has achieved great success.

[0003] Deep neural networks have been widely applied, but there are still problems of slow training speed and high computational cost. For some complex deep neural network learning tasks, even if expensive and high-performance graphics processors are used, it may take a long time to learn, often several days or even longer. Therefore, people need to explore methods to improve the learning speed of deep neural networks while maintaining performance. From another perspective, using better optimization methods can also help limited hardware devices save a lot of time and memory, making deep neural networks have more extensive applications. Therefore, people need a more effective deep neural network optimization method to update network parameters and achieve better convergence. Duchi et al. proposed an adaptive gradient-based optimization algorithm called AdaGrad in the literature (Adaptive Subgradient Methods for Online Learning and Stochastic Optimization, JMLR, 2011). This method uses the sum of the squares of the historical gradient size as a correction coefficient, so that each parameter can adaptively adjust its learning rate, which can assign a larger learning rate to sparse parameters to update them substantially, and assign a smaller learning rate to dense parameters to update them slightly. On this basis, Kingma et al. proposed an adaptive optimization algorithm based on gradient moment estimation called Adam in the literature (Adam: A Method for Stochastic Optimization, Computer Science, 2014). This method calculates the adaptive learning rate in the parameter update process by recording the first and second moment estimates of the historical gradient, so that the algorithm has the advantages of fast convergence speed, good optimization effect, wide application range and high computational efficiency. Wang et al. applied the classic proportional-integral-derivative (PID) method in the field of automatic control to the design of optimization algorithms in the literature (A PID Controller Approach for Stochastic Optimization of Deep Networks, CVPR, 2018), and proposed a PID optimization algorithm. This method adds gradient difference information to the parameter update to alleviate the overshoot problem caused by the momentum term in the training process, thereby accelerating convergence.

[0004] The above-mentioned optimization algorithms only use a single technology of adaptive learning rate or PID control idea, and cannot well solve the problems of slow algorithm convergence speed, severe oscillation and poor global optimization effect in deep learning.

[0005] Therefore, it is necessary to design an adaptive convolutional neural network optimization method to combine the characteristics of adaptive learning rate and the advantages of reducing overshoot, accelerate the convergence speed of the algorithm, improve the accuracy of the trained model, and enhance the generalization ability, so as to save the calculation resources and training time of the deep learning training process under the premise of ensuring the effect. SUMMARY

[0006] Therefore, the present application provides an image classification method based on adaptive optimization of convolutional neural network, which has faster convergence speed, and the trained model has higher accuracy and stronger generalization ability.

[0007] To achieve the above purpose, the technical scheme of the present application includes the following steps:

[0008] Step 1, prepare a training image set; construct a convolutional neural network model M, and randomly initialize the parameters in M.

[0009] Step 2, randomly extract a sample B from the training image set, input B into M to obtain the output result of M, calculate the loss function according to the output result of M, and the gradient of the loss function to the model parameters; and calculate the unbiased second moment estimation of the gradient and the gradient difference.

[0010] Step 3, use the obtained unbiased second moment estimation of the gradient and the gradient difference to construct a parameter update expression.

[0011] If the number of convolutional layers in M is not more than 5, directly use the parameter update expression to update the parameters of M.

[0012] If the number of convolutional layers in M is more than 5, add the influence of the proportion term of the current gradient in the parameter update expression to construct a second parameter update expression, and update the parameters of M.

[0013] After updating the parameters, the current training period is updated at the same time, and step 2 is returned to repeat steps 2 to 3 until convergence or the maximum training period is reached, and a trained convolutional neural network model is obtained.

[0014] Step 4, use the trained convolutional neural network model to classify the image set to be classified.

[0015] Further, the parameters in M are randomly initialized, specifically:

[0016] The initial value of the parameters of M is randomly initialized as θ0, the internal parameters of the initialization algorithm are initialized, including the initial value of the second moment estimation of the gradient v0=0, the initial value of the second moment estimation of the gradient difference d0=0, the initial value of the gradient accumulation I0=0, the initial value of the gradient difference accumulation D0=0, and the initial value of the training period t=0;

[0017] Step 1 also includes setting the hyperparameters of the model training algorithm, including the original learning rate r, the momentum factor a, the exponential decay coefficient b, the gain coefficient K of the gradient part i , the gain coefficient K of the gradient difference part d , and the maximum training period epochs.

[0018] Further, the unbiased second moment estimation of the gradient and the gradient difference is calculated, specifically:

[0019] The gradient g t of the loss function f(x) with respect to the deep neural network model parameter 9 in the extracted sample is calculated.

[0020] The accumulated item of the gradient part is calculated, and the accumulation is in the form of a traditional momentum item, that is, I t = a · I t-1 + g t ; I t and I t-1 are the gradient accumulation items of the current training period t and the last training period t-1;

[0021] The accumulated item of the gradient difference part is calculated using the weighted moving average method, that is, D t = a · D t-1 + (1-a) (g t -g t-1 ); D t and D t-1 are the gradient difference accumulation items of the current training period t and the last training period t-1;

[0022] Where a is the momentum factor, and the value range is (0, 1).

[0023] The biased second moment estimation v t of the gradient is calculated, that is, v t-1 = b · v t + (1-b) g 2 ; v t and v t-1 are the biased second moment estimations of the gradient of the current training period t and the last training period t-1;

[0024] The biased second moment estimation d t of the gradient difference is calculated, that is, d t-1 = b · d t + (1-b) (g t-1 -g 2 ); d t and d t-1 are the biased second moment estimations of the gradient difference of the current training period t and the last training period t-1;

[0025] Where b is the exponential decay coefficient, and the value range is (0, 1).

[0026] The bias correction processing is performed on the biased second moment estimation of the gradient and the gradient difference, so as to obtain the unbiased second moment estimation of the gradient and the gradient difference, i.e.

[0027] is the unbiased second moment estimation of the gradient. is the unbiased second moment estimation of the gradient difference.

[0028] Further, the parameter update expression is constructed using the obtained unbiased second moment estimation of the gradient and the gradient difference, specifically:

[0029]

[0030] θ t+1 is the parameter of the next training period t+1 after the update; θ t is the parameter of the current training period t; σ is a constant, and the value is between 10 -7 ~ 10 -9 .

[0031] The learning rate r therein is attenuated using the index-based learning rate attenuation method.

[0032] The training loss in the model training process is taken as the index: let the waiting time be 5, the threshold be 0.02, and the attenuation rate be 0.5, so when the reduction value of the training loss within 5 training periods is not more than 0.02, the learning rate r is reduced to half of the original.

[0033] Further, the influence of the proportional term of the current gradient is added in the parameter update expression, the second parameter update expression is constructed, and the parameters of M are updated, specifically:

[0034]

[0035] θ t+1 is the parameter of the next training period t+1 after the update; θ t is the parameter of the current training period t; σ is a constant, and the value is between 10 -7 ~ 10 -9 .

[0036] The learning rate r therein is attenuated using the time table-based learning rate attenuation method, and let the maximum training period be t max , then the learning rate time table is designed as:

[0037] The training period is [0, 0.5t max , and the learning rate is r;

[0038] The training period is [0.5t max , 0.75tmax ), learning rate is 0.5r;

[0039] Training period is [0.75t max ,t max ), learning rate is 0.25r.

[0040] Beneficial effects:

[0041] 1. The image classification method based on the convolutional neural network of adaptive optimization is provided, the use of the gradient difference by the simple PID optimizer is reserved, the adaptive learning rate is designed by using the second moment estimation of the gradient difference, the algorithm can have a faster convergence speed, and the model obtained by training has higher accuracy and stronger generalization ability.

[0042] 2. The image classification method based on the convolutional neural network of adaptive optimization is designed, the PID control idea and the design idea of adaptive learning rate are combined for the first time, the characteristics of reducing oscillation and relieving overshoot of the PID optimizer are reserved, and the advantages of flexible step and fast convergence of the adaptive learning rate type optimization algorithm are combined, the training effect of the algorithm on the deep neural network is effectively improved, the network model with higher accuracy and stronger generalization ability can be trained faster, and it has important significance for fully utilizing the computing resources and expanding the application scenarios of deep learning.

[0043] 3. The image classification method based on the convolutional neural network of adaptive optimization is provided, in order to improve the effect of the optimization method in actual use, when the learning rate is selected, in addition to using the fixed original learning rate r, the method also designs the decay learning rate skill which is beneficial to improving the algorithm effect for two update modes on this basis, mainly including two kinds: index decay and time table decay. The index decay refers to that if a selected index does not change obviously (such as training loss or test loss) within a certain training period interval, the learning rate r is reduced by a preset proportion. And the time table decay refers to that when the training period reaches a preset time point, the learning rate r will be reduced by a preset proportion. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is an application structure schematic diagram of the image classification method based on the convolutional neural network of adaptive optimization provided by the application;

[0045] Figure 2 It is a specific flowchart diagram of the image classification method based on the convolutional neural network of adaptive optimization provided by the application. DETAILED DESCRIPTION

[0046] The application will be described in detail below in combination with the drawings and examples.

[0047] The application provides an image classification method based on adaptive optimization of a convolutional neural network, which uses gradient difference in an optimization algorithm, and simultaneously designs an adaptive learning rate based on gradient and gradient difference second moment estimation, so that the optimization algorithm can reduce oscillation, accelerate convergence, and has strong global optimization capability.

[0048] Considering that, in the optimization process of deep learning, the optimization algorithm based on stochastic gradient descent usually has the problems of slow convergence speed and serious overshoot due to a large problem size. Although the simple PID optimizer can alleviate the overshoot problem and improve the optimization performance to a certain extent by using the gradient difference, it only considers the correction of the gradient direction and does not consider the design of an adaptive learning rate.

[0049] The optimization algorithm for deep learning in the application refers to the search strategy or rule used in the process of finding the network parameters that minimize the error function value of the deep neural network. Specifically, in the training process of the deep neural network, there are usually two parts of forward calculation and error backpropagation. The optimization algorithm for deep learning referred to in the application refers to the strategy or rule for modifying the parameters in the network according to the error during the backpropagation of the error.

[0050] As shown in Figure 1 , the embodiment of the application provides an adaptive optimization training method for a convolutional neural network used for image classification, which includes the following steps:

[0051] Step 1, preparing a training data set D and a deep neural network model M to be trained. Randomly initializing the neural network model parameters as θ0, initializing the algorithm internal parameters, including the gradient second moment estimation v0=0, the gradient difference second moment estimation d0=0, the gradient accumulation value I0=0, the gradient difference accumulation value D0=0, and the training period t=0. Setting the algorithm hyperparameters, including the original learning rate r, the momentum factor α, the exponential decay coefficient β, the gain coefficient K i of the gradient part, the gain coefficient K d of the gradient difference part, and the maximum training period epochs.

[0052] Step 2, if the training period t reaches the maximum training period, the training is ended. Otherwise, randomly extracting a small batch of samples B from the training data set D, inputting the sample B into the neural network model M to obtain the output result of the model. Calculating the loss function Loss corresponding to the model according to the model output result and the sample label, usually using the cross-entropy function where y is the label value, and the model output value. Then calculating the gradient of the loss function with respect to the model parameters θ The gradient obtained by calculating the small batch of samples is used to replace the full sample gradient for subsequent calculation.

[0053] The accumulated item of the gradient part is calculated by the form of the traditional momentum item, that is, I t = a I t-1 + g t , so that the algorithm can maintain a downward trend and help the algorithm jump out of the local optimum. The accumulated item of the gradient difference part is calculated by using the weighted moving average method, that is, D t = a D t-1 + (1-a) (g t -g t-1 ), which avoids the problem of learning fatigue of the algorithm in the later iteration period and also alleviates the influence of noise in the small sample era. Wherein a is the momentum factor, the value of which represents the influence degree of the past information on the current parameter update, generally a = 0.9.

[0054] The biased second moment estimation of the gradient is calculated: v t = beta v t-1 + (1-beta) g t 2 , and the biased second moment estimation of the gradient difference is calculated: d t = beta d t-1 + (1-beta) (g t -g t-1 ) 2 . Wherein beta is the exponential decay coefficient, generally beta = 0.99. The second moment estimation of the gradient and the gradient difference calculated in this step is the core value in the process of designing the adaptive learning rate of the present application.

[0055] After obtaining the moment estimation value obtained by the exponential weighted average, the corresponding bias correction processing is also required, so as to obtain more accurate unbiased second moment estimation of the gradient and the gradient difference, that is Through bias correction, the algorithm can also obtain more accurate moment estimation value in the early training period, so as to speed up the convergence speed of the algorithm in the early iteration period. The second unbiased moment estimation of the gradient and the gradient difference in this step will be directly used to construct the adaptive learning rate in the parameter update process.

[0056] Step 3, taking image processing which is the most common in deep learning tasks as an example, a convolutional neural network is usually used to realize image classification. For the neural network with the number of convolutional layers not more than 5, the present application uses the unbiased second moment estimation of the gradient and the gradient difference to design the adaptive learning rate to update the parameters of the deep neural network: is a constant, the value of which is in the range of 10 -7 ~ 10 -9between the training loss and the threshold value. The learning rate r is reduced to half of the original value when the training loss is reduced by less than the threshold value within the waiting period.

[0057] For a neural network with more than 5 convolutional layers or a more complex structure (such as a residual neural network or other modern neural network structures), in addition to the gradient partial accumulation term I t and the gradient difference part D t , the influence of the proportion of the current gradient is also added to the parameter update expression: to make the entire network converge faster. Meanwhile, a learning rate decay method based on a time table is used. Let t max be the maximum training period. The learning rate time table is shown in the following table.

[0058] Training period [[0, 0.5t max )]]> [[0.5t max ,0.75t max )]]> [[0.75t max ,t max )]]> Learning rate r 0.5r 0.25r

[0059] After updating the parameters, the current training period is updated at the same time, and steps 2 to 6 are repeated until the algorithm converges or the maximum training period is reached.

[0060] Since the model parameters are updated in the direction that reduces the loss function Loss, the neural network model will have more accurate results after parameter updating. After each parameter update, the training period t = t + 1, and steps 2 to 6 are repeated until the algorithm exits, i.e., the training period reaches the maximum training period limit.

[0061] Step 4: Using the trained convolutional neural network model to classify the image set to be classified.

[0062] Further, in order to improve the effect of the optimization method in actual use, in addition to using the fixed original learning rate r, the method also designs decay learning rate techniques that are beneficial to improving the algorithm effect for the two update methods, mainly including two kinds: index decay and time table decay. Index decay refers to reducing the learning rate r by a predetermined proportion if a selected index (such as the training loss or test loss) does not change significantly within a certain training period interval. Time table decay refers to reducing the learning rate r by a predetermined proportion when the training period reaches a predetermined time point.

[0063] Figure 2 The flowchart of the adaptive optimization training method of the convolutional neural network for image classification provided by the present application.

[0064] The application provides an instance of an image classification method based on an adaptive optimization convolutional neural network for training a convolutional neural network on a PetImage data set. The convolutional neural network is a kind of artificial neural network with a deep structure, and its main feature is that convolution calculation is used in the feature extraction process. At present, it has been widely used in the field of computer vision such as image recognition and classification. The PetImage data set is the data set of the big data competition of Kaggle platform in 2013, which contains the size of the RGB color image of cat and dog. The training set contains 12500 ordered divided images of cat and dog, while the test set contains 12500 unordered mixed images of cat and dog. The convolutional neural network used in the instance is a relatively complex residual neural network ResNet50. The hyperparameter setting of the adaptive PID optimizer is learning rate r = 0.0006, gain coefficient K i = 0.4 of the gradient part, gain coefficient K d = 1.0 of the gradient difference part, batch sample quantity batchsize = 256 each time, and maximum training period epochs = 100. The adaptive PID optimization method provided by the application is compared with other three optimization methods (Adam, PID, SGD-M) in effect, and the image classification method based on the adaptive optimization convolutional neural network provided by the application has certain advantages in the accuracy and generalization ability of the trained model.

[0065] In summary, the above is only a preferred embodiment of the application, and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method for adaptive optimization training of a convolutional neural network for image classification, characterized in that, The method comprises the following steps: Step 1, prepare a training image set; construct a convolutional neural network model M , the parameters in the M are randomly initialized; Step 2: Randomly select samples from the training image set B ,Will B Input to M In the middle, we get M The output result, based on M The output results are used to calculate the loss function and the gradient of the loss function with respect to the model parameters; and to calculate the unbiased second moment estimates of the gradient and gradient difference. Step 3, using the obtained gradient and the unbiased second moment estimation of the gradient difference to construct a parameter update expression, which is: ； wherein, is the updated next training cycle t convolutional neural network model parameter; is the current training cycle t parameter; is a constant with a value between 10 -7 ~ 10 -9 ; and are unbiased second moment estimations of the gradient and the gradient difference, respectively; and are the gradient accumulation term and the gradient difference accumulation term in the unbiased second moment estimations for the current training cycle t ; and are the gain coefficients for the gradient part and the gradient difference part in the model training algorithm, respectively; is the learning rate in the model training algorithm; If M the number of convolution layers in the middle is no more than 5, the parameter updating expression is directly used to update the parameters of M . If M the number of convolution layers exceeds 5, the influence of the proportion term of the current gradient is added in the parameter update expression, a second parameter update expression is constructed, and the parameters of the M are updated. After updating the parameters, the current training period is updated at the same time, and step 2 is returned to repeat steps 2 to 3 until convergence or the maximum training period is reached, and a trained convolutional neural network model is obtained; Step 4, using the trained convolutional neural network model to classify the image set to be classified.

2. The self-adaptive optimization training method of claim 1, wherein, The pair M The parameters in the file are randomly initialized, specifically as follows: Randomly initialize M the parameter initial values of , initialize the algorithm internal parameters, including the gradient second moment estimation initial value , the gradient difference second moment estimation initial value , the gradient accumulation value initial value , the gradient difference accumulation value initial value , and the training period initial value ; The step 1 further includes setting hyperparameters of the model training algorithm, including an initial learning rate , a momentum factor , an exponential decay coefficient , a gain coefficient for the gradient part , a gain coefficient for the gradient difference part , and a maximum training period .

3. The self-adaptive optimization training method of claim 1, wherein, The calculation of the gradient and the unbiased second moment estimation of the gradient difference is specifically: Computing loss function in extraction sample For gradient of convolutional neural network model parameters ;​ The accumulated terms of the gradient part are computed, accumulated in the form of a conventional momentum term, i.e. ; and are the gradient accumulated terms of the current training cycle t and the previous training cycle t -1. The accumulated term of the gradient difference part is calculated using the weighted moving average method, that is ; and the gradient difference accumulated term of the current training period t and the last training period t -1; wherein is a momentum factor, having a value in the range of ; a biased second moment estimate of the gradient ; and for the current training epoch t and the previous training epoch t -1; a biased second moment estimate of the gradient difference ; and for the current training cycle t and the previous training cycle t -1; wherein is an exponential decay coefficient, having a value in the range of ; The bias correction is performed on the biased second moment estimation of the gradient and the gradient difference, so as to obtain the unbiased second moment estimation of the gradient and the gradient difference, i.e. ; is an unbiased estimate of the gradient of the second moment; is an unbiased estimate of the gradient difference of the second moment.

4. The self-adaptive optimization training method of claim 3, wherein, the learning rate is decayed using an indicator-based learning rate decay method; Taking the training loss in the model training process as an index: setting the waiting time as 5, the threshold as 0.02, and the decay rate as 0.5, when the reduction of the training loss in 5 training periods is not more than 0.02, the learning rate is reduced to half of the original one. reduction to half of the original one.

5. The self-adaptive optimization training method of claim 3 or 4, wherein, The influence of incorporating the scaling term of the current gradient into the parameter update expression is used to construct a second parameter update expression. M The parameters are updated as follows: for the next training cycle t parameters of +1; for the current training cycle t parameters; is a constant with value in the interval 10 -7 ~10 -9 ; For the learning rate in it is decayed using a schedule-based learning rate decay method, where the maximum training epoch is denoted as The learning rate schedule is designed as: The training period is , and the learning rate is ; The training period is , and the learning rate is ; The training period is , and the learning rate is .

Images