Large-scale image classification method based on deep learning optimizer

Abstract

The present invention discloses a large-scale image classification method based on a deep learning optimizer. By constructing a new deep learning optimizer, a deep neural network model is trained using image training data and the constructed deep learning optimizer to obtain a trained image classification deep neural network model; then a color image to be classified containing predefined categories is input into the trained image classification deep neural network model, and the output of the model is the category label of the image, thereby realizing large-scale image classification. The use of the method of the present invention for large-scale image classification can avoid the additional calculation and memory overhead caused by calculating the gradient at the extrapolation point, has great tolerance to the batch size of the data volume, and can improve the efficiency and image classification accuracy of large-scale image classification.

Description

Technical Field

[0001] The present invention belongs to the field of pattern recognition, machine learning and artificial intelligence technology, and relates to an image classification optimization method, and in particular to a large-scale image classification method based on a deep learning optimizer (Adan). Background Art

[0002] Today, deep learning (DL) technology has achieved effective results in a wide range of artificial intelligence applications, such as large-scale computer vision tasks (e.g., image classification, object detection and segmentation), natural language processing (e.g., language modeling and machine translation). The scale of neural networks and data sets is one of the main reasons why DL shows strong performance. Larger data sets and neural networks often show better performance on most tasks, but the price to pay is longer training time and larger computational workload. Therefore, in recent years, there has been a surge in demand from academia and industry for distributed DL training on large cluster devices (such as TPUs and GPUs). Data parallelism has become the main implementation method for distributed training. It can distribute large batches of data to multiple or even thousands of GPUs, each of which has a copy of the same network parameters, and each GPU calculates the gradients generated by the obtained data. Finally, the gradients on all GPUs are collected for synchronous updates at each iteration.

[0003] Based on this high-parallel mode, the Adam optimizer (see formula 0-2) is generally used as the default neural network training optimizer by default. However, this large-scale image classification method based on Adam training generally has an unavoidable problem: when using a small batch size to train the network, it often takes too long. However, when using a large batch size for training, when the batch size exceeds a certain value (such as 4k), the model accuracy will drop significantly, that is, blindly increasing the batch size will usually lead to a decrease in generalization performance and reduce computational efficiency. This is mainly because: 1) As the batch size increases, the number of neural network parameter updates will decrease sharply, and the DL optimizer with a slow convergence speed cannot obtain a satisfactory convergence result under a limited number of updates; 2) A large batch of data will have a certain implicit influence on the optimizer, causing it to converge to a relatively sharp local minimum, thereby damaging the generalization performance of the network.

[0004] At present, for large-scale image classification problems, there is still a lack of deep learning optimizers that can be efficient and stable in the case of large batch data volumes while ensuring accuracy. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a deep learning optimizer that can be used for large-scale image classification tasks, and is named Adan (Adaptive Nesterov Momentum Algorithm). For large-scale image classification tasks, Adan can simultaneously use only half the computational effort on large-scale image classification tasks based on deep networks (convolutional networks or Transformers) to achieve improved classification accuracy.

[0006] The core of the present invention is to design a deep learning optimizer and use it for large-scale image classification tasks, including: firstly, improving the traditional Nesterov acceleration algorithm, and proposing a new momentum estimation method, which can avoid the additional calculation and memory overhead caused by calculating the gradient at the extrapolation point in the original method; then, by combining the new momentum estimation method with the adaptive gradient method in the deep optimizer, a new deep learning optimizer Adan is designed. Using the deep learning optimizer training algorithm provided by the present invention, a large-scale image classification method based on Adan can be realized.

[0007] In the method of the present invention, the input of the deep learning optimizer is a color image containing predefined categories as the main body, and the output is the category label of the image. Due to the learnability of the method of the present invention, the input image can be a single image or a batch image. Specific experiments show that on the benchmark data set, the optimizer designed by the present invention achieves more excellent image classification performance at a lower computational cost than the optimizer in the existing image classification method.

[0008] The technical solution provided by the present invention is:

[0009] A large-scale image classification method based on a deep learning optimizer (Adan) is proposed. A new deep learning optimizer Adan is constructed. A deep neural network model is trained using image training data and the constructed deep learning optimizer Adan to obtain a trained image classification deep neural network model. A color image containing predefined categories to be classified is input into the trained image classification deep neural network model to output the category label of the image, thereby realizing large-scale image classification. The method comprises the following steps:

[0010] 1) Perform image preprocessing to obtain training image data and image data to be classified;

[0011] 2) Build a new deep learning optimizer Adan, the input of which is a single or batch color image, and the output is the category label of the image. It includes the following processes:

[0012] 21) Improve the Nesterov acceleration algorithm and design a new momentum estimation method. The new momentum estimation method can avoid the extra computational and memory overhead caused by calculating the gradient at the extrapolation point in the original method.

[0013] The momentum estimation method based on the Nesterov acceleration algorithm is expressed as formula 1:

[0014]

[0015] Among them, g k and g k-1 is the stochastic gradient estimate of the kth and k-1th iterations, i.e., the stochastic gradient estimate of the color image batch data, ξ k is random noise with mean 0, m k is the first-order momentum of the algorithm's kth iteration. η>0 is the basic learning rate, and β1 is the accumulation coefficient of the first-order momentum, which takes a value in the interval 0-1.

[0016] 22) A new deep learning optimizer Adan is constructed by using a new momentum estimation method and the adaptive gradient method in the deep optimizer;

[0017] The improved momentum estimation method given in Formula 1 is used for the first-order and second-order momentum in the adaptive gradient update method in the deep learning optimizer, and a separate accumulation coefficient β2 is introduced for the gradient difference term to further establish the deep learning optimizer Adan, which is expressed as Formula 3:

[0018]

[0019] Among them, m k , n k are the first-order momentum and second-order momentum of the kth iteration, v k is the stochastic gradient difference momentum of the color image batch data at the kth iteration, η k is the generalized learning rate at the kth iteration, θ k is the network parameter for the kth iteration; the formula in the last line is used for parameter update. Here, the square of the vector refers to the square of each dimension, where ο is the Hadamard inner product. ε>0 is the smoothness parameter, β1, β2 and β3 are the accumulation coefficients of the first-order momentum, gradient difference momentum and second-order momentum, respectively, and belong to the interval 0-1.

[0020] 23) Further, weight decay is performed through the proximal operator to obtain the deep learning optimizer Adan with weight decay;

[0021] A unique weight decay strategy is introduced into the deep learning optimizer Adan obtained in step 22), and a parameter update formula with weight decay is obtained, which is expressed as Formula 6:

[0022]

[0023] The symbols are defined the same as in Formula 3, and λ is the coefficient of weight decay.

[0024] This results in the Adan optimizer with weight decay, expressed as Equation 7:

[0025]

[0026] The definitions of the symbols are the same as those in Formula 3.

[0027] 3) Using the training image data and the constructed deep learning optimizer Adan, the deep neural network model is trained to obtain a trained image classification deep neural network model; the training image data (color image) is input into the deep learning optimizer, and the output is the category label of the image.

[0028] Adan or Adan with weight decay can be used to train a given deep neural network model (which may include two network structures, ResNet-50 and ResNet-101). When the large-scale image classification method of the present invention is implemented, a stochastic depth mechanism is set for network model training; the loss function is set to cross entropy loss. When Adan with weight decay is used for training, the weight decay coefficient is selected as λ=0.02, and the basic learning rate η=1.5e-2; and the cosine learning rate decay method is used to decay the size of η during training. The batch size is 4096. The number of warmup epochs is selected as 60.

[0029] 4) Input the color image containing predefined categories to be classified into the trained image classification deep neural network model, and then output the category label of the image to achieve large-scale image classification.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) The optimizer Adan of the present invention shows great tolerance to the batch size of data, and generally a larger batch size. Therefore, the image classification method using the Adan optimizer can use more batch data at a time, while ensuring training efficiency and training accuracy. Therefore, it is more suitable for large-scale image classification methods.

[0032] (2) The large-scale image classification method based on Adan proposed in the present invention has a faster training speed. In a unit of time, more training samples can be used than the model using a common optimizer.

[0033] (3) Adan only consumes one-fourth of the training cost (number of parameter updates) of the default optimizer in the original large-scale image classification task and achieves higher or comparable classification accuracy in the classification task. DETAILED DESCRIPTION

[0034] The present invention is further described below in conjunction with examples, but the scope of the present invention is not limited in any way.

[0035] The present invention proposes a large-scale image classification method based on a deep learning optimizer (Adan). First, the traditional Nesterov acceleration algorithm is improved, and a new equivalent momentum estimation method is proposed; a new degree learning optimizer Adan is obtained by combining the proposed momentum estimation method and the adaptive gradient method; Adan is used to train the deep network model. The trained deep network model is used to classify the input image.

[0036] Specifically, the present invention first improves the traditional Nesterov acceleration algorithm for large-scale image classification, and proposes a new momentum estimation method. The method is used for large-scale image classification, which can avoid the additional calculation and memory overhead caused by calculating the gradient at the extrapolation point, and can improve the efficiency and image classification accuracy of large-scale image classification. Then, by combining the new momentum estimation method with the adaptive gradient algorithm of the manifold in the current deep optimizer, we obtain the final updated form of Adan. In addition, it can be proved that Adan's convergence speed can match the currently known optimal convergence speed on non-convex random problems (such as deep network training problems).

[0037] A large number of experimental results show that in large-scale image classification tasks based on convolutional network architectures and Transformer architectures, networks trained with the Adan optimizer have achieved improved classification accuracy.

[0038] It is noted that Adan only consumes a quarter of the training cost (number of parameter updates) of the default optimizer in the original large-scale image classification task to achieve higher or comparable classification accuracy in the classification task. In addition, the large-scale image classification method based on Adan shows great tolerance to the batch size of the data volume, which is a feature not possessed by previous image classification methods.

[0039] The method of the present invention comprises:

[0040] 1. Design a new deep learning optimizer Adan.

[0041] 2. Using the image training set data, use Adan to train a given deep neural network model (such as ResNet-50, Vision Transformer) to obtain a trained deep neural network model.

[0042] 3. Input the image to be classified into the trained deep neural network model and output the category label of the image, thus realizing large-scale image classification based on the deep learning optimizer (Adan).

[0043] The present invention defines the following models and parameters:

[0044] (1) For the training objective of large-scale image classification problems using neural networks, the objective function is modeled as Equation 0-1:

[0045]

[0046] in, Indicates taking the minimum value of the optimization objective function F(θ); represents the expectation of the loss function f(·,·) with respect to the training data ζ, which is differentiable and possibly non-convex; the training data ζ is sampled from an arbitrary distribution θ is a learnable network parameter; ||·|| is the classic l2 norm; λ is the regularization term coefficient. Specifically, for large-scale image classification problems, f(·,·) uses the cross entropy loss function; the training data ζ uses color image data.

[0047] (2) The adaptive gradient update method in the deep learning optimizer is expressed as Equation 0-2:

[0048]

[0049] in, is the stochastic gradient estimate, i.e., the batch gradient, is the loss function f(·,·) on the network parameters θ k Taking the first-order differential, ξ k is random noise with mean 0; m k , n k are the first-order and second-order momentum of the k-th iteration respectively; θ k is the network parameter for the kth iteration. Here in is the Hadamard inner product. η>0 is the basic learning rate, ε>0 is the smoothing parameter, β1 and β3 are the accumulation coefficients of the first and second order momentum, which belong to the interval 0-1.

[0050] (3) The classic Nesterov acceleration algorithm is defined as Equation 0-3:

[0051]

[0052] Among them, g k is the stochastic gradient estimate, i.e., the batch gradient, ξ k is random noise with mean 0, is the first-order momentum of the algorithm at the kth iteration. η>0 is the basic learning rate, and β1 is the accumulation coefficient of the first-order momentum, which belongs to the interval 0-1.

[0053] The present invention provides a large-scale image classification method based on a deep learning optimizer (Adan). The traditional Nesterov acceleration algorithm is improved, and a new equivalent momentum estimation method is proposed; a new degree learning optimizer Adan is obtained by combining the proposed momentum estimation method and the adaptive gradient method in the degree optimizer; Adan is used to train the deep network model. The trained deep network model is used to classify the input image. The large-scale image classification method based on the deep learning optimizer (Adan) includes the following steps:

[0054] 1. A new deep learning optimizer Adan is proposed, which includes:

[0055] 1) Improved “momentum estimation method” based on Nesterov acceleration algorithm;

[0056] 2) Combining momentum estimation method with adaptive gradient method;

[0057] 3) Weight attenuation through proximal operators;

[0058] 2. Using the image training set data, Adan is used to train the given deep neural network model to obtain a trained network model.

[0059] 3. Input the image to be classified into the trained deep neural network model and output the category label of the image, thus realizing large-scale image classification based on the deep learning optimizer (Adan).

[0060] The following describes the specific process:

[0061] 1. Propose a new deep learning optimizer Adan, including steps 1)-3):

[0062] 1) Design an improved momentum estimation method based on the Nesterov acceleration algorithm;

[0063] From equation 0-3, the Nesterov acceleration algorithm at the extrapolation point θ k ′=θ k -(1-β1)(θ k -θ k-1) to calculate the gradient. For deep networks, this will bring additional storage and computational overhead. To solve this problem, we design a new neural network parameter update point and the improved first-order momentum By introducing the above new neural network parameter update point θ in equation 0-3 k+1 and the first-order momentum m k , we can get the improved "momentum estimation method" based on the Nesterov acceleration algorithm, expressed as formula 1:

[0064]

[0065] Among them, g k is the stochastic gradient estimate, i.e., the batch gradient, ξ k is random noise with mean 0, m k is the first-order momentum of the algorithm's kth iteration. η>0 is the basic learning rate, β1 is the accumulation coefficient of the first-order momentum, and its value is in the interval 0-1. It can be proved that the improved "momentum estimation method" based on the Nesterov acceleration algorithm (Formula 1) and the output of the Nesterov acceleration algorithm given by Formula 0-3 can be converted to each other. Secondly, when the algorithm converges, the output is the same iteration point, that is, we can get Here the subscript ∞ refers to the convergence point of the algorithm.

[0066] 2) Combining the momentum estimation method with the adaptive gradient method, we obtain the initial deep learning optimizer Adan;

[0067] By replacing the first-order and second-order momentum in the adaptive gradient update method in the deep learning optimizer given in Equation 0-2 with the improved momentum estimation method given in Equation 1, we can get the initial version of the Adan algorithm:

[0068]

[0069] Note that the biggest difference between the Adan algorithm given in Equation 2 and the adaptive gradient update method in the deep learning optimizer given in Equation 0-2 comes from the gradient difference term (g k -g k-1 ). In order to better treat the differential term, we separate it out and introduce a separate cumulative coefficient β2 for the differential term, thereby further establishing the deep learning optimizer Adan, which is expressed as Equation 3:

[0070]

[0071] in, is the stochastic gradient estimate, i.e., the batch gradient, ξ k is random noise with mean 0, m k , nk are the first-order and second-order momentum of the kth iteration, v k is the gradient difference momentum of the kth iteration, θ k is the network parameter of the kth iteration. Here, the square of the vector refers to the square of each dimension, where is the Hadamard inner product. η>0 is the base learning rate, ε>0 is the smoothing parameter, β1, β2 and β3 are the accumulation coefficients of the first order, gradient difference and second order momentum, which belong to the interval 0-1.

[0072] 3) By introducing weight decay technology, we obtain Adan, a deep learning optimizer with weight decay;

[0073] Formula 3 does not consider the weight decay strategy. However, weight decay has been proven to greatly improve the performance of large-scale image classification tasks. Therefore, the present invention introduces a unique weight decay strategy for Adan. The last line of the update formula in Formula 3 can be equivalently expressed as Formula 4:

[0074]

[0075] Here, for any given vector x, we define Equation 4 can intuitively illustrate that each iteration of the Adan algorithm is optimizing the objective function given by Equation 0-1 at the parameter point θ k Considering the quadratic characteristics of the classic l2 norm, Adan does not need to make a first-order approximation for the regularization term (the square of the l2 norm). Therefore, the present invention replaces the optimization objective in Formula 4 with the optimization objective function in Formula 5:

[0076]

[0077] According to Formula 5, we can derive the parameter update formula with weight decay, which is expressed as Formula 6:

[0078]

[0079] Replacing the last line of Formula 3 (update formula) with Formula 6, we can get the Adan optimizer with weight decay, expressed as Formula 7:

[0080]

[0081] 2. Use the deep learning optimizer Adan (or use the deep learning optimizer Adan with weight decay) to train the given deep neural network model to obtain a trained deep neural network model;

[0082] 3. After obtaining the trained deep neural network model, the image to be classified is input into the trained model, and the output of the network model is the category vector of the image to be classified.

[0083] In the specific implementation, the deep learning optimizer Adan proposed in the present invention is used to train the network to process the image to be classified task and realize image classification, which specifically includes the following steps:

[0084] 1) Prepare training data The data comes from the public dataset ImageNet, which contains N = 1,000,000 pictures, of which the i-th picture ζ i The image size is 224×224×3, which contains the color image of the classified object. i Indicates the category label corresponding to the i-th image, which is a 1,000-dimensional vector that can only take 0 or 1. i The position k with a value of 1 indicates that the image belongs to the kth category, and the value of 0 indicates that it does not belong to the corresponding category.

[0085] 2) Set up the deep neural network training process, including:

[0086] 21) Process the image training data:

[0087] For data expansion of training data, in addition to using random cropping and horizontal flipping, the data augmentation methods of Mixup and CutMix are also used with a probability of 0.5 during implementation.

[0088] For data enhancement of training data, the RandAugment method is specifically used.

[0089] 22) Network model training, a stochastic depth mechanism is set. The loss function f(·,·) in equation 0-1 is set to the cross entropy loss. In specific implementation, the batch size of the large-scale image classification method of the present invention is 4096. The number of warmup epochs is selected as 60.

[0090] 23) Use the Adan algorithm with weight decay given in Formula 7 to train the given deep neural network (including two network structures, ResNet-50 and ResNet-101). We select the weight decay coefficient as λ=0.02, the basic learning rate η=1.5e-2, and use the cosine learning rate decay method to decay the size of η during training. In addition, the cumulative coefficients are selected as β1=0.02, β2=0.08, and β3=0.01.

[0091] 3) After obtaining the trained network model, the image to be classified is input into the trained neural network, and the output layer of the network model is the category label of the image to be classified.

[0092] The efficiency of the method of the present invention can be further verified. In the specific implementation, 5 deep learning optimizers and 2 different neural network structures were selected. Different large-scale image classification methods were obtained by combining them in pairs. The classification accuracy of several different image classification methods under different training theorem numbers is now tested. Table 1 is a comparison of the image classification accuracy (%) of the optimizer of the present invention and the existing deep learning optimizer under different neural network structures. Note that all models use the same training settings.

[0093] Table 1 ImageNet image classification accuracy of the optimizer of the present invention and the existing deep learning optimizer

[0094]

[0095] According to the ImageNet image classification accuracy in Table 1, it can be concluded that: (1) In the large-scale image classification method based on the ResNet network structure, the Adan training method provided by the present invention only requires 200 training rounds in most cases to achieve higher or comparable classification accuracy on the ImageNet dataset (compared with the results of 300 training rounds of other optimizers); (2) When the number of training rounds is small (for example, when the number of training rounds is 100), the advantage of the image classification method trained by Adan is more obvious. It can be further seen that the large-scale image classification method based on Adan provided by the present invention has obvious technical advantages in terms of training efficiency, classification accuracy and convergence speed of the training method.

[0096] It should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

Claims

1. A large-scale image classification method based on a deep learning optimizer, characterized in that: A new deep learning optimizer is constructed, and a deep neural network model is trained using image training data and the constructed deep learning optimizer to obtain a trained image classification deep neural network model; then, a color image containing predefined categories to be classified is input into the trained image classification deep neural network model, and the output of the model is the category label of the image, thereby realizing large-scale image classification; the steps include: 1) Perform image preprocessing to obtain training image data and image data to be classified; 2) Construct a new deep learning optimizer. The input of the deep learning optimizer is a single color image or a batch of color images, and the output is the category label of the image. Constructing a deep learning optimizer includes the following processes: 21) The Nesterov acceleration algorithm is improved, and a momentum estimation method based on the Nesterov acceleration algorithm is designed, which is expressed as formula 1: Where k is the number of iterations; g k and g k-1 is the stochastic gradient estimate for the kth and k-1th iterations; is the loss function f(·,·) on the network parameters θ k Take the first-order differential; the training data ζ is sampled from an arbitrary distribution ξ k is random noise with mean 0; m k is the first-order momentum of the kth iteration; η>0 is the basic learning rate; β1 is the accumulation coefficient of the first-order momentum, which takes the value in the interval 0-1; θ is the learnable network parameter; θ k represents the network parameters of the kth iteration; 22) A new deep learning optimizer is constructed by using a new momentum estimation method and an adaptive gradient method in a deep optimizer; The improved momentum estimation method of formula 1 is used for the first-order momentum and second-order momentum in the adaptive gradient update method in the deep learning optimizer, and a separate accumulation coefficient β2 is introduced for the gradient difference term to further establish the deep learning optimizer, which is expressed as formula 3: Among them, m k , n k are the first-order momentum and second-order momentum of the kth iteration respectively; v k is the random gradient difference momentum of the color image at the kth iteration; η k is the generalized learning rate for the kth iteration; the formula in the last row is used for parameter update; the square of the vector is squared by dimension; is the Hadamard inner product; ε>0 is the smoothness parameter; β1, β2 and β3 are the accumulation coefficients of first-order momentum, gradient difference momentum and second-order momentum, respectively, and their values ​​are in the interval 0-1; 3) Using the training image data and the constructed deep learning optimizer, the deep neural network model is trained to obtain a trained image classification deep neural network model; the training image data is input into the deep learning optimizer, and the output is the category label of the image; A random depth mechanism is set to train the deep neural network model; the training loss function is set to the cross entropy loss function; the cosine learning rate decay method is used to decay the size of the generalized learning rate during the training process; 4) Input the color image containing predefined categories to be classified into the trained image classification deep neural network model to output the category label of the image; Through the above steps, large-scale image classification based on deep learning optimizer is achieved.

2. The large-scale image classification method based on deep learning optimizer as claimed in claim 1, characterized in that: After the deep learning optimizer is established in step 22), further, by introducing the weight decay technology, a deep learning optimizer with weight decay is obtained; Specifically: A weight decay strategy is introduced into the deep learning optimizer established in step 22), and a parameter update formula with weight decay is obtained, which is expressed as Formula 6: The Adan optimizer with weight decay is obtained, which is expressed as Equation 7: Among them, λ is the coefficient of weight decay.

3. The large-scale image classification method based on deep learning optimizer as claimed in claim 1, characterized in that: Utilize the training image data and use the constructed deep learning optimizer or the deep learning optimizer with weight decay to train a given deep neural network model to obtain a trained image classification deep neural network model; the deep neural network model may include a ResNet-50 network model and a ResNet-101 network model.

4. The large-scale image classification method based on deep learning optimizer as claimed in claim 3, characterized in that: When using Adan with weight decay for training, the weight decay coefficient is selected as λ=0.02 and the basic learning rate η=1.5e-2; and the cosine learning rate decay method is used to decay the size of η during the training process.

5. The large-scale image classification method based on deep learning optimizer as claimed in claim 3, characterized in that: The batch size for model training is 4096; the number of hot start rounds is set to 60.

6. The large-scale image classification method based on deep learning optimizer as claimed in claim 3, characterized in that the image data used for model training is represented as where ζ i is the i-th color picture containing the classified object; y i Represents the category label corresponding to the i-th color image; the value of the category label is 0 or 1.

7. The large-scale image classification method based on deep learning optimizer as claimed in claim 1, characterized in that: The image training data is processed, including: data expansion and data enhancement of the training data.

8. The large-scale image classification method based on deep learning optimizer as claimed in claim 7, characterized in that: Perform data augmentation on the training data, including random cropping and horizontal flipping; you can also use the data augmentation methods of Mixup and CutMix.

9. The large-scale image classification method based on deep learning optimizer as claimed in claim 7, characterized in that: Perform data enhancement on the training data, specifically using the RandAugment method.

Images