Image classification method based on adaptive momentum and automatic gradient descent optimizer

Abstract

The invention relates to an image classification method based on an adaptive momentum and an automatic gradient descent optimizer. The method comprises the following steps: acquiring an image to be classified; using the trained image classification model to carry out identification processing on a to-be-classified image to obtain an image category of the to-be-classified image; wherein the image classification model is trained by adopting an optimizer based on adaptive momentum and automatic gradient descent; an adaptive momentum and automatic gradient descent optimizer is established on the basis of a learning rate and a momentum framework designed for the hyper-parameter-free SGDM, a way is laid for new progress of hyper-parameter-free optimization, the method is more stable due to the hyper-parameter-free property, and faster momentum convergence is achieved. By adopting the method, the problem of computing resource waste caused by too long model training time is solved, the generalization ability of the image classification model is enhanced, and the image classification accuracy is improved.

Description

Technical Field

[0001] This application relates to the technical field of image classification, and particularly to an image classification method based on an adaptive momentum and automatic gradient descent optimizer. Background Art

[0002] The application of neural networks and optimizers in image classification has developed under the combined action of various technical backgrounds. They cooperate with each other and continuously promote the progress of image classification technology.

[0003] Optimizers play a crucial role in training deep neural networks (DNNs). Optimizers may get stuck in local optima during training rather than global optima, especially in deep neural networks with complex non-convex loss functions. In this way, the image classification model may not achieve its theoretical best performance. Similarly, optimizers may lead to overfitting, especially when there is less training data. When overfitting occurs, the image classification model performs well on the training set but poorly on the validation set or test set, resulting in poor generalization of the image classification model. Existing optimizers usually require hyperparameters, such as the learning rate or momentum, which often need to be fine-tuned repeatedly, making the training time too long and causing waste of computing resources. In addition, even well-calibrated default hyperparameters may not achieve the best performance, resulting in unsatisfactory accuracy of image classification. Summary of the Invention

[0004] Based on this, it is necessary to provide an image classification method based on an adaptive momentum and automatic gradient descent optimizer for the above technical problems.

[0005] An image classification method based on an adaptive momentum and automatic gradient descent optimizer, the method includes: Obtain the image to be classified.

[0006] Use the trained image classification model to perform recognition processing on the image to be classified, and obtain the image category of the image to be classified.

[0007] Among them, the training process of the image classification model includes: Obtain training samples and the image classification model to be trained; the training samples include labeled images.

[0008] Input the training samples into the image classification model to be trained, and obtain the predicted output.

[0009] According to the predicted output and the corresponding annotations, the image classification model is trained using an optimizer based on adaptive momentum and stochastic gradient descent to obtain a trained image classification model; the optimizer based on adaptive momentum and stochastic gradient descent is used to determine the automatic learning rate according to the total number of layers of the image classification model to be trained, the weighted gradient average of each layer in the current iteration process, and the average gradient of a preset batch in the current iteration process, and to determine the adaptive momentum according to a preset momentum threshold, the automatic learning rate, and the sum of the gradients of a preset batch in the current iteration process and the previous iteration process. The weights of the image classification model to be trained are updated according to the automatic learning rate and the adaptive momentum.

[0010] In one embodiment, the automatic learning rate is:

[0011]

[0012]

[0013] Where is the automatic learning rate, L is the total number of layers of the image classification model to be trained, is the weighted gradient average of each layer in the current iteration process, is the average gradient of a preset batch in the current iteration process, , are the output dimension and the input dimension of the current network layer respectively, is the gradient of the current network for sample i, are the parameters of the network at iteration k, m is the value of a preset batch in the current iteration process.

[0014] In one embodiment, the adaptive momentum is:

[0015] Where is the adaptive momentum, L is the total number of layers of the image classification model to be trained, is the preset momentum threshold, is the automatic learning rate, , are the average gradients of a preset batch in the current iteration process and the previous iteration process respectively, is a projection operation with threshold , is the maximum function, is the minimum function.

[0016] In one embodiment, the weight update equation of the image classification model to be trained is as follows:

[0017]

[0018] Wherein, and are the weights of the image classification model to be trained in the next iteration process and the current iteration process respectively, is the adaptive momentum, is the automatically adjusted learning rate, is the average gradient of the preset batch in the current iteration process, and are the moving averages of all past gradients at the iteration k moment and k -1 moment respectively.

[0019] In one embodiment, the method further includes: adding weight decay to the optimizer based on adaptive momentum and automatic gradient descent; the loss function after adding weight decay is:

[0020] Wherein, is the weight decay hyperparameter, is the loss function after adding weight decay, is the loss function before adding weight decay, is the weight of the image classification model to be trained, is the output of the image classification model, ([[]] s i , y i ) represents the training data.

[0021] In one embodiment, the loss function during the training process of the image classification model to be trained is the cross-entropy loss function.

[0022] In one embodiment, the method further includes initializing the weights before training the image classification model to be trained; the initialization process is:

[0023]

[0024] Wherein, is the re-adjusted weight, is the weight after orthogonal initialization, and are the output dimension and the input dimension of the current network layer respectively, ​​is an orthonormalization function.

[0025] The above image classification method based on the adaptive momentum and automatic gradient descent optimizer includes: obtaining an image to be classified; using a trained image classification model to perform recognition processing on the image to be classified to obtain the image category of the image to be classified; wherein, the image classification model is trained by using the adaptive momentum and automatic gradient descent optimizer; the adaptive momentum and automatic gradient descent optimizer is built on the learning rate and momentum framework designed for the hyperparameter-free SGDM, paving the way for new progress in hyperparameter-free optimization, being more robust due to its hyperparameter-free nature, and achieving faster momentum convergence. Using this method overcomes the problem of waste of computing resources caused by too long model training time, enhances the generalization ability of the image classification model, and improves the image classification accuracy. Brief Description of the Drawings

[0026] Figure 1 is a schematic flowchart of the image classification method based on the adaptive momentum and automatic gradient descent optimizer in one embodiment; Figure 2 is a schematic diagram of the test results of the CIFAR-10 / 100 test accuracy and the learning rate on different networks in another embodiment, where Figure 2 (a), Figure 2 (b) and Figure 2 (c) are respectively schematic diagrams of the test results of the CIFAR-10 test accuracy and the learning rate on the VGG16, ResNet18, and PreResNet18 networks, Figure 2 (d), Figure 2 (e) and Figure 2 (f) are respectively schematic diagrams of the test results of the CIFAR-100 test accuracy and the learning rate on the VGG16, ResNet18, and PreResNet18 networks; Figure 3 is a schematic diagram of the test results of the β accuracy on the CIFAR-10 / 100 dataset and the VGG / ResNet network structure in another embodiment, where Figure 3 (a) is a schematic diagram of the test results of the β accuracy on the CIFAR-10 dataset and the ResNet18 network structure, Figure 3 (b) is a schematic diagram of the test results of the β accuracy on the CIFAR-100 dataset and the PreResNet18 network structure. Detailed Embodiments

[0027] To make the objectives, technical solutions and advantages of this application more clear and understandable, the following further details this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely used to explain this application and are not used to limit this application.

[0028] Symbol definition: In this article, lowercase letters represent scalars, and bold letters represent vectors or matrices.

[0029] Based on the Adaptive Momentum and Automatic Gradient Descent optimizer, abbreviated as: AGDAM optimizer.

[0030] In one embodiment, as Figure 1 shown, an image classification method based on the Adaptive Momentum and Automatic Gradient Descent optimizer is provided, and the method includes the following steps: Step 100: Obtain the image to be classified; Step 102: Use the trained image classification model to perform recognition processing on the image to be classified to obtain the image category of the image to be classified; wherein, the training process of the image classification model includes: obtaining training samples and the image classification model to be trained; the training samples include labeled images; input the training samples into the image classification model to be trained to obtain a predicted output; according to the predicted output and the corresponding label, use the Adaptive Momentum and Automatic Gradient Descent optimizer to train the image classification model to obtain a trained image classification model; the Adaptive Momentum and Automatic Gradient Descent optimizer is used to determine the automatic learning rate according to the total number of layers of the image classification model to be trained, the weighted gradient average of each layer in the current iteration process, and the average gradient of a preset batch in the current iteration process, and determine the adaptive momentum according to the preset momentum threshold, the automatic learning rate, and the gradient sum of the preset batch in the current iteration process and the previous iteration process, and update the weights of the image classification model to be trained according to the automatic learning rate and the adaptive momentum.

[0031] Specifically, when training a deep neural network (DNN) model using Stochastic Gradient Descent (SGD), selecting an appropriate learning rate sequence is crucial for effective convergence. A too high learning rate will hinder convergence, while a too low learning rate will greatly slow down the training speed. Usually, hyperparameters (such as the learning rate) are adjusted using grid search or similar methods. However, as the model scale grows, hyperparameter tuning becomes increasingly expensive and complex, making traditional search methods less feasible. To address these challenges, Automatic Gradient Descent (AGD) introduced an adaptive step size schedule that aims to work hierarchically, thus simplifying the process of selecting the best learning rate. However, in actual training, the convergence speed of AGD is usually slow. Drawing on the heavy ball and acceleration techniques in convex optimization, momentum-based SGD variants incorporate momentum terms that use information from previous updates to achieve faster convergence. The momentum update rule is given by: , is given, where represents the weighted average of past gradients.

[0032] In practice, adding momentum to SGD introduces an additional sequence of momentum hyperparameters . Currently, most momentum-based SGD methods adopt fixed momentum hyperparameters , but this hyperparameter significantly affects the convergence speed and the final model quality. Inspired by quadratic optimization insights, Adaptive Regularized Stochastic Heavy Ball (ANSHB) analyzes the optimal choice of the momentum hyperparameter and proposes an adaptive momentum sequence that can dynamically converge to this optimal momentum hyperparameter value. Despite the improvements, the performance of ANSHB still depends largely on the choice of the learning rate, which can be challenging to tune effectively across different architectures and datasets.

[0033] The learning rate and the momentum hyperparameter play crucial roles in the effectiveness of SGD and its variants in training deep neural networks (DNNs). Traditionally, these parameters are either set to fixed values or adjusted through heuristic adaptive methods based on empirical results. However, this approach lacks a strong theoretical foundation and may suffer from inefficiencies and suboptimal convergence. There is a great demand for optimizers that do not require parameter tuning, have solid theoretical properties, and can achieve fast convergence speeds. This application paves the way for new advancements in hyperparameter-free optimization by developing an optimizer that does not require any fine-tuning of the learning rate and the momentum hyperparameter.

[0034] Momentum has shown significant empirical success in accelerating training and reducing the variance of SGD. However, the theoretical proof of accelerating momentum convergence is not perfect. Research has shown that tuning the momentum hyperparameter can achieve optimal performance. In addition to adaptive learning rates, efforts have also been made to establish schemes for the momentum hyperparameter . For example, Wen et al. 1 (Cutkosky & Mehta, 2020) analyzed the stochastic gradient oracle and proposed an adaptive momentum hyperparameter scheme based on past learning rates and Lipschitz bounds. The Adaptive Stochastic Heavy Ball (ASHB) method determines the optimal momentum hyperparameter of the heavy ball algorithm in quadratic optimization problems . Similarly, the ANSHB method developed a scheme to approximate the minimum eigenvalue of the Hessian and used the past gradients and updates to estimate the optimal momentum hyperparameter However, these optimizers still require the learning rate hyperparameter, and the training performance still highly depends on this parameter. In the practical application of SGDM, the learning rate (denoted by lr , with a default value of 0.1) and the momentum hyperparameter (default value of 0.9) must be specified. To develop a truly hyperparameter-free SGDM optimizer, these two parameters must be designed to operate without manual adjustment. The learning rate plays a crucial role in the convergence speed of gradient descent, both theoretically and empirically. To avoid the fine-tuning process of hyperparameters lr , AGD introduced a step size schedule that explicitly adapts to the neural architecture, thus eliminating the need for fine-tuning in SGD. AGD is also a variant of SGD with normalized updates.

[0035] Despite the great potential of AGD, its experimental performance still lags behind that of mainstream optimizers such as SGDM and Adam, especially in terms of convergence speed, due to problems such as oscillations and being prone to getting stuck in local minima. This highlights the improvement potential of AGD. To improve the convergence speed, a common method is to integrate momentum into the update rule. Momentum is crucial for accelerating DNN training and improving generalization ability. It helps to stabilize the updates by reducing oscillations, thus guiding the gradient in a clearer direction. However, directly applying a fixed momentum parameter to AGD introduces heuristic tuning, which undermines the goal of a hyperparameter-free optimizer. Momentum contains the weighted average of past gradients, and the momentum parameter controls this weight. This hyperparameter greatly affects the training performance. Based on this, ANSHB proposed the optimal momentum hyperparameter for the momentum update rule , where is the smallest eigenvalue of the Hessian matrix. Although is usually unknown before training, it can be estimated during training. Therefore, ANSHB developed an adaptive sequence that converges to an approximation of this optimal , thus facilitating adaptive momentum without additional tuning. To improve the convergence speed of AGD while maintaining its hyperparameter-free property, this method introduces adaptive momentum into AGD, thus resulting in the Adaptive Momentum Automatic Gradient Descent Optimizer (AGDAM optimizer).

[0036] The image classification model can be but is not limited to ResNet model, VGG model, FCN model, and ViT model.

[0037]

[0038] ​In the above image classification method based on the adaptive momentum and automatic gradient descent optimizer, the method includes: obtaining an image to be classified; using a trained image classification model to perform recognition processing on the image to be classified to obtain the image category of the image to be classified; wherein, the image classification model is trained by using the adaptive momentum and automatic gradient descent optimizer; The adaptive momentum and automatic gradient descent optimizer are based on the learning rate and momentum framework designed for the hyperparameter-free SGDM, paving the way for new progress in hyperparameter-free optimization, being more robust due to its hyperparameter-free nature, and achieving faster momentum convergence. Using this method overcomes the problem of waste of computing resources caused by the too long model training time, enhances the generalization ability of the image classification model, and improves the image classification accuracy.

[0039] In one embodiment, the automatic learning rate is:

[0040]

[0041]

[0042] Wherein, is the automatic learning rate, L is the total number of layers of the image classification model to be trained, is the weighted gradient average of each layer in the current iteration process, is the average gradient of the preset batch in the current iteration process, 、 are the output dimension and input dimension of the current network layer respectively, is the gradient of the current network for sample i, is the parameter of the network at iteration k, m is the value of the preset batch in the current iteration process, represents the Frobenius norm of a given matrix.

[0043] In one embodiment, the adaptive momentum is:

[0044] Wherein, is the adaptive momentum, L is the total number of layers of the image classification model to be trained, is the preset momentum threshold, is the automatic learning rate, 、 are the average gradients of the preset batch in the current iteration process and the previous iteration process respectively, is the projection operation with threshold As a preference , is the maximum value function, is the minimum value function.

[0045] In one embodiment, the weight update equation of the image classification model to be trained is:

[0046]

[0047] Wherein, and are the weights of the image classification model to be trained in the next iteration process and the current iteration process respectively, is the adaptive momentum, is the automatic learning rate, is the average gradient of the preset batch in the current iteration process, and are the moving averages of all past gradients at iteration k time and k -1 time respectively.

[0048] In one embodiment, the method further includes: adding weight decay to the adaptive momentum and automatic gradient descent optimizer; the loss function after adding weight decay is:

[0049] Wherein, is the loss function after adding weight decay, is the loss function before adding weight decay, is the weight of the image classification model to be trained, is the output of the image classification model, ([[]] s i , y i ) represents the training data, is the weight decay hyperparameter. The larger the value of λ, the greater the penalty imposed on the weight, and the smaller the value, the smaller the penalty imposed.

[0050] Specifically, weight decay is another key hyperparameter in the training of deep neural networks (DNNs). By adding an additional ℓ2 penalty to the weights during training, weight decay aims to produce a well-optimized DNN with smaller weights.

[0051] To analyze the AGDAM optimizer with added weight decay, the AGDAM optimizer and the baseline optimizer are evaluated on the CIFAR-10 / 100 using ResNet series models, using common weight decay values To evaluate the impact of weight decay on this method. From the experimental results, it can be seen that: the larger the weight decay value (e.g., ), the greater it is, especially for SGD and Adam, and the greater the performance difference compared to that observed when the weight decay value is small. These fluctuations stem from the stronger constraints imposed by higher weight decay. In contrast, the AGDAM optimizer exhibits higher stability under these conditions. Among different weight decay settings, the AGDAM optimizer can always achieve performance equivalent to or better than the baseline, demonstrating its robustness. Fine-tuning this hyperparameter can further enhance the experimental results.

[0052] In one embodiment, the loss function during the training process of the image classification model to be trained is the cross-entropy loss function.

[0053] In one embodiment, the method further includes initializing the weights before training the image classification model to be trained; the initialization process is:

[0054]

[0055] where, is the re-adjusted weight, is the weight after orthogonal initialization, , are respectively the output dimension and the input dimension of the current network layer, is the orthogonalization function.

[0056] Specifically, the AGD method advocates weight initialization, which involves sampling a semi-orthogonal matrix and re-adjusting the singular values of each weight layer in a deep neural network (DNN) to conform to theoretical suggestions. Here, the necessity of this orthogonal initialization for the AGDAM optimizer proposed in this application will be explored.

[0057] Although the initial parameters of the network may have limited impact on the results when the number of training iterations is large enough, when the number of iterations is limited, good initialization can significantly affect the training results. We used ResNet and VGG series models to check the performance of the AGDAM optimizer with and without orthogonal initialization for CIFAR-10 / 100 image classification. The results show that in most cases, the presence or absence of orthogonal initialization does not have a significant impact on the results after 100 iterations. However, there are significant differences in PreResNet18 on CIFAR-100, indicating that 100 iterations may not be sufficient to meet the requirements of this architecture. This observation emphasizes the reason for not using orthogonal initialization by default in the AGDAM optimizer.

[0058] It should be understood that although Figure 1The steps in the flowchart are shown in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least some of the steps in Figure 1 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least some of the sub-steps or stages of other steps.

[0059] In a validation example, a series of experiments will be conducted to evaluate the performance of the method proposed in this application. The datasets used are: CIFAR-10 / 100 dataset, MNIST dataset, and ImageNet-1k dataset. The CIFAR-10 / 100 dataset contains 60,000 32*32*3 images of 10 and 100 classes, of which 50,000 are for training and 10,000 are for testing; the MNIST dataset contains 70,000 28*28 grayscale images of 10 classes, of which 60,000 are for training and 10,000 are for testing. The ImageNet-1k dataset contains approximately 1.2 million training images, 50,000 validation images, and 100,000 test images of 1000 classes.

[0060] For the MNIST and CIFAR-10 / 100 datasets, this example uses the cross-entropy loss function and the default batch size of 128. For the ImageNet-1k dataset, the cross-entropy loss function and a batch size of 512 are used. Several popular network architectures are used to perform the classification task, including ResNet series models, VGG series models, AlexNet, and several custom-designed fully connected networks (FCNs).

[0061] To evaluate the performance of this method, this example compares the AGDAM optimizer with the optimizers commonly used in the training of deep neural networks (DNNs). The baselines include SGDM, Adam with the default PyTorch (Paszke et al., 2017) implementation, AGD, and AGD with fixed momentum which is denoted as AGD+M (using = 0.9 as the default value). Except for AGD, all test baselines include momentum. In this embodiment, the performance is mainly evaluated according to the test set accuracy of each algorithm in the dataset. In addition, for the ImageNet-1k dataset, this embodiment gives the top-5 accuracy to provide a more comprehensive evaluation. To ensure a clear comparison of the original performance of the optimizers and minimize the impact of hyperparameters, no weight decay is used in all experiments in this embodiment.

[0062] All experiments in this embodiment are carried out on a server equipped with 4 NVIDIA 3090 GPUs and an Intel(R) Xeon(R) Gold 6226R CPU. Python (Van Rossum & Drake Jr, 1995) version 3.9.19 and PyTorch (Paszke et al., 2017) version 1.12.1 are used. The experimental results are obtained using random seeds from 42 to 46. By default, this embodiment adopts a batch size of 128 and a PyTorch random seed of 42.

[0063] (1) CIFAR / MNIST Classification In this embodiment, a custom-designed 16-layer FCN is first trained for MNIST classification, and ResNet18 and ResNet18 with pre-activation (PreResNet18) are used for CIFAR-10 / 100 classification. For FCN training, the default learning rates are 0.1 (SGD) and 0.001 (Adam). It can be seen from the experimental results that neither SGD nor Adam can effectively train the FCN using their default hyperparameters.

[0064] For ResNet18 and PreResNet18, this instance performs a grid search on the learning rates {0.0001, 0.001, 0.01, 0.1} of SGD and Adam. The final test accuracy achieved by the AGDAM optimizer proposed in this application can be comparable to the adjusted Adam and SGD. In all test networks of CIFAR10 / 100, the method of this application has a fixed momentum hyperparameter is always superior to AGD and AGD+M. In addition, comparing AGD+M with AGD shows that adding momentum can improve the convergence speed. It can be seen that the learning curve of the AGDAM optimizer has been leading AGD+M on CIFAR-10 / 100 and is only slightly behind Adam on CIFAR-100. Although the final learning accuracy of the AGDAM optimizer is lower than that of Adam, the AGDAM optimizer can achieve a lower test loss than Adam in the CIFAR-100 classification task. To evaluate the performance of the optimizer proposed in this application under different batch sizes and mitigate the impact of random seed changes, a series of experiments were conducted using the common batch sizes {16, 32, 64, 128, 256, 512}. Table 1 shows the final average test accuracy and deviation for each batch size. For CIFAR-10 / 100 classification, this embodiment uses a learning rate of 0.01 for SGD (0.1 cannot train an unbiased ResNet18) and a learning rate of 0.001 for Adam.

[0065] Table 1 Final test accuracy of CIFAR-10 / 100 image classification with different batch sizes

[0066] The results show that, except for Adam on CIFAR-100, all optimizers generally use smaller batch sizes to obtain higher test accuracies than larger batch sizes. The AGDAM optimizer shows more robust performance than AGD+M. If larger batch sizes are adopted, such as 64, 28, 256, and 512, the AGDAM optimizer performs better than AGD+M. In addition, compared with other baselines, the AGDAM optimizer shows less deviation. It can be seen that the experimental effects of AGD+M (from 73.6% when the batch size is 16 on CIFAR-100 to 60.6% when the batch size is 256) and AGD (from 72.8% to 47.8%) are greatly affected by the batch size. Only the fine-tuned Adam on CIFAR-10 and SGD on CIFAR-100 show less deviation than the AGDAM optimizer. The smaller the deviation, the higher the stability against the variance caused by the data volume. Overall, the method of this application performs as well as or even exceeds the fine-tuned Adam and SGD in the CIFAR-10 and CIFAR-100 image classification tasks for all tested batch sizes. At the same time, it also demonstrates the advantages of adaptive learning rates and momentum parameters.

[0067] (2)Hyperparameter study This embodiment also conducts a series of CIFAR-10 / 100 image classification experiments to analyze the performance of baseline optimizers with various hyperparameters. The results are as Figure 2 shown.Figure 2 (a), Figure 2 (b) and Figure 2 (c) are respectively the schematic diagrams of the test results of the test accuracy and learning rate of CIFAR-10 on the VGG16, ResNet18, and PreResNet18 networks. Figure 2 (d), Figure 2 (e) and Figure 2 (f) are respectively the schematic diagrams of the test results of the test accuracy and learning rate of CIFAR-100 on the VGG16, ResNet18, and PreResNet18 networks. For clarity, the learning curve of AGD+M on CIFAR-10 is omitted in this embodiment.

[0068] Figure 2 The advantages of no hyperparameter optimization are demonstrated. It can be observed that in all tested network architectures, the performance of SGD and Adam is highly sensitive to the learning rate hyperparameter. A too large learning rate will lead to a sharp decline in model performance, while a too small learning rate may lead to non-convergence. For example, SGDM with a learning rate of 0.1 only achieved a test accuracy of 10%, while when the learning rate was 0.05, it achieved an accuracy of approximately 80% on the CIFAR-10 classification task. These results emphasize that default hyperparameters sometimes lead to suboptimal results because they are more based on experience rather than theory.

[0069] In contrast, the AGDAM optimizer and AGD show higher robustness due to their no-hyperparameter characteristics. This reflects the advantage of not requiring parameter adjustment because not all default hyperparameters can produce ideal performance.

[0070] Next, an ablation study is conducted on the momentum hyperparameter . ResNet18 and VGG13 are trained for CIFAR-10, and PreResNet18 is trained for CIFAR-100. Using SGD with a learning rate of 0.01 for CIFAR-10, 0.1 for CIFAR-100, and 0.001 for Adam, we examined the test accuracies of AGD+M, SGD, and Adam with various momentum values. The results are as Figure 3 shown, where Figure 3 (a) is the schematic diagram of the test results of the β accuracy on the CIFAR-10 dataset and the ResNet18 network architecture, Figure 3 (b) is the schematic diagram of the test results of the β accuracy on the CIFAR-100 dataset and the PreResNet18 network architecture. For the momentum hyperparameter , in this embodiment, the momentum hyperparameter is increased by 0.02.

[0071] In the CIFAR-10 image classification task, increasing the momentum hyperparameter to a certain extent usually improves the test accuracy. However, when the momentum hyperparameter exceeds 0.95, the results drop sharply. The AGDAM optimizer proposed in this application performs best on VGG13 and only slightly outperforms Adam on ResNet18. This indicates that momentum has a significant impact on the performance of the Adam optimizer. Compared with AGD with fixed momentum, our adaptive momentum method shows higher robustness and excellent performance.

[0072] In the CIFAR-100 dataset, the final performance of Adam is highly sensitive to momentum, as evidenced by the large fluctuations in the test accuracy curve. In contrast, SGD shows strong robustness to changes in this hyperparameter, although the underlying reasons for these phenomena are not fully understood. A direct comparison of AGD+M and AGD reveals that fine-tuning the momentum can significantly improve the final performance. Although not perfect theoretically, the practical improvement brought by momentum is significant.

[0073] (3)ImageNet Classification Evaluate our method on the ImageNet-1k classification task. Train ResNet34 using SGD, Adam, AGD, AGD+M, and AGDAM optimizers. For SGD and Adam, we use the default hyperparameters provided by PyTorch (learning rate of 0.1 for SGD and 0.001 for Adam, both optimizers use β = 0.9). To mitigate the impact of randomness, multiple experiments were conducted using different random seeds, and the results are summarized in Table 2.

[0074] Table 2 Summary of Experimental Results

[0075] As can be seen from Table 2, although the AGDAM optimizer proposed in this application performs worse than AGD+M, SGD, and Adam in terms of training loss, the AGDAM optimizer achieves the best results in terms of Top-1 and Top-5 accuracy in the later stage of training and has the lowest test loss. It is worth noting that while the test losses of other methods tend to increase in the later stage of training, the test loss of the AGDAM optimizer continues to decline, indicating that our method shows the least overfitting in the test baseline. In addition, the AGDAM optimizer consistently outperforms other methods in various experiments (see Table 2). Moreover, compared with the CIFAR-10 / 100 tasks, the performance gap between AGD and AGD+M in these experiments has narrowed, which may be due to the larger number of training epochs. Overall, the experiments confirm that adding momentum is an effective strategy to improve performance.

[0076] It should be noted that the AGDAM optimizer proposed in this application is extended to NLP tasks. For NLP tasks, we evaluate the AGDAM optimizer on traditional CNN and LSTM architectures for NLP on the IMDB dataset. The IMDB dataset contains 50,000 movie reviews labeled as positive or negative, evenly divided into 25,000 for training and testing. All test networks are non-regularized, and biases and affine parameters are disabled. First, we train a custom-designed 2-layer LSTM without additional dropout for IMDB dataset classification. We use the default hyperparameters for SGD (learning rate lr = 0.1) and Adam (lr = 0.001), and a batch size of 128 for IMDB classification.

[0077] During training, we can see that the AGD with fixed momentum fails to capture the hidden patterns in the training set, resulting in a lower training accuracy. In addition, AGD converges more slowly during training than other optimizers, requiring more than 5,000 iterations to achieve an accuracy of 80%. Moreover, SGD is successful during training (training accuracy is approximately 100%), but fails to accurately classify the sentences in the test set (accuracy is approximately 75%), which leads to severe overfitting. Among the optimizers tested, only the Adam and AGDAM optimizers successfully train the custom-designed CNN under the default settings. The AGD-like optimizers other than the AGDAM optimizer perform worse during training. This also demonstrates the adaptability of adaptive momentum to network structures other than CNN.

[0078] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0079] The above-described embodiments only represent several implementation manners of this application, and their descriptions are relatively specific and detailed. However, it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several deformations and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application should be subject to the appended claims.

Claims

1. An image classification method based on an adaptive momentum and automatic gradient descent optimizer, characterized in that, The method includes: Obtain the image to be classified; Use the trained image classification model to perform recognition processing on the image to be classified, and obtain the image category of the image to be classified; Among them, the training process of the image classification model includes: Obtain the training samples and the image classification model to be trained; the training samples include the labeled images; Input the training samples into the image classification model to be trained, and obtain the predicted output; According to the predicted output and the corresponding annotation, use the optimizer based on adaptive momentum and automatic gradient descent to train the image classification model, and obtain the trained image classification model; the optimizer based on adaptive momentum and automatic gradient descent is used to determine the automatic learning rate according to the total number of layers of the image classification model to be trained, the weighted gradient average of each layer in the current iteration process, and the average gradient of the preset batch in the current iteration process, and determine the adaptive momentum according to the preset momentum threshold, the automatic learning rate, and the gradient sum of the preset batch in the current iteration process and the previous iteration process, and update the weights of the image classification model to be trained according to the automatic learning rate and the adaptive momentum.

2. The image classification method based on the adaptive momentum and automatic gradient descent optimizer according to claim 1, wherein The automatic learning rate is: wherein, is the automatic learning rate, L is the total number of layers of the image classification model to be trained, is the weighted gradient average of each layer in the current iteration process, is the average gradient of the preset batch in the current iteration process, in the k-th iteration process, l is the gradient of the -th layer, and are the output dimension and the input dimension of the current network layer respectively, i is the gradient of the current network for the sample are the parameters of the network at iteration k, m is the value of the preset batch in the current iteration process.

3. The image classification method based on the adaptive momentum and automatic gradient descent optimizer according to claim 1, characterized in that The adaptive momentum is: Among them, is the adaptive momentum, L is the total number of layers of the image classification model to be trained, is the preset momentum threshold, is the automatic learning rate, 、 are the average gradients of the preset batches in the current iteration process and the previous iteration process respectively, is the projection operation with the threshold , is the maximum value function, is the minimum value function.

4. The image classification method based on the adaptive momentum and automatic gradient descent optimizer according to claim 1, wherein The weight update equation of the image classification model to be trained is: Among them, and are the weights of the image classification model to be trained in the next iteration process and the current iteration process respectively, is the adaptive momentum, is the automatically learned rate, is the average gradient of the preset batch in the current iteration process, and are the moving averages of all past gradients at the iteration k moment and k -1 moment respectively.

5. The image classification method based on the adaptive momentum and automatic gradient descent optimizer according to claim 1, characterized in that, The method further includes: adding weight decay to the optimizer based on adaptive momentum and automatic gradient descent; the loss function after adding weight decay is: Among them, is the weight decay hyperparameter, is the loss function after adding weight decay, is the loss function before adding weight decay, is the weight of the image classification model to be trained, is the output of the image classification model,( s i , y i ) represents the training data.

6. The image classification method based on the adaptive momentum and automatic gradient descent optimizer according to claim 1, wherein The loss function during the training process of the image classification model to be trained is the cross-entropy loss function.

7. The image classification method based on the adaptive momentum and automatic gradient descent optimizer according to claim 1, characterized in that The method further includes initializing the weights before training the image classification model to be trained; the initialization process is: Among them, is the re-adjusted weight, is the weight after orthogonal initialization, , are the output dimension and the input dimension of the current network layer respectively, is the orthogonalization function.

Images