Deep neural network parameter optimizer, updating method and storage medium

By using the DiffGM optimizer to adjust the learning rate based on the difference between the target gradient and the first-order moment vector, the inaccuracy and noise sensitivity issues of algorithms such as DiffGrad during training are resolved, resulting in faster and more stable deep neural network optimization.

CN119026649APending Publication Date: 2024-11-26AIR FORCE UNIV PLA
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410774798.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-17
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing deep neural network optimization algorithms such as DiffGrad suffer from problems during training, such as inaccurate actual update direction, sensitivity to noisy gradients, and conservative learning rate at saddle points, resulting in low optimization efficiency.

Method used

A deep neural network parameter optimizer, DiffGM, is employed. By calculating the target gradient, first-order moment vector, and second-order moment vector, and combining the difference between the current gradient and the first-order moment vector, the learning rate is dynamically adjusted to better adapt to the model optimization process.

Benefits of technology

It improves model optimization performance, reduces the impact of noise gradients, achieves faster convergence and a more stable optimization process, and adapts to complex non-convex optimization landscapes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026649A_ABST
    Figure CN119026649A_ABST
Patent Text Reader

Abstract

The invention discloses a deep neural network parameter optimizer, an updating method and a storage medium, relates to the field of artificial intelligence, and is used for improving the optimization efficiency and effectiveness of a deep neural network model. According to the method, when network parameters are updated in each time step, the learning rate of the current time step is updated based on the difference between the gradient of the target function under the current time step and the first-order moment vector of the current time step; the updated learning rate of the current time step, the corrected first-order moment vector of the current time step and the corrected second-order moment vector of the current time step are utilized to calculate the update quantity of the deep neural network parameters in the current time step; and updating the deep neural network parameters of the previous time step by using the update quantity. According to the method, the adjustment of the learning rate is consistent with the actual optimization process of the model, the actual updating direction of the model is fully considered, and the short-term and long-term gradient differences are considered in the optimization process, so that faster convergence is realized while the optimal solution is prevented from being exceeded.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a deep neural network parameter optimizer, a deep neural network parameter updating method, and a computer-readable storage medium. Background Art

[0002] As an important field of artificial intelligence, deep learning can effectively solve various complex problems and has significantly changed various industries. In the field of computer vision, convolutional neural networks (CNNs) have achieved remarkable progress, for example, detecting skin cancer with an accuracy comparable to that of dermatologists. Advances in natural language processing with Transformers and recurrent neural networks (RNNs), especially long short-term memory (LSTM) networks, have enabled chatbots and virtual assistants to understand and generate human-like language. The field of autonomous driving uses CNNs and RNNs, coupled with reinforcement learning, to achieve safe navigation in complex environments. In the field of medical imaging, deep learning models help radiologists detect cancerous tumors with increasing accuracy. Power grids are becoming smarter due to deep learning algorithms that predict energy demand and supply, thereby improving scheduling efficiency. In e-commerce, recommendation systems based on deep learning algorithms study user behavior and provide personalized product recommendations, thereby promoting sales and customer satisfaction.

[0003] At the heart of these breakthroughs are deep neural networks, which have demonstrated an extraordinary ability to learn complex representations from huge datasets. However, training these networks is a difficult task because they require optimizing a complex and high-dimensional parameter space. Currently, stochastic gradient descent (SGD) remains the basic algorithm in this process, which iteratively updates the model parameters to minimize the loss function (objective function).

[0004] Despite its popularity, SGD has its limitations, in particular, the global learning rate it uses cannot adapt to the different frequencies and scales of the parameters being optimized. This can lead to slow convergence and poor performance. To overcome these problems, scholars have developed several adaptive learning rate optimization algorithms, each with a unique way to dynamically adjust the learning rate based on the training process. AdaGrad is one of the earliest adaptive learning rate methods, which adjusts the learning rate by making small but continuous updates to the accumulated squared gradients. While AdaGrad improves SGD by adapting to the data, it tends to reduce the learning rate over time too aggressively, leading to premature convergence and underutilization of the parameter space. AdaDelta was introduced as an improvement to AdaGrad, which solves the problem of overly aggressive reductions in the learning rate by modifying the accumulation term to be an exponentially weighted sum of squared gradients. This change allows AdaDelta to more robustly normalize gradients and maintain a more reasonable learning rate. RMSProp (root mean square propagation) expands on the idea of ​​AdaDelta and uses an exponentially weighted moving average of the squared gradients. This method further stabilizes the learning process by turning the accumulation into a moving average and introducing a decay factor that controls the speed of the moving average. RMSProp is popular for its practical effects and faster convergence speed compared to AdaGrad and AdaDelta. Adam (Adaptive Moment Estimation, proposed by KINGMA DPB et al. in "Adam: AMethod for Stochastic Optimization") combines the best features of AdaGrad and RMSProp, fusing the exponentially decaying average of the gradient (first moment) and the exponentially decaying average of the squared gradient (second moment). By using these two moments, Adam provides more powerful normalization and effectively adjusts the learning rate for each parameter. This makes Adam the first choice for many deep learning practitioners due to its efficiency and minimal configuration requirements.

[0005] Although these adaptive learning rate methods provide improvements over SGD, they often overlook the importance of local gradient changes, which has a significant impact on the efficiency of the optimization process. They focus on historical gradient information but do not fully exploit short-term gradient behavior, which can provide valuable information for the immediate optimization direction. DiffGrad was developed to fill this gap by considering the role of recent gradient changes in the learning rate adjustment strategy. DiffGrad introduces a novel approach that directly uses the difference between the current and past instantaneous gradients to adjust the step size for each parameter. Larger step sizes are assigned to parameters with fast gradient changes to speed up convergence, while smaller step sizes are assigned to parameters with slow gradient changes to prevent oscillations.

[0006] However, DiffGrad only relies on the most recent gradient changes, which may not accurately reflect the actual update direction of the model during the optimization process. Specifically, DiffGrad has the following shortcomings due to its excessive reliance on the most recent gradient changes:

[0007] 1) Inaccurate representation of the actual update direction: DiffGrad focuses on characterizing the changes in the true gradient of the model, but it does not accurately represent the gradient changes in the actual update direction based on the first-order moment estimate. This difference may lead to suboptimal performance and slower convergence.

[0008] 2) Sensitivity to noisy gradients: DiffGrad can be sensitive to noisy gradients caused by outliers in the mini-batch data during model training. In these cases, DiffGrad may make large learning rate adjustments in response to the noise, potentially leading to unstable or oscillatory behavior.

[0009] 3) Conservative learning rate at saddle points: In cases where the gradients of two consecutive iterations are close to zero, such as at saddle points, DiffGrad may adopt a smaller learning rate due to minimization of the gradient change. This conservative approach may lead to slow convergence or even stagnation at saddle points. In these cases, a larger learning rate would be more appropriate to escape the saddle point and continue the optimization process.

[0010] Addressing these limitations is crucial for developing more efficient and robust optimization algorithms that can better adapt to the complexities of deep learning tasks. Summary of the invention

[0011] The object of the present invention is to provide a deep neural network parameter optimizer, update method and storage medium to address all or part of the above-mentioned problems, so as to more accurately capture short-term gradient behavior and better align with the actual optimization process of the model, thereby achieving more effective optimization and faster convergence.

[0012] The technical solution adopted by the present invention is as follows:

[0013] A deep neural network parameter optimizer, the optimizer is used to update the deep neural network parameters at each time step in a plurality of consecutive time steps; the optimizer comprises:

[0014] The target gradient calculation circuit module is configured to: calculate the gradient of the target function at the current time step;

[0015] The first-order moment vector calculation circuit module is configured to: update the first-order moment vector of the previous time step based on the gradient at the current time step to obtain the first-order moment vector of the current time step;

[0016] The second-order moment vector calculation circuit module is configured to: update the second-order moment vector of the previous time step based on the gradient at the current time step to obtain the second-order moment vector of the current time step;

[0017] The first-order moment vector correction circuit module is configured to: correct the first-order moment vector of the current time step;

[0018] The second-order moment vector correction circuit module is configured to: correct the second-order moment vector of the current time step;

[0019] The parameter updating circuit module is configured to: update the learning rate of the current time step, and update the deep neural network parameters of the previous time step using the corrected first-order moment vector and the corrected second-order moment vector of the current time step;

[0020] The parameter updating circuit module updates the learning rate of the current time step based on the difference between the gradient of the current time step and the first-order moment vector of the current time step.

[0021] Furthermore, the first-order moment vector calculation circuit module calculates the first-order moment vector of the current time step according to the following configuration:

[0022] m t ←β1·g t +(1-β1)·m t-1 ,

[0023] Among them, m t represents the first-order moment vector of the current time step, β1 is the first exponential decay rate, g t represents the gradient of the current time step, m t-1 Represents the first-order moment vector at the previous time step.

[0024] Furthermore, the second-order moment vector calculation circuit module calculates the second-order moment vector of the current time step according to the following configuration:

[0025] v t ←β2·g t 2 +(1-β2)·v t-1 ,

[0026] Among them, v t represents the second-order moment vector of the current time step, β2 is the second exponential decay rate, v t-1 Represents the second-order moment vector at the previous time step.

[0027] Furthermore, the first-order moment vector correction circuit module corrects the first-order moment vector of the current time step according to the following configuration:

[0028]

[0029] in, Represents the corrected first-order moment vector for the current time step.

[0030] Furthermore, the second-order moment vector correction circuit module corrects the second-order moment vector of the current time step according to the following configuration:

[0031]

[0032] in, Represents the corrected second-order moment vector for the current time step.

[0033] Furthermore, the parameter updating circuit module calculates a dynamic correction factor of the current time step by the difference between the gradient of the current time step and the first-order moment vector of the current time step, and uses the dynamic correction factor to update the learning rate of the current time step.

[0034] Furthermore, the parameter update circuit module calculates the dynamic correction factor for the current time step according to the following configuration:

[0035] ξ t =AbsSig(g t -m t ),

[0036] Among them, ξ t represents the dynamic correction factor for the current time step, g t represents the gradient of the current time step, m t represents the first-order moment vector of the current time step, and t represents the current time step.

[0037] Furthermore, the parameter update circuit module updates the parameters of the deep neural network at the previous time step according to the following configuration:

[0038]

[0039] Among them, θ t represents the deep neural network parameters at the current time step, θ t-1 represents the deep neural network parameters of the previous time step, α t represents the learning rate of the current time step, and They represent the first-order moment vector and the second-order moment vector after the current time step update, and ∈ is the numerical stability constant of the configuration to avoid the denominator being zero.

[0040] To solve the above problem, the present invention also provides a method for updating parameters of a deep neural network, which is used to update the parameters of the deep neural network at each time step in a plurality of consecutive time steps. The method comprises:

[0041] The objective function is used to iteratively update the parameters of the deep neural network at each time step, where, at each time step, there are:

[0042] Update the learning rate of the current time step based on the difference between the gradient of the objective function at the current time step and the first-order moment vector of the current time step;

[0043] Utilizing the learning rate updated at the current time step, as well as the corrected first-order moment vector and the corrected second-order moment vector of the current time step, the update amount of the deep neural network parameters at the current time step is calculated; wherein the corrected first-order moment vector and the corrected second-order moment vector of the current time step are obtained by respectively correcting the first-order moment vector of the current time step and the second-order moment vector of the current time step, and the first-order moment vector of the current time step and the second-order moment vector of the current time step are respectively updated based on the gradient at the current time step to the first-order moment vector of the previous time step and the second-order moment vector of the previous time step;

[0044] The update amount is used to update the deep neural network parameters of the previous time step.

[0045] To solve the above problems, the present invention also provides a computer-readable storage medium, which stores a computer program. Running the computer program can execute the above-mentioned deep neural network parameter updating method.

[0046] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0047] 1. The present invention updates the learning rate based on the difference between the gradient and the first-order moment vector, so that the adjustment of the learning rate is consistent with the actual optimization process of the model, and fully considers the actual update direction of the model, thereby improving the optimization effect of the model.

[0048] 2. The present invention uses the first-order moment vector to update the learning rate, which helps to filter out the noise in the current gradient, making the learning rate adjustment process less sensitive to fluctuations caused by the noise gradient, and achieving a more stable and reliable optimization process.

[0049] 3. The present invention uses the difference between the current gradient and the first-order moment as the control of the learning rate, taking into account the short-term and long-term gradient differences. By considering the short-term and long-term gradient differences, the present invention can make more intelligent decisions on the learning rate, thereby achieving faster convergence while avoiding exceeding the optimal solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0051] Figure 1 It is a flow chart of the deep neural network parameter update method.

[0052] Figure 2 This is a diagram of the Transformer-based model architecture for 6G CSI feedback.

[0053] Figure 3 Here are diagrams of three non-convex functions and diagrams of empirical results.

[0054] Figure 4 It is a diagram of the 1D-CNN architecture for HRRP target recognition and inter-shaft bearing fault diagnosis.

[0055] Figure 5 Figure 2 is a graph showing the experimental results on four image classification datasets.

[0056] Figure 6 Figure 2 is a diagram of experimental results on the HIT and HRRP datasets. DETAILED DESCRIPTION

[0057] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0058] Any feature disclosed in this specification (including any additional claims and abstract), unless otherwise stated, may be replaced by other equivalent or alternative features having similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0059] Example 1

[0060] This embodiment introduces a deep neural network parameter optimizer. In this embodiment of the present invention, the optimizer is called DiffGM. The optimizer, like other optimizers, is used to update the deep neural network parameters at each time step in a plurality of consecutive time steps until the objective function converges or until a predetermined number of time steps is reached.

[0061] DiffGM includes a target gradient calculation circuit module, a first-order moment vector calculation circuit module, a second-order moment vector calculation circuit module, a first-order moment vector correction circuit module, a second-order moment vector correction circuit module and a parameter update circuit module. These circuit modules realize corresponding configuration functions at each time step through the cooperation of circuits and programs.

[0062] The target gradient calculation circuit module is configured to calculate the gradient of the target function at the current time step.

[0063] Assuming that the number of iteration time steps is T, the noise objective function is defined as f(θ). The purpose of optimizing the parameters of the deep neural network is to minimize the value of the objective function, where θ represents the parameters of the deep neural network. The realization of the objective function at each time step is expressed as f1(θ),...,f T(θ), where 1,...,T denotes the iteration time step. The variability in these realizations may be attributed to sampling data points from random subpopulations or to the inherent noise in the function itself.

[0064] The deep neural network model includes a large number of deep neural network parameters. i is used to represent the i-th deep neural network parameter. For each deep neural network parameter, its gradient corresponding to the t-th time step is expressed as g t,i , the gradient is defined as the gradient of the i-th deep neural network parameter θ i Location Assessment t (θ i ), expressed as Ignoring the parameter number, it is expressed as Correspondingly, the first-order moment vector of this parameter corresponding to the tth time step is expressed as m t,i . Then the parameter used to adjust the learning rate when optimizing the i-th deep neural network parameters is expressed as:

[0065] Δg t,i =g t,i -m t,i .

[0066] Since the optimization logic for each parameter is the same, in the subsequent description, no single deep neural network parameter will be marked separately, that is, the subscript i will no longer be reflected in the relevant parameters, that is, the gradient is represented by g t , the first-order moment vector is represented by m t .

[0067] The first-order moment vector calculation circuit module is configured to update the first-order moment vector of the previous time step based on the gradient at the current time step to obtain the first-order moment vector of the current time step.

[0068] t represents the current time step, m t represents the first-order moment vector of the current time step, β1 is the first exponential decay rate, g t represents the gradient of the current time step, m t-1 represents the first-order moment vector of the previous time step, then the first-order moment vector calculation circuit module calculates the first-order moment vector m of the current time step according to the following configuration t :

[0069] m t ←β1·g t +(1-β1)·m t-1 . Among them, m0←0.

[0070] The second-order moment vector calculation circuit module is configured to update the second-order moment vector of the previous time step based on the gradient at the current time step to obtain the second-order moment vector of the current time step.

[0071] With v t represents the second-order moment vector of the current time step, β2 is the second exponential decay rate, v t-1 represents the second-order moment vector of the previous time step, then the second-order moment vector calculation circuit module calculates the second-order moment vector of the current time step according to the following configuration:

[0072] v t ←β2·g t 2 +(1-β2)·v t-1 . Among them, v0←0.

[0073] The first-order moment vector correction circuit module is configured to correct the first-order moment vector of the current time step.

[0074] The first-order moment vector correction circuit module can correct the first-order moment vector of the current time step according to the following configuration:

[0075] Represents the corrected first-order moment vector for the current time step.

[0076] The second-order moment vector correction circuit module is configured to correct the second-order moment vector of the current time step.

[0077] The second-order moment vector correction circuit module can correct the second-order moment vector of the current time step according to the following configuration:

[0078] Represents the corrected second-order moment vector for the current time step.

[0079] The parameter updating circuit module is configured to: update the learning rate of the current time step and update the deep neural network parameters of the previous time step using the corrected first-order moment vector and the corrected second-order moment vector of the current time step.

[0080] As mentioned above, the basis for adjusting the learning rate in the present invention is Δg t =g t -m t (The description of the subscript i is omitted), that is, the parameter update circuit module updates the learning rate of the current time step based on the difference between the gradient of the current time step and the first-order moment vector of the current time step. It can be proved that g t -m t =(1-β1)·(g t -m t-1 ), which indicates that not only the short-term gradient difference but also the long-term gradient difference is taken into account when adjusting the learning rate.

[0081] In some embodiments, the data may be used to calculate a nonlinear parameter to update the learning rate. For example, the parameter update circuit module calculates the dynamic correction factor of the current time step by the difference between the gradient of the current time step and the first-order moment vector of the current time step, and uses the dynamic correction factor to update the learning rate of the current time step.

[0082] The dynamic correction factor is calculated as:

[0083] ξ=AbsSig(Δg),

[0084] The parameter update circuit module calculates the dynamic correction factor for the current time step according to the following configuration:

[0085] ξ t =AbsSig(g t -m t ),

[0086] Among them, ξ t represents the dynamic correction factor for the current time step,

[0087] Then use this dynamic correction factor to adjust the learning rate α of the current time step t Update, expressed as: α t ·ξ t α0 is obtained according to the initialization configuration.

[0088] Then the parameter update circuit module updates the deep neural network parameters of the previous time step according to the following configuration:

[0089]

[0090] Among them, θ t represents the deep neural network parameters at the current time step, θ t-1 represents the deep neural network parameters of the previous time step, ∈ is the numerical stability constant of the configuration, usually taking the value of 10 -7 , to avoid the denominator in the above formula being zero. θ0 is initialized randomly.

[0091] Through the cooperation of the above-mentioned circuit modules at each time step, the parameters of the deep neural network are updated at multiple time steps until the objective function converges, thus completing the optimization of the model.

[0092] Example 2

[0093] This embodiment introduces a deep neural network parameter updating method, or it can also be called a deep neural network optimization method, and its process is as follows: Figure 1The method is used to update the parameters of the deep neural network at each time step in a plurality of consecutive time steps until the objective function of the deep neural network model converges, or until a predetermined number of time steps are iterated.

[0094] The method of this embodiment includes the following process:

[0095] The objective function is used to iteratively update the parameters of the deep neural network at each time step, where, at each time step, there are:

[0096] Update the learning rate of the current time step based on the difference between the gradient of the objective function at the current time step and the first-order moment vector of the current time step;

[0097] Utilizing the learning rate updated at the current time step, as well as the corrected first-order moment vector and the corrected second-order moment vector of the current time step, the update amount of the deep neural network parameters at the current time step is calculated; wherein the corrected first-order moment vector and the corrected second-order moment vector of the current time step are obtained by respectively correcting the first-order moment vector of the current time step and the second-order moment vector of the current time step, and the first-order moment vector of the current time step and the second-order moment vector of the current time step are respectively updated based on the gradient at the current time step to the first-order moment vector of the previous time step and the second-order moment vector of the previous time step;

[0098] The update amount is used to update the deep neural network parameters of the previous time step.

[0099] Using f(θ) to represent the objective function of the model, the gradient at the tth time step is expressed as g t , which is defined as the objective function f evaluated at the tth time step t The gradient of (θ).

[0100] t represents the current time step, m t represents the first-order moment vector of the current time step, β1 is the first exponential decay rate, g t represents the gradient of the current time step, m t-1 represents the first-order moment vector of the previous time step, then the first-order moment vector m of the current time step is calculated according to the following method t :

[0101] m t ←β1·g t +(1-β1)·m t-1 . Among them, m0←0.

[0102] Then the update base of the learning rate of the current time step is: Δg t =g t -m t .

[0103] With vt represents the second-order moment vector of the current time step, β2 is the second exponential decay rate, v t-1 represents the second-order moment vector of the previous time step, then the second-order moment vector of the current time step is calculated according to the following method:

[0104] v t ←β2·g t 2 +(1-β2)·v t-1 . Among them, v0←0.

[0105] by Represents the first-order moment vector after correction at the current time step, which is calculated as:

[0106]

[0107] by Represents the second-order moment vector after correction at the current time step, which is calculated as:

[0108]

[0109] As described above, the learning rate of the current time step is updated based on the difference between the gradient of the objective function at the current time step and the first-order moment vector of the current time step. In some embodiments, the dynamic correction factor of the current time step is first calculated by the difference between the gradient of the current time step and the first-order moment vector of the current time step, and then the dynamic correction factor is used to update the learning rate of the current time step.

[0110] The dynamic correction factor is calculated as:

[0111] ξ=AbsSig(Δg).

[0112] Then the calculation method of the dynamic correction factor of the current time step is:

[0113] ξ t =AbsSig(g t -m t ),

[0114] Among them, ξ t represents the dynamic correction factor for the current time step,

[0115] Then use this dynamic correction factor to adjust the learning rate α of the current time step t Update, expressed as: α t ·ξ t α0 is obtained according to the initialization configuration.

[0116] Then, using the learning rate updated at the current time step, as well as the corrected first-order moment vector and the corrected second-order moment vector at the current time step, the method for calculating the update amount of the deep neural network parameters at the current time step is:

[0117]

[0118] Where ∈ is the numerical stability constant of the configuration, which is usually set to 10 -7 , to avoid the denominator being zero.

[0119] Finally, the update amount is used to update the deep neural network parameters of the previous time step:

[0120] θ t represents the deep neural network parameters at the current time step, θ t-1 Represents the deep neural network parameters at the previous time step.

[0121] The pseudo code of the method introduced in this embodiment is expressed as:

[0122]

[0123] The initial stage of the DiffGM algorithm involves setting the first-order moment vector m0 and the second-order moment vector v0 to zero, and initializing the time step t. The process then enters a recursive loop until the parameter vector θ t Convergence. In each cycle, the time step t is incremented, and then the gradient g of the random target corresponding to the current time step is determined t , then use g t Update the first-order moment vector and the second-order moment vector estimate. After that, obtain the correction estimate of the first-order moment vector and the second-order moment vector, and perform parameter vector θ t After completion, the algorithm outputs the resulting parameter vector θ t .

[0124] The DiffGM algorithm uses the difference between the gradient at the current time step and the first-order moment vector estimate to adjust the learning rate, essentially incorporating the actual update direction of the model. This approach provides a more accurate representation of the true optimization process. In addition, the first-order moment vector estimate acts as an EMA (exponential moving average) of the historical gradients, essentially reducing the impact of noisy gradients. In addition, the introduction of the first-order moment vector estimate allows a non-zero difference between the true gradient and the estimate when the model update is at a saddle point, thereby alleviating the problem of conservative learning rates in this case.

[0125] The DiffGM algorithm has the following characteristics:

[0126] 1) DiffGM makes the learning rate adjustment consistent with the actual optimization process.

[0127] In DiffGrad, the adjustment of the learning rate is based on the most recent gradient change, which may not accurately reflect the actual update direction of the model during the optimization process. In contrast, DiffGM uses the difference between the current gradient and the first-order moment as the control of the learning rate, which takes into account the actual update direction of the model. This modification keeps the learning rate adjustment consistent with the actual optimization process, making DiffGM more reasonable in theory.

[0128] 2) DiffGM alleviates the impact of noisy gradients.

[0129] The first-order moment vector is an exponentially weighted average of past gradients, which provides a more accurate representation of recent gradient history than using only recent gradient changes. DiffGM may reduce the impact of noisy gradients. Since the first-order moment vector takes into account past gradients through exponential decay, including past gradients g t-1 , provides a consistent and accurate measure of short-term gradient behavior, which may help filter out noise in the current gradient. As a result, the learning rate adjustment in DiffGM is less sensitive to fluctuations caused by noisy gradients, leading to a more stable and reliable optimization process. This property makes DiffGM more robust in theory and more effective in dealing with noisy gradients in practice.

[0130] 3) DiffGM considers long-term gradient differences to fully understand the gradient behavior.

[0131] It can be proved that g t -m t =(1-β1)·(g t -m t-1 ), which shows that DiffGM takes into account not only short-term gradient differences, but also long-term gradient differences. This approach has potential advantages in capturing a more comprehensive gradient behavior and providing a more accurate measure of the optimal learning rate. By considering long-term gradient differences, DiffGM may capture the overall trend of gradient behavior instead of just focusing on recent changes. This may lead to a more stable and reliable learning rate because it is less sensitive to noise or fluctuations between short-term gradients. In addition, considering long-term gradient differences may help to better capture the curvature of the optimization landscape, which may be important when avoiding local minima or saddle points.

[0132] 4) DiffGM balances rapid progress and cautious steps to achieve faster convergence.

[0133] DiffGM uses the difference between the current gradient and the first-order moment vector as a control for the learning rate, which takes into account both short-term and long-term gradient differences. This approach allows DiffGM to strike a balance between the need to make rapid progress toward the optimal solution and taking cautious steps in areas where the gradient changes are small. By taking into account both short-term and long-term gradient differences, DiffGM is able to make more informed decisions about the learning rate, leading to faster convergence while avoiding overshooting the optimal solution. This is especially important in non-convex optimization settings, where the optimization landscape may have many local minima or saddle points.

[0134] The DiffGM algorithm faces significant challenges in optimizing the loss function landscape of a particular neural network. These difficulties mainly stem from the inherent non-convexity of neural networks and their high-dimensional variable space, which hinders intuitive understanding using visualization tools. To circumvent these obstacles and facilitate more manageable investigations, this example constructs three univariate non-convex functions, specifically designed to have zero minima. The three functions are denoted as f1(x), f2(x), and f3(x). Figure 3 (a) Figure 3 (b) and Figure 3 (c) and their mathematical formulas are provided in Equation (1), Equation (2) and Equation (3). Figure 3 (a) has a global minimum and a local optimum, f2(x) Figure 3 (b) has a global optimum and two local optima, f3(x) Figure 3 (c) also has a global optimum and two local optima.

[0135]

[0136] A comparative evaluation was performed between Adam, DiffGrad, and the DiffGM proposed in this embodiment. To ensure a fair comparison of the algorithm performance, the experiments were conducted under a consistent set of hyperparameters: initial learning rate a0 = 0.1, total number of iterations T = 400, first-order moment vector estimation decay parameter β1 = 0.95, second-order moment vector estimation decay parameter β2 = 0.999, and numerical stability constant ∈ = 10 -7 All three algorithms start iterating from an initial value x0 = -3.

[0137] The results of these iterations are Figure 3 (d) Figure 3 (e) and Figure 3 (f) shows the relationship between the number of iterations and the variable x after 400 iterations for each algorithm. Figure 3 (g) Figure 3 (h) and Figure 3(i) shows the relationship between the number of iterations and the function value f(x).

[0138] For f1(x), both DiffGrad and DiffGM successfully converged to the global optimal solution x = -0.3, while the Adam algorithm converged to a suboptimal local optimal solution.

[0139] In the case of f2(x), all three algorithms failed to find the global optimal solution. DiffGrad stopped at x=0, while Adam and DiffGM converged to relatively better solutions. DiffGM achieved convergence faster than Adam and DiffGrad. Figure 3 (b) reveals that the gradient changes near x=0 are relatively small. This observation is consistent with the previously identified limitation of DiffGrad: it tends to take a conservative learning rate at saddle points. Specifically, when the gradients of consecutive iterations approach zero, as encountered at saddle points, DiffGrad may choose to reduce the learning rate due to the minimal gradient fluctuations. This cautious approach may hinder the convergence rate and even cause the optimization process to stagnate at these saddle points. Ideally, a larger learning rate would help escape these points and effectively resume optimization. The DiffGM method proposed in this embodiment addresses this challenge by adjusting the step size based on the difference between the current gradient and the first-order moment. This approach enables DiffGM to bypass local minima, such as the minimum at x=0, without getting stuck in them. Therefore, DiffGM performs well in avoiding stagnation at these critical points, thereby enhancing the overall optimization process.

[0140] For f3(x), Adam exhibits oscillatory behavior around the local optimal solution of x=5, while DiffGrad and DiffGM eventually oscillate around the global optimal solution of x=0. Figure 3 As shown in (i), the oscillation amplitude of DiffGM is significantly reduced compared with that of DiffGrad.

[0141] Overall, DiffGM performs well in experiments on three non-convex functions, outperforming the compared algorithms. This verifies the effectiveness of the method of adjusting the learning rate based on the difference between the current gradient and the first-order moment estimate in terms of optimization effect and efficiency, which is more effective than the DiffGrad method that only considers short-term gradient changes. The motivation behind DiffGM is to address the limitations of short-term memory of gradient information, which can lead to premature convergence or oscillation around suboptimal solutions. By incorporating longer-term gradient memory, DiffGM enhances the algorithm's ability to navigate the complex non-convex landscapes unique to neural networks, thereby providing a more stable, robust, and efficient optimization strategy.

[0142] Example 3

[0143] This embodiment verifies the optimizer / optimization method proposed in the present invention in a deep learning task.

[0144] The models and algorithms involved in this embodiment are implemented using PyTorch 2.1 and Python 3.11. In order to ensure a fair comparison between different optimizers under the same conditions, the experimental hyperparameters are set as follows: the initial learning rate α0 is set to 0.001, β1 = 0.9, β2 = 0.999, and ∈ = 10 -8 This example trains all models for 200 cycles and reduces the learning rate α0 by a factor of 0.1 every 60 cycles. For the image classification task, this example adopts a weight decay strategy and sets the weight decay hyperparameter to 0.001. For other tasks, this example does not apply weight decay.

[0145] In order to verify the effectiveness of the DiffGM algorithm proposed in the present invention, this embodiment conducts a series of experiments on various tasks and data sets, including image classification, radar target recognition, inter-axle bearing fault diagnosis, and 6G channel state information feedback. The tasks considered in the experimental evaluation include classification and regression problems. Image classification tasks using ResNet18 on CIFAR10, CIFAR100, PathMNIST, and SVHN datasets, as well as radar high-resolution range image (HRRP) target recognition and inter-axle bearing fault diagnosis tasks using 1D-CNN are all examples of classification problems. In these tasks, the goal is to correctly assign input samples to their respective categories based on the learned representation. On the other hand, the 6G channel state information (CSI) feedback task using the Transformer architecture is a regression problem. In this task, the goal is to accurately predict the continuous-valued CSI feedback based on the input data, rather than assigning it to a discrete category. By including classification and regression tasks in the experimental evaluation, a comprehensive evaluation of the performance of the DiffGM algorithm in different problem areas is ensured.

[0146] Experimental setup:

[0147] (1) Performance evaluation of ResNet18 on image classification tasks.

[0148] This experiment uses a compact residual network architecture ResNet18 with 18 layers to evaluate the performance of DiffGM on image classification tasks. ResNet18 is a convolutional neural network (CNN) that uses residual connections to alleviate the gradient vanishing problem, enabling the training of deeper networks. The ResNet18 architecture contains 11,220,132 trainable parameters. In this experiment, four image datasets were used: CIFAR10, CIFAR100, PathMNIST, and SVHN.

[0149] The CIFAR10 dataset consists of 60,000 32x32 color images divided into 10 categories with 6,000 images per category. The dataset is divided into 50,000 training images and 10,000 test images. CIFAR100 is similar to CIFAR10 but contains 100 categories with 600 images per category.

[0150] The PathMNIST dataset is a set of 100,000 28×28 grayscale images of histopathological scans of lymph node tissue classified into 9 categories. The dataset is divided into 89,999 training images and 10,008 test images.

[0151] The SVHN dataset is a real-world image dataset obtained from house numbers in Google Street View images. It contains more than 600,000 32×32 color images divided into 10 categories. The dataset is divided into 73,257 training images and 26,032 test images.

[0152] (2) Application of 1D-CNN in radar HRRP target recognition and inter-axle bearing fault diagnosis tasks.

[0153] The experiment uses the HRRP dataset proposed by Liu L et al. in “On the Variance of the Adaptive Learning Rate and Beyond; proceedings of the Eighth International Conference on Learning Representations” to test the DiffGM method on the radar target recognition task, and uses the HIT dataset from an aircraft engine system to test the method for the inter-shaft bearing fault diagnosis task.

[0154] The description of HRRP and HIT datasets is as follows:

[0155] The HRRP dataset contains simulated radar echo data for five types of medium-range ballistic targets: warheads, high-fidelity decoys, simple decoys, spherical decoys, and motherships. The dataset was generated using a physical optics method that involves simulating the electromagnetic scattering properties of these targets under different conditions. The dataset contains a total of 3601 HRRP samples for each target category, corresponding to different azimuths. Each HRRP sample consists of 256 range cells, providing detailed information about the target range profile. 20\% of the samples from each target category are randomly selected to form the test dataset, and the remaining 80% are used for training.

[0156] The HIT dataset provides a comprehensive dataset for fault diagnosis research in aero-engine systems. The dataset contains vibration signals from actual aero-engines, including rotor and casing signals of inter-shaft bearings under different fault conditions. It has multiple speeds and fault types, including healthy, inner race fault, and outer race fault, covering the rotor speed range of 1000-6000 rpm. The dataset provides signals from displacement and acceleration sensors, with a total of 2412 sets of data, each containing 20,480 data points from 6 channels, and classifies the fault states of inter-shaft bearings into three different categories: healthy inter-shaft bearings, inter-shaft bearings with inner race faults, and inter-shaft bearings with outer race faults. There are 9648 samples in the test dataset and 2412 samples in the training dataset. This dataset is very valuable for benchmarking and evaluating the performance of fault diagnosis methods in realistic and challenging aero-engine environments.

[0157] In this experiment, the widely used one-dimensional convolutional neural network (1D-CNN) architecture is adopted for both tasks, such as Figure 4 As shown, the architecture contains four convolution modules. The specific configuration of 1D-CNN is as follows: each convolution module uses a standard convolution operation, followed by a batch normalization layer (BN layer), a nonlinear activation layer, and a pooling layer. For the nonlinear activation layer, this embodiment selects the Mish activation function, which performs well in various deep learning tasks. Given that both the HRRP and HIT datasets contain samples with only one channel, the output channels of the convolution layer, from left to right, are set to 32, 64, 128, and 256, respectively. The convolution kernel size is uniformly set to 3, the stride is 1, the group is set to 1, and the zero padding is 0. For the pooling layer, we used the maximum pooling method, with a window size of 3 and a stride of 1 for each pooling layer. The entire 1D-CNN architecture has a total of 131,328 trainable parameters.

[0158] (3) Application of Transformer in 6G CSI feedback task.

[0159] For the 6G CSI feedback task, this experiment used the open source dataset provided by OPPO Research Institute and used the Transformer-1Decoder model described in the paper “CsiTransformer: A Limited-sample 6G Channel State InformationFeedback Model” by XIANG Q et al. Figure 2As shown in Figure 1, (a) is the detailed structure of the Transformer, and (b) is the overall flow chart of the CSI feedback task. This task involves using a Transformer-based encoder to encode the original CSI data at the base station and convert it into a 30-bit bit stream. The user equipment then uses a Transformer-based decoder to decode this bit stream back to the original CSI data. The Transformer backbone in the encoder and decoder has the same Figure 2 The dataset includes training and validation sets for two scenes. This embodiment randomly selects 20% of the training set as the test set for model selection. The model is trained using data from two scenes, and the experimental results report all three subsets: training set, validation set, and test set. It is worth noting that the test and validation results are presented for the two scenes respectively. Unlike the classification model that uses accuracy evaluation, the CSI feedback task uses generalized cosine similarity square (SGCS), mean absolute error (MAE), and mean square error (MSE) as evaluation indicators. A higher SGCS value indicates a better CSI reconstruction quality, while lower MAE and MSE values ​​indicate better reconstruction quality.

[0160] The above three experiments cover a variety of neural network architectures, data sets, and tasks. This rich experimental evaluation enables this embodiment to comprehensively evaluate the effectiveness and robustness of the DiffGM algorithm compared to other state-of-the-art optimization methods such as Adam, RAdam, CAME, and DiffGrad. By comparing the performance of these algorithms on different tasks and data sets, it aims to provide a comprehensive understanding of the strengths and weaknesses of each method, as well as the potential benefits of using DiffGM to solve various optimization problems.

[0161] Experimental results:

[0162] (1) Results of ResNet18 on image classification tasks.

[0163] Table 1 shows the results of ResNet18 on four image classification tasks: CIFAR10, CIFAR100, PathMNIST, and SVHN. The performance of the model is evaluated using two key metrics: Top-1 accuracy and cross entropy loss on the test dataset.

[0164] Table 1: Results of ResNet18 on CIFAR10, CIFAR100, PathMNIST and SVHN image classification tasks. The symbol ↑ indicates that the higher the metric value, the better the model performance, and the symbol ↓ indicates that the lower the metric value, the better the model performance.

[0165]

[0166] On all datasets, the optimizer DiffGM shows superior performance compared to other considered optimizers. Specifically, DiffGM achieves the highest test Top-1 accuracy on all four datasets, 92.85% on CIFAR10, 70.38% on CIFAR100, 92.16% on PathMNIST, and 95.724% on SVHN. Similarly, DiffGM also achieves the lowest test loss on three of the four datasets (CIFAR10, CIFAR100, and SVHN), indicating better model fit and generalization ability.

[0167] It is worth noting that although CAME achieves high accuracy (92.61%) on CIFAR10, its performance on other datasets and in terms of test loss is not strong. This indicates that CAME is inferior to DiffGM in generalization across different tasks. On the other hand, RAdam shows competitive performance, especially on PathMNIST, where it achieves the lowest test loss (0.286). However, DiffGM still outperforms RAdam in terms of accuracy on all datasets except PathMNIST, where the difference is marginal.

[0168] The most widely used optimizer, Adam, exhibits the lowest performance among all the considered optimizers, highlighting the need for more advanced optimization techniques like DiffGM to fully exploit the capabilities of deep learning models like ResNet18.

[0169] Figure 5 The accuracy and loss function curves of the five optimizers on the training and test datasets during 200 training cycles are also shown. Obviously, the convergence results of the DiffGM algorithm on the test sets of the four image datasets are significantly better than those of other methods. Although the convergence speed of DiffGM during training is significantly slower than that of CAME, the generalization performance of DiffGM is significantly better than that of CAME. This is shown in the fourth row of the subfigure ( Figure 5 (m), Figure 5 (n) Figure 5 (o) and Figure 5 (p)) supports that these figures reveal that CAME exhibits a clear degradation process on the test set even with learning rate decay.

[0170] In summary, the optimizer DiffGM consistently outperforms other state-of-the-art optimizers on multiple image classification tasks, demonstrating its potential as a powerful tool for training deep learning models.

[0171] (2) Results of 1D-CNN on radar HRRP target recognition and inter-axle bearing fault diagnosis tasks.

[0172] Table 2 shows the results of 1D-CNN models trained with different optimizers on two datasets: the HRRP dataset for radar target recognition and the HIT dataset for inter-shaft bearing fault diagnosis. The performance metrics reported include top-1 accuracy and loss during training and testing phases. Among the tested optimizers, RAdam achieves the highest training accuracy and the lowest training loss on both datasets, indicating its superiority in fitting the training data. However, when it comes to testing performance—a more reliable indicator of the effectiveness of the optimizer, DiffGM performs best. Specifically, DiffGM achieves the highest testing accuracy on both the HIT and HRRP datasets, outperforming all other optimizers, including the widely used Adam and the recently proposed RAdam and CAME. In terms of testing loss, DiffGM also performs well, achieving the lowest loss on the HRRP dataset and a competitive loss with RAdam on the HIT dataset, with RAdam having the lowest loss on HIT. This shows that DiffGM not only improves accuracy but also maintains a low loss during testing, indicating better generalization and less overfitting to the training data.

[0173] Table 2: Results of 1D-CNN on the HRRP dataset for radar target recognition and the HIT dataset for inter-shaft bearing fault diagnosis. The symbol ↑ indicates that the higher the metric value, the better the model performance, and the symbol ↓ indicates that the lower the metric value, the better the model performance.

[0174]

[0175] from Figure 6 It can be observed that RAdam performs best in terms of convergence asymptotes in terms of training accuracy and loss on the HIT and HRRP datasets. However, when it comes to the test set, DiffGM is shown to be superior, further validating the strong generalization ability of the proposed DiffGM throughout the process. In addition, it is worth noting that CAME performs well in training on the image datasets, but performs relatively poorly on the training and test sets of the HIT and HRRP datasets. This suggests that although CAME may facilitate fast convergence on image-based tasks, its applicability and versatility is limited in a wider range of deep learning tasks compared to DiffGM.

[0176] Overall, the results demonstrate the effectiveness of DiffGM in improving the performance of 1D-CNN models on radar target recognition and bearing fault diagnosis tasks. DiffGM not only achieves excellent training results, but also maintains strong generalization capabilities on unseen data, making it a more robust and widely applicable optimization method than other algorithms tested in this study. Its strong test performance shows that DiffGM is a promising optimizer for improving the generalization capabilities of deep learning models.

[0177] (3) Transformer results on the 6G channel state information feedback task.

[0178] Table 3 compares the performance of various optimization algorithms, namely Adam, RAdam, CAME, DiffGrad, and DiffGM, on the 6G channel state information feedback task. The evaluation metrics include SGCS (higher is better), MAE (mean absolute error, lower is better), and MSE (mean square error, lower is better) for both training and test datasets, as well as validation metrics for two different scenarios and their combinations.

[0179] Table 3: Transformer results on 6G channel state information feedback. The symbol ↑ indicates that higher metric values ​​indicate better model performance, while the symbol ↓ indicates that lower metric values ​​indicate better model performance.

[0180]

[0181] From Table 3, we can conclude that DiffGrad slightly outperforms DiffGM and RAdam in training SGCS, indicating that it has a slightly stronger ability to model the training data. However, when considering the training MAE and MSE, DiffGM significantly outperforms all other methods, indicating that it provides more accurate predictions during training.

[0182] For the test metrics, DiffGM again demonstrates its superiority on all evaluation criteria (SGCS, MAE, MSE), highlighting its strong generalization ability on unseen data. RAdam is close behind DiffGM in terms of testing SGCS, but is less accurate than DiffGM in terms of MAE and MSE.

[0183] In the validation sets of both scenarios and their combination, DiffGM consistently outperforms all other methods in all metrics. This further reinforces the robustness and versatility of DiffGM in handling different scenarios. This shows that DiffGM is able to handle changes in data distribution and provide robust performance in different scenarios.

[0184] Overall, DiffGM emerged as the clear winner in this comparative analysis, demonstrating superior performance during training and testing, as well as strong generalization capabilities on different validation sets. Its consistent superiority across different metrics and datasets highlights the versatility and effectiveness of DiffGM as an optimization algorithm for 6G CSI feedback tasks, especially in complex scenarios where generalization and robustness are critical.

[0185] The present invention is not limited to the above-mentioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A deep neural network parameter optimizer, the optimizer is used to update the deep neural network parameters at each time step in a plurality of consecutive time steps; characterized in that: The optimizer comprises: The target gradient calculation circuit module is configured to: calculate the gradient of the target function at the current time step; The first-order moment vector calculation circuit module is configured to: update the first-order moment vector of the previous time step based on the gradient at the current time step to obtain the first-order moment vector of the current time step; The second-order moment vector calculation circuit module is configured to: update the second-order moment vector of the previous time step based on the gradient at the current time step to obtain the second-order moment vector of the current time step; The first-order moment vector correction circuit module is configured to: correct the first-order moment vector of the current time step; The second-order moment vector correction circuit module is configured to: correct the second-order moment vector of the current time step; The parameter updating circuit module is configured to: update the learning rate of the current time step, and update the deep neural network parameters of the previous time step using the corrected first-order moment vector and the corrected second-order moment vector of the current time step; The parameter updating circuit module updates the learning rate of the current time step based on the difference between the gradient of the current time step and the first-order moment vector of the current time step.

2. The deep neural network parameter optimizer according to claim 1, characterized in that The first-order moment vector calculation circuit module calculates the first-order moment vector of the current time step according to the following configuration: m t ←β1·g t +(1-β1)·m t-1 , Among them, m t represents the first-order moment vector of the current time step, β1∈(0,1] is the first exponential decay rate, g t represents the gradient of the current time step, β t-1 Represents the first-order moment vector at the previous time step.

3. The deep neural network parameter optimizer according to claim 2, characterized in that: The second-order moment vector calculation circuit module calculates the second-order moment vector of the current time step according to the following configuration: v t ←β2·g t 2 +(1-β2)·v t-1 , Among them, v t represents the second-order moment vector of the current time step, β2∈(0,1] is the second exponential decay rate, v t-1 Represents the second-order moment vector at the previous time step.

4. The deep neural network parameter optimizer according to claim 2, characterized in that: The first-order moment vector correction circuit module corrects the first-order moment vector of the current time step according to the following configuration: in, Represents the corrected first-order moment vector for the current time step.

5. The deep neural network parameter optimizer according to claim 3, characterized in that: The second-order moment vector correction circuit module corrects the second-order moment vector of the current time step according to the following configuration: in, Represents the corrected second-order moment vector for the current time step.

6. The deep neural network parameter optimizer according to claim 1, characterized in that: The parameter updating circuit module calculates the dynamic correction factor of the current time step through the difference between the gradient of the current time step and the first-order moment vector of the current time step, and uses the dynamic correction factor to update the learning rate of the current time step.

7. The deep neural network parameter optimizer according to claim 6, characterized in that The parameter update circuit module calculates the dynamic correction factor for the current time step according to the following configuration: x t =AbsSig(g t -m t ), Among them, ξ t represents the dynamic correction factor for the current time step, g t represents the gradient of the current time step, m t represents the first-order moment vector of the current time step, and t represents the current time step.

8. The deep neural network parameter optimizer according to claim 7, characterized in that: The parameter update circuit module updates the deep neural network parameters of the previous time step according to the following configuration: Among them, θ t represents the deep neural network parameters at the current time step, θ t-1 represents the deep neural network parameters of the previous time step, α t represents the learning rate of the current time step, and They represent the first-order moment vector and the second-order moment vector after the current time step update, and ∈ is the numerical stability constant of the configuration to avoid the denominator being zero.

9. A method for updating parameters of a deep neural network, for updating the parameters of the deep neural network at each time step in a plurality of consecutive time steps, characterized in that: The method includes: The objective function is used to iteratively update the parameters of the deep neural network at each time step, where, at each time step, there are: Update the learning rate of the current time step based on the difference between the gradient of the objective function at the current time step and the first-order moment vector of the current time step; Utilizing the learning rate updated at the current time step, as well as the corrected first-order moment vector and the corrected second-order moment vector of the current time step, the update amount of the deep neural network parameters at the current time step is calculated; wherein the corrected first-order moment vector and the corrected second-order moment vector of the current time step are obtained by respectively correcting the first-order moment vector of the current time step and the second-order moment vector of the current time step, and the first-order moment vector of the current time step and the second-order moment vector of the current time step are respectively updated based on the gradient at the current time step to the first-order moment vector of the previous time step and the second-order moment vector of the previous time step; The update amount is used to update the deep neural network parameters of the previous time step.

10. A computer-readable storage medium having a computer program stored therein, characterized in that: Running the computer program can execute the deep neural network parameter updating method as claimed in claim 9.

Citation Information

Cited By

  • F5G all-optical substation real-time monitoring method and system fused with neural network, and medium

    CN119763038A