Hyper-parameter optimization and performance evaluation method of NAG based on phase plane method

Abstract

The invention provides an NAG hyper-parameter optimization and performance evaluation method based on a phase plane method. The method comprises the following specific steps: S11, establishing an iterative formula based on a process of updating network parameters by NAG; s12, based on an NAG iterative formula, establishing an NAG transfer function and an NAG system dynamic model; s13, deriving an NAG differential equation from the NAG transfer function; s14, based on the NAG differential equation, establishing a phase plane graph of the NAG system to analyze the influence of hyper-parameters on the convergence rate and stability of the NAG system; and obtaining dynamic behaviors of the NAG system under different parameter values through the phase trajectory. According to the method, system optimization is carried out on hyper-parameters of the NAG algorithm through a phase plane analysis tool, the optimized NAG algorithm is excellent in performance in a deep learning task, and the convergence speed and stability of the algorithm are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and deep learning optimizers, and in particular to a hyperparameter optimization and performance evaluation method of NAG based on a phase plane method. Background Art

[0002] With the rapid development of artificial intelligence, deep learning, as a branch of artificial intelligence, is widely used in image recognition, control and other fields. As a key component in the training of deep learning models, the optimization algorithm directly affects the convergence speed and final performance of the model. The NAG optimizer is an indispensable computing link in the deep learning model, which is used to optimize the network model and realize the parameterized expression of the model for specific problems. The NAG optimizer is a basic tool for deep learning models. Compared with the traditional gradient descent method, NAG introduces a momentum term, which can effectively speed up the convergence speed and reduce gradient oscillation to a certain extent. However, the performance of NAG depends on the selection of its hyperparameters, such as learning rate and momentum factor, and the selection of these hyperparameters often depends on experience or a large number of trial and error experiments. In addition, the existing research on gradient descent optimization algorithms focuses on empirical practice, and the research on the role of parameters has the following three key issues that need to be solved urgently:

[0003] (1) The basic discrete description method of the optimizer has a narrow applicability, focuses on program implementation, and lacks a mathematical explanation of the basic mechanism of action;

[0004] (2) Improper settings of optimizer parameters may cause problems such as gradient vanishing, training oscillation, and long training time. There is a lack of theoretical explanation for scientific analysis of the effects of optimizer parameters.

[0005] (3) The parameter control of the optimizer mostly adopts trial and error and experience methods. The training and learning process generally adopts the loss function curve based on the experiment and analyzes the trend of the initial and final values. There is a lack of methods to characterize the dynamic process of the system.

[0006] Therefore, it is of great significance to develop a systematic hyperparameter optimization and performance evaluation method for NAG. The phase plane method is a dynamic system analysis tool, and it is also a mathematical tool for studying the behavior of dynamic systems by analyzing the relationship between system state variables. In the study of optimization algorithms, the phase plane method is used to describe the gradient changes in the optimization process, providing an intuitive way to analyze the convergence characteristics and stability of the optimization algorithm. Through the phase plane method, the impact of different hyperparameter combinations on the optimization trajectory is observed, thereby providing guidance for the selection of hyperparameters. However, the existing NAG hyperparameter optimization research based on the phase plane method has the following shortcomings:

[0007] (1) The phase plane method involves a lot of mathematical calculations and graphics drawing. For high-dimensional deep learning models, the computational complexity and resource consumption are large, making it difficult to be widely promoted in practical applications;

[0008] (2) Existing research focuses on the optimization of individual hyperparameters, but lacks a systematic approach to the joint optimization of multiple hyperparameters, which results in repeated debugging on different problems and datasets, which is time-consuming and labor-intensive.

[0009] (3) Although the phase plane method provides an intuitive analysis tool, it lacks in-depth integration with the deep learning model optimization theory, making it difficult to systematically explain the various phenomena and results in the optimization process from a theoretical perspective. Summary of the invention

[0010] In order to solve the above problems, the present invention proposes a NAG hyperparameter optimization and performance evaluation method based on the phase plane method, which systematically optimizes its hyperparameters by constructing a phase plane model of the optimizer and analyzing its dynamic behavior.

[0011] The first aspect of the present invention proposes a method for optimizing the hyperparameters and evaluating the performance of NAG based on a phase plane method, comprising the following steps:

[0012] S11, based on the process of updating the deep learning network parameters of NAG (Nesterov Accelerated Gradient), establish a NAG iteration formula, wherein the NAG iteration formula is a discrete form time domain expression of NAG;

[0013] S12, establishing a NAG transfer function and a NAG system dynamic model based on the NAG iterative formula, wherein the NAG transfer function is used to describe the input-output relationship of the NAG system, and the NAG system dynamic model is used to solve the expression problem of the internal structure of the NAG system;

[0014] S13, establishing a NAG differential equation based on the NAG transfer function, and obtaining the relationship between the control coefficients in the NAG differential equation and the coefficients in its time domain expression;

[0015] S14. Based on the NAG differential equation, the phase plane diagram of the NAG system is established to obtain the influence of hyperparameters on the convergence speed and stability of the NAG system; the phase trajectories of underdamped, critically damped and overdamped systems are established to obtain the dynamic behavior of the NAG system under different parameter values, providing guidance for the selection of hyperparameters;

[0016] Furthermore, in step S11, the steps of establishing the NAG iteration formula are as follows:

[0017] S21. Calculate the loss function at the current moment t about θ t The gradient is:

[0018]

[0019] S22, calculate the next time t+1 parameters:

[0020] θ t+1 =θ t -m t+1

[0021] S23, taking a step before calculating the current gradient, the step being to perform a tentative update before updating the parameters to evaluate the effectiveness or stability of the update direction:

[0022]

[0023] but

[0024]

[0025] S24. Calculate momentum:

[0026] m t+1 =αm t +rg t+1

[0027] S25, calculate the next moment t+1 parameter before the current gradient,

[0028]

[0029] Then the NAG iteration formula is:

[0030]

[0031] Where t represents the number of iterations; r is the learning rate, and r>0; L(θ t ) is the parameter θ t The loss function of It is at the estimated parameter position The gradient calculated at θ t is the updated parameter before step t; θ t+1 is the updated parameter of step t+1; α is the momentum coefficient or decay weight, which is used to adjust the weight of historical gradient and current gradient; Indicates the use of momentum term αm at step t t Estimated parameters; g t+1 Indicates that in the forward parameter position The gradient calculated at ; is the gradient of the loss function L with respect to the parameter θ; m t is the momentum vector of the t-th step, which is the weighted sum of all the gradients of the previous t steps, attenuated by α; m t+1 is the momentum vector at step t+1; represents the forward-looking parameter position at step t+1, which is obtained by forward-looking adjustment of the parameters using the current momentum term and is used to calculate the gradient in the next step; α t-i represents the momentum coefficient used in step ti; Represents the loss function L for the parameter at the i-th moment The gradient calculated at .

[0032] Furthermore, in step S12, the steps of establishing the NAG transfer function are:

[0033] S31. Construct an open-loop transfer function of the SGD controller. The open-loop transfer function refers to the functional relationship between the ratio of the output to the input of an open-loop system and the frequency, and its expression is:

[0034]

[0035] Among them, K 0 represents the gain or amplification factor of the control system; r is the learning rate; parameter s is equivalent to the error accumulation effect of the model;

[0036] S32, make an advance correction to the SGD controller, wherein the advance correction is that NAG updates the momentum by using the future gradient, that is, the G n Set to advance correction, the expression is:

[0037]

[0038] Where α is the lead correction factor (α>0), T 1 is the time constant of the correction factor;

[0039] S33. Let g(μ) = αT, then the expression of the open-loop transfer function of the NAG controller is:

[0040]

[0041] Among them, g(μ) is the momentum control function;

[0042] S34 and NAG controllers are typical second-order control systems with zero points, and their closed-loop transfer functions are expressed as follows:

[0043]

[0044] Furthermore, in step S34, the dynamic process of the NAG system is controlled by:

[0045]

[0046] Among them, ω nis the undamped natural oscillation frequency of the NAG system, ξ is the damping ratio of the NAG system;

[0047] Then the convergence speed and stability of the NAG dynamic equation are controlled by K 0 , r, g(μ) and α, the expression of NAG dynamic equation is:

[0048]

[0049] Among them, x′ 1 is the state variable x 1 The derivative with respect to time t, which is a one-dimensional scalar; x′ 2 is the state variable x 2 The derivative with respect to time t is also a one-dimensional scalar; x 1 and x 2 are all one-dimensional state variables; u is a 2-dimensional input vector.

[0050] Further, in step S13, the NAG differential equation is the denominator of the NAG transfer function. The denominator of the NAG transfer function is the characteristic polynomial of the NAG system, which reflects the dynamic behavior and stability characteristics of the NAG system. From step S34, it can be seen that the differential equation of the motion of the NAG system is expressed as:

[0051] g(μ)s 2 +s+K 0 rg(μ)s+K 0 αr=0

[0052] Furthermore, in step S13, the relationship between the control coefficient in the NAG differential equation and the coefficient in its time domain expression is determined by:

[0053] S41. Construct a proportional-integral controller. The output y(t) of the controller is proportional to the error signal e(τ) and also proportional to the integral of e(τ) over time. Then the output signal of the proportional-integral controller is:

[0054]

[0055] Among them, y(t) is the output of the controller, e(τ) is the error between the reference input and the actual system output, that is, the input of the controller, K 0 is the proportionality coefficient, τ 1 is the integral time constant, t is the time interval from the start of adjustment to the output of the current control quantity, and e(t) is the error between the reference input and the actual system output;

[0056] The relationship between the iteration formula of S42 and NAG system and the proportional integral controller is:

[0057] e(t) in the proportional-integral controller and NAG iteration formula is the current gradient,

[0058] ∫′ in the proportional-integral controller 0 e(τ)dτ and NAG iteration formula is the historical gradient;

[0059] S43, the negative sign of the NAG iteration formula represents the negative direction. From the NAG differential equation, we can see that the order of the NAG differential equation is 2, indicating that the NAG system is a second-order control model with zero points;

[0060] New parameter K introduced in the second-order control model of S44 and NAG system 0 , g(μ) corresponds to the parameters in the NAG iteration formula:

[0061] K 0 =-(1+α)r

[0062]

[0063] Where n is the number of steps of parameter update during the optimization process; α is the momentum coefficient or decay weight.

[0064] Furthermore, in step S14, the phase trajectories of underdamped, critically damped and overdamped systems are established as follows:

[0065] When the NAG system is an underdamped system and a stable system, the eigenvalue of the NAG system is 0<ξ<1, and the phase trajectory of the NAG system is a cluster of convergent logarithmic spirals, and it approaches the stable focus (0, 0) in an oscillating manner along the spirals.

[0066] When the NAG system is a critical damping system, the eigenvalue of the NAG system is ξ=1, and the phase trajectory of the NAG system is a cluster of parabolas tending to the origin of the phase plane and tending to the stable node (0, 0) in a non-positive oscillation manner;

[0067] When the NAG system is an overdamped system, the eigenvalue of the NAG system is ξ>1, and the phase trajectory of the NAG system is a cluster of parabolas tending to the origin of the phase plane and tending to the stable node (0, 0) in a non-positive oscillation manner;

[0068] When the NAG system is a stable system, the eigenvalue of the NAG system is ξ≥1, and the phase trajectory of the NAG system is a cluster of parabolas tending to the origin of the phase plane and tending to the stable point node (0, 0) in a non-positive oscillation manner.

[0069] A second aspect of the present invention provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0070] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0071] A fourth aspect of the present invention provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0072] The beneficial effects of the present invention are as follows:

[0073] 1. The present invention proposes to use dynamic equations to solve the problem of all independent variables changing in the optimizer system, improve the stability and convergence speed of the NAG algorithm, and provide a more reliable and efficient deep learning model training solution.

[0074] 2. The phase plane method proposed in the present invention can visualize the optimization process more intuitively, and guide the selection of hyperparameters by analyzing the gradient, which can effectively improve the performance and robustness of the optimizer, thereby improving the convergence speed and performance of the deep learning model and reducing the complexity of parameter adjustment.

[0075] 3. The present invention uses the phase plane method to analyze the hyperparameter optimization and performance evaluation of NAG, solves the parameter setting problem of the unexplored optimizer, and comprehensively evaluates the performance of the optimized NAG algorithm in different deep learning tasks, thereby promoting the development of artificial intelligence technology in various fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0077] Figure 1 The present invention is a flow chart of a method for hyperparameter optimization and performance evaluation of NAG based on the phase plane method. DETAILED DESCRIPTION

[0078] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0079] like Figure 1As shown, a method for optimizing the hyperparameters and evaluating the performance of NAG based on the phase plane method comprises the following steps:

[0080] S11, based on the process of updating the parameters of the deep learning network by the NAG accelerated gradient descent method (Nesterov Accelerated Gradient), a NAG iteration formula is established, wherein the NAG iteration formula is a discrete form time domain expression of NAG;

[0081] SGD (Stochastic Gradient Descent) is the most common form of parameter optimization in practical applications. SGD selects appropriate initial values ​​and continuously iterates and updates them. Minimize until convergence. When the training data sample size is large, the network parameters can be iterated once by randomly selecting a batch of training data. In order to suppress the oscillation during SGD training, SGD-M adds inertia to the gradient descent process, and introduces the accumulation of historical gradient information by increasing the first-order momentum coefficient, thereby accelerating the SGD learning process. NAG (Newton Accelerated Gradient Descent) is an improvement on SGD and SGD-M, further suppressing the oscillation of SGD staying only at the local optimal point and accelerating the convergence speed.

[0082] In step S11, the steps of establishing the NAG iteration formula are as follows:

[0083] S21: Calculate the loss function at the current moment t with respect to θ t The gradient is:

[0084]

[0085] S22: Calculate the parameters for the next time t+1:

[0086] θ t+1 =θ t -m t+1

[0087] S23, taking a step before calculating the current gradient, the step being to perform a tentative update before updating the parameters to evaluate the effectiveness or stability of the update direction:

[0088]

[0089] but

[0090]

[0091] S24: Calculate momentum:

[0092] m t+1 =αm t +rg t+1

[0093] S25: Calculate the parameters of the next moment t+1 before the current gradient.

[0094]

[0095] Then the NAG iteration formula, that is, the discrete form of the NAG iteration formula in the time domain is:

[0096]

[0097] Where t represents the number of iterations; r is the learning rate, and r>0; L(θ t ) is the parameter θ t The loss function of It is at the estimated parameter position The gradient calculated at θ t is the updated parameter before step t; θ t+1 is the updated parameter of step t+1; α is the momentum coefficient or decay weight, which is used to adjust the weight of historical gradient and current gradient; Indicates the use of momentum term αm at step t t Estimated parameters; g t+1 Indicates that in the forward parameter position The gradient calculated at ; is the gradient of the loss function L with respect to the parameter θ; m t is the momentum vector of the tth step, which is the weighted sum of all the gradients of the previous t steps, attenuated by α; m t+1 is the momentum vector at step t+1; represents the forward-looking parameter position at step t+1, which is obtained by forward-looking adjustment of the parameters using the current momentum term and is used to calculate the gradient in the next step; α t-i represents the momentum coefficient used in step ti; Represents the loss function L for the parameter at the i-th moment The gradient calculated at .

[0098] S12. Based on the NAG iteration formula, a NAG transfer function and a NAG system dynamic model are established. The NAG transfer function is used to describe the input-output relationship of the NAG system, and the NAG system dynamic model is used to solve the problem of expressing the internal structure of the NAG system. Since the NAG differential equation and transfer function only express the input and output relationship of the system, they do not indicate the internal structure of the system. Therefore, the transfer function is calculated based on the NAG iteration formula, and the NAG system dynamic model is established, which not only solves the problem of fully expressing the internal structure of the system, but also maintains the input-output dynamic relationship problem determined by the original NAG transfer function.

[0099] In step S12, the steps of establishing the NAG transfer function are:

[0100] Under zero initial conditions, the ratio of the pull-type transformation of the system output variable to the pull-type transformation of the input variable is called the transfer function of the system, which is recorded as:

[0101]

[0102] Among them, Y(s) and U(s) are the pull transformations of the output variable and input variable respectively.

[0103] S31: Construct the open-loop transfer function of the SGD controller, which refers to the functional relationship between the ratio of the output to the input of an open-loop system and the frequency. A single variable linear steady-state system, whose equation of motion is an n-order linear constant coefficient differential equation, then formula (1) is:

[0104]

[0105] Among them, u(t) and y(t) are input variables and output variables respectively; n≥m is a positive integer, n is the system order; the derivatives of y(t) and u(t) are arranged on both sides of the equal sign in descending order; a i (i=0,1,2,…,n) are the structural parameters of the system, b j (j=0, 1, 2, ..., m) are structural parameters of the input function, all of which are constant coefficients;

[0106] The initial values ​​of y(t) and derivatives of each order are:

[0107]

[0108] According to the definition of transfer function and formula (1), let U(s) = L[u(t)], Y(s) = L[y(t)], the incremental differential equation describes the essence of the dynamic characteristics of the system, differential equation (1) is the incremental equation with the Δ symbol omitted, then formula (2) is:

[0109]

[0110] Use formula (2) and calculate the Laplace transform on both sides of formula (1). According to the differential theorem, when the initial condition is 0, after Laplace transform, formula (3) is:

[0111] (a n s n +a n-1 s n-1 +...+a 1 s+a 0 )Y(s)=

[0112] (b m s m +bm-1 s m-1 +...+b 1 s+b 0 )U(s)(3)

[0113] Then the NAG system transfer function formula (4) is:

[0114]

[0115] For the unit negative feedback system transfer function, the value of H(s) is 1, then the NAG system closed-loop system transfer function formula (5) is:

[0116]

[0117] Formula (5) can be viewed as formula (6):

[0118]

[0119] There are two cases for the decomposition of the NAG transfer function of formula (6):

[0120] For the inner layer of the NAG system transfer function, we have:

[0121]

[0122] For the outer layer of the NAG system transfer function, we have:

[0123] Y(s)=b(s)·Y 1 (s) = b(s)·X 1 (s)

[0124] Based on the above theory, NAG is a transfer function with a numerator of 1, and we derive their dynamic equations. Since the NAG transfer function does not consider the dynamic movement of the internal state of the system, it only describes the relationship between the input and output of the system. Therefore, the dynamic equation is introduced to fully reflect the changes of all independent variables of the system. According to the actual connection of each link of the system, the corresponding NAG dynamic equation is established to realize the mutual conversion between the dynamic equation and transfer function of the NAG system.

[0125] Since the open-loop transfer function of the SGD controller is expressed as:

[0126]

[0127] Among them, K 0 Represents the gain or amplification factor of the control system; r is the learning rate; parameter s is equivalent to the error accumulation effect of the model; in control theory, the gain coefficient determines the degree of response of the system to the error signal.

[0128] S32: SGD-M calculates the gradient first and then adds momentum; NAG adds the momentum term first and then calculates the gradient; the NAG system updates the momentum by using the future gradient, so the G of the NAG system n If set to advance correction,

[0129] The SGD is corrected in advance. The forward correction is that the NAG system updates the momentum by using the future gradient, that is, the G n Set to advance correction, the expression is:

[0130]

[0131] Where α is the lead correction factor (α>0), T 1 is the time constant of the correction factor; then the transfer function of NAG is:

[0132]

[0133] S33: Let g(μ) = αT, then the expression of the open-loop transfer function of the NAG controller is:

[0134]

[0135] Among them, g(μ) is the momentum control function.

[0136] S34: NAG controller is a typical second-order control system with zero point, and its closed-loop transfer function is expressed as:

[0137]

[0138] Since the NAG transfer function does not consider the dynamic movement of the internal state of the system, it only describes the relationship between the input and output of the system. Therefore, the dynamic equation is introduced to fully reflect the changes of all independent variables of the system. According to the actual connection of each link from step S21 to S24, the corresponding NAG dynamic equation is established to realize the mutual conversion between the dynamic equation and transfer function of the NAG system.

[0139] In step S34, the dynamic process of the NAG system is controlled by:

[0140]

[0141] Among them, ω n is the undamped natural oscillation frequency of the NAG system, and ξ is the damping ratio of the NAG system.

[0142] Then the convergence speed and stability of the NAG dynamic equation are controlled by K 0 , r, g(μ) and α, the expression of NAG dynamic equation is:

[0143]

[0144] Among them, x′ 1 is the state variable x 1 The derivative with respect to time t, which is a one-dimensional scalar; x′ 2 is the state variable x 2 The derivative with respect to time t is also a one-dimensional scalar; x 1 and x 2 are all one-dimensional state variables; u is a 2-dimensional input vector.

[0145] S13: Establishing NAG differential equation based on NAG transfer function, and obtaining the relationship between the control coefficient in NAG differential equation and the coefficient in its time domain expression;

[0146] Any system can be described by differential equations. The differential equations of the control system are mathematical models that describe the dynamic performance of the system in the time domain. Under given external actions and initial conditions, solving the differential equations can obtain the analytical expression of the system's output response. Through this analytical expression, the system output response curve can be drawn, which can intuitively reflect the dynamic process of the system and analyze the dynamic characteristics of the system. However, when the structure or a certain parameter of the system changes, the differential equations must be rewritten and solved, which is not conducive to the analysis and design of the system. For this reason, the transfer function of the control system is introduced to characterize the dynamic characteristics of the system, and the transfer function is used to study the influence of the structure or parameter changes of the system on the system performance.

[0147] In step S13, the NAG transfer function describes the relationship between the input and output of the NAG system, and its denominator represents the characteristic equation of the NAG system, reflecting the dynamic characteristics of the NAG system itself. The NAG differential equation is the denominator of the NAG transfer function, and the denominator of the NAG transfer function is the characteristic polynomial of the NAG system, reflecting the dynamic behavior and stability characteristics of the NAG system. From step S34, it can be seen that the differential equation of the NAG system motion is expressed as:

[0148] g(μ)s 2 +s+K 0 rg(μ)s+K 0 αr=0

[0149] In step S13, the relationship between the control coefficient in the NAG differential equation and the coefficient in its time domain expression is determined by:

[0150] S41: Construct a proportional-integral controller. The output y(t) of the controller is proportional to the error signal e(τ) and also proportional to the integral of e(τ) over time. The output signal of the proportional-integral controller is:

[0151]

[0152] Among them, y(t) is the output of the controller, e(τ) is the error between the reference input and the actual system output, that is, the input of the controller, K 0 is the proportionality coefficient, τ 1 is the integral time constant, t is the time interval from the start of adjustment to the output of the current control quantity, and e(t) is the error between the reference input and the actual system output;

[0153] S42: The iterative formula of the NAG system is related to the proportional-integral controller as follows:

[0154] e(t) in the proportional-integral controller and NAG iteration formula is the current gradient,

[0155] ∫′ in the proportional-integral controller 0 e(τ)dτ and NAG iteration formula is the historical gradient;

[0156] S43: The negative sign of the NAG iteration formula represents the negative direction. From the NAG differential equation, we can see that the order of the NAG differential equation is 2, indicating that the NAG system is a second-order control model with zero points;

[0157] New parameter K introduced in the second-order control model of S44 and NAG system 0 , g(μ) corresponds to the parameters in the NAG iteration formula:

[0158] K 0 =-(1+α)r

[0159]

[0160] Where n is the number of steps of parameter update during the optimization process; α is the momentum coefficient or decay weight.

[0161] S14: Based on the NAG differential equation, the phase plane diagram of the NAG system is established to obtain the influence of hyperparameters on the convergence speed and stability of the NAG system; the phase trajectories of underdamped, critically damped and overdamped systems are established to obtain the dynamic behavior of the NAG system under different parameter values, providing guidance for the selection of hyperparameters. The convergence speed and stability of the NAG system are controlled by K 0 , r, g(μ) and α, the phase trajectory method is used to qualitatively discuss the responses of underdamped, overdamped and critically damped NAG systems to the initial state.

[0162] In step S14, the phase trajectory of the underdamped, critically damped and overdamped systems is established as follows:

[0163] When the NAG system is an underdamped system and a stable system, the eigenvalue of the NAG system is 0<ξ<1, and the phase trajectory of the NAG system is a cluster of convergent logarithmic spirals, and it approaches the stable focus (0, 0) in an oscillating manner along the spirals; at this time, the NAG system will oscillate, and the model parameter update will fluctuate greatly, but it will eventually stabilize;

[0164] When the NAG system is a critical damping system, the eigenvalue of the NAG system is ξ=1, the phase trajectory of the NAG system is a cluster of parabolas tending to the origin of the phase plane, and tending to the stable node (0, 0) in a non-positive oscillation manner, and the model parameter update converges quickly and stably;

[0165] When the NAG system is an overdamped system, the eigenvalue of the NAG system is ξ>1, and the phase trajectory of the NAG system is a cluster of parabolas tending to the origin of the phase plane and tending to the stable node (0, 0) in a non-positive oscillation manner. The system response speed is slow but stable.

[0166] When the NAG system is a stable system, the eigenvalue of the NAG system is ξ≥1, and the phase trajectory of the NAG system is a cluster of parabolas tending to the origin of the phase plane and tending to the stable point node (0, 0) in a non-positive oscillation manner.

[0167] The influence of α on the performance and stability of the NAG system is studied through the x-x′ phase trajectory curve.

[0168] in:

[0169]

[0170] when When , NAG is an underdamped system because:

[0171]

[0172] The characteristic values ​​of the NAG system are:

[0173]

[0174] Since the eigenvalues ​​of the NAG state matrix are two complex numbers, and the real parts of the eigenvalues ​​are both:

[0175]

[0176] Then NAG is a stable system, and its equilibrium point is a stable focus. 1 ,λ 2 If there is an imaginary part, the NAG system will vibrate. The phase trajectory of the NAG system will approach the stable focus (0, 0) in an oscillating manner along the spiral line. NAG is an oscillating and decaying system.

[0177] when When , NAG is a critical damping system, and the eigenvalue of the NAG system is:

[0178]

[0179] Since the eigenvalues ​​of the NAG state matrix are two equal negative real numbers, NAG is a stable system, its equilibrium point is a stable node, and the phase trajectory of the NAG system tends to the stable node (0, 0) in a non-positive oscillation manner.

[0180] when When , NAG is an overdamped system because:

[0181]

[0182] λ 1 <0,λ 2 <0, the characteristic value of NAG system is:

[0183]

[0184] Since the eigenvalues ​​of the NAG state matrix are all negative, NAG is a stable system, and its equilibrium point is a stable node. The phase trajectory of the NAG system tends to the stable node (0, 0) in a non-positive oscillation manner.

[0185] The NAG system updates the network parameters at a fixed learning rate. The present invention analyzes the effect of the α value of the NAG system on the stability and rapidity of the model from the phase trajectory method, and analyzes the effect of different hyperparameter settings on the performance of the deep learning model, providing theoretical support for the hyperparameter design of NAG. When using an optimizer to adjust the parameters of the deep learning model, the method of first increasing the order and then reducing the overshoot can be considered.

[0186] First, select smaller parameters to roughly analyze the dynamic response characteristics of the deep learning model.

[0187] Secondly, select larger parameters and analyze the degree of hyperparameter tuning.

[0188] Finally, a compromise is made in selecting control parameters to weaken hyperparameter tuning and improve stability, which requires balancing the joint regulation of appropriate parameters.

[0189] As an optional embodiment, the present invention further provides a computer program product, including a computer program, which implements the steps of the method described in the aforementioned method embodiment when executed by a processor.

[0190] As another optional embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the aforementioned method embodiment are implemented.

[0191] As another optional embodiment, the present invention also provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the aforementioned method embodiment.

[0192] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for hyperparameter optimization and performance evaluation of NAG based on phase plane method, characterized in that: The following steps are involved: S11, based on the process of updating the deep learning network parameters of NAG (Nesterov Accelerated Gradient), a NAG iteration formula is established, wherein the NAG iteration formula is a discrete form time domain expression of NAG; S12, establishing a NAG transfer function and a NAG system dynamic model based on the NAG iterative formula, wherein the NAG transfer function is used to describe the input-output relationship of the NAG system, and the NAG system dynamic model is used to solve the expression problem of the internal structure of the NAG system; S13, establishing a NAG differential equation based on the NAG transfer function, and obtaining the relationship between the control coefficients in the NAG differential equation and the coefficients in its time domain expression; S14. Based on the NAG differential equation, the phase plane diagram of the NAG system is established to obtain the influence of hyperparameters on the convergence speed and stability of the NAG system; the phase trajectories of underdamped, critically damped and overdamped systems are established to obtain the dynamic behavior of the NAG system under different parameter values, providing guidance for the selection of hyperparameters.

2. The method for hyperparameter optimization and performance evaluation of NAG based on phase plane method according to claim 1, characterized in that: In step S11, the steps of establishing the NAG iteration formula are as follows: S21. Calculate the loss function at the current moment t about θ t The gradient is: S22, calculate the next time t+1 parameters: i t+1 =θ t -m t+1 S23, taking a step before calculating the current gradient, the step being to perform a tentative update before updating the parameters to evaluate the effectiveness or stability of the update direction: but S24. Calculate momentum: m t+1 =αm t +rg t+1 S25, calculate the next moment t+1 parameter before the current gradient, Then the NAG iteration formula is: Where t represents the number of iterations; r is the learning rate, and r>0; L(θ t ) is the parameter θ t The loss function of It is at the estimated parameter position The gradient calculated at θ t is the updated parameter before step t; θ t+1 is the updated parameter of step t+1; α is the momentum coefficient or decay weight, which is used to adjust the weight of historical gradient and current gradient; Indicates the use of momentum term αm at step t t Estimated parameters; g t+1 Indicates that in the forward parameter position The gradient calculated at ; is the gradient of the loss function L with respect to the parameter θ; m t is the momentum vector of the tth step, which is the weighted sum of all the gradients of the previous t steps, attenuated by α; m t+1 is the momentum vector at step t+1; represents the forward-looking parameter position at step t+1, which is obtained by forward-looking adjustment of the parameters using the current momentum term and is used to calculate the gradient in the next step; α t-i represents the momentum coefficient used in step ti; Represents the loss function L for the parameter at the i-th moment The gradient calculated at .

3. The method for hyperparameter optimization and performance evaluation of NAG based on phase plane method according to claim 1, characterized in that: In step S12, the steps of establishing the NAG transfer function are: S31. Construct an open-loop transfer function of the SGD controller. The open-loop transfer function refers to the functional relationship between the ratio of the output to the input of an open-loop system and the frequency, and its expression is: Among them, K0 represents the gain or amplification factor of the control system; r is the learning rate; the parameter s is equivalent to the error accumulation effect of the model; S32, make an advance correction to the SGD controller, wherein the advance correction is that NAG updates the momentum by using the future gradient, that is, the G n Set to advance correction, the expression is: Where α is the lead correction factor (α>0), and T1 is the time constant of the correction factor; S33. Let g(μ) = αT, then the expression of the open-loop transfer function of the NAG controller is: Among them, g(μ) is the momentum control function; S34 and NAG controllers are typical second-order control systems with zero points, and their closed-loop transfer functions are expressed as follows:

4. The method for hyperparameter optimization and performance evaluation of NAG based on phase plane method according to claim 3, characterized in that: In step S34, the dynamic process of the NAG system is controlled by: and Among them, ω n is the undamped natural oscillation frequency of the NAG system, ξ is the damping ratio of the NAG system; The convergence speed and stability of the NAG dynamic equation are controlled by K0, r, g(μ) and α. The expression of the NAG dynamic equation is: Among them, x′1 is the derivative of the state variable x1 with respect to time t, which is a one-dimensional scalar; x2 is the derivative of the state variable x2 with respect to time t, which is also a one-dimensional scalar; x1 and x2 are both one-dimensional state variables; u is a two-dimensional input vector.

5. The method for hyperparameter optimization and performance evaluation of NAG based on phase plane method according to claim 3, characterized in that: In step S13, the NAG differential equation is the denominator of the NAG transfer function. The denominator of the NAG transfer function is the characteristic polynomial of the NAG system, which reflects the dynamic behavior and stability characteristics of the NAG system. From step S34, it can be seen that the differential equation of the NAG system motion is expressed as: g(μ)s 2 +s+K0rg(μ)s+K0αr=0.

6. The method for hyperparameter optimization and performance evaluation of NAG based on phase plane method according to claim 2, characterized in that: In step S13, the relationship between the control coefficient in the NAG differential equation and the coefficient in its time domain expression is determined by: S41. Construct a proportional-integral controller. The output y(t) of the controller is proportional to the error signal e(τ) and also proportional to the integral of e(τ) over time. Then the output signal of the proportional-integral controller is: Among them, y(t) is the output of the controller, e(τ) is the error between the reference input and the actual system output, that is, the input of the controller, K0 is the proportional coefficient, τ1 is the integral time constant, t is the time interval from the beginning of adjustment to the output of the current control quantity, and e(t) is the error between the reference input and the actual system output; The relationship between the iteration formula of S42 and NAG system and the proportional integral controller is: e(t) in the proportional-integral controller and NAG iteration formula is the current gradient, ∫′0e(τ)dτ in the proportional-integral controller and NAG iterative formula is the historical gradient; S43, the negative sign of the NAG iteration formula represents the negative direction. From the NAG differential equation, we can see that the order of the NAG differential equation is 2, indicating that the NAG system is a second-order control model with zero points; S44. The corresponding relationship between the new parameters K0, g(μ) introduced in the second-order control model of the NAG system and the parameters in the NAG iteration formula is: K0＝-(1+α)r Where n is the number of steps of parameter update during the optimization process; α is the momentum coefficient or decay weight.

7. The method for hyperparameter optimization and performance evaluation of NAG based on phase plane method according to claim 4, characterized in that: In step S14, the phase trajectory of the underdamped, critically damped and overdamped systems is established as follows: When the NAG system is an underdamped system and a stable system, the eigenvalue of the NAG system is 0<ξ<1, and the phase trajectory of the NAG system is a cluster of convergent logarithmic spirals, and it approaches the stable focus (0, 0) in an oscillating manner along the spirals. When the NAG system is a critical damping system, the eigenvalue of the NAG system is ξ=1, and the phase trajectory of the NAG system is a cluster of parabolas tending to the origin of the phase plane and tending to the stable node (0, 0) in a non-positive oscillation manner; When the NAG system is an overdamped system, the eigenvalue of the NAG system is ξ>1, and the phase trajectory of the NAG system is a cluster of parabolas tending to the origin of the phase plane and tending to the stable node (0, 0) in a non-positive oscillation manner; When the NAG system is a stable system, the eigenvalue of the NAG system is ξ≥1, and the phase trajectory of the NAG system is a cluster of parabolas tending to the origin of the phase plane and tending to the stable point node (0, 0) in a non-positive oscillation manner.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for hyperparameter optimization and performance evaluation of NAG based on the phase plane method described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for hyperparameter optimization and performance evaluation of NAG based on the phase plane method described in any one of claims 1 to 7 are implemented.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the hyperparameter optimization and performance evaluation method of NAG based on the phase plane method as described in any one of claims 1 to 7.

Images