SGD-M optimizer control order regulation and control method based on transfer function

Abstract

The invention provides an SGD-M optimizer control order regulation and control method based on a transfer function, and the method comprises the following steps: S1, carrying out the deduction according to the process of updating a network parameter theta of SGD-M, and obtaining a discrete-form time domain expression of the SGD-M; s2, through a time domain expression of the SGD-M, deriving to obtain a transfer function of the SGD-M; s3, deducing a coefficient relationship between the regulation and control coefficient and the time domain expression through an SGD-M transfer function; and S4, through the regulation and control coefficient of the SGD-M transfer function, analyzing the convergence speed and stability of the SGD-M transfer function to network training. According to the method, the transfer function and the dynamic equation of the optimizer are established, the control model order of the optimizer is provided, and then the property of the influence of the regulation and control coefficient under different optimizer control model orders on the dynamic performance of the system is given. According to the method, mutual crossing fusion between a deep learning optimization algorithm and a control theory is successfully realized, a novel description method for network process analysis is expanded, and important theoretical support is provided for optimization problems in engineering application.

Description

Technical Field

[0001] The present invention relates to the technical field of optimizer order regulation, and in particular to a SGD-M optimizer control order regulation method based on transfer function. Background Art

[0002] In recent years, with the widespread application of deep learning in various fields, the research on optimizers has gradually become an important topic in the field of artificial intelligence. Optimizers are basic tools for adjusting model parameters and are an indispensable part of deep learning network training. Optimizers are used to calculate network parameters that affect deep learning model training and model output, so that they approach the optimal value, thereby minimizing the loss function. The optimizer in deep learning is an algorithm used in the backpropagation algorithm, which can adjust the parameters of the deep learning network model so that the deep learning network performs better on the training set. Stochastic gradient descent (SGD) and its variants (such as momentum stochastic gradient descent SGD-M) are one of the most commonly used optimizers and are widely used because of their high computational efficiency and simple implementation. However, the SGD-M algorithm faces the problem of balancing convergence speed and stability during training, which is particularly prominent in the training of large-scale data and complex models.

[0003] SGD-M is derived from the momentum improvement of SGD. The main idea is to introduce a momentum that accumulates historical gradient information to accelerate the SGD learning process. In order to suppress the oscillation of SGD, SGD-M adds inertia to the gradient descent process. SGD-M introduces a momentum term on the basis of traditional SGD to reduce gradient fluctuations, thereby accelerating convergence and improving stability. However, in practical applications, the performance of SGD-M is affected by multiple parameters (such as learning rate, momentum factor, etc.). Improper parameter selection may cause algorithm oscillation, slow convergence or even failure to converge. Therefore, how to scientifically adjust the parameters of SGD-M to optimize algorithm performance is a hot topic and difficulty in current research.

[0004] The optimizer adjusts the parameters of the deep learning model in a step-by-step feedback manner. First, the data is input into the model for forward propagation. During the forward propagation process, the input data is processed by convolution and pooling of multiple layers of convolutional layers to extract feature vectors, which are passed to the fully connected layer and then output labels. Secondly, the error between the model's predicted output value and the true target value is calculated by the loss function. The deep learning network uses the obtained loss value as a feedback signal and sends it to the optimizer for calculation. Finally, the convolution kernel is updated by the optimizer to achieve back propagation. Therefore, understanding the optimizer is the basis for improving the training efficiency of deep learning models. The optimization algorithms for deep learning have different orders and have different effects on the training process of the network. Among them, momentum is a function related to the gradient itself, and the difference of momentum is a function related to the speed of the gradient. The network parameters are updated by selecting different momentums. When using high-order algorithms, the current work has the following difficulties:

[0005] (1) When using the Hessian matrix method, it is difficult to fit the function expression of the deep learning network and the matrix is ​​difficult to solve; (2) Even if the gradient and Hessian matrix are obtained, when the dimension of the input vector is large, the memory required for calculation is large; (3) When using the robust stochastic gradient descent algorithm based on t distribution, it is easy to fall into a saddle point when looking for a point where the gradient is zero.

[0006] The optimizer reduces the value of the loss function by iterating the model parameters a finite number of times. Research on the order of the gradient descent optimizer has been carried out with the improvement of the control coefficient. However, there is little research on the impact of the control coefficient on the training process. The lack of necessary parameter range pre-selection tools has restricted the scientific application of optimizers in deep learning networks.

[0007] As an important tool in control theory, transfer function can systematically analyze and design the behavior of dynamic systems. Based on the transfer function method, the SGD-M optimization process can be regarded as a dynamic system, and its order can be controlled by adjusting the system parameters, thereby achieving precise control of the dynamic response characteristics of the system.

[0008] The present invention proposes a control order regulation method for an SGD-M optimizer based on a transfer function. By introducing a transfer function model and analyzing the dynamic characteristics of the SGD-M system, a set of scientific parameter regulation methods is proposed, aiming to improve the convergence speed and stability of SGD-M, solve the problems existing in the prior art, and provide a new theoretical basis and technical means for deep learning optimization algorithms. Summary of the invention

[0009] In view of this, the object of the present invention is to provide a SGD-M optimizer control order regulation method based on transfer function to at least solve the above problems.

[0010] The technical solution adopted by the present invention is as follows:

[0011] A transfer function-based SGD-M optimizer control order regulation method, the method comprising the following steps:

[0012] S1. According to the process of updating the network parameter θ by SGD-M, its discrete form time domain expression is derived;

[0013] S2. Through the time domain expression of SGD-M, derive its transfer function;

[0014] S3. Through the SGD-M transfer function, derive the relationship between its control coefficient and the coefficient of the time domain expression;

[0015] S4. Analyze the convergence speed and stability of network training through the control coefficient of the SGD-M transfer function.

[0016] Furthermore, in step S1, according to the process of updating the network parameter θ by SGD-M, its discrete form time domain expression is derived as follows:

[0017] The time domain expression of SGD-M is:

[0018]

[0019] Where t represents the number of iterations, r is the learning rate, and r>0, L(θ t ) is the parameter θ of L t The loss function is is the parameter θ at time t t The loss gradient, θ t is the parameter of the tth round (parameter before update), θ t+1 is the parameter of the t+1th round; α is the decay weight, which is used to regulate the weight of the historical gradient and the current gradient. Its initialization setting is 0.9, α∈[0,1].

[0020] Furthermore, in step S2, the transfer function of SGD-M is derived through the time domain expression:

[0021] The proportional integral controller PI, whose output y(t) is proportional to the error signal e(τ) and at the same time proportional to the integral of e(τ) over time, is expressed as

[0022]

[0023] Among them, y(t) is the output signal of the controller, e(τ) is the error between the reference input and the actual system output, that is, the input of the controller, K0 is the proportional coefficient, τ is the integral time constant, and t is the time interval from the beginning of adjustment to the output of the current control quantity;

[0024] Since SGD-M is equivalent to the proportional integral PI link in the PID controller, there is a decay term α in the integral term. The introduction of α is to make the gradient away from the current value to reduce noise. The iterative format of SGD-M considers the relationship between the next moment and the previous moment. The relationship between SGD-M formula (1) and PI formula (2) is: the current gradient, Historical gradient accumulation term, According to the formula

[0025]

[0026] The momentum factor

[0027]

[0028] In SGD-M, the inertia link is introduced through the momentum control function g(μ), making SGD-M equivalent to a second-order control system. In order to derive the closed-loop transfer function of SGD-M, the open-loop transfer function is analyzed.

[0029] Since the SGD open-loop transfer function is

[0030]

[0031] Let T = g(μ), then T is a function of g(μ), and the SGD-M open-loop transfer function is

[0032]

[0033] Among them, g(μ) is the momentum control function. The open-loop transfer function of SGD-M is obtained by adding an inertia link on the basis of the SGD open-loop transfer function. Then the SGD-M controller is a second-order control system, and the SGD-M closed-loop transfer function is

[0034] .

[0036] Furthermore, in step S3, the relationship between the control coefficient and the coefficient of the time domain expression is derived through the SGD-M transfer function:

[0037] There is a relationship between the coefficients of the SGD-M transfer function and the time domain expression of SGD-M:

[0038]

[0039] The negative sign in the SGD-M iterative format represents the negative direction. From formula (4), we know that the order of the denominator of the SGD-M transfer function is 2, and SGD-M is a second-order control model without zero points. Therefore, the corresponding relationship between the new parameters K0, g(μ) introduced by the SGD-M control model and the parameters in the SGD-M iterative format is:

[0040] .

[0042] Furthermore, in step S4, the convergence speed and stability of SGD-M are derived through the control coefficient of the SGD-M transfer function as follows:

[0043] Parameters ξ and ω n The dynamic response characteristics of the second-order system are determined. The convergence speed and stability of SGD-M are controlled by the parameters K0, r and g(μ). The phase trajectory method is used to qualitatively discuss the response of the SGD-M system to the initial state. The influence of different values ​​of the parameters K0, r and g(μ) on the SGD-M eigenvalues ​​λ1 and λ2 is discussed in a classified manner, and then the influence of the eigenvalues ​​on the performance and stability of the system is analyzed.

[0044] When K0rg(μ)>0.25, SGD-M is an underdamped system, the system is stable, its equilibrium point is the stable focus, and it is accompanied by oscillation phenomenon;

[0045] When K0rg(μ)=0.25, SGD-M is a critical damping system, the system is stable, and its equilibrium point is a stable node;

[0046] When 0<K0rg(μ)<0.25, SGD-M is an overdamped system, the system is stable, and its equilibrium point is a stable node.

[0047] Furthermore, when K0rg(μ)>0.25, the damping ratio ξ satisfies 0<ξ<1. At this time, SGD-M is an underdamped system. Then the SGD-M system continues to vibrate and the amplitude gradually decreases and eventually stops. The characteristic equation of SGD-M is:

[0048] g(μ)s 2 +s+K0r=0

[0049]

[0050] but

[0051]

[0052] The eigenvalues ​​of the SGD-M state matrix are two complex numbers, and the real parts of the eigenvalues ​​are

[0053]

[0054] Then SGD-M is a stable system, and its equilibrium point is a stable focus. Since the eigenvalues ​​λ1 and λ2 have imaginary parts, the SGD-M system vibrates. The phase trajectory of the SGD-M system will approach the stable focus (0,0) in an oscillating manner along the spiral line, showing a system that oscillates and decays at the same time.

[0055] Furthermore, when K0rg(μ)=0.25, ξ=1, and SGD-M is a critical damping system. From formula (7), we can get the eigenvalue of the SGD-M system:

[0056]

[0057] At this time, the eigenvalues ​​of the SGD-M state matrix are two equal negative real numbers, then SGD-M is a stable system, its equilibrium point is a stable node, the phase trajectory of the SGD-M system will tend to the stable node (0,0) along a parabola in a non-positive oscillation manner, and the two state variables x1(t) and x2(t) of the SGD-M system converge to the equilibrium point at the same speed.

[0058] Furthermore, when 0<K0rg(μ)<0.25, ξ>1, then SGD-M is an overdamped system, due to

[0059]

[0060] From formula (7), we know that λ1<0, λ2<0, and its eigenvalue

[0061]

[0062] At this time, the eigenvalues ​​of the SGD-M state matrix are all negative, then SGD-M is a stable system, its equilibrium point is a stable node, the phase trajectory of the SGD-M system will approach the stable node (0,0) along a non-oscillatory path, and the two state variables x1(t) and x2(t) of the system converge to the equilibrium point at different speeds.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] The present invention proposes a control order regulation method for an SGD-M optimizer based on a transfer function. Aiming at the problems of how to use the optimizer and design the optimal parameters of the optimizer in a specific optimization problem, compared with the parameter setting scheme relying on experience and random selection, a control order regulation method for an SGD-M gradient descent optimizer is proposed.

[0065] This paper proposes a control order regulation method for SGD-M optimizer based on transfer function. Based on the transfer function and dynamic equation of the optimizer, the control model order of the optimizer is given, and then the nature of the influence of the regulation coefficient under different optimizer control model orders on the dynamic performance of the system is given. This study successfully cross-integrated deep learning optimization algorithms with control theory, expanded the description method of network process analysis, and provided a solid theory for engineering application problems.

[0066] The transfer function-based SGD-M optimizer control order regulation method proposed in the present invention not only provides a novel optimization algorithm regulation method, but also provides theoretical support and practical application value for the training of deep learning models on large-scale data and complex tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only preferred embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 Schematic diagram of the connection between SGD-M and proportional integral link in an embodiment of the present invention.

[0069] Figure 2 Schematic diagram of SGD-M phase trajectory curve when the parameter g(μ) takes different values ​​in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The enumerated embodiments are only used to explain the present invention and are not used to limit the scope of the present invention.

[0071] The present invention provides a SGD-M optimizer control order regulation method based on transfer function, the method comprising the following steps:

[0072] S1. According to the process of updating the network parameter θ by SGD-M, its discrete form time domain expression is derived;

[0073] S2. Through the time domain expression of SGD-M, derive its transfer function;

[0074] S3. Through the SGD-M transfer function, derive the relationship between its control coefficient and the coefficient of the time domain expression;

[0075] S4. Analyze the convergence speed and stability of network training through the control coefficient of the SGD-M transfer function.

[0076] In step S1, according to the process of SGD-M updating the network parameter θ, its discrete form time domain expression is derived as follows:

[0077] The time domain expression of SGD-M is:

[0078]

[0079] Where t represents the number of iterations, r is the learning rate, and r>0, L(θ t ) is the parameter θ of L t The loss function is is the parameter θ at time t t The loss gradient, θ t is the parameter of the tth round (parameter before update), θ t+1 is the parameter of the t+1th round; α is the decay weight, which is used to regulate the weight of the historical gradient and the current gradient. Its initialization setting is 0.9, α∈[0,1].

[0080] In step S2, the transfer function of SGD-M is derived through the time domain expression:

[0081] The proportional integral controller PI, whose output y(t) is proportional to the error signal e(τ) and at the same time proportional to the integral of e(τ) over time, is expressed as

[0082]

[0083] Among them, y(t) is the output signal of the controller, e(τ) is the error between the reference input and the actual system output, that is, the input of the controller, K0 is the proportional coefficient, τ is the integral time constant, and t is the time interval from the beginning of adjustment to the output of the current control quantity;

[0084] Since SGD-M is equivalent to the proportional integral PI link in the PID controller, there is a decay term α in the integral term. The introduction of α is to make the gradient away from the current value to reduce noise. The iterative format of SGD-M considers the relationship between the next moment and the previous moment. The relationship between SGD-M formula (1) and PI formula (2) is: the current gradient, Historical gradient accumulation term, According to the formula

[0085]

[0086] The momentum factor

[0087]

[0088] In SGD-M, the inertia link is introduced through the momentum control function g(μ), making SGD-M equivalent to a second-order control system. In order to derive the closed-loop transfer function of SGD-M, the open-loop transfer function is analyzed.

[0089] Since the SGD open-loop transfer function is

[0090]

[0091] Let T = g(μ), then T is a function of g(μ), and the SGD-M open-loop transfer function is

[0092]

[0093] Among them, g(μ) is the momentum control function. The open-loop transfer function of SGD-M is obtained by adding an inertia link on the basis of the SGD open-loop transfer function. Then the SGD-M controller is a second-order control system, and the SGD-M closed-loop transfer function is

[0094] .

[0096] Exemplary, the relationship between SGD-M formula (1) and PI formula (2): Current gradient, Historical gradient, Therefore, it has two characteristics: (1) selecting historical gradients and current gradients for weight calculation, that is, there is a certain delay / lag for the current input error signal; (2) the integral link is attenuated to a certain extent, and the iteration is more stable. It can be seen that SGD-M is equivalent to adding an inertia link on the basis of the SGD method.

[0097] In step S3, the relationship between the control coefficient and the coefficient of the time domain expression is derived through the SGD-M transfer function:

[0098] There is a relationship between the coefficients of the SGD-M transfer function and the time domain expression of SGD-M:

[0099]

[0100] The negative sign in the SGD-M iterative format represents the negative direction. From formula (4), we know that the order of the denominator of the SGD-M transfer function is 2, and SGD-M is a second-order control model without zero points. Therefore, the corresponding relationship between the new parameters K0, g(μ) introduced by the SGD-M control model and the parameters in the SGD-M iterative format is:

[0101] .

[0103] In step S4, the convergence speed and stability of SGD-M are derived through the control coefficient of the SGD-M transfer function as follows:

[0104] Refer to Table 1 for parameters ξ and ω n The dynamic response characteristics of the second-order system are determined. The convergence speed and stability of SGD-M are controlled by the parameters K0, r and g(μ). The phase trajectory method is used to qualitatively discuss the response of the SGD-M system to the initial state. The influence of different values ​​of the parameters K0, r and g(μ) on the SGD-M eigenvalues ​​λ1 and λ2 is discussed in a classified manner, and then the influence of the eigenvalues ​​on the performance and stability of the system is analyzed.

[0105] When K0rg(μ)>0.25, SGD-M is an underdamped system, the system is stable, its equilibrium point is the stable focus, and it is accompanied by oscillation phenomenon;

[0106] When K0rg(μ)=0.25, SGD-M is a critical damping system, the system is stable, and its equilibrium point is a stable node;

[0107] When 0<K0rg(μ)<0.25, SGD-M is an overdamped system, the system is stable, and its equilibrium point is a stable node.

[0108] Table 1 Determining the type and stability of system equilibrium points by the eigenvalue of A

[0109]

[0110] Reference Figure 2 (a) When K0rg(μ)>0.25, the damping ratio ξ satisfies 0<ξ<1. At this time, SGD-M is an underdamped system. The SGD-M system vibrates continuously and the amplitude gradually decreases and eventually stops. The characteristic equation of SGD-M is:

[0111]

[0112] but

[0113]

[0114] The eigenvalues ​​of the SGD-M state matrix are two complex numbers, and the real parts of the eigenvalues ​​are

[0115]

[0116] Then SGD-M is a stable system, and its equilibrium point is a stable focus. Since the eigenvalues ​​λ1 and λ2 have imaginary parts, the SGD-M system vibrates. The phase trajectory of the SGD-M system will approach the stable focus (0,0) in an oscillating manner along the spiral line, showing a system that oscillates and decays at the same time.

[0117] Reference Figure 2(b) When K0rg(μ)=0.25, ξ=1, SGD-M is a critical damping system. From formula (7), we can get the eigenvalue of SGD-M system:

[0118]

[0119] At this time, the eigenvalues ​​of the SGD-M state matrix are two equal negative real numbers, then SGD-M is a stable system, its equilibrium point is a stable node, the phase trajectory of the SGD-M system will tend to the stable node (0,0) along a parabola in a non-positive oscillation manner, and the two state variables x1(t) and x2(t) of the SGD-M system converge to the equilibrium point at the same speed.

[0120] Reference Figure 2 (c) When 0<K0rg(μ)<0.25, ξ>1, then SGD-M is an overdamped system, because

[0121]

[0122] From formula (7), we know that λ1<0, λ2<0, and its eigenvalue

[0123]

[0124] At this time, the eigenvalues ​​of the SGD-M state matrix are all negative, then SGD-M is a stable system, its equilibrium point is a stable node, the phase trajectory of the SGD-M system will approach the stable node (0,0) along a non-oscillatory path, and the two state variables x1(t) and x2(t) of the system converge to the equilibrium point at different speeds.

[0125] The influence of g(μ) on the performance and stability of the SGD-M system is studied through the x—x′ phase trajectory curve.

[0126] Figure 2 x represents position and x' represents velocity. The x-x' plane is also called the phase plane. Given a number of initial conditions, the x-x' plane will form a number of phase trajectories, which constitute a plane graph. The differential equation of motion of the SGD-M system is g(μ)s 2 +s+K0r=0, then under different initial conditions, the differential equation can obtain different solutions of x and x', and x and x' form Figure 2 There are multiple different curves drawn on the plane shown, each curve (also called phase trajectory) represents a different motion of the SGD-M system. The phase trajectory point represents a state of the system, and the phase trajectory represents the change of the motion state of the system over time, representing a solution of the differential equation under given initial conditions. The arrow on the phase trajectory indicates the direction of movement of the phase trajectory point as time increases. Figure 2In the example, the parameters K0 and r are quantitative (K0 = 50, r = 0.01), and g(μ) is 0.93, 0.5, and 0.07, respectively, corresponding to the underdamped SGD system ( Figure 2 (a)), critical damping ( Figure 2 (b)), overdamping ( Figure 2 (c)). Figure 2 The phase trajectory of (a) is a cluster of converging logarithmic spirals, and the process of moving toward the equilibrium state is a periodic oscillation process. Figure 2 The phase trajectory of (b,c) is a group of parabolas tending to the origin of the phase plane, and when x′>0, the parabola opens to the left. Similarly, when x′<0, the parabola opens to the right. Figure 2 The process of (b,c) moving toward equilibrium is a non-periodic oscillation process. Since the SGD-M system movement converges to (0,0), the system is stable. This is consistent with the theoretical analysis of Corollary 1.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A control order regulation method of SGD-M optimizer based on transfer function, characterized in that: The method comprises the following steps: S1. According to the process of updating the network parameter θ by SGD-M, its discrete form time domain expression is derived; S2. Through the time domain expression of SGD-M, derive its transfer function; S3. Through the SGD-M transfer function, derive the relationship between its control coefficient and the coefficient of the time domain expression; S4. Analyze the convergence speed and stability of network training through the control coefficient of the SGD-M transfer function.

2. The control order regulation method of the SGD-M optimizer based on transfer function according to claim 1, characterized in that: In step S1, according to the process of SGD-M updating the network parameter θ, its discrete form time domain expression is derived as follows: The time domain expression of SGD-M is: Where t represents the number of iterations, r is the learning rate, and r>0, L(θ t ) is the parameter θ of L t The loss function is is the parameter θ at time t t The loss gradient, θ t is the parameter of the tth round (parameter before update), θ t+1 is the parameter of the t+1th round; α is the decay weight, which is used to regulate the weight of the historical gradient and the current gradient. Its initialization setting is 0.9, α∈[0,1].

3. The control order regulation method of the SGD-M optimizer based on transfer function according to claim 2 is characterized in that: In step S2, the transfer function of SGD-M is derived through the time domain expression: The proportional integral controller PI, whose output y(t) is proportional to the error signal e(τ) and at the same time proportional to the integral of e(τ) over time, is expressed as Among them, y(t) is the output signal of the controller, e(τ) is the error between the reference input and the actual system output, that is, the input of the controller, K0 is the proportional coefficient, τ is the integral time constant, and t is the time interval from the beginning of adjustment to the output of the current control quantity; Since SGD-M is equivalent to the proportional integral PI link in the PID controller, there is a decay term α in the integral term. The introduction of α is to make the gradient away from the current value to reduce noise. The iterative format of SGD-M considers the relationship between the next moment and the previous moment. The relationship between SGD-M formula (1) and PI formula (2) is: the current gradient, Historical gradient accumulation term, According to the formula The momentum factor In SGD-M, the inertia link is introduced through the momentum control function g(μ), making SGD-M equivalent to a second-order control system. In order to derive the closed-loop transfer function of SGD-M, the open-loop transfer function is analyzed. Since the open-loop transfer function of SGD is Let T = g(μ), then T is a function of g(μ), and the SGD-M open-loop transfer function is Among them, g(μ) is the momentum control function. The open-loop transfer function of SGD-M is obtained by adding an inertia link on the basis of the SGD open-loop transfer function. Then the SGD-M controller is a second-order control system, and the SGD-M closed-loop transfer function is 。 4. The control order regulation method of the SGD-M optimizer based on transfer function according to claim 3 is characterized in that: In step S3, the relationship between the control coefficient and the coefficient of the time domain expression is derived through the SGD-M transfer function: There is a relationship between the coefficients of the SGD-M transfer function and the time domain expression of SGD-M: The negative sign in the SGD-M iterative format represents the negative direction. From formula (4), we know that the order of the denominator of the SGD-M transfer function is 2, and SGD-M is a second-order control model without zero points. Therefore, the corresponding relationship between the new parameters K0, g(μ) introduced by the SGD-M control model and the parameters in the SGD-M iterative format is:

5. The control order regulation method of the SGD-M optimizer based on transfer function according to claim 1, characterized in that: In step S4, the convergence speed and stability of SGD-M are derived through the control coefficient of the SGD-M transfer function: Parameters ξ and ω n The dynamic response characteristics of the second-order system are determined. The convergence speed and stability of SGD-M are controlled by the parameters K0, r and g(μ). The phase trajectory method is used to qualitatively discuss the response of the SGD-M system to the initial state. The influence of different values ​​of the parameters K0, r and g(μ) on the SGD-M eigenvalues ​​λ1 and λ2 is discussed in a classified manner, and then the influence of the eigenvalues ​​on the performance and stability of the system is analyzed. When K0rg(μ)>0.25, SGD-M is an underdamped system, the system is stable, its equilibrium point is the stable focus, and it is accompanied by oscillation phenomenon; When K0rg(μ)=0.25, SGD-M is a critical damping system, the system is stable, and its equilibrium point is a stable node; When 0<K0rg(μ)<0.25, SGD-M is an overdamped system, the system is stable, and its equilibrium point is a stable node.

6. The control order regulation method of the SGD-M optimizer based on transfer function according to claim 5, characterized in that: When K0rg(μ)>0.25, the damping ratio ξ satisfies 0<ξ<1. At this time, SGD-M is an underdamped system. Then the SGD-M system vibrates continuously and the amplitude gradually decreases and finally stops. The characteristic equation of SGD-M is: but The eigenvalues ​​of the SGD-M state matrix are two complex numbers, and the real parts of the eigenvalues ​​are Then SGD-M is a stable system, and its equilibrium point is a stable focus. Since the eigenvalues ​​λ1 and λ2 have imaginary parts, the SGD-M system vibrates. The phase trajectory of the SGD-M system will approach the stable focus (0,0) in an oscillating manner along the spiral line, showing a system that oscillates and decays at the same time.

7. The control order regulation method of the SGD-M optimizer based on transfer function according to claim 5, characterized in that: When K0rg(μ)=0.25, ξ=1, SGD-M is a critical damping system. From formula (7), we can get the eigenvalue of the SGD-M system: At this time, the eigenvalues ​​of the SGD-M state matrix are two equal negative real numbers, then SGD-M is a stable system, its equilibrium point is a stable node, the phase trajectory of the SGD-M system will tend to the stable node (0,0) along a parabola in a non-positive oscillation manner, and the two state variables x1(t) and x2(t) of the SGD-M system converge to the equilibrium point at the same speed.

8. The control order regulation method of the SGD-M optimizer based on transfer function according to claim 5, characterized in that: When 0<K0rg(μ)<0.25, ξ>1, then SGD-M is an overdamped system, because From formula (7), we know that λ1<0, λ2<0, and its eigenvalue At this time, the eigenvalues ​​of the SGD-M state matrix are all negative, then SGD-M is a stable system, its equilibrium point is a stable node, the phase trajectory of the SGD-M system will approach the stable node (0,0) along a non-oscillatory path, and the two state variables x1(t) and x2(t) of the system converge to the equilibrium point at different speeds.

Images