Hydraulic multi-way valve intelligent shaft control method based on reinforcement learning

Abstract

The application discloses a hydraulic multi-way valve intelligent shaft control method based on reinforcement learning, which is based on an execution-evaluation neural network reinforcement learning framework, fuses a dynamic surface control thought, and designs a position shaft control intelligent controller which considers unknown friction dynamic active compensation. For the position shaft control problem of the hydraulic multi-way valve, the application can not only guarantee active learning compensation of unknown friction dynamics of the system, improve the anti-interference ability and intelligent level of the system, but also can avoid the problem of differential explosion in traditional backstepping control of an electro-hydraulic system, reduce the influence of measurement noise on control precision, and realize asymptotic tracking performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electromechanical servo control, and particularly relates to an intelligent hydraulic multi-way valve shaft control method based on reinforcement learning (RLAC). BACKGROUND

[0002] The hydraulic multi-way valve shaft control system has a great position in the fields of robots, heavy machinery and high-performance loading test equipment due to its large power density, large force / torque output and fast dynamic response. The hydraulic multi-way valve shaft control system is a typical nonlinear system, which contains many nonlinear characteristics and modeling uncertainties. The nonlinear characteristics include input nonlinearities such as hysteresis and saturation, multi-way valve flow pressure nonlinearities, friction nonlinearities, etc., and the modeling uncertainties include parameter uncertainties such as load mass, viscous friction coefficient of actuators, leakage coefficient, multi-way valve flow gain, hydraulic oil elastic modulus, etc., and uncertain nonlinearities such as unmodeled friction dynamics, system high-order dynamics, external disturbances and unmodeled leakage, etc. When the hydraulic multi-way valve shaft control system develops towards high precision and high frequency response, the nonlinear characteristics of the system have a more significant impact on the system performance, and the existence of modeling uncertainties will make the controller designed based on the nominal model of the system unstable or reduced order. Therefore, the nonlinear characteristics and modeling uncertainties of the hydraulic multi-way valve shaft control system are important factors that limit the performance improvement of the system. With the continuous progress of technology in the industrial and defense fields, the controllers based on traditional linear theory in the past have gradually failed to meet the high performance requirements of the system, so it is necessary to study more advanced nonlinear control strategies for the nonlinear characteristics in the hydraulic multi-way valve shaft control system.

[0003] For the nonlinear control problem of hydraulic multi-way valve shaft control system, many methods have been proposed. Among them, the adaptive control method is very effective for dealing with parameter uncertainty problems, and can obtain asymptotic tracking steady-state performance, but it is not good at dealing with uncertain nonlinearities such as external load disturbances, and when the uncertain nonlinearities are too large, the system may be unstable. Therefore, the adaptive control method cannot obtain high-precision control performance in practical applications. As a robust control method, the classical sliding mode control can effectively deal with any bounded modeling uncertainty and obtain asymptotic tracking steady-state performance, but the discontinuous controller designed by the classical sliding mode control can easily cause the chattering problem of the sliding surface, thereby deteriorating the tracking performance of the system. In order to solve the problems of parameter uncertainty and uncertain nonlinearity at the same time, an adaptive robust control method is proposed. This control method can make the system obtain certain transient and steady-state performance when both kinds of modeling uncertainty exist. In order to obtain high-precision tracking performance, the feedback gain must be increased to reduce the tracking error. Due to the existence of measurement noise, the gain is too large, which often leads to high-gain feedback and causes chattering of the control input, thereby deteriorating the control performance and even causing the system to be unstable. SUMMARY

[0004] The purpose of the present application is to provide a hydraulic multi-way valve intelligent shaft control method with intelligent learning ability, strong anti-interference ability and high tracking performance, which can not only ensure active learning compensation for unknown friction of the system, improve the anti-interference ability and intelligent level of the system, but also avoid the problem of differential explosion in traditional backstepping control of electro-hydraulic systems, reduce the influence of measurement noise on control precision, and realize asymptotic tracking performance.

[0005] The technical solution for achieving the purpose of the present application is: a hydraulic multi-way valve intelligent shaft control method based on reinforcement learning, comprising the following steps:

[0006] Step 1, establish a mathematical model of the hydraulic multi-way valve shaft control system, and go to step 2.

[0007] Step 2, based on the mathematical model of the hydraulic multi-way valve shaft control system, design a position shaft control controller based on reinforcement learning, and go to step 3.

[0008] Step 3, use Lyapunov stability theory to prove the stability of the position shaft control controller, and obtain the result that the system tracking error is asymptotically stable.

[0009] Compared with the prior art, the present application has the following advantages: (1) high intelligent level; (2) unknown disturbance active compensation; (3) high precision tracking performance; and (4) strong anti-interference ability. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a schematic diagram of the principle of the hydraulic multi-way valve intelligent shaft control method based on reinforcement learning.

[0011] Figure 2 is a schematic diagram of the principle of the hydraulic multi-way valve shaft control system.

[0012] Figure 3 is a tracking process curve diagram of the system output to the expected command under the action of the RLAC controller designed in the present application.

[0013] Figure 4 is a curve diagram of the tracking error of the system changing with time under the action of the RLAC controller designed in the present application.

[0014] Figure 5 is a curve diagram of the tracking error of the system under the action of the RLAC controller and the traditional PID controller designed in the present application.

[0015] Figure 6 is a control input curve diagram of the system under the action of the RLAC controller designed in the present application. DETAILED DESCRIPTION

[0016] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0017] In combination with Figure 1 and Figure 2 , the hydraulic multi-way valve intelligent shaft control method based on reinforcement learning comprises the following steps:

[0018] Step 1, establishing a mathematical model of the hydraulic multi-way valve shaft control system, specifically as follows:

[0019] Step 1-1, the hydraulic multi-way valve shaft control system is applied to linear motion of large industrial heavy load mechanical equipment, wherein the load is fixedly connected with the piston rod on the hydraulic cylinder, and the hydraulic multi-way valve controls the movement of the piston rod on the hydraulic cylinder, thereby driving the load to move.

[0020] According to Newton's second law, the force balance equation of the hydraulic multi-way valve shaft control system is:

[0021]

[0022] In formula (1), m represents the mass of the load, y represents the displacement of the hydraulic cylinder piston rod, represents the velocity of the hydraulic cylinder piston rod, represents the acceleration of the hydraulic cylinder piston rod, A represents the effective action area of the hydraulic cylinder piston, P1 represents the oil pressure in the inlet chamber of the hydraulic cylinder, and P2 represents the oil pressure in the outlet chamber of the hydraulic cylinder, represents the friction force borne by the load, d1(t) represents the system mechanical unmodeled disturbance, and t represents time.

[0023] Formula (1) is then rewritten as:

[0024]

[0025] In the hydraulic multi-way valve shaft control system, the pressure dynamic equation is:

[0026]

[0027] In formula (3), β e represents the effective elastic modulus of the oil, C t represents the leakage coefficient in the hydraulic cylinder, the pressure difference P L =P1-P2 between the inlet and outlet chambers of the oil cylinder, the control volume V1 of the inlet chamber =V 01 +Ay, the control volume V2 of the outlet chamber =V 02 -Ay, V 01 represents the initial volume of the inlet chamber, V 02 represents the initial volume of the outlet chamber, Q1 represents the flow rate of the inlet chamber, Q2 represents the flow rate of the outlet chamber, q1 represents the unmodeled disturbance of P1, and q2 represents the unmodeled disturbance of P2, represents the first derivative of P1, represents the first derivative of P2.

[0028] Q1 and Q2 are respectively related to the displacement x v of the hydraulic multi-way valve spool as follows:

[0029]

[0030] wherein the hydraulic multi-way valve coefficient C d represents the flow coefficient of the hydraulic multi-way valve, w0 represents the area gradient of the spool of the hydraulic multi-way valve, and p represents the oil density, P s represents the supply oil pressure, and P r represents the return oil pressure, and sg(·) represents a function of the intermediate variable ·, which is defined as:

[0031]

[0032] Neglecting the dynamics of the hydraulic multi-way valve spool, assume that the control input u acting on the spool and the spool displacement x satisfy v a proportional relationship, i.e. satisfy x v = ku (4) i where k i represents the voltage-spool displacement gain coefficient, thus formula (4) is rewritten as:

[0033]

[0034] formula (6), the intermediate variable k u = ku (7) q k i , the intermediate variable the intermediate variable

[0035] Step 1-2, define state variables: where the intermediate variable x1=y, the intermediate variable the intermediate variable x3=(AP1-AP2) / m, then formula (2) is converted into a state equation:

[0036]

[0037] formula (7), represents the first derivative of x1, represents the first derivative of x2, represents the first derivative of x3, the system unknown dynamics D1=d1(t) / m, the intermediate variable F(x2)=F f (x2) / m, the intermediate variable the intermediate variable the intermediate variable the system unknown dynamics

[0038] For the convenience of designing the controller, make the following assumptions:

[0039] Assumption 1: the system is expected to track the position command x d is second-order continuous, and the system expected position command, speed command and acceleration command are all bounded;

[0040] Assumption 2: the system unknown dynamics D1 and D2 satisfy:

[0041] |D1|≤δ1,|D2|≤δ2 (8)

[0042] formula (8), δ1 and δ2 are both unknown positive constants.

[0043] Turn to Step 2.

[0044] Step 2, based on the mathematical model of the hydraulic multi-way valve shaft control system, a position shaft control controller based on reinforcement learning is designed, as follows:

[0045] Step 2-1, for the convenience of designing the controller, the tracking error of the system is defined as z1=x1-x d , x d is the desired tracking position command of the system, and the following nonlinear filter is designed:

[0046]

[0047] Formula (9), filter gain τ1>0, s1represents the virtual control of x2, s 1f represents the filtered signal of s1, s 1f and the error z2=x2-s 1f of x2, s1filtering error ε1=s 1f -s1, gain l1>0 represents the upper bound of , σ(t) represents a function that is always positive, and satisfies where ν represents the integral variable, represents a constant that is always positive, represents the first derivative of s1, represents the first derivative of s 1f .

[0048] Taking the derivative of z1, we get:

[0049]

[0050] The virtual control s1is designed as:

[0051]

[0052] Formula (11), gain k1>0, then

[0053]

[0054] Step 2-2, taking the derivative of z2, we get:

[0055]

[0056] The following nonlinear filter is designed:

[0057]

[0058] Formula (14), filter gain τ2>0, s2represents the virtual control of x3, s 2f represents the filtered signal of s2, s 2f and the error z3=x3-s 2f of x3, s2filtering error ε2=s 2fs2, gain l2 > 0, represents the upper bound of s2, represents the first derivative of s2, represents the first derivative of s 2f .

[0059] The virtual control s2 is designed as:

[0060]

[0061] Equation (15), gain k2 > 0, s 2s represents an intermediate variable, δ2 represents a constant positive number, represents the estimated value of F(x2), The specific form of s2 is:

[0062]

[0063] Equation (16), represents the estimated value of W a , W a represents the weight of the execution neural network, represents the activation function of the execution neural network, X a represents the input of the execution neural network.

[0064] Correspondingly, the reinforcement learning signal R(t) can be designed as

[0065]

[0066] Equation (17), represents the estimated value of the evaluation neural network weight W c , represents the activation function of the evaluation neural network, χ represents an intermediate variable, represents the first derivative of χ, B s represents an intermediate variable, B s1 represents an intermediate variable, B s2 represents an intermediate variable, represents the first derivative of s , δ0 represents a constant positive number.

[0067] Then the weight update law of the execution-evaluation neural network can be designed as:

[0068]

[0069] Equation (18), represents the first derivative of W a , Γ a represents the weight gain matrix of the execution neural network, represents the first derivative of W c , Γc denotes the evaluation of the neural network, Proj(·) denotes a discontinuous mapping function.

[0070] Substituting equations (15) to (18) into equation (13) gives:

[0071]

[0072] Equation (19), denotes the estimation error of the neural network weights W a a denotes the approximation error of the neural network.

[0073] Step 2-3, the derivative of z3 is:

[0074]

[0075] According to equation (20), the control input of the spool, i.e. the position axis controller u based on reinforcement learning, is:

[0076]

[0077] Equation (21), gain k3> 0, u s denotes an intermediate variable, and δ3 denotes a constant that is always positive.

[0078] Substituting equation (21) into equation (20) gives:

[0079]

[0080] Go to step 3.

[0081] Step 3, the Lyapunov stability theory is used to prove the stability of the position axis controller, and the result that the system tracking error is asymptotically stable is obtained, as follows:

[0082] Define the Lyapunov function V as follows:

[0083]

[0084] where, denotes the estimation error of the neural network weights W c .

[0085] Differentiate equation (23) and substitute equations (9), (12), (14), (18), (19) and (22) into it to obtain:

[0086]

[0087] Considering that and the expression is obtained:​

[0088]

[0089] It is noted that

[0090]

[0091] Substituting equation (26) into equation (25), we have

[0092]

[0093] Define the intermediate variable z and Λ as follows:

[0094] z = [z1; z2; z3; ε1; ε2] (28)

[0095]

[0096] Equation (29), the intermediate variables Λ1 and Λ2 are

[0097]

[0098] By adjusting the gains k1, k2, k3 and filter gains τ1, τ2, the symmetric matrix Λ can be made positive definite, then we have:

[0099]

[0100] Equation (31), the intermediate variable Φ = z T Λz, T represents the transpose.

[0101] Integrating both sides of equation (31), we have:

[0102]

[0103] From equation (32), we know that V is bounded, and the integral of Φ is bounded. Further, we can conclude that all signals of the system are bounded. Therefore, Φ is uniformly continuous. According to the Barbalat lemma, we can conclude that the tracking error z1 tends to 0 as time tends to positive infinity.

[0104] Therefore, the conclusion is that by adjusting the gains k1, k2, k3 and filter gains τ1, τ2, the position axis control controller based on reinforcement learning designed for the hydraulic multi-way valve axis control system can make the system obtain the result that the tracking error converges to 0 gradually. The principle diagram of the position axis control controller based on reinforcement learning of the hydraulic multi-way valve axis control system is shown in Figure 1 .

[0105] Embodiment

[0106] To evaluate the performance of the designed controller, the physical parameters of the hydraulic multi-way valve axis control system in the simulation are shown in Table 1:

[0107] Table 1 System physical parameters

[0108] Physical parameter Value Physical parameter Value A(m 2 )]]> 2 x 10 -4 ]] β e (Pa)]]> 2 x 10 8 ]] m (kg) 40 B (N-s / m) 80 C t (m 5 / (N·s)) 7 x 10 -12 ]] k u (m / V) 4 x 10 -8 ]] V 01 (m 3 )]]> 1 x 10 -3 ]]> V 02 (m 3 )]]> 1 x 10 -3 ]] P s (MPa) 7 P r (MPa) 0

[0109] The desired instruction of the given system is x d = 0.1 sin (πt) x (1 - e -0.1t ) m.

[0110] The following controller is taken in the simulation for comparison:

[0111] RLAC based on reinforcement learning: the gain k1 = 10, k2 = 1, k3 = 1, τ1 = 2000, τ2 = 2000, l1 = l2 = 1.

[0112] PID controller: the selection steps of the PID controller parameters are as follows: first, a set of controller parameters are obtained by the PID parameter self-tuning function in Matlab by ignoring the nonlinear dynamics of the hydraulic multi-way valve shaft control system, and then the self-tuned parameters are fine-tuned to make the system obtain the best tracking performance after adding the nonlinear dynamics of the system. The selected controller parameters are k P = 10, k I = 1, k D = 1.

[0113] The desired instruction of the system, the tracking error of the RLAC controller, and the tracking error comparison of the RLAC controller and the PID controller are shown in Figure 3 , Figure 4 and Figure 5 . As can be seen from Figure 4 , under the action of the RLAC controller, the position output of the hydraulic multi-way valve shaft control system has high tracking accuracy for the instruction, and the amplitude of the steady-state tracking error is about 6 x 10 -5 m. From the tracking error comparison of the two controllers in Figure 5 , it can be seen that the tracking error of the RLAC controller proposed in the application is much smaller than that of the PID controller, and the tracking performance is more superior.

[0114] Figure 6 is the curve of the control input of the hydraulic multi-way valve shaft control system changing with time under the action of the RLAC controller, and from the figure it can be seen that the obtained control input is a low-frequency continuous signal, which is more conducive to execution in actual application.

Claims

1. A hydraulic multi-way valve intelligent axle control method based on reinforcement learning, characterized in that, Comprising the following steps: Step 1, a mathematical model of the hydraulic multi-way valve shaft control system is established, and step 2 is entered; Step 2, based on the mathematical model of the hydraulic multi-way valve shaft control system, a position shaft control controller based on reinforcement learning is designed, as follows: Step 2-1, To facilitate the design of the controller, define the tracking error of the system z1 = x1 - x d , x1 is an intermediate variable, x d is the desired tracking position command of the system, and the following nonlinear filter is designed: Equation (9), filter gain τ1>0, s1 represents the virtual control of intermediate variable x2, s 1f The filtered signal s1 represents s1. 1f The error between x2 and x2 is z2 = x2 - s 1f The filtering error ε1 of s1 is s 1f -s1; This represents the first derivative of s1. s 1f The first derivative; gain l1 > 0 indicates The upper bound of ; σ(t) represents a function that is always positive and satisfies Where ν represents the integration variable, A constant that is always positive; Taking the derivative of z1 gives wherein represents x d first derivative The virtual control s1 is designed as: Formula (11), gain k1>0, then Step 2-2, differentiate z2 with respect to Wherein, x3 represents an intermediate variable, D1 is the unknown dynamic of the system, and F(x2) represents an intermediate variable; The following nonlinear filter is designed: (14), filter gain τ2 > 0, s2 represents a virtual control of x3, s 2f represents a filtered signal of s2, s 2f represents an error z3 = x3 - s 2f of x3, s2 represents a filtered error ε2 = s 2f -s2, gain l2 > 0, represents an upper bound of s2, represents a first derivative of s2, represents a first derivative of s 2f , T represents a transpose; The virtual control s2 is designed as: Equation (15), gain k2 > 0, s 2s denotes an intermediate variable, δ2 denotes a constant that is positive, denotes an estimate of F(x2), In particular, the following holds: Equation (16), denotes W a an estimate of the weight W a denotes a weight of the neural network, denotes an activation function of the neural network, X a denotes an input to the neural network; Correspondingly, the reinforcement learning signal R(t) is designed as Formula (17), represents an evaluation of the neural network weight W c of the estimated value, represents an activation function of the evaluation neural network, χ represents an intermediate variable, represents a first derivative of χ, B s represents an intermediate variable, B s1 represents an intermediate variable, B s2 represents an intermediate variable, represents a first derivative of χ, δ0 represents a constant positive constant; Then the weight updating law of the execution-evaluation neural network is designed as: a first derivative of W represents W a a first derivative of W a represents a weight gain matrix of a neural network, a first derivative of W c a first derivative of W c represents a weight gain matrix of a neural network, Proj(·) represents a discontinuous mapping function; Substitute formula (15) to formula (18) into formula (13), and get: Equation (19), the estimation error of the neural network weights W a ε ε a denotes the approximation error of the neural network Step 2-3, differentiate with respect to z3 gives where f 31 , f 32 , f 33 are intermediate variables, and D2 is the unknown dynamics of the system. According to formula (20), the control input of the valve core, that is, the position shaft control controller u based on reinforcement learning is: Equation (21), gain k3 > 0, u s denotes an intermediate variable, and δ3 denotes a constant positive constant; Substitute formula (21) into formula (20), and get: Step 3 is entered; Step 3, the Lyapunov stability theory is used to prove the stability of the position shaft control controller, and the result that the tracking error of the system is asymptotically stable is obtained.

2. The method of claim 1, wherein, In step 1, the mathematical model of the hydraulic multi-way valve shaft control system is established, as follows: Step 1-1, the hydraulic multi-way valve shaft control system is applied to the linear motion of large industrial heavy load mechanical equipment, wherein the load is fixedly connected with the upper piston rod of the hydraulic cylinder, and the hydraulic multi-way valve controls the movement of the upper piston rod of the hydraulic cylinder, thereby driving the load to move; Step 1-2, the state variable is defined, and the state equation of the hydraulic multi-way valve shaft control system is obtained.

3. The method of claim 2, wherein, In step 1-1, the hydraulic multi-way valve shaft control system is applied to the linear motion of large industrial heavy load mechanical equipment, wherein the load is fixedly connected with the upper piston rod of the hydraulic cylinder, and the hydraulic multi-way valve controls the movement of the upper piston rod of the hydraulic cylinder, thereby driving the load to move, as follows: According to Newton's second law, the force balance equation of the hydraulic multi-way valve shaft control system is: In formula (1), m represents the mass of the load, y represents the displacement of the hydraulic cylinder piston rod, represents the velocity of the hydraulic cylinder piston rod, represents the acceleration of the hydraulic cylinder piston rod, A represents the effective action area of the hydraulic cylinder piston, P1 represents the oil pressure in the inlet chamber of the hydraulic cylinder, and P2 represents the oil pressure in the outlet chamber of the hydraulic cylinder, represents the friction force borne by the load, and d1(t) represents the system mechanical unmodeled disturbance, and t represents time. Then formula (1) is rewritten as: In the hydraulic multi-way valve shaft control system, the oil leakage of the cylinder is ignored, and the pressure dynamic equation is: Equation (3), β e represents the effective elastic modulus of the oil, C t represents the leakage coefficient in the hydraulic cylinder, the oil pressure difference P L = P1-P2, the control volume V1 of the inlet oil chamber = V 01 + Ay, the control volume V2 of the outlet oil chamber = V 02 -Ay, V 01 represents the initial volume of the inlet oil chamber, V 02 represents the initial volume of the outlet oil chamber, Q1 represents the flow of the inlet oil chamber, Q2 represents the flow of the outlet oil chamber, q1 represents the unmodeled disturbance of P1, q2 represents the unmodeled disturbance of P2, represents the first derivative of P1, represents the first derivative of P2; Q1, Q2 are respectively hydraulic multi-way valve spool displacement x v has the following relationship: wherein the hydraulic multi-way valve coefficient C d denotes a flow coefficient of the hydraulic multi-way valve, w0denotes a spool area gradient of the hydraulic multi-way valve, p denotes an oil density, P s denotes a supply oil pressure, P r denotes a return oil pressure, sg(·) denotes a function of an intermediate variable ·, defined as: Neglecting the dynamics of the hydraulic multi-way valve spool, assume that the control input u acting on the spool and the spool displacement x v are in a proportional relationship, i.e. satisfy x v = k i u, where k i denotes the voltage-to-spool displacement gain coefficient, so that equation (4) is rewritten as: Equation (6), intermediate variable k u = k q k i , intermediate variable intermediate variable 4. The method of claim 3, wherein, In steps 1-2, define state variables: where the intermediate variable x1 = y, the intermediate variable The intermediate variable x3 = (AP1 - AP2) / m, and then transform equation (2) into a state equation: Formula (7), denotes the first derivative of x1, denotes the first derivative of x2, denotes the first derivative of x3; System unknown dynamics D1 = d1(t) / m, intermediate variable F(x2) = F f (x2) / m, intermediate variable intermediate variable intermediate variable System unknown dynamics Step 2 is entered.

5. The method of claim 4, wherein, For the convenience of designing the controller, the following assumptions are made: Assumption 1 : The system is expected to track position command x d is second order continuous, and the system is expected to have bounded position command, velocity command, and acceleration command; Assumption 2: the unknown dynamics D1 and D2 of the system satisfy: |D1|≤δ1,|D2|≤δ2 (8) In formula (8), δ1 and δ2 are unknown positive constants.

6. The method of claim 5, wherein, In step 3, the Lyapunov stability theory is used to prove the stability of the position shaft control controller, and the result that the tracking error of the system is asymptotically stable is obtained, as follows: The Lyapunov function V is defined as follows: wherein the evaluation neural network weight W c is estimated error The Lyapunov stability theory is used to prove the stability, and the result that the tracking error of the system is asymptotically stable is obtained.

Images