Uncertain system Lyapunov stability reinforcement learning control algorithm based on worst case

Abstract

The invention discloses an uncertain system Lyapunov stability reinforcement learning control algorithm based on the worst situation. The uncertain system Lyapunov stability reinforcement learning control algorithm comprises the following steps: S1, modeling a system dynamics model and an uncertain boundary through a relu activation network; s2, determining a robustness condition, and pre-determining the area of an attraction domain by using the robustness condition; s3, determining the uncertainty of the dynamic model and the robustness guarantee RL under state estimation; s4, establishing network parameterization; and S5, carrying out numerical simulation on the inverted pendulum and the quadrotor unmanned aerial vehicle. According to the uncertainty system Lyapunov stability reinforcement learning control algorithm based on the worst situation, the most violating state can still be accurately found, so that the stability under uncertainty is forced, a geometric view of the existence of a robust RL problem solution is given to explain the robustness and the ability thereof, and the robustness and the ability of the robustness RL problem solution are improved. The numerical simulation of the inverted pendulum and the four-rotor aircraft under various uncertainties proves the effectiveness of the proposed method.

Description

Technical Field

[0001] The present invention relates to the field of automatic control technology, and in particular to a Lyapunov stable reinforcement learning control algorithm for an uncertain system based on the worst case. Background Art

[0002] Robotic systems are susceptible to stability degradation and catastrophic failures caused by various uncertainties, including modeling uncertainty and state uncertainty. Existing techniques on this topic include robust control, adaptive control, stochastic control, sliding mode control, and h-∞, among others. However, existing approaches may be limited by the models they use to describe uncertainty and system dynamics. RL algorithms, which rely on function approximation via neural networks, have emerged as a highly flexible framework for solving Markov decision processes (MDPs). However, classical RL struggles with robustness to uncertainty, disturbances, or changes in environmental structure. To this end, robust RL has been studied to enhance the robustness of policies with respect to various aspects: (1) uncertain dynamics models; (2) external disturbances; (3) controllable noise; and (4) uncertain observations and state estimates. A min-max problem or optimality condition based on the Hamilton-Jacobi-Isaacs equations is then formulated and solved to obtain a controller that suppresses the effects of uncertainty in the worst case.

[0003] For example, the small gain theorem is used to construct the objective function to minimize the sensitivity of the stabilization to disturbances or uncertainties. If the gain is bounded by the inverse of the sensitivity, disturbances caused by modeling errors can be suppressed. When the uncertainty set is defined as centered on the wrong MDP that produces a single sample trajectory and estimated by sampling, an alternative adversarial action or perturbation from the adversary is added to the selected action, and the controller is trained in the manner of a generative adversarial network (GAN).

[0004] However, it is difficult for robust RL to provide strict convergence guarantees under uncertainty in both the dynamics model and the state estimate, hindering its credibility in practical applications. Uncertainty is compensated by minimizing the expectation of some designed reward function. Modeled uncertainty can still lead to suboptimal or even unsafe behavior, especially when the agent explores new or unfamiliar states. Noisy observations can lead to uncertainty in the state estimate, and the stability of the system may be compromised.

[0005] Auxiliary loss functions based on Lyapunov theory and barrier functions have partially solved the convergence problem of RL methods. However, their applicability to uncertain dynamic models and system states has not been explored. Summary of the invention

[0006] The purpose of the present invention is to provide a worst-case Lyapunov-stabilized reinforcement learning control algorithm for uncertain systems for synthesizing neural network controllers to maintain the stability of the system under bounded multiplicative and additive uncertainties in the dynamics model and state estimation.

[0007] To achieve the above object, the present invention provides a Lyapunov stable reinforcement learning control algorithm for an uncertain system based on the worst case, comprising the following steps:

[0008] The following steps are involved:

[0009] S1, modeling the system dynamics model and uncertainty boundary through the relu activation network;

[0010] S2, determining a robustness condition and using it to predetermine the area of ​​the attraction domain;

[0011] S3, determine the robustness guarantee RL under the uncertainty of dynamic model and state estimation;

[0012] S4, network parameterization establishment;

[0013] S5. Perform numerical simulation on inverted pendulum and quadrotor drone.

[0014] Preferably, in step S1, the system dynamics model and the uncertainty boundary are modeled, and the specific process is as follows:

[0015] Consider a discrete-time robot system whose dynamics model is:

[0016]

[0017] Among them, x t ∈R nx and u t ∈R nu They represent the robot's state and control input at time t respectively, and the control saturation is given by the lower bound u min Or upper bound u max given;

[0018] The kinetic model is, assuming that its approximate value It is obtained through system identification as shown below:

[0019]

[0020] Among them, δ m and δ a denote the multiplicative and additive uncertainties between the actual dynamics and its approximation, respectively, represents the total model uncertainty, u t =π(x t) indicates that the controller π is in state x t The control given at

[0021] However, the system state x is estimated from some estimator and thus corrupted by noise, as shown below:

[0022]

[0023] where ∈ m and ∈ a denote the multiplicative uncertainty and additive uncertainty of the state estimation, ∈(x t ) represents the aggregate uncertainty in the state estimate;

[0024] Therefore, the predicted state at time t+1 is:

[0025]

[0026] in, It means that the controller π is used in the state estimation The control given at

[0027] For an approximate kinetic model State estimator that outputs system estimates Control saturation u min and u max , and a domain of attraction D, find a controller Let the system stabilize at the equilibrium point under model uncertainty δ and state uncertainty ∈;

[0028] Assumption 1: An approximate dynamics model is given in the form of a relu activated neural network

[0029] Assumption 2: Assume that the uncertainty of the system model dynamics δ(x) and the uncertainty of the state estimation ∈(x) are bounded, and their bounds are denoted by Δ δ (x) and Δ ∈ (x), then:

[0030]

[0031] Among them, |·| is the infinite norm, Δ δ (x) and Δ ∈ (x) are all given in the form of a relu activated neural network.

[0032] Preferably, in step S2, the robustness condition is a stability condition under the worst case uncertainty, which is derived from Lyapunov neural control. When uncertainty is not involved, the coupled Lyapunov function V(x t ): Rnx →R to ensure the stability of the system:

[0033] V(x * )=0 (6);

[0034]

[0035] Where D is the attractive state area, x* is the equilibrium state, V max (x t ) and V min (x t ) belongs to the set of K functions, and the stability conditions (6) to (8) guarantee that the system starting from any state in D converges to the equilibrium state x*.

[0036] Preferably, in step S2, the stability condition under the worst case uncertainty includes a robustness condition under the uncertainty of the dynamic model and a robustness condition under the uncertainty of the state estimation:

[0037] Under the model uncertainty δ(x) in the dynamics described, the state at time t+1 is not completely determined by the control ut and the current state x t Therefore, Equation (8) must be used for each possible state at time t+1 are all valid, then:

[0038]

[0039] in,

[0040] By x t+1 The Taylor expansion of the Lyapunov function at , its robustness condition (8) becomes:

[0041]

[0042] Given The randomness of , and based on assumption 2, a stronger and more sufficient robustness condition is obtained as follows:

[0043]

[0044] in, and The ratio is negligible, taking into account the worst case uncertainty in the dynamics modeling and forcing the Lyapunov value to decrease along the state trajectory;

[0045] With a given ground truth state x t The control input u obtained t =π(x t ) On the contrary, the controller π actually estimates is calculated to determine the control input Therefore, the stability condition needs to take into account the randomness of the state estimate; considering the worst-case uncertainty in the state estimate, The Lyapunov value at is obtained by Taylor expansion as follows:

[0046]

[0047] The derivative of the Lyapunov function is obtained by the chain rule, as follows:

[0048]

[0049] Among them, x t+1 =f(x t ,u t ),u t =π(x t );

[0050] Then the robustness condition under uncertainty in state estimation is as follows:

[0051]

[0052] Under assumption 2, the stability condition (8) becomes the new robustness condition as follows:

[0053]

[0054] in, is ignored, the derivative and are the piecewise constants in x, u, and x respectively.

[0055] Preferably, in step S3, let α represent the trainable parameters in the Lyapunov network, β represent the trainable parameters in the controller network, and use the robustness conditions (7), (12) and (16) to design the following loss function to adjust the training of the controller network and the Lyapunov network, and give the loss function that guarantees the semi-positive definition of the Lyapunov network (7):

[0056]

[0057] When only uncertainty δ is considered in the model dynamics, the Lyapunov value along the trajectory should increase strictly monotonically and be guaranteed by satisfying the robustness condition (12). The corresponding loss function is designed as:

[0058]

[0059] When only the uncertainty v in the state estimation is considered, the loss function (16) established by the robustness condition is expressed as:

[0060] Based on the robustness condition (16), for a system with only uncertainty in the dynamic model, the loss function is constructed:

[0061]

[0062] When the above two uncertainties are considered simultaneously, the robustness condition (8) becomes:

[0063]

[0064] Then, the loss function corresponding to the robustness condition eq is obtained, which is expressed as:

[0065]

[0066] Express the objective function and all constraints as piecewise linear functions of x:

[0067] Let θ = (α, β) represent the set of all trainable parameters. In each iteration of training, the value θ of the previous iteration is used, and the division of the attraction domain S is:

[0068] Let y i is a binary if and only if x is equal to 1 in the i-th partition, then x is represented as a convex combination of the extreme points of the partition. By introducing slack variables, the optimization problem of the loss function is transformed into a mixed integer linear programming:

[0069]

[0070] stAθ s =b θ (22b);

[0071] Among them, the problem coefficient c θ d θ , A θ and b θ are all explicit functions of θ, and the optimization variable vector s consists of x and y1..., then the optimal value of the objective function (22) is written as:

[0072]

[0073] Using Gradients The trainable parameters θ are optimized via back-propagation.

[0074] Preferably, in step S4, the network parameterization establishment process is as follows:

[0075] Using the HL-PWL method, the following expression is used to represent any piecewise linear function F(x):

[0076]

[0077] in, is a coefficient vector, is a stack of basis functions, basis functions The stack includes all nested functions at nesting level i; the coefficient vector a i With nested functions associated;

[0078] Robustness conditions under HL-PWL representation:

[0079] Correct characterization of the Lyapunov function, formula (7) is rewritten as:

[0080]

[0081] Where ξ1(|x|) and ξ2(|x|) are K functions and increase with the increase of |x|. Both the Lyapunov function V(x) and the K function are rewritten in HL-PWL form (23), expressed as:

[0082]

[0083] Among them, |·| is the infinite norm, and is the parameter vector, ζ a and i is the number of corresponding parameter vectors, upper and lower bounds and μ(|x|)=[μ 0 (|x|), μ 1 (|x|)] T is a vector function, including nested functions at nesting level 0 and nesting level 1. It is easy to get (i.e. |x|max);

[0084] Since μ is a K function, we have:

[0085] ξ i (0) = 0 (27);

[0086] ξ i (|x t+1 |)>ξ i (|x i |), if|x t+1 |>|x t | (28);

[0087] The robustness condition (6) and positivity condition (7) under equilibrium conditions are written as:

[0088]

[0089] and

[0090]

[0091] Given an attraction domain D, partition and define vertices. Then, condition (30) is transformed into the form of 6 in Matrix W 11 and W 12 is obtained by superimposing all x in V j Vector obtained by superposition, where V is the set of D partitions;

[0092] Similarly, the upper bound condition of formula (24) can be rewritten as Then the upper bound ξ1(|x|) and the lower bound ξ2(|x|) are K functions that satisfy equation (27), that is:

[0093] μ T (0)b i =0.i=1,2 (31);

[0094] And the function that satisfies (28), that is:

[0095] W2 T b i <x * , i=1,2 (32);

[0096] where W2 = {μ(0)-μ(γ), …, μ((m-1)γ)-μ(mγ)} and γ is the grid size of the partition. When considering the uncertainties in dynamic modeling and state estimation simultaneously, the robustness conditions (20) and (14) yield another condition:

[0097]

[0098] It is further assumed that both the dynamical system and the controller satisfy the following Lipschitz condition:

[0099]

[0100] where u and F represent the upper bounds of the controller and positive dynamics changes, respectively;

[0101] Then, the term V(x t+1 )-V(x t ) in state x t+1 The Taylor expansion produced at is:

[0102]

[0103] By negation And using HL-PWL representation, formula (36) is written as:

[0104]

[0105] in, express The derivative of x is t+1 , then the robust condition (33) is written as:

[0106]

[0107] It can be written as the following quadratic inequality:

[0108]

[0109] Where (x t+1 -x t ) is a column vector, is a real symmetric matrix; since X is a rank 1 matrix, it has a nonzero eigenvalue equal to (x i+1 -x t ) is the square of the norm of |x t+1 -x t | 2 , the remaining eigenvalues ​​are zero, let

[0110]

[0111] but is rewritten as:

[0112]

[0113] where Q is an orthogonal identity matrix QQ T =I, so formula (39) is further rewritten as:

[0114] make Then it satisfies:

[0115]

[0116] Formula (43) shows that Convert to standard quadratic form, There must be a correct solution;

[0117] However, when the control is saturated at the lower and upper bounds, if for all u min ≤u t ≤u max , Then no matter how the coefficient a changes, the entire quadratic form is less than zero. Therefore, the necessary condition for the existence of the solution, that is, the robust condition (33), is written as:

[0118]

[0119] The above necessary conditions require that the controller has sufficient ability to exceed the t to x t+1 The uncertainty range of the method is used to find the infeasible state by using the necessary condition, and the attraction domain D is improved to eliminate the state that cannot be guaranteed by the method.

[0120] Since the system (2) is piecewise linear, the control evaluates at its boundary vertices to obtain each state x i The reachable domain vertex set Get a cone for each vertex control, then construct the union of these cones, and get a new cone by approximation

[0121] Now, For each vertex x in i Corresponding to a cone in space The union of these cones C is the solution space of a. Consider the cones corresponding to the four vertex states. For the cones to be aligned, and considering the constraints imposed by formula (30), only four cones are shown. The intersection of these four cones is approximately a linear cone.

[0122] Without loss of generality, assume that the cone has N intersection points v1, v2....v N , these intersection points form N planes in the solution space, with normal vectors n1, n2, ....n N The linear cone formed by these N intersection points is represented by computing for each plane:

[0123] n i =v i ×v i+1 (45);

[0124]

[0125] Then, the Lyapunov candidate function The solution for the coefficients is:

[0126]

[0127] Where b1, b2, a are the coefficients of the HL-PWL representation of the Lyapunov function and its boundary. Before applying the proposed RGRL to learn the controller and the coupled Lyapunov function, the attraction domain is preliminarily verified using the solution space (47).

[0128] Therefore, the present invention adopts the above-mentioned Lyapunov stable reinforcement learning control algorithm for uncertain systems based on the worst case, and the beneficial effects are as follows:

[0129] (1) By modeling the uncertainty of the ReLU activation network and the system dynamics model, the present invention can still accurately find the most violating state, thereby enforcing its stability under uncertainty.

[0130] (2) The present invention discusses the solvability of the problem of solving the robust controller and the connected Lyapunov function, and derives the necessary conditions for determining the attraction domain before solving the problem.

[0131] (3) This paper presents a geometric view of the existence of solutions to robust RL problems to explain robustness and its capabilities. Numerical simulations of inverted pendulums and quadrotors under various uncertainties demonstrate the effectiveness of the proposed method.

[0132] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0133] Figure 1 It is a schematic diagram of a solution space composed of cones according to an embodiment of the Lyapunov stable reinforcement learning control algorithm for an uncertain system based on the worst case of the present invention;

[0134] Figure 2 It is the simulation result of the inverted pendulum in the precision system using NNC according to the embodiment of the Lyapunov stable reinforcement learning control algorithm for the uncertain system based on the worst case of the present invention, wherein (a) is the trajectory diagram, (b) is the Lyapunov value, and (c) is the control value;

[0135] Figure 3 is a comparison between NNC and RNC under observation uncertainty of an embodiment of the Lyapunov-stabilized reinforcement learning control algorithm for an uncertain system based on the worst case of the present invention, wherein (a) is the trajectory of NCC, (b) is the trajectory of RNC, (c) is the Lyapunov value of NCC, (d) is the Lyapunov value of RNC, (e) is the control signal of NCC, and (f) is the control signal of RNC;

[0136] Figure 4 is a comparison between NNC and RNC controllers under observation uncertainty of an embodiment of the Lyapunov-stabilized reinforcement learning control algorithm for an uncertain system based on the worst case of the present invention, wherein (a) is the trajectory and Lyapunov value generated by NCC, (b) is the trajectory and Lyapunov value generated by RNC, (c) is the control signal of NCC, and (d) is the control signal of RNC;

[0137] Figure 5is a comparison of the NNC and RNC controllers of the embodiment of the Lyapunov-stabilized reinforcement learning control algorithm for the uncertain system based on the worst case of the present invention in the form of observation uncertainty multiplication, wherein (a) is the NCC trajectory, (b) is the RNC trajectory, (c) is the Lyapunov value of the NCC, (d) is the Lyapunov value of the RNC, (e) is the control signal of the NCC, and (f) is the control signal of the RNC;

[0138] Figure 6 is a comparison of NNC and RNC in the form of multiplication of observation uncertainty of an embodiment of the Lyapunov-stabilized reinforcement learning control algorithm for an uncertain system based on the worst case of the present invention, wherein (a) is the NCC trajectory, (b) is the RNC trajectory, (c) is the NCC control signal, and (d) is the RNC control signal;

[0139] Figure 7 is a comparison of controllers under multiplication and addition uncertainty based on the worst-case Lyapunov-stabilized reinforcement learning control algorithm for uncertain systems of the present invention, where (a) is δ a The trajectory under (b) is δ m The trajectory under (c) is δ a The Lyapunov value under , (d) is δ m The Lyapunov value under , (e) is δ a The control signal under (f) is δ m The control signal of

[0140] Figure 8 It is the convergence efficiency of the compressed space under the state space condition of the embodiment of the Lyapunov stabilized reinforcement learning control algorithm for the uncertain system based on the worst case of the present invention;

[0141] Fig. 9 is the loss function of the entire state space during the training process of the embodiment of the Lyapunov-stabilized reinforcement learning control algorithm for the uncertain system based on the worst case of the present invention, wherein (a) is the loss function of the entire state space after 1 training cycle, (b) is the loss function of the entire state space after 20 training cycles, and (c) is the loss function of the entire state space after 100 training cycles;

[0142] Fig.10 is a comparison of the NCC and RNC controllers under observation uncertainty of the embodiment of the Lyapunov-stabilized reinforcement learning control algorithm for the worst-case uncertainty system of the present invention, where (a) is δ m The controller under (b) is ρδ m The following controller;

[0143] Fig.11is a comparison of a two-dimensional quadcopter under multiplicative uncertainty and additive uncertainty based on the worst-case uncertainty system Lyapunov stabilized reinforcement learning control algorithm embodiment of the present invention, where (a) is δ m The trajectory under (b) is δ a The trajectory under (c) is δ m The Lyapunov value under , (d) is δ a The Lyapunov value under , (e) is δ m The control signal under (f) is δ a The following control signal. DETAILED DESCRIPTION

[0144] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0145] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0146] Robust reinforcement learning (RL) optimizes the expectation of the worst-case cumulative reward through two-level optimization. However, it is difficult to obtain strict robustness guarantees when uncertainties coexist, which limits the credibility of RL in practical applications. In the absence of uncertainty, the stability problem of RL methods has been partially solved by auxiliary loss functions or barrier functions based on Lyapunov theory. The present invention presents a stable RL under bounded uncertainty of dynamic models and state estimates in a unified way.

[0147] As shown in the figure, the Lyapunov stable reinforcement learning control algorithm for the worst-case uncertain system includes the following steps:

[0148] S1. Model the system dynamics model and uncertainty boundary through the relu activation network. The specific process is as follows:

[0149] Consider a discrete-time robot system whose dynamics model is:

[0150] x t+1 =f(x t ,u t ),u min ≤u t ≤u max (1);

[0151] Among them, x t ∈R nx and u t ∈R nuThey represent the robot's state and control input at time t respectively, and the control saturation is given by the lower bound u min Or upper bound u max given;

[0152] The kinetic model is, assuming that its approximate It is obtained through system identification as shown below:

[0153]

[0154] Among them, δ m and δ a denote the multiplicative and additive uncertainties between the actual dynamics and its approximation, respectively, represents the total model uncertainty, u t =π(x t ) indicates that the controller π is in state x t The control given at

[0155] However, the system state x is estimated from some estimator and thus corrupted by noise, as shown below:

[0156]

[0157] where ∈ m and ∈ a denote the multiplicative uncertainty and additive uncertainty of the state estimation, ∈(x t ) represents the aggregate uncertainty in the state estimate;

[0158] Therefore, the predicted state at time t+1 is:

[0159]

[0160] in, It means that the controller π is used in the state estimation The control given at

[0161] For an approximate kinetic model State estimator that outputs system estimates Control saturation u min and u max , and a domain of attraction D, find a controller Let the system stabilize at the equilibrium point under model uncertainty δ and state uncertainty ∈;

[0162] The controller π is usually constructed based on the approximate dynamic model The training is done in a simulator and then applied to the actual system f, which creates a gap between simulation and reality in the dynamic model. In addition, the state estimation The controller π is used to obtain the control input. Both of these factors may lead to the instability of the closed-loop robot system.

[0163] Assumption 1: An approximate dynamics model is given in the form of a relu activated neural network

[0164] Assumption 2: Assume that the uncertainty of the system model dynamics δ(x) and the uncertainty of the state estimation ∈(x) are bounded, and their bounds are denoted by Δ δ (x) and Δ ∈ (x), then:

[0165]

[0166] Among them, |·| is the infinite norm, Δ δ (x) and Δ ∈ (x) are all given in the form of a relu activated neural network.

[0167] Among them, the relu activated network allows flexible modeling of system dynamics and uncertainty boundaries, making the proposed RGRL method applicable to many complex systems. More importantly, through assumptions 1 and 2, the robustness conditions can be converted into a set of linear constraints, and the states that violate the constraints can be effectively found through mixed integer linear programming (MILP). In addition to solving the robust controller, the attraction domain D is also refined because the adversarial relationship between control saturation and uncertainty may cause the state to fail to meet the robustness conditions.

[0168] S2, determining a robustness condition and using it to predetermine the area of ​​the attraction domain;

[0169] Robustness conditions are used to design the RGRL loss function. Specifically, stability conditions under the worst-case uncertainty are derived to regulate the training of the controller and the Lyapunov function.

[0170] The robustness condition is the stability condition under the worst case uncertainty, which is derived from Lyapunov neural control. When no uncertainty is involved, the coupled Lyapunov function V(x) is trained to satisfy the following stability condition: t ): R nx →R to ensure the stability of the system:

[0171] V(x * )=0 (6);

[0172]

[0173] Where D is the attraction domain, x* is the equilibrium state, V max (x t ) and V min(x t ) belongs to the set of K functions, and the stability conditions (6) to (8) guarantee that the system starting from any state in D converges to the equilibrium state x*.

[0174] The uncertainty of a given dynamics model and the robustness conditions of state estimation are discussed below.

[0175] The present invention explicitly considers bounds on the uncertainties in the dynamics modeling and specifically guarantees the worst-case scenario. This is in contrast to domain randomized RL methods, which optimize the expectation of reward over these uncertainties and do not provide guaranteed robustness. More importantly, the derived robustness conditions are linear in the system states, allowing the use of MILP to find states that violate these conditions.

[0176] The stability conditions under the worst case uncertainty include the robustness conditions under the uncertainty of the dynamic model and the robustness conditions under the uncertainty of the state estimation:

[0177] Under the model uncertainty δ(x) in dynamics, the state at time t+1 is not completely determined by the control ut and the current state x t Therefore, Equation (8) must be used for each possible state at time t+1 are all valid, then:

[0178]

[0179] in,

[0180] By x t+1 The Taylor expansion of the Lyapunov function at , its robustness condition (8) becomes:

[0181]

[0182] Given The randomness of , and based on assumption 2, a stronger and more sufficient robustness condition is obtained as follows:

[0183]

[0184] in, and The worst case uncertainty in the dynamics modeling is considered and the Lyapunov value is forced to decrease along the state trajectory.

[0185] Note that given the parameters of the Lyapunov network, the derivative dV / dx(x) of the Lyapunov function V(x) is expressed as (wrt), and the state x is a piecewise constant. In addition, the uncertainty bound Δσ(x) and the Lyapunov function are piecewise linear in x. Therefore, condition (12) is piecewise linear in x.

[0186] Robustness Conditions Under Uncertainty in State Estimation

[0187] With a given ground truth state x t The control input u obtained t =π(x t ) On the contrary, the controller π actually estimates is calculated to determine the control input Therefore, the stability condition needs to take into account the stochastic nature of the state estimates.

[0188] Taking into account the worst-case uncertainty in the state estimate, The Lyapunov value at is obtained by Taylor expansion as follows:

[0189]

[0190] The derivative of the Lyapunov function is obtained by the chain rule, as follows:

[0191]

[0192] Among them, x t+1 =f(x t ,u t ),u t =π(x t );

[0193] Then the robustness condition under uncertainty in state estimation is as follows:

[0194]

[0195] Under assumption 2, the stability condition (8) becomes the new robustness condition as follows:

[0196]

[0197] in, is ignored.

[0198] Note 3: Derivative and are piecewise constants in x, u, and x, respectively. Therefore, condition (16) is piecewise linear in x. S3, robustness under uncertainty in the dynamics model and state estimation guarantees RL;

[0199] Robustness-guaranteed RL (RGRL) under uncertainty in dynamic modeling and state estimation is similar to stability-guaranteed RL, in which the controller network π(x) and the Lyapunov network V(x) are trained simultaneously.

[0200] Assuming that α represents the trainable parameters in the Lyapunov network and β represents the trainable parameters in the controller network, the following loss function is designed using the robustness conditions (7), (12) and (16) to regulate the training of the controller network and the Lyapunov network. The loss function that guarantees the semi-positive definition of the Lyapunov network (7) is given as:

[0201]

[0202] When only uncertainty δ is considered in the model dynamics, the Lyapunov value along the trajectory should increase strictly monotonically and be guaranteed by satisfying the robustness condition (12). The corresponding loss function is designed as:

[0203]

[0204] When only the uncertainty ∈ in the state estimation is considered, the loss function (16) established by the robustness condition is expressed as:

[0205] Based on the robustness condition (16), for a system with only uncertainty in the dynamic model, the loss function is constructed:

[0206]

[0207] When the above two uncertainties are considered simultaneously, the robustness condition (8) becomes:

[0208]

[0209] Then, the loss function corresponding to the robustness condition eq is obtained, which is expressed as:

[0210]

[0211] Note that the calculation of the above loss function involves solving the optimal system state x(x t ), inspired by Lyapunov neural control, the activation function in the controller network, Lyapunov network and uncertainty bound network is “leaky-ReLU”, making the controller, Lyapunov function and uncertainty bound piecewise linear. Let θ = (α, β) denote the set of all trainable parameters. In each training iteration, when solving a state x that mainly violates the loss function (18), (19) or (21), the trainable parameters θ of the previous iteration are fixed and used. This state is also called a “counterexample”. The loss functions (18), (19) and (21) are all piecewise linear functions in x.

[0212] The attraction domain D is decomposed into multiple partitions, determined by θ. Let yi be a binary that is equal to 1 if and only if x is equal to 1 in the i-th partition, then x is represented as a convex combination of the extreme points of the partition. By introducing slack variables, finding a counterexample x with respect to the loss function (18), (19) or (21) can be converted into a mixed integer linear programming MLIP problem as follows:

[0213] Express the objective function and all constraints as piecewise linear functions of x:

[0214] Let θ = (α, β) represent the set of all trainable parameters. In each iteration of training, the value θ of the previous iteration is used, and the division of the attraction domain S is:

[0215] Let y i is a binary if and only if x is equal to 1 in the i-th partition, then x is represented as a convex combination of the extreme points of the partition. By introducing slack variables, the optimization problem of the loss function is transformed into a mixed integer linear programming:

[0216]

[0217] sA θ s=b θ (22b);

[0218] The problem coefficient c θ , d θ , A θ , b θ are all explicit functions of θ, and the optimization variable vector s consists of x and y. Then the optimal cost of the above equations (26a) and (26b) is written in closed form: γ(θ) = cTθA-θ1bθ+dθ. The gradient is calculated by backpropagating this closed form expression

[0219] Among them, the problem coefficient c θ d θ , A θ and b θ are all explicit functions of θ and are considered fixed when looking for “counterexamples”. The optimization variable vector s consists of x and y1..., then the optimal value of the objective function (22) is written as:

[0220]

[0221] Using Gradients The trainable parameters θ are optimized via back-propagation.

[0222] Algorithm 1 below summarizes the proposed RGRL, which finds a “counterexample” state x where θ is fixed.c (Step 3 in Algorithm 1) and use the “counterexample” x found c The gradient calculated on Iterate between updating θ (step 5 in Algorithm 1).

[0223] Algorithm 1 Robust Guaranteed RL:

[0224] 1. Initialize the controller π(x) and the Lyapunov function V(x);

[0225] 2. When iterations ≤ maximum value:

[0226] 3. Use the Gurobi solver to maximize γ(θ) in equation (22) to find x c ;

[0227] 4. Calculate the gradient

[0228] 5. Update where κ is the step size.

[0229] Note that the solution of robust guaranteed RL depends heavily on the uncertainty, system dynamics, control saturation, attraction domain, and flexibility of the neural network structure. In fact, the solvability of Problem 1 has been embedded into the MILP problem in (22) and will be investigated in subsequent studies.

[0230] Solubility analysis

[0231] The RGRL proposed in Algorithm 1 involves finding “counterexample” states via MLIP and optimizing trainable parameters θ. Due to the adversarial relationship between the uncertainty size and control saturation, there may not be such a controller that can stabilize certain states and guarantee robustness. Therefore, before using RGRL to find a robust controller and coupled Lyapunov function, the domain of attraction, solvability given uncertainty and control saturation should be carefully checked.

[0232] Since there is always a PWL function corresponding to a relu-activated neural network. PWL functions have been widely used in many fields such as control systems, signal processing, optimization problems, circuit design, etc. A rich set of analysis tools are available for PWL systems. When there is no uncertainty or control saturation, if the PWL system is affine, there exists a piecewise linear Lyapunov function and a controller that stabilizes the system. In addition, if the state at the vertex of the partition satisfies the stability condition, then all states in the partition satisfy the stability condition. Let H denote the vertex of the state partition. This step advances this line of work by considering control saturation and uncertainty.

[0233] S4. Network parameterization is established. The process is as follows:

[0234] In order to further formulate problem (22) into a more compact form without resorting to the form of slack variables, a high-level canonical representation of the PWL function, called HL-PWL, is employed. HL-PWL has been shown to use fewer parameters than the classical PWL while maintaining the same accuracy, expressed in the form:

[0235] Using the HL-PWL method, the following expression is used to represent any piecewise linear function F(x):

[0236]

[0237] in, is a coefficient vector, is a stack of basis functions, basis functions The stack includes all nested functions at nesting level i, with coefficient vector a i With nested functions associated;

[0238] Robustness conditions under HL-PWL representation:

[0239] Correct characterization of the Lyapunov function, formula (7) is rewritten as:

[0240]

[0241] Where ξ1(|x|) and ξ2(|x|) are K functions and increase with the increase of |x|. Both the Lyapunov function V(x) and the K function are rewritten in HL-PWL form (23), expressed as:

[0242]

[0243] Among them, |·| is the infinite norm, and is the parameter vector, ζ a and i is the number of corresponding parameter vectors, upper and lower bounds and μ(|x|)=[μ 0 (|x|), μ 1 (|x|)] T is a vector function, including nested functions at nesting level 0 and nesting level 1. It is easy to obtain (i.e. |x|max).

[0244] Since μ is a K function, we have:

[0245] ξ i (0) = 0 (27);

[0246] ξ i(|x t+1 |)>ξ i (|x t |), if|x t+1 |>|x t | (28);

[0247] The robustness condition (6) and positivity condition (7) under equilibrium conditions are written as:

[0248]

[0249] and

[0250]

[0251] Given an attraction domain D, partition and define vertices. Then, condition (30) is transformed into the form of 6 in Matrix W 11 and W 12 is obtained by superimposing all x in V j Vector obtained by superposition, where V is the set of D partitions.

[0252] Similarly, the upper bound condition of formula (24) can be rewritten as Then the upper bound ξ1(|x|) and the lower bound ξ2(|x|) are K functions that satisfy equation (27), that is:

[0253] μ T (0)b i =0. i=1,2 (31);

[0254] And the function that satisfies (28), that is:

[0255] W2 T b i <x * , i = 1, 2 (32);

[0256] Where W2 = {μ(0)-μ(γ), ..., μ((m-1)γ)-μ(mγ)} and γ is the grid size of the partition. For the PWL system, it is sufficient to analyze the state at vertex v. Note that in the rest of the discussion, the state x t is assumed to be one of the vertices.

[0257] When considering the uncertainties in dynamics modeling and state estimation simultaneously, the robustness conditions (20) and (14) yield another condition:

[0258]

[0259] It is further assumed that both the dynamical system and the controller satisfy the following Lipschitz condition:

[0260]

[0261] where u and F represent the upper bounds of the controller and positive dynamics changes, respectively;

[0262] Then, the term V(x t+1 )-V(x t ) in state x t+1 The Taylor expansion produced at is:

[0263]

[0264] By negation And using HL-PWL representation, formula (36) is written as:

[0265]

[0266] in, express The derivative of x is t+1 , then the robust condition (33) is written as:

[0267]

[0268] It can be written as the following quadratic inequality:

[0269]

[0270] Among them, (x t+1 -x t ) is a column vector, is a real symmetric matrix. Since X is a rank 1 matrix, it has a non-zero eigenvalue equal to (x t+1 -x t ) is the square of the norm of |x t+1 -x t | 2 , the remaining eigenvalues ​​are zero, let

[0271]

[0272] but is rewritten as:

[0273] where Q is an orthogonal identity matrix QQ T =I, so formula (39) is further rewritten as:

[0274]

[0275] make Then it satisfies:

[0276]

[0277] Formula (43) shows that Convert to standard quadratic form, There must be a correct solution;

[0278] However, when the control is saturated at the lower and upper bounds, if for all u min ≤u t ≤u max , Then no matter how the coefficient a changes, the entire quadratic form is less than zero. Therefore, the necessary condition for the existence of the solution, that is, the robust condition (33), is written as:

[0279]

[0280] The above necessary conditions require that the controller has sufficient ability to exceed the t to x t+1 The uncertainty range of the proposed method is used to find infeasible states by utilizing the necessary condition, and the attraction domain D is improved to eliminate the states that cannot be guaranteed by this method.

[0281] Since the system (2) is piecewise linear, the control evaluates at its boundary vertices to obtain each state x i The reachable domain vertex set Get a cone for each vertex control, then construct the union of these cones, and get a new cone by approximation

[0282] Now, For each vertex x in i Corresponding to a cone in space The union of these cones C is the solution space of a, with Figure 2 As an example, the cones corresponding to the four vertex states are considered. For the cones to be aligned, and considering the constraints imposed by formula (30), only four cones are shown. The intersection of these four cones is approximately a linear cone, such as Figure 2 As shown by the red dots in .

[0283] Without loss of generality, assume that the cone has N intersection points v1, v2....v N (like Figure 1 These intersection points form N planes in the solution space, with normal vectors n1, n2, ....n N The linear cone formed by these N intersection points is represented by computing for each plane:

[0284] n i =vi ×v i+1 (45);

[0285]

[0286] Then, the Lyapunov candidate function The solution for the coefficients is:

[0287]

[0288] Where b1, b2, a are the coefficients of the HL-PWL representation of the Lyapunov function and its boundary. Before applying the proposed RGRL to learn the controller and the coupled Lyapunov function, a preliminary verification of the attraction domain is performed using the solution space (47). So far, the control problem has been linearized for approximation. Before training the neural network, this planning problem can be used for preliminary verification and the characteristics of linear programming can be used to analyze this control problem.

[0289] Determination of attraction area

[0290] In the existing literature, the attraction region D is often given. Some vertices in V may not satisfy the necessary conditions (44), which makes the problem unsolvable. Before applying the proposed RGRL, the necessary conditions are used to refine the given attraction region D. Given the upper and lower bounds u min u max and uncertainty bound Δ ∈ (x) and Δ δ (x), the vertices that satisfy the necessary condition (44) in the attraction region D form a new set Then we get a fine attraction area if The vertices in satisfy the necessary condition (44), then All states in satisfy this condition.

[0291] Robust Conditional Linearization Inference

[0292] Each vertex has a reachable domain From each vertex x t ,Right now Attracting regions by refining Any state x in t , both satisfy the necessary condition (44).

[0293] Proposition 1: When the proposed RGRL is used, when ρ < ρ max The uncertainty bound Δ ∈ (x) and ΔΔ δ (x) is enlarged, the controller becomes more robust, where ρmax is the maximum value, any ρ>ρ max will make RL unsolvable, where:

[0294]

[0295] Proof 1: Due to the necessary condition, the first term in equation (44) is positive, while the others are negative. This shows that the quadratic form forms a cone of the second kind (also called a hyperbolic cone) in space. To calculate the cone angle of this cone, the cone angle can be approximated by considering the angle between the cone and the major axis. In addition, the major axis of the cone is aligned with the direction of the eigenvector corresponding to the positive eigenvalue, then the half angle θ of the cone can be approximated by the following relationship:

[0296]

[0297] When the uncertainty of the system changes from Increase to When Then the new half angle θ′ becomes the following formula:

[0298]

[0299] Since the arc tangent function is monotonically increasing, when the uncertainty of the system is magnified by the proportionality factor ρ, the cone angle of the quadratic shape decreases:

[0300]

[0301] in, and represents the solution space. In addition, the size of the cone angle is equivalent to the solution space As the cone angle decreases, the solution space decreases to This makes the proposed RGRL approach challenging but results in a controller that is able to handle the increase in uncertainty.

[0302] Then we obtain the necessary condition (44) for the maximum value of the coefficient ρ. By reducing the above state space, all vertex states should satisfy the necessary condition. u min ≤u≤u max , the upper limit of the amplification factor ρ max The following conditions should be met:

[0303]

[0304] S5. Perform numerical simulation on inverted pendulum and quadrotor drone.

[0305] The proposed robustness-guaranteed RGRL is compared with Lyapunov neural network control (NNC) through numerical simulations of an inverted pendulum and a quadrotor drone, where the multiplicative and additive uncertainties in the dynamic model and state estimation are specifically considered. The form of the multiplicative and additive uncertainties is shown in Equation (3). Both training and testing are performed on an Intel i7 CPU.

[0306] A. Inverted Pendulum

[0307] The simulated inverted pendulum is 1 meter long, weighs 1 kg, and the acceleration due to gravity is set to 9.81 m / s 2 The simulation is performed in Gym, and the torque control range is -100Nm to 100Nm. Let θ represent the angle between the pendulum and the horizontal x-axis. Let the equilibrium point be θ = 0, The attraction domain D of the state space (angle and angular velocity) is specified as [0, 2π] × [0, 4]. The controller and Lyapunov function are synthesized using the RGRL proposed in the present invention, and the resulting controller is called a robust neural controller RNC. Three scenarios are simulated to illustrate the effectiveness of RNC in handling various uncertainties.

[0308] In scenario I, only the multiplicative observation uncertainty δ is considered m In scenario II, the multiplicative observation uncertainty δ m and the multiplicative model uncertainty δ a Scenario III considers various uncertainties δ m , δ a ,∈ m and ∈ a .

[0309] 1) Scenario I: Multiplicative Observation Uncertainty of State Estimation:

[0310] In scenario I, the multiplicative observation uncertainty ∈ m Set to [0.01 0.01] T , NNC and RNC are obtained from Lyapunov neural control and the proposed robust guaranteed RL, respectively. NNC and RNC converge in 3 and 5 hours. The controller is tested with 30 different initial states and the trajectories obtained are Figure 3 In the figure, various colors are used to visually represent the equilibrium point. In the case of NNC, the system is stable at Figure 3 (a) The blue star state shown.

[0311] The difference between the two stars highlights the multiplicative observational uncertainty ∈ m In the case of, NNC cannot effectively stabilize the inverted pendulum at the desired equilibrium point. Figure 3As shown in (b), the trajectories derived by RNC converge to the desired equilibrium point. Figure 3 The same conclusion can be drawn from the evolution of the Lyapunov functions over time shown in (c) and (d). Initially, the Lyapunov values ​​generated by both methods decrease. The Lyapunov values ​​generated by RNC show a consistent monotonic decrease and eventually approach zero. In contrast, the Lyapunov values ​​generated by NNC show an increase before converging to a non-zero constant value.

[0312] The control signals corresponding to the two methods are as follows: Figure 3 (e) and (f). Similar to the Lyapunov value, the control signal from the NNC converges to a non-zero value, while the control signal from the RNC converges to zero. The RNC proposed in the present invention strictly regulates the training process of the controller by a robustness loss based on the uncertainty of the state estimation. This method effectively offsets the impact of this uncertainty and ensures that the system converges to the desired equilibrium state. The robustness guarantee provided by the RNC can be evaluated by evaluating the value function based on the robustness constraint.

[0313] Figure 4 (a) and (b) show example trajectories generated by NNC and RNC and their corresponding Lyapunov functions, respectively. The Lyapunov functions of both methods are visualized as 3D cones with contours of equal values ​​projected onto the base. These contours are centered at the equilibrium point. The red star indicates the initial state and the blue star indicates the converged state. The trajectories are shown next to the Lyapunov function values, and the system state evolves along the Lyapunov function cone and converges toward the base. In the example Figure 4 In the presence of observation uncertainty in the state estimate shown, NNC has difficulty reaching the desired equilibrium point.

[0314] 2) Scenario II, multiplicative uncertainty in dynamics model and state estimation:

[0315] The uncertainty of the dynamic model δ m Set to [0.01 0.01] T , the uncertainty in the state estimate ∈ m Set to [0.01 0.01] T .like Figure 5 As shown in (a), the NNC has difficulty handling uncertainties in the dynamic model and state estimation, causing the trajectory to gradually deviate from the desired equilibrium point. As a result, the inverted pendulum continues to rotate without reaching a stationary state. This perpetual motion causes the overall energy of the controlled system to continue to increase, resulting in an exponential increase in the Lyapunov value and control saturation, respectively. Figure 5 (c) and Figure 5 (e) as shown.

[0316] In contrast, RNC cleverly mitigates the effects of these two uncertainties, effectively stabilizing the control system and guiding it toward the desired equilibrium state. Figure 5 (d) is shown. Figure 5 In (b), all trajectories converge quickly to a diagonal trajectory, while in Figure 5 In (b), the trajectory starting from the initial state of [0.5π, 1.5π] deviates a little and then converges to the diagonal line.

[0317] This is because the multiplicative uncertainty requires a larger control input when the state is far from the equilibrium point, e.g. Figure 5 (f) At the beginning of the trajectory, the control is usually saturated, resulting in a rapid convergence to the diagonal trajectory. Figure 5 The monotonic decrease of the Lyapunov value in (d) verifies the guaranteed robustness. An example trajectory generated by NNC and RNC and its corresponding Lyapunov function are shown in the figure.

[0318] like Figure 6 An example trajectory generated by NNC and RNC and its corresponding Lyapunov function are shown in (a) and (b), respectively. The Lyapunov functions of both methods are visualized as 3D surfaces, and the contours of isovalues ​​are projected onto the base. These contours are centered at the equilibrium point. The red star indicates the initial state and the blue star indicates the desired equilibrium state. The trajectory is shown next to the Lyapunov function value. The system state progresses along the Lyapunov function cone and converges to the base.

[0319] Under the combined influence of the dynamic model uncertainty and the state estimation uncertainty, the NNC proves to be ineffective in guiding the trajectory of the inverted pendulum along the Lyapunov surface, resulting in unstable motion with oscillating trajectories, causing the controller output to remain in a saturated state. In contrast, the RNC effectively mitigates the effects of these uncertainties, causing the trajectory to descend smoothly along the Lyapunov surface and reach the target position.

[0320] 3) Scenario III, Multiplicative and Additive Uncertainties in Dynamic Models and State Estimation:

[0321] In addition to the multiplicative uncertainty, scenario III also models the additive uncertainty in the dynamics model and state estimation. a and δ a Set to [0.01 0.01] respectively T and 0.01. exist Figure 7 The graphs correspond to the performance of the two controllers under additive and multiplicative uncertainty. The individual trajectories in the graph correspond to Figure 5A specific trajectory from the set of trajectories of the controller is shown. It can be observed that the trajectories under multiplicative uncertainty converge smoothly to the equilibrium point, where both the Lyapunov function and the control function demonstrate the ability to stably converge to the equilibrium state.

[0322] Next, we focus on the trajectory under additive uncertainty; as the trajectory approaches the equilibrium state, oscillations in the state are observed instead of stable convergence. In addition, the Lyapunov function fluctuates near the convergence point, indicating that the controller can only guarantee stable convergence within a certain range. Within this range, there is no guarantee that the value of the Lyapunov function will continue to decrease.

[0323] In general, in an uncertain system containing a constant term, the method provided by the present invention can ensure that the overall energy is reduced. The convergence range that cannot be guaranteed in the end is related to the constant term in the uncertainty.

[0324] B. Refinement of the Attraction Domain

[0325] In this example, the uncertainty ∈ in the dynamics model and state estimation is set to [0.04 0.04] T , the attraction region is refined before applying the proposed RGRL. The results are shown in Figure 8 As shown, the blue regions fail to meet the necessary conditions and are subsequently excluded, and the refined attraction regions are depicted as red dashed boxes.

[0326] C. Enhanced robustness

[0327] During training, the uncertainty is intentionally amplified by a factor of ρ = 1.1, resulting in a second RNC (called enhanced RNC). Fig.10 In the figure, the blue circle indicates the state where the Lyapunov value increases when the conventional RNC is used, while the red circle indicates the state where the Lyapunov value increases when it is amplified by the amplification factor. The results show that using the enhanced RNC performs better than the conventional one, resulting in a smaller unstable area.

[0328] D. Learning Lyapunov functions

[0329] Visualize the learning process of the Lyapunov function. In RGRL, the Lyapunov function is trained based on the loss in the formula. Formulas (17), (18), and (19) implement the maximization operation in the two-level optimization by solving the MILP problem. The degree of violation of the robustness condition is analyzed. Fig. 9In Figure 1, the present invention describes three loss snapshots of the entire state space after 1, 20, and 100 training epochs, respectively. Initially, the state space is mainly yellow-green, indicating that the robustness conditions of the initial controller network and the Lyapunov network are violated. Initially, the "counterexample" states are those far from the equilibrium state. As training progresses, the "counterexample" states gradually approach the equilibrium state. Fig. 9 This is supported by the example in , where during training we enforce prioritization of peripheral states to satisfy the robustness condition.

[0330] E. 2D quadcopter

[0331] Simulate a two-dimensional quadcopter with a rotor arm length of 0.25m, a quadcopter mass of 0.486kg, and a moment of inertia of 0.0083kgm 2 , the acceleration due to gravity is 9.81 m / s 2 The activation area is designed to The region verifies the Lyapunov condition, and the NNC and RNC converge within 6 and 12 hours, respectively.

[0332] Two scenarios were simulated: scenario VI included only the multiple uncertainties in the dynamics model and state estimation, while scenario V included only additive uncertainties, where δ m , δ m , δ m , and δ m is set to [0.04 0.04] T , [0.04 0.04] T , [0.40.4] T and [0.4 0.04] T In both scenarios, the aircraft is launched in a box domain, such as Fig.11 The black box shown, the equilibrium state is set to [0, 0] T .

[0333] The trajectories obtained by NNC and RNC in scenario VI are as follows Fig.11 (a). Under RNC control, the aircraft can stably fly to the equilibrium point shown in red, while NNC cannot stabilize the aircraft and later flew out of the black box. This is also obvious in the comparison of Lyapunov values. The Lyapunov value of NNC increases from the beginning, while the Lyapunov value of RNC decreases monotonically and converges to zero after 10 seconds. The control signals of NNC and RNC are shown in Fig.11 As shown in (e), it can be seen from the figure that NNC cannot stabilize the aircraft, and its control is continuously saturated after 2 seconds, while RNC reaches saturation at the beginning and drops to a constant value after the aircraft is stabilized.

[0334] The trajectories obtained by NNC and RNC in scenario V are as follows Fig.11 (b). In this case, both NNC and RNC are able to bring the vehicle to the desired equilibrium point. However, RNC enables the vehicle to fly stably to the equilibrium point shown in red, while NNC fails and cannot stabilize the vehicle at the equilibrium point. This is also evident in the comparison of the Lyapunov values. The Lyapunov values ​​of both methods decrease, but the Lyapunov value of the NNC method may increase from time to time, indicating that robustness is no longer guaranteed. The Lyapunov value of the RNC decreases monotonically and converges to zero after 10 seconds. The control signals of the NNC and RNC are shown in Fig.11 As shown in (f), the results show that the output control of NNC has frequent fluctuations, while the output signal of RNC is smooth and has a certain robustness.

[0335] Therefore, the present invention adopts the above-mentioned Lyapunov-stabilized reinforcement learning control algorithm for uncertain systems based on the worst case scenario for synthesizing a neural network controller to maintain the stability of the system under bounded multiplicative and additive uncertainties in the dynamic model and state estimation. In order to ensure the robustness of the neural network, a set of inverse business losses are proposed to train the Lyapunov function of the neural network. Since the maximization of the loss function is converted into a mixed integer linear programming, the two-level optimization is solved by the Gurobi solver. A robustness criterion is proposed to ensure the solvability of the RL problem and is used to determine the possible attraction domain. Guaranteed robustness is shown by testing on a simulated inverted pendulum and a two-dimensional quadcopter and comparing the performance with existing methods. The robustness criterion strictly requires that the Lyapunov value decreases along the system trajectory, making the solution too conservative.

[0336] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. Based on the worst-case uncertainty system Lyapunov stable reinforcement learning control algorithm, it is characterized by: The following steps are involved: S1, modeling the system dynamics model and uncertainty boundary through the relu activation network; S2, determining a robustness condition and using it to predetermine the area of ​​the attraction domain; S3, determine the robustness guarantee RL under the uncertainty of dynamic model and state estimation; S4, network parameterization establishment; S5. Perform numerical simulation on inverted pendulum and quadrotor drone.

2. The Lyapunov-stabilized reinforcement learning control algorithm for the worst-case uncertainty system according to claim 1, characterized in that: In step S1, the system dynamics model and uncertainty boundary are modeled, and the specific process is as follows: Consider a discrete-time robot system whose dynamics model is: x t+1 =f(x t ,u t ),u min ≤u t ≤u max (1); Among them, x t ∈R nx and u t ∈R nu They represent the robot's state and control input at time t respectively, and the control saturation is given by the lower bound u min Or upper bound u max given; The dynamic model is f(x t ,u t ), assuming its approximate value It is obtained through system identification as shown below: Among them, δ m and δ a denote the multiplicative uncertainty and additive uncertainty between the actual dynamics and its approximation, respectively, and δ xt represents the total model uncertainty, u t =π(x t ) indicates that the controller π is in state x t The control given at However, the system state x is estimated from some estimator and is therefore corrupted by noise, as shown below: where ∈ m and ∈ a denote the multiplicative uncertainty and additive uncertainty of the state estimation, ∈(x t ) represents the aggregate uncertainty in the state estimate; Therefore, the predicted state at time t+1 is: in, It means that the controller π is used in the state estimation The control given at For an approximate kinetic model State estimator that outputs system estimates Control saturation u min and u max , and a domain of attraction D, find a controller Let the system stabilize at the equilibrium point under model uncertainty δ and state uncertainty ∈; Assumption 1: An approximate dynamics model is given in the form of a relu activated neural network Assumption 2: Assume that the uncertainty of the system model dynamics δ(x) and the uncertainty of the state estimation ∈(x) are bounded, and their bounds are denoted by Δ δ (x) and Δ ∈ (x), then: Among them, |·| is the infinite norm, Δ δ (x) and Δ ∈ (x) are all given in the form of a relu activated neural network.

3. The Lyapunov-stabilized reinforcement learning control algorithm for the worst-case uncertainty system according to claim 2, characterized in that: In step S2, the robustness condition is the stability condition under the worst case uncertainty, which is derived from Lyapunov neural control. When uncertainty is not involved, the coupled Lyapunov function V(x t ): R nx →R to ensure the stability of the system: V(x * )=0 (6); Where D is the attractive state area, x* is the equilibrium state, V max (x t ) and V min (x t ) belongs to the set of K functions, and the stability conditions (6) to (8) guarantee that the system starting from any state in D converges to the equilibrium state x*.

4. The Lyapunov-stabilized reinforcement learning control algorithm for the worst-case uncertainty system according to claim 3, characterized in that: In step S2, the stability condition under the worst case uncertainty includes the robustness condition under the uncertainty of the dynamic model and the robustness condition under the uncertainty of the state estimation: Under the model uncertainty δ(x) in the described dynamics, the state at time t+1 is not completely determined by the control u t and the current state x t Therefore, Equation (8) must be used for each possible state at time t+1 are all valid, then: in, By x t+1 The Lyapunov function at is subjected to the Zinle expansion, and its robustness condition (8) becomes: Given The randomness of , and based on assumption 2, a stronger and more sufficient robustness condition is obtained as follows: in, and The ratio is negligible, taking into account the worst case uncertainty in the dynamics modeling and forcing the Lyapunov value to decrease along the state trajectory; With a given ground truth state x t The control input u obtained t =π(x t ) On the contrary, the controller π actually estimates is calculated to determine the control input Therefore, the stability condition needs to take into account the randomness of the state estimate; considering the worst-case uncertainty in the state estimate, The Lyapunov value at is obtained by Qinle expansion as follows: The derivative of the Lyapunov function is obtained by the chain rule, as follows: among them,x t+1 =f(x t ,in t ),in t =π(x t ); Then the robustness condition under uncertainty in state estimation is as follows: Under assumption 2, the stability condition (8) becomes the new robustness condition as follows: in, is ignored, the derivative and are the piecewise constants in x, u, and x respectively.

5. The Lyapunov-stabilized reinforcement learning control algorithm for the worst-case uncertainty system according to claim 4, characterized in that: In step S3, let α represent the trainable parameters in the Lyapunov network, β represent the trainable parameters in the controller network, and use the robustness conditions (7), (12) and (16) to design the following loss function to adjust the training of the controller network and the Lyapunov network. The loss function that guarantees the semi-positive definition of the Lyapunov network (7) is given as: When only uncertainty δ is considered in the model dynamics, the Lyapunov value along the trajectory should increase strictly monotonically and be guaranteed by satisfying the robustness condition (12). The corresponding loss function is designed as: When only the uncertainty ∈ in the state estimation is considered, the loss function (16) established by the robustness condition is expressed as: Based on the robustness condition (16), for a system with only uncertainty in the dynamic model, the loss function is constructed: When the above two uncertainties are considered simultaneously, the robustness condition (8) becomes: Then, the loss function corresponding to the robustness condition eq is obtained, which is expressed as: Express the objective function and all constraints as piecewise linear functions of x: Let θ = (α, β) represent the set of all trainable parameters. In each iteration of training, the value θ of the previous iteration is used, and the division of the attraction domain S is: Let y i is a binary if and only if x is equal to 1 in the i-th partition, then x is represented as a convex combination of the extreme points of the partition. By introducing slack variables, the optimization problem of the loss function is transformed into a mixed integer linear programming: stA θ s=b θ (22b); Among them, the problem coefficient c θ d θ , A θ and b θ are all explicit functions of θ, and the optimization variable vector s consists of x and y1..., then the optimal value of the objective function (22) is written as: Using Gradients The trainable parameters θ are optimized via back-propagation.

6. The Lyapunov-stabilized reinforcement learning control algorithm for the worst-case uncertainty system according to claim 5, characterized in that: In step S4, the network parameterization establishment process is as follows: Using the HL-PWL method, the following expression is used to represent any piecewise linear function F(x): in, is a coefficient vector, is a stack of basis functions, basis functions The stack includes all nested functions at nesting level i; the coefficient vector a i With nested functions associated; Robustness conditions under HL-PWL representation: Correct characterization of the Lyapunov function, formula (7) is rewritten as: Where ξ1(|x|) and ξ2(|x|) are K functions and increase with the increase of |x|. Both the Lyapunov function V(x) and the K function are rewritten in HL-PWL form (23), expressed as: Among them, |·| is the infinite norm, and is the parameter vector, ζ a and i is the number of corresponding parameter vectors, the upper and lower bounds ξx(|x|), and μ(|x|)=[μ 0 (|x|), μ 1 (|x|)] T is a vector function, including nested functions at nesting level 0 and nesting level 1. It is easy to get (i.e. |x|max); Since μ is a K function, we have: g i (0)=0 (27); ξ i (|x t+1 |)>ξ i (|x t |),if|x t+1 |>|x t | (28); The robustness condition (6) and positivity condition (7) under equilibrium conditions are written as: and Given an attraction domain D, partition and define vertices. Then, condition (30) is transformed into the form of 6 in Matrix W 11 and W 12 is obtained by superimposing all x in V j Vector obtained by superposition, where V is the set of D partitions; Similarly, the upper bound condition of formula (24) can be rewritten as Then the upper bound ξ1(|x|) and the lower bound ξ2(|x|) are K functions that satisfy equation (27), that is: m T (0)b i =0.i=1,2 (31); And the function that satisfies (28), that is: W2 T b i <x * ,i=1,2 (32); where W2 = {μ(0)-μ(γ), …, μ((m-1)γ)-μ(mγ)} and γ is the grid size of the partition. When considering the uncertainties in dynamic modeling and state estimation simultaneously, the robustness conditions (20) and (14) yield another condition: It is further assumed that both the dynamical system and the controller satisfy the following Lipschitz condition: where u and F represent the upper bounds of the controller and positive dynamics changes, respectively; Then, the term V(x t+1 )-V(x t ) in state x t+1 The resulting Qinle expansion is: By negation And using HL-PWL representation, formula (36) is written as: in, express The derivative of x is t+1 , then the robust condition (33) is written as: It can be written as the following quadratic inequality: Where (x t+1 -x t ) is a column vector, is a real symmetric matrix: Since X is a rank 1 matrix, it has a nonzero eigenvalue equal to (xt +1 -x t ) is the square of the norm of |x t+1 -x t | 2 , the remaining eigenvalues ​​are zero, let but is rewritten as: where Q is an orthogonal identity matrix QQ T =I, so formula (39) is further rewritten as: make Then it satisfies: Formula (43) shows that Convert to standard quadratic form, There must be a correct solution; However, when the control is saturated at the lower and upper bounds, if for all u min ≤u t ≤u max , Then no matter how the coefficient a changes, the entire quadratic form is less than zero. Therefore, the necessary condition for the existence of the solution, that is, the robust condition (33), is written as: The above necessary conditions require that the controller has sufficient ability to exceed the t to x t+1 The uncertainty range of the method is used to find the infeasible state by using the necessary condition, and the attraction domain D is improved to eliminate the state that cannot be guaranteed by the method. Since the system (2) is piecewise linear, the control evaluates the reachable domain vertex set of each state x at its boundary vertex Get a cone for each vertex control, then construct the union of these cones, and get a new cone by approximation Now, For each vertex x in i Corresponding to a cone in space The union of these cones C is the solution space of a. Consider the cones corresponding to the four vertex states. For the cones to be aligned, and considering the constraints imposed by formula (30), only four cones are shown. The intersection of these four cones is approximately a linear cone. Without loss of generality, assume that the cone has N intersection points v1, v2....v N , these intersection points form N planes in the solution space, with normal vectors n1, n2....n N The linear cone formed by these N intersection points is represented by computing for each plane: n i =v i ×v i +1 (45); Then, the Lyapunov candidate function The solution for the coefficients is: Where b1, b2, a are the coefficients of the HL-PWL representation of the Lyapunov function and its boundary. Before applying the proposed RGRL to learn the controller and the coupled Lyapunov function, the attraction domain is preliminarily verified using the solution space (47).

Images