An unmanned system online self-adaptive learning tracking control method and device

Abstract

The application discloses a tracking control method and device for online self-adaptive learning of an unmanned system. The application can learn an optimal tracking control input by using only data information of an intelligent unmanned system under the condition that a mathematical model of the unmanned system is unknown, so that an optimal tracking control task is realized. The application does not need an initial stable control strategy, a full-rank data matrix array, a restriction condition of continuous excitation and a data storage unit, has low calculation complexity and is easy to implement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent unmanned system, and particularly relates to a tracking control method and device for online adaptive learning of unmanned system. BACKGROUND

[0002] An unmanned system is a kind of autonomous system without human intervention, and is a product of high combination of informatization and mechanization. On one hand, the unmanned system is autonomous, and on the other hand, the unmanned system realizes a collaborative operation mode of “no one / no one”. The unmanned system includes unmanned aerial, ground and marine systems, and is represented by systems such as unmanned aerial vehicles, unmanned vehicles, unmanned ships, robots, unmanned clusters and unmanned manned hybrid systems. At present, intelligent unmanned systems of various types have formed application products, mainly including robot exoskeleton systems, intelligent ground vehicles, underwater vehicles, unmanned aerial vehicles, unmanned vehicles, service-oriented robots, intelligent industrial robots, space robots, marine robots, unmanned workshop / intelligent factory, intelligent mine unmanned inspection system, ground unmanned system, intelligent inspection robot of marine unmanned platform, intelligent unmanned loading system and intelligent unmanned workshop. With the diversification and refinement of control tasks of the unmanned system, on one hand, a mechanism modeling method cannot accurately establish a mathematical model of the unmanned system, and on the other hand, a traditional steady-state control method cannot realize optimal control tasks. In order to solve the two problems, an optimal control method based on reinforcement learning emerges as the times require. The optimal control method based on reinforcement learning generates a reward by letting the controlled system interact with the external environment, and then selects a behavior action according to the reward obtained at present, and finally learns an optimal behavior action. At present, the optimal control method based on reinforcement learning has been widely researched and applied, but most of the results require storage of historical system data, require continuous incentive conditions and require known initial stable control strategies. However, the requirement of storage of historical system data means that additional storage resources need to be occupied, the requirement of continuous incentive conditions will have a negative impact on the system, and the requirement of known initial stable control strategies means that the system model needs to be known in advance. Therefore, from the above analysis, it can be seen that these requirements cannot adapt to actual systems, so how to relax these requirements is a problem that needs to be further studied. SUMMARY

[0003] Therefore, the present application provides an online adaptive learning tracking control method for unmanned system, which does not need to occupy a large amount of additional storage resources to store historical data, does not need continuous incentive conditions and initial stable strategies, has low computational complexity and is easy to implement.

[0004] The present application provides an online adaptive learning tracking control method for unmanned system, which includes the following steps:

[0005] Step 1: an intelligent unmanned system model in the following form is established:

[0006]

[0007] in and These represent the states of the follower and leader intelligent unmanned systems, respectively. This indicates the need for further design of a follower-based intelligent unmanned system control input. and Let A, B, C, E, and F represent the outputs of the follower and leader intelligent unmanned systems, respectively, and let F represent unknown coefficient matrices. Let represent n-dimensional, q-dimensional, g-dimensional, p-dimensional, and r-dimensional real vector spaces, respectively.

[0008] Step 2: Based on the intelligent unmanned system model established in Step 1, design the control input u(t) to achieve optimal control performance by solving a static optimization problem and a dynamic optimization problem. The dynamic optimization problem is defined as follows:

[0009]

[0010]

[0011] In the above formula, Q and R represent the given weight matrices, and

[0012] The optimal feedback control input can be obtained by solving the dynamic optimization problem. for

[0013]

[0014] Where K x =R -1 B T P, where P is a positive definite matrix and is an algebraic Riccati equation A. T P+PA-PBR -1 B T The unique solution to P+Q=0.

[0015] The static optimization problem is defined as follows:

[0016]

[0017] In the above formula M represents the given weight matrix, and X and U are the matrices to be solved.

[0018] The optimal feedforward control input can be obtained by solving the static optimization problem. for

[0019]

[0020] where K v = U - K x X.

[0021] The actual optimal control input u(t) can be obtained by combining the optimal feedforward control input and the optimal feedback control input

[0022]

[0023] Step 3: Based on the actual optimal control input obtained in step 2, further design a reinforcement learning optimal control learning algorithm based on system data to calculate the optimal control input under the condition that the system model is unknown.

[0024] Specifically, the following sub-steps are included:

[0025] Step 3.1: Select parameters K0=0, Q≥0, R>0, α0>0, σ∈(0,1), ε>0 and matrix sequence where K0 and α0 represent the control gain and transition operator at the initial iteration, respectively, and σ and ε are two given constants.

[0026] Step 3.2: Apply an arbitrary control input to the system, where is a given exploration noise.

[0027] Step 3.3: Construct the dynamic equation where h l (t) and m l (t) are two variables. Then find the time t that makes the determinant of Z f (t) greater than 0, and let T←t.

[0028] Step 3.4: Select an arbitrary parameter γ>0, let i=0, l=0, and solve P i+1 from the following equation:

[0029]

[0030] where

[0031]

[0032] and vec(·) represents converting an m×n matrix into an mn column vector, and vech(·) represents converting a symmetric matrix by column block and converting it into a column vector while removing symmetric elements.

[0033] Step 3.5: Check if the condition P1>0 is true, if true then go to Step 3.6, otherwise continue to Step 3.4 until the condition P1>0 is true.

[0034] Step 3.6: Let i = 1.

[0035] Step 3.7: Solve P i+1 and K i+1 from the equation in Step 3.4.

[0036] Step 3.8: Update the transition operator α i+1 according to the following equation:

[0037]

[0038] Step 3.9: Let i <- i + 1, check if the condition α i+1 >γ is true, if not true then repeat Steps 3.7 to 3.8 until the condition is true, otherwise go to Step 3.10.

[0039] Step 3.10: Let

[0040] Step 3.11: Continue to solve P i+1 and K i+1 from the equation in Step 3.4.

[0041] Step 3.12: Let i <- i + 1, check if the condition ||P i+1 -P i || <ε is true, if not true then repeat Steps 3.11 to 3.12 until the condition is true, otherwise go to Step 3.13.

[0042] Step 3.13: Let K x = K i+1 , and l = 1, and the optimal feedback control input is

[0043]

[0044] Step 3.14: Solve (X l ) from the equation in Step 3.4. A

[0045] Step 3.15: Let l <- l + 1, check if the condition l = m is true, if not true then repeat Steps 3.14 to 3.15, otherwise go to Step 3.16.

[0046] Step 3.16: Construct the output regulator equation:

[0047] Λζ = Υ ​

[0048] where

[0049]

[0050] ζ = [α2... α m vec T (X)vec T (U)] T

[0051]

[0052] and I qm and I nqm are the qm x qm and n x n identity matrices, respectively, and α2,...,α m are constants, A (X l ) = X l E - AX l , l = 1, 2,... m.

[0053] Step 3.17: Apply row transformations to the output regulator equation in Step 3.16 and separate vec T (X) and vec T (U) from ζ to obtain the following equation:

[0054]

[0055] where Θ 11 , Θ 12 , Θ 21 , Θ 22 , Γ1 and Γ2 are the transformed block matrices.

[0056] Step 3.18: Solve for vec

[0057]

[0058] where λ is a given Lagrange multiplier and M is a given weight matrix.

[0059] Step 3.19: Calculate the optimal feedforward control input

[0060] Step 3.20: Calculate the optimal control input

[0061] Step 4: Apply the control input calculated in Step 3 to the system to achieve the optimal tracking control task for the unmanned system.

[0062] The application also provides a tracking control device which adopts the above method to perform optimal tracking control.

[0063] Beneficial effects:

[0064] The online data-driven optimal tracking controller learning method provided by the application can learn an optimal tracking control input by using only data information of an intelligent unmanned system in the case that a mathematical model of the unmanned system is unknown, so as to realize an optimal tracking control task. The algorithm has the advantages of not needing initial stable control strategies, data matrix array full rank and continuous excitation limit conditions and data storage units, so that the calculation complexity is low and the algorithm is easy to implement. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 The application provides a method flowchart.

[0066] Figure 2 The application provides an optimal tracking control effect diagram of an unmanned system. DETAILED DESCRIPTION

[0067] The application will be described in detail below in combination with the drawings and embodiments.

[0068] The application provides an online adaptive learning tracking control method of an unmanned system, a flowchart of which is shown in Figure 1 The method specifically includes the following steps.

[0069] Step S1: an intelligent unmanned system model is established as

[0070]

[0071] Wherein x(t) and v(t) respectively represent states of follower and leader intelligent unmanned systems, u(t) represents a control input of the follower intelligent unmanned system to be further designed, and y(t) and r(t) respectively represent outputs of the follower and leader intelligent unmanned systems. C=[1,0], F=[-1,0].

[0072] Step S2: according to the intelligent unmanned system model established in step S1, a static optimization problem and a dynamic optimization problem are designed to design the control input u(t) to realize optimal control performance, wherein the static optimization problem is designed as

[0073]

[0074] s.t.AX+BU=XE

[0075] CX+F=0

[0076] In the above formula denotes a given weight matrix, X and U are matrices to be solved.

[0077] The optimal feedforward control input can be obtained by solving the static optimization problem

[0078]

[0079] The dynamic optimization problem is defined as

[0080]

[0081] In the above equation, R = 5 respectively denote weight matrices, and

[0082]

[0083] The optimal feedback control input can be obtained by solving the dynamic optimization problem

[0084]

[0085] The actual optimal control input u(t) can be obtained by combining the optimal feedforward control input and the optimal feedback control input as

[0086]

[0087] Step S3: Design a reinforcement learning optimal control algorithm based on system data according to the actual optimal control input u(t) obtained in step S2, and calculate the optimal control input without using the system model.

[0088] The parameters selected in step 3.1 are K0 = 0, a0 = 0.2, e = 0.001 and the matrix sequence

[0089]

[0090] Apply in step 3.2 where

[0091] In step 3.3, by judging the determinant of Z f (t), it can be obtained that when T = 3s, its determinant is not equal to 0.

[0092] The parameter selected in step 3.4 is g = 0.1.

[0093] In step 3.5, when g gradually increases to 6, a P1 which is a positive definite matrix can be obtained.

[0094] ​​Let i = 1, l = 0 in Step 3.6.

[0095] The solutions P6and K6in Step 3.7 are

[0096]

[0097] The parameter α6= 6.1414 in Step 3.8.

[0098] In Step 3.9, it can be judged that the condition is satisfied when i = 6, i.e., α6> γ.

[0099] In Step 3.10, let

[0100] The solutions P 21 and K 21 in Step 3.11 are

[0101]

[0102] In Step 3.12, it can be judged that the condition is satisfied when i = 21, i.e., ||P 21 -P 20 ||≤ 0.001, where P 20 and K 20 are

[0103]

[0104] In Step 3.13, let K x = K 21 = [0.13577 0.13882], and l = 1, and the optimal feedback control input is

[0105]

[0106] The solutions A (X0), A (X1), A (X2) in Step 3.14 and Step 3.15 are

[0107]

[0108] The optimal feedforward control input in Step 3.16 to Step 3.19 is

[0109]

[0110] The optimal control input in Step 3.20 is

[0111]

[0112] Step 4: optimal control input obtained according to step S3 The optimal tracking control task is applied to the system to realize an unmanned system, and an effect diagram is as shown in Figure 2 It can be seen from the diagram that the black dashed line (the trajectory of the intelligent unmanned system) can gradually track the red solid line (representing the trajectory to be tracked).

[0113] To sum up, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A tracking control method for online adaptive learning of an unmanned system, characterized in that, Comprise: Step 1: Establish intelligent unmanned system model: Wherein x(t) and v(t) respectively represent the state of the follower and the leader intelligent unmanned system, u(t) represents the control input of the follower intelligent unmanned system to be further designed, y(t) and r(t) respectively represent the output of the follower and the leader intelligent unmanned system, A, B, C, E and F represent unknown coefficient matrices respectively; Step 2, the optimal feedback control input is obtained by solving a dynamic optimization problem The optimal feedforward control input is obtained by solving a static optimization problem The actual optimal control input u(t) is where The dynamic optimization problem is: where Q and R represent given weight matrices, respectively, and Optimal feedback control input For where K x = R -1 B T P, P is a positive definite matrix, and is the unique solution of the following algebraic Riccati equation A T P + PA - PBR -1 B T P + Q = 0; The static optimization problem is s.t.AX+BU=XE CX+F=0 wherein, M represents a given weight matrix, X and U are matrices to be solved. Optimal feedforward control input For where K v = U - K x X; Step 3, the reinforcement learning optimal control learning algorithm based on system data calculates the optimal control input under the condition that the system model is unknown; Specifically: Step 3.1: Choose parameters K0= 0, Q > 0, R > 0, a0> 0, σ ∈ (0, 1), ε > 0 and a sequence of matrices where K0and a0denote the control gain and the transition operator at the initial iteration, respectively, and σ and ε are two given constants; Step 3.2: Apply arbitrary control input to the system, where is a given exploration noise; Step 3.3: Constructing the dynamic equation where h l (t) and m l (t) are two variables; then find the time t that makes the determinant of Z f (t) greater than 0, and let T <- t; Step 3.4: Choose an arbitrary parameter γ > 0, let i = 0, l = 0, and solve for P from the following equation i+1 : Wherein And vec(·) represents converting an m*n dimensional matrix into an mn dimensional column vector, vech(·) represents converting a symmetric matrix by column block and converting into a column vector while removing symmetric elements; Step 3.5: Verify whether the condition P1>0 is established, if it is established, execute step 3.6, otherwise continue to execute step 3.4 until the condition P1>0 is established; Step 3.6: Let i=1; Step 3.7: Solve for P from the equation in step 3.4 i+1 and K i+1 ; Step 3.8: Update the migration operator a according to the following formula i+1 : Step 3.9: Let i <- i + 1, check if condition α i+1 > γ is true, if not, then repeat steps 3.7 to 3.8 until the condition is true, otherwise execute step 3.10; Step 3.10: Let Step 3.11: Continue solving P from Step 3.4 i+1 and K i+1 : Step 3.12: Let i <- i + 1, test condition || P i+1 - P i || < ε is true, if not, then steps 3.11 to 3.12 are repeated until this condition is true, otherwise step 3.13 is executed; Step 3.13: Let K x = K i+1 , and l = 1, and the optimal feedback control input is given by Step 3.14: Solve from step 3.4 A (X l ); Step 3.15: Let l←l+1, judge whether the condition l=m is established, if not, repeatedly execute step 3.14 to step 3.15, otherwise execute step 3.16; Step 3.16: Construct output regulator equation: Λζ=Υ Wherein ζ = [α2... α m vec T (X) vec T (U)] T and I qm and I nqm denote the qm-dimensional and nqm-dimensional identity matrix, respectively, and a2,..., a m denote constants, A (X l ) = X l E - AX l , l = 1, 2,... m; Step 3.17: Apply row transformation to the output regulator equations in Step 3.16 and pull vec T (X) and vec T (U) out, resulting in the following equations: wherein Θ 11 ,Θ 12 ,Θ 21 ,Θ 22 , Γ1 and Γ2 are transformed block matrices; Step 3.18: The equation obtained from step 3.17 is solved using the Lagrange multiplier method Wherein λ is a given Lagrange multiplier, M is a given weight matrix; Step 3.19: Calculate optimal feedforward control input Step 3.20: Calculate optimal control input Step 4, based on the optimal control input obtained in step 3, realize the optimal tracking control of the unmanned system.

2. The method of claim 1, wherein, Said step 3.2 in, 3. The method of claim 1, wherein, In the step 3.4, γ=0.1 is selected; If step 3.5 is not satisfied, increase γ, return to step 3.4, until step 3.5 is satisfied, execute step 3.

6.

4. An unmanned system online adaptive learning tracking control device, characterized in that, The optimal tracking control of the unmanned system is carried out by the method according to any one of claims 1-3. The optimal tracking control of the unmanned system is carried out by the method according to any one of claims 1-3.

Images