Tracking control method and device for online adaptive learning of unmanned system

Abstract

The invention discloses a tracking control method and device for online adaptive learning of an unmanned system. According to the invention, under the condition that a mathematical model of the unmanned system is unknown, only data information of the intelligent unmanned system is used for learning an optimal tracking control input, so that an optimal tracking control task is realized. According to the method, an initial stability control strategy, limiting conditions of data matrix column full rank and continuous excitation and a data storage unit are not needed, the calculation complexity is low, and implementation is easy.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent unmanned systems, and in particular to a tracking control method and device for online adaptive learning of an unmanned system. Background Art

[0002] Unmanned systems are autonomous systems without human intervention. They are the product of a high degree of integration of informatization and mechanization. On the one hand, these systems must be autonomous, and on the other hand, they must achieve "unmanned / unmanned" collaborative operation. Unmanned systems include three parts: unmanned aerial, ground, and marine systems, represented by systems consisting of drones, unmanned vehicles, unmanned ships, robots, unmanned clusters, and unmanned and manned hybrids. At present, there are many types of intelligent unmanned systems that have formed application products, mainly including robot exoskeleton systems, intelligent ground vehicles, underwater vehicles, drones, unmanned cars, service-oriented robots, intelligent industrial robots, space robots, navigation robots, unmanned workshops / intelligent factories, unmanned inspection systems for intelligent mines, ground unmanned systems, intelligent inspection robots for unmanned garrison platforms at sea, intelligent unmanned loading systems, and intelligent unmanned workshops. With the diversification and refinement of the control tasks of unmanned systems, on the one hand, the mechanism modeling method can no longer accurately establish its mathematical model, and on the other hand, the traditional steady-state control method cannot achieve the optimal control task. In order to solve these two problems, the optimal control method based on reinforcement learning came into being. It generates rewards by allowing the controlled system to interact with the external environment, and then selects the next action according to the current reward, and finally learns the optimal action. At present, the optimal control method based on reinforcement learning has been widely studied and applied, but most of the results require the storage of historical system data, the application of continuous incentive conditions, and the known initial stable control strategy. However, the requirement to store historical system data means that additional storage resources are required, the requirement to apply continuous incentive conditions will have a negative impact on the system, and the requirement to know the initial stable control strategy means that the system model needs to be known in advance. Therefore, from the above analysis, it can be seen that these requirements cannot be adapted to actual systems, so how to relax these requirements is a problem that needs further research. Summary of the invention

[0003] In view of this, the present invention provides an unmanned system online adaptive learning tracking control method, which does not require a large amount of additional storage resources to store historical data, nor does it require continuous excitation constraints and initial stabilization strategies. It has low computational complexity and is easy to implement.

[0004] The present invention provides an unmanned system online adaptive learning tracking control method, comprising the following steps:

[0005] Step 1: Establish an intelligent unmanned system model as follows:

[0006]

[0007] in and Represent the states of the follower and leader intelligent unmanned systems respectively, It indicates the control input of the follower intelligent unmanned system to be further designed. and They represent the outputs of the follower and leader intelligent unmanned systems respectively, and A, B, C, E and F represent the unknown coefficient matrices respectively. They represent real vector spaces of n, q, g, p, and r dimensions respectively.

[0008] Step 2: Based on the intelligent unmanned system model established in step 1, the control input u(t) is designed to achieve the optimal control performance by solving a static optimization problem and a dynamic optimization problem. The dynamic optimization problem is defined as

[0009]

[0010]

[0011] In the above formula, Q and R represent the given weight matrices, and

[0012] The optimal feedback control input can be obtained by solving the dynamic optimization problem for

[0013]

[0014] Where K x =R -1 B T P, P is a positive definite matrix and is the following algebraic Riccati equation A T P+PA-PBR -1 B T The only solution where P+Q=0.

[0015] The static optimization problem is defined as

[0016]

[0017] In the above formula M represents the given weight matrix, X and U are the matrices to be solved.

[0018] The optimal feedforward control input can be obtained by solving the static optimization problem for

[0019]

[0020] Where K v =UK x X.

[0021] Combining the optimal feedforward control input and the optimal feedback control input, the actual optimal control input u(t) can be obtained as

[0022]

[0023] Step 3: Based on the actual optimal control input obtained in step 2, further design a reinforcement learning optimal control learning algorithm based on system data to calculate the optimal control input when the system model is unknown.

[0024] The specific steps include the following:

[0025] Step 3.1: Select parameters K0=0,Q≥0,R>0,α0>0,σ∈(0,1),ε>0 and matrix sequence Where K0 and α0 represent the control gain and migration operator at the initial iteration, respectively, and σ and ε are two given constants.

[0026] Step 3.2: Apply arbitrary control input to the system, where is the given exploration noise.

[0027] Step 3.3: Construct the dynamic equation in h l (t) and m l (t) are two variables. Then find the f (t) is greater than 0 at time t, and let T←t.

[0028] Step 3.4: Choose an arbitrary parameter γ>0, set i=0, l=0, and solve P from the following equation i+1 :

[0029]

[0030] in

[0031]

[0032] And vec(·) means converting an m×n dimensional matrix into an mn dimensional column vector, and vech(·) means converting a symmetric square matrix into a column vector after dividing it into blocks by columns and removing the symmetric elements at the same time.

[0033] Step 3.5: Verify whether the condition P1>0 is true. If so, execute step 3.6. Otherwise, continue to execute step 3.4 until the condition P1>0 is true.

[0034] Step 3.6: Let i=1.

[0035] Step 3.7: Solve the equation in step 3.4 for P i+1 and K i+1 .

[0036] Step 3.8: Update the migration operator α according to the following formula i+1 :

[0037]

[0038] Step 3.9: Let i←i+1, and judge the condition α i+1 >γ is true, if not, then repeatedly execute step 3.7 to step 3.8 until the condition is met, otherwise execute step 3.10.

[0039] Step 3.10: Order

[0040] Step 3.11: Continue solving for P from step 3.4 i+1 and K i+1 :

[0041] Step 3.12: Let i←i+1, and judge the condition ||P i+1 -P i ||<ε is true. If not, repeatedly execute steps 3.11 to 3.12 until this condition is met, otherwise execute step 3.13.

[0042] Step 3.13: Let K x =K i+1 , and l = 1, and the optimal feedback control input is

[0043]

[0044] Step 3.14: Solve from step 3.4 A (X l ).

[0045] Step 3.15: Let l←l+1, and determine whether the condition l=m is established. If not, repeatedly execute steps 3.14 to 3.15, otherwise execute step 3.16.

[0046] Step 3.16: Construct the output regulator equation:

[0047] Λζ=Υ

[0048] in

[0049]

[0050] ζ=[α2…α m vec T (X)vec T (U)] T

[0051]

[0052] And I qm and I nqm denote the identity matrices of qm and nqm dimensions respectively, α2,…,α m represents a constant, A (X l )=X l E-AX l ,l=1,2,…m.

[0053] Step 3.17: Apply a row transform to the output regulator equation in step 3.16 and convert vec from ζ T (X) and vec T (U) is separated and the following equation is obtained:

[0054]

[0055] in Θ 11 ,Θ 12 ,Θ 21 ,Θ 22 , Γ1 and Γ2 are the transformed block matrices.

[0056] Step 3.18: Use the Lagrange multiplier method to solve the equation obtained in step 3.17

[0057]

[0058] Where λ is the given Lagrange multiplier and M is the given weight matrix.

[0059] Step 3.19: Calculate the optimal feedforward control input

[0060] Step 3.20: Calculate the optimal control input

[0061] Step 4: The control input calculated in step 3 is applied to the system to achieve the optimal tracking control task of the unmanned system.

[0062] The present invention also provides a tracking control device, which adopts the above method to perform optimal tracking control.

[0063] Beneficial effects:

[0064] The online data-driven optimal tracking controller learning method proposed in the present invention can learn an optimal tracking control input using only the data information of the intelligent unmanned system when the mathematical model of the unmanned system is unknown, thereby achieving the optimal tracking control task. The advantages of the proposed algorithm are that it does not require an initial stable control strategy, a data matrix column full rank and continuous excitation constraints, and a data storage unit, so the computational complexity is low and it is easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 The figure is a flow chart of the method of the present invention.

[0066] Figure 2 This is a diagram showing the optimal tracking control effect of an unmanned system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0068] The present invention provides a tracking control method for unmanned system online adaptive learning, the flow chart is as follows Figure 1 As shown, the specific steps include:

[0069] Step S1: Establish an intelligent unmanned system model

[0070]

[0071] Where x(t) and v(t) represent the states of the follower and leader intelligent unmanned systems respectively, u(t) represents the control input of the follower intelligent unmanned system to be further designed, and y(t) and r(t) represent the outputs of the follower and leader intelligent unmanned systems respectively. C=[1,0], F=[-1,0].

[0072] Step S2: According to the intelligent unmanned system model established in step S1, the following static optimization problem and dynamic optimization problem are designed to design the control input u(t) to achieve the optimal control performance, where the static optimization problem is designed as

[0073]

[0074] stAX+BU=XE

[0075] CX+F=0

[0076] In the above formula represents the given weight matrix, X and U are the matrices to be solved.

[0077] The optimal feedforward control input can be obtained by solving the static optimization problem for

[0078]

[0079] The dynamic optimization problem is defined as

[0080]

[0081] In the above formula R=5 represents the weight matrix, and

[0082]

[0083] The optimal feedback control input can be obtained by solving the dynamic optimization problem for

[0084]

[0085] Combining the optimal feedforward control input and the optimal feedback control input, the actual optimal control input u(t) can be obtained as

[0086]

[0087] Step S3: According to the actual optimal control input u(t) obtained in step S2, a reinforcement learning optimal control algorithm based on system data is designed to calculate the optimal control input without using a system model.

[0088] The parameters selected in step 3.1 are K0=0, α0=0.2, ε=0.001 and matrix sequence

[0089]

[0090] Step 3.2 applies in

[0091] In step 3.3, by judging Z f The determinant of (t) shows that when T = 3s, its determinant is not equal to 0.

[0092] The parameter selected in step 3.4 is γ = 0.1.

[0093] In step 3.5, when γ gradually increases to 6, a matrix P1 is positive definite.

[0094] In step 3.6, let i=1, l=0.

[0095] The solutions P6 and K6 obtained in step 3.7 are

[0096]

[0097] The parameter α6 calculated in step 3.8 is 6.1414.

[0098] In step 3.9, it can be determined that the condition is met when i=6, that is, α6>γ.

[0099] In step 3.10,

[0100] The solution P obtained in step 3.11 is 21 and K 21 They are

[0101]

[0102] In step 3.12, it can be determined that the condition is met when i=21, that is, ||P 21 -P 20 ||≤0.001, where P 20 and K 20 They are

[0103]

[0104] In step 3.13, let K x =K 21 =[0.13577 0.13882], and l = 1, and the optimal feedback control input is

[0105]

[0106] The solution in steps 3.14 and 3.15 is A (X0), A (X1), A (X2) are

[0107]

[0108] The optimal feedforward control input calculated in steps 3.16 to 3.19 is

[0109]

[0110] The optimal control input calculated in step 3.20 is

[0111]

[0112] Step 4: Optimal control input obtained from step S3 Applied to the system to achieve the optimal tracking control task of the unmanned system, the effect diagram is as follows Figure 2 As shown, it can be seen from the figure that the black dotted line (the trajectory of the intelligent unmanned system) can gradually track the red solid line (indicating the trajectory to be tracked).

[0113] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A tracking control method for unmanned system online adaptive learning, characterized in that: include: Step 1: Establish an intelligent unmanned system model: Where x(t) and v(t) represent the states of the follower and leader intelligent unmanned systems, respectively; u(t) represents the control input of the follower intelligent unmanned system to be further designed; y(t) and r(t) represent the outputs of the follower and leader intelligent unmanned systems, respectively; A, B, C, E and F represent the unknown coefficient matrices, respectively; Step 2: Obtain the optimal feedback control input by solving a dynamic optimization problem The optimal feedforward control input is obtained by solving a static optimization problem The actual optimal control input u(t) is in, The dynamic optimization problem is: Among them, Q and R represent the given weight matrices, and Optimal feedback control input for Where K x =R -1 B T P, P is a positive definite matrix and is the following algebraic Riccati equation A T P+PA-PBR -1 B T The only solution of P+Q=0; The static optimization problem is stAX+BU=XE CX+F=0 in, M represents the given weight matrix, X and U are the matrices to be solved; Optimal feedforward control input for Where K v =UK x X; Step 3: The reinforcement learning optimal control learning algorithm based on system data calculates the optimal control input when the system model is unknown; specifically: Step 3.1: Select parameters K0=0,Q≥0,R>0,α0>0,σ∈(0,1),ε>0 and matrix sequence Where K0 and α0 represent the control gain and migration operator at the initial iteration, respectively, and σ and ε are two given constants; Step 3.2: Apply arbitrary control input to the system, where is the given exploration noise; Step 3.3: Construct the dynamic equation in h l (t) and m l (t) are two variables; then find the f (t) is greater than 0 at time t, and let T←t; Step 3.4: Choose an arbitrary parameter γ>0, set i=0, l=0, and solve P from the following equation i+1 : in And vec(·) means converting an m×n dimensional matrix into an mn dimensional column vector, and vech(·) means converting a symmetric square matrix into a column vector after dividing it into columns and removing the symmetric elements at the same time; Step 3.5: Verify whether the condition P1>0 is true. If so, execute step 3.

6. Otherwise, continue to execute step 3.4 until the condition P1>0 is true. Step 3.6: Let i=1; Step 3.7: Solve the equation in step 3.4 for P i+1 and K i+1 ; Step 3.8: Update the migration operator α according to the following formula i+1 : Step 3.9: Let i←i+1, and judge the condition α i+1 >Whether γ is true, if not, repeatedly execute step 3.7 to step 3.8 until the condition is true, otherwise execute step 3.10; Step 3.10: Order Step 3.11: Continue solving for P from step 3.4 i+1 and K i+1 : Step 3.12: Let i←i+1, and judge the condition ||P i+1 -P i ||<ε is true, if not, then repeatedly execute steps 3.11 to 3.12 until this condition is met, otherwise execute step 3.13; Step 3.13: Let K x =K i+1 , and l = 1, and the optimal feedback control input is Step 3.14: Solve from step 3.4 A (X l ); Step 3.15: Let l←l+1, and determine whether the condition l=m is established. If not, repeatedly execute steps 3.14 to 3.15, otherwise execute step 3.16; Step 3.16: Construct the output regulator equation: Λζ=Υ in ζ=[α2 ... α m vec T (X) vec T (U)] T And I qm and I nqm denote the identity matrices of qm and nqm dimensions respectively, α2,…,α m represents a constant, A (X l )=X l E-AX l ,l=1,2,…m; Step 3.17: Apply a row transform to the output regulator equation in step 3.16 and convert vec from ζ T (X) and vec T (U) is separated and the following equation is obtained: in Θ 11 ,Θ 12 ,Θ 21 ,Θ 22 ,Γ1 and Γ2 are the transformed block matrices; Step 3.18: Use the Lagrange multiplier method to solve the equation obtained in step 3.17 Where λ is the given Lagrange multiplier and M is the given weight matrix; Step 3.19: Calculate the optimal feedforward control input Step 3.20: Calculate the optimal control input Step 4: Based on the optimal control input obtained in step 3, optimal tracking control of the unmanned system is achieved.

2. The method according to claim 1, characterized in that In step 3.2, 3. The method according to claim 1, characterized in that In step 3.4, select γ=0.1; if step 3.5 is not satisfied, increase γ and return to step 3.4 until step 3.5 is satisfied, and then execute step 3.

6.

4. A tracking control device for online adaptive learning of an unmanned system, characterized in that: The optimal tracking control of the unmanned system is performed by using the method described in any one of claims 1 to 3.

Images