A multi-unmanned aerial vehicle control method based on reinforcement learning

Abstract

The application discloses a multi-unmanned aerial vehicle control method based on reinforcement learning and belongs to the technical field of unmanned aerial vehicle cooperative control. The application designs a control algorithm for a double-time-scale multi-agent, effectively avoids possible calculation errors in the process of solving a matrix equation by introducing a novel coefficient matrix, and proposes a hybrid iteration algorithm which is independent of initial stable controller gain and has a fast convergence speed and is used for solving an approximate solution of a game algebraic Riccati equation based on a communication topology between agents. The application can realize consistency between unmanned aerial vehicles, enables the unmanned aerial vehicles to work cooperatively, and completes a task.

Description

Technical Field

[0001] This invention relates to the field of collaborative control technology for unmanned aerial vehicles (UAVs), and specifically to a multi-UAV control method based on reinforcement learning. Background Technology

[0002] In collaborative drone operations, distributed control methods are widely used to solve the problem of coordinated control among multiple drones. Distributed control methods can be divided into two categories: those based on integrated intelligence and those based on multi-agent systems. Integrated intelligence-based methods treat all drones as a whole for control, requiring the integration and processing of state information from each drone at a central node before decision-making and control. In contrast, multi-agent-based distributed control methods treat each drone as an individual agent, exchanging information through communication protocols and making decisions and controlling based on the received information. Multi-agent-based distributed control methods exhibit good robustness and scalability, do not require a central node, and can adapt to the needs of large-scale drone systems.

[0003] However, a major challenge in practical applications of multi-agent distributed control methods is achieving consistency among unmanned aerial vehicles (UAVs). Consistency refers to the ability of multiple UAVs to reach a consensus on decisions when performing a task, enabling them to work collaboratively and complete the mission. In real-world applications, achieving complete consistency is difficult due to communication delays and uncertainties among UAVs. Therefore, this paper proposes a multi-UAV control method based on reinforcement learning. Summary of the Invention

[0004] The technical problem to be solved by this invention is: how to achieve consistency among unmanned aerial vehicles (UAVs) so that they can work together and complete tasks. This invention provides a multi-UAV control method based on reinforcement learning, which treats the multi-UAV system as a dual-timescale multi-agent system, combines reinforcement learning and hybrid algorithms, and achieves control by sharing information among the various UAVs.

[0005] The present invention solves the above-mentioned technical problems through the following technical solution, and the present invention includes the following steps:

[0006] S1: Obtain a continuous-time motion control model for a multi-UAV system;

[0007] S2: The optimal consensus problem of a multi-agent system is transformed into solving the game algebra Riccati equation related to its neighboring agents.

[0008] S3: Use an offline hybrid iterative algorithm to iteratively optimize the control strategy and obtain the optimal control strategy;

[0009] S4: Apply the optimal control strategy to a multi-UAV system to achieve consistent scheduling control.

[0010] Furthermore, in step S1, each drone in the multi-drone system is treated as an intelligent agent, and multiple intelligent agents form a multi-agent system.

[0011] Furthermore, in step S1, the motion control model of the multi-UAV system is represented as follows:

[0012]

[0013] in, Represents the system state, where x i1 (t) represents the position variable of the i-th UAV, x i2 (t) represents the velocity variable of the i-th UAV, representing the slow state and the fast state respectively; ∈ represents the singular perturbation parameter; For the input speed command; A 11 A 12 A 21 A 22 B i1 B i2 All are constant matrices of suitable size, defined as follows:

[0014]

[0015]

[0016] Among them, I n Describes an n-order identity matrix. Represents the Kronecker product, l i >0 indicates the control gain.

[0017] Furthermore, in step S2, the norm of the state vector difference between any i-th UAV and j-th UAV eventually converges to 0, that is:

[0018]

[0019] in, This ensures consistency between all drones.

[0020] Furthermore, in step S2, the specific processing procedure is as follows:

[0021] S21: Define the local neighborhood error of each agent as:

[0022]

[0023] The dynamics of the i-th local neighborhood error are constructed according to the following expression:

[0024]

[0025] in, For the local neighborhood control input of agent i, Represents a set of nodes. The set d representing the edges i , is the in-degree matrix The element in, a ij It is a connection matrix For any element in the set, if (j,i)∈ε, then a ij >0, otherwise a ij =0; B j The definition is as follows:

[0026]

[0027] S22: The control strategy and corresponding performance indicators for agent i are designed as follows:

[0028]

[0029]

[0030]

[0031] Among them, Q i ≥0, R ii >0 and R ij >0 are all weight matrices, γ is a positive scalar, and K and L are the gain matrices to be determined;

[0032] S23: The value function in a linear system is expressed in quadratic form, i.e. Where P i >0 is a symmetric matrix. Combining this with step S22, the Hamiltonian function can be expressed as:

[0033]

[0034] S24: Based on and Based on this, the optimal control strategy is calculated. and worst interference They are represented as follows:

[0035]

[0036]

[0037] in, This is the optimal solution to the Riccati equation in game algebra, which is as follows:

[0038]

[0039] The expression is as follows:

[0040]

[0041] in,

[0042] This transforms the optimal consensus problem of multi-agent systems into obtaining it from the Riccati equation of game algebra.

[0043] S25: Due to the direct solution There are calculation errors in the process, therefore the following conversion relationship is given:

[0044]

[0045] in,

[0046] The Riccati equation in game algebra can be rewritten as:

[0047]

[0048] in,

[0049] Furthermore, in step S3, the offline hybrid iterative algorithm is implemented by combining the value iteration algorithm and the policy iteration algorithm.

[0050] Furthermore, in step S3, the specific processing procedure is as follows:

[0051] S31: Let the iteration number k = 0, and the step size be... The iterative decision domain and counter q = 0;

[0052] S32: Update using the following two expressions

[0053]

[0054] Subsequently, in order to obtain Determine what is obtained in the above formula. The range, when Belongs to the iterative decision domain Iterate in time, Belongs to the iterative decision domain Time remains unchanged:

[0055]

[0056] S33: Reuse the two expression pairs from step S32 Update, where k = k + 1, until || Where χ1>0 is a predefined threshold, selected As the optimal controller gain;

[0057] S34: Solve using the following Lyapunov equations

[0058]

[0059] in:

[0060]

[0061]

[0062] S35: Update using the following two expressions and

[0063]

[0064]

[0065] S36: Reuse the two expressions from step S35 and Update, where k = k + 1, when || Where χ²>0 is a predefined threshold, selected... As the optimal controller gain.

[0066] Compared with the prior art, the present invention has the following advantages: This multi-UAV control method based on reinforcement learning designs a control algorithm for multi-agent systems with dual time scales. By introducing a novel coefficient matrix, it effectively avoids possible computational errors in solving matrix equations. It proposes a hybrid iterative algorithm that does not depend on the initial stable controller gain and has a fast convergence speed, used to solve approximate solutions to the game algebra Riccati equation based on the communication topology between agents. It can achieve consistency among UAVs, enabling them to work collaboratively and complete tasks. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating the multi-UAV control method based on reinforcement learning in Embodiment 1 of the present invention.

[0068] Figure 2This is a schematic diagram illustrating the implementation process of the reinforcement learning-based multi-UAV control method in Embodiment 2 of the present invention;

[0069] Figure 3a This is the position change curve of the UAV in Embodiment 2 of the present invention (in the figure, "Agent" refers to a single UAV);

[0070] Figure 3b This is the speed change curve of the UAV in Embodiment 2 of the present invention (in the figure, "Agent" refers to a single UAV);

[0071] Figure 4 This is a 3D schematic diagram of the changes in various state variables of the UAV in Embodiment 2 of the present invention.

[0072] Figure 2 middle:

[0073]

[0074] Detailed Implementation

[0075] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0076] Example 1

[0077] like Figure 1 As shown, this embodiment provides a technical solution: a multi-UAV control method based on reinforcement learning, comprising the following steps:

[0078] S1: Obtain a continuous-time motion control model for a multi-UAV system;

[0079] S2: The optimal consensus problem of a multi-agent system is transformed into solving the game algebra Riccati equation related to its neighboring agents.

[0080] S3: Use an offline hybrid iterative algorithm to iteratively optimize the control strategy and obtain the optimal control strategy;

[0081] S4: Apply the optimal control strategy to a multi-UAV system to achieve consistent scheduling control.

[0082] In step S1, each drone in the multi-drone system is treated as an intelligent agent, and multiple intelligent agents form a multi-agent system.

[0083] In step S1, the motion control model of the multi-UAV system is represented as follows:

[0084]

[0085] in, Represents the system state, where x i1 (t) represents the position variable of the i-th UAV, x i2 (t) represents the velocity variable of the i-th UAV, representing the slow state and the fast state respectively; ∈ represents the singular perturbation parameter; For the input speed command; A 11 A 12 A 21 A 22 B i1 B i2 These are all constant matrices of a given size, defined as follows:

[0086]

[0087]

[0088] Among them, I n Describes an n-order identity matrix. Represents the Kronecker product, l i >0 indicates the control gain.

[0089] In step S2, the norm of the state vector difference between any i-th UAV and j-th UAV eventually converges to 0, that is:

[0090]

[0091] in, This ensures consistency between all drones.

[0092] In step S2, the specific processing procedure is as follows:

[0093] S21: Define the local neighborhood error of each agent as:

[0094]

[0095] The dynamics of the i-th local neighborhood error are constructed as follows:

[0096]

[0097] in, For the local neighborhood control input of agent i, Represents a set of nodes. The set d representing the edges i , is the in-degree matrix The element in, a ij It is a connection matrix For any element in the set, if (j,i)∈ε, then a ij >0, otherwise a ij =0; B j The definition is as follows:

[0098]

[0099] S22: The control strategy and corresponding performance indicators for agent i are designed as follows:

[0100]

[0101]

[0102]

[0103] Among them, Q i ≥0, R ii >0 and R ij >0 are all weight matrices, γ is a positive scalar, and K and L are the gain matrices to be determined;

[0104] S23: The value function in a linear system is expressed in quadratic form, i.e. Where P i >0 is a symmetric matrix. Combining this with step S22, the Hamiltonian function can be expressed as:

[0105]

[0106] S24: Based on and Based on this, the optimal control strategy is calculated. and worst interference They are represented as follows:

[0107]

[0108]

[0109] in, This is the optimal solution to the Riccati equation in game algebra, which is as follows:

[0110]

[0111] The expression is as follows:

[0112]

[0113] in,

[0114] This transforms the optimal consensus problem of multi-agent systems into obtaining it from the Riccati equation of game algebra.

[0115] S25: Because in directly solving P i There are calculation errors in the process, therefore the following conversion relationship is given:

[0116]

[0117] in,

[0118] The Riccati equation in game algebra can be rewritten as:

[0119]

[0120] in,

[0121] In step S3, the offline hybrid iterative algorithm is implemented by combining the value iteration algorithm and the policy iteration algorithm.

[0122] In step S3, the specific processing procedure is as follows:

[0123] S31: Let the iteration number k = 0, and the step size be... The iteration decision domain and counter q = 0;

[0124] S32: Update using the following two expressions

[0125]

[0126] Subsequently, in order to obtain Determine what is obtained in the above formula. The range, when Belongs to the iterative decision domain Iterate in time, As an intermediate variable in the calculation, Belongs to the iterative decision domain Time remains unchanged:

[0127]

[0128] S33: Reuse the two expression pairs from step S32 Update, where k = k + 1, until || Where χ1>0 is a predefined threshold, selected As the optimal controller gain;

[0129] S34: Solve using the following Lyapunov equations

[0130]

[0131] in:

[0132]

[0133]

[0134] S35: Update using the following two expressions and

[0135]

[0136]

[0137] S36: Reuse the two expressions from step S35 and Update, where k = k + 1, when || Where χ²>0 is a predefined threshold, selected... As the optimal controller gain.

[0138] Example 2

[0139] like Figure 2 As shown, this embodiment provides a more detailed explanation of the steps in Embodiment 1. To improve the efficiency of combat missions, it is necessary for the drone swarm to fly in a specific formation. This can increase the exploration range during coordinated reconnaissance, search, and patrol operations, and also allow for circling enemy targets through time-varying formations during standoff tracking and positioning. Considering that current autopilots can control drones by controlling the speed of quadcopter drones, the internal dynamics of the drones are ignored in the design process, and only the external control of the formation control layer is considered. Treating each drone as a point mass allows for an approximate description of its motion characteristics.

[0140] Therefore, the i-th multi-agent system for motion control of the drone formation can be represented as follows:

[0141]

[0142] in, Represents the system state, where x i1 (t) represents the position variable of the i-th UAV, x i2 (t) represents the velocity variable of the i-th UAV, representing the slow state and the fast state respectively; ∈ represents the singular perturbation parameter; For the input speed command; A 11 A 12 A21 A 22 B i1 B i2 All are constant matrices of appropriate size;

[0143] The definition is as follows:

[0144]

[0145]

[0146] Among them, I n Describes an n-order identity matrix. Represents the Kronecker product, l i >0 indicates the control gain.

[0147] The main objective of this invention is to design a set of distributed optimal controllers that enable all agents to eventually reach the same state trajectory. This means that the norm of the state vector difference between any i-th and j-th drones eventually converges to 0.

[0148]

[0149] in,

[0150] This ensures consistency between all drones.

[0151] Now, for the purpose of subsequent analysis, the local neighborhood error of each agent is defined as:

[0152]

[0153] Therefore, the dynamics of the i-th local neighborhood error can be constructed as follows:

[0154]

[0155] in, For the local neighborhood control input of agent i, Represents a set of nodes. The set d representing the edges i It is the in-degree matrix The element in, a ij It is a connection matrix For any element in the set, if (j,i)∈ε, then a ij >0, otherwise a ij =0; B j The definition is as follows:

[0156]

[0157] Each agent is only allowed to use its own information and the input information of neighboring agents to design the controller. At the same time, each agent must minimize its performance metrics and overcome interference from neighboring agents. Therefore, the control strategy and corresponding performance metrics of agent i are as follows:

[0158]

[0159]

[0160]

[0161] Among them, Q i ≥0, R ii >0 and R ij All values ​​>0 represent weight matrices, γ is a positive scalar, and K and L are the gain matrices to be determined.

[0162] The value function in a linear system can be represented in quadratic form, i.e. Where P i >0 is a symmetric matrix. Combining with (3), the Hamiltonian function can be expressed as:

[0163]

[0164] Based on and Based on this, the optimal control strategy can be calculated. and worst interference They are represented by formulas (4) and (5), respectively.

[0165]

[0166]

[0167] in, It is the optimal solution to the following game algebra Riccati equation.

[0168]

[0169] Meanwhile, it can be learned from page 555 of the document "A revised Kleinman algorithm to solve algebraic Riccati equation of singularly perturbed systems," that P i,f The following expression exists:

[0170]

[0171] in,

[0172] Based on the above analysis, the core problem of this invention is transformed into obtaining from formula (6) However, due to Represented as formula (7), in direct solution There are calculation errors in the process. Therefore, another major objective of this invention is to eliminate calculation errors caused by singular perturbation parameters when designing the controller.

[0173] For subsequent analysis, the following transformations are given:

[0174]

[0175] in:

[0176]

[0177] Now, equation (6) can be rewritten as:

[0178]

[0179] in,

[0180] Since (8) is nonlinear, this means that P can be obtained directly. i It's very difficult.

[0181] To overcome this problem, an algorithm will be proposed in the next step to approximate the solution of equation (8). A model-based offline hybrid iterative algorithm will then be provided, combining the advantages of policy iteration and value iteration algorithms to obtain a distributed optimal controller.

[0182] Specifically, let the number of iterations k = 0, and the step size be... The iterative decision domain and counter q = 0;

[0183] Then, update using equations (9) and (10).

[0184]

[0185] Subsequently, in order to obtain Determine what was obtained in (9) The range, when Belongs to the iterative decision domain Iterate in time, Belongs to the iterative decision domain Time remains unchanged:

[0186]

[0187] Repeat equations (9) and (10), where k = k + 1, until... Where χ1>0 is a predefined small threshold of 10. -3 .choose As the optimal controller gain

[0188] Although the value iteration algorithm does not require an initial stable control gain, it requires multiple iterations to converge to the optimal value. Now, a policy iteration method with a fast convergence rate is provided to solve (8).

[0189] Solve using the following Lyapunov equations.

[0190]

[0191] in:

[0192]

[0193]

[0194] Then, update in the following way and

[0195]

[0196]

[0197] when Where χ²>0 is a predefined small threshold of 10. -3 Repeat (11), (12) and (13), and let k = k + 1, then choose As the optimal controller gain.

[0198] The strategy iterative algorithm obtains the solution by iteratively solving the Lyapunov equation (11). Equation (8) can be approximately solved. However, the algorithm requires a stable initial gain matrix as a prerequisite to function correctly. To mitigate this condition, a model-based hybrid iterative method is adopted, combining the advantages of policy iteration and value iteration algorithms to solve the problem in (8). Figure 2 The diagram shows the implementation flow of the algorithm of this invention.

[0199] In this algorithm, the initial stable control strategy is no longer the starting condition, because in multi-UAV systems, the initial conditions are typically uncertain and change over time. The algorithm consists of two phases. In the first phase, the multi-UAV system acquires a stable control strategy through offline learning, enabling the UAVs to work collaboratively and achieve consistent control. In the second phase, the algorithm iteratively optimizes to gradually approach the optimal control strategy, achieving a higher level of control performance and further improving the collaborative efficiency of the multi-UAV system. The advantage of this algorithm lies in its ability to quickly converge to the optimal solution and its capacity to handle the uncertainties and variability in the control of multi-UAV systems.

[0200] like Figure 3a , Figure 3b As shown, the controller is added at 2 seconds to quickly keep the speed and position of each UAV consistent. This demonstrates that the multi-agent hybrid iterative control algorithm based on reinforcement learning in this invention can effectively solve the consistency problem in the control of multi-UAV systems, enabling each UAV to work collaboratively and complete the task.

[0201] In summary, the reinforcement learning-based multi-UAV control method described in the above embodiments designs a control algorithm for multi-agent systems with dual time scales. By introducing a novel coefficient matrix, it effectively avoids computational errors that may exist in the process of solving matrix equations. It proposes a hybrid iterative algorithm that does not depend on the initial stable controller gain and has a fast convergence speed to solve approximate solutions to the game algebra Riccati equation based on the communication topology between agents. It can achieve consistency among UAVs, enabling them to work collaboratively and complete tasks.

[0202] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A multi-UAV control method based on reinforcement learning, characterized in that, Includes the following steps: S1: Obtain a continuous-time motion control model for a multi-UAV system; S2: The optimal consensus problem of a multi-agent system is transformed into solving the game algebra Riccati equation related to its neighboring agents. S3: Use an offline hybrid iterative algorithm to iteratively optimize the control strategy and obtain the optimal control strategy; S4: Apply the optimal control strategy to a multi-UAV system to achieve consistent scheduling control; In step S2, the specific processing procedure is as follows: S21: Define the local neighborhood error of each agent as: ； in, Indicates the system status. For the first The position variables of the drone For the first The speed variable of the drone represents the slow state and the fast state, respectively; No. The dynamics of a local neighborhood error are constructed using the following expression: ； in, For intelligent agents Local neighborhood control input, For singular perturbation parameters, , is the input speed command. Represents a set of nodes. Set representing edges , is the in-degree matrix The elements in It is a connection matrix The elements in, if ,So ,otherwise ; The definition is as follows: ； S22: Designing Intelligent Agents The control strategy and corresponding performance indicators are as follows: ； ； in, ≥0, >0 and All values ​​greater than 0 are weight matrices. It is a positive scalar. and The gain matrix to be determined; S23: The value function in a linear system is expressed in quadratic form, i.e. ,in >0 is a symmetric matrix. Combining this with step S22, the Hamiltonian function can be expressed as: ； S24: Based on and Based on this, the optimal control strategy is calculated. and worst interference They are represented as follows: ； ； in, This is the optimal solution to the Riccati equation in game algebra, which is as follows: ； The expression is as follows: ； in, ; This transforms the optimal consensus problem of multi-agent systems into obtaining it from the Riccati equation of game algebra. ; S25: Due to the direct solution There are calculation errors in the process, therefore the following conversion relationship is given: ； in, ; The Riccati equation in game algebra can be rewritten as: ； in, ; All are constant matrices of suitable size, defined as follows: ； ； in, Describes an n-order identity matrix. Indicates the Kronecker product. To control the gain.

2. The multi-UAV control method based on reinforcement learning according to claim 1, characterized in that: In step S1, each drone in the multi-drone system is treated as an intelligent agent, and multiple intelligent agents form a multi-agent system.

3. The multi-UAV control method based on reinforcement learning according to claim 2, characterized in that: In step S1, the motion control model of the multi-UAV system is represented as follows: 。 4. The multi-UAV control method based on reinforcement learning according to claim 3, characterized in that: In step S2, for any first... drones and the first The norm of the state vector difference between the drones eventually converges to 0, that is: ； in, This ensures consistency between all drones.

5. The multi-UAV control method based on reinforcement learning according to claim 4, characterized in that: In step S3, the offline hybrid iterative algorithm is implemented by combining the value iteration algorithm and the policy iteration algorithm.

6. The multi-UAV control method based on reinforcement learning according to claim 5, characterized in that: In step S3, the specific processing procedure is as follows: S31: Let the iteration number k=0, and the step size be... , For iterative determination domain and counter ; S32: Update using the following two expressions : ； Subsequently, in order to obtain Determine the result obtained in the above formula. The range, when Belongs to the iterative decision domain Iterate in time, Belongs to the iterative decision domain Time remains unchanged: ； S33: Reuse the two expression pairs from step S32 Update, among which ,until ,in >0 is a predefined threshold; select As the optimal controller gain; S34: Solve using the following Lyapunov equations : ； in: ； ； S35: Update using the following two expressions and : ； ； S36: Reuse the two expressions from step S35 and Update, among which ,when ,in >0 is a predefined threshold; select As the optimal controller gain.

Images