A multi-UAV formation control method and system based on deep reinforcement learning

Abstract

The present invention discloses a multi-UAV formation control method and system based on deep reinforcement learning, which relates to the field of UAV control and aims to solve the problem that UAVs in existing formations are difficult to ensure high robustness, safety and reliability when dealing with model uncertainty and multi-source interference. The method comprises the following steps: Step 1: Considering the external airflow interference faced by UAVs when flying through narrow channels, an external airflow interference model is constructed; Step 2: A UAV flight dynamics model affected by external airflow interference is constructed; Step 3: Using a PD controller as the base controller, and using a UDE-based controller to estimate and compensate for model uncertainty and multi-source interference; Step 4: Constructing a state space and action space for UAV robust control optimization based on deep reinforcement learning, and constructing a piecewise reward function that focuses on position tracking, attitude stability, control smoothness and safety, so as to enhance the UAV's adaptive ability under complex interference.

Description

Technical Field

[0001] The present invention relates to the field of drone control technology, and in particular to a multi-drone formation optimization control method, system, and storage medium based on deep reinforcement learning. Background Art

[0002] In urban environments or complex terrain in the wild, multi-UAV formations need to have higher safety and stronger robustness when performing tasks of crossing narrow channels (such as gaps between buildings, canyons in mountainous areas). UAV formations are inevitably restricted by air resistance during flight, and this resistance will directly affect the flight trajectory and stability of the UAVs. At the same time, the distance between UAVs is greatly shortened in a narrow space during formation flight, resulting in increased airflow coupling interference between UAVs, posing a threat to the robustness and coordination capabilities of the formation. Currently, there are two main methods for designing robust control of UAVs, one is a non-learning robust control method, and the other is a learning-based control method. Non-learning robust control methods include "Proportional-Integral-Differential" control (PID), H ∞ control, control based on disturbance observer (DO), control based on uncertainty and disturbance estimator (UDE), etc.

[0003] Reinforcement learning is a method for learning control strategies through the interaction between an intelligent agent and its environment. Based on a trial-and-error mechanism, it enables the agent to explore unknown or complex environments and continuously optimize its behavior based on the rewards it receives. The goal of reinforcement learning is to maximize the agent's cumulative rewards over time. Reinforcement learning does not require specific models or labeling information. Because it learns control strategies through real-time interaction with the environment, it possesses strong adaptability and flexibility, giving it significant advantages in dealing with complex, uncertain, and dynamically changing environments.

[0004] Current robust control research primarily addresses model uncertainty and multi-source interference in simple tasks and scenarios. However, with the widespread use of drones, their mission scenarios are becoming increasingly complex, and external interference is multi-source, diverse, and variable. For example, when a multi-drone formation navigates a narrow passage, it faces a complex challenge of robustness, safety, and reliability. This places higher demands on robust control of drones. Ensuring collision avoidance and effective obstacle avoidance have become key research topics. Summary of the Invention

[0005] The technical problems to be solved by the present invention are:

[0006] Existing drone formations have difficulty ensuring high robustness, safety, and reliability when traversing narrow passages and dealing with model uncertainty and multi-source interference.

[0007] The present invention is to solve the above technical problems using the following technical solutions:

[0008] The present invention provides a multi-UAV formation control method based on deep reinforcement learning, comprising the following steps:

[0009] Step 1: Considering the external airflow interference faced by the UAV when flying through a narrow channel, a mathematical model of the external airflow interference is constructed;

[0010] Step 2: Construct a UAV flight dynamics model affected by external airflow interference;

[0011] Step 3: Using the PD controller as the base controller, a UDE-based controller is used to estimate and compensate for model uncertainties and multi-source disturbances.

[0012] Step 4: Construct the state space and action space for robust control optimization of the UAV based on deep reinforcement learning, and construct a piecewise reward function that focuses on position tracking, attitude stabilization, control smoothness, and safety.

[0013] Furthermore, the external airflow interference in step 1 includes: air resistance, ground effect and downdraft.

[0014] Furthermore, the mathematical model of external airflow interference constructed in step 1 is specifically as follows:

[0015] Construct the air resistance generated by the friction between the drone and the air during flight The model is:

[0016] (1)

[0017] in For speed, is the angular velocity of the nth motor, is the coefficient matrix;

[0018] Building the ground effect generated by the motor The model is:

[0019] (2)

[0020] Where h is the flight altitude, is the geo-efficiency coefficient, is the propeller radius;

[0021] The downwash airflow model is constructed as:

[0022] (3)

[0023] in, UAVs are divided into x 、 y 、 z Distance in direction, The drones are x 、 y 、 z Downward thrust coefficient in the direction.

[0024] Furthermore, the construction of the UAV flight dynamics model affected by external airflow interference described in step 2 includes the following processes:

[0025] (4)

[0026] (5)

[0027] Where p= Represents the position of the drone in the inertial coordinate system, Represents the speed of the drone in the inertial coordinate system, Indicates the quality of the drone, are the unit basis vectors of the inertial coordinate system and the body coordinate system, respectively, and , is the rotation matrix from the machine system to the inertial system, is the acceleration due to gravity; = is the attitude angle of the UAV in the body coordinate system, is the angular velocity of the UAV in the body coordinate system, , Indicates the rotational inertia of the drone along the x, y, and z directions of the system. Indicates the torque generated by the propeller on the fuselage axis, including rolling torque , pitching moment and yaw moment , is the propeller lift, is the triaxial component of air resistance in the inertial system; is the external disturbance torque; is the conversion matrix between the body's rotational angular velocity and the Euler angular velocity;

[0028] The attitude angle of the UAV is approximated as , , , expand equation (4) and equation (5) together to obtain:

[0029] (6)

[0030] in, is the attitude angle uncertainty interference, is the expected attitude angle of the UAV, with the superscript des represents expectations;

[0031] The air resistance, ground effect and wind disturbance between drones are regarded as uncertain interference terms of external forces and expressed as:

[0032] (7)

[0033] in, It is the uncertainty interference of position;

[0034] Assume that the control inputs of the position and angle of the UAV in each direction are:

[0035] (8)

[0036] in, It is the position loop control input of the UAV in the x, y, and z directions. It is the attitude loop control input of the UAV. It is the real control input of the drone;

[0037] The linearization of formula (6) at the equilibrium point of the drone's "hovering" state is:

[0038] (9)

[0039] Where, d is the original interference signal, ;

[0040] .

[0041] Furthermore, step 3 includes the following process:

[0042] Taking the PD controller as the basic controller, the controller structure based on UDE is:

[0043] (10)

[0044] in is the overall control input, It is the basic controller. is the amount of interference compensation provided by UDE;

[0045] Define the attitude angle error of the drone for:

[0046] (11)

[0047] (12)

[0048] in is the outer loop control input of the drone, is the desired yaw angle;

[0049] Through PD control, the desired angular velocity of the drone is obtained:

[0050] (13)

[0051] in, is the feedback gain of the proportional term and differential term of the angle loop;

[0052] Define the rotational angular velocity error of the drone for:

[0053] (14)

[0054] Through PD control, the control output of the UAV inner loop basic controller is obtained:

[0055] (15)

[0056] in, is the proportional term and differential term feedback gain of the angular velocity loop;

[0057] Defining the drone's position error for:

[0058] (16)

[0059] in, is the desired position of the drone;

[0060] Define the velocity error of the drone for:

[0061] (17)

[0062] in, is the desired speed of the drone;

[0063] by As the input of the speed controller, the control output of the outer loop basic controller is:

[0064] (18)

[0065] in is the feedback gain of the outer loop proportional term and differential term;

[0066] Use UDE to estimate and compensate for UAV interference;

[0067] Assuming the interference estimation signal With the original interference signal The following relationship exists:

[0068] (19)

[0069] In the formula It is a low-pass filter;

[0070] Substituting formula (10) into formula (9), we get the following identity transformation:

[0071] (20)

[0072] Substituting (19) into (20) we get:

[0073] (twenty one)

[0074] in is the Laplace transform operator, E is the identity matrix, and by rearranging equations (19)-(21), we can obtain:

[0075] (twenty two)

[0076] Consider designing filters for:

[0077] (twenty three)

[0078] in, is the first-order time constant of the filter, is the complex independent variable after Laplace transformation;

[0079] Combining equations (22) and (23), the frequency domain expression of UDE is:

[0080] (twenty four)

[0081] in, is the gain coefficient of the UDE integral link;

[0082] The time domain expression of UDE obtained by Laplace inverse transform is:

[0083] (25).

[0084] Among them, η(t) and η(0) are the posture states at time t and the initial time respectively;

[0085] Furthermore, the step 4 constructs a piecewise reward function that focuses on position tracking, posture stability, control smoothness and safety, specifically:

[0086] (28)

[0087] Among them, W1~W5 are the weights of each segment reward function;

[0088] is the cost item for the UAV to track the desired trajectory, specifically:

[0089] (29)

[0090] in, is the position of the drone at time t, is the expected position of the UAV at time t;

[0091] The cost item for the UAV to maintain speed while tracking the desired trajectory is:

[0092] (30)

[0093] Among them, v t is the speed of the drone at time t, is the expected speed of the UAV at time t;

[0094] This is the cost of stable flight of the UAV, specifically:

[0095] (31)

[0096] in, It's a drone t The posture angle of the moment, It's a drone t The desired attitude angle at the moment;

[0097] The cost item to prevent the drone from hovering for a long time to maintain stable flight is as follows:

[0098] (32)

[0099] in, is the attitude angular velocity of the UAV at time t, is the expected attitude angular velocity of the UAV at time t;

[0100] The cost of ensuring the safety of drone flights and maintaining long-term flight time is as follows:

[0101] (33).

[0102] Further, expect ~ The weights W1~W5 are:

[0103] .

[0104] The present invention provides a multi-UAV formation control system based on deep reinforcement learning. The system has a program module corresponding to the steps of the method described in any of the above technical solutions, and executes the steps in the above-mentioned multi-UAV formation control method based on deep reinforcement learning during operation.

[0105] The present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program is configured to implement the steps of the multi-UAV formation control method based on deep reinforcement learning described in any one of the above technical solutions when called by a processor.

[0106] Compared with the prior art, the present invention has the following beneficial effects:

[0107] 1. Enhanced interference suppression capabilities for multi-UAV systems: This invention effectively estimates and compensates for external airflow disturbances encountered by UAVs traversing narrow corridors, improving flight stability in multi-source interference scenarios. This also further enhances the controller's adaptive capabilities in extreme scenarios with extremely close distances and strong airflow disturbances, mitigating crash risks.

[0108] 2. Improving the robustness and training efficiency of learning control methods: By combining uncertainty system theory with reinforcement learning optimization decision-making methods, a robust and adaptive control framework was constructed. This framework integrates model prior knowledge and data-driven optimization to reduce the exploration space of deep reinforcement learning and improve training efficiency.

[0109] 3. Solve the difficult problem of handling complex multi-source interference in UAVs: A control structure combining a basic controller (PD) and an uncertainty and disturbance estimator (UDE) is proposed. The UDE estimates multi-source interference in real time and generates compensation signals. This solves the problem that traditional control methods have difficulty modeling complex interference or are highly conservative, and also provides a foundation for reinforcement learning input. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 Schematic diagram of the effect of external airflow interference on a drone in an embodiment of the present invention;

[0111] Figure 2 Schematic diagram of the impact of ground effect on a UAV in an embodiment of the present invention;

[0112] Figure 3 Schematic diagram of the effect of downwash on a drone in an embodiment of the present invention;

[0113] Figure 4 This is a force analysis diagram of a UAV in an embodiment of the present invention;

[0114] Figure 5 PD control block diagram in an embodiment of the present invention;

[0115] Figure 6 A control block diagram based on UDE in an embodiment of the present invention;

[0116] Figure 7 Schematic diagram of the combination of reinforcement learning and control input in an embodiment of the present invention;

[0117] Figure 8 This is a block diagram of a drone control system integrating reinforcement learning in an embodiment of the present invention;

[0118] Figure 9 2 is a comparison chart of position error simulation results in an embodiment of the present invention. DETAILED DESCRIPTION

[0119] In order to enable those skilled in the art to better understand the present invention, exemplary embodiments or examples of the present invention will be described below with reference to the accompanying drawings. Obviously, the described embodiments or examples are only some of the embodiments or examples of the present invention, and not all of them. Based on the embodiments or examples of the present invention, all other embodiments or examples obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0120] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0121] Specific implementation plan 1: The present invention provides a multi-UAV formation control method based on deep reinforcement learning, comprising the following steps:

[0122] Step 1: Considering the external airflow interference faced by the UAV when flying through a narrow channel, a mathematical model of the external airflow interference is constructed;

[0123] Step 2: Construct a UAV flight dynamics model affected by external airflow interference;

[0124] Step 3: Using the PD controller as the base controller, a UDE-based controller is used to estimate and compensate for model uncertainties and multi-source disturbances.

[0125] Step 4: Construct the state space and action space for robust control optimization of the UAV based on deep reinforcement learning, and construct a piecewise reward function that focuses on position tracking, attitude stabilization, control smoothness, and safety.

[0126] Specific implementation scheme 2: The external airflow interference effects in step 1 include: air resistance, ground effect and downdraft. This implementation scheme is otherwise the same as the specific implementation scheme 1.

[0127] Specific implementation plan three: Step 1 includes the following steps:

[0128] Step 1: If Figures 1 to 4 As shown in the figure, considering the impact of external airflow interference faced by UAVs when flying through narrow channels on the performance of UAVs, a corresponding mathematical model is constructed to quantify these impacts, focusing on three types of airflow: air resistance, ground effect, and downwash coupling.

[0129] like Figure 1 As shown in the figure, when a drone flies forward, the air moves relative to the drone's wings, fuselage and other surfaces, generating friction. The friction is always opposite to the direction of the drone's movement, hindering the drone's movement. The rotating propellers of a quadcopter generate drag, which is a force opposite to the direction of movement. , the angular velocity of the nth motor and coefficient matrix Construct the resistance generated by the friction between the drone and the air during flight The model is:

[0130] (1)

[0131] In this implementation, the inertial coordinate system is a coordinate system fixed to the Earth's surface, with a point on the Earth's surface chosen as the origin. The body coordinate system is a coordinate system fixed to the drone, with the drone's center of mass chosen as the origin.

[0132] like Figure 2 As shown in Figure 1, when a drone approaches the ground or other boundaries, the ground or boundaries hinder the airflow, causing the drone to be affected by the ground effect, and the lift or drag it experiences increases. The ground effect often causes the drone to fly unsteadily, especially when flying at low altitudes. Considering this effect, the ground effect generated by the rotor airflow is constructed. The model is:

[0133] (2)

[0134] Where h is the flight altitude, is the geo-efficiency coefficient, is the propeller radius;

[0135] like Figure 3 As shown in Figure 1, when two drones intersect at different altitudes, the downdraft will cause the lift of the lower aircraft to decrease, causing the drone to lose stability and even crash to the ground. Assuming that the force generated by the downdraft between the aircraft acts on the center of mass of the lower drone, the force generated is It depends on the distance between the two aircraft in the x, y, and z directions. x , y , z The distance in the direction is ,along x , y , z The downward force coefficients in the direction are , the downwash airflow model is constructed as:

[0136] (3)

[0137] The rest of this implementation plan is the same as the second specific implementation plan.

[0138] Specific implementation plan 4: Step 2 includes the following steps:

[0139] The quadcopter is a nonlinear, multivariable, strongly coupled underactuated control system. The quadcopter is driven by four symmetrically arranged propellers to generate thrust, enabling hovering and autonomous flight. During the flight of the drone, it is subject to various forces, including propeller lift. , k is the propeller number, gravity ( is the weight of the drone, is the combined force of gravity acceleration and resistance When a drone formation flies through a narrow passage, it will also be affected by the ground effect. and inter-machine wind disturbance Unsure about the impact of interference.

[0140] The UAV flight dynamics model affected by external airflow interference is constructed as follows:

[0141] (4)

[0142] (5)

[0143] Where p= Represents the position of the drone in the inertial coordinate system, Represents the speed of the drone in the inertial coordinate system, Indicates the quality of the drone, are the unit basis vectors of the inertial coordinate system and the body coordinate system, respectively, and , is the rotation matrix from the machine system to the inertial system, is the acceleration due to gravity; = is the attitude angle of the UAV in the body coordinate system, is the angular velocity of the UAV in the body coordinate system, , Indicates the rotational inertia of the drone along the x, y, and z directions of the system. Indicates the torque generated by the propeller on the fuselage axis, including rolling torque , pitching moment and yaw moment , is the propeller lift, is the triaxial component of air resistance in the inertial system; is the external disturbance torque; is the conversion matrix between the body's rotational angular velocity and the Euler angular velocity:

[0144]

[0145] The mathematical model of a quadcopter drone exhibits nonlinearity, high coupling of position dynamics and angles, which is not conducive to controller design and theoretical analysis. To simplify the derivation process, it is assumed that the attitude angle of the drone is always kept within a small range during flight, thereby linearizing the motion equation with small deviations. Specifically, the attitude angle (pitch angle) of the drone is , roll angle and yaw angle ) is considered as a smaller amount, that is, , , , expand equation (4) and equation (5) together to obtain:

[0146] (6)

[0147] in, is the attitude angle uncertainty interference, is the expected attitude angle of the UAV, with the superscript des represents expectations;

[0148] The air resistance, ground effect and wind disturbance between drones are regarded as uncertain interference items of external forces and expressed as:

[0149] (7)

[0150] in, It is the uncertainty interference of position;

[0151] To facilitate controller design, the control inputs of the drone’s three positions and three angle directions are assumed to be:

[0152] (8)

[0153] in, It is the position loop control input of the UAV in the x, y, and z directions. It is the attitude loop control input of the UAV. It is the actual control input of the drone.

[0154] The linearization of formula (6) at the equilibrium point of the drone's "hovering" state is:

[0155] (9)

[0156] Where, d is the original interference signal,

[0157]

[0158] The rest of this implementation plan is the same as the specific implementation plan three.

[0159] Specific implementation plan five: Step 3 includes the following steps:

[0160] First, a basic PD controller is designed, and then the UDE method is proposed based on the PD controller. Relying on the UDE method, model uncertainty and multi-source interference are estimated and effectively compensated to improve the robustness of the UAV.

[0161] The controller structure based on UDE is:

[0162] (10)

[0163] in is the overall control input, It is the basic controller. is the amount of interference compensation provided by UDE.

[0164] like Figure 5 As shown in the figure, the basic controller PD design is divided into inner loop attitude control and outer loop position control. Each control loop includes two sub-loops, so the overall control structure is multi-loop control. The outer loop position controller is mainly used to generate the total thrust of the UAV. , desired pitch angle , expected roll angle , while the inner loop attitude controller tracks the expected value given by the outer loop controller and , and produces the desired pitching moment , rolling moment , and the yaw moment .

[0165] Define the attitude angle error of the drone for:

[0166] (11)

[0167] (12)

[0168] in is the outer loop control input of the drone, is the desired yaw angle, given by the upper-level planner.

[0169] Through PD control, the desired angular velocity of the drone is obtained:

[0170] (13)

[0171] in, is the feedback gain of the proportional term and differential term of the angle loop;

[0172] Define the rotational angular velocity error of the drone for:

[0173] (14)

[0174] Through PD control, the control output of the UAV inner loop basic controller is obtained:

[0175] (15)

[0176] in, is the proportional term and differential term feedback gain of the angular velocity loop; the corresponding PD feedback gain;

[0177] Defining the drone's position error for:

[0178] (16)

[0179] in, is the desired position of the drone;

[0180] Define the velocity error of the drone for:

[0181] (17)

[0182] in As the input of the speed controller, the control output of the outer loop basic controller is obtained:

[0183] (18)

[0184] in is the feedback gain of the outer loop proportional term and differential term;

[0185] The UDE-based controller estimates and compensates for interference in the drone. The proposed UDE-based control scheme exhibits two degrees of freedom, meaning the basic controller and interference compensation mechanisms are decoupled from each other, allowing for flexible design adjustments based on actual needs. Furthermore, the proposed UDE-based control method is not only applicable to single-input, single-output (SISO) systems but can also effectively address interference estimation and compensation issues in multi-input, multi-output (MIMO) systems.

[0186] like Figure 6 As shown, the core idea of ​​UDE is to assume that the interference estimation signal With the original interference signal The following relationship exists:

[0187] (19)

[0188] In the formula It is a low-pass filter.

[0189] Substituting formula (10) into formula (9), we get the following identity transformation:

[0190] The following identity transformation is obtained through the relationship between system structure (9) and control law (10):

[0191] (20)

[0192] Substituting (19) into (20) we get:

[0193] (twenty one)

[0194] in is the Laplace transform operator, E is the identity matrix, and rearranging equations (19-21) yields:

[0195] (twenty two)

[0196] Consider designing filters for:

[0197] (twenty three)

[0198] in, is the first-order time constant of the filter, is the complex independent variable after Laplace transformation;

[0199] Combining equations (22) and (23), the frequency domain expression of UDE is:

[0200] (twenty four)

[0201] in, is the gain coefficient of the UDE integral link;

[0202] The time domain expression of UDE obtained by Laplace inverse transform is:

[0203] (25)

[0204] Among them, η(t) and η(0) are the posture states at time t and the initial time respectively;

[0205] The rest of this implementation plan is the same as the specific implementation plan four.

[0206] Specific implementation plan six: Step 4 includes the following steps:

[0207] To address the lack of robustness of the "PD+UDE" controller in extreme scenarios described in Step 3, we conducted research on robust control optimization for UAVs using deep reinforcement learning (DRL). We first designed a state space consisting of the UAV state and UDE estimates, as well as an action space for the UDE parameters or control input residuals. We then constructed a piecewise reward function (with reward objectives focusing on position tracking, attitude stability, control smoothness, and safety).

[0208] The action space of the quadrotor drone corresponds to the lift, roll, pitch and yaw control inputs, such as Figure 7 As shown in , these control inputs directly determine the flight attitude and trajectory of the drone. However, the method using action space as control input is mainly based on a trial-and-error mechanism, and its learning strategy is less reliable. In addition, since a large amount of training data is required to support the trial-and-error process, this increases the cost and time of training and reduces the efficiency in practical applications. To solve this problem, the present invention adopts the idea of ​​residual reinforcement learning, such as Figure 8 As shown, the UDE is used as part of the state as a method to optimize the robust control algorithm in step 3. It is possible to estimate the multi-source interference in the environment and optimize the control performance without changing the original control structure.

[0209] Taking into account the information availability, dimensionality, and task relevance of the reinforcement learning state space, the present invention selects the reinforcement learning state space for UAV robust control optimization as:

[0210] (26)

[0211] in, Represents the position, velocity, attitude angle, and angular acceleration to reflect the status of the drone itself; Represents the expected trajectory of multiple UAVs; Estimation of team interference,From the design and verification of UDE, it can be seen that UDE can estimate external multi-source disturbances,relatively well; represents the state at the previous moment. The state space in Equation (26) has a total of 44 dimensions, which comprehensively reflects the drone's state and environmental information. Position and velocity are obtained using GPS, motion capture systems, and optical flow sensors through direct, differential, and integral methods; attitude angles and angular velocities are obtained using IMU sensors; and the UDE is calculated from measurable states and basic controller outputs. Therefore, all states are accessible.

[0212] In addition, the action space design needs to consider the UAV's maximum speed, acceleration, and angular velocity constraints to ensure that the UAV does not violate physical limitations when performing actions. In addition, it is also necessary to avoid selecting actions that may cause the UAV to lose control or collide to ensure the safety of UAV operations. Therefore, reinforcement learning is added to the basic controller as a supplement to the basic controller to solve the above problems. Set the action space to , enabling it to directly participate in the control process of the UAV, thereby achieving fine-tuning of the UAV's behavior.

[0213] In practical applications, the design of a reward function is closely tied to the actual tasks performed by the multi-UAV system. It needs to directly reflect the relationship between the agent's actions and goals, helping the agent understand which actions are beneficial and which should be avoided. The design of a reward function also requires interpretability to help understand the agent's behavior and adjust it when necessary.

[0214] Reward Function The general form is:

[0215] (27)

[0216] in is the weight, This is a requirement for drones. If you want the drone to have a stable attitude, then Designed for , the more stable the drone's attitude is, The larger the value, the larger the reward function. The segmented design is:

[0217] (28)

[0218] is the cost term for the UAV to track the desired trajectory, The bigger, The closer it is to 0, the better the drone tracks the desired trajectory. The mathematical expression is:

[0219] (29)

[0220] in, is the position of the drone at time t, is the expected position of the UAV at time t.

[0221] While tracking the desired trajectory, the UAV should maintain corresponding speed to prevent the UAV from reducing its flight speed in order to maintain the accuracy of tracking the desired trajectory when flying in a straight line. is the cost of the UAV maintaining its speed while tracking the desired trajectory, The bigger, The closer it is to 0, the better the drone tracks the desired trajectory. The mathematical expression is:

[0222] (30)

[0223] Among them, v t is the speed of the drone at time t, is the expected speed of the UAV at time t;

[0224] is the cost of stable flight of the UAV, The bigger, The closer it is to 0, the closer the drone's attitude is to the "hovering" state, and the more stable the drone's flight is. The mathematical expression is:

[0225] (31)

[0226] in, It's a drone t The posture angle of the moment, It's a drone t The desired attitude angle at the moment;

[0227] To prevent the drone from hovering for a long time to maintain a stable flight, the mathematical expression is:

[0228] (32)

[0229] The cost of ensuring the drone's flight safety and maintaining long-term flight time prevents the drone from taking high-risk actions due to excessive pursuit of short-term rewards. The mathematical expression is:

[0230] (33)

[0231] The rest of this implementation plan is the same as the specific implementation plan five.

[0232] Specific implementation plan seven:

[0233] According to the above analysis, the position and posture This is the main consideration for drones, so the weight of these two should be higher than and The weight of the drone is changed by changing the posture. If the weight is too high, the drone will seek high stability and reduce the accuracy of position tracking. The weight should be higher than the posture Based on the above analysis, the weight design is:

[0234]

[0235] The rest of this implementation plan is the same as the specific implementation plan six.

[0236] The training parameters used in the reinforcement learning in this implementation are shown in Table 1.

[0237] Table 1 Reinforcement learning training parameters

[0238]

[0239] The multi-UAV formation control method (algorithm) based on deep reinforcement learning proposed in this invention is the underlying technical core of this invention, and various products can be derived based on the algorithm.

[0240] Based on the method proposed in the present invention, a multi-UAV formation control system based on deep reinforcement learning is developed using a programming language. The system has program modules corresponding to the steps of the above-mentioned technical solution, and executes the steps in the above-mentioned multi-UAV formation control method based on deep reinforcement learning during operation.

[0241] The developed system (software) is stored on a computer-readable storage medium as a computer program. When invoked by a processor, the computer program is configured to implement the steps of the multi-UAV formation control method based on deep reinforcement learning. This materializes the present invention on a carrier, becoming a computer program product.

[0242] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0243] The computer programs (also referred to as programs, software, software applications, or code) herein comprise machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., a magnetic disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0244] The beneficial effects of the present invention will be described below with reference to specific embodiments.

[0245] Example 1

[0246] This example verifies the performance advantage of the proposed method in wind disturbance resistance through simulation comparison. In order to better simulate the impact of internal and external environmental interference on tracking control, a time-varying interference term is introduced in the simulation:

[0247]

[0248] In the simulation experiment, the motion trajectories of the two UAVs are set as and Whenever a drone crashes during simulation, or the drone maintains safe flight for 1000 time steps, the simulation ends, i.e. done = True. A new round of simulation begins, and the height d of the lower drone changes, causing the distance s between the drones to change. Under the above repeated simulation settings, the reinforcement learning algorithm explores the optimal action to respond to wind disturbances in different directions and heights. Set the random number , each change interval is [0.01, 0.1].

[0249] When a UDE is designed as part of a state space, such as Figure 9 The comparison of position error and control attitude of the three schemes, namely PD control only, PD+UDE control and PD+UDE+deep reinforcement learning control proposed in the present invention, under the same conditions is shown in Table 2; the simulation comparison results of the crash risk of the three control schemes are shown in Table 2.

[0250] Table 2 Crash statistics

[0251]

[0252] from Figure 9 Comparison results show that the control method using UDE as part of the reinforcement learning state exhibits significant performance advantages over the PD controller and PD+UDE controller. Regarding position error, the results of the PD+RL_UDE controller are closer to zero than those of both the PD control and PD+UDE controllers. Table 2 shows that in the simulated flight experiments, the controller using UDE as part of the reinforcement learning state effectively avoided crashes and increased the success rate of multi-UAV formation flight, demonstrating that the introduction of UDE significantly improves the robustness of the controller.

[0253] Although the present invention is disclosed as above, the scope of protection disclosed by the present invention is not limited thereto. Those skilled in the art of the present invention may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A multi-UAV formation control method based on deep reinforcement learning, characterized in that: The steps include: Step 1: Considering the external airflow interference faced by the UAV when flying through a narrow channel, a mathematical model of the external airflow interference is constructed; Step 2: Construct a UAV flight dynamics model affected by external airflow interference; Step 3: Using the PD controller as the base controller, a UDE-based controller is used to estimate and compensate for model uncertainties and multi-source disturbances. Step 4: Construct the state space and action space for robust control optimization of the UAV based on deep reinforcement learning, and construct a piecewise reward function that focuses on position tracking, attitude stability, control smoothness, and safety; Step 3 includes the following process: Taking the PD controller as the basic controller, the controller structure based on UDE is: Where u is the overall control input, u0 is the basic controller, is the amount of interference compensation provided by UDE; Define the attitude angle error e of the drone γ for: e γ =(Θ des -I) (11) I des =[φ des ,the des ,ψ des ] T =[u x ,u y ,ψ des ] T (12) where u x ,u y is the outer loop control input of the UAV, ψ des is the desired yaw angle, Θ = [θ, φ, ψ] T is the attitude angle of the UAV in the body coordinate system, and the superscript des represents expectation; Through PD control, the desired angular velocity of the drone is obtained: Among them, k p，γ , k d，γ is the feedback gain of the proportional term and differential term of the angle loop; Define the rotation angular velocity error e of the drone σ for: e σ =(ω des -ω) (14) Where ω = [p, q, r] T is the angular velocity of the UAV in the body coordinate system; Through PD control, the control output of the UAV inner loop basic controller is obtained: Among them, k p，σ , k d，σ is the proportional term and differential term feedback gain of the angular velocity loop; Define the position error e of the UAV α for: e α =(p des -p) (16) Where p = [x, y, z] T Represents the position of the drone in the inertial coordinate system; Define the velocity error e of the drone β for: e β =(v des -v) (17) Among them, Represents the speed of the drone in the inertial coordinate system; With v des As the input of the speed controller, the control output of the outer loop basic controller is: where k p，β , k d，β is the feedback gain of the outer loop proportional term and differential term; Use UDE to estimate and compensate for UAV interference; Assuming the interference estimation signal There is the following relationship with the original interference signal d: Where G(s) is a low-pass filter; Substituting formula (10) into formula (9), we get the following identity transformation: Where, η = [x, y, z, θ, φ, ψ], g = [0, 0, g, 0, 0, 0] T ; Substituting (19) into (20) we get: in is the Laplace transform operator, E is the identity matrix, and by rearranging equations (19)-(21), we can obtain: Consider designing filter G(s) = [G x (s), G y (s), G z (s), G φ (s), G θ (s), G ψ (s)] is: Among them, T λ is the first-order time constant of the filter, and s is the complex independent variable after Laplace transformation; Combining equations (22) to (23), the frequency domain expression of UDE is: Among them, T is the gain coefficient of the UDE integral link; The time domain expression of UDE obtained by Laplace inverse transform is: Among them, η(t) and η(0) are the posture states at time t and the initial time, respectively.

2. The multi-UAV formation control method based on deep reinforcement learning according to claim 1 is characterized in that: The external airflow interference in step 1 includes: air resistance, ground effect and downdraft.

3. The multi-UAV formation control method based on deep reinforcement learning according to claim 2 is characterized in that: The mathematical model of external airflow interference described in step 1 is constructed as follows: Construct the air resistance F generated by the friction between the drone and the air during flight D The model is: Where v is the velocity, ω n is the angular velocity of the nth motor, C D is the coefficient matrix; Building the ground effect generated by the motor The model is: Where h is the flight altitude, c G is the geopotential efficiency coefficient, r P is the propeller radius; The downwash airflow model is constructed as: Among them, δ x , δ y , δ z are the distances of the drone in the x, y, and z directions, respectively. are the downward force coefficients of the UAV along the x, y, and z directions, respectively.

4. The multi-UAV formation control method based on deep reinforcement learning according to claim 3 is characterized in that: The construction of the UAV flight dynamics model affected by external airflow interference described in step 2 includes the following steps: Where m represents the mass of the drone, {e x , e y , e z } and {b x , b y , b z } are the unit basis vectors of the inertial coordinate system and the body coordinate system respectively, and is the rotation matrix from the machine system to the inertial system, g is the acceleration of gravity; J = diag(I x , I y , I z ), I x , I y , I z It represents the moment of inertia of the drone along the x, y, and z directions of the aircraft system. τ represents the torque generated by the propeller on the aircraft axis, including the rolling torque τ x , pitching moment τ y and the yaw moment τ z , F T is the propeller lift, is the triaxial component of air resistance in the inertial system; d τ is the external disturbance torque; S is the conversion matrix between the body rotation angular velocity and Euler angular velocity; Approximate the attitude angle of the UAV to θ≈0, φ≈0, ψ≈0, and expand Equation (4) and Equation (5) together to obtain: Among them, d φ , d θ , d ψ is the attitude angle uncertainty disturbance, θ des ,φ des , ψ des is the desired attitude angle of the UAV; The air resistance, ground effect and wind disturbance between drones are regarded as uncertain interference terms of external forces and expressed as: Among them, d x , d y , d z It is the uncertainty interference of position; Assume that the control inputs of the position and angle of the UAV in each direction are: Among them, [u x ,u y ,u z ] T is the position loop control input of the UAV in the x, y, and z directions, [u φ ,u θ ,u Ψ ] T is the attitude loop control input of the UAV, [F T , τ φ , τ θ , τ ψ ] T It is the real control input of the drone; The linearization of formula (6) at the equilibrium point of the drone's "hovering" state is: Where d is the original interference signal, u=Bu des ; 5. The multi-UAV formation control method based on deep reinforcement learning according to claim 4 is characterized in that: The step 4 describes the construction of a piecewise reward function that focuses on position tracking, posture stability, control smoothness and safety. Specifically: R t =W1R1+W2R2+W3R3+W4R4+W5R5 (28) Among them, W1~W5 are the weights of each segment reward function; R1 is the cost term for the drone to track the desired trajectory, specifically: Among them, p t is the position of the drone at time t, is the expected position of the UAV at time t; R2 is the cost of the drone maintaining its speed while tracking the desired trajectory, specifically: Among them, v t is the speed of the drone at time t, is the expected speed of the UAV at time t; R3 is the cost item for stable flight of the drone, specifically: R3=-||Θ t -I t des || (31) Among them, Θ t is the attitude angle of the UAV at time t, Θ t des is the expected attitude angle of the UAV at time t; R4 is the cost item to prevent the drone from hovering for a long time to maintain stable flight. Specifically: R4=-||ω t -oh t des || (32) Among them, ω t is the attitude angular velocity of the UAV at time t, is the expected attitude angular velocity of the UAV at time t; R5 is the price item for the drone's flight safety and long-term flight, specifically:

6. The multi-UAV formation control method based on deep reinforcement learning according to claim 5 is characterized in that: The expected weights W1 to W5 of R1 to R5 are:

7. A multi-UAV formation control system based on deep reinforcement learning, characterized in that: The system has a program module corresponding to the steps of the method described in any one of claims 1 to 6 above, and executes the steps in the above-mentioned multi-UAV formation control method based on deep reinforcement learning when running.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is configured to implement the steps in the multi-UAV formation control method based on deep reinforcement learning according to any one of claims 1 to 6 when called by a processor.

Images