Unmanned aerial vehicle formation keeping control method based on deep reinforcement learning

Abstract

This invention discloses a deep reinforcement learning-based method for UAV formation-keeping control, belonging to the interdisciplinary field of intelligent control and UAV technology. The method includes: establishing a flight dynamics model and kinematic model for the UAVs, and designing a relative motion model for the UAVs based on a virtual lead aircraft topology; designing a stabilization-attitude-track PID cascade controller for the UAVs; designing a Markov decision process for each UAV's MAPPO agent, including state space, action space, reward function, and termination condition; designing a neural network structure suitable for conforming to the Markov decision model; training the designed MAPPO agents, where the input to the agent is the state space, and the output is the UAV control command. The PID cascade controller receives the control command and then controls the UAVs to maintain formation. This invention improves the intelligence, robustness, and accuracy of UAV formation-keeping control.

Description

Technical Field

[0001] This invention relates to the intersection of intelligent control and unmanned aerial vehicle (UAV) technology, and in particular to a UAV formation-keeping control method based on deep reinforcement learning. Background Technology

[0002] Traditional methods for generating flight path commands for UAV formations, such as the PID (Proportional-Integral-Derivative) algorithm, often require manual parameter tuning and exhibit poor robustness. Their control performance tends to degrade sharply or even fail when faced with complex environments or severe disturbances. Model predictive control and other methods generally require accurate models of the controlled object and environmental factors such as disturbances to design control and guidance laws. However, these models are typically nonlinear and extremely complex, making it difficult to establish an accurate control model, and ensuring model robustness is also challenging. Against this backdrop, deep reinforcement learning methods, which excel at handling complex stochasticities, have attracted considerable attention.

[0003] Deep reinforcement learning is a crucial component of machine learning and a powerful tool for handling sequential decision-making problems. It is currently widely applied in robotics, gaming, finance, transportation, and other fields. Training with deep reinforcement learning often requires establishing its Markov decision process. The goal is to train an agent to obtain an optimal policy guided by the accumulated reward function value during interaction with the training environment. This policy enables the agent to rationally select actions based on changes in the state space. When using this algorithm for training, strong randomness can be actively designed into the environment model, enabling the solution of decision-making and control problems in complex stochastic environments. It is also suitable for solving the problem of generating flight path commands for UAV formations. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a UAV formation-keeping control method based on deep reinforcement learning, so as to improve the intelligence, robustness and accuracy of UAV formation-keeping control.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] A method for drone formation keeping control based on deep reinforcement learning, comprising:

[0007] Step 1: Based on the principles of flight mechanics, establish the flight dynamics model and kinematic model of the UAV, and design the relative motion model of the UAV based on the virtual lead aircraft topology;

[0008] Step 2: Based on the PID control principle, design a PID cascade controller for stabilization, attitude, and trajectory of the UAV;

[0009] Step 3: Design the Markov decision process for the MAPPO agent of each UAV, including the state space, action space, reward function, and termination condition;

[0010] Step 4: Design a neural network structure suitable for this Markov decision model;

[0011] Step 5: Train the designed MAPPO agent. The input of the agent is the state space, and the output is the UAV control command. The PID cascade controller receives the control command and then controls the UAV to complete the formation maintenance.

[0012] The present invention has the following beneficial effects:

[0013] This invention presents a deep reinforcement learning-based UAV formation-keeping control method that combines an independent learning paradigm with a near-end policy optimization algorithm to address multi-UAV formation (multi-agent) problems. First, kinematic and dynamic equations for individual UAVs are established based on flight mechanics principles, and a relative motion model for the UAV formation is built based on a virtual leader structure. Next, PID cascade controllers for each UAV are designed to enable them to accurately and quickly track commands. Then, a Markov decision model, neural network structure, and algorithm flow for the formation-keeping process are designed based on the MAPPO algorithm, allowing multiple UAVs to maintain formation in complex environments such as wind disturbances. Furthermore, by incorporating key UAV state variables such as pitch angular velocity into the state space, the control accuracy of this method is significantly improved. This invention utilizes deep reinforcement learning algorithms to establish a mapping relationship between complex environments and UAV commands, enhancing the intelligence, robustness, and accuracy of UAV formation-keeping control. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the structure of an unmanned aerial vehicle (UAV) formation system applying the method of the present invention;

[0015] Figure 2 This is a schematic diagram of the formation coordinate system and the relative positions of the UAVs in this invention;

[0016] Figure 3 This is a schematic diagram of the structure of the UAV flight control, i.e., the PID cascade controller, in this invention;

[0017] Figure 4 This is a schematic diagram of the neural network structure in this invention;

[0018] Figure 5 This is a block diagram of the formation system training in this invention;

[0019] Figure 6 This is a reward curve diagram of the training process in this invention;

[0020] Figure 7This is an example formation diagram from the present invention;

[0021] Figure 8 This is a flight path diagram showing the formation maintenance in this invention;

[0022] Figure 9 This is a graph showing the formation maintenance error in this invention.

[0023] Figure 10 The diagram shows the V, γ, and χ responses of the UAV maintaining formation in this invention. Detailed Implementation

[0024] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0025] This invention is based on deep reinforcement learning algorithms. It utilizes the repeated training mechanism of reinforcement learning technology to improve the experience of the agent and uses deep neural networks to fit the relationship between the environmental state and the command output, thereby enabling the agent to guide UAVs to maintain formation flight.

[0026] This invention provides a method for unmanned aerial vehicle (UAV) formation-keeping control based on deep reinforcement learning, such as... Figure 1-10 As shown, it includes:

[0027] Step 1: Based on the principles of flight mechanics, establish the flight dynamics model and kinematic model of the UAV, and design the relative motion model of the UAV based on the virtual lead aircraft topology;

[0028] In this step, the dynamic equation of the UAV's center of mass can be:

[0029]

[0030] And the rotational dynamics equation can be:

[0031]

[0032] In the formula, m is the mass of the UAV, g is the local gravitational acceleration, Ix, Iy, Iz, Izx are the moment of inertia and product of inertia of the UAV; (u, v, w) is the projection of the UAV velocity V onto the body axis, (p, q, r) is the projection of the UAV angular velocity ω onto the body axis; θ and φ are the pitch angle and roll angle of the UAV, respectively. (Tx, Ty, Tz) is the thrust of the UAV engine, (X, Y, Z) is the aerodynamic force, and (L, M, N) is the aerodynamic torque.

[0033] In this step, the velocity of the UAV's center of mass is projected onto the ground coordinate system to obtain the UAV's kinematic equations of mass:

[0034]

[0035] And the rotational kinematic equations of the UAV about its center of mass:

[0036]

[0037] In the formula, ψ is the yaw angle.

[0038] To facilitate spatial location determination, as an optional embodiment, step 1 may include:

[0039] A formation coordinate system is designed to describe the position of the UAV relative to the virtual lead aircraft. The virtual lead aircraft is used as the origin to establish the formation coordinate system O. f x f y f z f O f x f The positive direction of the axis represents the velocity of the virtual salvo in the horizontal plane O. g x g y g The projection direction within, O f z f The axis is perpendicular to the horizontal plane and downwards, O f y f Located in the horizontal plane, it is determined by the right-hand rule, such as Figure 2 As shown;

[0040] Figure 2 In the coordinate system O g x g y g z g Using a ground coordinate system, the velocity V and heading (χ, γ) of the UAV are defined, where γ is the distance between the aircraft's ground velocity vector V and the horizontal plane O. g x g y g The angle between them, χ is the projection of the aircraft's ground velocity vector V onto the horizontal plane and O. g x g The included angle of the axis; O i Let O be the actual position of the UAV i (i = 1, 2, ..., m). di Let i be the desired formation position of UAV i; the speed and heading of each UAV are (V i , χ i γ i ), (x if y if , z if Let be the relative position coordinates of UAV i in the formation coordinate system; the ideal formation is represented as {(x... dif y dif , z dif ), i = 1, 2, ..., m}, where m is the total number of drones in the formation.

[0041] Taking a three-aircraft formation as an example, the following matrix F can be used. j This represents the formation, where j represents the task type.

[0042]

[0043] Step 2: Based on the PID control principle, design a PID cascade controller for stabilization, attitude, and trajectory of the UAV;

[0044] In this step, based on the PID control method, a stabilization, attitude and trajectory tracking controller for the UAV was designed, enabling the UAV to effectively track guidance commands.

[0045] PID cascade controllers can employ various structures readily conceived by those skilled in the art. For improved control accuracy, preferred structures include... Figure 3 As shown, the PID cascade controller may include a trajectory PID controller, an attitude angle PID controller, control surfaces, and a linearized motion model of the UAV (small) disturbance, connected in sequence, wherein:

[0046] The input terminal of the trajectory PID controller is used to receive guidance commands, and the output terminal is used to output attitude angle commands to the attitude angle PID controller.

[0047] The output of the attitude angle PID controller is used to output control surface commands to the control surfaces;

[0048] The UAV disturbance linear motion model outputs velocity and position signals, which are fed back to the input of the trajectory PID controller. The UAV disturbance linear motion model also outputs attitude angle signals, which are fed back to the input of the attitude angle PID controller. The UAV disturbance linear motion model also outputs attitude angle angular velocity signals, which are fed back to the input of the control surfaces via a stabilizer.

[0049] Step 3: Design the Markov decision process for the MAPPO agent of each UAV, including the state space, action space, reward function, and termination condition;

[0050] In this step, a Markov decision process for the agent is established, including the state space, action space, reward function, and termination condition, allowing the current environment to be validated using other algorithms. The MAPPO (Multi-agent Proximal Policy Optimization) agent can specifically be an IMAPPO (Independence Multi-agent Proximal Policy Optimization) agent. The state space S can contain the three-axis deviations (x, y, x) of the UAV from its ideal formation position. eif y eif , z eif ) and its differential and integral, basic state quantities of UAVs Specifically, the expression for the state space S can be:

[0051]

[0052] In the formula, (x eif ,y eif ,z eif Let be the error between the drone i (i = 1, 2, 3...) and the desired position; For x eif The differential term, ∫x eif For x eif The integral term, For y eif The differential term, ∫y eif For y eif The integral term, For z eif The differential term, ∫z eif For z eif The integral term, where Vi is the velocity of drone i, and θ i Let α be the pitch angle of UAV i. i Let q be the angle of attack of UAV i. i Let β be the pitch angular velocity of UAV i. i Let r be the sideslip angle of the drone i. i Let φ be the yaw rate of UAV i. i Let be the roll angle of drone i.

[0053] All of the above variables can be normalized before being input into the agent's observations. The purpose of this is to prevent any observation from being too large and affecting the efficiency of the agent's gradient descent process.

[0054] In this step, the action space can include discrete speed, track yaw, and track tilt commands, with the following expressions:

[0055] ΔV=[-ΔV min ,0,ΔV max m / s

[0056] Δχ=[-Δχ max ,0,Δχ max ]°

[0057] Δγ=[-Δγ max ,0,Δγ max ]°.

[0058] It should be noted that the upper and lower limits of the ΔV command can differ, primarily depending on the thrust-drag characteristics of the UAV, i.e., its acceleration and deceleration performance. Furthermore, the three types of commands in the action library only include the maximum positive value, the minimum negative value, and 0. This is because various complex commands of the UAV can be composed of combinations of these three basic commands, and fewer actions are more conducive to the agent's rapid learning. In fact, extensive testing has shown that more actions do not significantly improve control accuracy; on the contrary, they reduce training speed and efficiency.

[0059] In this step, the expression for the reward function can be:

[0060]

[0061] In the formula, R f The total reward function during formation maintenance; r i (i = 1, 2, ..., 5) represent different reward functions; k i (i = 1, 2, 3) is the reward coefficient, which is set to a negative value, specifically a small negative value.

[0062] In this step, the termination condition for the single-round training of the i-th agent can be:

[0063]

[0064] In the formula, [V min V max ] represents the set speed range for the drone; d set This represents the upper limit of the positional error between the drone and the ideal formation.

[0065] Step 4: Design a neural network structure suitable for this Markov decision model;

[0066] In this step, the preferred neural network structure is the Actor-Critic framework. As shown in Table 1-2, the Actor network consists of three fully-connected (FC) layers with 128, 128, and 27 hidden nodes, respectively. The first two layers use the ReLU activation function, and the last layer uses the Softmax activation function. The final output is the probability distribution of each action in the action library under the current policy. The Critic network consists of four fully connected layers. The first three layers have 128, 128, and 128 hidden nodes, respectively, and all use the ReLU activation function. The last layer has 1 node, and the output value is a function of the current state.

[0067] Table 1 Actor Network Structure

[0068]

[0069] Table 2 Critic Network Structure

[0070]

[0071] The specific structure of the Actor-Critic framework is as follows: Figure 4 As shown.

[0072] Step 5: Train the designed MAPPO agent. The input of the agent is the state space, and the output is the UAV (wingman) control command. The PID cascade controller receives the control command and then controls the UAV to complete the formation maintenance.

[0073] In this step, a formation command generator based on the MAPPO algorithm is trained, and speed commands, track deviation commands, and track tilt commands are output simultaneously. This enables the UAVs (wingmen) to follow the virtual lead aircraft and maintain a preset formation under the guidance of the intelligent agent, thereby achieving intelligent formation of the UAV swarm.

[0074] As an optional embodiment, step 5 may include:

[0075] like Figure 5 As shown, the UAV swarm forms a joint state space S based on its three-axis errors relative to the ideal formation position and its own state and position information. N (s1,s2,…,s N The input is fed into the MAPPO agent, which selects the control commands (A) to be input to the UAV (wingman) based on the outputs of each Actor network. N (a1,a2,…,a N In the policy training process of the agent's actioner, the policy gradient method is adopted, and a pruning-based objective function is introduced to ensure the fast convergence of the policy.

[0076] As another optional embodiment, step 5 may further include:

[0077] Control commands A for the drone (wingman) N (a1,a2,…,a N The PID cascade controllers of each UAV are input to generate control surface and throttle commands, which are then input into the UAV motion model to obtain the state of the UAV swarm at the next moment; the reward function value array R N (r1,r2,…,r N ) and the next state of the system S′ N =(s′1,s′2,...,s′) N The tuple data (S) obtained during the interaction process can also be obtained accordingly. N A N ,R N ,S′ N All data are saved to the experience pool of each agent. At each time step, random samples are taken from the experience pool to update the network parameters in MAPPO in batches. When the time step of each round reaches the preset value or the training forced termination condition is triggered, the training of that round ends and the training process continues to the next round until the number of training rounds reaches the preset value or the reward function condition is met.

[0078] In this step, an offline training method is used to obtain a neural network that can be used for the formation maintenance process of UAVs. This neural network can calculate the aircraft control commands for maintaining formation by using flight data of the local aircraft and nearby aircraft detected in the airspace.

[0079] In summary, the UAV formation-keeping control method based on deep reinforcement learning of this invention combines the independent learning paradigm with the near-end policy optimization algorithm to handle multi-UAV formation (multi-agent) problems. First, the kinematic and dynamic equations of individual UAVs are established based on flight mechanics principles, and a relative motion model of the UAV formation is established based on a virtual leader structure. Next, PID cascade controllers for each UAV are designed to enable them to accurately and quickly track commands. Then, a Markov decision model, neural network structure, and algorithm flow for the formation-keeping process are designed based on the MAPPO algorithm, enabling multiple UAVs to maintain formation in complex environments such as wind disturbances. Simultaneously, key UAV state variables such as pitch angular velocity are added to the state space, significantly improving the control accuracy of this method. This invention utilizes deep reinforcement learning algorithms to establish a mapping relationship between complex environments and UAV commands, improving the intelligence, robustness, and accuracy of UAV formation-keeping control.

[0080] This invention designs a UAV formation-maintaining trajectory guidance law based on a multi-agent near-end policy optimization algorithm and a virtual navigation method, thereby achieving precise and intelligent control of the UAV formation geometry. It has advantages such as strong robustness and controller parameters obtained through training rather than manual tuning.

[0081] Furthermore, to verify the method of the present invention, step 5 may be followed by:

[0082] Step 6: Based on the digital virtual flight simulation calculation method, establish a UAV formation maintenance scenario, use the offline established neural network to generate control commands for multiple aircraft online, and conduct simulation verification of the coordination control technology for UAV formation maintenance.

[0083] a) Parameter settings

[0084] A MAPPO network was constructed in the Matlab 2021 / Simulink environment. Iterative optimization of its neural network parameters was achieved using the Adam (adaptive moment estimation) algorithm, and the advantage value was estimated using the GAE (Generalized Advantage Estimation) method. MiniBatchsize(M) was set to 64, ExperienceHorizon(EH) to 200, ClipFactor(Cf) to 0.2, EntropyLossWeight(ELW) to 0.01, NumEpoch(K) to 3, and GAEFactor(GAE) to 3. f The learning rate α of the Actor and Critic networks is set to 0.95, the discount factor γ is set to 0.99, and the learning rate α of the Actor and Critic networks is set to 0.95. A α C The values ​​are set to 0.0001 and 0.001 respectively. The number of agents N = 3, and the training runs for 3000 rounds, which is the maximum number of training rounds N. max =3000, the simulation time T per round is 50s, the simulation step size Ts is 0.1s, and the maximum time step N per round is... s =500. See the table below for empirical values ​​of the parameters required during training.

[0085] Table 3 Training Parameters for Formation Maintenance

[0086]

[0087]

[0088] b) Training process

[0089] Depend on Figure 6It can be seen that the reward function curves of the three drones show a consistent trend. In the initial stage, the agents of the three drones repeatedly tried and failed, and the curves oscillated around 0. After about 800 rounds of training, the network parameters of the agents began to update in a positive direction, and the reward function curves of the three drones rose rapidly. When the training reached 1500 rounds, the reward values ​​obtained by the three drones tended to stabilize, eventually converging at a reward value of around 1450, with a convergence mean of 1452. This means that the MAPPO algorithm can effectively and quickly learn the policy, causing the neural network parameters to gradually converge to near the optimal value, thus effectively realizing the training process of multiple agents for this task.

[0090] c) Numerical simulation experiment

[0091] After completing the training process in step b), a numerical simulation experiment was conducted on the agent's instruction generation strategy to verify its effectiveness. In the simulation environment, the drone formation was set up with one virtual lead drone and three unmanned wingmen (UAVs). Every 0.1 seconds, the MAPPO agent outputs the speed command, trajectory deviation command, and trajectory tilt command with the highest probability. The specific demonstration and verification task requirements were: the three UAVs flew at an altitude of 1500 meters and a speed of 125 km / h, initially setting the formation to a straight line with 100-meter intervals, as shown below. Figure 7 As shown, the formation matrix is ​​F keep The simulation lasted 1000 seconds, during which the three drones had to maintain their formation and fly in a spiraling ascent. It should be noted that the formation interval was set to 100 meters to more clearly show the shape of the drone formation and prevent the three drones from overlapping at a single point in the image.

[0092]

[0093] Formation flight path see Figure 8 The curve showing the change in the position error between the wingman and the target formation during flight is shown below. Figure 9 The changes in speed V, path deviation χ, and path inclination γ of the virtual lead aircraft and its three wingmen are shown in the figure. Figure 10 .

[0094] Figure 8 This visually demonstrates the effectiveness of the command generation strategy based on the MAPPO algorithm. The three triangular arrows of different colors / grayscales within the black box indicate the positions of the three aircraft at the same moment. Combined with the trajectory curves of the three aircraft, it can be seen that throughout the entire circling ascent, the three wingmen were able to maintain formation well and follow the virtual lead aircraft.

[0095] During formation flight, the formation position error Er of UAV i (i=1,2,3) i It can be represented as:

[0096]

[0097] To more clearly display the error changes, the error curves of the three machines are extracted from the first 50 seconds of the 1000-second simulation time, as shown below. Figure 9 As shown.

[0098] from Figure 9 As can be seen, within the first 50 seconds of the simulation, the formation error of the three UAVs remained within 0.2m, indicating high control precision. It should be noted that the formation-keeping process studied in this application did not involve any preset flight paths input into the UAVs. The target formation position (i.e., a point in space) of the UAVs needed to be calculated in real-time from the position of the lead UAV. Therefore, there was a time delay in the calculation and following process, inevitably leading to formation position errors.

[0099] from Figure 10 As can be seen, the initial speed of all three drones was 125 km / h. Due to the different curvatures of the inner and outer drones during the circling process, slight adjustments to their speeds were necessary to maintain formation. Drone No. 2 maintained a speed roughly consistent with the virtual leader, while the speed of the outermost drone No. 3 was adjusted to be slightly greater than that of the virtual leader, and the speed of the innermost drone No. 1 was adjusted to be slightly less than that of the virtual leader. The track tilt commands also remained synchronized with the virtual leader, fluctuating around 0.573° to ensure stable changes in drone altitude. The track deflection angles of all three drones effectively tracked the track deflection angle of the virtual leader, achieving consistent heading for the formation and enabling the circling effect. The track commands generated by each of the three drone agents were essentially consistent with those of the virtual leader, indirectly demonstrating the effectiveness of the proposed method in maintaining drone formation.

[0100] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for unmanned aerial vehicle (UAV) formation-keeping control based on deep reinforcement learning, characterized in that, include: Step 1: Based on the principles of flight mechanics, establish the flight dynamics model and kinematic model of the UAV, and design the relative motion model of the UAV based on the virtual lead aircraft topology; Step 2: Based on the PID control principle, design a PID cascade controller for stabilization, attitude, and trajectory of the UAV; Step 3: Design the Markov decision process for the MAPPO agent of each UAV, including the state space, action space, reward function, and termination condition; Step 4: Design a neural network structure suitable for this Markov decision model; Step 5: Train the designed MAPPO agent. The input of the agent is the state space, and the output is the UAV control command. The PID cascade controller receives the control command and then controls the UAV to complete the formation maintenance. Step 1 includes: A formation coordinate system is designed to describe the position of the UAV relative to the virtual lead aircraft. The virtual lead aircraft is used as the origin to establish the formation coordinate system. , The positive direction of the axis represents the velocity of the virtual salvo in the horizontal plane. The projection direction within, The axis is perpendicular to the horizontal plane and downwards. Located in the horizontal plane, it is determined by the right-hand rule; coordinate system Using a ground coordinate system to define the UAV's velocity V and heading. ,in, The aircraft's ground velocity vector V is perpendicular to the horizontal plane. The angle between them The projection of the aircraft's ground velocity vector V onto the horizontal plane and The included angle of the axis; The actual location of drone i Let i be the desired formation position of UAV i; and let the speed and heading of each UAV be... , Let be the relative position coordinates of UAV i in the formation coordinate system; the ideal formation is represented as . m is the total number of drones in the formation; In step 2, the PID cascade controller includes a trajectory PID controller, an attitude angle PID controller, control surfaces, and a UAV disturbance linearization motion model connected in sequence, wherein: The input terminal of the trajectory PID controller is used to receive guidance commands, and the output terminal is used to output attitude angle commands to the attitude angle PID controller. The output of the attitude angle PID controller is used to output control surface commands to the control surface; The output terminal of the UAV disturbance linear motion model outputs velocity and position signals, and these velocity and position signals are fed back to the input terminal of the trajectory PID controller. The UAV disturbance linear motion model also outputs attitude angle signals, which are fed back to the input terminal of the attitude angle PID controller. The UAV disturbance linear motion model also outputs attitude angle angular velocity signals, which are fed back to the input terminal of the control surface via a stabilizer. Step 5 includes: The drone swarm forms a joint state space S based on its three-axis errors relative to the ideal formation position and its own state and position information. N (s1,s2,…,s N The input is fed into the MAPPO agent, which selects the control commands (A) to be input to the UAV based on the outputs of each Actor network. N (a1,a2,…,a N In the policy training process of the agent's actioner, the policy gradient method is adopted, and a pruning-based objective function is introduced to ensure the fast convergence of the policy. Step 5 further includes: Control command A for the drone N (a1,a2,…,a N The PID cascade controllers of each UAV are input to generate control surface and throttle commands, which are then input into the UAV motion model to obtain the state of the UAV swarm at the next moment; the reward function value array R N (r1,r2,…,r N ) and the next step of the system status You can also obtain tuple data during the interaction process. All data are saved to the experience pool of each agent. At each time step, random samples are taken from the experience pool to update the network parameters in MAPPO in batches. When the time step of each round reaches the preset value or the training forced termination condition is triggered, the training of that round ends and the training process continues to the next round until the number of training rounds reaches the preset value or the reward function condition is met.

2. The method according to claim 1, characterized in that, In step 3, the expression for the state space S is: In the formula, ( x eif ,y eif ,z eif ) represents the error between the drone i and the desired position; for The differential term, for The integral term, for The differential term, for The integral term, for The differential term, for The integral term, V i For the speed of drone i, Let i be the pitch angle of the drone. Let q be the angle of attack of UAV i. i Let i be the pitch angular velocity of the UAV. Let i be the sideslip angle of the drone. Let i be the yaw rate of the unmanned aerial vehicle (UAV). Let be the roll angle of drone i.

3. The method according to claim 1, characterized in that, In step 3, the action space includes discrete speed, track yaw, and track tilt commands, with the following expressions: 。 4. The method according to claim 1, characterized in that, In step 3, the expression for the reward function is: In the formula, R f The total reward function during formation maintenance; For different reward functions; The reward coefficient is set to a negative value.

5. The method according to claim 1, characterized in that, In step 3, the termination condition for the single-round training of the i-th agent is: In the formula, [This refers to the set speed range for the drone;] This represents the upper limit of the positional error between the drone and the ideal formation.

6. The method according to claim 1, characterized in that, In step 4, the neural network structure adopts the Actor-Critic framework. The Actor network consists of three fully connected layers with 128, 128, and 27 hidden nodes, respectively. The first two layers use the ReLU activation function, and the last layer uses the Softmax activation function. The final output is the probability distribution of each action in the action library under the current policy. The Critic network contains four fully connected layers. The first three layers have 128, 128, and 128 hidden nodes, respectively, and all use the ReLU activation function. The last layer has 1 node, and the output value is a function of the current state.

Images