Unmanned aerial vehicle cluster formation control method and system based on deep learning, and medium

Abstract

The invention discloses an unmanned aerial vehicle cluster formation control method and system based on deep learning, and a medium, and belongs to the technical field of intelligent control and unmanned aerial vehicles, and the method comprises the steps: observing the flight environment of a plurality of unmanned aerial vehicles through an environment interaction module; sampling actions of the unmanned aerial vehicles in the flight environment and actions of the unmanned aerial vehicles in the area to obtain sampling actions of the unmanned aerial vehicles in the flight environment; modeling is carried out on unmanned aerial vehicle cluster formation control by adopting a Markov decision process; carrying out strategy analysis and evaluation on sampling actions of the unmanned aerial vehicles by adopting an unmanned aerial vehicle formation model which is trained in a centralized manner and executed in a distributed manner; and optimizing the action of the unmanned aerial vehicle according to a strategy analysis and evaluation result. The optimal course angle of the unmanned aerial vehicle is determined according to the obtained state information of the unmanned aerial vehicle and other unmanned aerial vehicles, and multi-target cooperative control such as autonomous obstacle avoidance, formation keeping and trajectory tracking of the unmanned aerial vehicle formation is achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent control and unmanned aerial vehicle, more specifically, and particularly relates to a method and system for controlling a cluster of unmanned aerial vehicles based on deep learning, and a medium. BACKGROUND

[0002] In recent years, quadrotor unmanned aerial vehicles have been widely used in civilian and military fields due to their simple structure and strong maneuverability. However, the operating capacity of a single unmanned aerial vehicle is limited, making it difficult to adapt to high-difficulty task requirements in complex environments.

[0003] Traditional multi-unmanned aerial vehicle formation control methods, such as PID control and Leader-Follower model, have certain limitations. These methods usually rely on preset formation rules and fixed control strategies, and have poor adaptability when facing dynamic environments or task changes. Moreover, they can only achieve specific shape formations, making it difficult to meet complex and changing requirements.

[0004] An unmanned aerial vehicle (UAV) is a kind of aircraft that does not require a pilot to directly operate, and it completes flight tasks through remote control or autonomous program control. It has been widely used in military and civilian fields. According to the flight principle and structural characteristics, unmanned aerial vehicles are mainly divided into four categories: unmanned helicopters, fixed-wing unmanned aerial vehicles, bionic flapping-wing vehicles, and rotor unmanned aerial vehicles. Among them, rotor unmanned aerial vehicles have shown significant advantages in aerial photography, inspection, rescue, and other scenarios due to their vertical take-off and landing, hovering in the air, and flexible maneuvering characteristics. With the continuous improvement of application requirements, single unmanned aerial vehicle systems have inherent defects such as limited operating capacity and small activity radius.

[0005] Current unmanned aerial vehicle formation control mainly adopts the following technical solutions: the leader-follower method achieves formation by establishing a master-slave control relationship, but has a single-point failure risk, with a system failure rate of over 60% when the leader fails; the virtual structure method relies on rigid geometric constraints to maintain formation, but it is difficult to adjust the structure in real time in dynamic obstacle scenarios, and the computational complexity increases nonlinearly with the number of unmanned aerial vehicles; the behavior rule-based method achieves cooperative control through predefined rules, but the target conflict between rules leads to control oscillation with a probability of over 40%; model predictive control (MPC) can optimize trajectory accuracy, but is sensitive to model accuracy, with a 5% deviation in aerodynamic parameters leading to a 1.2m trajectory deviation; graph theory methods optimize robustness through communication topology, but position error increases by 37% when communication delay exceeds 150ms. These methods generally face three core defects: poor adaptability to dynamic environments (failure rate exceeds 55% in sudden obstacle scenarios), dependence on environmental models, and insufficient computational scalability (MPC takes 35 times longer than DRL for 20-machine scale), and weak communication robustness (delay sensitivity threshold < 150ms).

[0006] Therefore, a more intelligent and more adaptive control method is needed to improve the flexibility and robustness of the multi-UAV formation system. SUMMARY

[0007] In view of the above or the deficiencies existing in the prior art, the present application provides a UAV cluster formation control method and system based on deep learning and a medium, which realizes end-to-end mapping from original sensor input to adaptive decision-making through a multi-agent collaborative learning mechanism, does not rely on an environment model, and is suitable for decision-making and control problems in unknown environments.

[0008] To solve the above technical problems, the present application provides the following technical solutions:

[0009] In a first aspect, the present application provides a UAV cluster formation control method based on deep learning, comprising:

[0010] observing the flight environment of the multi-UAV through an environment interaction module;

[0011] sampling the actions of the UAVs in the flight environment and the UAVs in the region to obtain the sampled actions of the UAVs in the flight environment;

[0012] modeling the UAV cluster formation control using a Markov decision process;

[0013] using a centralized training and distributed execution UAV formation model to analyze and evaluate the sampled actions of the UAVs;

[0014] optimizing the actions of the UAVs according to the results of the policy analysis and evaluation.

[0015] As a further technical solution of the present application, the sampling of the actions of the UAVs in the flight environment and the UAVs in the region specifically comprises:

[0016] The overall kinematic model of the UAV is simplified to two translational degrees of freedom and one rotational degree of freedom, where the rotational degree of freedom is the angle between the flight direction of the UAV and the x-axis, i.e., the kinematic model of the UAV is:

[0017]

[0018] wherein:

[0019] x - horizontal coordinate of the UAV;

[0020] v - speed of the UAV;

[0021] y - vertical coordinate of the UAV;

[0022] μ - angle between the heading of the UAV and the x-axis;

[0023] f(μ d ,μ)---change of angle μ of the UAV.

[0024] As a further technical solution of the present application, the action sampling of the UAV adopts a strategy network and a value function, wherein:

[0025] The value function is used to learn the mapping from the state to the value;

[0026] The strategy function is used to learn the mapping from the observation to the action distribution, or the action mean and variance mapped to the Gaussian function for action sampling.

[0027] As a further technical solution of the present application, the UAV centralized formation control adopts Markov decision process modeling; specifically comprising:

[0028] A state space determination unit is used to represent the state of the UAV through a three-dimensional array, wherein x represents the horizontal coordinate, y represents the vertical coordinate, and μ represents the angle of the UAV with the horizontal direction; the state of the multi-UAV is represented by a multi-dimensional three-dimensional array;

[0029] An action space determination unit is used to define the maneuvering action space of the UAV, and the formation control is realized by controlling the heading and speed of the UAV, and the deflection angle action space is defined as:

[0030]

[0031] In the formula:

[0032] A μ ---deflection angle action space set;

[0033] ---maximum angle of the UAV;

[0034] A reward function determination unit is used to determine the reward function, and the reward function is:

[0035]

[0036] In the formula:

[0037] r i (i=0,1,2,3)---reward value;

[0038] time---time coefficient;

[0039] d i (i=1,2,3,4)---reward coefficient;

[0040] x t ---collision detection coefficient;

[0041] y---arrival detection coefficient;

[0042] In formula (3): r0 represents a time penalty, which is negative, forcing the UAV to constantly reduce this item and constantly try; time represents the time spent from the start of training to the current step; r1 represents a distance penalty, which is the negative number of the distance of the UAV from the destination; distance represents the distance of the UAV from the end point; r2 represents a collision out-of-bounds penalty; x t represents whether a collision occurs, whether it flies out of the active area; r3 represents a reward for reaching, which is an immediate reward that can be obtained when reaching the destination; y represents whether the destination is reached; d i is the weight.

[0043] As a further technical solution of the application, the x t represents whether a collision occurs, whether it flies out of the active area, and is 1 if so and 0 if not; y represents whether the destination is reached, and is 1 if so and 0 if not; d1=1, d2=8, d3=12, d4=100.

[0044] As a further technical solution of the application, the MAPPO module, in the training phase, each agent's Critic network can access global state and action information of other agents;

[0045] In the execution phase, each agent only uses its own observed local state and policy to select actions.

[0046] As a further technical solution of the application, in the execution phase, each agent only uses its own observed local state and policy to select actions; specifically including:

[0047] Each agent i uses the objective function of formula (4) to update the policy, but the policy of each agent only depends on its own observation O i , that is:

[0048]

[0049] In the formula:

[0050] The clipped loss function of agent (i);

[0051] ∈ - clipping threshold, used to control the amplitude of policy update;

[0052] clip(·) - clipping function, limits the input value to the interval [1-∈, 1+∈];

[0053] r t (θ i ) - policy update rate of agent (i);

[0054] the advantage estimate of the agent (i);

[0055] observation O i is calculated and r t (θ i ) is the premise input, by affecting the two core variables, indirectly participate in the calculation of formula , is the key link of "environment perception → policy optimization" in the formula logic closed loop.

[0056] As a further technical solution of the application, the Critic network is trained by minimizing the mean square error loss function: specifically including:

[0057] The value function V i (s) of each agent i is estimated by a centralized Critic network, and the Critic network uses the global state s and the actions (a1, a2, a3,....a N ) of all agents to estimate the global value function; MAPPO ensures that each agent considers the strategies of other agents during training by using a centralized Critic, so as to learn more effective strategies; The goal of Critic is to minimize the mean square error loss function:

[0058]

[0059] In the formula:

[0060] L(φ i ) represents the loss function;

[0061] φ i are the parameters of the target network;

[0062] s t is the global state of the agent (i) at time t;

[0063] s t+1 is the global state of the agent (i) at time t+1;

[0064] r t is the immediate reward obtained by the agent (i) after performing the action at time t;

[0065] V i (s t ; φ i ) is the value estimate of the Critic network for the agent (i) at time t, state s t ;

[0066] R tThe actual discounted return of the agent (i) at time t and thereafter;

[0067] The expectation calculation of the state transition process, that is, traversing all possible s t t t+1 Combination, average value of the expression in the parentheses.

[0068] In a second aspect, the present application provides a multi-UAV formation control system, comprising:

[0069] An environment interaction module for observing the flight environment of the multi-UAV;

[0070] An action sampling module for sampling the actions of the UAVs in the flight environment and the UAVs in the region, and obtaining the sampled actions of the UAVs in the flight environment;

[0071] A Markov decision-making process modeling module for modeling the multi-UAV formation control using a Markov decision-making process (MDP);

[0072] A Multi-Agent Proximal Policy Optimization (MAPPO) module for performing policy analysis and evaluation on the sampled actions of the UAVs using a centralized training and decentralized execution UAV formation model;

[0073] An action optimization module for optimizing the actions of the UAVs according to the results of the policy analysis and evaluation.

[0074] In a third aspect, the present application further provides a computer storage medium comprising a processor and a memory, wherein the memory stores a computer program, and the processor implements the deep learning-based multi-UAV formation control method when executing the computer program.

[0075] The present application has the following advantages:

[0076] 1. The present application enables all UAVs to learn a control strategy for autonomously avoiding other UAVs and reaching a designated location without any prior knowledge; based on the obtained state information of the UAVs themselves and others, the UAVs determine their optimal heading angles and maintain a safe distance from other UAVs, thereby achieving the effect of multi-UAV formation control through deep learning.

[0077] 2. The Proximal Policy Optimization (PPO) algorithm framework is used to achieve multi-objective collaborative control of autonomous obstacle avoidance, formation maintenance, and trajectory tracking of UAV formation through carefully designed state space, action space, and reward functions.​​

[0078] 3、The application verifies through multiple groups of comparison, and shows that the control algorithm has excellent adaptability in a dynamic environment, and can guarantee the stability and efficiency of the formation control; at the same time, the algorithm architecture shows good scalability, and can adapt to the formation requirements of unmanned aerial vehicles of different scales.

[0079] 4、The deep learning of the application realizes end-to-end mapping from original sensor input to adaptive decision through a multi-agent collaborative learning mechanism, does not depend on an environment model, and is suitable for decision control problems in unknown environments. BRIEF DESCRIPTION OF DRAWINGS

[0080] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0081] Figure 1 A structure diagram of the unmanned aerial vehicle cluster formation control method based on deep learning provided by the embodiment of the application is provided.

[0082] Figure 2 A flowchart of the unmanned aerial vehicle formation control training using two networks provided by the embodiment of the application is provided.

[0083] Figure 3 A reward curve diagram obtained from disordered state to a specified figure (triangle) provided by the embodiment of the application is provided.

[0084] Figure 4 A system structure diagram of the unmanned aerial vehicle cluster formation control method based on deep learning provided by the embodiment of the application is provided.

[0085] Figure 5 A human-computer interaction interface system design interface diagram provided by the embodiment of the application is provided.

[0086] Figure 6 A triangle formation interface diagram of the unmanned aerial vehicle cluster formation provided by the embodiment of the application is provided.

[0087] Figure 7 A triangle-to-diamond interface diagram provided by the embodiment of the application is provided.

[0088] Figure 8 A triangle-to-rectangle interface diagram provided by the embodiment of the application is provided.

[0089] Figure 9 A rectangle-to-diamond interface diagram provided by the embodiment of the application is provided.

[0090] Figure 10The reduction is provided for the embodiment of the application to set the discrete state interface diagram;

[0091] Figure 11 The custom training function display interface diagram provided by the embodiment of the application is provided.

[0092] Figure 12 The training function display interface diagram of the system provided by the embodiment of the application. DETAILED DESCRIPTION

[0093] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0094] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the concept of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0095] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0096] In the present application, unless otherwise specifically defined and limited, the terms "mounting", "connection", "connection", "fixing" and other terms should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise specifically limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0097] Secondly, "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments. It should be noted that the implementation of the present application can be applied to any applicable scenario.

[0098] Reference Figure 1 The deep learning-based unmanned aerial vehicle swarm formation control method provided by an embodiment of the present application comprises:

[0099] Step 101, observing the flight environment of the multiple unmanned aerial vehicles through an environmental interaction module;

[0100] Step 102, sampling the actions of the unmanned aerial vehicles in the flight environment and the unmanned aerial vehicles in the region to obtain the sampled actions of the unmanned aerial vehicles in the flight environment;

[0101] Step 103, modeling the unmanned aerial vehicle cluster formation control using a Markov decision process;

[0102] Step 104, performing policy analysis and evaluation on the sampled actions of the unmanned aerial vehicles using a centralized training and distributed execution unmanned aerial vehicle formation model;

[0103] Step 105, optimizing the actions of the unmanned aerial vehicles according to the policy analysis and evaluation results.

[0104] The present application realizes unmanned aerial vehicle cluster formation control by combining deep learning with unmanned aerial vehicle clusters, adopts a hierarchical design including an algorithm environment adaptation layer, a neural network layer and a formation implementation layer, takes a multi-agent collaborative training framework as a basis, and realizes distributed decision-making through environment-model-algorithm ternary coupling. The top algorithm engine is responsible for initializing the unmanned aerial vehicle cluster environment and kinematic constraints, the middle neural network layer constructs an Actor-Critic double-flow architecture, and the bottom formation implementation layer completes policy gradient calculation and action space mapping. This hierarchical structure design forms a closed-loop path for observation data flow, policy update flow and action execution flow.

[0105] The present application realizes unmanned aerial vehicle cluster formation to form a specific image using deep learning technology, assumes that all quadrotor unmanned aerial vehicles are at the same horizontal height, that is, moving in the Euclidean plane, and mainly considers the process of forming a specific topological graph from a discrete unmanned aerial vehicle cluster, without considering adding obstacles in this plane or actual wind disturbance factors, only considering the collision between each unmanned aerial vehicle, whether each unmanned aerial vehicle reaches the destination without collision with other unmanned aerial vehicles to form a designated topological graph.

[0106] The purpose of the present application is to enable all unmanned aerial vehicles to learn a control strategy for autonomously avoiding other unmanned aerial vehicles and reaching a designated location without any prior knowledge. The strategy can determine the best heading angle of the unmanned aerial vehicle according to the obtained state information of the unmanned aerial vehicle itself and other unmanned aerial vehicles, and maintain a safe distance from other unmanned aerial vehicles to achieve the effect of unmanned aerial vehicle cluster formation control through deep learning.

[0107] The present application only considers the formation of unmanned aerial vehicle clusters at a specific height, and does not consider the overall Z-axis, so in step 102, the actions of the unmanned aerial vehicles in the flight environment and the unmanned aerial vehicles in the region are sampled to obtain a kinematic model of the unmanned aerial vehicles, specifically:

[0108] The overall kinematic model of the UAV is simplified into two translational degrees of freedom (x, y directions) and one rotational degree of freedom, wherein the rotational degree of freedom is the angle between the flight direction of the UAV and the x axis, that is, the kinematic model of the UAV is:

[0109]

[0110] In the formula:

[0111] x - horizontal coordinate of the UAV;

[0112] v - sailing speed of the UAV;

[0113] y - vertical coordinate of the UAV;

[0114] μ - angle between the sailing direction of the UAV and the x axis;

[0115] f(μ d , μ) - change of the angle μ of the UAV.

[0116] Wherein, the action sampling of the UAV is performed by using a policy network and a value function, wherein:

[0117] The value function is used to learn the mapping from the state to the value;

[0118] The policy function is used to learn the mapping from the observation to the action distribution, or the action mean and variance mapped to the Gaussian function for action sampling.

[0119] The present application uses two networks to realize the UAV formation control, the policy network π θ and the value function The present application is for the same type of intelligent agent, which is a UAV, and the action space is the same, all network parameters are the same, the whole algorithm uses the value function to learn a mapping S→R, and uses the policy function π θ to learn a mapping from the observation to a range of distribution or to the action mean and variance mapped to the Gaussian function for subsequent action sampling. The whole algorithm framework runs in a loop iteration mode, and the training process is as shown in Figure 2 .

[0120] The present application models the UAV centralized formation control problem as a Markov decision-making process (MDP) in the background of model-free reinforcement learning, and models the UAV centralized formation control by using the Markov decision-making process; specifically including:

[0121] The state space determination unit represents the state of the UAV by a three-dimensional array S=(x, y, μ), wherein x represents the horizontal coordinate, y represents the vertical coordinate, and μ represents the angle between the UAV and the horizontal direction; the state of the multi-UAV is represented by a multi-dimensional three-dimensional array; the state of the multi-UAV can be represented by a multi-dimensional three-dimensional array, wherein each three-dimensional array represents the state of one UAV. During training, x and y are random positions given by the system, and μ is set to 0 by default, that is, the initial motion direction is the same as the horizontal direction. In order to prevent cliff exploration, the maximum value of the change of μ is set during training, thereby reducing the uncertainty of training.

[0122] The action space determination unit controls the UAV by controlling the heading and speed of the UAV. By controlling the angle between the heading of the UAV and the horizontal direction, the coordinates of the UAV are obtained by multiplying each step by the speed. In order to prevent the UAV from appearing cliff exploration and falling into a local optimum, the deflection angle action space is defined as:

[0123]

[0124] In the formula:

[0125] A μ The deflection angle action space set;

[0126] The maximum angle of the UAV;

[0127] The reward function determination unit determines the reward function, and the reward function is:

[0128]

[0129] In the formula:

[0130] r i The reward value (i=0, 1, 2, 3);

[0131] time--time coefficient;

[0132] d i The reward coefficient (i=1, 2, 3, 4);

[0133] x t The collision detection coefficient;

[0134] The arrival detection coefficient;

[0135] In formula (3): r0 represents the time penalty, the application does not want the agent drone to always explore without meaning, the faster the better to reach the target, so r0 is negative, which can force the drone to constantly reduce this item and constantly try; time represents the time spent from the beginning of training to the current step; r1 represents the distance penalty, the application hopes that the drone can finally reach the destination after training, and this reward represents the negative number of the distance between the drone and the destination, when the drone is closer to the destination, the coefficient is larger, which can constantly make the drone try to reach the destination and constantly increase the reward; distance represents the distance between the drone and the terminal; r2 represents the collision out-of-bounds penalty, when the drone obviously flies out of the activity area or collides with other drones, these are not research expectations, so a certain penalty will be given; x t represents whether a collision occurs, whether it flies out of the activity area, 1 if yes, 0 if no, through x t , the size of r2 can be effectively controlled, so that the drone can constantly learn correctly and reduce the probability of collision and flying out of the activity area; r3 represents the arrival reward, which is the immediate reward obtained when reaching the destination; y represents whether the destination is reached, 1 if yes, 0 if no, through y, the size of r3 can be effectively controlled, so that the drone can constantly learn the way to reach the destination; d i is the weight, which is a fixed given default value, the default is d1=1, d2=8, d3=12, d4=100.

[0136] In the embodiment of the application, MAPPO inherits the core idea of PPO and combines the characteristics of multi-agent systems, adopts a centralized training and distributed execution architecture:

[0137] During the training phase, the Critic network of all agents can access global state and action information of other agents to learn more accurate value functions.

[0138] During the execution phase, each agent only uses its own observed local state and policy to select actions, ensuring distributed control of the system.

[0139] Further, during the execution phase, each agent only uses its own observed local state and policy to select actions; specifically including:

[0140] Each agent i in MAPPO uses the objective function of formula (4) for policy update, but the policy of each agent only depends on its own observation O i , that is:

[0141]

[0142] In the formula:

[0143] clipped loss function of the agent (i);

[0144] ∈—clipping threshold, used to control the magnitude of policy update;

[0145] clip(·)—clipping function, limits the input value within the interval [1-∈, 1+∈];

[0146] r t (θ i )—policy update rate of the agent (i);

[0147] advantage estimate of the agent (i);

[0148] observation O i is the prerequisite input for calculating and r t (θ i ), by affecting these two core variables, indirectly participates in the calculation of formula , is the key link of "environment perception→policy optimization" in the formula logic closed loop.

[0149] The Critic network is trained by minimizing the mean square error (MSE) loss function: Specifically, the value function V i (s) of each agent (i) is estimated by a centralized Critic network, which uses the global state (s) and the actions of all agents (a1, a2, a3,....a N ) to estimate the global value function; MAPPO ensures that each agent considers the strategies of other agents during training by using a centralized Critic, so as to learn more effective strategies; The goal of Critic is to minimize the mean square error (MSE) loss function:

[0150]

[0151] In the formula:

[0152] L(φ i )—denotes the loss function;

[0153] φ i —parameters of the target network;

[0154] s t —global state of the agent (i) at time t;

[0155] s t+1 —global state of the agent (i) at time rt+1;

[0156] r t — the immediate reward obtained by the agent (i) after performing action at time t;

[0157] V i t i — the value estimate of the agent (i) at time t, state s t by the Critic network;

[0158] R t — the actual discounted return of the agent (i) from time t onwards;

[0159] — the expected computation of the state transition process, that is, the average value of the expression in the parentheses for all possible s t t t+1 combinations; the purpose is to make the loss function more stable and avoid the influence of noise of a single sample.

[0160] The experimental hardware platform used by the present application is a computer equipped with an RTX 3060 GPU (12G) and a Windows 10 operating system, and the compilation environment is python 3.10 and PyTorch 2.6.0. The experimental training parameters are set as follows according to the official website: a training plan of 3125 rounds, an initial learning rate of 0.005, and an episode length of 100 for each round. In order to save training time, multiple environments are run in parallel, and 64 parallel environments are used by default.

[0161] Referring to Figure 3 , the reward curve from the disordered state to the specified figure (triangle) when the number of unmanned aerial vehicles is 8. The curve trend completely conforms to the ideal convergence form of the deep learning model, the steep rising stage reflects the efficient exploration mechanism, and the later stable characteristics reflect the strong generalization ability of the strategy network. The experimental results fully prove the learning efficiency and strategy stability advantages of the algorithm in complex environments. It can fully meet the multi-unmanned aerial vehicle formation control requirements.

[0162] The formation control accuracy and convergence time are specifically used for comparison, wherein the formation control accuracy is the position error: the deviation between the actual position of the unmanned aerial vehicle and the expected position of the formation, and the root mean square error (RMSE) is usually used as a statistical indicator, and the calculation formula is:

[0163]

[0164] In the formula:

[0165] N — the number of unmanned aerial vehicles;

[0166] ​​​​——The actual position of the i-th drone;

[0167] ——The expected position of the i-th UAV.

[0168] The essence of formation control is to accurately maintain the relative positions between drones, and the position error directly reflects the degree to which the algorithm satisfies spatial constraints.

[0169] Convergence time refers to the time it takes for all drones, starting from their initial random distribution, to settle into and stabilize in the desired formation position. It is typically defined as the first time the formation's position error consistently falls below a preset threshold (e.g., 0.1 meter). UAV formations often require rapid response in dynamic environments (e.g., emergency deployment, target tracking), and prolonged convergence time can lead to mission failure.

[0170] To validate the effectiveness of this method, the pilot-follower method was selected as a comparison benchmark. This method is currently the most mature formation control technology. Its core is to achieve coordination through a master-slave control architecture: one UAV is designated as the pilot, generating a reference trajectory based on global path planning; the remaining followers calculate their relative position (distance, angle, and velocity difference) with the pilot in real time and use PID or MPC controllers to adjust their flight strategy to maintain the formation. The experimental comparison results are shown below.

[0171] This paper compares eight drone swarms and six different formations: from discrete to rectangular, diamond, triangle, rectangle to diamond, diamond to triangle, and triangle to rectangle. The RMSE of these six formations is analyzed. Table 1 shows the comparison of the two algorithms (to four decimal places).

[0172] Table 1

[0173] Formation state Algorithm (m) Leader-follower method (m) Discrete -> rectangle 0.7242 1.1605 Discrete -> diamond 0.8625 1.0305 Discrete -> triangle 0.8245 1.2243 Rectangle to diamond 0.7639 0.5916 Diamond to triangle 0.8196 0.7981 Triangle to rectangle 0.8034 0.6675

[0174] In the core scenario of converting discrete formations to basic geometric configurations, the present invention demonstrated breakthrough performance: the rectangle construction error was reduced by 36.03% compared to the traditional pilot-follower method, the diamond configuration accuracy was improved by 19.44%, and the triangle construction error was significantly optimized by 46.33%. These data verify the adaptive ability of the deep learning model in the environmental perception-decision-control closed loop. The algorithm of the present invention effectively overcomes the modeling limitations of traditional methods in nonlinear coupled systems by autonomously exploring the intrinsic connections in the state space.

[0175] The speed of each drone in this invention is set to the same value, so when comparing time here, the number of steps is used instead of hours, minutes, and seconds. For each topology, the step size of the last drone to reach the endpoint is selected as the formation convergence time for that topology. Table 2 shows the convergence time comparison of the two algorithms.

[0176] Table 2

[0177]

[0178] The UAV formation control algorithm provided by the present application exhibits dynamic environment adaptation advantage in the dimension of formation convergence time. Experimental data shows that in the scene of converting discrete state to basic configuration, the algorithm realizes 5.1% convergence speed improvement in the discrete→rhombus task, but the step number increases by about 10% in the discrete→rectangle and discrete→triangle scenes, which maps the initial learning cost in the exploration-exploitation balance of the deep learning model. It is worth noting that in the complex task involving dynamic reconfiguration of geometric topology, the algorithm exhibits significant progress: the convergence speed is improved by 7.7% in the rhombus to triangle scene, and the step number difference in the rectangle to rhombus and triangle to rectangle reveals the sensitivity of the model to specific geometric transformations. This performance differentiation confirms that the algorithm is better at handling time-dependent problems of multi-agent collaborative decision-making.

[0179] The present application proposes a MAPPO algorithm based on deep learning to solve the problem of poor adaptability of traditional multi-UAV formation control methods in dynamic environments. By constructing a state space containing position and heading angle and a multi-objective function that integrates time penalty and obstacle avoidance reward, and combining the "centralized training-decentralized execution" architecture, autonomous obstacle avoidance and dynamic configuration maintenance of UAV formation are realized. Experiments show that, in an 8-UAV cluster scenario, the algorithm realizes a 36.03% reduction in position error compared to the traditional Leader-Follower method, and the convergence time is improved by 7.7% in complex formation transformation. Moreover, a training-demonstration integrated system is realized through PyQt-Matplotlib integration, supporting custom trajectory generation and dynamic visualization. The research results provide a solution with both theoretical innovation and engineering feasibility for multi-UAV collaborative tasks. In the future, the research will be expanded to three-dimensional space formation control, introducing wind disturbance models and dynamic obstacle avoidance mechanisms to further improve the generalization ability of the algorithm in complex real environments and promote the practicalization process of deep learning in UAV cluster control.

[0180] Referring to Figure 4 The present application also provides a multi-UAV formation control system, comprising:

[0181] An environment interaction module 201 is configured to observe the flight environment of the multi-UAV;

[0182] An action sampling module 202 is configured to sample the actions of the UAVs in the flight environment and the UAVs in the region, and obtain the sampled actions of the UAVs in the flight environment;

[0183] A Markov decision process modeling module 203 is configured to model the multi-UAV cluster formation control using a Markov decision process.

[0184] A MAPPO (Multi-Agent Proximal Policy Optimization) module 204 is used to analyze and evaluate the sampled actions of the UAVs according to the UAV formation model with centralized training and distributed execution;

[0185] An action optimization module 205 is used to optimize the actions of the UAVs according to the results of the policy analysis and evaluation.

[0186] The system is built based on the QtWidgets module of PyQt5, uses QMainWindow as the main container, and realizes the grid interface layout through QGridLayout. The designed human-computer interaction interface is as shown in Figure 5 .

[0187] The interface starts with the default display of the results of an 8-UAV cluster. After pressing the corresponding button on the right, the demonstration area will automatically demonstrate the formation process of the UAV cluster. The initial state of the UAV cluster set in this paper is on a line, (10, 2), (15, 2), (20, 2), (25, 2), (45, 2), (50, 2), (55, 2), and (60, 2) are the initial states of the 8 UAVs. After pressing the button, the current state will be automatically recognized. The trained state is a triangle, a rectangle, and a diamond. The 6-UAV cluster is a triangle, a rectangle, and a hexagon. The number of UAVs, the state of the UAV cluster, and the end point of the UAV cluster can be customized. As shown in Figure 5 , after pressing the triangle button, the 8-UAV cluster changes from a discrete state to a triangular path display.

[0188] The system uses different colors to identify the motion trajectories of different UAVs, making it easy to observe whether the UAV cluster has collided and the training effect of the motion trajectory.

[0189] The system also provides a formation change function. After pressing the diamond button again, it will recognize that the current state is a triangle and will execute a formation change from a triangle to a diamond, as shown in Figure 6 .

[0190] Other formation changes are also feasible. As shown in Figure 7 , the triangle changes to a rectangle, Figure 8 , and the rectangle changes to a diamond.

[0191] Clicking the restore button of the system interface can restore the initial position of the UAV cluster to the set discrete state, as shown in Figure 9 . The system also provides a UAV cluster formation demonstration with different numbers of UAVs. Clicking the 6-UAV demonstration button can jump to the function interface, and the operation is the same as that of the 8-UAV cluster.

[0192] The system provides a custom training function, by inputting the starting point and end point coordinates of the UAV cluster (format (x, y) for a group, enter the next UAV coordinate input), as shown in Figure 10 , for its function display.

[0193] As shown in Figure 10 , the example takes 6 UAV clusters, after inputting the starting coordinates and end coordinates of the 6 UAVs, determine the number of rounds that need to be trained (default no less than 400 rounds) click start training, the system will automatically judge whether the input value is correct corresponding, whether the number error is generated, and then it will enter the model training, and the results in the training will be output to a QTextEdit, which is convenient for observing the current training progress, as shown in Figure 11 , for the system training function display.

[0194] As shown in Figure 11 , the system will automatically identify whether the current environment can use gpu acceleration, and can use it to speed up the training progress, and then display the current training round, which is convenient for querying the current progress, and also output the reward value, frame number and other important information, which is convenient for judging the current training effect.

[0195] The application also provides a computer storage medium, comprising a processor and a memory, the memory stores a computer program, and the processor executes the computer program to realize the deep learning-based UAV cluster formation control method according to any one of claims 1-9.

[0196] In addition, although the operations of the method of the application are described in a particular order in the drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, combined into a single step, and / or divided into multiple steps.

[0197] It should also be noted that in the device of the application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the application. The above description of the disclosed aspects is provided so that any person skilled in the art can make or use the application. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the application. Therefore, the application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0198] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the application to the forms disclosed herein. Although several example aspects and embodiments have been discussed, those skilled in the art will recognize certain variations, modifications, changes, additions, and sub-combinations thereof.

Claims

1. A UAV swarm formation control method based on deep learning, characterized by: include: Observe the flight environment of multiple UAVs through the environmental interaction module; Sampling the movements of drones in the flight environment and the area, and obtaining the sampled movements of drones in the flight environment; The Markov decision process is used to model the UAV swarm formation control; A UAV formation model with centralized training and distributed execution is used to strategically analyze and evaluate the sampling actions of UAVs. The drone's actions are optimized based on the strategy analysis and evaluation results.

2. The UAV swarm formation control method based on deep learning according to claim 1 is characterized in that: The sampling of the actions of drones in the flight environment and drones in the area specifically includes: The overall kinematic model of the UAV is simplified to two translational degrees of freedom and one rotational degree of freedom, where the rotational degree of freedom is the angle between the UAV's flight direction and the x-axis. The kinematic model of the UAV is: Where: x – horizontal coordinate of the drone; v – the speed of the drone; y – vertical coordinate of the drone; μ——the angle of the drone’s heading with the x-axis; f(μ d ,μ)——the change of the angle μ of the UAV.

3. The UAV swarm formation control method based on deep learning according to claim 1 is characterized in that: The sampling of the drone's actions uses a policy network and a value function, where: The value function is used to learn the mapping from state to value; The policy function is used to learn a mapping from observations to action distributions, or to the action mean and variance of a Gaussian function for action sampling.

4. The UAV swarm formation control method based on deep learning according to claim 1, characterized in that: The centralized UAV formation control adopts Markov decision process modeling; specifically includes: A state space determination unit is used to represent the state of the UAV using a three-dimensional array, where x represents the horizontal coordinate, y represents the vertical coordinate, and μ represents the angle between the UAV and the horizontal direction; the state of multiple UAVs is represented by a multi-dimensional three-dimensional array; The action space determination unit is used to define the control action space of the UAV and realize formation control by controlling the heading and speed of the UAV. The deflection action space is defined as: Where: A μ ——Declination action space set; - The maximum angle of the drone; The reward function determining unit determines a reward function, where the reward function is: Where: r i (i=0,1,2,3)——return value; time——time coefficient; d i (i=1,2,3,4)——reward coefficient; x t ——collision detection coefficient; y——arrival detection coefficient; In formula (3), r0 represents the time penalty; time represents the time taken from the start of training to the current step; r1 represents the distance penalty, which is the negative of the distance between the drone and the destination; distance represents the distance between the drone and the destination; r2 represents the collision penalty; x t Indicates whether a collision occurs or whether the flight leaves the activity area; r3 indicates arrival reward, which is the instant reward obtained upon reaching the destination; y indicates whether the destination is reached; d i is the weight.

5. The UAV swarm formation control method based on deep learning according to claim 4 is characterized in that: The x t Indicates whether a collision occurs and whether the vehicle leaves the active area. If yes, it is 1, otherwise it is 0. y indicates whether the vehicle reaches the destination. If yes, it is 1, otherwise it is 0. d1=1, d2=8, d3=12, d4=100.

6. The UAV swarm formation control method based on deep learning according to claim 1, characterized in that: The UAV formation model adopts centralized training and distributed execution to conduct strategic analysis and evaluation of the sampling actions of UAVs; specifically, During the training phase, each agent's critic network can access the global state and action information of other agents; During the execution phase, each agent only uses its own observed local state and strategy to select actions.

7. The UAV swarm formation control method based on deep learning according to claim 6, characterized in that: During the execution phase, each agent uses only its own observed local state and strategy to select actions; specifically: Each agent i uses the objective function of formula (4) to update its strategy, but the strategy of each agent only depends on its own observation O i ,Right now: Where: — the clipped loss function of agent (i); ∈——clipping threshold, used to control the amplitude of policy update; clip(·)——truncation function, limiting the input value to the interval [1-∈,1+∈]; r t (θ i ) — the policy update rate of agent (i); — the advantage estimate of agent (i); Observation O i It is calculated and r t (θ i ) is the premise input, and indirectly participates in the formula by affecting these two core variables. The calculation is the key link of "environmental perception → strategy optimization" in the formula logic closed loop.

8. The UAV swarm formation control method based on deep learning according to claim 6, characterized in that: The critic network is trained by minimizing the mean square error loss function: The value function V of each agent i i (s) is estimated by a centralized Critic network, which uses the global state s and the actions of all agents (a1, a2, a3, ... a N ) to estimate the global value function; MAPPO uses a centralized critic to ensure that each agent considers the strategies of other agents during training, thereby learning more effective strategies; the goal of the critic is to minimize the mean squared error loss function: Where: L(φ i )——represents the loss function; φ i ——Parameters of the target network; s t — the global state of agent (i) at time t; s t+1 — the global state of agent (i) at time t+1; r t — the immediate reward obtained by agent (i) after performing the action at time t; V i (s t ;φ i )——Critic network’s evaluation of agent (i) at time t and state s t The estimated value of R t — the actual discounted reward of agent (i) at time t and thereafter; ——Expected calculation of the state transition process, that is, traversing all possible s t ,r t ,s t+1 Combine and find the average value of the expressions in the brackets.

9. UAV swarm formation control system based on deep learning, characterized by: The method for controlling a swarm of drones based on deep learning as claimed in any one of claims 1 to 8 comprises: Environmental interaction module, used to observe the flight environment of multiple UAVs; The motion sampling module is used to sample the motions of drones in the flight environment and the drones in the area, and obtain the sampled motions of drones in the flight environment; Markov decision process modeling module, used to model UAV swarm formation control using Markov decision process; The MAPPO module uses a centralized training and distributed execution UAV formation model to perform strategic analysis and evaluation of UAV sampling actions; The motion optimization module is used to optimize the UAV's motion based on the strategy analysis and evaluation results.

10. A computer storage medium, characterized in that It includes a processor and a memory, the memory stores a computer program, and when the processor executes the computer program, it implements the drone cluster formation control method based on deep learning as described in any one of claims 1 to 8.

Images