Multi-aircraft cooperative formation route planning method based on leader and follower model

Abstract

The invention discloses a multi-aircraft cooperative formation flight path planning method based on a leader and follower model, and relates to the technical field of environment perception and unmanned aerial vehicle cluster cooperation. The method comprises the following steps of: firstly, designing a multi-agent double delay depth deterministic strategy gradient (LFMATD3) based on a leader-follower model, and converting an optimization problem model into a Markov decision process model by introducing an artificial potential field model and a reward function; and secondly, constructing an independent agent for each unmanned aerial vehicle, optimizing a behavior strategy of the unmanned aerial vehicle by combining a reward function based on an algorithm framework of deep reinforcement learning, and performing flight path planning and realizing dynamic formation control. The method provided by the invention can effectively improve the formation stability and collaboration of the unmanned aerial vehicles in a complex environment.

Description

Technical Field

[0001] The present invention relates to the field of environmental perception and UAV cluster collaboration technology, and in particular to a method and system for obstacle avoidance and trajectory planning of multi-UAV collaborative formation in a complex environment. Background Art

[0002] Unmanned aerial vehicles (UAVs) are small, unmanned aircraft. Their flexibility, efficiency, and relatively low cost have made them widely adopted across multiple sectors. However, as missions become more challenging and operational scopes expand, single UAVs are increasingly unable to meet the demands of mission execution and operational requirements. Their limited payload capacity, difficulty handling multiple tasks simultaneously, and low fault tolerance restrict their application. To address this, researchers have turned their attention to multi-UAV systems (MUSs). By deploying multiple UAVs and enabling information sharing and collaborative operations, they overcome the limitations of a single UAV and achieve distributed interaction and collaborative operations. Distributed collaborative trajectory planning is crucial in MUSs. Its goal is to tailor safe and efficient flight paths and maneuvers for each UAV, taking into account multiple factors such as mission priorities, UAV performance differences, and environmental constraints. This requires close coordination between UAVs to effectively avoid collisions while ensuring optimal system performance.

[0003] Multi-UAV systems utilize swarm intelligence strategies to perform more complex missions, but this also introduces a host of new challenges. For example, how to allocate UAV resources based on mission requirements to ensure each drone performs at its maximum capacity; how to achieve real-time communication and information sharing between UAVs to enable the swarm to adjust its strategies in real time; and how to coordinate the movements and positions of UAVs to ensure they maintain safe distances and successfully complete the mission. To address these challenges, researchers have conducted in-depth research on trajectory planning algorithms and developed a series of intelligent algorithms.

[0004] Traditional trajectory planning algorithms, such as the Artificial Potential Field (APF), the Probabilistic Roadmap Method (PRM), and the A* algorithm (Astar), have limitations when dealing with complex, dynamic, and large-scale environments. These algorithms often struggle in scenarios where environmental information is scarce or dynamically changing. Consequently, a number of bio-inspired intelligent optimization algorithms have been proposed, such as Ant Colony Optimization (ACO), Particle Swarm Optimization (PSO), and Artificial Bee Colony (ABC). However, most bio-inspired algorithms focus on intuitive strategies in their design. When directly applied to the distributed trajectory planning task of multiple UAVs, they often fail to achieve the expected efficiency and accuracy. While these algorithms can optimize the trajectory of a single UAV, they often overlook the interaction between the motion states of UAVs, especially those between neighboring UAVs. This limits their effectiveness in achieving coordination and dynamic adaptation across the entire UAV network.

[0005] Subsequently, some scholars proposed a path planning algorithm that combines reinforcement learning (RL) and incorporates the feature perception capabilities of deep learning, and further proposed a path planning algorithm based on deep reinforcement learning (DRL). These algorithms fully leverage the advantages of deep reinforcement learning in environmental adaptability and feature extraction, treating each drone as an intelligent agent and using a distributed or centralized framework to train drones in complex and changing environments, optimize their action strategies, ensure smooth information sharing and collaboration between drones, and maintain system stability to efficiently complete trajectory planning tasks. This type of algorithm improves the drone's ability to analyze complex environmental information and learns the optimal collaborative strategy through continuous exploration and trial and error, thereby demonstrating higher autonomy and adaptability in trajectory planning.

[0006] Current research in drone trajectory planning technology focuses on three core areas: drone swarm trajectory planning, drone collaborative task allocation, and drone collaborative formation control. The latter aims to address the formation maintenance and control issues of drones in flight. By fine-tuning formation structure, formation transformations, and obstacle avoidance strategies, it achieves efficient coordination of swarm actions.

[0007] The challenges of cooperative drone formations primarily focus on maintaining formation stability and obstacle avoidance capabilities. Formation maintenance requires the drone formation to be able to quickly respond to environmental factors such as wind speed changes and turbulence, and to maintain the consistency and stability of the formation by adjusting flight attitude and movements. For example, methods based on artificial potential fields exist. However, in actual scenarios, the non-zero distribution of potential field forces often causes drones to oscillate near that location, thus affecting the stability of the formation. Formation obstacle avoidance requires the drone formation to make real-time obstacle avoidance decisions when encountering obstacles. This requires algorithms that can quickly process sensor data and generate effective obstacle avoidance paths. During the obstacle avoidance process, the drone formation must not only ensure individual safety but also maintain the integrity of the formation. This requires precise collaborative control algorithms to coordinate the actions of each drone to ensure that the drone formation can perform its mission efficiently and safely. Summary of the Invention

[0008] Technical solution: To achieve the above-mentioned purpose, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a method for distributed collaborative trajectory planning of a swarm of unmanned aerial vehicles (UAVs) based on deep reinforcement learning, comprising the following steps:

[0010] Based on the UAV swarm system architecture for formation operations, under environmental and inter-machine constraints, the individual UAV motion decision-making, efficient information sharing and decision-making coordination of multi-machine systems are used as optimization variables, and a joint optimization problem model is constructed with the optimization goals of maximizing formation integrity and minimizing risk obstacles. The optimization problem model is converted into a Markov decision process model, and an independent intelligent agent is constructed for each UAV. Based on the deep reinforcement learning algorithm framework, the reward function is combined to optimize the UAV's behavior strategy, perform trajectory planning, and realize dynamic formation control.

[0011] Furthermore, the UAV swarm's collaborative obstacle avoidance and trajectory planning decisions are designed based on a leader-follower model. The leader UAV is responsible for avoiding obstacles and planning the optimal flight path, while the follower UAVs dynamically adjust their position and speed based on the leader's status to maintain the formation structure and avoid obstacles. The obstacle avoidance strategy is implemented using an artificial potential force model, and a customized reward function is constructed to assist in adjusting the follower UAVs' formation. This ensures that the follower UAVs can effectively perform formation-following tasks in the leader-follower model.

[0012] Furthermore, the state space of the drone cluster includes a leader state space and a follower state space, wherein the leader state space includes obstacle information, leader information and target information, and the follower state space includes lidar detection information and follower drone information itself; the action space includes the position adjustment vector and speed and heading control strategy of each drone in three-dimensional space to achieve collaborative obstacle avoidance and target task completion of the cluster.

[0013] Furthermore, the dynamic formation reward function design incorporates unique reward functions for different roles, ensuring that the reward function encourages leaders and followers to collaborate to achieve their goals. The leader drone's reward scheme consists of three components: a reward for moving toward the target point, a reward for avoiding obstacles, and a time-step penalty. The follower drone's reward scheme consists of four components: an angle constraint reward, a distance constraint reward, a following constraint reward, and a potential field force reward.

[0014] Furthermore, this deep reinforcement learning-driven collaborative control mechanism is based on a Leader-Follower Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (LFMATD3) algorithm. Each drone has two critic networks evaluating the value function of its given state and action, more accurately guiding the update of the actor network and accelerating the convergence of the entire algorithm. The actor network makes action decisions by observing the surrounding environment.

[0015] Furthermore, the dynamic formation control strategy adjusts the flight trajectory of each drone in real time through the relative position and speed feedback between the leader and the follower to maintain the stability of the formation; at the same time, after each drone moves, the flight path is adjusted to ensure the coordination and stability of the entire formation in complex environments.

[0016] Furthermore, the collaborative trajectory planning establishes a simulation environment model, divides the study area into multiple grids, sets a path perturbation mechanism, and constructs diversified scenarios by changing the target area position and the starting formation structure of the UAV to enhance the diversity and environmental adaptability of the training samples.

[0017] In a second aspect, the present invention provides a UAV swarm collaborative obstacle avoidance and trajectory planning system based on deep reinforcement learning, comprising:

[0018] Problem building module: Establish a multi-UAV cooperative communication system model in a complex environment, divide the UAV cluster into at least one leader and several followers, and jointly optimize the UAV trajectory planning and formation collaboration to construct a joint optimization problem with the goal of maximizing formation integrity and minimizing risk obstacles;

[0019] The neural network building module adopts the multi-agent delayed dual deterministic policy gradient (LFMATD3-PER) framework to build a distributed decision-making model: a centralized critic network and a distributed actor network are designed. The critic network receives the global state, including the positions of all drones, the obstacle distance array, and the leader-follower relative information, and uses a dual Q-network structure to output a joint Q value to reduce the overestimation bias. Each drone independently deploys a set of actors, inputs the local observation state, including its own speed / heading, 16-line lidar obstacle distance, and the relative distance to the leader and the nearest follower, and outputs a continuous action vector for real-time trajectory adjustment.

[0020] The network training module is used to define the reward function. The reward function is divided into a leader reward function and a follower reward function based on different roles. The leader's reward function consists of three parts: a positive reward for approaching the target, a negative penalty for being too close to an obstacle, and a time step penalty. The follower's reward function consists of a penalty for formation angle deviation, a penalty for distance deviation within the formation, a penalty for being too close to an obstacle, and a potential field force reward. The loss function is defined as minimizing the error between the output Q value of the critic network and the target value. Delayed actor updates and soft updates are used to synchronize the target network weights.

[0021] The communication module is used to realize the status broadcast and reception between each drone in the cluster. In particular, the leader periodically sends position and posture information to all followers, and at the same time receives local perception data sent back by followers to ensure formation coordination and information synchronization.

[0022] In a third aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the multi-aircraft collaborative formation trajectory planning method based on the leader-follower model are implemented.

[0023] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method for collaborative obstacle avoidance and trajectory planning of a drone cluster based on deep reinforcement learning.

[0024] Beneficial effects: The present invention proposes a multi-machine collaborative formation trajectory planning method based on the leader-follower model. This method effectively improves the adaptability of the UAV formation to various complex environments and the formation coordination. First of all, the leader-follower model plays a vital role in the UAV formation, and its flexible organizational structure significantly improves the coordination efficiency. The designed method shows higher robustness and environmental adaptability when facing a dense obstacle environment. Under a variety of different obstacle density conditions, the method can maintain a high mission execution success rate, while the speed consistency and formation continuity rate are better than other algorithms. This means that the method can cope with challenges such as formation maintenance, collision avoidance and collaborative control that arise as the number of UAVs increases. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of the UAV formation path based on deep reinforcement learning in an embodiment of the present invention.

[0026] Figure 2 This is a framework diagram of the LFMATD3-PER algorithm in an embodiment of the present invention.

[0027] Figure 3 1 is the convergence performance curve of three different algorithms: MATD3, MADDPG and LFMATD3-PER in the embodiment of the present invention.

[0028] Figure 4 This is the trajectory planned by the LFMATD3-PER algorithm in the embodiment of the present invention.

[0029] Figure 5 This is a comparison chart of algorithm success rates under different obstacle densities in an embodiment of the present invention.

[0030] Figure 6 This is a comparison chart of algorithm formation persistence rates under different obstacle densities in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0032] Combine Figure 1In the scenario shown, an embodiment of the present invention discloses a multi-drone collaborative formation trajectory planning method LFMATD3-PER based on a leader-follower model. For the UAV collaborative formation in complex environments, the MATD3 strategy is introduced to solve the problem of UAV collaborative formation maintaining formation stability and obstacle avoidance. This method maximizes the maintenance of formation integrity, improves the formation collaborative operation capability, and enhances environmental adaptability under constraints such as following distance constraints, angle constraints, and collision constraints. In order to solve this multi-objective optimization problem, a deep reinforcement learning algorithm for distributed collaborative control of multiple UAVs is designed. This algorithm combines the dynamic formation artificial potential field force design with the dynamic formation reward function to solve the problems of information sharing and decision-making coordination in multi-drone system trajectory planning and formation obstacle avoidance in complex environments.

[0033] The LFMATD3-PER method, a leader-follower model-based multi-drone collaborative formation trajectory planning method, describes an embodiment of the present invention. First, based on a UAV swarm system architecture for formation operations, it comprehensively considers UAV collaborative formation, distance constraints, angle constraints, and collision constraints in complex environments. Using individual UAV motion decisions and efficient information sharing and decision-making coordination across multiple UAV systems as optimization variables, it constructs a joint optimization problem model with the optimization objectives of maximizing formation integrity and minimizing risk obstacles. The optimization problem model is then transformed into a Markov decision process model, with independent agents constructed for each UAV. Based on a deep reinforcement learning algorithm framework, the UAV's behavior strategy is optimized in conjunction with a reward function to perform trajectory planning and achieve dynamic formation control. Specifically, in this implementation, the UAV formation coordination problem is represented by an artificial potential field model, the multi-drone formation navigation problem is represented by a dynamic formation reward function, and a critic network and actor network are combined to accelerate algorithm convergence and provide action decisions based on the surrounding environment. Simulation results show that compared with the baseline algorithm, the proposed method improves the average success rate by 5.1% and 7.0%, respectively, and the formation persistence rate by 28.5% and 42.6%, respectively. Furthermore, in environments with varying obstacle densities, the proposed method maintains an average success rate of 90%, and its formation persistence rate far outperforms the other two baseline algorithms. Experiments have verified that the proposed method exhibits excellent formation stability and coordination in complex environments.

[0034] The specific steps of the embodiment of the present invention are described in detail below in conjunction with a specific system model.

[0035] Step (1) is to establish a UAV collaborative formation and path planning architecture, including a leader-follower model and an artificial potential field model; the specific steps are as follows:

[0036] (1a) Design a deep reinforcement learning algorithm for distributed cooperative control of multiple UAVs. This algorithm requires multiple UAVs to form and maintain a certain formation in a confined space, while avoiding obstacles along the way, and ultimately reach a predetermined target area. This embodiment of the present invention sets the target area as a 1600m×1600m square scene, including a leader UAV and two follower UAVs completing a triangular formation navigation task. The distance between obstacles is 250m, and the obstacle radius is 50m.

[0037] (1b) Establish an artificial potential field model. This model consists of two parts: following gravity and anti-collision repulsion, ensuring that the follower UAV can effectively perform the formation following task in the leader-follower model. The distance between the leader and the follower is divided into two stages, and a distance-dependent following gravity function is designed, with different gravity values ​​for each stage. The following gravity function is as follows:

[0038]

[0039] Among them, p j To follow the current position of UAV j, P leader is the leader drone position, d(p j , p leader ) is the relative distance between follower j and the leader, D0 is the distance threshold, ξ is the gravitational coefficient, and the following gravitational The direction of the follower drone points toward the leader drone. When the distance between the follower and the leader is too large, the follower drone experiences a greater gravitational pull, causing it to quickly approach the leader drone. However, when the distance between the follower and the leader is too close, posing a collision risk, the gravitational force is reduced through nonlinear attenuation. This reduces the follower's acceleration and avoids collisions or oscillations caused by excessive speed.

[0040] When solving the collision problem between followers, we can treat other followers as dynamic obstacles and design an anti-collision repulsion function. The anti-collision repulsion function is as follows:

[0041]

[0042] Among them, ε is the repulsion coefficient, D1 is the safety distance threshold between machines, d(p j , p k ) is the relative distance between follower j and follower k. Therefore, the total repulsive force of the inter-machine collision avoidance on follower j is:

[0043]

[0044] Among them, N is the other follower drones other than follower j, and their total repulsive force is

[0045] (1c), the present invention uses state space and action space to describe the information of the leader and the follower. Specifically, in the leader-follower model, the leader state space S leader Including obstacle information Leader Information self =[v t ,θ t ], target information It contains the relative distance value and relative angle value between the leader and the target. Therefore, the state space S of the leader drone is leader ={S self , S obsstacle , S goal}.

[0046] In the follower state space Represents the laser radar detection information, S self =[v t ,θ t ] represents the follower drone’s own information. The state space S related to the leader l The definition is as follows:

[0047]

[0048] in, Indicates follower j 和 The horizontal distance between leaders, represents the angle between the heading of UAV j and the leader.

[0049] The last part of the status information S other is the distance between follower j and other followers in the neighborhood, defined as follows:

[0050]

[0051] in, and Indicates follower j 和 The distance between two adjacent follower drones. Therefore, the state space of follower j is expressed as:

[0052] S j ={S self , S obstacle , S l , S other}

[0053] At the same time, designing a suitable action space can not only promote effective collaboration between drones, but also improve their overall efficiency and flexibility when executing trajectory planning. The action equation of drone i is defined as:

[0054]

[0055] in, and are the increases and decreases in the speed and heading of drone i, respectively, and are normalized to [-1, 1]. The action space of drone i is:

[0056] A i =[a i ,φ i ]

[0057] Therefore, the action space of all drones is:

[0058] A={A l ,…,A N}

[0059] Step (2) is to design a dynamic formation reward function, including the leader’s reward function and the follower’s reward function. The specific steps are as follows:

[0060] (2a), design the reward function of the leader UAV. Specifically, the reward scheme of the leader UAV i consists of the following three parts.

[0061] The first part is the reward for moving towards the goal point:

[0062]

[0063] in, is the distance between UAV i and the target at time t; is the distance between drone i and the target at time t-1; this reward encourages the leader to reach the target point as quickly as possible.

[0064] The second part is the reward for avoiding obstacles:

[0065]

[0066] As the leader drone, it should keep a safe distance from obstacles. obstacle to ensure the safety of the flight.

[0067] The third part is the time step penalty:

[0068]

[0069] Therefore, the total reward of leader drone i is:

[0070]

[0071] (2b), design the reward function of the leader UAV. Specifically, the reward function of the follower UAV j is composed of the following four parts:

[0072] The first part is the angle constraint reward function:

[0073]

[0074] Among them, α is the angle weight coefficient, θ j is the angle between UAV j and the line connecting the two adjacent UAVs, and θ0 is the angle that UAV j is expected to maintain with its adjacent UAVs. This expected angle is calculated based on the number of UAVs in the mission scenario.

[0075] The second part is the distance constraint reward function:

[0076]

[0077] In the formation process, in order to ensure that the follower drones can effectively avoid collisions and obstacles, the reward function is set to a larger penalty value, and the collision distance threshold D c It should also be adjusted accordingly according to different mission scenarios.

[0078] The third part is the following constraint reward function:

[0079]

[0080] in is the distance between the follower UAV and the leader at time f; is the distance between the follower and the leader at time t-1. The above formula ensures that each follower UAV remains within a reasonable distance from the leader UAV, avoiding falling behind and collisions.

[0081] The fourth part of the reward is the potential field force reward function:

[0082]

[0083] During the follower drone's navigation, the reward value at the current moment t is determined by the gravitational difference and the total repulsive force. By introducing the gravitational and repulsive reward mechanisms, the drone formation can dynamically adjust its formation and avoid obstacles, maintaining a stable formation and improving mission execution efficiency and coordination.

[0084] Therefore, the total reward of the follower drone is:

[0085]

[0086] Step (3): Based on the above model, construct the LFMATD3-PER algorithm for formation operation; including the following specific steps:

[0087] (3a), in Figure 2 The LFMATD3-PER algorithm framework process is described in

[15] : Assuming that there are N drones in a multi-drone formation navigation task, the formation trajectory planning problem is modeled as a Markov decision process. In the training phase, a batch of samples are first sampled from the priority experience replay buffer and these samples are used to update the network parameters of all drones. During this process, drone i will be updated based on its observation value o at the current time step. i,t To determine the action, and output the action a i，t Feedback to the environment to obtain the reward value r of the current time step i and the observation value o at the next time step i，t+1 , and save it in the priority experience back pool, repeating the training until the end. In order to simulate the impact of uncertain factors in the environment on the action, Ornstein-Uhlenbeck noise is introduced in the output action, as shown in the following formula:

[0088] a i，t =π i (o i，t |θ i，t +oU(μ1,θ1,σ1)

[0089] In the formula, μ1 represents the mean, θ1 is the regression coefficient, and σ1 is the diffusion factor. OU noise can improve the exploration capability of drones. Its characteristics enable the drone to achieve smooth transitions in action during the exploration phase and help avoid falling into local optimal solutions. However, attention must be paid to the parameter setting of OU noise, because excessive noise may lead to excessive exploration, which in turn affects learning efficiency. Therefore, the appropriate selection of OU noise parameters is crucial to achieving ideal noise effects. The MATD3 algorithm introduces a priority experience replay mechanism to speed up training by calculating the priority of sample k from the TD error. As shown in the following formula:

[0090] p k =|δ k |+ρ

[0091] where δ k is the TD error, and ρ is a constant greater than 0, ensuring that the sampling probability is not 0.

[0092] The sampling probability of sample k is as follows:

[0093]

[0094] Where N is the total number of samples, Indicates the priority of the k-th sample.

[0095] (3b), two targets Q are calculated by two target critic networks, and the corresponding target value y can be calculated as:

[0096]

[0097] In the subsequent update process, the smaller Q value is used as the final target value:

[0098] y=r+γmin(y1,y2)

[0099] The Critic network updates the network parameters by minimizing the error between the output Q value and the target value. The error is calculated by the following loss function:

[0100]

[0101] (3c) A policy update delay mechanism is adopted, which first optimizes the Critic network n times, and then synchronously updates the Actor network and its target network. This strategy ensures that the Critic network reaches a more stable state before assisting the Actor network, which reduces the frequency of update operations and improves the overall stability of the algorithm. In addition, this method encourages the drone to focus more on exploring the current strategy instead of over-relying on historical strategies. The Actor network parameter update formula is as follows:

[0102]

[0103] And the parameters of each target network are soft-updated by the following formula:

[0104]

[0105] Step (4): Based on the above modeling, the specific process of executing the method is as follows:

[0106] (4a), in the initialization stage, the Actor network and the dual Critic network (Q1, Q2) of each agent and their corresponding target network are initialized, and the experience replay pool D is initialized to store the (state, action, reward, next state) quadruple. Here, Prioritized Experience Replay (PER) is used to assign sampling probability to samples according to TD error and reward size.

[0107] (4b) In the episode-level training loop, the outer loop controls the total number of training episodes. At the beginning of each training episode, the initial scene state is randomly generated, including the positions of the leader and followers, the distribution of obstacles, and the location of the target area. Each episode consists of multiple time steps, simulating a continuous decision-making process.

[0108] (4c), during the leader drone’s decision execution process, each leader inputs the current state into its Actor network and outputs an action (such as speed or angle change). At the same time, the environment feeds back rewards and next states based on the action and saves samples into the experience pool.

[0109] (4d), during the behavior execution process of the follower drone, each follower outputs an action based on the current state (including the state of neighbors, the distance to the leader, etc.), the environment updates the state, and uses the artificial potential field model to calculate the reward function, and adds the data sample to the experience pool D.

[0110] (4e), in the network training part, a number of samples are sampled from the experience pool D for each UAV according to the priority weight, and the TD target is calculated using the target Q value. After the calculation is completed, the TD error is minimized and the critic network parameters are updated.

[0111] (4f) In the actor update section, a delayed update strategy is adopted, whereby actors are updated every few steps instead of every step to avoid policy instability. Soft updates are performed on the target actor network and the target critic network to gradually and smoothly copy the current network parameters to the target network to avoid severe fluctuations.

[0112] (4g), after completing one round of training process, enter the next round of environment initialization and training.

[0113] The following simulation experiments are conducted to compare the present invention with traditional algorithms to verify the beneficial effects of the present invention.

[0114] exist Figure 3 The performance comparison between the embodiment of the present invention and the existing mainstream solutions MATD3 algorithm and MADDPG algorithm is described. Figure 3 As can be seen from the graph, the LFMATD3-PER algorithm begins to converge after the 1,000th cycle, while the MADDPG algorithm requires 2,000 cycles to reach convergence. This means that the LFMATD3-PER algorithm not only converges faster but also exhibits more robust and adaptable behavior in practical applications. In contrast, the MADDPG and MATD3 algorithms both have lower convergence speed and final converged average reward than the LFMATD3-PER algorithm.

[0115] exist Figure 4In the figure, the trajectory planned for a drone formation by an embodiment of the present invention is described. The drone formation starts from the same starting position and arrives at the same target point. In the figure, the leader drone flies along the pink path, while the blue and yellow paths are planned by the follower drones. The LFMATD3-PER algorithm of the embodiment of the present invention shows excellent performance. It can effectively avoid falling behind during the drone formation flight and can quickly adjust and recover the formation when encountering obstacles. The advantage of this algorithm is that it can respond quickly and accurately to emergencies while maintaining the integrity of the formation. This shows that the distributed LFMATD3-PER algorithm of the leader-follower model proposed in the present invention can effectively improve the overall coordination and flight efficiency of the formation, and has broad application prospects.

[0116] exist Figure 5 and Figure 6 The article describes a comparison of the algorithm success rates of the embodiment of the present invention and the MATD3 algorithm under different obstacle densities. As the obstacle density increases, the average success rate of the MATD3 algorithm decreases more significantly. In contrast, the LFMATD3-PER algorithm performs more stably. This comparison fully demonstrates that the LFMATD3-PER algorithm exhibits greater robustness and environmental adaptability when faced with dense obstacle environments. Under various obstacle density conditions, the LFMATD3PER algorithm is able to maintain a high task execution success rate, averaging 90%, while the MATD3 algorithm's success rate is relatively low, averaging only 83%. This shows that the LFMATD3-PER algorithm has superior environmental adaptability.

[0117] Based on the description of the present invention, it should not be difficult for those skilled in the art to see that the multi-aircraft collaborative formation trajectory planning method LFMATD3-PER designed by the present invention, which is based on the leader-follower model, can effectively improve the adaptability and formation coordination of UAV formations to various complex environments.

[0118] Based on the same inventive concept, an embodiment of the present invention also discloses a multi-aircraft collaborative formation trajectory planning system based on a leader-follower model. The system comprises: a leader-follower model that generates a simulated square environment with a length and width of 1600 meters, in which a leader drone and two follower drones complete a triangular formation navigation task. The distance between obstacles is 250 meters, and the obstacle radius is 50 meters. An artificial potential field model is used to construct a customized reward function to assist in adjusting the formation of the follower drones. This ensures that the follower drones can effectively perform the formation following task within the leader-follower model. A deep reinforcement learning decision model utilizes an iterative mechanism of a critic network and an actor network. The two critic networks evaluate the value function for a given state and action. These networks have the same structure but different network parameters. This aims to reduce overestimation of Q values, more accurately guide the updates of the actor network, and accelerate the convergence of the entire algorithm. The actor network provides action decisions by observing the surrounding environment. A reward function module ensures that the reward function encourages the leader and follower drones to collaborate to achieve the goal.

[0119] The embodiment of the present invention also discloses a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the method for multi-aircraft cooperative formation trajectory planning based on the leader-follower model is implemented.

[0120] An embodiment of the present invention further discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multi-aircraft cooperative formation trajectory planning method based on the leader-follower model.

[0121] The program code for implementing the inventive method can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that the program code, when executed by the processor or controller, causes the steps of the inventive method to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server. The present invention is not described in detail herein, and all of these are known techniques to those skilled in the art.

Claims

1. A multi-aircraft cooperative formation trajectory planning method based on a leader-follower model, characterized in that: The following steps are involved: Establish a multi-UAV cooperative communication system model in a complex environment, divide the UAV cluster into at least one leader and several followers, and jointly optimize the UAV trajectory planning and formation collaboration to construct a joint optimization problem with the goal of maximizing formation integrity and minimizing risk obstacles; Design a dynamic formation reward function based on an artificial potential field, taking into account factors such as target gravity, obstacle repulsion, following angle error, and formation maintenance error; A distributed decision-making model for multiple UAV agents is constructed using the Leader-Follower Multi-Agent TwinDelayed Deep Deterministic Policy Gradient (LFMATD3) algorithm framework. A critic network and an actor network are designed with centralized training and distributed execution. The critic network evaluates the agent's state and actions and outputs Q-value evaluations. The actor network provides action decisions by observing the surrounding environment. Define the state space of the leader and follower. The leader's state space includes its own speed, heading angle, obstacle distance array, and target relative position information; the follower's state space includes its own information, relative position and angle with the leader and adjacent followers; Define the reward functions for the leader and follower. The leader reward function is the weighted sum of the target distance factor, risk aversion factor, and time step penalty factor. The follower reward function is the weighted sum of the angle constraint factor, distance constraint factor, following constraint factor, and potential field force factor. Define the loss function as the error between the minimized output Q value of the critic network and the target value. During the execution phase, the drone generates formation obstacle avoidance trajectories based on the trained LFMATD3 model. At the same time, through priority sampling, it achieves collaborative optimization of improving data utilization efficiency and enhancing the overall performance of the learning process.

2. The multi-aircraft cooperative formation trajectory planning method based on the leader-follower model according to claim 1 is characterized in that: The LFMATD3 model models the formation trajectory planning problem as a Markov decision process. In the training phase, a batch of samples are first sampled from the priority experience replay buffer and the network parameters of all drones are updated using these samples. In this process, drone i will update the network parameters of all drones according to its observation value o at the current time step. i，t To determine the action, and output the action a i，t Feedback to the environment to obtain the reward value r of the current time step i and the observation value o at the next time step i，t+1 , and save it in the priority experience pool, repeating the training until the end.

3. The multi-aircraft cooperative formation trajectory planning method based on the leader-follower model according to claim 1 is characterized in that: The artificial potential field model consists of two parts: following gravity and anti-collision repulsion. The following gravity term and the anti-collision repulsion term are nonlinearly scaled according to the change in distance based on the following formula to avoid collision or oscillation problems.

4. The multi-aircraft cooperative formation trajectory planning method based on the leader-follower model according to claim 1 is characterized in that: The dynamic formation reward function includes the leader's reward function and the follower's reward function. The leader's reward function r consists of three parts: a positive reward for approaching the target, a negative penalty for being too close to an obstacle, and a time step penalty. The follower's reward function r consists of four parts: a penalty for formation angle deviation, a penalty for distance deviation within the formation, a penalty for being too close to an obstacle, and a potential field force reward.

5. The multi-aircraft cooperative formation trajectory planning method based on the leader-follower model according to claim 1 is characterized in that: The described experience replay mechanism combines TD error and immediate reward value to calculate sample priority and reduces bias through importance sampling method.

6. The multi-aircraft cooperative formation trajectory planning method based on the leader-follower model according to claim 1 is characterized in that: In this method, each agent consists of two critic networks and one actor network, and uses a "soft update" approach to update the target network to improve network stability. First, two target Q values ​​are calculated using the two target critic networks. The smaller Q value is used as the final target value in subsequent updates. The critic network updates its parameters by minimizing the error between the output Q value and the target value. The critic network is then iteratively optimized n times, and then the actor network and its target network are simultaneously updated.

7. The multi-aircraft cooperative formation trajectory planning method based on the leader-follower model according to claim 1 is characterized in that: During the training phase, a simulation platform is used for environmental modeling, a path perturbation mechanism is set up, and diversified scenarios are constructed by changing the target area location and the starting formation structure of the UAVs to enhance the diversity of training samples and environmental adaptability. The simulation environment supports dynamic interaction and formation structure transformation, which can effectively evaluate the robustness and generalization performance of the method.

8. A multi-aircraft cooperative formation trajectory planning method based on a leader-follower model that implements the method described in claims 1-7, characterized in that: include: Problem building module: Establish a multi-UAV cooperative communication system model in a complex environment, divide the UAV cluster into at least one leader and several followers, and jointly optimize the UAV trajectory planning and formation collaboration to construct a joint optimization problem with the goal of maximizing formation integrity and minimizing risk obstacles; The neural network building module adopts the multi-agent delayed dual deterministic policy ladder (LFMATD3-PER) framework to build a distributed decision-making model: a centralized critic network and a distributed actor network are designed. The critic network receives the global state, including the positions of all drones, the obstacle distance array, and the leader-follower relative information, and uses a dual Q-network structure to output a joint Q value to reduce the overestimation bias. Each drone independently deploys a set of actors, inputs the local observation state, including its own speed / heading, 16-line lidar obstacle distance, and the relative distance to the leader and the nearest follower, and outputs a continuous action vector for real-time trajectory adjustment. The network training module is used to define the reward function. The reward function is divided into a leader reward function and a follower reward function based on different roles. The leader's reward function consists of three parts: a positive reward for approaching the target, a negative penalty for being too close to an obstacle, and a time step penalty. The follower's reward function consists of a penalty for formation angle deviation, a penalty for distance deviation within the formation, a penalty for being too close to an obstacle, and a potential field force reward. The loss function is defined as minimizing the error between the output Q value of the critic network and the target value. Delayed actor updates and soft updates are used to synchronize the target network weights. The communication module is used to realize the status broadcast and reception between each drone in the cluster. In particular, the leader periodically sends position and posture information to all followers, and at the same time receives local perception data sent back by followers to ensure formation coordination and information synchronization.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images