Unmanned aerial vehicle formation keeping method and system based on Attn-MADDPG algorithm

Abstract

The application discloses a UAV formation keeping method and system based on an Attn-MADDPG algorithm, relates to the technical field of UAV formation, and comprises the following steps: constructing a kinematics model of a UAV and a formation target optimization model under the condition of a set cost constraint, with the goal of reaching the fastest speed, along the shortest driving path, and meeting the whole-process collision-free safety requirement; constructing a Markov decision process in a navigation path planning, which is used for autonomous judgment and speed control adjustment of a navigator UAV; setting and training a deep deterministic policy gradient algorithm model based on an attention mechanism, wherein the model comprises a centralized evaluation network, multiple distributed execution networks and a replay experience pool group; the model learns a decision strategy through centralized training, and the execution is distributed, and an attention mechanism is added in the evaluation network; the trained model is used for executing multi-UAV system formation and formation keeping; and the method realizes path reliability planning of multi-UAV formation.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) formation technology, and in particular to a method and system for maintaining UAV formation based on the Attn-MADDPG algorithm. Background Technology

[0002] In recent years, with the continuous advancement of sensor, wireless communication, and intelligent control technologies, unmanned aerial vehicle (UAV) technology has developed rapidly. Due to its low cost, flexible operation, and lack of scene constraints, UAVs have been widely used in numerous fields. Compared to a single UAV, multiple UAVs can form a formation to execute missions, effectively compensating for the performance deficiencies of a single aircraft and increasing the likelihood of success in complex combat missions.

[0003] When multiple UAV swarms perform battlefield support, reconnaissance, and coordinated attacks, they need to maintain stability and coordination in complex environments. For example, in battlefield support missions, rapid switching between multiple targets by UAVs can reduce the stability of the swarm; during reconnaissance or attack missions, UAVs need to avoid collisions and plan suitable flight routes in complex terrain. Traditional path planning algorithms struggle to cope with sudden obstacles or changes in target position in dynamic environments, which places higher demands on the autonomy and intelligence of multi-UAV swarms.

[0004] Reinforcement learning is an experience-based machine learning method that learns optimal decisions through continuous interaction and trial and error. It helps drones make the best decisions in complex environments to accomplish predetermined mission objectives. Deep reinforcement learning, which combines reinforcement learning with deep neural networks, has outstanding capabilities and can efficiently handle high-dimensional state spaces composed of complex factors such as the positions of multiple drones.

[0005] Currently, research in UAV formation control mainly focuses on two directions: maintaining formation among multiple UAVs and path planning for multi-agent systems. Regarding UAV formation path planning algorithms, domestic and international scholars have largely focused on improving classical path planning algorithms, such as achieving path planning under heterogeneous constraints by incorporating the RRT fast search strategy; and developing... A collaborative architecture coupling algorithms with multi-objective potential fields addresses issues such as configuration maintenance and member collision avoidance. These classic algorithms demonstrate good architectural compatibility and ease of implementation in engineering practice, but their inherent limitations remain significant: computational complexity increases exponentially with the expansion of task dimensions, and they are highly dependent on global environment awareness. In dynamic adversarial scenarios, there is often an irreconcilable contradiction between real-time performance and optimality guarantees, which directly affects the effectiveness of formation control in complex task scenarios. Summary of the Invention

[0006] Based on the technical problems existing in the background technology, this invention proposes a method and system for maintaining UAV formation based on the Attn-MADDPG algorithm, which realizes path reliability planning for multi-UAV formations.

[0007] The UAV formation maintenance method based on the Attn-MADDPG algorithm proposed in this invention includes:

[0008] With the goal of achieving the fastest arrival speed, the shortest travel path, and meeting collision-free safety requirements throughout the entire process under a set cost constraint, a kinematic model of the drones and a formation target optimization model are constructed. A Markov decision process is constructed for navigation path planning, which is used for the navigator UAV to autonomously judge and adjust speed control. A deep deterministic policy gradient algorithm model based on an attention mechanism is set up and trained. The model includes a centralized evaluation network, multiple distributed execution networks, and a replay experience pool. It learns decision policies through centralized training, while execution is distributed, and an attention mechanism is incorporated into the evaluation network. The trained model is used to perform multi-UAV system formation and formation maintenance.

[0009] Furthermore, the kinematic model of the UAV is as follows: ; in, , These represent the drone body. x direction and y linear velocity in the direction, For velocity vectors, For heading angle, For the speed of drones axis, The components of the axis.

[0010] Furthermore, in discrete time, the state update formula for the UAV is: ; in, For action networks at time The given output action, Sampling time, For drones at all times Location, For drones at all times Location, For a moment The velocity vector, For a moment The heading angle.

[0011] Furthermore, the objective function of the formation target optimization model is as follows: ; in, Represented as a loss function, represent The formation at any moment Indicates the desired formation. This is the strategy adopted by drones. This is the initial state of the entire system. For task duration, This represents the optimal strategy that minimizes the objective function. It expresses expectation.

[0012] Furthermore, the Markov decision process in constructing the navigation path planning is specifically as follows: The navigator's leadership strategy is set by utilizing the relative positional and heading errors between the followers and the navigator. Utilize the leader's action space to set up the follower's continuous action space; A distributed reward function is set up as an evaluation metric for drone strategies.

[0013] Furthermore, the distributed reward function as follows: ; in, The reward functions are for different segments. For the set error requirements, These are the weighting coefficients for the system center error and the corresponding point error, respectively. , To meet the error requirements , The fixed reward value obtained, These respectively indicate their importance in the reward. For drone indexing, The total number of drones, This is a termination condition to prevent the risk of collisions caused by adjacent drones being too close together.

[0014] Furthermore, the number of replay experience pool groups is the same as the number of followers; Each replay experience pool group includes a short-term experience pool and a long-term experience pool. The short-term experience pool stores experience samples from the most recently set rounds, while the long-term experience pool stores experience from all rounds since the formation began.

[0015] Furthermore, each time experience is stored, it is first stored in the short-term experience pool. When the short-term experience pool is full, the earliest stored data is transferred to the long-term experience pool. Experience data in the short-term experience pool is periodically transferred to the long-term experience pool.

[0016] Furthermore, the centralized evaluation network is shared by all followers, and the evaluation network includes an embedding layer, a Transformer encoder, and a multilayer perceptron connected in sequence. Both the embedding layer and the multilayer perceptron are composed of forward fully connected layers.

[0017] A computer system includes a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the method described above.

[0018] The advantages of the UAV formation-keeping method and system based on the Attn-MADDPG algorithm provided by this invention are as follows: Deep modeling of the UAV formation system and path targets is performed, and mathematical models of the state space and action space of the leader UAV are designed. Furthermore, to accelerate the UAV learning process and improve overall learning efficiency, a distributed training reward function is designed, aiming to construct a more continuous and smooth reward distribution. This provides an important mathematical model foundation for the development of Attn-MADDPG-based control algorithms. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 Diagram showing formation maintenance; Figure 3 This is a schematic diagram of the basic framework of the Attn-MADDPG algorithm; Figure 4 To evaluate the network diagram; Figure 5 This is a schematic diagram of an action network; Figure 6 Here is a diagram of the GRU architecture; Figure 7 This is a schematic diagram of the average reward function; Figure 8 This is a schematic diagram of the velocity variation curve (Attn-MADDPG); Figure 9 This is a schematic diagram of the velocity variation curve (MADDPG); Figure 10 This is a schematic diagram of the velocity variation curve (DDQN); Figure 11 This is a schematic diagram of the speed variation curve (PID). Detailed Implementation

[0020] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0021] like Figures 1 to 11 As shown, the UAV formation-keeping method based on the Attn-MADDPG algorithm proposed in this invention includes: Step 1: With the goal of achieving the fastest arrival speed, following the shortest travel path, and meeting collision-free safety requirements throughout the entire process, while setting cost constraints, construct the kinematic model of the drones and the formation target optimization model. Step 2: Construct a Markov decision process for navigation path planning, which is used by the navigator drone to autonomously judge and adjust speed control; Step 3: Set up and train a deep deterministic policy gradient algorithm model based on an attention mechanism. The model includes a centralized evaluation network, multiple distributed execution networks, and a replay experience pool. It learns decision policies through centralized training, while execution is distributed. An attention mechanism is incorporated into the evaluation network. Step 4: Use the trained model to perform multi-UAV system formation and formation maintenance.

[0022] This embodiment performs in-depth modeling of the UAV formation system and path targets, and designs mathematical models of the state space and action space for the navigator UAV. Furthermore, to accelerate the UAV learning process and improve overall learning efficiency, a distributed training reward function is designed, aiming to construct a more continuous and smooth reward distribution. This provides an important mathematical model foundation for developing the Attn-MADDPG-based control algorithm.

[0023] Based on this embodiment, through deep mathematical modeling of the UAV formation system and path targets, and related mathematical models, an Attn-MADDPG control algorithm based on a multi-head self-attention mechanism is designed. Compared with existing formation algorithms, it effectively solves the convergence difficulty of networks in complex high-dimensional spaces, significantly improves policy search efficiency and formation cooperation performance, and ensures that UAVs can complete tasks efficiently and safely in complex environments.

[0024] In one embodiment, step one involves constructing a kinematic model of the drones and a formation target optimization model, with the goal of achieving the fastest arrival speed, following the shortest travel path, and meeting collision-free safety requirements throughout the entire process, under a set cost constraint. Specifically: This embodiment is based on the "Virtual Leader-Follower" (VLF) control framework. The core objective of the entire planning task is to ensure that, for a randomly set target location on the map, the drone formation can complete the mission with the fastest arrival speed, along the shortest travel path, and in compliance with collision-free safety requirements throughout the entire process, while meeting acceptable cost constraints. To this end, an objective function is constructed to mathematically describe the problem.

[0025] (a1) System modeling; To enable the UAV system to follow the leader's speed in real time while maintaining the required formation, a kinematic model and a formation target optimization model for the UAV were designed. The kinematic model of the UAV is represented as follows: ,(1); in, , These represent the drone body. x direction and y linear velocity in the direction, Let the velocity vector of the drone be... For heading angle, For the velocity vector in axis, The axis components, the position of the drone are .

[0026] By controlling the speed of the drone To control its movement, therefore at every moment Action vector Therefore, the state update of the UAV in discrete time is as follows: (2); in, For action networks at time The given output action, Sampling time, For drones at all times Location, For drones at all times Location, For a moment The velocity vector, For a moment The heading angle.

[0027] (a2) Formation target optimization model; According to the virtual navigator-follower framework, the leader is only responsible for path planning strategy, and the leader's strategy is defined as follows: Given a randomly selected target location on a map, the goal is for the drone to reach it safely and without collisions, using the shortest path and fastest speed, within an acceptable cost range. This optimization problem can be expressed by the following formula: (3); in, Represented as a loss function, represent The formation at any moment Indicates the desired formation. This is the strategy adopted by drones. This is the initial state of the entire system. For task duration, This represents the optimal strategy that minimizes the objective function. It expresses expectation.

[0028] This embodiment is... and The restrictions are , , For the speed of drones Minimum and maximum values ​​of axis components, For the speed of drones Minimum and maximum values ​​of the axis components.

[0029] In one embodiment, step two, constructing a Markov decision process for navigation path planning, is used for the navigator UAV to autonomously determine and adjust speed control, specifically as follows: Given a randomly selected objective point on a map, the navigator drone is required to autonomously determine and adjust its speed control to ensure successful mission completion. To this end, a navigator leadership strategy, a follower continuous action space, and a distributed reward function are designed to construct a more continuous and smooth reward distribution, thereby accelerating the drone's learning process and improving its overall learning efficiency.

[0030] (b1) Use the relative positional error and relative heading error between the follower and the navigator to set the navigator leadership strategy; Assuming that in the global coordinate system, some states of the virtual leader are... The follower's partial states are ,in These correspond to the positional status of the leader and followers, respectively. and Given the heading angles corresponding to the leader and followers respectively, the positional deviation of the follower relative to the leader is... for: (4); The relative heading error between the follower and the leader is In addition to the relative information mentioned above, followers also need the trend of the navigator to achieve real-time tracking, so the speed and heading of the navigator are also required.

[0031] (b2) Utilize the leader's action space to set up the follower's continuous action space; In many fields, such as reinforcement learning, the design of high-dimensional action spaces exponentially increases the convergence difficulty of algorithms, which undoubtedly greatly hinders the efficient progress of tasks. Therefore, carefully designing a concise and effective action control strategy is a key prerequisite for ensuring the efficient completion of tasks. Designing the leader's action space... As shown below: (5); (b3) Set up a distributed reward function as an evaluation metric for the drone strategy; The reward function, as the sole rule and evaluation metric for the UAV strategy, directly influences the choice of action strategy during network training, which directly determines whether the UAV system can learn the optimal strategy. Therefore, the reward function is set as a piecewise function. The distributed function design is shown in equation (6): ; in, These respectively indicate their importance in the reward. For drone indexing, The total number of drones, To terminate the process and prevent the risk of collisions due to insufficient distance between adjacent drones, The reward functions are for different segments. For the set error requirements, These are the weighting coefficients for the system center error and the corresponding point error, respectively. Adjusting their values ​​can change... and The importance of the reward.

[0032] In one embodiment, step three involves setting up and training a deep deterministic policy gradient algorithm model based on an attention mechanism. The model includes a centralized evaluation network, multiple distributed execution networks, and a replay experience pool. It learns decision policies through centralized training, while execution is distributed. An attention mechanism is incorporated into the evaluation network, specifically: Based on the Markov decision process constructed in step two, this embodiment proposes a novel method for multi-UAV collaborative decision-making: the Improved Multi-Agent Deep Deterministic Policy Gradient Algorithm (Attn-MADDPG) based on an attention mechanism. The algorithm comprises a centralized evaluation network, multiple distributed execution networks, and a corresponding number of replay experience pools (i.e., the number of replay experience pools is consistent with the number of followers). It learns decision policies through centralized training, while execution is distributed. Furthermore, an attention mechanism is incorporated into its evaluation network to accelerate operation while improving policy learning capabilities.

[0033] (c1) The principle of the control algorithm design based on Attn-MADDPG; The Deep Deterministic Policy Gradient (MADDPG) algorithm, proposed by researchers at OpenAI, is designed to address the complexities of multi-agent systems, particularly in fully cooperative environments. MADDPG utilizes two main networks in each agent: a judge network and an action network. Additionally, each network has a corresponding target network. The judge network evaluates the value of the current policy, aiming to minimize the mean squared error between the predicted Q-value and the target Q-value.

[0034] (c2) Double replay queue; Cooperative decision-making in multi-UAV formation maintenance is characterized by long timeframes, high complexity, and sparse environmental rewards, making it difficult for agents to obtain sufficient rewards in the early stages of training to drive the acquisition of excellent policies, resulting in poor policy convergence. Therefore, this embodiment designs a dual experience replay pool to improve the sample efficiency and policy stability of reinforcement learning. The replay experience pool consists of two pools: a short-term experience pool and a long-term experience pool. The short-term experience pool stores experience samples from the most recently set rounds to capture the latest changes in the current policy, while the long-term experience pool stores experience from all rounds since the formation began to maintain the memory of earlier policies.

[0035] Each time an experience is stored, it is first stored in the short-term experience pool. When the short-term experience pool is full, it is then transferred to the long-term experience pool. Data in the short-term experience pool is periodically transferred to the long-term experience pool to ensure the diversity of samples in the long-term experience pool.

[0036] (c3) Network structure design; The Transformer is a model entirely based on an attention mechanism. This embodiment combines the Transformer model with a centralized evaluation network, which is shared by all follower drones. In terms of network structure, a Transformer-encoder structure is used, such as... Figure 4As shown, the judging network consists of an embedding layer, a Transformer-encoder structure, and a multilayer perceptron. Both the embedding layer and the multilayer perceptron are composed of feedforward fully connected layers.

[0037] Action networks such as Figure 5 As shown. It is responsible for calculating the current action of each drone. The action network consists of a perception layer, a decision layer, and an action encoding layer. Each UAV sends its state and previous actions to the action network and receives the actions for the current situation. The decision layer includes a GRU network and two multilayer perceptrons. This indicates that the decision-making level receives the current state. And output the action , can be represented as The GRU module is shown in Figure 6.

[0038] In one embodiment, simulation experiments and analysis are conducted. (d1) Parameter settings; Within the experimental map, the virtual leader and three followers are given fixed initial positions and velocities. The positions of the three followers form an equilateral triangle, as shown below. Figure 2 As shown, the virtual navigator is positioned at the center of an equilateral triangle. The total experiment duration is set to... Sampling time is To simulate the randomness in complex environments, a random function is used with noise perturbation at intervals of sampling time. Assign a speed component to the virtual navigator, and set its size range to [value missing]. , The follower's speed is determined by a deep reinforcement learning algorithm, with its range set to match the leader's speed range. Using the same experimental parameters, the Attn-MADDPG algorithm is compared with the original MADDPG algorithm, the DDQN algorithm, and the PID algorithm. The algorithm parameter settings are shown in Table 1. The reward function parameter settings are shown in Table 2.

[0039] Table 1 Algorithm Parameters

[0040] Where Steps is the number of steps. Sampling time, As a discount factor, Here, the coefficients are the learning rate, the iteration step size is the learning rate, and the episodes are the interpolation values. For the total empirical sample, For the experience sample drawn from the short-term experience pool, The experience sample is drawn from the long-term experience pool. and Let be the sample size in the short-term experience pool and the sample size in the long-term experience pool, respectively.

[0041] Table 2 Reward Function Parameters

[0042] in, and These represent the minimum and maximum distances between adjacent drones.

[0043] (d2) Analysis of training results; This embodiment employs an experimental setup of 10,000 training epochs. The performance evolution trend of the strategy is quantified using a sliding window averaging method (window size = 100 epochs), and the curve is smoothed. To verify the effectiveness of the algorithm improvement, a comparative experimental framework is designed, including the proposed method and three benchmark algorithms. All baseline algorithms use a consistent basic network architecture, and optimization modules such as experience replay are strictly disabled to ensure fairness in the comparison. The experimental process and results are as follows: Figures 7 to 11 As shown.

[0044] The MADDPG algorithm has a slow learning speed in the early stages of training, such as... Figure 7 As shown, the reward value gradually increases in the first 20 rounds, but fluctuates significantly. After 50 rounds, the reward value continues to rise, eventually stabilizing at around 4000, lower than MAPPO and Attn-MADDPG. Considering the performance of both MAPPO and DDQN algorithms, the Attn-MADDPG algorithm not only has the fastest convergence speed but also a higher convergence reward value, demonstrating its excellent learning ability and performance.

[0045] (d3) Analysis of simulation results; First, the core of maintaining multi-UAV formation lies in dynamic speed coordination consistency, which needs to meet two constraints: (1) relative speed synchronization between adjacent UAVs; (2) global formation speed matching with the navigator reference speed. To quantitatively evaluate the performance of the coordination strategy, different algorithms were compared by collecting speed time-series data during formation maneuvers. The simulation results of each algorithm are shown below: Figures 8 to 11 The graphs show the speed tracking curves for each algorithm, with the first graph for each algorithm showing its speed at [specific speed range]. The second image shows the algorithm's velocity variation along the axial direction. Change in velocity along the axial direction. Figure 8 In the Attn-MADDPG algorithm, the speeds of each follower are close to the speed of the virtual navigator, indicating that it has learned the speed strategy of the virtual navigator very well. Figure 9In the original MADDPG algorithm, although the speed of each follower can track the speed trend of the virtual navigator, the curve fluctuates greatly, indicating that the speed error is large and the algorithm accuracy is not high. Figure 10 In this context, the DDQN algorithm has a discrete action space, with velocity actions set to (-1, -0.9…0.8, 0.9, 1). (The rest of the text appears to be a continuation of the previous sentence and can be left as is.) Figure 10 It can be seen that the follower's speed change trend under the DDQN algorithm is consistent with that of the virtual navigator, and the fluctuation is smaller than that of the original MADDPG. Its discretized action space results in lower precision in speed changes, thus causing slightly larger fluctuations compared to the Attn-MADDPG algorithm. Figure 11 In the PID algorithm, the speed change trends of the three followers are similar, but the followers fail to track the speed of the virtual navigator, and the speed fluctuations are too large, resulting in severe overshoot. Considering the above performance, the Attn-MADDPG algorithm shows the best performance in terms of accuracy and learning efficiency compared to the other three algorithms when the navigator exhibits complex speed changes.

[0046] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A UAV formation-keeping method based on the Attn-MADDPG algorithm, characterized in that, include: With the goal of achieving the fastest arrival speed, the shortest travel path, and meeting collision-free safety requirements throughout the entire process under a set cost constraint, a kinematic model of the drones and a formation target optimization model are constructed. A Markov decision process is constructed for navigation path planning, which is used for the navigator UAV to autonomously judge and adjust speed control. A deep deterministic policy gradient algorithm model based on an attention mechanism is set up and trained. The model includes a centralized evaluation network, multiple distributed execution networks, and a replay experience pool. It learns decision policies through centralized training, while execution is distributed, and an attention mechanism is incorporated into the evaluation network. The trained model is used to perform multi-UAV system formation and formation maintenance.

2. The method according to claim 1, characterized in that, The kinematic model of the UAV is as follows: ； in, , These represent the drone body. x direction and y linear velocity in the direction, For velocity vector, For heading angle, For the speed of drones axis, The components of the axis.

3. The method according to claim 2, characterized in that, In discrete time, the state update formula for the UAV is: ； in, For action networks at time The given output action, Sampling time, For drones at all times Location, For drones at all times Location, For a moment The velocity vector, For a moment The heading angle.

4. The method according to claim 1, characterized in that, The objective function of the formation target optimization model is as follows: ； in, Represented as a loss function, represent The formation at any moment Indicates the desired formation. This is the strategy adopted by drones. This is the initial state of the entire system. For task duration, This represents the optimal strategy that minimizes the objective function. It expresses expectation.

5. The method according to claim 1, characterized in that, The Markov decision process in constructing the navigation path planning is specifically as follows: The navigator's leadership strategy is set by utilizing the relative positional and heading errors between the followers and the navigator. Utilize the leader's action space to set up the follower's continuous action space; A distributed reward function is set up as an evaluation metric for drone strategies.

6. The method according to claim 5, characterized in that, The distributed reward function as follows: ； in, The reward functions are for different segments. For the set error requirements, These are the weighting coefficients for the system center error and the corresponding point error, respectively. , To meet the error requirements , The fixed reward value obtained, These respectively indicate their importance in the reward. For drone indexing, The total number of drones, This is a termination condition to prevent the risk of collisions caused by adjacent drones being too close together.

7. The method according to claim 1, characterized in that, The number of replay experience pool groups is the same as the number of followers; Each replay experience pool group includes a short-term experience pool and a long-term experience pool. The short-term experience pool stores experience samples from the most recently set rounds, while the long-term experience pool stores experience from all rounds since the formation began.

8. The method according to claim 7, characterized in that, Each time experience is stored, it is first stored in the short-term experience pool. When the short-term experience pool is full, the earliest stored data is transferred to the long-term experience pool. Experience data in the short-term experience pool is periodically transferred to the long-term experience pool.

9. The method according to claim 1, characterized in that, The centralized evaluation network is shared by all followers. The evaluation network includes an embedding layer, a Transformer encoder, and a multilayer perceptron connected in sequence. Both the embedding layer and the multilayer perceptron are composed of forward fully connected layers.

10. A computer device / equipment / system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-9.

Images