Deep reinforcement learning multi-unmanned aerial vehicle path planning method fusing artificial potential field

Abstract

The invention belongs to the technical field of unmanned aerial vehicle path planning, and particularly relates to a deep reinforcement learning multi-unmanned aerial vehicle path planning method fused with an artificial potential field, which comprises the following steps: S1, constructing a three-dimensional simulation environment containing static and dynamic obstacles, and constructing a system model for performing path planning on an unmanned aerial vehicle (UAV) in an unknown environment, and constructing an optimization problem of UAV path planning based on the constructed system model, and aiming at minimizing task completion time and avoiding obstacles. According to the method, the artificial potential field resultant force is embedded into the state space of the TD3 algorithm, and the multi-dimensional reward function is designed, so that the problems of low training efficiency and insufficient obstacle avoidance capability of traditional deep reinforcement learning are solved. Experiments show that the task success rate of the method in a dynamic obstacle scene is improved by 10%-30%, the collision rate is reduced by 20%-50%, a smooth collision-free trajectory can be generated in real time, and the method is suitable for collaborative navigation and emergency rescue tasks of a multi-unmanned aerial vehicle cluster.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) path planning technology, specifically to a deep reinforcement learning method for multi-UAV path planning that integrates artificial potential fields. Background Technology

[0002] Unmanned aerial vehicle (UAV) technology is widely used in military reconnaissance, logistics delivery, and disaster relief due to its low cost, high maneuverability, and adaptability. The core objective of multi-UAV cooperative path planning is to generate collision-free trajectories while satisfying both task and physical constraints. Traditional methods such as the Artificial Potential Field (APF), A algorithm, and Rapid Expanding Random Tree (RRT) show certain advantages in static environments, but have significant limitations when facing dynamic obstacles: APF is prone to getting trapped in local minima, causing UAVs to oscillate or stagnate near obstacles; A and RRT require frequent replanning, resulting in high computational complexity and difficulty in meeting real-time requirements.

[0003] Deep reinforcement learning (DRL) provides a new approach to path planning. Algorithms such as DQN, DDPG, and TD3 learn the optimal strategy autonomously through environmental interaction. However, they still face three major problems in multi-UAV scenarios: low training efficiency, requiring a large number of samples to converge; sparse rewards, with non-zero rewards only obtained when reaching the target or colliding, resulting in low exploration efficiency; and insufficient safety guarantees, as pure data-driven approaches cannot guarantee a 100% obstacle avoidance success rate. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To address the shortcomings of existing technologies, this invention provides a deep reinforcement learning-based multi-UAV path planning method that integrates artificial potential fields, thus solving the problems mentioned in the background section.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0008] A deep reinforcement learning-based multi-UAV path planning method incorporating artificial potential fields includes the following steps:

[0009] S1. Construct a 3D simulation environment containing static and dynamic obstacles, build a system model for UAV path planning in an unknown environment, and construct an optimization problem for UAV path planning based on the constructed system model, with the goal of minimizing task completion time and avoiding obstacles. The system model includes the scene setting target point and UAV physical constraints for UAV path planning in an unknown environment, including maximum speed and turning radius.

[0010] S2, Based on the system model constructed in step S1, the optimization problem is modeled as a Markov decision process, where the UAV is regarded as an intelligent agent that makes decisions based on the current state and adjusts its actions based on feedback rewards;

[0011] S3. Based on the system model constructed in step S1 and the Markov decision process constructed in step S2, design an improved APF-TD3 path planning framework based on the TD3 algorithm. The specific improvements include:

[0012] A virtual potential force calculated by artificial potential field (APF) is embedded into the state space of the TD3 algorithm to form an extended state space;

[0013] Design a multi-dimensional reward function, including transition reward, collision penalty, target reward, step size penalty, and artificial potential field guided reward;

[0014] Optimize the network structure by combining APF potential field information with the TD3 Actor-Critic network to achieve policy optimization;

[0015] S4 will use the APF-TD3 algorithm proposed in this chapter to test in environments 1 and 2. In these two environments, the positions of the drones, obstacles, and target points will be randomly initialized to test the generalization of the algorithm. Training was conducted in two typical environments: Environment 1, a fixed configuration scenario with 3 drones and 12 obstacles, and Environment 2, a dynamic complex scenario with 5 drones and 11 obstacles. During the testing process, 2000 scenarios were randomly generated for each environment, divided into a baseline scenario with 3 drones and 12 obstacles and a complex scenario with 5 drones and 11 obstacles. Each scenario contained 1000 independent test cases, for a total of 4000 tests.

[0016] Furthermore, a three-dimensional simulation environment containing static and dynamic obstacles is constructed, and target points and physical constraints of the UAV are set, including maximum speed and turning radius;

[0017] (1) Environmental dimensions: 200m×200m×200m, including static obstacles of 10m×10m×100m and dynamic obstacles of random cylinders;

[0018] (2) Target point: a circular platform with a diameter of 2m, the safe distance of the UAV is set to 2m, and the detection range of the lidar is 15m;

[0019] (3) UAV constraints: maximum speed 20m / s, action space is three-dimensional speed and yaw angle, range [-1.5m / s, 1.5m / s] and [-90°, 90°];

[0020] The goal of completing a ensemble task is to get multiple drones to converge at the same target point in the shortest possible time. Therefore, the training objective of the APF-TD3 model in this section is to minimize T_{sum}, where the number of drones is set to n∈[0...N]. The total time T_{sum} can be expressed as the formula:

[0021]

[0022] Furthermore, based on the system model constructed in step S1, the optimization problem is modeled as a Markov decision process, where the UAV is regarded as an agent that makes decisions based on the current state and adjusts its actions based on feedback rewards. The Markov decision process (MDP) can be defined as a five-element (S, A, P, R, γ), where S is the state space, A is the action space, P is the policy, R is the reward function, and γ is the reward discount factor, and γ∈[0,1].

[0023] Furthermore, based on the system model constructed in step S1 and the Markov decision process constructed in step S2, an improved APF-TD3 path planning framework based on the TD3 algorithm is designed. The specific improvements include:

[0024] Traditional state space includes the drone's position s pos Speed ​​s vel Obstacle position s obs Target location s goal Based on this, the resultantforce vector calculated by APF is added, which includes the resultant force of target attraction, obstacle repulsion, and attraction between UAVs, as shown in the following formula:

[0025] S = [s pos ,s vel ,s obs ,s goal ,F APF ]

[0026] Where F APF =F att +F rep +F a2 ,F att For the target gravity, F rep F is the repulsive force of the obstacle. a2 The attractive force between drones; the formula for calculating the target's gravity is:

[0027]

[0028] U, where ξ is the gravity coefficient in the equation, and q represents the current position of the AUV. g Indicates the position of the target point, d(q,q) gThe distance between the current position of the drone and the target position is represented by ; the gravitational field function U is taken. att The negative gradient of (q) is used to obtain gravity, which is then expressed as F. att (q) indicates;

[0029]

[0030] The formula includes the repulsive field coefficient η and the obstacle position q. obs d(q,q) obs The distance between the drone's current position and the obstacle, and the maximum impact distance (minimum safe distance) of the obstacle on the drone; force F rep (q) is the force field function U rep The negative gradient of (q);

[0031]

[0032] The designated target location is 1. There are multiple obstacles within the navigation area. Therefore, the total potential field is the sum of the gravitational field and the repulsive field, i.e.:

[0033]

[0034] The resultant force of attraction and repulsion is represented by F(q), that is:

[0035]

[0036] As can be seen from the above formula, traditional APF only considers the target position and obstacle position in the environment during the planning process; however, it also considers the attractive potential field generated between UAVs, denoted as U. a2 This is used to make drones gather together with surrounding drones;

[0037]

[0038] The action space A, as the output of the TD3 algorithm, represents the interaction between the agent and the environment. In the UAV's control strategy, the main control variables are the velocities in different directions and the UAV's yaw angle, used to control the UAV's horizontal navigation and pitch motion. Therefore, the action space A is defined as A = [u, v, w, θ, ψ]. Based on the actual performance of the UAV's motion, the magnitude of each action in the action space is subject to certain limitations, as follows:

[0039]

[0040] A policy P is based on the current state and describes the conditional probability distribution of choosing an action given the current state. Such a policy is called a stochastic policy. This policy allows different actions to be taken in the same state and assigns a probability to each possible action.

[0041] P = π(a|s) = p(a|s)

[0042] The reward discount factor γ is used to calculate the expected cumulative reward in the future. If the discount factor is set to 1, it means that each future moment is considered as important as the current moment. If the discount factor is 1, only the immediate reward is considered. Introducing the discount factor can effectively control the impact of future rewards on the current state decision, which helps to balance the consideration of long-term and short-term rewards, enabling the agent to make better decisions and learn. Here, the discount factor γ is set to 0.95.

[0043] A reward is the feedback provided by the environment after an agent takes an action. This feedback is typically a scalar function used to evaluate the quality of the action or the result of taking a specific action in a given environment. The reward depends on the current state, the action performed, and the state at the next time step. Accumulating the reward over time steps introduces the concept of a trajectory; the sum of all rewards along the trajectory is called the reward function.

[0044] R = 0.01 * r trans -4*r collision +5*r tgt +r step +R APF

[0045] Where r trans As a transitional reward, it reflects changes in the distance between the drone and the target; the calculation method is as follows:

[0046]

[0047] r collision This is a collision penalty, 1 for collisions and 0 otherwise; r tgt The reward is 1 if the goal is achieved, and 0 otherwise; r step =-0.01 is the step size penalty to avoid cyclic oscillation; R APF =cos(d agent_APF The reward is guided by the artificial potential field and is calculated based on the cosine value of the closest point on the path between the UAV and the APF. agent_APF The distance between the agent's current position and the path of the artificial potential field is calculated as follows:

[0048]

[0049] Among them, (x APF y APF () represents the path coordinates of the artificial potential field;

[0050] The execution flow of the APF-TD3 algorithm includes:

[0051] Initialize TD3 network parameters and APF potential field parameters, including the gravitational coefficient ξ and repulsive coefficient η; collect UAV state and environmental information in real time, calculate the APF resultant force, and construct an extended state space; generate control actions through the Actor network, including three-dimensional velocity and yaw angle, limited to [-1.5m / s, 1.5m / s] and [-90°, 90°]; after executing the action, obtain the new state and reward, and store them in the experience replay buffer; optimize the Actor / Critic network parameters based on the TD3 dual-delay update mechanism, with an update cycle of 10 steps; network parameters are: batch size 128, maximum step size 500 / episode, experience replay pool capacity 100,000, soft update factor τ = 0.01, discount factor γ = 0.95, learning rate 0.001, and Adam optimizer.

[0052] Furthermore, the proposed APF-TD3 algorithm was tested in environments 1 and 2. In both environments, the positions of the drones, obstacles, and target points were randomly initialized to test the generalization ability of the proposed algorithm. Training was conducted in two typical environments: Environment 1, a fixed configuration scenario with 3 drones and 12 obstacles, and Environment 2, a dynamic complex scenario with 5 drones and 11 obstacles. During testing, 2000 scenarios were randomly generated for each environment, divided into a baseline scenario with 3 drones and 12 obstacles and a complex scenario with 5 drones and 11 obstacles. Each scenario contained 1000 independent test cases, for a total of 4000 tests. Path planning was performed using the configured environments to generate collision-free smooth trajectories.

[0053] (III) Beneficial Effects

[0054] Compared with existing technologies, this invention provides a deep reinforcement learning-based multi-UAV path planning method that integrates artificial potential fields, which has the following beneficial effects:

[0055] Improved planning accuracy: In the 3-drone + 12-obstacle scenario, the mission success rate increased from 82.14% in TD3 to 89.57%, and the collision rate decreased from 5.85% to 4.62%; in the 5-drone + 11-dynamic-obstacle scenario, the success rate increased from 65.28% to 76.92%, and the local minimum trap problem was solved.

[0056] Training efficiency optimization: Embedding APF prior knowledge reduces training time by 25.7% (from 120.48s to 96.36s) and the number of training iterations by 15%, meeting the needs of real-time decision-making.

[0057] Trajectory quality optimization: The average path length was shortened by 16.7% (3 drone scenarios) and 12.3% (5 drone scenarios), the trajectory smoothness (standard deviation of curvature) was improved by 30%, and unnecessary detours were reduced.

[0058] Enhanced generalization ability: When the dynamic obstacle density increases by 40%, the success rate fluctuates by only ±3.2%, which is significantly better than TD3's ±8.5%, making it suitable for complex disaster relief scenarios. Attached Figure Description

[0059] Figure 1 A flowchart of a deep reinforcement learning method for multi-UAV path planning that integrates artificial potential fields;

[0060] Figure 2 A diagram of the APF-TD3 algorithm framework;

[0061] Figure 3 A rendering of the two-dimensional path planning for three drones;

[0062] Figure 4 A rendering of the two-dimensional path planning for five drones;

[0063] Figure 5 A rendering of the 3D path planning for three drones;

[0064] Figure 6 A rendering of the 3D path planning for five drones;

[0065] Figure 7 This chart compares the test data of the APF-TD3 algorithm with the DDPG and TD3 algorithms in terms of success rate, collision rate, average reward, and running time. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Example

[0068] like Figure 1-7 As shown in the figure, an embodiment of the present invention proposes a deep reinforcement learning multi-UAV path planning method that integrates artificial potential fields. The specific steps are as follows:

[0069] S1. Construct a 3D simulation environment containing static and dynamic obstacles, build a system model for UAV path planning in an unknown environment, and construct an optimization problem for UAV path planning based on the constructed system model, with the goal of minimizing task completion time and avoiding obstacles. The system model includes the scene setting target point and UAV physical constraints for UAV path planning in an unknown environment, including maximum speed and turning radius.

[0070] (1) Environmental dimensions: 200m×200m×200m, including static obstacles (10m×10m×100m) and dynamic obstacles (random cylinders);

[0071] (2) Target point: a circular platform with a diameter of 2m, the safe distance of the UAV is set to 2m, and the detection range of the lidar is 15m;

[0072] (3) UAV constraints: maximum speed 20m / s, action space is three-dimensional speed and yaw angle, range [-1.5m / s, 1.5m / s] and [-90°, 90°].

[0073] The goal of completing a ensemble task is to get multiple drones to converge at the same target point in the shortest possible time. Therefore, the training objective of the APF-TD3 model in this section is to minimize T_{sum}. Let the number of drones be n∈[0...N], and the total time T_{sum} can be expressed as the formula:

[0074]

[0075] S2, Based on the system model constructed in step S1, the optimization problem is modeled as a Markov decision process, where the UAV is regarded as an intelligent agent that makes decisions based on the current state and adjusts its actions based on feedback rewards;

[0076] A Markov Decision Process (MDP) can be defined as a five-element system (S, A, P, R, γ), where S is the state space, A is the action space, P is the policy, R is the reward function, and γ is the reward discount factor, and γ∈[0,1].

[0077] S3: Based on the system model constructed in step S1 and the Markov decision process constructed in step S2, design a path planning framework based on the TD3 algorithm to construct an improved APF-TD3 path planning framework. Specific improvements include:

[0078] Traditional state space includes the drone's position s pos Speed ​​s vel Obstacle position s obs Target location s goalThis patent adds the resultantforce vector calculated by APF, which includes the resultant force of target attraction, obstacle repulsion, and attraction between UAVs, as shown in the following formula:

[0079] S = [s pos ,s vel ,s obs ,s goal ,F APF ]

[0080] Where F APF =F att +F rep +Fa2,F att For the target gravity, F rep F is the repulsive force of the obstacle. a2 For the attraction of drones;

[0081] The formula for calculating the target gravity is:

[0082]

[0083] U, where ξ is the gravity coefficient in the equation, and q represents the current position of the AUV. g Indicates the position of the target point, d(q,q) g This represents the distance between the drone's current position and the target position. The gravitational field function U is taken. att The negative gradient of (q) is used to obtain gravity, which is then expressed as F. att (q) represents.

[0084]

[0085] The formula includes the repulsive field coefficient η and the obstacle position q. obs d(q,q) obs The distance between the drone's current position and the obstacle, and the maximum impact distance (minimum safe distance) of the obstacle on the drone. Force F rep (q) is the force field function U rep The negative gradient of (q).

[0086]

[0087] The target location specified in this paper is 1. There are multiple obstacles within the navigation area. Therefore, the total potential field is the sum of the gravitational field and the repulsive field, that is:

[0088]

[0089] The resultant force of attraction and repulsion is represented by F(q), that is:

[0090]

[0091] As can be seen from the above formulas, traditional APF only considers the target and obstacle positions in the environment during the planning process. This paper considers the attractive potential field generated between UAVs, denoted as U. a2 It is used to make drones gather together with drones in the vicinity.

[0092]

[0093] The action space A, as the output of the TD3 algorithm, represents the interaction between the agent and the environment. In the UAV's control strategy, the main control variables are the velocities in different directions and the UAV's yaw angle, used to control the UAV's horizontal navigation and pitch motion. Therefore, the action space A is defined as A = [u, v, w, θ, ψ]. Based on the actual performance of UAV motion, the magnitude of each action in the action space is subject to certain limitations, as follows:

[0094]

[0095] A policy P describes the conditional probability distribution of choosing an action given the current state. Such a policy is called a stochastic policy, which allows different actions to be taken in the same state and assigns a probability to each possible action.

[0096] P = π(a|s) = p(a|s)

[0097] The reward discount factor γ is used to calculate the expected cumulative reward in the future. A discount factor of 1 indicates that each future moment is considered equally important as the current moment, while a discount factor of 1 considers only the immediate reward. Introducing a discount factor can effectively control the impact of future rewards on the current state decision, helping to balance the consideration of long-term and short-term rewards, enabling the agent to make better decisions and learn. Here, the discount factor γ is set to 0.95.

[0098] A reward is the feedback provided by the environment after an agent takes an action. This feedback is typically a scalar function, used to evaluate the quality of the action or the result of taking a specific action in a given environment. The reward depends on the current state, the action performed, and the state at the next time step. Accumulating the reward over time steps introduces the concept of a trajectory; the sum of all rewards along the trajectory is called the reward function.

[0099] R = 0.01 * r trans -4*r collision +5*r tgt +r step +R APF

[0100] Where r trans As a transitional reward, it reflects changes in the distance between the drone and the target; the calculation method is as follows:

[0101]

[0102] r collision This is a collision penalty, 1 for collisions and 0 otherwise; r tgt The reward is 1 if the goal is achieved, and 0 otherwise; r step =-0.01 is the step size penalty to avoid cyclic oscillation; R APF =cos(d agent_APF The reward is guided by the artificial potential field and is calculated based on the cosine value of the closest point on the path between the UAV and the APF. agent_APF The distance between the agent's current position and the path of the artificial potential field is calculated as follows:

[0103]

[0104] Among them, (x APF y APF ) represents the path coordinates of the artificial potential field.

[0105] The execution flow of the APF-TD3 algorithm includes:

[0106] Initialize TD3 network parameters and APF potential field parameters, including the attraction coefficient ξ and repulsion coefficient η; collect UAV state and environmental information in real time, calculate the APF resultant force, and construct an extended state space; generate control actions through the Actor network, including three-dimensional velocity and yaw angle, limited to [-1.5m / s, 1.5m / s] and [-90°, 90°]; after executing the action, obtain the new state and reward, and store them in the experience replay buffer; optimize the Actor / Critic network parameters based on the TD3 dual-delay update mechanism, with an update cycle of 10 steps. Network parameters are: batch size 128, maximum step size 500 / episode, experience replay pool capacity 100,000, soft update factor τ = 0.01, discount factor γ = 0.95, learning rate 0.001, and Adam optimizer.

[0107] S4: The APF-TD3 algorithm proposed in this chapter will be tested in environments 1 and 2. In both environments, the positions of the drones, obstacles, and target points will be randomly initialized to test the generalization ability of the algorithm. Training was performed in two typical environments: Environment 1, a fixed configuration scenario with 3 drones and 12 obstacles, and Environment 2, a dynamic complex scenario with 5 drones and 11 obstacles. During testing, 2000 scenarios were randomly generated for each environment, divided into a baseline scenario (3 drones and 12 obstacles) and a complex scenario (5 drones and 11 obstacles). Each scenario contained 1000 independent test cases, for a total of 4000 tests.

[0108] Performance metrics: Using visualized loss functions and evaluation metrics, the performance of APF-TD3 is compared with that of DDPG and TD3.

[0109] In a scenario with 3 drones and 12 obstacles, the APF-TD3 achieved a success rate of 89.57% and an average running time of 96.36 seconds, which is 7.43% better than the TD3.

[0110] In a scenario with 5 drones and 11 dynamic obstacles, the success rate was 76.92% and the collision rate was 18.42%, both of which were better than the comparison algorithms.

[0111] The model training environment for this invention is as follows: CPU: Intel(R) Xeon(R) Platinum 8362; GPU: RTX 3090; RAM: 24GB; Operating System: Ubuntu 20.04; PyTorch 2.0.0 deep learning framework and Python 3.8 programming language are used.

[0112] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep reinforcement learning-based multi-UAV path planning method incorporating artificial potential fields, characterized in that: Includes the following steps, S1. Construct a 3D simulation environment containing static and dynamic obstacles, build a system model for UAV path planning in an unknown environment, and construct an optimization problem for UAV path planning based on the constructed system model, with the goal of minimizing task completion time and avoiding obstacles. The system model includes the scene setting target point and UAV physical constraints for UAV path planning in an unknown environment, including maximum speed and turning radius. S2, Based on the system model constructed in step S1, the optimization problem is modeled as a Markov decision process, where the UAV is regarded as an intelligent agent that makes decisions based on the current state and adjusts its actions based on feedback rewards; S3. Based on the system model constructed in step S1 and the Markov decision process constructed in step S2, design an improved APF-TD3 path planning framework based on the TD3 algorithm. The specific improvements include: A virtual potential force calculated by artificial potential field (APF) is embedded into the state space of the TD3 algorithm to form an extended state space; Design a multi-dimensional reward function, including transition reward, collision penalty, target reward, step size penalty, and artificial potential field guided reward; Optimize the network structure by combining APF potential field information with the TD3 Actor-Critic network to achieve policy optimization; S4 will use the APF-TD3 algorithm proposed in this chapter to test in environments 1 and 2. In these two environments, the positions of the drones, obstacles, and target points will be randomly initialized to test the generalization of the algorithm. Training was conducted in two typical environments: Environment 1, a fixed configuration scenario with 3 drones and 12 obstacles, and Environment 2, a dynamic complex scenario with 5 drones and 11 obstacles. During the testing process, 2000 scenarios were randomly generated for each environment, divided into a baseline scenario with 3 drones and 12 obstacles and a complex scenario with 5 drones and 11 obstacles. Each scenario contained 1000 independent test cases, for a total of 4000 tests.

2. The deep reinforcement learning multi-UAV path planning method integrating artificial potential fields according to claim 1, characterized in that, Construct a 3D simulation environment that includes static and dynamic obstacles, and set target points and physical constraints for the UAV, including maximum speed and turning radius; (1) Environmental dimensions: 200m×200m×200m, including static obstacles of 10m×10m×100m and dynamic obstacles of random cylinders; (2) Target point: a circular platform with a diameter of 2m, the safe distance of the UAV is set to 2m, and the detection range of the lidar is 15m; (3) UAV constraints: maximum speed 20m / s, action space is three-dimensional speed and yaw angle, range [-1.5m / s, 1.5m / s] and [-90°, 90°]; The goal of completing a ensemble task is to get multiple drones to converge at the same target point in the shortest possible time. Therefore, the training objective of the APF-TD3 model in this section is to minimize T_{sum}, where the number of drones is set to n∈[0...N]. The total time T_{sum} can be expressed as the formula:

3. The deep reinforcement learning multi-UAV path planning method integrating artificial potential fields according to claim 2, characterized in that, Based on the system model constructed in step S1, the optimization problem is modeled as a Markov decision process, where the UAV is regarded as an intelligent agent that makes decisions based on the current state and adjusts its actions based on feedback rewards. A Markov Decision Process (MDP) can be defined as a five-element system (S, A, P, R, γ), where S is the state space, A is the action space, P is the policy, R is the reward function, and γ is the reward discount factor, and γ∈[0,1].

4. The deep reinforcement learning multi-UAV path planning method integrating artificial potential fields according to claim 3, characterized in that, Based on the system model constructed in step S1 and the Markov decision process constructed in step S2, an improved APF-TD3 path planning framework based on the TD3 algorithm is designed. The specific improvements include: Traditional state space includes the drone's position s pos Speed ​​s vel Obstacle position s obs Target location s goal Based on this, the resultantforce vector calculated by APF is added, which includes the resultant force of target attraction, obstacle repulsion, and attraction between UAVs, as shown in the following formula: S=[s pos ,s vel ,s obs ,s goal ,F APF ] Where F APF =F att +F rep +F a2 ,F att For the target gravity, F rep F is the repulsive force of the obstacle. a2 The attractive force between drones; the formula for calculating the target's gravity is: U, where ξ is the gravity coefficient in the equation, and q represents the current position of the AUV. g Indicates the position of the target point, d(q,q) g The distance between the current position of the drone and the target position is represented by ; the gravitational field function U is taken. att The negative gradient of (q) is used to obtain gravity, which is then expressed as F. att (q) indicates; The formula includes the repulsive field coefficient η and the obstacle position q. obs d(q,q) obs The distance between the drone's current position and the obstacle, and the maximum impact distance (minimum safe distance) of the obstacle on the drone; Force F rep (q) is the force field function U rep The negative gradient of (q); The designated target location is 1. There are multiple obstacles within the navigation area. Therefore, the total potential field is the sum of the gravitational field and the repulsive field, i.e.: The resultant force of attraction and repulsion is represented by F(q), that is: As can be seen from the above formula, traditional APF only considers the target position and obstacle position in the environment during the planning process; however, it also considers the attractive potential field generated between UAVs, denoted as U. a2 This is used to make drones gather together with surrounding drones; The action space A, as the output of the TD3 algorithm, represents the interaction between the agent and the environment. In the UAV's control strategy, the main control variables are the velocities in different directions and the UAV's yaw angle, used to control the UAV's horizontal navigation and pitch motion. Therefore, the action space A is defined as A = [u, v, w, θ, ψ]. Based on the actual performance of the UAV's motion, the magnitude of each action in the action space is subject to certain limitations, as follows: A policy P is based on the current state and describes the conditional probability distribution of choosing an action given the current state. Such a policy is called a stochastic policy. This policy allows different actions to be taken in the same state and assigns a probability to each possible action. P = π(a|s) = p(a|s) The reward discount factor γ is used to calculate the expected cumulative reward in the future. If the discount factor is set to 1, it means that each future moment is considered as important as the current moment. If the discount factor is 1, only the immediate reward is considered. Introducing the discount factor can effectively control the impact of future rewards on the current state decision, which helps to balance the consideration of long-term and short-term rewards, enabling the agent to make better decisions and learn. Here, the discount factor γ is set to 0.

95. A reward is the feedback provided by the environment after an agent takes an action. This feedback is typically a scalar function used to evaluate the quality of the action or the result of taking a specific action in a given environment. The reward depends on the current state, the action performed, and the state at the next time step. Accumulating the reward over time steps introduces the concept of a trajectory; the sum of all rewards along the trajectory is called the reward function. R=0.01*r trans -4*r collision +5*r tgt +r step +R APF Where r trans As a transitional reward, it reflects changes in the distance between the drone and the target; the calculation method is as follows: r collision This is a collision penalty, 1 for collisions and 0 otherwise; r tgt The reward is 1 if the goal is achieved, and 0 otherwise; r step =-0.01 is the step size penalty to avoid cyclic oscillation; R APF =cos(d agent_APF The reward is guided by the artificial potential field and is calculated based on the cosine value of the closest point on the path between the UAV and the APF. agent_APF The distance between the agent's current position and the path of the artificial potential field is calculated as follows: Among them, (x APF y APF () represents the path coordinates of the artificial potential field; The execution flow of the APF-TD3 algorithm includes: Initialize TD3 network parameters and APF potential field parameters, including the gravitational coefficient ξ and repulsive coefficient η; collect UAV state and environmental information in real time, calculate the APF resultant force, and construct an extended state space; generate control actions through the Actor network, including three-dimensional velocity and yaw angle, limited to [-1.5m / s, 1.5m / s] and [-90°, 90°]; after executing the action, obtain the new state and reward, and store them in the experience replay buffer; optimize the Actor / Critic network parameters based on the TD3 dual-delay update mechanism, with an update cycle of 10 steps; network parameters are: batch size 128, maximum step size 500 / episode, experience replay pool capacity 100,000, soft update factor τ = 0.01, discount factor γ = 0.95, learning rate 0.001, and Adam optimizer.

5. The deep reinforcement learning multi-UAV path planning method integrating artificial potential fields according to claim 2, characterized in that, The proposed APF-TD3 algorithm was tested in environments 1 and 2. In both environments, the positions of the drones, obstacles, and target points were randomly initialized to test the generalization ability of the proposed algorithm. Training was performed in two typical environments: Environment 1, a fixed configuration scenario with 3 drones and 12 obstacles, and Environment 2, a dynamic complex scenario with 5 drones and 11 obstacles. During testing, 2000 scenarios were randomly generated for each environment, divided into a baseline scenario with 3 drones and 12 obstacles and a complex scenario with 5 drones and 11 obstacles. Each scenario contained 1000 independent test cases, for a total of 4000 tests. Path planning was performed using the configured environments to generate collision-free smooth trajectories.

Images