Autonomous obstacle avoidance and path optimization control method for unmanned aerial vehicle in complex environment

Abstract

The invention discloses an autonomous obstacle avoidance and path optimization control method for an unmanned aerial vehicle in a complex environment. The method comprises the following steps: A, establishing a simulation environment containing static obstacles and dynamic obstacles; b, obtaining a group of path schemes approaching an optimal solution and a plurality of candidate paths based on a particle swarm algorithm and the initial position of the unmanned aerial vehicle; c, based on a multi-agent depth deterministic strategy gradient algorithm, constructing a state space, an action space and a reward function used for training an obstacle avoidance motion strategy of multiple unmanned aerial vehicles; d, a multi-agent depth deterministic strategy gradient algorithm is used for training, and a path planning strategy is optimized; and E, evaluating scores of all candidate paths through a reinforcement learning mechanism, and calculating to obtain an unmanned aerial vehicle flight path with the highest comprehensive score as an optimal path. According to the invention, a safe and efficient flight path can be quickly and accurately planned for the unmanned aerial vehicle in a complex dynamic scene, and efficient obstacle avoidance of the unmanned aerial vehicle in the complex dynamic scene is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of UAV path planning, and particularly to an autonomous obstacle avoidance and path optimization control method for UAVs in complex environments. Background Art

[0002] With the continuous progress and wide application of UAV technology, the research on complex obstacle avoidance algorithms has become an important topic in the UAV field. Traditional obstacle avoidance algorithms are mainly designed for simple static obstacles, such as walls, furniture, etc. These algorithms usually rely on predefined rules or geometric models to plan the flight path of the UAV by avoiding known obstacles. However, in the real world, the environment faced by UAVs is often complex and changeable, and the shape, position, and quantity of obstacles are often variable and complex, which may include moving people or objects, irregular terrains, and buildings, etc.

[0003] Currently, existing multi-UAV motion planning methods generally adopt reinforcement learning algorithms. For example, the technical solution disclosed in the patent application with the application number 202411766682.7 and the patent name "Multi-UAV Target Exploration and Tracking Method and System Based on Multi-Agent Ant Colony Reinforcement Learning": The actual environment where the UAV is located is divided into several grid cells, and the target states of these grid cells are initialized. At the same time, the initial observation position and state of the UAV are set; the exploration pheromone and tracking pheromone in the ant colony algorithm are set, and the secretion state of the pheromone is updated according to the path of the UAV. Based on the pheromone concentration, the transfer probability of the UAV between grid cells is calculated; the reward function of reinforcement learning is set, including exploration reward, collision reward, tracking reward, and pursuit reward. The multi-UAVs are centrally trained using the MAPPO algorithm, and combined with the existing exploration information and tracking information, an optimal joint decision is generated.

[0004] Although the existing path planning technologies based on reinforcement learning can achieve the exploration and tracking of targets in unknown areas, these methods usually rely on pre-set fixed routes and are difficult to flexibly respond to the changes in dynamic environments. Especially in high-density aircraft and complex environment scenarios, this static planning method cannot effectively adapt to the real-time changing airspace management requirements, resulting in low utilization rate of airspace resources. In addition, the lack of real-time response ability to dynamic environments also increases the risk of flight conflicts, limiting the overall efficiency and safety of the multi-UAV system. Therefore, we use a path planning method based on the particle swarm algorithm and the multi-agent deep deterministic policy gradient algorithm. This method has the capabilities of environmental perception and decision-making, can effectively cope with environmental uncertainties and dynamic changes, and also has the global path planning ability. In addition, this algorithm has high real-time performance and computational efficiency, which can ensure that the UAV quickly responds to environmental changes and avoids obstacles during flight, thereby improving the overall performance and safety of the system. Summary of the Invention

[0005] The object of the present invention is to provide a method for autonomous obstacle avoidance and path optimization control of an unmanned aerial vehicle (UAV) in a complex environment, which can quickly and accurately plan a safe and efficient flight path for the UAV in a complex dynamic scene, and realize efficient obstacle avoidance of the UAV in the complex dynamic scene.

[0006] The present invention adopts the following technical solutions:

[0007] A method for autonomous obstacle avoidance and path optimization control of an unmanned aerial vehicle in a complex environment, comprising the following steps:

[0008] A: Establish a simulation environment including static obstacles and dynamic obstacles, and initialize the motion state of the UAV and the position information of the target point; subsequently, obtain the state data of the UAV in real time and dynamically update the trajectory and obstacle avoidance actions of the UAV;

[0009] B: Based on the particle swarm optimization algorithm and the initial position of the UAV, by calculating the individual optimum and the global optimum of the particles, obtain a set of path schemes approaching the optimum solution and several candidate paths;

[0010] C: Based on the multi-agent deep deterministic policy gradient algorithm, define the state space, action space and reward function for training the obstacle avoidance motion strategy of multiple UAVs;

[0011] D: Use the multi-agent deep deterministic policy gradient algorithm for training to optimize the path planning strategy;

[0012] E: Evaluate the scores of all candidate paths through a reinforcement learning mechanism, finally converge to the optimal strategy, and calculate the UAV flight path with the highest comprehensive score as the optimal path.

[0013] Step A includes the following steps:

[0014] A1: Use the simulation platforms AirSim and Pygame to establish a simulation environment, model the static obstacles and dynamic obstacles, obtain the position information, direction and planned path data of the UAV in the form of three-dimensional coordinates in real time, and construct a three-dimensional plane visualization interface;

[0015] A2: Initialize the position information of the initial position and the target point of the UAV in the simulation environment, and set the collision distance;

[0016] A3: Establish a motion model of the UAV in the simulation environment by using the position, speed, motion direction, target angle and angular velocity;

[0017] A4: Establish a motion model of the obstacle, and use Pygame to synchronously update the UAV trajectory and obstacle avoidance actions in real time.

[0018] Step B includes the following steps:

[0019] B1: Initialize based on the particle swarm algorithm, randomly generating the initial positions and velocities of different particles;

[0020] B2: Update the positions and velocities of the particles according to the inertia weight and learning factor;

[0021] B3: Update the individual optimal solutions and the global optimal solution of the particles, finally obtaining a set of path schemes approaching the optimal solution and several candidate paths.

[0022] Step C includes the following steps:

[0023] C1: Model the state space of the multi - UAV system in the multi - agent deep deterministic policy gradient algorithm; the state of each UAV includes position, velocity, direction, distance to the target point, and the spatial distribution information of obstacles around the UAV;

[0024] C2: Model the action space of the multi - UAV system in the multi - agent deep deterministic policy gradient algorithm; the actions of each UAV include angular velocity and linear velocity;

[0025] C3: Model the reward function and penalty function in the multi - agent deep deterministic policy gradient algorithm.

[0026] The reward function includes: giving a reward when the distance between the UAV and the target point is less than the target area radius, and giving a reward when the UAV moves towards the target point; the penalty function includes: giving a penalty when the distance between the UAV and the obstacle is less than the safety distance, and giving a penalty when the UAV moves away from the target point.

[0027] In step D, use the multi - agent deep deterministic policy gradient algorithm for training, adopting a training paradigm of centralized training and decentralized execution, storing the interaction records of the UAVs through a distributed experience pool, using the Actor network to make action decisions based on local observations, and the Critic network to perform value evaluation using global state - action information, jointly optimizing the path planning strategy.

[0028] Step D includes the following steps:

[0029] D1: Store the trajectory data of the interaction between each agent and the environment through a distributed experience replay pool;

[0030] D2: Equip each UAV with an Actor network for generating actions according to the current state and a Critic network for evaluating the quality of the current policy;

[0031] D3: Update the Actor network and the Critic network in the multi - agent deep deterministic policy gradient algorithm to optimize the path planning strategy.

[0032] In step E, the multi-agent deep deterministic policy gradient algorithm is iteratively trained. Through the reinforcement learning mechanism, the scores of all candidate paths are evaluated, the learning parameters of the Actor-Critic network are dynamically adjusted, and an iterative loop for policy evaluation and improvement is carried out, finally converging to the optimal policy, and the UAV flight path with the highest comprehensive score is calculated and obtained.

[0033] Step E includes the following steps:

[0034] E1: Configure the total number of training iterations and the maximum duration of each iteration;

[0035] E2: In each training iteration, update the current time step, and then perform parameter optimization of the Actor-Critic network;

[0036] E3: Evaluate the matching degree between the current state and the preset termination condition, and decide whether to trigger the initialization process of a new training cycle;

[0037] E4: When the number of training iterations is reached, terminate the training, save the weight parameters of the Actor-Critic network and the optimal path trajectory, and select the path with the highest comprehensive score as the optimal path.

[0038] In step A, the UAV and the obstacles are both modeled as spheres, and the target position is modeled as a spherical area.

[0039] The present invention adopts the reinforcement learning technology. By constructing a simulation environment on the AirSim and Pygame platforms, the particle swarm optimization algorithm is used for path planning to generate multiple high-quality candidate paths, and the multi-agent deep deterministic policy gradient algorithm is used to optimize the paths output by the particle swarm algorithm, so as to achieve efficient obstacle avoidance of the UAV in complex dynamic scenarios and improve the task success rate. Brief Description of the Drawings

[0040] Figure 1 is the flow diagram of the present invention;

[0041] Figure 2 is the flow chart of the particle swarm optimization algorithm in the present invention;

[0042] Figure 3 is the structural schematic diagram of the multi-agent deep deterministic policy gradient algorithm in the present invention. Detailed Embodiment

[0043] The present invention will be described in detail below with reference to the drawings and embodiments:

[0044] As Figures 1 to 3 shown, the method for autonomous obstacle avoidance and path optimization control of a UAV in a complex environment according to the present invention includes the following steps:

[0045] A: Use the simulation platforms AirSim and Pygame to establish a simulation environment. Model static and dynamic obstacles in the simulation environment, and at the same time initialize the motion state of the drone and the position information of the target point. Obtain the state data of the drone in real time through AirSim, and use Pygame to achieve three-dimensional visualization, dynamically updating the trajectory and obstacle avoidance actions of the drone.

[0046] In this embodiment, step A includes the following specific steps:

[0047] A1: Use the simulation platforms AirSim and Pygame to establish a simulation environment.

[0048] In the present invention, first, a virtual city environment is constructed based on the AirSim simulation platform to simulate the real city environment for testing the autonomous obstacle avoidance and path optimization algorithms of the unmanned helicopter in complex scenarios. The simulation environment mainly simulates static and dynamic obstacles. Static obstacles include buildings, billboards, street lights, and trees, etc., and dynamic obstacles simulate vehicles, pedestrians, etc. in the real scene. The above dynamic elements configure the movement trajectory and interaction behavior through the blueprint script of the Unreal Engine, so as to truly reproduce the dynamic changes in the complex environment and further increase the complexity of the simulation environment. In addition, use the API provided by AirSim to obtain the position information, direction, and planned path data of the drone in real time, and convert the above data into three-dimensional coordinate form. After collecting the position information of static and dynamic obstacles, the data is transferred to Pygame to construct a three-dimensional plane visualization interface.

[0049] A2: Initialize the initial position of the drone and the position information of the target point in the simulation environment.

[0050] In three-dimensional space, model the drone i as a spherical agent with a radius of r i and model the shape of the obstacle as a sphere with a radius of r o . To ensure that the drone does not collide with the obstacle, the present invention sets the collision distance D collision = r i + r o . When the distance between the drone and the obstacle is greater than D collision , it is considered that the drone has not collided. In addition, set the target position of the drone i, and the target position is a target spherical area with a radius of r g . When the distance D ig between the center position of the drone i and the center position of the target spherical area is = r i + r g , it is determined that the drone i has successfully reached its target position.

[0051] A3: Model the position, velocity, direction of motion, target angle, and angular velocity of the UAV in the simulation environment.

[0052] In three-dimensional space, the motion state of the UAV is described by state variables in multiple dimensions, including position, velocity, direction of motion, target angle, and angular velocity; the combined description of the above state variables can provide a complete motion model for the three-dimensional space navigation, obstacle avoidance, and path planning of the UAV.

[0053] Among them, the position of UAV i is represented by the three-dimensional coordinate P i = [x i , y i , z i T ; x i , y i , z i represent the positions of the UAV on the X, Y, and Z axes respectively;

[0054] The velocity of UAV i is represented by the three-dimensional vector ; represent the velocity components of the UAV in the X, Y, and Z axis directions respectively; the magnitude of the velocity of UAV i is

[0055] The direction of motion of UAV i is represented by the pitch angle φ i , yaw angle ψ i and roll angle θ i , and the direction is uniformly represented as Direction = [φ i , ψ i , θ i ;

[0056] The angular velocity of UAV i is represented by the vector ; represent the rotation rates around the velocity vector axis, horizontal axis, and vertical axis respectively.

[0057] A4: Model the motion model of the obstacle.

[0058] The motion model of the obstacle is the same as that of the UAV, and different obstacles are represented by Obstacle = [d1, d2, d3,..., d n .

[0059] Through the above steps, the construction of the simulation environment can be completed. During the flight of the UAV, the Pygame is used to synchronously refresh the screen, update the trajectory and obstacle avoidance actions in real time, synchronously present the dynamic changes of the UAV trajectory and real-time obstacle avoidance decisions, and provide intuitive visual support for the evaluation of the path planning effect.

[0060] B: Based on the particle swarm optimization algorithm and the initial position of the UAV, by calculating the individual optimum and global optimum of the particles, a set of path plans approaching the optimum solution and several candidate paths are obtained.

[0061] In the present invention, a set of path plans approaching the optimum solution is obtained based on the particle swarm optimization algorithm, mainly by updating the velocity of the particles, updating the positions of the particles, updating the individual optimum solutions and the group optimum solutions of the particles, and finally obtaining a set of path plans approaching the optimum solution and several candidate paths.

[0062] B1: Initialize based on the particle swarm optimization algorithm, and randomly generate the initial positions and velocities of different particles.

[0063] First, at the beginning of the PSO algorithm, a group of particles are randomly generated. These particles are distributed in the search space. The path of each particle is represented by the coordinates of a series of path points, P = [p1, p2, …, p n , where each path point contains the position coordinates p i = [x i , y i , z i T , and each particle is also randomly assigned an initial velocity for adjusting its position in subsequent iterations. The setting of the initial position and velocity of the particles determines the diversity of the search, thus affecting the global search ability of the algorithm.

[0064] B2: In the PSO algorithm, update the positions and velocities of the particles according to the inertia weight and learning factors.

[0065] The PSO algorithm finds the optimum solution through iteration. In each iteration, the particles track the individual optimum solution and the global optimum solution, and update the positions of the particles through the inertia weight and learning factors.

[0066] The movement of the particles in the search space is realized through the iterative update of their positions and velocities. The update of the velocity consists of three parts: the influence of the particle's velocity at the previous moment, the distance between the particle and its individual optimum position, and the distance between the particle and the global optimum position. The specific formula is as follows:

[0067] v i (t + 1) = w·v i (t) + c1·rand()·(pBest i - P i (t)) + c2·rand()·(gBest i

[0068] - P i (t))

[0069] ​Among them, w is the inertia weight, which controls the search range of the particle, and v i (t) represents the velocity of UAV i at time t, and P i (t) represents the position of UAV i at time t. c1 and c2 are learning factors, which respectively measure the influence of the particle's own experience pBest i and the global optimal experience gBest i . rand() is a random number, which increases the randomness of exploration.

[0070] The position update consists of the current position and the velocity of the particle at the next moment, and is carried out according to the following formula:

[0071] P i (t + 1) = P i (t) + v i (t + 1)

[0072] Among them, P i (t + 1) represents the position of UAV i at time t + 1, P i (t) represents the position of UAV i at time t, and v i (t + 1) represents the velocity of UAV i at time t + 1.

[0073] Through continuous iterative updates, the particles gradually converge to the optimal solution. In the initial stage of the algorithm, the inertia weight w is usually set to be larger to enhance the global search ability of the particles and help the particles explore a wider solution space; while in the later stage of the iteration, the inertia weight w gradually decreases to enhance the local search accuracy of the particles so that they can make fine adjustments near the optimal solution.

[0074] B3: In the PSO algorithm, update the individual optimal solution and the population optimal solution of the particles.

[0075] In the particle swarm algorithm, the search direction of the particles is guided by updating the individual optimal and global optimal of the particles.

[0076] In each iteration, calculate the fitness value (i.e., the objective function value) of the current particle, compare the current fitness value with the historical optimal fitness value of the particle. If the current fitness value is better than the historical optimal fitness value, then update the individual optimal position of the particle to the current position and update the individual optimal fitness value to the current fitness value. Specifically expressed as:

[0077]

[0078] Among them, P i (t) represents the position of UAV i at time t, f(P i (t)) is the fitness value of UAV i at time t, and pBest iis the individual optimal position of the UAV i, and f(pBest i ) is the individual optimal fitness value of the UAV.

[0079] The global optimal (gBest) is the optimal position among all particle individuals in the entire particle swarm. In each iteration, the individual optimal fitness values of all particles are compared. The individual optimal position with the optimal fitness value is selected as the global optimal position. Specifically, it is expressed as:

[0080]

[0081] Among them, pBest i is the individual optimal position of the UAV, and f(pBest i ) is the individual optimal fitness value of the UAV. represents finding the parameter at which the objective function f(pBest i ) obtains the minimum value, and gBest is the global optimal position.

[0082] When the PSO optimization process ends, the algorithm first screens out the global optimal particle with the optimal fitness value from the particle swarm, and at the same time retains the high-quality particle paths with the top N (such as Top 20) fitness values to construct a candidate path set, which is used to provide diverse inputs for further optimization of the multi-agent deep deterministic policy gradient algorithm.

[0083] C: Based on the multi-agent deep deterministic policy gradient algorithm, define the state space, action space, and reward function of the algorithm for training the obstacle avoidance motion strategy of multiple UAVs.

[0084] In the present invention, since the PSO algorithm calculates an approximate optimal solution, a multi-agent deep deterministic policy gradient (MADDPG) algorithm is further introduced for global optimal path screening. MADDPG is a reinforcement learning algorithm that mainly conducts deep learning and intelligent evaluation on the candidate path set generated by the PSO algorithm by constructing an agent interaction system including a state space, an action space, and a reward function.

[0085] C1: Model the state space in the MADDPG algorithm.

[0086] In the MADDPG algorithm, the state space is the core basis of path planning and is used to comprehensively describe the current state and environmental information of the UAV. For each UAV i, its state includes the position (p i = [x i , y i , z i ), speed (v i , ωi )、Direction (Direction = [φ i , ψ i , θ i ), the distance to the target point (D ig ) and the spatial distribution information of the obstacles around UAV i The state of each UAV i in the MADDPG algorithm is represented as s i = [p i , v i , ω i , Direction, D ig , Obstacle i , and the state space of the multi - UAV system is jointly represented by the states of all UAVs, that is, S = [s1, s2, …, s n ; The above representation method can accurately reflect the overall operation state of the system and the mutual influence between UAVs.

[0087] C2: Model the action space in the MADDPG algorithm.

[0088] The action space describes all possible behavior options of the UAV in each step, including operations such as speed adjustment and flight direction change. For the multi - UAV system, the action space is the combination of the actions of all UAVs, expressed as: A = [a1, a2, …, a n , where a i represents the action of UAV i, including the angular velocity v i and the linear velocity w i . This design enables the algorithm to select the optimal flight actions in different situations, thereby realizing the dynamic adjustment of path planning and adapting to the real - time changes in complex environments.

[0089] C3: In the MADDPG algorithm, model the reward function and the penalty function.

[0090] The reward function is the optimization basis for the UAV path planning, used to guide its behavior towards the target. The reward function mainly includes three parts: giving rewards when approaching the target point, giving penalties when approaching obstacles or colliding, and reducing the path length to optimize the overall performance. By designing a joint reward function, it is possible to effectively balance the independent goals of each UAV and the collaborative efficiency of the overall system, thereby improving the effect of path planning and the task completion rate.

[0091] If the distance between the UAV and the obstacle is less than the safety distance, a penalty is given to reduce the score of the path, and no points are deducted in other cases. The collision penalty score is specifically expressed as follows:

[0092]

[0093] Among them, represents the minimum distance between the UAV i and all obstacles, D collision represents the collision distance. When the distance between the UAV and the obstacle is less than D collision , it is considered that the UAV has a collision.

[0094] If the distance between the UAV and the target point is less than the target area radius, a reward is given to increase the score of the path, and no points are added in other cases. The reward score for reaching the target Specifically, it is expressed as follows:

[0095]

[0096] Among them, and represent the central position of the target spherical area, x i , y i , z i represents the central position of the UAV i, represents the distance between the UAV i and the target point, D ig represents the distance between the UAV and the target point, that is, the distance between the UAV i and the central position of the target spherical area. When the distance between the UAV and the obstacle is less than D ig , it is considered that the UAV i has successfully reached its target position.

[0097] In the MADDPG algorithm, a dynamic reward function based on distance change is used to guide the UAV to move towards the target point. Specifically, when the UAV approaches the target point, the system gives a positive reward according to the approaching speed. If the UAV deviates from the target point and the distance increases, a negative penalty is imposed to avoid ineffective exploration. The reward for a single-step movement Specifically, it is expressed as follows:

[0098]

[0099] Among them, represents the position with the target point at time t - 1, represents the position with the target point at time t, and η is the reward coefficient used to adjust the weight of the reward for a single-step movement.

[0100] The joint reward function R is composed of the rewards of all UAVs and is specifically expressed as follows: R = [r1, r2, …, r n . Among them, the reward function of the UAV i is expressed as

[0101] D: It is trained using the multi-agent deep deterministic policy gradient algorithm, adopting a training paradigm of centralized training and decentralized execution. The interaction records of the drones are stored in a distributed experience pool. The Actor network makes action decisions based on local observations, and the Critic network uses global state-action information for value evaluation to jointly optimize the path planning strategy.

[0102] In the centralized training stage, we adopt a paradigm that combines centralized training and decentralized execution. The trajectory data of the interaction between each agent and the environment (including state observations, joint actions, immediate rewards, and transition states) is stored in a distributed experience replay pool. Among them, the Actor network of each agent makes distributed decisions based on local observation information, while the Critic network uses global state-action information for centralized value evaluation. Among them, the parameter update of the Actor-Critic network structure adopts the target network mechanism and the policy gradient method to ensure the stability of the training process.

[0103] Step D includes the following specific steps:

[0104] D1: The trajectory data of the interaction between each agent and the environment is stored in a distributed experience replay pool.

[0105] In the MADDPG algorithm, the experience pool is used to store the interaction records of multiple drones in the environment, denoted as (S, A, R, S ′ '); S represents the joint state space of the multi-drone system, A represents the joint action space of the multi-drone system, R represents the joint reward function, and S ′ represents the next moment's joint state after executing action A.

[0106] To improve the training efficiency, a prioritized experience replay mechanism is adopted, that is, different sampling probabilities are assigned according to the importance of the samples. The prioritized experience replay mechanism samples according to the TD-error priority of the samples, preferentially selecting samples that are more valuable for training. The sampling probability M(i) of drone i is calculated as follows:

[0107]

[0108] Among them, represents the priority metric value of the current drone, δ i is the TD-error, which reflects the deviation of the current action value estimation. ∈ is a small constant (e.g., 10 -5 ), to avoid zero priority. α is a parameter to adjust the priority degree, represents the normalized sum of the priority metric values of all drones. High-priority experience samples will be assigned higher sampling probabilities, thus accelerating the algorithm's learning of key scenarios and improving the training effect.

[0109] D2: Equip each drone with an Actor network and a Critic network.

[0110] In the MADDPG algorithm, each drone is equipped with an Actor network and a Critic network. The Actor network is responsible for generating actions based on the current state, while the Critic network is used to evaluate the quality of the current policy and help the agent optimize its behavior.

[0111] The Actor is responsible for generating actions based on the current state of the drone. The input is the state s i of the drone, including position, speed, direction, distance to the target point, and distribution information of surrounding obstacles. The output is the action a i = [v i , ω i . To ensure that the action values are within a reasonable range, the Sigmoid function is used to limit the linear velocity v i within the range of [0, 1], and at the same time, the Tanh function is used to limit the angular velocity ω i within the range of [-1, 1].

[0112] The Critic is used to evaluate the quality of the current policy. The input is the joint state S and joint action A of all drones, and the output is the action value Q i (S, A), which represents the expected return of executing the joint state S and joint action A in the current joint state. By receiving the state and action information of all drones, the Critic network can comprehensively evaluate the complex interaction relationships between multiple agents, thereby providing a more accurate action value evaluation for each agent.

[0113] D3: Update the Actor network and the Critic network in the MADDPG algorithm.

[0114] During the training process, for the Critic network, the value function estimation is optimized by minimizing the Temporal Difference Error. For the Actor network, based on the value evaluation provided by the Critic network, the policy parameters are adjusted along the direction of increasing the expected return through the Policy Gradient Ascent method.

[0115] The Critic network updates its parameters by minimizing the Temporal Difference (TD) error to more accurately evaluate the action value. The TD error represents the difference between the current action value Q i (S, A) and the target value y i . The loss function L Critic of the Critic network is defined as:

[0116]

[0117] Among them, represents the expected value, which is used to calculate the average value of the squared error over all samples, Q i (S, A) represents the current state and action, and the target value y i is composed of the current reward R i and the future discounted reward, and the calculation formula is:

[0118] y i = R i + γQ i (S ′ , A ′ )

[0119] Among them, γ is the discount factor, which is used to balance the importance of the current reward and the future reward. R i represents the current reward, and Q i (S ′ , A ′ ) represents the state and action at the next moment. By minimizing the TD error, the Critic network can gradually approximate the true action value function, thereby more accurately evaluating the quality of the policy.

[0120] The Actor network optimizes its parameters by the gradient ascent method to maximize the target value Q i (S, A). Specifically, the optimization objective of the Actor network is to adjust the policy parameter θ i such that the action value Q i (S, A) evaluated by the Critic network is maximized. The update formula of the Actor network is specifically expressed as:

[0121]

[0122] Among them, represents the expected value, which is used to calculate the average value of the squared error over all samples, represents the gradient of the action value function with respect to the action, μ i (s i ) is the deterministic policy output by the Actor network, represents the gradient of the policy function with respect to the parameter. By this way of gradient ascent, the Actor network can gradually optimize the policy, generate better actions, and thus improve the path planning effect of the drone.

[0123] E: Iteratively train the MADDPG algorithm, evaluate the scores of all candidate paths through the reinforcement learning mechanism, dynamically adjust the learning parameters of the Actor-Critic network, and perform an iterative loop for policy evaluation and improvement, finally converging to the optimal policy, thereby calculating the UAV flight path with the highest comprehensive score.

[0124] By iteratively training the network and updating the learning parameters, finally calculate the UAV path with the highest score.

[0125] The specific steps are as follows:

[0126] E1: Configure the total number of training iterations and the maximum duration of each iteration.

[0127] First, set the total number of training epochs E and the maximum training time T for each epoch. The total number of epochs E determines the overall number of iterations for training, and the maximum training time T for each epoch limits the duration of a single training session. At the same time, initialize the training parameters, including the epoch counter (e = 0) and the time counter (t = 0), to prepare for the subsequent training process.

[0128] E2: In each training epoch, update the current time step, and then perform parameter optimization of the Actor-Critic network.

[0129] After each training, update the training time (t = t + Δt), where Δt is the time step. Then, judge t. If (t < T), continue to train all UAVs in the current epoch, enter step D3, and continue the training process; if (t ≥ T), the current epoch ends, enter step E3, and prepare to start the training of the next epoch.

[0130] E3: Evaluate the matching degree between the current state and the preset termination condition, and decide whether to trigger the initialization process of a new training cycle.

[0131] After the current epoch ends, update the epoch counter (e = e + 1), and judge e: If (e < E), enter the training of the next epoch, return to step D1, re-initialize the environment and start a new round of training; if (e ≥ E), the training ends, enter step E4, and complete the entire training process.

[0132] E4: Abort the MADDPG training when the number of training epochs is reached, and save the weight parameters of the Actor-Critic network and the optimal path trajectory.

[0133] When the number of training rounds reaches E, terminate the training process of the target neural network, and save the parameter data of the current Actor network and Critic network. These parameters will be used in the subsequent test and execution phases. According to the score results recorded during the training process, select the path with the highest score as the optimal path. The path with the highest score usually has the shortest path length and the highest task completion rate.

[0134] So far, using the method provided in this embodiment, the UAV has completed efficient obstacle avoidance in a complex environment and reached the destination. The overall algorithm flow of the algorithm is as Figure 1 shown, demonstrating the complete process from environment construction to path planning and optimization; the specific implementation process of the particle swarm optimization algorithm is as Figure 2 shown, detailing the steps of particle position update, velocity adjustment, and dynamic optimization of the inertia weight; the structure of the multi-agent deep deterministic policy gradient algorithm is as Figure 3 shown, demonstrating the application of the Actor-Critic network architecture, joint reward function design, and prioritized experience replay mechanism.

[0135] This embodiment adopts reinforcement learning technology. By constructing a high-fidelity environment on the AirSim and Pygame platforms, combining the particle swarm optimization algorithm to generate multiple high-quality candidate paths, and using the multi-agent deep deterministic policy gradient algorithm to collaboratively optimize the paths, ultimately achieving efficient obstacle avoidance of the UAV in complex dynamic scenarios and improving the task success rate. The UAV path planning method provided in this embodiment brings new breakthroughs to the field of path planning through reinforcement learning, and at the same time provides a more reliable solution for social and technological development.

[0136] It should be noted that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. An autonomous obstacle avoidance and path optimization control method for drones in complex environments, characterized in that, It includes the following steps: A: Establish a simulation environment including static and dynamic obstacles, and initialize the motion state of the UAV and the position information of the target point; subsequently, obtain the state data of the UAV in real time and dynamically update the UAV's trajectory and obstacle avoidance actions; B: Based on the particle swarm optimization algorithm and the initial position of the UAV, calculate the individual optimal and global optimal of the particles to obtain a set of path plans approaching the optimal solution and several candidate paths; C: Based on the multi-agent deep deterministic policy gradient algorithm, construct the state space, action space, and reward function for training the obstacle avoidance motion strategy of multiple UAVs; D: Use the multi-agent deep deterministic policy gradient algorithm for training to optimize the path planning strategy; E: Evaluate the scores of all candidate paths through a reinforcement learning mechanism, finally converge to the optimal strategy, and calculate the UAV flight path with the highest comprehensive score as the optimal path.

2. The method for autonomous obstacle avoidance and path optimization control of an unmanned aerial vehicle in a complex environment according to claim 1, wherein Step A includes the following steps: A1: Use the simulation platforms AirSim and Pygame to establish a simulation environment, model static and dynamic obstacles, obtain the position information, orientation, and planned path data of the UAV in the form of three-dimensional coordinates in real time, and construct a three-dimensional plane visualization interface; A2: Initialize the position information of the initial position of the UAV and the target point in the simulation environment, and set the collision distance; A3: Build a motion model of the UAV in the simulation environment using position, velocity, motion direction, target angle, and angular velocity; A4: Model the motion model of the obstacle, and use Pygame to synchronously update the UAV trajectory and obstacle avoidance actions in real time.

3. The autonomous obstacle avoidance and path optimization control method for drones in complex environments according to claim 1, wherein, Step B includes the following steps: B1: Initialize based on the particle swarm optimization algorithm, and randomly generate the initial positions and velocities of different particles; B2: Update the positions and velocities of the particles according to the inertia weight and learning factors; B3: Update the individual optimal solutions and population optimal solutions of the particles, and finally obtain a set of path plans approaching the optimal solution and several candidate paths.

4. The autonomous obstacle avoidance and path optimization control method for drones in complex environments according to claim 1, wherein, Step C includes the following steps: C1: Model the state space of the multi-UAV system in the multi-agent deep deterministic policy gradient algorithm; the state of each UAV includes position, velocity, orientation, distance to the target point, and the spatial distribution information of obstacles around the UAV; C2: Model the action space of the multi-UAV system in the multi-agent deep deterministic policy gradient algorithm; the actions of each UAV include angular velocity and linear velocity; C3: Model the reward function and penalty function in the multi-agent deep deterministic policy gradient algorithm.

5. The autonomous obstacle avoidance and path optimization control method for drones in complex environments according to claim 1, characterized in that: The reward function includes: giving a reward when the distance between the UAV and the target point is less than the target area radius, and giving a reward when the UAV moves towards the target point; the penalty function includes: giving a penalty when the distance between the UAV and the obstacle is less than the safety distance, and giving a penalty when the UAV moves away from the target point.

6. The autonomous obstacle avoidance and path optimization control method for drones in complex environments according to claim 1, characterized in that: In step D, use the multi-agent deep deterministic policy gradient algorithm for training, adopt a training paradigm of centralized training and decentralized execution, store the interaction records of the UAVs through a distributed experience pool, use the Actor network to make action decisions based on local observations, and the Critic network to perform value evaluation using global state-action information, and jointly optimize the path planning strategy.

7. The method for autonomous obstacle avoidance and path optimization control of an unmanned aerial vehicle in a complex environment according to claim 6, characterized in that, Step D includes the following steps: D1: Store the trajectory data of each agent interacting with the environment through a distributed experience replay pool; D2: Equip each drone with an Actor network for generating actions based on the current state and a Critic network for evaluating the quality of the current policy; D3: Update the Actor network and the Critic network in the multi-agent deep deterministic policy gradient algorithm to optimize the path planning strategy.

8. The autonomous obstacle avoidance and path optimization control method for drones in complex environments according to claim 6, characterized in that: In step E, iteratively train the multi-agent deep deterministic policy gradient algorithm, evaluate the scores of all candidate paths through the reinforcement learning mechanism, dynamically adjust the learning parameters of the Actor-Critic network, perform an iterative loop of policy evaluation and improvement, and finally converge to the optimal policy to calculate the drone flight path with the highest comprehensive score.

9. The autonomous obstacle avoidance and path optimization control method for drones in complex environments according to claim 8, characterized in that Step E includes the following steps: E1: Configure the total number of training iterations and the maximum duration of each iteration; E2: In each training iteration, update the current time step and then perform parameter optimization of the Actor-Critic network; E3: Evaluate the matching degree between the current state and the preset termination condition and decide whether to trigger the initialization process of a new training cycle; E4: Abort the training when the number of training iterations is reached, save the weight parameters of the Actor-Critic network and the optimal path trajectory, and select the path with the highest comprehensive score as the optimal path.

10. The autonomous obstacle avoidance and path optimization control method for an unmanned aerial vehicle in a complex environment according to claim 1, wherein: In step A, the drones and obstacles are both modeled as spheres, and the target position is modeled as a spherical region.

Images