Unmanned aerial vehicle flight path planning method based on hybrid WOA-CEM-DRL algorithm

Abstract

The invention discloses an unmanned aerial vehicle flight path planning method based on a hybrid WOA-CEM-DRL algorithm, and the method is based on a hybrid WOA (whale optimization algorithm), a cross entropy method (CEM) and deep reinforcement learning (DRL), and aims at solving the problems that an existing algorithm is slow in convergence speed and is liable to fall into a local optimal solution in a complex three-dimensional environment. The invention provides a three-stage dynamic coevolution framework. In the early stage of evolution, population diversity is enhanced through global disturbance of CEM; in the middle stage of evolution, exploration and development are balanced; and in the later stage of evolution, performing fine tuning on the elite solution by using gradient information of the DRL. According to the method, global search of WOA, efficient sampling of CEM and rapid convergence capability of DRL are fused, and the method is mainly used for accurately generating a safe and smooth optimal track for an unmanned aerial vehicle in a complex three-dimensional environment.

Description

Technical Field

[0001] This invention relates to the field of UAV trajectory planning technology, and in particular to a UAV trajectory planning method based on the hybrid WOA-CEM-DRL algorithm, for efficient and reliable UAV trajectory planning in complex environments. Background Technology

[0002] Unmanned Aerial Vehicles (UAVs) are increasingly used in reconnaissance, logistics, agriculture, and rescue, and their autonomous navigation and decision-making capabilities are crucial for completing complex missions. Path planning, as a core technology of autonomous navigation, aims to generate an optimal or near-optimal path from the starting point to the destination for the UAV while satisfying various complex constraints (such as obstacle avoidance, flight performance limitations, and no-fly zones). This problem is essentially a complex, high-dimensional, nonlinear optimization problem with multiple constraints, and is typically proven to be NP-hard (Non-deterministic Polynomial-hard).

[0003] Traditional path planning algorithms, such as graph search algorithms like A* and Dijkstra, as well as Extended Random Tree (RRT) and Artificial Potential Field (APF) methods, perform reasonably well in handling simple, low-dimensional environments. However, when faced with increasingly complex real-world operating environments, these traditional algorithms often reveal their limitations due to their massive computational demands and strong dependence on environmental models.

[0004] In recent years, with the development of artificial intelligence technology, DRL (Depth-Range Learning) has been introduced into the field of trajectory planning, demonstrating great potential. DRL algorithms, such as Deep Q-Networks (DQN), Deep Deterministic Policy Gradient (DDPG), and their improved versions (such as TD3 and SAC), can directly learn decision policies from high-dimensional sensor inputs. However, DRL algorithms generally face bottlenecks such as low sample efficiency, slow convergence speed, unstable training process, and susceptibility to getting trapped in local optima. In particular, policy gradient-based algorithms are sensitive to hyperparameters, while Q-value learning algorithms may suffer from overestimation of Q-values.

[0005] On the other hand, nature-inspired evolutionary algorithms (EAs) offer another approach to solving complex optimization problems. Metaheuristic algorithms, such as genetic algorithms (GAs) and particle swarm optimization (PSOs), maintain a population of candidate solutions for parallel search by simulating the collective intelligence behavior of the wild, thus possessing powerful global search capabilities. However, EAs typically have slow convergence speeds and fail to fully utilize gradient information during the optimization process for fine-grained local searches.

[0006] To combine the advantages of both, Evolutionary Deep Reinforcement Learning (EvoDRL) has emerged as an emerging hybrid intelligent approach. Its core idea is to leverage the global exploration capabilities of Evolutionary Exploration (EA) to compensate for the limitations of Depth-Range Learning (DRL), while simultaneously utilizing the gradient information of DRL to accelerate the convergence of EA. Woven Occultation (WOA), as an advanced metaheuristic algorithm, achieves a good balance between global exploration and local exploitation by simulating the humpback whale's unique "bubble web" predation strategy. Concentrated Evolutionary Learning (CEM), as a powerful distribution estimation algorithm, can efficiently guide the population to evolve towards high-reward regions through learning from elite samples.

[0007] Therefore, this invention aims to address the shortcomings of existing technologies by deeply integrating the unique search mechanism of WOA, the efficient sampling capability of CEM, and the fast local optimization capability of the DRL algorithm to design a hybrid trajectory planning algorithm with superior performance and faster convergence. Summary of the Invention

[0008] To address the issues of low DRL sample efficiency and weak EA local development capabilities, this invention proposes a UAV trajectory planning method based on a hybrid WOA-CEM-DRL algorithm. The purpose of this invention is to overcome the limitations of single algorithms by constructing a framework that deeply integrates WOA, CEM, and DRL at multiple levels, thereby achieving accurate and robust UAV trajectory planning in complex environments.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A UAV trajectory planning method based on a hybrid WOA-CEM-DRL algorithm is proposed. Its core technology involves constructing a multi-level, heterogeneous optimization system and dynamically controlling the optimization process through an adaptive three-stage evolutionary strategy. The method specifically includes the following steps: Step 1: Formulate the UAV trajectory planning problem as a constrained multi-objective optimization problem. Initialize a whale population of size M, where each individual represents a candidate trajectory. Simultaneously, initialize a DRL agent based on the Actor-Critic architecture.

[0010] Step 2: At the maximum number of iterations Within the process, iterative optimization is performed. Each iteration dynamically executes a collaborative update strategy for WOA, CEM, and DRL based on the current stage.

[0011] Step 2.1: Macro-Global Search. First, all individuals perform standard WOA updates, extensively exploring the entire solution space by simulating behaviors such as encirclement, spiral updates, and random search, thus maintaining the algorithm's global exploration capability.

[0012] Step 2.2: Adaptive Collaborative Update. Subsequently, based on the different stages of the current iteration, a specific CEM-DRL collaborative update strategy is initiated to dynamically balance exploration and utilization.

[0013] Step 2.2.1: Early Evolution (Exploration First): The goal of this stage is to maximize population diversity. Based on elite solutions, the probability distribution of high-reward regions is estimated using CEM and sampled to generate a new, more diverse population. Then, DRL is used to perform gradient boosting on poorly fit individuals in the new population, quickly improving inferior solutions and raising the lower bound of the average quality of the entire population.

[0014] Step 2.2.2: Mid-Evolutionary Phase (Exploration vs. Exploitation Balance): The goal of this phase is to achieve a balance between exploration and exploitation. First, the population is divided into elite populations. (Top 50%) and the general population (The last 50%). Using CEM to analyze elite populations. Perturbation injection is performed to introduce new high-quality solutions. Simultaneously, DRL is used to optimize the general population. Perform gradient optimization to accelerate its convergence towards the elite region.

[0015] Step 2.2.3: Late Evolutionary Stage (Utilization Priority): The core objective of this stage is to finely develop the optimal solution region. This involves using CEM on the general population. Small-scale updates are performed to maintain necessary population diversity. Simultaneously, DRL is used to monitor elite populations. Perform fine-tuning of the gradient to quickly approach the local optimum.

[0016] Step 3: Output the optimal trajectory. After iteration, the individual with the highest fitness in the population is selected. This is the optimal flight path that is ultimately planned.

[0017] Compared with the prior art, the beneficial effects of the present invention are reflected in: 1. Innovative Multi-Level Heterogeneous Fusion Framework: This invention does not simply combine multiple algorithms linearly, but constructs a multi-level heterogeneous optimization framework. This framework configures optimization operators at different scales: the WOA operator performs global search at the macro scale; the CEM algorithm estimates and guides population distribution at the meso scale; and the DRL agent utilizes gradient information for local fine-tuning at the micro scale. This hierarchical collaborative mechanism organically connects the three stages of global exploration, distribution guidance, and local utilization, solving the problems of functional redundancy and low collaborative efficiency in traditional hybrid algorithms.

[0018] 2. Adaptive Exploration-Utilization Control Strategy: This invention's unique three-stage evolutionary strategy is an adaptive control mechanism based on the optimization process. It dynamically and non-uniformly allocates limited computing resources to exploration and utilization tasks, focusing on global exploration in the early stages, balancing the two in the middle stages, and concentrating on local utilization in the later stages. This intelligent resource scheduling strategy effectively resolves the core contradiction of "exploration-utilization" in optimization algorithms.

[0019] 3. Superior Performance and Robustness: Due to its unique framework and strategy, this invention achieves breakthroughs in solution speed, accuracy, and stability. Compared to single algorithms or other hybrid methods, this method converges to higher-quality solutions faster and exhibits stronger robustness and adaptability to various complex 3D environments and task constraints. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the example description will be briefly introduced below.

[0021] Figure 1 This is a schematic diagram of the overall process of a UAV trajectory planning method based on the hybrid WOA-CEM-DRL algorithm proposed in this invention.

[0022] Figure 2 This is a flowchart illustrating the three-stage collaborative update strategy for pre-evolution, mid-evolution, and late-evolution stages proposed in this invention.

[0023] Figure 3 This is a schematic diagram of the trajectory planning results of the present invention in simulation environment 1.

[0024] Figure 4 This is a schematic diagram of the trajectory planning results of the present invention in simulation environment 2.

[0025] Figure 5 This is a schematic diagram of the trajectory planning results of the present invention in simulation environment 3. Detailed Implementation

[0026] The following will refer to Figures 1 to 5 This paper provides a complete and detailed description of the specific technical solution and implementation process of the UAV trajectory planning method based on the hybrid WOA-CEM-DRL algorithm proposed in this invention, including the following steps: Step 1, Planning Environment Modeling and System Initialization: This step involves mathematically defining and formalizing the physical environment, trajectory, optimization objective, and algorithmic agent.

[0027] Step 1.1, Track and Environment Representation: To verify the effectiveness of the algorithm, a 3D simulation environment was constructed. The environment includes... For ease of calculation, each obstacle is simplified into a sphere. An obstacle can be represented as ,in Let the coordinates be the center of the sphere. Let the radius be denoted as . The drone is considered as a point mass, and its radius is denoted as . The coordinates of the time are Drones and the first Distance of the obstacle Greater than its radius When this occurs, it is considered that no collision has occurred.

[0028] A three-dimensional track Depend on An ordered sequence of waypoints represents: in It is the first The spatial coordinates of each track point. Starting from, The endpoint.

[0029] Step 1.2, Mathematical Model of Track Cost: This invention constructs trajectory planning as a constrained multi-objective optimization problem, whose cost function... The fitness function, which aims to minimize the sum of path length and steering angle, is defined as follows: .

[0030] The constraints are: 1. Obstacle avoidance constraints: For all and Established.

[0031] 2. Flight range constraints: , , .

[0032] in, and These are the corresponding weighting coefficients. The path length cost is used to calculate the sum of the Euclidean norms of all segments of the track, and its calculation formula is: in, The total number of waypoints. and The coordinates of adjacent waypoints Let be the Euclidean distance between the two points.

[0033] As a smoothness penalty, the smoothness of the trajectory is quantified by accumulating the angle between adjacent velocity vectors, penalizing sharp turns. The calculation formula is as follows: in, The total number of waypoints. Represents connecting waypoints and The direction vector.

[0034] Step 1.3, DRL Agent Design: In order to integrate with the DRL framework, its interface for interacting with the environment needs to be precisely defined, including the state space, action space, and reward function.

[0035] State-space design: At any given moment, the state of the DRL agent Defined as comprising three key components: the Euclidean distance between the drone and the target point. Normalized direction vector of the UAV pointing to the target point The current velocity vector of the drone .

[0036] This state-space design can effectively capture the relative relationship between the UAV and the target, as well as its own motion state, providing sufficient basis for intelligent agent decision-making.

[0037] Action space design: Assuming the drone travels at a constant speed Exercise, its movements Defined as two angles: the yaw angle in the horizontal direction. and vertical pitch angle .

[0038] The three-dimensional velocity vector of the drone Completely determined by its rate and actions The decision is made using the following formula: in, For constant flight speed, Yaw angle The pitch angle, The three components correspond to the velocity components in the X, Y, and Z axis directions, respectively.

[0039] Reward function design: Reward function Designed as a combination of continuous cost and discrete event reward, it guides the agent's learning with dense signals, and its calculation formula is as follows: in, , These are the weighting coefficients for each cost. The target distance cost is defined as the Euclidean distance between the current position of the UAV and the target point. The obstacle risk cost is defined as the sum of the reciprocals of the distances to all obstacles within the perception range, and its calculation formula is as follows: in, A collection of obstacles. for Time drones and the first The distance to the obstacle.

[0040] The energy cost is defined as the square norm of the difference between the previous action and the action at the previous moment. It is used to punish drastic changes in action and encourage smooth flight.

[0041] in, and These are the action vectors for the current time step and the previous time step, respectively.

[0042] Event rewards are awarded as follows: A large positive reward of +100 is given when the drone successfully reaches the target point; a large negative reward of -100 is given when the drone collides with an obstacle.

[0043] Step 1.4, System Initialization: WOA initialization: Randomly generate a WOA containing The population of individual whales, such as =50. Position vector of each whale. This corresponds to a parameterized candidate track.

[0044] DRL Initialization: Initialize a DRL agent based on the Actor-Critic framework.

[0045] Network Structure: Both the Actor and Critic networks employ a Multilayer Perceptron (MLP) with two hidden layers. The Actor network's input layer dimension is the same as the state space dimension, and its output layer dimension is the same as the action space dimension; the Critic network's input layer dimension is the sum of the state and action space dimensions, and it outputs a Q-value; all hidden layers use the ReLU activation function.

[0046] Target Networks: Create corresponding target networks for the Actor and Critic networks respectively, and initialize their weights to be the same as those of the main network.

[0047] Hyperparameter settings: Initialize the experience replay buffer The capacity is set to 1e6; the training batch size is... Set to 256; Discount factor Set to 0.99; Target network soft update coefficient Set the learning rate to 0.005; set the learning rates for the Actor and Critic networks to 1e-4 and 1e-3, respectively.

[0048] Step 2, Iterative optimization: Reference Figure 1 After initialization, the algorithm enters its main loop, with a total number of iterations set to [value missing]. Each generation executes the following multi-level optimization process.

[0049] Step 2.1, WOA Global Update: At the start of each iteration, the entire population is updated according to the WOA evolutionary rules; this is a macroscopic global search. For each individual whale... Based on a random probability between [0,1] Choose an update strategy: in, For the first The individual whale's position vector, It is the optimal solution in the current population. A random number in the range [-1, 1] This is a constant for the spiral shape. Coefficient vector. and The calculation is as follows: in, It is a random vector between [0,1]. The value decreases linearly from 2 to 0 as the number of iterations increases. Let the step size vector be the vector that surrounds the prey. This is the distance vector between the individual and the optimal solution. This mechanism allows the algorithm to perform well in the early stages ( ) tends to explore randomly, while in the later stages ( It tends to converge toward the current optimal solution.

[0050] Step 2.2, Three-stage collaborative update: Reference Figure 2 After the WOA update, a specific CEM-DRL collaborative update strategy is executed based on the current iteration number t and the stage at which it is located.

[0051] Step 2.2.1, Exploration Phase ( < / 3): The core objective at this stage is to conduct extensive global exploration to maximize population diversity. First, starting with the current population... Selecting based on fitness An elite set is formed by a number of elite individuals. Then, a weighted mean of the parameters of this elite set is calculated. and weighted covariance Thus, a Gaussian distribution is constructed. .

[0052] in For the number of elite individuals, For the first The position vector of an elite individual For the first The weight corresponding to each elite individual This is the updated distribution mean. This is the updated covariance matrix.

[0053] Subsequently, a completely new distribution of size is generated by sampling from this Gaussian distribution. population and completely replace the current population with it. This step is equivalent to a guided global population reset, which can effectively escape local optima.

[0054] To rapidly improve the overall quality of the new population, select the new population. The individuals ranked in the bottom 50% of fitness are selected. The parameters of these individuals are then sequentially loaded into the Actor network of the DRL agent. In the process, a complete trajectory planning round is executed by interacting with the environment, and experience data such as status, actions, and rewards are collected and stored in the experience replay buffer. When the buffer has sufficient data, mini-batch data is sampled from it to update the gradients of the Critic and Actor networks. The loss function of the Critic network... Policy gradient of the Actor network ( )as follows: in, For the goal value, For instant rewards, As a discount factor, For the next state, For the target network in state The following output action, This provides the Q-value estimates for the target Critic network's next state s' and the target Actor network's output action. and These are the parameters for the target Critic and Actor networks, respectively. For the first The error loss function of a Critic network. The policy gradient of the Actor network, passed through the first Critic network. To guide the Actor network parameters The direction of updates. For experience replay buffer, Indicates the state of sampling in the buffer. Find the expected value.

[0055] This step utilizes the gradient information of DRL to quickly improve poor-quality solutions, thereby raising the lower limit of the average quality of the entire population.

[0056] Step 2.2.2, Equilibrium Period ( / 3 <= < 2 / 3): The goal at this stage is to strike a balance between exploration and development. First, the current population is sorted by fitness, dividing it into the top 50% elite populations. and the bottom 50% of the general population Then, based on... The distribution of Gaussian model was calculated. And sample from them to generate new individuals, and use these new individuals to randomly replace One-third of the members. The purpose of this step is to conduct a detailed exploration around the dominant region, introducing potential better solutions.

[0057] Subsequently, For all individuals, perform a DRL gradient update process similar to step 2.2.1 to guide them to evolve into an elite group and accelerate convergence.

[0058] Step 2.2.3, Utilization period ( >= 2 / 3): The core objective of this stage is to refine the optimal solution region. First, the population is divided into elite populations. and ordinary population Then, a Gaussian model is calculated based on the distribution of the entire population. And randomly replace new individuals from them. One-third of the members are selected to prevent the entire population from converging to the same point too early.

[0059] Subsequently, All individuals in the dataset undergo DRL gradient updates. Since these individuals are already in high-reward regions, the gradient information from the DRL can be used to fine-tune them, allowing them to quickly approach a local optimum.

[0060] Step 2.3, Update the optimal solution: After each iteration of the collaborative update, the fitness of all individuals in the entire population is re-evaluated, and the current best individual is compared with the global optimal solution. Compare and update.

[0061] Step 3, Output the result: When the number of iterations achieve Then, the main loop terminates. At this point, the global optimal solution is found. The corresponding trajectory parameters are the final planned optimal trajectory, which is then output.

[0062] Reference Figure 3 The figure illustrates the optimal trajectory planned by this method in a 3D simulation environment containing multiple spherical obstacles. In the figure, the yellow dots represent the starting points, the red dots represent the ending points, and the blue solid line represents the planned optimal trajectory, which successfully and smoothly bypasses all obstacles.

[0063] Example 1 The method of this invention was simulated in simulation environment 1. For example... Figure 3 As shown, the optimal three-dimensional trajectory planned by the method of the present invention successfully avoids all obstacles and accurately reaches the destination.

[0064] Example 2 The method of this invention was simulated in simulation environment 2, where the obstacle distribution differed from that in Example 1. Figure 4 As shown, this method also plans a safe and smooth flight path, verifying its adaptability under different environmental layouts.

[0065] Example 3 The method of this invention was simulated in simulation environment 3. The obstacle layout in this environment is a different random configuration than those in the aforementioned embodiments. Figure 5 As shown, this method still successfully plans a safe and smooth trajectory, fully demonstrating its adaptability to different obstacle distributions.

[0066] In summary, simulation results show that the method proposed in this invention can plan safe and smooth optimal trajectories in different complex three-dimensional environments, effectively avoiding all threat areas, and demonstrating good robustness and practicality.

[0067] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A UAV trajectory planning method based on hybrid WOA-CEM-DRL, characterized in that: Includes the following steps: A mathematical model for the UAV 3D trajectory planning problem is established, and candidate trajectories are parameterized into optimization vectors. Initialize the population representing candidate paths and the deep reinforcement learning agent network; The loop optimization is performed repeatedly within a preset number of iterations, and the loop optimization includes: The whale optimization algorithm is used to perform a global search of the population. Combine cross-entropy method with deep reinforcement learning for collaborative updates; The focus of exploration and development should be dynamically adjusted according to the pre-, mid-, and late-stage evolution; After the iteration is completed, the individual with the best fitness is selected from the final population as the output of the optimal trajectory.

2. The method according to claim 1, characterized in that: The mathematical model constructs UAV trajectory planning as a constrained multi-objective optimization problem, with its cost function... The path length cost is defined as the path length cost. And the cost of track smoothness Weighted sum: in The total number of waypoints. For the first The coordinates of each waypoint This is the direction vector of the track segment. and These are weighting coefficients that balance the importance of path length and smoothness.

3. The method according to claim 1, characterized in that: The state of the deep reinforcement learning agent This includes the distance between the drone and the target point, the direction vector pointing towards the target point, and the current velocity vector, as well as its actions. The reward function includes the yaw and pitch angles that control the direction. It includes continuous costs related to the distance to the target point and obstacles and control energy consumption, combined with event rewards triggered when discrete events such as collisions or reaching the target occur.

4. The method according to claim 1, characterized in that, The position update of the whale optimization algorithm is based on the current optimal solution. The encirclement or spiral update mechanism is mathematically expressed as: in For the first The individual whale's position vector, This is the current optimal solution position. The random probability value is within the range [0,1]. For the encirclement distance, This is the straight-line distance to the optimal solution. A random number in the range [-1, 1] Let be the helical shape constant, and be the coefficient. and The process is dynamically adjusted during iteration to balance global and local searches.

5. The method according to claim 1, characterized in that: The cross-entropy method updates by calculating the mean of elite individuals. Covariance Let's build a Gaussian probability model New individuals are generated by sampling from this model, and their mean and covariance are calculated using the following formulas: in For the number of elite individuals, For the first The position vector of an elite individual For the first The weight corresponding to each elite individual This is the updated distribution mean. This is the updated covariance matrix.

6. The method according to claim 5, characterized in that: The deep reinforcement learning update employs a policy gradient method based on the Actor-Critic framework, updating the parameters of the Critic network by minimizing its loss function. The Critic network is then used to guide the parameters of the Actor network. Update the policy gradient in the direction of improving policy performance. The calculation method is as follows: in Let the objective function of the policy network be... The objective function with respect to the Actor network parameters policy gradient, For state, For the Actor network in state The following output action, For the Critic network to state and strategy Output action Value estimation, For experience replay buffer, This represents the mathematical expectation of the sampled state s in the buffer.

7. The method according to claim 6, characterized in that: The dynamic adjustment exploration and development in the early stage of evolution focuses on the following: based on the elite solution, the probability distribution of the high-reward region is estimated by the cross-entropy method and a new, more diverse population is generated by sampling. Then, deep reinforcement learning is used to perform gradient boosting on individuals with poor fitness in the new population, so as to quickly improve the inferior solution and increase the lower limit of the average quality of the entire population.

8. The method according to claim 7, characterized in that: The dynamic adjustment exploration and development in the mid-evolutionary stage focuses on the following: First, the population is divided into an elite population and a normal population. The cross-entropy method is used to inject perturbation into the elite population to introduce new high-quality solutions. At the same time, deep reinforcement learning is used to perform gradient optimization on the normal population to accelerate its approach to the elite region.

9. The method according to claim 8, characterized in that: The dynamic adjustment exploration and development in the later stage of evolution focuses on: using the cross-entropy method to update the ordinary population in a small range to maintain the necessary population diversity, while using deep reinforcement learning to finely optimize the gradient of the elite population so that it can quickly approach the local optimum.

Images