Unmanned aerial vehicle path planning method based on reinforcement learning and particle swarm optimization algorithm

Abstract

This invention discloses a UAV path planning method based on reinforcement learning and particle swarm optimization (PSO) algorithms, belonging to the field of intelligent UAV path planning technology. Addressing the challenges of dense low-altitude urban buildings and complex constraints, traditional algorithms are prone to getting trapped in local optima, while PSO algorithms suffer from premature convergence and policy rigidity. This invention first constructs a three-dimensional urban environment and a multi-objective cost function including path length, threat, altitude, and smoothness, taking into account both UAV kinematics and airspace constraints. Then, it groups particles based on fitness and activity and designs differentiated update strategies, introducing mechanisms such as chaotic perturbation and elite guidance to improve optimization performance. Finally, it builds an adaptive policy selection framework based on Q-learning and designs a composite reward function to dynamically balance global exploration and local development. Validated by test functions and multi-complexity urban scenarios, this invention demonstrates high convergence accuracy, good obstacle avoidance performance, and path planning that satisfies multiple constraints, effectively overcoming premature convergence and adapting to low-altitude UAV operations such as urban inspection, logistics delivery, and traffic monitoring.

Description

Technical Field

[0001] This invention belongs to the field of intelligent path planning technology for unmanned aerial vehicles (UAVs), specifically relating to a UAV path planning method based on reinforcement learning and particle swarm optimization algorithms. Background Technology

[0002] Unmanned aerial vehicles (UAVs), with their flexibility, efficiency, and low cost, have shown great application potential in areas such as urban public safety inspection, emergency logistics delivery, and traffic monitoring. UAV path planning is a core technology to ensure their safe and efficient operation in urban low-altitude environments. However, the urban environment is characterized by dense buildings and stringent flight constraints, posing severe challenges to path planning problems due to multi-objective conflicts and high-dimensional nonlinear constraints. Currently, commonly used path planning methods mainly fall into two categories: traditional algorithms and metaheuristic algorithms. Traditional graph search-based algorithms such as A* and Dijkstra's algorithm are effective in specific, simple scenarios, but they suffer from the following drawbacks when dealing with the unique high-dimensional, nonlinear, multi-modal, and complex constraint optimization problems of urban environments:

[0003] 1. High computational complexity, prone to getting trapped in local optima, and insufficient adaptability to dynamic environments;

[0004] 2. Existing methods only focus on solving the shortest path for obstacle avoidance, without refining the actual flight scenario. The flight area often uses a simplified 3D model, resulting in insufficient path planning accuracy and poor usability in practical applications.

[0005] 3. Although existing research considers obstacle avoidance, it does not adequately set up safety redundancy, which makes it easy for drones to get close to the ground or obstacles, increasing flight safety risks;

[0006] 4. Most methods only take the shortest path and obstacle avoidance as the single optimization objective, without comprehensively balancing multiple factors. It is difficult to achieve the global optimality of safety, energy consumption and flight stability, and the universality of different missions and aircraft types is insufficient. Summary of the Invention

[0007] The present invention aims to solve the above-mentioned problems in the prior art and provide a UAV path planning method based on reinforcement learning and particle swarm optimization algorithm.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] The UAV path planning method based on reinforcement learning and particle swarm optimization includes the following steps:

[0010] Step 1: Environment Modeling

[0011] The urban low-altitude UAV path planning problem is transformed into a multi-constraint optimization problem. The path consists of a start point, an end point, and several nodes to be optimized. The feasible solution space is constrained by the flight altitude range and environmental boundaries. The total cost function is a weighted sum of path length cost, threat cost, altitude cost, and smoothness cost. The path length cost is defined as the sum of the Euclidean distances of all segments to save energy and mission time. The threat cost models obstacles such as buildings as cylinders and penalizes them based on the distance between the path segments and obstacles. The altitude cost is used to constrain the UAV's flight altitude within a specified range by penalizing deviations from the target cruising altitude and forcibly limiting the altitude boundary. The smoothness cost includes turning angle cost and climb angle cost to ensure that the path meets the UAV's kinematic constraints such as minimum turning radius and maximum climb angle.

[0012] Step 2: Constructing a hybrid particle swarm optimization algorithm

[0013] Particles are categorized into four types—elite, ordinary, active, and dormant—based on both fitness and activity levels, implementing a differentiated evolutionary strategy. Elite particles perform refined local searches, ordinary particles introduce chaotic perturbations to enhance their exploration capabilities, active particles expand their search radius to cover a wider area, and dormant particles are reactivated by leveraging high-quality external information. Simultaneously, an elite-center-guided approach, neighborhood learning, and bidirectional search strategies are integrated to achieve a dynamic balance between exploration and development.

[0014] Step 3: Design a framework combining reinforcement learning and particle swarm optimization

[0015] Q-learning is employed to construct a state-action Q-table for each particle, using multiple candidate update strategies as the action space. Each row of the Q-table corresponds to a particle, representing its current state; each column corresponds to an optional action, i.e., a candidate update strategy. The particle selects an action based on its current state, executes the selected strategy, and generates a new individual. The system then calculates the corresponding reward and updates the Q-value of the corresponding unit in the Q-table. Based on the updated Q-value, the individual continues to select the strategy for the next iteration, repeating this process until the algorithm reaches its termination condition. Through continuous interactive learning, the algorithm can autonomously select the optimal update strategy based on real-time feedback during the evolutionary process, avoiding frequent reliance on a single strategy that could lead to local optima, thus achieving intelligent adaptive adjustment of the search behavior.

[0016] Step 4: Design the reward function for reinforcement learning

[0017] The composite reward function includes a fitness improvement reward and a diversity reward. The former measures the improvement of a particle's current position compared to the historical best, while the latter assesses the population dispersion through the Euclidean distance between particles. This design guides the algorithm to maintain a balance between convergence accuracy and exploration capability, avoiding premature convergence.

[0018] Step 5: Path Planning

[0019] In three types of urban scenarios with 40, 60, and 80 obstacles, the optimal path generated by RLHPSO demonstrates excellent obstacle avoidance capability and smoothness.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] This invention addresses the complex characteristics of low-altitude environments in cities, such as dense buildings, numerous no-fly zones, restricted airspace altitude, and multiple coupled constraints. It constructs a multi-dimensional cost function that includes path length, obstacle threat, flight altitude, and path smoothness, while also taking into account UAV kinematic constraints (minimum turning radius, maximum climb angle) and civil aviation airspace altitude restrictions. This allows for precise adaptation to urban three-dimensional low-altitude path planning scenarios with multiple obstacles and constraints, solving the problem of poor adaptability of traditional algorithms to complex urban environments.

[0022] By grouping particles into elite, ordinary, active, and dormant groups based on both fitness and activity dimensions, and designing differentiated particle velocity and position update strategies, this algorithm incorporates mechanisms such as chaotic perturbation, elite center guidance, neighborhood optimum competition, and simulated annealing probability acceptance. This effectively improves upon the shortcomings of the standard particle swarm optimization algorithm, such as a sharp drop in population diversity in the later stages of evolution, susceptibility to local optima, and premature convergence. The algorithm's ability to balance global exploration and local development is significantly enhanced. Furthermore, by integrating Q-learning reinforcement learning to construct an adaptive strategy selection framework, a Q-table is built to enable particles to dynamically and autonomously optimize and update strategies based on population evolution status, fitness changes, and population diversity. This eliminates the limitations of traditional fixed rules and manually preset strategies, and does not rely on human experience. The algorithm can intelligently switch search modes at different optimization stages, dynamically adapting to planning needs with varying terrain complexity.

[0023] A composite reward function is constructed that integrates fitness improvement rate, population diversity, elite dispersion, and exploration of new regions. Differentiated rewards are given to elite and non-elite particles respectively, which not only accelerates the convergence of high-quality solutions but also incentivizes particles to explore unknown regions, thereby further balancing the algorithm's convergence speed and global search capability.

[0024] Through simulation verification using standard test functions and various comparative algorithms (PSO, EAPSO, SDPSO, MPSO, CSA, GMO, CCE, MSMA), the RLHPSO algorithm of this invention has significant advantages in optimization accuracy, stability, and convergence performance, and can still maintain excellent solution performance under high-dimensional complex optimization problems.

[0025] In simple, medium, and complex city simulation scenarios with 40 / 60 / 80 obstacles, it can quickly plan safe, collision-free, optimally lengthed, smooth-flying, and airspace and maneuvering constraints in three-dimensional flight paths. The paths can directly meet the actual needs of UAV city inspection, emergency delivery, traffic monitoring, and other tasks. It has strong resistance to environmental interference and good engineering feasibility.

[0026] Compared to traditional graph search algorithms such as A* and Dijkstra, it avoids the problems of increased computational load and poor real-time performance under high-dimensional complex constraints. At the same time, relying on the reinforcement learning self-learning decision mechanism, it has better adaptive adjustment capabilities for changing scenarios such as dynamic obstacles in the city and temporary airspace restrictions compared to hybrid algorithms with fixed improvement strategies. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the UAV path planning environment of the present invention;

[0028] Figure 2 This is a schematic diagram of the turning and climb angles of the drone of the present invention;

[0029] Figure 3 This is a 3D diagram of the optimal paths of the four algorithms under 80 obstacles according to the present invention;

[0030] Figure 4 This is a top view of the optimal paths of the four algorithms under 80 obstacles according to the present invention;

[0031] Figure 5 This is a side view of the optimal paths of the four algorithms under 80 obstacles according to the present invention;

[0032] Figure 6 This is the algorithm flowchart of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0034] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0035] In this embodiment of the invention, the UAV path planning method based on reinforcement learning and particle swarm optimization algorithm includes the following steps:

[0036] Step 1: Environment Modeling

[0037] like Figure 1As shown, the goal of UAV path planning in urban environments is to find an optimal or suboptimal flight path from the starting point to the destination in a three-dimensional solution space, while satisfying its own performance constraints and geographical constraints. This problem can be formally defined as...

[0038]

[0039] in, Indicates by The flight path consists of several waypoints. and They are fixed starting and ending points, respectively. These are the path nodes that need to be optimized. It is a feasible solution space, limited by the flight altitude range of the drone. Together with environmental boundaries, they constitute a constraint.

[0040] In urban environments, The standard value is 120m (as stipulated by the Civil Aviation Administration of China), but this article uses 100m to ensure compliance.

[0041] Step Two: Flight Costs

[0042] A cost function that comprehensively considers path optimality, safety, and feasibility. Path The total cost function consists of four parts: path length cost, threat cost, altitude cost, and smoothness cost. The weighted sum of these four parts is the total cost function.

[0043]

[0044] in, These are weighting coefficients used to balance the importance of different optimization objectives. It is the first The first path That comes at a cost.

[0045] When performing missions, drones typically aim to reach their destination via the shortest path to save energy and mission time. Path length cost. Defined as the sum of the Euclidean distances of all flight segments

[0046]

[0047] Buildings, no-fly zones, and other obstacles in urban environments pose a direct threat to the flight safety of drones. This paper models obstacles as cylinders, with each obstacle... From its horizontal projection center coordinates ,radius and height Uniquely determined. The mathematical model of the obstacle is defined as follows:

[0048]

[0049] in It is the projection of a point onto a horizontal plane. It is the corresponding height of that point. and Indicates the first The center coordinates of the obstacles It is the number of obstacles. Indicates the first The height of the obstacle.

[0050] For a given path segment The associated threat costs and their impact on distance Proportional. By considering the diameter of the drone and the dangerous distance between the collision zone For obstacle sets and all waypoints Threat and cost Calculation as follows

[0051]

[0052]

[0053] Among them, diameter Danger distance is determined by the size of the drone. It depends on several factors, such as application type, operating environment, and positioning accuracy. In urban environments, the danger distance is taken as 10 times the size of the drone.

[0054] Drones must maintain a prescribed flight altitude; they cannot fly too low to avoid collisions with ground obstacles, nor too high to exceed airspace restrictions. Therefore, altitude comes at a cost. Defined as

[0055]

[0056]

[0057] in, The target cruising altitude is defined by equation (16). Equation (16) encourages stable flight by penalizing deviations from the target altitude, and also constrains the flight altitude by imposing an infinite penalty. Within the range.

[0058] like Figure 3 As shown, the maneuverability of a drone is limited by physical constraints such as minimum turning radius and maximum climb angle; therefore, the planned path must be sufficiently smooth to ensure kinematic feasibility. Smoothness Cost It includes two items: turning angle cost and climb angle cost. Turning angle They are two consecutive path segments and On the horizontal plane The angle between the upper projections. Let for If the unit vector is along the axis, then the projection vector can be calculated as follows:

[0059]

[0060] Therefore, the turning angle is calculated as follows:

[0061]

[0062] Angle of climb It is a path segment Its projection on the horizontal plane The included angle between them is given by the following formula.

[0063]

[0064] The smoothness cost is calculated as follows:

[0065]

[0066] in, and These are the penalty coefficients for turning angle and climb angle, respectively.

[0067] Step 3: Constructing a hybrid particle swarm optimization algorithm

[0068] First, particles are grouped based on two dimensions: fitness and activity. Different particles play different roles in the evolutionary process: elite particles are responsible for localized in-depth searches, while ordinary particles are responsible for global exploration; active particles maintain search momentum, while stagnant particles require external incentives. By implementing differentiated strategies through grouping, the potential of each particle can be more fully realized.

[0069] Secondly, different update strategies are implemented for different particles.

[0070] Elite particles represent the group of individuals with the best fitness in the current population, and high-quality solution regions usually exist near them. Therefore, a locally refined search strategy is adopted for elite particles: randomly selected non-elite particles are introduced as a difference guide term in the velocity update, allowing elite particles to retain their advantageous direction while being slightly perturbed by the diversity information of non-elite individuals. In addition, a weak guide term for the elite center is added to further promote the convergence of elite particles to high-potential regions. The update formula for elite particles is as follows:

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077] Among them, learning factor and They are 2 and 1 respectively. It is a random particle among ordinary particles. Ordinary particles have relatively poor fitness and require stronger exploration capabilities to improve their position. Randomly selected elite particles provide high-quality directional guidance while introducing chaotic perturbations to enhance the ability to escape local traps. This strategy encourages ordinary particles to move towards elite and optimal solution regions while maintaining sufficient randomness to cover a wider search space.

[0078]

[0079]

[0080]

[0081]

[0082] in, and It is a random particle among elite particles. It is the fitness value ratio A low-level random particle, It is a directional random factor.

[0083] Active particles possess strong exploration potential. For these particles, the strategy deliberately expands their search radius, introduces the directional difference of another active particle into the velocity update, and adds the position of a stationary particle as a contrast perturbation, thereby generating more diverse trial step sizes. At the same time, a small amount of random noise is superimposed to expand the search area.

[0084]

[0085]

[0086] in, It is a random particle among active particles. and It is a random particle among dormant particles. yes OK A matrix of random numbers in columns. It is 1 line The row vectors of the column conform to a standard normal distribution.

[0087] Dormant particles, having made no progress for a long time, have lost their intrinsic value. The strategy no longer relies on their historical trajectories, but instead forcibly activates them with the help of high-quality external information. It uses randomly active particles and global optimum as learning factors, supplemented by chaotic perturbations, to jointly drive dormant particles back into an effective search trajectory.

[0088]

[0089]

[0090]

[0091] in, It is a random particle among active particles. It is the fitness value ratio A low-level random particle.

[0092] When there are a certain number of elite particles in the population, their spatial centroids represent the clustering center of the current optimal solution region. Strategy 3 utilizes this information as an additional guiding term, causing particles to simultaneously move towards both the elite center and the global optimal position, thus accelerating population convergence.

[0093]

[0094]

[0095] Strategy 4 introduces nonlinear inertial weights with chaotic modulation, enabling the algorithm to dynamically switch between exploration and development. Velocity updates employ a neighborhood-optimal competition mechanism: two particles are randomly selected, and the better individual's optimal value is used as the guiding direction. Simultaneously, the mean of the best individual in the population is introduced as a social learning term. Position updates incorporate a simulated annealing probabilistic acceptance mechanism, accepting the perturbed position with a certain probability.

[0096]

[0097]

[0098]

[0099]

[0100] in, It is the best in the neighborhood.

[0101] Step 4: Design a framework combining reinforcement learning and particle swarm optimization

[0102] To dynamically balance the exploration and development needs at different optimization stages, this invention proposes a reinforcement learning-based selection mechanism to assist the particle swarm optimization algorithm in selecting candidate update strategies during the optimization process. In complex terrain, frequently relying on a single strategy to update candidate solutions may cause the algorithm to get stuck in local optima. Q-learning can dynamically adjust its strategy through reward feedback in UAV path planning, ensuring more intelligent decision-making throughout the optimization process.

[0103] To improve the update policy selection mechanism of the PSO algorithm using reinforcement learning, we construct an N-row, 4-column Q-table to store the expected cumulative reward for each individual in the population under different states and actions. Each row in the Q-table corresponds to an individual in the population, representing its current state; each column corresponds to an optional action, i.e., a candidate update policy. When an individual executes the selected policy and generates a new individual, the system calculates the corresponding reward and then updates it based on the Q-value of the corresponding unit in the Q-table. Based on the updated Q-value, the individual continues to select the policy for the next iteration, and this process is repeated until the algorithm reaches the termination condition.

[0104] Step 4: Design the reward function for reinforcement learning

[0105] To improve the performance of the adaptive selection strategy, this paper considers the fitness improvement rate and population diversity, as well as the appropriate dispersion of elite particles and the need for non-elite particles to explore new regions in the reward function. Therefore, a composite reward function is designed as shown in Equation (34), and the rewards for elite particles and non-elite particles are given by Equations (37) and (38), respectively.

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112] in, It is the total reward. It is an adaptation improvement reward. It is a variety of rewards. This represents the extent of improvement in the particle's fitness. It is the normalized distance. denoted as Euclidean distance between particles. From the perspective of the problem dimension, and These represent the upper and lower bounds of the search space, respectively.

[0113] Step 5: Path Planning

[0114] This invention validated the algorithm's effectiveness in urban environments of varying complexity. Three urban simulation scenarios were designed, with 40, 60, and 80 obstacles respectively, corresponding to simple, medium, and complex terrain complexities. In all scenarios, obstacles were modeled as cylinders, randomly distributed across a 750m × 750m two-dimensional plane, with heights randomly generated between 10m and 100m. The start and end points for the three scenarios were set as follows: Simple Scenario 1: Start (1,1,0), End (500,600,50), 40 obstacles; Medium Scenario 2: Start (0,0,0), End (600,400,50), 60 obstacles; Complex Scenario 3: Start (1,1,0), End (500,600,50), 80 obstacles. The progression in scenario complexity is reflected in: the increased number of obstacles narrows the feasible path; and the changing end point position forces the algorithm to search for paths in different areas.

[0115] A comparative experiment was conducted based on an urban terrain scenario to evaluate the performance of the RLHPSO algorithm in the UAV path planning problem. In this part of the test, CCE, CSA, and EAPSO algorithms were selected as comparison algorithms for testing.

[0116] from Figures 3-6 As can be seen, none of the algorithms' planned paths collided with the terrain. However, the algorithms differed in path quality. CCE and EAPSO generated paths with more unnecessary curves, exhibiting significant detours when avoiding obstacles, leading to increased path length and more convoluted paths for the drone. The paths generated by CCE and EAPSO also had higher flight altitudes, indicating that the planned paths were not stable in flight altitude. Figure 5 This confirms the point, resulting in a higher average path length. In contrast, the paths planned by CSA and RLHPSO are smoother, with gentler turns and flatter flight altitudes. However, CSA did not find a sufficiently short path, while RLHPSO's path avoided redundant detours. RLHPSO adapts very well to urban environments of varying complexity.

Claims

1. A method for UAV path planning based on reinforcement learning and particle swarm optimization algorithms, comprising the following steps: Step 1: Environment Modeling The path planning problem for urban low-altitude UAVs is transformed into a multi-constraint optimization problem. The path consists of a starting point, an ending point, and several nodes to be optimized. The feasible solution space is constrained by the flight altitude range and environmental boundaries. The total cost function is a weighted sum of path length cost, threat cost, altitude cost, and smoothness cost. The path length cost is defined as the sum of the Euclidean distances of all segments to save energy and mission time. The threat cost models obstacles such as buildings as cylinders and penalizes them based on the distance between the path segments and obstacles. The altitude cost is used to constrain the UAV's flight altitude within a specified range by penalizing deviations from the target cruising altitude and forcibly limiting the altitude boundary. The smoothness cost includes turning angle cost and climb angle cost to ensure that the path meets the kinematic constraints of the UAV, such as the minimum turning radius and maximum climb angle. Step 2: Constructing a hybrid particle swarm optimization algorithm Particles are divided into four categories—elite, ordinary, active, and dormant—based on two dimensions: fitness and activity. A differentiated evolution strategy is implemented, with elite particles performing local fine-grained searches, ordinary particles introducing chaotic perturbations to enhance their exploration capabilities, active particles expanding their search radius to cover a wider area, and dormant particles being reactivated with the help of high-quality external information. At the same time, the strategy integrates elite center guidance, neighborhood learning, and bidirectional search to achieve a dynamic balance between exploration and development. Step 3: Design a framework combining reinforcement learning and particle swarm optimization Q-learning is used to construct a state-action Q-table for each particle, and multiple candidate update strategies are used as the action space. Each row of the Q-table corresponds to a particle, representing its current state; each column corresponds to an optional action, i.e., a candidate update strategy. The particle selects an action according to its current state, executes the selected strategy and generates a new individual. The system calculates the corresponding reward and then updates the Q value of the corresponding unit in the Q-table. Based on the updated Q value, the individual continues to select the strategy for the next iteration. This process is repeated until the algorithm reaches the termination condition. Through continuous interactive learning, the algorithm can autonomously select the optimal update strategy based on real-time feedback during the evolution process, avoiding frequent reliance on a single strategy that could lead to getting stuck in local optima, and achieving intelligent adaptive adjustment of search behavior. Step 4: Design the reward function for reinforcement learning The composite reward function includes fitness improvement reward and diversity reward. The former measures the improvement of the particle's current position compared to the historical best, while the latter evaluates the population dispersion through the Euclidean distance between particles. This design guides the algorithm to maintain a balance between convergence accuracy and exploration ability, and avoids premature convergence. Step 5: Path Planning In three types of urban scenarios with 40, 60, and 80 obstacles, the optimal path generated by RLHPSO demonstrates excellent obstacle avoidance capability and smoothness.

2. The UAV path planning method based on reinforcement learning and particle swarm optimization algorithm according to claim 1, characterized in that, Construction of the particle grouping framework in the particle swarm optimization algorithm: Grouping particles based on both fitness and activity levels, and implementing differentiated strategies through grouping, allows for a more comprehensive utilization of each particle's potential. Elite particle swarm optimization sorts the population in ascending order of fitness value, selecting the top 20% of particles by fitness value as elite particles, and the rest as ordinary particles. However, this grouping does not consider the potential exhibited by ordinary particles during evolution. Therefore, this algorithm proposes a new grouping method, detailed as follows: in, , , Based on the evaluation of the new fitness, the top 20% of particles are selected as the elite group, a proportion consistent with the elite particle swarm optimization algorithm. The activity level of a particle reflects the drastic nature of its position updates. Active particles have strong exploration capabilities and should maintain their search momentum; stagnant particles, on the other hand, require external incentives to help them escape local optima. in, , , Based on the activity level evaluation, the top 50% of particles are selected as the active group.

3. The UAV path planning method based on reinforcement learning and particle swarm optimization algorithm according to claim 1, characterized in that, Construction of the particle update formula in the particle swarm optimization algorithm: The update formula for elite particles is: The update formula for ordinary particles is: The update formula for active particles is: The update formula for dormant particles is: Among them, learning factor and They are 2 and 1 respectively. It is a random particle among ordinary particles. and It is a random particle among elite particles. It is the fitness value ratio A low-level random particle, As a directional random factor, It is a random particle among active particles. It is the fitness value ratio A low-level random particle, It is a random particle among active particles. and It is a random particle among dormant particles. yes OK A matrix of random numbers in columns. It is 1 line The row vectors of the column conform to a standard normal distribution.

4. The UAV path planning method based on reinforcement learning and particle swarm optimization algorithm according to claim 1, characterized in that, The construction of the aforementioned particle swarm optimization algorithm and strong bad learning fusion framework: To dynamically balance the exploration and development needs at different optimization stages, this invention proposes a reinforcement learning-based selection mechanism to assist the PSO algorithm in selecting candidate update strategies during the optimization process. In complex terrain, frequently relying on a single strategy to update candidate solutions may cause the algorithm to get stuck in local optima. Q-learning can dynamically adjust its strategy through reward feedback in UAV path planning, ensuring more intelligent decision-making throughout the optimization process. When applying reinforcement learning to improve the update policy selection mechanism of the PSO algorithm, we construct a Q-table with N rows and 4 columns to store the expected cumulative reward of each individual in the population under different states and actions. Each row in the Q-table corresponds to an individual in the population and represents its current state. Each column corresponds to an optional action, i.e., a candidate update strategy. When an individual executes the selected strategy and generates a new individual, the system calculates the corresponding reward, and then updates it based on the Q-value of the corresponding cell in the Q-table. Based on the updated Q-value, the individual continues to select the strategy for the next iteration, and so on, until the algorithm reaches the termination condition. To improve the performance of the adaptive selection strategy, this invention considers the fitness improvement rate and population diversity, as well as the appropriate dispersion of elite particles and the need for non-elite particles to explore new regions in the reward function. Therefore, a composite reward function is designed, with separate rewards for elite and non-elite particles. in, It is the total reward. It is an adaptation improvement reward. It is a variety of rewards. This represents the extent of improvement in the particle's fitness. It is the normalized distance. Let be the Euclidean distance between particles. From the perspective of the problem dimension, and These represent the upper and lower bounds of the search space, respectively.

Images