A multi-strategy particle swarm method for unmanned aerial vehicle path planning based on reinforcement learning
By introducing a reinforcement learning mechanism and a multi-strategy particle swarm optimization algorithm with adaptive inertial weights, the problems of computational complexity and unstable planning results in UAV trajectory planning under complex environments are solved, and efficient and safe trajectory planning is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENYANG AEROSPACE UNIVERSITY
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-09
AI Technical Summary
Existing UAV trajectory planning methods suffer from high computational complexity, unstable planning results, and a tendency to get trapped in local optima under complex environments and multiple constraints, making it difficult to simultaneously meet the requirements of path quality and planning efficiency.
By combining reinforcement learning and particle swarm optimization algorithms, a multi-strategy particle swarm optimizer is constructed. By adaptively adjusting the learning factor and inertia weight, and combining it with a multi-objective path evaluation function, the UAV trajectory planning is optimized.
It improves the safety and reliability of flight path planning results, generates safer, smoother and higher quality flight paths, and enhances the flight stability and efficiency of UAVs in complex environments.
Smart Images
Figure CN122170870A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent control and path planning for unmanned aerial vehicles (UAVs), and specifically relates to a multi-strategy particle swarm UAV trajectory planning method based on reinforcement learning. Background Technology
[0002] Unmanned aerial vehicle (UAV) trajectory planning is a key technology to ensure UAV flight safety and mission execution efficiency. Its goal is to plan a flight path for UAVs from the starting position to the target position under complex environmental conditions, so that UAVs can avoid obstacles and threat areas while meeting constraints such as flight altitude and turning radius, and taking into account performance indicators such as flight time and energy consumption.
[0003] Existing UAV trajectory planning methods can be mainly classified into the following categories.
[0004] The first category comprises graph search-based trajectory planning methods, typically including the A* algorithm and Dijkstra's algorithm. These methods usually construct a corresponding topological graph by modeling the flight environment in a grid or discretized manner, and then search for the shortest or optimal path from the starting point to the destination within the graph. While these methods can obtain feasible trajectories in relatively static environments or scenarios with simple obstacle distributions, the computational complexity of environment modeling and path search is high in dynamic environments or scenarios with complex obstacle distributions, making it difficult to simultaneously meet the requirements of planning efficiency and path quality.
[0005] The second category comprises sampling-based trajectory planning methods, with representative methods including Rapidly-exploring Random Tree (RRT) and its improved algorithms. These methods construct the path search structure by randomly sampling in the state space or configuration space, making them suitable for high-dimensional spaces and complex constrained environments. However, the trajectories generated by these methods typically exhibit poor continuity and smoothness, and the planning results are easily affected by sampling strategies and environmental uncertainties, thus requiring improvement in stability.
[0006] The third category comprises trajectory planning methods based on intelligent optimization algorithms, including Particle Swarm Optimization (PSO), Ant Colony Optimization (ACO), and Genetic Algorithm (GA). These methods iteratively optimize candidate trajectories by simulating the cooperative or evolutionary behavior of natural populations, achieving superior planning results under complex constraints.
[0007] Among the aforementioned intelligent optimization algorithms, particle swarm optimization (PSO) is widely used in UAV trajectory planning due to its simple structure, ease of implementation, and relatively small number of parameters. However, standard PSO still has some shortcomings in practical applications: on the one hand, its algorithm parameters are usually fixed or empirically set, making it difficult to adaptively adjust them according to different stages of the optimization process; on the other hand, under complex environments and multiple constraints, the algorithm is prone to a decrease in convergence speed or getting trapped in local optima, thus affecting the overall performance of trajectory planning. Summary of the Invention
[0008] In view of this, the present invention provides a multi-strategy particle swarm unmanned aerial vehicle (UAV) trajectory planning method based on reinforcement learning, comprising the following steps:
[0009] Step 1: Obtain environmental information of the UAV flight site, obtain 3D elevation map data, model the mountain environment based on the 3D elevation map data, establish a 3D environmental threat model for UAV trajectory planning, and determine the boundary constraints and obstacle distribution of the flight area;
[0010] Step 2: Taking into account factors such as path length, flight altitude, path smoothness, and collision threat, construct a global fitness function to evaluate the quality of the flight path, that is, establish a multi-objective path evaluation function, and transform the flight path planning into a multi-objective optimization problem;
[0011] Step 3: Introduce Q-learning reinforcement learning mechanism into the standard particle swarm optimization algorithm, establish Q table to store the mapping relationship between state and action, adaptively adjust the learning factor of the algorithm through the interaction between the agent and the environment, and combine the adaptive inertia weight mechanism to balance the global and local search capabilities, thereby constructing a multi-strategy particle swarm optimizer based on reinforcement learning mechanism.
[0012] Step 4: Under the constraints of the 3D environmental threat model and the global fitness function, the particle swarm is initialized. The multi-strategy particle swarm optimizer is used to iteratively search for path points in the 3D environmental threat model. During the iteration process, the reward signal is calculated based on the global fitness function value in Step 2 and the Q table is updated. The trajectory planning optimization is performed, and finally the UAV 3D flight trajectory with the best fitness is output.
[0013] Furthermore, in step 1, the construction of the environmental threat model is specifically as follows:
[0014] ;
[0015] in, A three-dimensional environmental threat model, , Represents the projected coordinates of the horizontal plane in the environmental model. This represents the terrain elevation value corresponding to the projection point. , and These represent the maximum boundary range in the three-dimensional space under the map coordinate system; based on the spatial distribution characteristics of the threat areas, several cylindrical threat areas are set in the simulation environment, specifically described as follows:
[0016] ;
[0017] in, Indicates the first There are several threat zones, with their center coordinates as follows: , radius is The height is ,
[0018] The establishment of the three-dimensional environmental threat model is based on the acquired three-dimensional elevation and terrain data, and is achieved by constructing the three-dimensional environmental threat model. and setting up various threat zones Together, they form the flight space constraints used for UAV trajectory planning, thus providing an environmental modeling foundation for subsequent reinforcement learning-based multi-strategy particle swarm UAV trajectory planning methods.
[0019] Furthermore, in step 2, the multi-objective path evaluation function, i.e., the global fitness function... The specific construction method is as follows:
[0020] ;
[0021] in, For path Length, For path Height coefficient, For path Smoothness coefficient, For path The collision coefficient; , , , Represents the weights corresponding to each evaluation function; where,
[0022] 1) Path length function : Calculate the sum of Euclidean distances between adjacent waypoints;
[0023] ;
[0024] in, Represents the total number of waypoints;
[0025] 2) Flight altitude function Assess whether waypoint altitudes are within safe and energy-efficient ranges, and penalize waypoints that exceed minimum or maximum flight altitudes;
[0026] ;
[0027] 3) Path smoothness function The curvature cost of the path is calculated by combining the changes in horizontal turning angle and vertical climb angle to ensure flight stability.
[0028] ;
[0029] ;
[0030] ;
[0031] ;
[0032] In the formula: and The horizontal turning angles for UAV 3D path planning are respectively. and vertical climbing angle The weight, Indicates the first [unmanned aerial vehicle] flight path The first waypoint and the first The displacement vector between each waypoint is used to describe the change in the flight direction of the UAV between adjacent waypoints;
[0033] 4) Collision Threat Function The area around the obstacle is divided into an obstacle zone, a threat zone, and a secondary threat zone; when the drone's waypoint falls into different zones, its distance from the center of the obstacle is used as the determining factor. Apply penalty values with different gradients; the closer the distance, the higher the penalty value.
[0034] ;
[0035] In the formula: Indicates the first [unmanned aerial vehicle] flight path The first waypoint to the second Euclidean distance from the center of each threat region; , , Let represent the radius of the danger zone, the radius of the threat zone, and the radius of the secondary threat zone, respectively, and satisfy . .
[0036] Furthermore, in step 3, the construction of the multi-policy particle swarm optimizer based on the reinforcement learning mechanism specifically includes:
[0037] 1) Establish the Q-table: Using the iterative process of the particle swarm optimization algorithm as the reinforcement learning environment, and the particle swarm used for trajectory planning and search as the agent, define the state space and action space of the reinforcement learning; wherein, the state space includes the iteration success state. (The fitness of the new generation of particle swarms is better than that of the previous generation) and iteration failure state (The fitness of the new generation of particle swarm optimization has not been improved); the action space includes various learning factors for particle swarm optimization algorithms. and The adjustment strategy includes at least increasing, decreasing, or keeping it unchanged, in order to achieve adaptive adjustment of particle swarm search behavior;
[0038] 2) Define the Q-value update rule: During the iterative process of the particle swarm optimization algorithm, the agent updates the Q-value according to the current iteration state. Select the appropriate parameter adjustment action. The Q-table is updated based on the reward signal generated after the action is applied to the particle swarm optimization algorithm. The Q-value update rule is as follows:
[0039] ;
[0040] in, Indicates the current state. Indicates the current action. It's the learning rate. It is a discount factor. Indicates the first State-action pairs at the next iteration The corresponding Q value, This represents the Q value after one update. It is in state Take action below The reward value, Indicates the next state Then, based on the current Q-table, take the maximum Q value for all optional actions;
[0041] 3) Design of adaptive inertia weight: Introduce a nonlinear dynamic update formula to decrease the inertia weight ω as the number of iterations increases, as shown in the following formula:
[0042] ;
[0043] In the formula, and These represent the current iteration number and the total number of iterations, respectively. and These represent the maximum and minimum values of the inertia weight, respectively, thus enabling a focus on global search in the early stages of iteration and local development in the later stages.
[0044] Furthermore, in step 4, the specific process of performing trajectory planning and optimization is as follows:
[0045] Step 4.1: Under the constraints of the UAV trajectory planning environment model, which includes a 3D environmental threat model and a global fitness function, the particle swarm is initialized, including initializing the population size, initial position and velocity of the particles, number of waypoints, and the Q-table and learning parameters required for reinforcement learning. Each particle is represented as a candidate flight trajectory consisting of several waypoints. Under the premise of satisfying the flight space constraints, the initial trajectory position of each particle is randomly generated, and the initial velocity parameters are set for the corresponding trajectory. At the same time, the Q-table and reinforcement learning related parameters are initialized.
[0046] Step 4.2: Substitute the initial trajectory position of the particles into the fitness function established in Step 2, calculate the fitness value of each particle, and update the individual historical optimal solution and the global optimal solution;
[0047] Step 4.3: The agent determines whether the current iteration state is successful or unsuccessful. Based on the iteration state, it selects an action to adjust the learning factor according to the Q table, calculates the reward value according to the change of fitness value in step 2, and updates the Q value in the Q table using the Q-learning algorithm, thereby achieving adaptive optimization of parameters.
[0048] Step 4.4: Update the particle's velocity and position using the adaptive inertia weight ω determined in Step 3 and the learning factor optimized by Q-learning;
[0049] Step 4.5: Determine if the maximum number of iterations has been reached. If not, return to step 4.2; otherwise, output the track coordinate sequence corresponding to the global optimal solution.
[0050] The advantages of this solution are:
[0051] This invention constructs a 3D trajectory planning environment model based on the terrain features and threat area distribution information of a real mountainous environment, reasonably constraining the UAV's flight space, making the environment modeling closer to actual application scenarios, and improving the safety and reliability of trajectory planning results. Combining UAV flight performance constraints and trajectory planning task requirements, a fitness function incorporating factors such as trajectory length, smoothness, and threat avoidance is constructed, making the optimization objectives more comprehensive and effectively improving the feasibility and effectiveness of trajectory planning results in actual flight tasks. A reinforcement learning mechanism is introduced into the particle swarm optimization algorithm. By establishing a state-action mapping relationship and adaptively adjusting the learning factors according to the reward signal, the particle swarm algorithm can dynamically change its search strategy according to different stages of the search process, thereby enhancing global search capabilities and improving local optimization performance, reducing the probability of the algorithm getting trapped in local optima, and improving convergence speed and planning accuracy. In summary, compared with existing technologies, this invention, by improving the parameter adjustment method of the particle swarm optimization algorithm and combining reasonable environment modeling and constraint design, can generate safer, smoother, and higher-quality UAV trajectory planning results under complex environmental conditions, improving the overall efficiency and stability of UAV flight missions. Attached Figure Description
[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0054] Figure 1 This is a schematic diagram of Q;
[0055] Figure 2 This is a schematic diagram illustrating the change in inertia weight.
[0056] Figure 3 Flowchart of the improved particle swarm optimization algorithm;
[0057] Figure 4 This is a schematic diagram showing the division of threatening obstacle zones;
[0058] Figure 5 This is a schematic diagram illustrating the mountain's environment.
[0059] Figure 6 This is a schematic diagram of the simulation results;
[0060] Figure 7 This is a schematic diagram of the process of the present invention. Detailed Implementation
[0061] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of systems consistent with some aspects of the invention as detailed in the appended claims.
[0062] refer to Figure 1-7 This invention provides a multi-strategy particle swarm unmanned aerial vehicle (UAV) trajectory planning method based on reinforcement learning, comprising the following steps:
[0063] Step 1: Obtain environmental information of the UAV flight site, obtain 3D elevation map data, model the mountain environment based on the 3D elevation map data, establish a 3D environmental threat model for UAV trajectory planning, and determine the boundary constraints and obstacle distribution of the flight area;
[0064] The environmental threat model is constructed as follows:
[0065] ;
[0066] in, A three-dimensional environmental threat model, , Represents the projected coordinates of the horizontal plane in the environmental model. This represents the terrain elevation value corresponding to the projection point. , and These represent the maximum boundary range in the three-dimensional space under the map coordinate system; based on the spatial distribution characteristics of the threat areas, several cylindrical threat areas are set in the simulation environment, specifically described as follows:
[0067] ;
[0068] in, Indicates the first There are several threat zones, with their center coordinates as follows: , radius is The height is ,
[0069] The establishment of the three-dimensional environmental threat model is based on the acquired three-dimensional elevation and terrain data, and is achieved by constructing the three-dimensional environmental threat model. and setting up various threat zones Together, they form the flight space constraints used for UAV trajectory planning, thus providing an environmental modeling foundation for subsequent reinforcement learning-based multi-strategy particle swarm UAV trajectory planning methods.
[0070] Step 2: When performing 3D path planning for a UAV, it is essential not only to ensure the path safely avoids obstacles but also to evaluate the overall quality of the path. Therefore, this paper adopts a multi-objective optimization function that comprehensively considers multiple factors such as path length, flight altitude, and turning angle to achieve a comprehensive evaluation of path quality. This multi-dimensional evaluation method can more effectively guide UAVs in efficient and safe path planning in complex environments. Assume the path is... It consists of 10 nodes, of which Indicates the first Each node position.
[0071] Taking into account factors such as path length, flight altitude, path smoothness, and collision threat, a global fitness function is constructed to evaluate the quality of the flight path, that is, a multi-objective path evaluation function is established, which transforms the flight path planning into a multi-objective optimization problem.
[0072] In step 2, when evaluating the effectiveness of the flight path planning, a weighted synthesis method with multiple objectives was used for path length, flight altitude, and path smoothness. The multi-objective path evaluation function, namely the global fitness function, is employed. The specific construction method is as follows:
[0073] ;
[0074] in, For path Length, For path Height coefficient, For path Smoothness coefficient, For path The collision coefficient; , , , Represents the weights corresponding to each evaluation function; where,
[0075] 1) Path length is one of the key metrics for evaluating the performance of UAV path planning algorithms. A shorter path means the UAV needs less time to complete the task, thus indicating higher quality path planning. When constructing the path length function, the starting and ending points of the path must first be determined. Then, three intermediate points are randomly selected between the starting and ending points. A continuous path is generated by smoothing the curve using cubic spline interpolation.
[0076] Path length function Calculate the sum of Euclidean distances between adjacent waypoints; the formula for calculating the path length is as follows:
[0077] ;
[0078] in, Represents the total number of waypoints;
[0079] 2) Ensuring the safe flight of drones necessitates limiting their flight altitude within specific areas. Furthermore, precise altitude control effectively reduces energy loss and operational complexity caused by altitude fluctuations. Optimizing flight altitude improves drone flight efficiency and ease of operation. (Flight altitude function) The assessment evaluates whether waypoint altitudes are within safe and energy-efficient ranges, penalizing waypoints that exceed minimum or maximum flight altitudes. This scheme considers flight altitude as a key objective function, as shown below:
[0080] ;
[0081] 3) In UAV 3D path planning, ensuring path smoothness is crucial for maintaining flight stability and safety. Especially in complex urban environments, a smooth path can reduce risks caused by sharp turns, rapid ascents, or descents. Path smoothness function. The curvature cost of the path is calculated by considering the changes in both horizontal turning angle and vertical climb angle to ensure flight stability; therefore, path planning should comprehensively consider the horizontal turning angle. and vertical climbing angle Given the smoothness in both directions, the objective function is defined as follows:
[0082] ;in,
[0083] ;
[0084] ;
[0085] ;
[0086] In the formula: and The horizontal turning angles for UAV 3D path planning are respectively. and vertical climbing angle The weight, Indicates the first [unmanned aerial vehicle] flight path The first waypoint and the first The displacement vector between each waypoint is used to describe the change in the flight direction of the UAV between adjacent waypoints;
[0087] 4) Ensuring a sufficient distance from obstacles is crucial for safe navigation, mitigating collision risks, and improving the overall reliability of drone operations. For example... Figure 6 As shown, in complex environments, drones must navigate around multiple obstacles while optimizing their flight paths to effectively achieve specific mission objectives.
[0088] This paper divides threats and obstacles into obstacle zones, threat zones, and secondary threat zones, each with different fitness function values, such as... Figure 7 As shown.
[0089] Collision threat function The area around the obstacle is divided into an obstacle zone, a threat zone, and a secondary threat zone; when the drone's waypoint falls into different zones, its distance from the center of the obstacle is used as the determining factor. Apply penalty values with different gradients; the closer the distance, the higher the penalty value.
[0090] ;
[0091] In the formula: Indicates the first [unmanned aerial vehicle] flight path The first waypoint to the second Euclidean distance from the center of each threat region; , , Let represent the radius of the danger zone, the radius of the threat zone, and the radius of the secondary threat zone, respectively, and satisfy . .
[0092] Step 3: Improve the standard particle swarm optimization algorithm. Specifically, introduce Q-learning reinforcement learning mechanism into the standard particle swarm optimization algorithm, establish Q table to store the mapping relationship between state and action, adaptively adjust the learning factor of the algorithm through the interaction between the agent and the environment, and combine adaptive inertia weight mechanism to balance global and local search capabilities, thereby constructing a multi-strategy particle swarm optimizer based on reinforcement learning mechanism.
[0093] Particle swarm optimization (PSO) simulates birds in a flock by designing massless particles. These particles have only two attributes: velocity and position. Velocity represents the speed of movement, and position represents the direction of movement. Each particle independently searches for the optimal solution in the search space, recording it as its current individual extreme value. This individual extreme value is shared with all other particles in the swarm. The optimal individual extreme value is then used as the current global optimal solution for the entire swarm. All particles in the swarm adjust their velocity and position based on their own current individual extreme value and the shared global optimal solution.
[0094] Particle position update expression:
[0095] ;
[0096] Particle velocity update expression:
[0097] ;
[0098] The inertial weight represents the degree of confidence in the current velocity direction; , This represents individual learning factors and group learning factors; , These represent two random numbers between (0,1) to enhance the randomness of the search.
[0099] To achieve adaptive adjustment of algorithm parameters, this scheme improves the particle swarm optimization (PSO) algorithm by constructing a multi-strategy PSO optimizer. Specifically, a reinforcement learning mechanism is introduced into the PSO algorithm, and the Q-learning algorithm is used to dynamically optimize the learning factor. Q-learning, as a model-free reinforcement learning method, continuously updates the state-action value function through interaction with the optimization environment, thereby learning the optimal parameter adjustment strategy. The learning results are stored in a Q-table, such as... Figure 1 As shown.
[0100] 1) Establish the Q-table: Using the iterative process of the particle swarm optimization algorithm as the reinforcement learning environment, and the particle swarm used for trajectory planning and search as the agent, define the state space and action space of the reinforcement learning; wherein, the state space includes the iteration success state. (The fitness of the new generation of particle swarms is better than that of the previous generation) and iteration failure state (The fitness of the new generation of particle swarm optimization has not been improved); the action space includes various learning factors for particle swarm optimization algorithms. and The adjustment strategy includes at least increasing, decreasing, or keeping it unchanged, in order to achieve adaptive adjustment of particle swarm search behavior;
[0101] 2) Define the Q-value update rule: During the iterative process of the particle swarm optimization algorithm, the agent updates the Q-value according to the current iteration state. Select the appropriate parameter adjustment action. The Q-table is updated based on the reward signal generated after the action is applied to the particle swarm optimization algorithm. The Q-value update rule is as follows:
[0102] ;
[0103] in, Indicates the current state. Indicates the current action. It's the learning rate. It is a discount factor. Indicates the first State-action pairs at the next iteration The corresponding Q value, This represents the Q value after one update. It is in state Take action below The reward value, Indicates the next state Then, based on the current Q-table, take the maximum Q-value for all optional actions.
[0104] In this algorithm, the state space and action space are defined as follows:
[0105] 1. State Space: Defines two states:
[0106] Iteration successful, meaning the optimal fitness of the new generation is better than that of the previous generation;
[0107] Iteration failure means that the optimal fitness of the new generation has not improved.
[0108] Action Space:
[0109] : + 0.1 * rand & - 0.05 * rand;
[0110] : and Remain unchanged;
[0111] : - 0.05 * rand & + 0.1 * rand
[0112] Here, rand represents a random number between (0,1).
[0113] In terms of action selection, the agent in Q-learning follows a SoftMax policy; for example, in the state... The agent selects an action. The probability can be calculated as follows:
[0114] ;
[0115] in Indicates the state Choose the action , This represents a defined parameter used to control randomness. The total number of actions. When When the size is large, the algorithm tends to explore; when When the Q value is smaller, there is a greater tendency to choose actions with high Q values.
[0116] After each iteration, the agent determines the reward signal based on the change in fitness:
[0117] ;
[0118] in, Representing the The fitness value of the next iteration. This represents the reward obtained in a single iteration.
[0119] The Q-table is then updated according to the following formula. When the algorithm continuously receives positive rewards, the Q-value of the corresponding action will be strengthened, thus giving it a higher selection probability in subsequent iterations. Conversely, when an action leads to a deterioration in fitness, its Q-value will be weakened, leading to its gradual elimination.
[0120] ;
[0121] Through the above mechanism, the algorithm can adaptively learn the optimal parameter adjustment strategy during the search process, thereby achieving individual learning factors. With group learning factor The dynamic balance adjustment improves the global convergence capability and local development accuracy of the particle swarm algorithm.
[0122] 3) Design adaptive inertia weights:
[0123] Analysis of the position and velocity iteration formulas in the standard particle swarm optimization algorithm reveals the inertia weight. The size of the inertia weight affects the size of the search space range of the particle swarm optimization algorithm. Assuming the learning factor remains constant, the inertia weight... The value of will affect the global search capability and local search capability of the particle swarm optimization algorithm: if A larger value results in a stronger global search capability, which is beneficial for escaping local optima but hinders convergence; conversely, a smaller value results in a stronger local search capability, which is beneficial for convergence but makes the algorithm prone to getting trapped in local optima. To optimize these shortcomings, a nonlinear dynamic update formula is introduced, which reduces the inertia weight ω as the number of iterations increases. The formula is as follows:
[0124] ;
[0125] In the formula, and These represent the current iteration number and the total number of iterations, respectively. and These represent the maximum and minimum values of the inertia weight, respectively. In this paper, they are set to 1.2 and 0.3, respectively, so as to focus on global search in the early stage of iteration and local development in the later stage.
[0126] according to Figure 2 It can be seen that by dynamically updating the inertia weight nonlinearly, the algorithm has a stronger global search capability in the early stage, focusing on the experience of individual particles, and a better local search capability in the later stage, focusing on the collective experience of the particle swarm, thus achieving a good trade-off between the algorithm's global search and local search capabilities.
[0127] The improved particle swarm optimization algorithm process is as follows: Figure 3 As shown, during the iteration process, the Q-table adaptively adjusts the parameters in the particle swarm optimization algorithm to enhance the algorithm's optimization capability.
[0128] Furthermore, as the number of iterations increases, the inertia weight is also adjusted, aiming to balance the algorithm's exploration and development capabilities.
[0129] Step 4: Under the constraints of the 3D environmental threat model constructed in Step 1 and the global fitness function established in Step 2, initialize the population size, initial particle positions and velocities, number of waypoints, and the Q-table and learning parameters required for reinforcement learning. Use a multi-strategy particle swarm optimizer to iteratively search for waypoints in the 3D environmental threat model. During the iteration process, calculate the reward signal based on the global fitness function value in Step 2 and update the Q-table. Perform trajectory planning optimization and finally output the UAV 3D flight trajectory with the best fitness.
[0130] The specific process of performing trajectory planning and optimization is as follows:
[0131] Step 4.1: Under the UAV trajectory planning environment model, initialize the particle swarm, representing each particle as a candidate flight trajectory consisting of several waypoints; under the premise of satisfying the flight space constraints, randomly generate the initial trajectory position of each particle and set the initial velocity parameters for the corresponding trajectory; at the same time, initialize the Q-table and reinforcement learning related parameters.
[0132] Step 4.2: Substitute the initial trajectory position of the particles into the fitness function established in Step 2, calculate the fitness value of each particle, and update the individual historical optimal solution and the global optimal solution;
[0133] Step 4.3: Determine whether the current iteration state is successful or failed; based on the iteration state, select the action to adjust the learning factor according to the Q table, and calculate the reward value r according to the change of fitness value in step 2, and then use the Q-learning algorithm to update the Q value in the Q table, thereby achieving adaptive optimization of parameters;
[0134] Step 4.4: Update the particle's velocity and position using the adaptive inertia weight ω determined in Step 3 and the learning factor optimized by Q-learning;
[0135] Step 4.5: Determine if the maximum number of iterations has been reached. If not, return to step 4.2; otherwise, output the track coordinate sequence corresponding to the global optimal solution.
[0136] This paper utilizes the threat posed by mountainous terrain for UAV flight path planning. The region has a maximum elevation difference of 2km, with undulating terrain and numerous ravines. A no-fly zone (purple cylinder) is established within this area. Figure 5 As shown. It is assumed that there are no tall trees in the mountainous area, only low shrubs. Furthermore, it is assumed that external factors such as weather will not affect the drone's flight.
[0137] Simulation results are as follows Figure 6 As shown, RLPSO can produce a smooth trajectory at a lower flight altitude, effectively minimizing energy consumption.
[0138] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these changes and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A multi-strategy particle swarm unmanned aerial vehicle (UAV) trajectory planning method based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Obtain environmental information of the UAV flight site, obtain 3D elevation map data, model the mountain environment based on the 3D elevation map data, establish a 3D environmental threat model for UAV trajectory planning, and determine the boundary constraints and obstacle distribution of the flight area; Step 2: Taking into account factors such as path length, flight altitude, path smoothness, and collision threat, construct a global fitness function to evaluate the quality of the flight path, that is, establish a multi-objective path evaluation function, and transform the flight path planning into a multi-objective optimization problem; Step 3: Improve the particle swarm optimization algorithm. Specifically, introduce Q-learning reinforcement learning mechanism into the standard particle swarm optimization algorithm, establish a Q table to store the mapping relationship between state and action, adaptively adjust the learning factor of the algorithm through the interaction between the agent and the environment, and combine an adaptive inertial weight mechanism to balance global and local search capabilities, thereby constructing a multi-strategy particle swarm optimizer based on reinforcement learning mechanism. Step 4: Under the constraints of the 3D environmental threat model and the global fitness function, the particle swarm is initialized. The multi-strategy particle swarm optimizer is used to iteratively search for path points in the 3D environmental threat model. During the iteration process, the reward signal is calculated based on the global fitness function value in Step 2 and the Q table is updated. The trajectory planning optimization is performed, and finally the UAV 3D flight trajectory with the best fitness is output.
2. The method as described in claim 1, characterized in that, In step 1, the construction of the environmental threat model is as follows: ； in, A three-dimensional environmental threat model, , Represents the projected coordinates of the horizontal plane in the environmental model. This represents the terrain elevation value corresponding to the projection point. , and These represent the maximum boundary range in the three-dimensional space under the map coordinate system; based on the spatial distribution characteristics of the threat areas, several cylindrical threat areas are set in the simulation environment, as detailed below: ； in, Indicates the first There are several threat zones, with their center coordinates as follows: , radius is The height is , The establishment of the three-dimensional environmental threat model is based on the acquired three-dimensional elevation and terrain data, and is achieved by constructing the three-dimensional environmental threat model. and setting up various threat zones Together, they form the flight space constraints used for UAV trajectory planning, thus providing an environmental modeling foundation for subsequent reinforcement learning-based multi-strategy particle swarm UAV trajectory planning methods.
3. The method as described in claim 1, characterized in that, In step 2, the multi-objective path evaluation function, i.e., the global fitness function... The specific construction method is as follows: ； in, For path Length, For path Height coefficient, For path Smoothness coefficient, For path The collision coefficient; , , , Represents the weights corresponding to each evaluation function; where, 1) Path length function : Calculate the sum of Euclidean distances between adjacent waypoints; ； in, Represents the total number of waypoints; 2) Flight altitude function Assess whether waypoint altitudes are within safe and energy-efficient ranges, and penalize waypoints that exceed minimum or maximum flight altitudes; ； 3) Path smoothness function : By combining the changes in horizontal turning angle and vertical climb angle, the curvature cost of the path is calculated to ensure flight stability; ； ； ； ； In the formula: and The horizontal turning angles for UAV 3D path planning are respectively. and vertical climbing angle The weight, Indicates the first [unmanned aerial vehicle] flight path The first waypoint and the first The displacement vector between each waypoint is used to describe the change in the flight direction of the UAV between adjacent waypoints; 4) Collision Threat Function The area around the obstacle is divided into an obstacle zone, a threat zone, and a secondary threat zone; when the drone's waypoint falls into different zones, its distance from the center of the obstacle is considered. Apply penalty values with different gradients; the closer the distance, the higher the penalty value. ； In the formula: Indicates the first [unmanned aerial vehicle] flight path The first waypoint to the second Euclidean distance from the center of each threat region; , , Let represent the radius of the danger zone, the radius of the threat zone, and the radius of the secondary threat zone, respectively, and satisfy . .
4. The method as described in claim 1, characterized in that, Step 3, the construction of the multi-policy particle swarm optimizer based on reinforcement learning mechanism specifically includes: 1) Establish the Q-table: Using the iterative process of the particle swarm optimization algorithm as the reinforcement learning environment, and the particle swarm used for trajectory planning and search as the agent, define the state space and action space of the reinforcement learning; wherein, the state space includes the iteration success state. and iteration failure status The action space includes various learning factors specific to the particle swarm optimization algorithm. and The adjustment strategy includes at least increasing, decreasing, or keeping it unchanged, in order to achieve adaptive adjustment of particle swarm search behavior; 2) Define the Q-value update rule: During the iterative process of the particle swarm optimization algorithm, the agent updates the Q-value according to the current iteration state. Select the appropriate parameter adjustment action. The Q-table is updated based on the reward signal generated after the action is applied to the particle swarm optimization algorithm. The Q-value update rule is as follows: ； in, Indicates the current state. Indicates the current action. It's the learning rate. It is a discount factor. Indicates the first State-action pairs at the next iteration The corresponding Q value, This represents the Q value after one update. It is in state Take action below The reward value, Indicates the next state Then, based on the current Q-table, take the maximum Q-value for all optional actions; 3) Design of adaptive inertia weight: Introduce a nonlinear dynamic update formula to decrease the inertia weight ω as the number of iterations increases, as shown in the following formula: ； In the formula, and These represent the current iteration number and the total number of iterations, respectively. and These represent the maximum and minimum values of the inertia weight, respectively, thus enabling a focus on global search in the early stages of iteration and local development in the later stages.
5. The method as described in claim 1, characterized in that, In step 4, the specific process of performing trajectory planning and optimization is as follows: Step 4.1: Under the constraints of the UAV trajectory planning environment model, which includes a 3D environmental threat model and a global fitness function, the particle swarm is initialized, including initializing the population size, initial position and velocity of the particles, number of waypoints, and the Q-table and learning parameters required for reinforcement learning. Each particle is represented as a candidate flight trajectory consisting of several waypoints. Under the premise of satisfying the flight space constraints, the initial trajectory position of each particle is randomly generated, and the initial velocity parameters are set for the corresponding trajectory. At the same time, the Q-table and reinforcement learning related parameters are initialized. Step 4.2: Substitute the initial trajectory position of the particles into the fitness function established in Step 2, calculate the fitness value of each particle, and update the individual historical optimal solution and the global optimal solution; Step 4.3: The agent determines whether the current iteration state is successful or unsuccessful. Based on the iteration state, it selects an action to adjust the learning factor according to the Q table, calculates the reward value according to the change of fitness value in step 2, and updates the Q value in the Q table using the Q-learning algorithm, thereby achieving adaptive optimization of parameters. Step 4.4: Update the particle's velocity and position using the adaptive inertia weight ω determined in Step 3 and the learning factor optimized by Q-learning; Step 4.5: Determine if the maximum number of iterations has been reached. If not, return to step 4.2; otherwise, output the track coordinate sequence corresponding to the global optimal solution.