Unmanned aerial vehicle multi-target path planning method based on hierarchical reinforcement learning
By combining hierarchical reinforcement learning and obstacle perception models with a soft actor-critic algorithm, the efficiency and accuracy issues in UAV path planning are solved, achieving optimal path planning and deadlock avoidance in complex environments, and is applicable to multi-objective path planning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF TECH
- Filing Date
- 2026-01-22
- Publication Date
- 2026-06-12
AI Technical Summary
Existing drone inventory path planning methods are inefficient and inaccurate in complex environments, difficult to generalize to different scenarios, and prone to deadlock.
A hierarchical reinforcement learning-based approach is adopted to model multi-objective path planning as a semi-Markov decision process. The path planning is combined with an obstacle perception attention model and a soft actor-critic (SAC) algorithm, and a deadlock breaking mechanism is introduced to ensure path optimality and reachability.
It improves the efficiency and accuracy of multi-target path planning for UAVs, can generalize in different environments, effectively avoids deadlock, and achieves path optimization and reachability.
Smart Images
Figure CN122192299A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a multi-objective path planning method based on hierarchical reinforcement learning. It provides a novel approach for multi-objective path planning in unmanned aerial vehicles (UAVs) in complex 3D environments, belonging to the field of UAV automatic control technology. Background Technology
[0002] In modern logistics and vehicle warehousing management, regular inventory checks are a crucial step in ensuring inventory accuracy. Traditional manual inventory checks are inefficient, prone to errors, and costly. With the development of drone technology and RFID identification technology, using RFID-based drone inventory checks can achieve non-contact, rapid identification of vehicles and other items in the warehouse area. However, in practical applications, the efficiency and accuracy of inventory checks are limited by factors such as the RFID reader's identification directionality, identification distance, obstacle obstruction, and vehicle distribution. Therefore, rationally planning the drone inventory check path becomes critical. In this invention, we abstract the inventory check path planning problem into a multi-objective path planning problem for research. Existing multi-objective path planning methods often adopt a path splicing paradigm of "global path planning + local obstacle avoidance," which lacks guarantees of path optimality and is difficult to generalize to different scenarios in practical applications.
[0003] This invention, "A Multi-Objective Path Planning for Unmanned Aerial Vehicles Based on Hierarchical Reinforcement Learning," takes the aforementioned problem as its starting point. The proposed path planning method can solve multi-objective path planning under different environments while ensuring path optimality. By modeling the multi-objective path planning problem as a semi-Markov decision process, multi-objective path planning is performed within an option-based hierarchical reinforcement learning framework. The higher layers use an obstacle perception attention model to perform order planning based on physical reachability, while the lower layers, considered as internal strategies of options, use the Soft Actor-Critic (SAC) algorithm for obstacle avoidance path planning. Simultaneously, an unmanned aerial vehicle kinematic model is introduced to make the path more realistic. Simulation experiments demonstrate the effectiveness and universality of the proposed method. Summary of the Invention
[0004] (1) Purpose: The purpose of this invention is to provide a multi-objective path planning method, which can plan a UAV flight path to visit multiple targets in a complex three-dimensional environment and generalize to different scenarios while ensuring the optimality of the path.
[0005] (2) Technical solution: The main contents and steps of the present invention "A Multi-Objective Path Planning for Unmanned Aerial Vehicles Based on Hierarchical Reinforcement Learning" are as follows: First, the multi-objective path planning scenario is mathematically represented. Under the hierarchical reinforcement learning framework based on options, the multi-objective path planning is modeled as a semi-Markov decision process. Then, the high layer uses an attention model based on obstacle perception to perform order planning based on physical accessibility. The low layer models the path planning as a Markov decision process and trains it using the soft actor-critic algorithm (SAC). A deadlock breaking mechanism is introduced to solve the deadlock problem that reinforcement learning methods often face in path planning.
[0006] The technical solution adopted in this invention is a multi-objective path planning method for unmanned aerial vehicles based on hierarchical reinforcement learning, and the specific steps are as follows:
[0007] Step 1: Formulate a mathematical expression for the multi-objective path planning problem and model it as a semi-Markov decision process within an option-based hierarchical reinforcement learning framework.
[0008] Step two: Use an obstacle perception attention model to perform sequence planning at higher levels.
[0009] Step 3: Model the low-level path planning as a Markov decision process and train it using the Soft Actor-Critic (SAC) algorithm.
[0010] Step four: Introduce a deadlock breaking mechanism at the lower level to ensure that an unobstructed path from the starting point to the target point can be successfully planned.
[0011] The mathematical expression of the multi-objective path planning environment described in step one is as follows:
[0012] Assume a three-dimensional flyable space is ,in The area represents a collision-free zone in the environment where the drone can move freely. The set of target points that the drone needs to access is specified as... The environment contains several cylindrical obstacles, collectively represented as... ,in The center of the obstacle. Let its equivalent radius in the configuration space be... Represents the height of the obstacle. Extract obstacle nodes. Together with the set of target points, they form a set of nodes. The drone configuration is provided by Description, in which, Represents the location of the drone. Represents the speed of the drone. This represents the drone's acceleration. The order of access can be determined by the arrangement of node indices. Describe this. Unlike the traditional method of using Euclidean distance as the path length between two points, given two target points... The collision-free path cost is defined as the actual path length:
[0013] ;
[0014] in Let be the number of time steps the drone takes to travel from the starting point to the target point. Therefore, the cost for the drone to visit all target points is:
[0015] ;
[0016] Therefore, the final global path planning problem can be expressed as:
[0017] ;
[0018] in It is the set of all permutations that contain the target point.
[0019] Among them, the obstacle perception attention model described in step two, given a set of nodes and obstacle sets Construct a visibility graph based on geometric relationships. If node and An edge is considered visible if the line connecting two points does not intersect any obstacle; define the set of edges. Invisible edges are removed, giving the visibility graph natural sparsity and structural priors. Graph neural networks (GNNs) are then introduced to extract structural prior knowledge, thereby explicitly integrating obstacle information into the attention distribution.
[0020] Construct a visibility adjacency matrix ,in, Represents a node and visible, Represents a node and The objects are occluded by obstacles and are not visible. The encoder uses a graph attention network to compute the initial embedding for each node. Input feature vector ,in, Let be the coordinates of the node. One-hot encoding is used to distinguish between target nodes and obstacle nodes. First, the input is mapped to a high-dimensional space:
[0021] ;
[0022] In each layer of the graph attention neural network, nodes Aggregate its neighbors The attention coefficient is defined as follows:
[0023] ;
[0024] Therefore, the node feature update equation is:
[0025] ;
[0026] go through Layer iterations output the final topological features. And by aggregating the embeddings of all nodes, the final graph embedding is obtained. :
[0027] ;
[0028] Decoder in The context of a time step comes from the encoder and consists of graph embeddings, the previous node embedding, and the first node embedding. When When using and As a placeholder, that is:
[0029] ;
[0030] Then, a new context embedding is computed using a multi-head attention mechanism. The keys and values come from the node embedding. We only embed through context. Calculate query vector :
[0031] ;
[0032] Then, the adjacency matrix mask is entered, and cross-attention is used to calculate the values for all candidate nodes. Selection priority :
[0033] ;
[0034] We interpret these compatibility values as non-normalized log probabilities and use softmax to compute the final output probability vector. :
[0035] ;
[0036] In step three, the low-level path planning employs the Soft Actor-Critic (SAC) algorithm for learning, designed to handle continuous control problems over a given period. SAC achieves both exploratory improvement and stabilization of the learning process by maximizing the entropy-regularized reward function.
[0037] ;
[0038] in This is a temperature coefficient used to balance the weights of reward maximization and policy entropy. For the strategy in the state The higher the entropy, the stronger the randomness of the strategy and the stronger the exploration ability.
[0039] The internal policy of each option acts as an independent continuous controller, and its training process can be regarded as a local optimization of the state transitions within the option, thereby enabling the execution performance of the lower layer to meet the requirements of the higher-level task.
[0040] The deadlock breaking mechanism described in step four consists of three modules that share a feature encoder: considering the complex geometry and obstacle distribution characteristics of the local feasible space in the 3D environment, a lightweight convolutional neural network (CNN) is used to analyze the current state. Departure to destination The local environment is encoded as a feature vector. A lightweight multilayer perceptron (MLP) is used as the risk prediction module for feature representation-based risk assessment. Estimate the potential risk of deadlock in the current state:
[0041] ;
[0042] in, This indicates a risk index. A risk threshold is set. Based on the risk prediction results, the system decides which strategy to implement. This mechanism allows the system to identify risks before a deadlock occurs, enabling proactive intervention rather than reactive handling.
[0043] The strategy module consists of escape strategies and exploration strategies. These are used to generate behavior under high-risk and low-risk conditions, respectively. If the risk index exceeds a threshold, i.e. If the current state indicates a risk of deadlock, an escape strategy will be executed; if the risk index is below a threshold... If the current state indicates a risk of deadlock, then an exploratory strategy is executed. The dual-policy network embodies the dynamic allocation of the decision space, achieving policy reconfiguration through risk-driven mechanisms.
[0044] ;
[0045] The symbols are explained as follows:
[0046] A three-dimensional, collision-free space where drones can fly;
[0047] A collection of obstacles;
[0048] Set of target points;
[0049] Target point;
[0050] Number of target points;
[0051] Set of obstacle nodes;
[0052] Obstacle nodes;
[0053] Number of obstacle nodes;
[0054] The set of all nodes;
[0055] Visibility map;
[0056] Number of obstacles;
[0057] No. One obstacle;
[0058] The center of the obstacle;
[0059] The equivalent radius of the obstacle in the configuration space;
[0060] No. The height of the obstacle;
[0061] The distance between nodes i and j;
[0062] Target point access sequence;
[0063] The index of the i-th target point in the access order;
[0064] The cost of drones accessing all target points;
[0065] The set of all permutations containing the target point;
[0066] Visibility adjacency matrix;
[0067] The final output is a probability vector;
[0068] node The input feature vector;
[0069] Initial embedding of nodes
[0070] No. Layer node embedding;
[0071] No. Layer node embedding;
[0072] Candidate Nodes Priority of selection;
[0073] Query vector;
[0074] The final output is a probability vector;
[0075] Local environmental feature vectors;
[0076] Risk index.
[0077] Set risk thresholds;
[0078] Obstacle avoidance strategies;
[0079] Exploration strategy;
[0080] Technical breakthroughs and performance advantages of this invention:
[0081] Regarding the technological breakthroughs of this method, firstly, this invention proposes a multi-objective path planning method for UAVs based on hierarchical reinforcement learning. This method solves the order planning and path planning involved in multi-objective path planning hierarchically, generalizing the strategy to different scenarios while ensuring path optimality. Unlike the traditional path splicing paradigm of "global planning + local obstacle avoidance," this invention models the multi-objective path planning problem as a semi-Markov decision process. At the high level, an obstacle perception attention model is used to solve for the optimal access order while considering physical reachability; the low-level path planning is modeled as a Markov decision process and trained using the soft actor-critic (SAC) algorithm. To address the deadlock problem, a deadlock breaking mechanism based on active prediction is proposed to ensure the reachability of low-level paths. Simulation experiments verify the effectiveness of the proposed method.
[0082] Regarding performance advantages, the present invention, "A Multi-Objective Path Planning for Unmanned Aerial Vehicles Based on Hierarchical Reinforcement Learning," has the following advantages compared to existing technologies:
[0083] 1) This method can effectively improve the efficiency of multi-target path planning for UAVs and solve the problem of insufficient path optimality in traditional methods.
[0084] 2) This method can be generalized to different scenarios and exhibits good performance in different environments.
[0085] 3) This method solves the deadlock problem faced by traditional reinforcement learning methods in path planning problems. Attached Figure Description
[0086] Figure 1 This is a schematic diagram of the multi-target path planning results for UAVs.
[0087] Figure 2 This is a graph of the reward function.
[0088] Figure 3 This is a speed curve graph for the drone. Detailed Implementation
[0089] The design methods of each part of this invention will be further explained below:
[0090] This invention discloses a multi-objective path planning method for unmanned aerial vehicles (UAVs) based on hierarchical reinforcement learning, the specific steps of which are as follows:
[0091] Step 1: Establish a mathematical model;
[0092] Assume a three-dimensional flyable space is The set of target points that the drone needs to visit is specified as Obstacles are uniformly represented as obstacle nodes Together with the set of target points, they form a set of nodes. The drone configuration is provided by The description states that the access order can be determined by the arrangement of node indices. Describe it.
[0093] Its collision-free path cost is defined as the actual path length:
[0094] ;
[0095] in Let be the number of time steps the drone takes to travel from the starting point to the target point. Therefore, the cost for the drone to visit all target points is:
[0096] ;
[0097] Therefore, the global path planning problem can ultimately be expressed as:
[0098] ;
[0099] in It is the set of all permutations that contain the target point.
[0100] Step 2: Use an obstacle-aware attention mechanism for sequence planning;
[0101] Construct a visibility graph based on the set of all nodes. Furthermore, a graph neural network (GNN) is introduced to extract prior structural knowledge. A visibility adjacency matrix is constructed. ,in, Represents a node and visible, Represents a node and The objects are occluded by obstacles and are not visible. The encoder uses a graph neural network to compute the initial embedding, for each node... Input feature vector ,in, Let be the coordinates of the node. One-hot encoding is used to distinguish between target nodes and obstacle nodes. First, the input is mapped to a high-dimensional space:
[0102] ;
[0103] In each layer of the Graph Attention Neural Network (GAT), nodes Aggregate its neighbors The attention coefficient is defined as follows:
[0104] ;
[0105] Therefore, the node feature update equation is:
[0106] ;
[0107] go through Layer iterations output the final topological features. And by aggregating the embeddings of all nodes, the final graph embedding is obtained. :
[0108] ;
[0109] Decoder in The context of a time step comes from the encoder and consists of graph embeddings, the previous node embedding, and the first node embedding. When When using and As a placeholder, that is:
[0110] ;
[0111] Then, a new context embedding is calculated using a multi-head attention mechanism. The keys and values come from the node embedding. We only embed through context. Calculate query vector :
[0112] ;
[0113] Then, the adjacency matrix mask is used, and cross-attention is calculated for all candidate nodes. Selection priority :
[0114] ;
[0115] These compatibility metrics are interpreted as non-normalized log probabilities, and the final output probability vector is computed using softmax. :
[0116] ;
[0117] Step 3: Use the Soft Actor-Critic (SAC) algorithm for low-level path planning;
[0118] The SAC algorithm is used for low-level path planning learning, and exploratory improvement and stabilization of the learning process are achieved by maximizing the reward function of entropy regularization:
[0119] ;
[0120] in This is a temperature coefficient used to balance the weights of reward maximization and policy entropy. For the strategy in the state The higher the entropy, the stronger the randomness of the strategy and the stronger the exploration ability.
[0121] The internal policy of each option acts as an independent continuous controller, and its training process can be regarded as a local optimization of the internal state transitions of the option, so that the execution performance of the option meets the requirements of the high-level task.
[0122] Step 4: Deadlock breaking mechanism ensures lower-level reachability;
[0123] A deadlock breaking mechanism based on risk prediction is introduced, comprising a shared feature encoding module, a risk prediction module, and a dual-policy execution module. The shared feature encoding module is used to extract features from the current UAV state and its local environment to form an environment representation vector.
[0124] The risk prediction module outputs a risk index of the current state being trapped in deadlock based on the environmental representation vector, and compares it with a preset risk threshold. When the risk index is lower than the threshold, the system executes a conventional exploration strategy to maintain the efficiency of path planning; when the risk index is higher than the threshold, the system automatically switches to an escape strategy to guide the drone to actively move away from the potential deadlock area.
[0125] By anticipating risks and switching strategies before deadlocks actually occur, deadlock problems can be proactively avoided, rather than through traditional passive handling, thereby significantly improving the accessibility of low-level path planning and the overall task completion rate.
[0126] Experimental charts and related data, such as Figure 1-3 As shown.
Claims
1. A multi-objective path planning method for unmanned aerial vehicles (UAVs) based on hierarchical reinforcement learning, characterized in that, The specific steps of this method are as follows: Step 1: Formulate a mathematical expression for the multi-objective path planning problem and model it as a semi-Markov decision process within an option-based hierarchical reinforcement learning framework. Step 2: Use an obstacle perception attention model for sequence planning at higher levels; Step 3: Model the low-level path planning as a Markov decision process and train it using the soft actor-critic algorithm (SAC). Step four: Introduce a deadlock breaking mechanism at the lower level to ensure that an unobstructed path from the starting point to the target point can be successfully planned.
2. The UAV multi-objective path planning method based on hierarchical reinforcement learning according to claim 1, characterized in that, The mathematical expression of the multi-objective path planning environment described in step one is as follows: Assume a three-dimensional flyable space is ,in The area in the middle represents the collision-free zone in the environment where the drone can move freely; the set of target points that the drone needs to visit is specified as... ; The environment contains several cylindrical obstacles, collectively represented as ,in The center of the obstacle. Let its equivalent radius in the configuration space be... Indicates the height of the obstacle; Extract obstacle nodes Together with the set of target points, they form a set of nodes. The configuration of the drone is provided by Description, in which, Represents the location of the drone. Represents the speed of the drone. Represents the acceleration of the drone; The access order is determined by the arrangement of node indices. Describe; given two target points The cost of its collision-free path is defined as the actual path length: ； in Let be the number of time steps the drone takes to travel from the starting point to the target point. The cost of the drone visiting all target points is: ； The final global path planning problem is formulated as follows: ； in It is the set of all permutations that contain the target point.
3. The UAV multi-objective path planning method based on hierarchical reinforcement learning according to claim 1, characterized in that, The obstacle perception attention model described in step two, given a set of nodes. and obstacle sets Construct a visibility graph based on geometric relationships. If node and An edge is considered visible if the line connecting two points does not intersect any obstacle; define the set of edges. Invisible edges are removed, giving the visibility graph natural sparsity and structural priors. Graph neural networks (GNNs) are then introduced to extract structural prior knowledge, thereby explicitly integrating obstacle information into the attention distribution. Construct a visibility adjacency matrix ,in, Represents a node and visible, Represents a node and The nodes are occluded by obstacles and are not visible; the encoder uses a graph attention network to compute the initial embedding for each node. Input feature vector ,in, Let be the coordinates of the node. It uses one-hot encoding to distinguish between target nodes and obstacle nodes; First, the input is mapped to a high-dimensional space: ； In each layer of the graph attention neural network, nodes Aggregate its neighbors The attention coefficient is defined as follows: ； The node feature update equation is: ； go through Layer iterations output the final topological features. And by aggregating the embeddings of all nodes, the final graph embedding is obtained. : ； Decoder in The context of a time step comes from the encoder and consists of graph embeddings, the previous node embedding, and the first node embedding; when When using and As a placeholder, that is: ； Then, a new context embedding is computed using a multi-head attention mechanism. The keys and values come from the node embedding. , through context embedding Calculate query vector : ； Then, the adjacency matrix mask is entered, and cross-attention is used to calculate the values for all candidate nodes. Selection priority : ； These compatibility metrics are interpreted as non-normalized log probabilities, and the final output probability vector is computed using softmax. : 。 4. The UAV multi-objective path planning method based on hierarchical reinforcement learning according to claim 1, characterized in that, The low-level path planning described in step three employs the Soft Actor-Critic Algorithm (SAC) for learning, designed to handle continuous control problems over a given duration. SAC achieves both exploratory enhancement and stabilization of the learning process by maximizing the entropy-regularized reward function. ； in This is a temperature coefficient used to balance the weights of reward maximization and policy entropy. For the strategy in the state The entropy is lowered by the policy. The higher the entropy, the stronger the randomness of the policy and the stronger the exploration ability. The policy within each option is an independent continuous controller. Its training process can be regarded as a local optimization of the state transition within the option, so that the execution performance of the lower layer meets the requirements of the higher layer task.
5. The UAV multi-objective path planning method based on hierarchical reinforcement learning according to claim 1, characterized in that, The deadlock breaking mechanism described in step four consists of three modules, sharing a feature encoder: considering the complex geometry and obstacle distribution characteristics of the local feasible space in the 3D environment, a lightweight convolutional neural network (CNN) is used to analyze the current state. Departure to destination The local environment is encoded as a feature vector. A lightweight multilayer perceptron (MLP) is used as the risk prediction module for feature representation-based risk prediction. Estimate the potential risk of deadlock in the current state: ； in, Indicates the risk index; sets the risk threshold. The appropriate strategy will be implemented based on the risk prediction results.
6. The UAV multi-objective path planning method based on hierarchical reinforcement learning according to claim 5, characterized in that, The strategy module consists of escape strategies and exploration strategies; these are used to generate behaviors under high-risk and low-risk conditions, respectively. If the risk index is higher than the threshold, that is If this indicates that the current state is at risk of deadlock, then an escape strategy will be executed; If the risk index is below the threshold If the current state indicates a risk of deadlock, an exploratory strategy is executed. The dual-policy network embodies the dynamic allocation of the decision space, achieving policy reconfiguration through risk-driven mechanisms. 。