Unmanned aerial vehicle path planning method and device based on heuristic monte carlo tree search

Abstract

This invention relates to a method and apparatus for UAV path planning based on heuristic Monte Carlo tree search. The method includes: modeling the UAV path planning task as a Markov decision problem, and selecting the optimal leaf node from the root node of an initial search tree using a heuristic UCB strategy; creating a new leaf node on the optimal leaf node; simulating the path from the new leaf node using a goal-oriented and smoothness-constrained action selection strategy until the task objective point is reached, and calculating the total reward value for each node; pruning and normalizing the total reward value, and then propagating it back to the root node along the task objective point, updating the visit count and total reward value of all nodes along the way, until the iteration terminates, obtaining an updated search tree; and extracting the optimal path based on the updated search tree. This invention enables efficient path optimization under complex constraints, meeting the real-time and safety requirements of autonomous navigation for intelligent UAVs.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a UAV path planning method and apparatus based on heuristic Monte Carlo tree search. Background Technology

[0002] With the rapid iteration of technologies such as unmanned platforms and mobile robots, autonomous navigation for UAVs has become a core support for realizing scenarios such as unmanned delivery, intelligent warehousing, and autonomous driving. Path planning, as a crucial component of autonomous navigation, directly determines the operational efficiency and safety of UAVs. Currently, UAV path planning technology has been widely applied in various practical scenarios, such as disaster relief, material delivery, and autonomous obstacle avoidance navigation. The goal of path planning is to find an optimal path for the UAV from its starting point to its destination, while ensuring collision-free operation, compliance with kinematic constraints, and minimization of overall cost. This cost assessment can be flexibly set according to specific needs, including multiple indicators such as path length, flight time, energy consumption, and smoothness.

[0003] Intelligent UAV path planning is a core component of autonomous navigation technology. Existing technologies can be broadly categorized into several types: The first type is traditional path search methods, such as... The first category consists of algorithms such as Dijkstra's algorithm, which uses heuristic functions to guide path generation based on grid or graph structures. While it can quickly find the shortest path in static environments, discretization of continuous spaces can lead to non-smooth paths, and it has poor adaptability in dynamic environments. It relies on manually designed cost functions, and its search complexity increases significantly under high-dimensional constraints. The second category comprises intelligent optimization algorithms, such as particle swarm optimization and genetic algorithms. These algorithms primarily handle multi-constraint scenarios through group iterative optimization. However, they suffer from convergence stability dependent on parameter tuning, susceptibility to local optima, and sensitivity to initial solutions. Their replanning response is slow in dynamic environments. The third category includes sampling-based methods, such as Rapidly Exploring Random Trees. Trees (RRT), etc., construct path trees in a continuous space through random sampling. They are suitable for high-dimensional state spaces, but have high path redundancy, low exploration efficiency, and are prone to getting stuck in local minima in dense obstacle environments. The cost of replanning is also high. The fourth category is deep learning and reinforcement learning methods, which are based on neural networks to fit state-action value functions. They can achieve end-to-end planning, but require massive amounts of labeled data. They have poor generalization ability in unseen environments, low transparency of decision logic, and difficulty in verifying security. The cost of retraining in dynamic environments is also high.

[0004] MCTS, as an intelligent method, finds the optimal decision in a given domain by sampling in the decision space and constructing a search tree based on the results. It has a significant impact on sequential decision problems such as games and planning. Although traditional MCTS performs well in discrete decision spaces, when directly applied to UAV path planning, it suffers from low search efficiency in continuous state-action spaces, insufficient constraint handling, poor path feasibility, and an imperfect exploration-exploitation balancing strategy, making it difficult to meet real-time requirements. Summary of the Invention

[0005] Therefore, it is necessary to provide a method and apparatus for UAV path planning based on heuristic Monte Carlo tree search that can balance computational efficiency, constraint adaptability, and dynamic robustness, in order to address the above-mentioned technical problems.

[0006] A method for UAV path planning based on heuristic Monte Carlo tree search, the method comprising: The UAV path planning task is obtained, the UAV path planning task is modeled as a Markov decision problem, and an initial search tree is constructed. Starting from the root node of the initial search tree, the optimal leaf node is selected through a heuristic UCB strategy. Generate effective actions based on the kinematic model, calculate the new position and heading angle, and create a new leaf node on the optimal leaf node; Starting from the new leaf node, a goal-oriented and smoothness-constrained action selection strategy is used to simulate the path until the task objective is reached, and the total reward value of each node is calculated during the path simulation process. After pruning and normalizing the total reward value, the algorithm propagates backward from the task objective point back to the root node, updating the visit count and total reward value of all nodes along the way until the iteration terminates, resulting in the updated search tree. Based on the updated search tree, starting from the root node, select the sequence of child nodes with the most visits to extract the optimal path.

[0007] On the other hand, a drone path planning device based on heuristic Monte Carlo tree search is also provided, including: The selection module is used to acquire the UAV path planning task, model the UAV path planning task as a Markov decision problem, construct an initial search tree, and select the optimal leaf node from the root node of the initial search tree through a heuristic UCB strategy. An extension module is used to generate effective actions based on the kinematic model, calculate new positions and heading angles, and create new leaf nodes on the optimal leaf node; The simulation module is used to simulate the path from the new leaf node, using a goal-oriented and smoothness-constrained action selection strategy, until the task objective point is reached, and to calculate the total reward value of each node during the path simulation process. The backtracking module is used to prune and normalize the total reward value, then propagate it back to the root node along the task target point, updating the number of visits and the total reward value of all nodes along the way, until the iteration terminates, and obtains the updated search tree. The path extraction module is used to extract the optimal path based on the updated search tree, starting from the root node and selecting the sequence of child nodes with the most visits.

[0008] Compared with existing technologies, the UAV path planning method and apparatus based on heuristic Monte Carlo tree search provided by this invention have the following advantages: This invention discretizes the UAV path planning problem into a Markov decision problem and accelerates node selection, improving search efficiency and iterative convergence speed, based on a heuristic UCB strategy for optimal leaf node selection. It employs a goal-oriented and smoothness-constrained action selection strategy for path simulation, ensuring each action aims to approach the mission objective while maintaining path smoothness, reducing unnecessary traversals, and making the simulation process more closely resemble actual mission requirements, further shortening iterative simulation time. Upon creating the optimal leaf node, effective actions are generated based on the UAV's kinematic model, calculating new positions and heading angles, ensuring that each generated action conforms to the UAV's physical control capabilities and guarantees the effectiveness of the actions.

[0009] The method proposed in this invention can balance computational efficiency, constraint adaptability, and dynamic robustness, achieving efficient path optimization under complex constraints and meeting the real-time and safety requirements of autonomous navigation for intelligent unmanned aerial vehicles. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention, and those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the UAV path planning method based on heuristic Monte Carlo tree search provided in Example 1. Figure 2 This is a structural block diagram of the UAV path planning device based on heuristic Monte Carlo tree search provided in Example 2; Figure 3 This is an internal structural diagram of the computer device provided in Example 3.

[0012] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0014] It is understood that the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0015] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0016] Example 1 This invention integrates the heuristic UCB strategy into the framework of heuristic Monte Carlo tree search to achieve autonomous obstacle avoidance path planning under constraints, thus meeting the real-time and safety requirements of UAV autonomous navigation.

[0017] like Figure 1 As shown, the UAV path planning method based on heuristic Monte Carlo tree search provided in this embodiment includes the following steps: Step 201: Obtain the UAV path planning task, model the UAV path planning task as a Markov decision problem, and construct an initial search tree. Starting from the root node of the initial search tree, select the optimal leaf node through the heuristic UCB strategy.

[0018] Step 202: Generate effective actions based on the kinematic model, calculate the new position and heading angle, and create a new leaf node on the optimal leaf node.

[0019] Step 203: Starting from the new leaf node, a path simulation is performed using a goal-oriented and smoothness-constrained action selection strategy until the task objective point is reached, and the total reward value of each node is calculated during the path simulation process.

[0020] Step 204: After pruning and normalizing the total reward value, propagate back to the root node along the task objective point, updating the visit count and total reward value of all nodes along the way, until the iteration terminates, and obtain the updated search tree.

[0021] Step 205: Based on the updated search tree, starting from the root node, select the sequence of child nodes with the most visits and extract the optimal path.

[0022] This invention discretizes the UAV path planning problem into a Markov decision problem and accelerates node selection, improving search efficiency and iterative convergence speed, based on a heuristic UCB strategy for optimal leaf node selection. It employs a goal-oriented and smoothness-constrained action selection strategy for path simulation, ensuring each action aims to approach the mission objective while maintaining path smoothness, reducing unnecessary traversals, and making the simulation process more closely resemble actual mission requirements, further shortening iterative simulation time. Upon creating the optimal leaf node, effective actions are generated based on the UAV's kinematic model, calculating new positions and heading angles, ensuring that each generated action conforms to the UAV's physical control capabilities and guarantees the effectiveness of the actions.

[0023] The method proposed in this invention can balance computational efficiency, constraint adaptability, and dynamic robustness, achieving efficient path optimization under complex constraints and meeting the real-time and safety requirements of autonomous navigation for intelligent unmanned aerial vehicles.

[0024] In the specific implementation of step 201, an environmental model is first constructed, and the map size, obstacle positions and shapes, task starting point, task target point and target area radius are set.

[0025] Initialize the UAV parameters, setting the UAV radius, maximum speed, initial heading angle, maximum turning angle, number of actions, and time step. The number of actions is a finite number of turning angle offsets uniformly sampled within the maximum turning angle range. The time step is used to discretize the continuous motion process into a time-series state transition.

[0026] Initialize Monte Carlo Tree Search (MCTS) parameters, setting the maximum number of iterations, maximum search depth, exploration constant, and discount factor.

[0027] In this embodiment, the environmental parameters are as follows: the map size is set to 100×100 units, the obstacles are 6 circular areas (such as [30, 30, 8], [50, 20, 6], etc., where the first two values ​​are the center coordinates and the last value is the radius), the task starting point coordinates are [10, 10], the task target point coordinates are [90, 90], and the target area radius is 5 units.

[0028] The specific parameters of the UAV are as follows: collision radius of 0.5 units, maximum speed of 5 units / second, initial heading angle of 0 radians, maximum turning angle of ±45°, number of actions of 7 (uniformly sampled within the turning angle range), and time step of 1.0 second.

[0029] The change in heading angle is discretized as Based on a simplified two-dimensional kinematic model, Always towards The kinematic model of the state at time step is: (1) In the formula, express Real-time drone location; Indicates the time step; Indicates the speed of the drone; express The heading angle at any given moment.

[0030] The Monte Carlo tree search parameters are as follows: maximum number of iterations 1000, maximum search depth 500, and exploration coefficient 1.4.

[0031] After initializing the environment and algorithm parameters, the UAV path planning task is obtained. This task is modeled as a sequential decision problem within a Markov framework, and based on the kinematic model, the path planning is determined using time steps. Discretize continuous actions into finite-dimensional decision variables.

[0032] Then, an initial search tree is constructed, where nodes represent the drone's state, including its location. Heading angle The edge represents a maneuver, set to a change in heading angle. Each node contains status information, parent node index, child node list, number of visits, total reward value, action that generated the node, depth, and terminal status identifier.

[0033] Subsequently, the path planning problem is solved. The heuristic MCTS solution framework is as follows: iteratively search by repeatedly executing sub-steps such as selection, expansion, simulation, and backtracking within a finite time, and further constructing the search tree based on the initial search tree to obtain the updated search tree.

[0034] In the selection phase, starting from the root node of the initial search tree, the optimal leaf node is selected using the heuristic UCB strategy.

[0035] Specifically, using distance and orientation alignment heuristics, the system prioritizes searching for nodes that are most efficient in moving closer to the task target point, constructs a heuristic UCB function, and selects the optimal leaf node based on this function. Here, distance heuristic refers to the normalized distance change from the reference node to the task target point; orientation alignment heuristic refers to the consistency between the reference motion direction and the direction of the task target point.

[0036] The expression for the heuristic UCB function is: (2) In the formula, Represents a node; Indicates the number of times a node has been accessed; This represents the average reward per node. This represents the exploration coefficient, used to balance exploration and exploitation in tree search; This indicates the number of times the node's parent node has been visited. Represents heuristic weights; Represents heuristic fractions. Preferably, choose... .

[0037] The heuristic expression for calculating fractions is: (3) in, (4) (5) (6) (7) In the formula, , These represent the heuristic score weights based on distance and direction, respectively. Represents the distance improvement heuristic function; This represents the orientation alignment heuristic function; This represents the Euclidean distance from the current location to the target point; It indicates the length of the map's diagonal, that is, the longest straight line on the map; Represents the target point coordinate vector; Represents the current position coordinate vector; Represents the coordinate vector of the target point; The coordinate vector representing the current position; This represents the coordinate vector of the starting point. Preferably, choose... , .

[0038] It can be seen that the smaller the distance to the target, The larger the value, the closer the velocity direction is to the target's line-of-sight direction. The larger the value, the better. The heuristic UCB function takes into account both the proximity to the target and the consistency of the movement direction, effectively guiding the search toward the target area. When facing obstacles, in order to avoid penalties for necessary detours, the direction heuristic score is subject to a non-negative constraint in formula (3), at which point the distance heuristic plays a dominant role.

[0039] In the specific implementation of step 202, which is mainly the expansion phase, effective actions are generated based on the kinematic model, the new position and heading angle are calculated, and a new leaf node is created on the optimal leaf node. Effective actions must satisfy collision-free and map boundary constraints.

[0040] It is understandable that when generating a candidate action set for the optimal leaf node, while considering the smoothness of the path, we want the action space to be as small as possible. The action set is selected as 7 values ​​that are uniformly discretely selected in the range of [-45°, 45°].

[0041] The specific implementation of step 203 mainly involves the simulation phase. In this phase, due to the enormous state space of the path planning problem, the search space of the stochastic exploration planning problem is quite inefficient, and experiments have shown that the probability of finding a solution through stochastic simulation is very low. Therefore, to encourage prioritizing actions with the potential to reach the destination, an optimized simulation process was designed. This process not only considers basic stochastic simulation but also incorporates goal-oriented strategies, smoothness constraints, and obstacle avoidance mechanisms.

[0042] The simulation process will simulate the path from the current state to the maximum depth and determine its value. In each action selection, first check whether there is an unobstructed straight path between the current position and the task target point. If there is, the unobstructed path is directly selected; otherwise, calculate the new state obtained after performing different actions according to formula (1) and evaluate each effective action by multiple factors.

[0043] Specifically, starting from the new leaf node, it is determined whether there is an unobstructed path between it and the task target point; if so, the path is directly selected as the feasible path for the current segment; if not, the new state obtained by the UAV after performing different actions is calculated based on the kinematic model, and then the comprehensive score from the new state to the task target point is calculated using the action selection strategy of goal orientation and smoothness constraint.

[0044] Select the node corresponding to the new state with the highest comprehensive score as the leaf node for the next round of path expansion and continue the simulation until the termination condition is met.

[0045] To encourage simulated exploration that leads to the goal, a goal-orientation score is calculated, expressed as: (8) In the smoothness constraint, the smoothness score is calculated using the following expression: (9) The obstacle penalty score is calculated using the following expression: (10) To predict the safety of a path after performing a certain action, a multi-step safety assessment score is also calculated, expressed as: (11) Based on goal-oriented scores Smoothness score Obstacle penalty points and multi-step security assessment score The comprehensive score from the new state to the task objective is calculated using the following expression: (12) In the formula, Represents the coordinate vector of the next new point; Represents the coordinate vector of the target point; This indicates the difference in turning angle between the current action and the previous action; This represents the maximum obstacle penalty score incurred upon a collision. This represents the baseline obstacle penalty score, indicating that no collision occurred but the distance to the obstacle was within the safe distance threshold. Indicates the safe distance threshold; Indicates the actual gap, where, Represents a coordinate vector; Indicates the first The coordinate vectors of the obstacles; Indicates the first The radius of the obstacle; This indicates the collision radius of the drone itself; Indicates a collision indicator (1 for a collision, 0 otherwise); Indicates the first The actual gap between the step and the nearest obstacle; This represents the safety margin penalty coefficient; This indicates a small amount of random perturbation, and To increase exploratory power and avoid getting stuck in local optima during simulation, a high probability of 90% is used for target-oriented selection to ensure search efficiency, while a 10% random exploration probability maintains the algorithm's global exploration capability; This indicates the maximum simulation process depth.

[0046] Preferably, , , .

[0047] It can be seen that the obstacle penalty function in formula (10) The exponential penalty related to the actual clearance of the obstacle not only applies a high penalty upon collision but also a progressive penalty upon approaching the obstacle, guiding the UAV away from it. The multi-step safety assessment function based on formula (11) simulates the future of the UAV. The movement of steps is used to assess the safety of the path.

[0048] After determining to adopt a goal-oriented approach, the Softmax formula is used to determine safe actions (here set to...). From the actions, specific actions are selected, using... -A variant of "greedy", but instead of a fixed probability, it dynamically adjusts the score calculated based on the environmental state. For actions The higher the overall score, the higher the probability of being selected, expressed as: (13) In the formula, This represents the temperature parameter; preferably, select... ; Represents a node.

[0049] Furthermore, when selecting the node corresponding to the new state with the highest comprehensive score as the leaf node for the next round of path expansion and continuing the simulation, the following steps are also included: The drone executes actions based on the new state, updates its position, and then performs constraint checks. The constraints are as follows: (14) In the formula, This represents the "OR" operation; Represents a coordinate vector; Indicates the first The coordinate vectors of the obstacles; Indicates the first The radius of the obstacle; This indicates the collision radius of the drone itself; This indicates that the drone is outside the map. It's worth noting that if a collision is detected, a penalty of -1000 is applied, and the simulation of the current action is terminated.

[0050] The simulation process terminates when any of the following conditions are met: when the target area is reached ( ),in, This indicates the distance from the current location to the task target point. This represents the radius of the target area; collisions occur (a penalty of -1000); no effective actions are taken (a penalty of -500); the maximum search depth is reached. The total reward value for each node is calculated. The expression is: (15) In the formula, Indicates the number of steps in the simulation process; Indicates step-by-step reward; Indicates smoothness reward; Indicates terminal reward; This indicates an efficiency reward.

[0051] It's understandable that there are step-by-step rewards. Negatively correlated with the distance from the current position to the target point, including straight-line path reward (positive feedback when the projected point lies on the straight line from the start point to the end point); smoothness reward. Negatively correlated with the difference in steering angle between adjacent actions; terminal reward A base reward is awarded upon reaching the target area, which is also related to the ratio of the actual path length to the theoretical shortest path length; efficiency reward. It is positively correlated with path efficiency (the ratio of the shortest straight-line distance to the actual path distance).

[0052] Among them, step rewards The specific expression is: (16) In the formula, This indicates the distance from the current location to the task target point; Indicates distance weight; This indicates the added movement costs incurred to urge UAVs to reduce ineffective actions; This represents the reward for a straight path. Preferably, , .

[0053] Straight path reward The specific calculation process is as follows: (17) (18) (19) (20) (twenty one) (twenty two) In the formula, Represents a vector from the starting point to the ending point; Represents the coordinate vector of the target point; The coordinate vector representing the starting point; This represents a vector from the starting point to the current position. Represents the coordinate vector of the current point; express exist The projection length on the vector represents the progress of the current position along the ideal path direction; express exist The coordinate vector of the projection point onto the vector; This represents the perpendicular distance from the current position to the straight line between the start and end points; This represents the penalty weight for deviating from the straight line direction; This indicates the indicator function (1 when the projection point is between the start and end points, 0 otherwise); This represents an additional bonus for being in the correct direction on the start-to-end line segment. Preferably, , .

[0054] Smoothness reward The specific calculation process is as follows: (twenty three) (twenty four) In the formula, It represents the change in the control variable (heading angle) within a step size; Indicates the control quantity for the current step; This indicates the control variable selected in the previous step; This represents the reward weight for the smoothness of the control quantity. Preferably, .

[0055] To incentivize UAVs to reach their targets, terminal rewards are provided. The specific expression is: (25) In the terminal rewards, regardless of efficiency, you can receive them as long as you reach the destination. S =2000 base success reward.

[0056] In the formula, Indicates the actual path length; This represents the theoretical shortest path length; Indicates the path length penalty weight; This represents the distance penalty weight when the target is not reached, and Distance from the mission objective The larger the value, the greater the penalty. Preferably, , ; This indicates a basic success reward.

[0057] Efficiency Rewards The specific calculation process is as follows: (26) (27) (28) (29) In the formula, Indicates the actual distance traveled; Indicates the first The position coordinate vector of the step; Indicates the first The position coordinate vector of the step; This represents the shortest straight-line distance between two points; The coordinate vector representing the location of the termination point; A coordinate vector representing the starting position; The weight representing the path efficiency reward; Path efficiency is expressed as the ratio of the shortest straight-line distance between two points to the actual distance traveled. UAVs are encouraged to take the shortest path. Preferably, .

[0058] After obtaining the total reward value of each node, to improve the stability of the heuristic Monte Carlo tree, a pruning function is used to normalize the total reward value of each node. This balances the exploration-exploitation term in the UCB formula and improves the accuracy of node value assessment. The specific expression for normalization using the pruning function is as follows: ; ; In the formula, This represents the total reward value after trimming; This represents the normalized total reward value; For coefficients. Preferably, .

[0059] Then, normalize the total reward value. Propagate backward from the task objective point back to the root node, updating the visit count and total reward value of all nodes along the way, until the iteration terminates, resulting in the updated search tree.

[0060] After the MCTS iteration is completed, each node obtains several cumulative normalized reward values. The Softmax function is then used to normalize and average these cumulative normalized reward values ​​to obtain the normalized average reward value, expressed as: (30) (31) In the formula, Indicates the first Softmax weights for each simulation process; Indicates the first The cumulative normalized reward value obtained from each simulation process; Indicates the first The cumulative normalized reward value obtained from each simulation process; This represents the largest reward value among all cumulative normalized reward values; Indicates temperature parameter; Represents a node Number of times it was visited; This indicates the total number of simulations that have been completed. Represents a node The normalized average reward value.

[0061] Based on the updated search tree, and according to the normalized average reward value Starting from the root node, select the sequence of child nodes with the most visits and extract the optimal path.

[0062] Calculate the normalized average reward value The goal is to give higher weight to high-quality simulated paths, thereby improving search efficiency and path quality.

[0063] It is worth noting that the action selection strategy in the simulation phase combines a 90% probability of goal-oriented selection with a 10% probability of random exploration to balance search efficiency and global exploration capability, and the goal-oriented selection also uses Softmax.

[0064] In the specific implementation of step 204, the main phase is the backtracking phase. The normalized reward value obtained from the final simulation is propagated back to the root node along the task objective point. The number of visits and the total reward value of all ancestor nodes are updated along the way. When the computing resources are exhausted or the terminal status flag is triggered, the MCTS simulation terminates and the updated search tree is obtained.

[0065] In the specific implementation of step 205, based on the updated search tree, starting from the root node, the sequence of child nodes with the most visits is selected, the optimal path is extracted, and the state-action pair sequence (including position and heading angle) is output, and the path, obstacles, starting point, target and MCTS search node distribution are visualized.

[0066] The method proposed in this invention views the process of finding the optimal path as the process of building a tree, starting from the initial state. Or, the starting point serves as the parent node of the search tree. The adjacent nodes of the initial position, representing the feasible space in path planning, are added to the search as children of the root node. Each node records the number of times it has been visited and its average state value. Through four phases—selection, expansion, simulation, and backtracking—the selection process is carried out at the state level. Actions to be performed transition to state The process continues until the endpoint is reached. After a given number of iterations, starting from the root node, the child node with the most visits is selected sequentially in the tree, forming the optimal state-action pair sequence, i.e., the optimal path. Based on the above data, a graph is plotted using commands, and the results are visualized. Thus, the optimal path under specific conditions can be obtained using the MCTS algorithm.

[0067] This invention utilizes the fusion mechanism of the UCT strategy and the goal-oriented heuristic function in the improved MCTS algorithm, and fully combines the advantages of the multi-objective reward function in path optimization. That is, it can balance exploration and utilization in each tree search iteration, and prioritize the generation of safe, smooth and near-optimal paths, ultimately realizing efficient path planning for UAV autonomous navigation in complex obstacle environments.

[0068] It should be understood that, although this embodiment Figure 1 The steps are shown sequentially as indicated by the arrows, but they are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are performed; they can be executed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0069] Example 2 Based on the UAV path planning method based on heuristic Monte Carlo tree search in Embodiment 1, this embodiment discloses a UAV path planning device based on heuristic Monte Carlo tree search, such as... Figure 2 As shown, the UAV path planning device based on heuristic Monte Carlo tree search includes: a selection module 401, an extension module 402, a simulation module 403, a backtracking module 404, and a path extraction module 405, wherein: The selection module 401 is used to obtain the UAV path planning task, model the UAV path planning task as a Markov decision problem, and construct an initial search tree. Starting from the root node of the initial search tree, the optimal leaf node is selected through a heuristic UCB strategy.

[0070] The extension module 402 is used to generate effective actions based on the kinematic model, calculate new positions and heading angles, and create new leaf nodes on the optimal leaf node.

[0071] The simulation module 403 is used to simulate the path from the new leaf node, using a goal-oriented and smoothness-constrained action selection strategy, until the task target point is reached, and to calculate the total reward value of each node during the path simulation process.

[0072] The backtracking module 404 is used to prune and normalize the total reward value, then propagate it back to the root node along the task objective point, updating the number of visits and the total reward value of all nodes along the way, until the iteration terminates, and obtains the updated search tree.

[0073] The path extraction module 405 is used to extract the optimal path based on the updated search tree, starting from the root node and selecting the sequence of child nodes with the most visits.

[0074] In this embodiment, the specific working process and working principle of the selection module 401, expansion module 402, simulation module 403, backtracking module 404, and path extraction module 405 are the same as those in Embodiment 1, and therefore will not be described again in this embodiment. Each unit module can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit module can be embedded in or independent of the processor in the computer device in hardware form, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above unit modules.

[0075] Example 3 like Figure 3 The diagram illustrates a terminal device disclosed in this embodiment, comprising a transmitter, a receiver, a memory, and a processor. The transmitter transmits instructions and data, the receiver receives instructions and data, the memory stores computer-executed instructions, and the processor executes the computer-executed instructions stored in the memory to implement the method described in Embodiment 1 above.

[0076] It is important to note that the aforementioned memory can be either standalone or integrated with the processor. When the memory is set up independently, the terminal device also includes a bus for connecting the memory and the processor.

[0077] Example 4 This embodiment discloses a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method in Embodiment 1 above.

[0078] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0079] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A heuristic Monte Carlo tree search based unmanned aerial vehicle path planning method, characterized in that, The method includes: The UAV path planning task is obtained, the UAV path planning task is modeled as a Markov decision problem, and an initial search tree is constructed. Starting from the root node of the initial search tree, the optimal leaf node is selected through a heuristic UCB strategy. Generate effective actions based on the kinematic model, calculate the new position and heading angle, and create a new leaf node on the optimal leaf node; Starting from the new leaf node, a goal-oriented and smoothness-constrained action selection strategy is used to simulate the path until the task objective is reached, and the total reward value of each node is calculated during the path simulation process. After pruning and normalizing the total reward value, the algorithm propagates back to the root node along the task objective point, updating the visit count and total reward value of all nodes along the way until the iteration terminates, resulting in the updated search tree. Based on the updated search tree, starting from the root node, select the sequence of child nodes with the most visits to extract the optimal path.

2. The UAV path planning method based on heuristic Monte Carlo tree search according to claim 1, characterized in that, The optimal leaf node is selected using a heuristic UCB strategy, including: By employing distance-based and direction-alignment-based heuristics, we prioritize searching for nodes that are most efficient in moving closer to the task objective, construct a heuristic UCB function, and select the optimal leaf node based on the heuristic UCB function.

3. The UAV path planning method based on heuristic Monte Carlo tree search according to claim 2, characterized in that, The expression for the heuristic UCB function is: ； In the formula, Indicates the number of times a node has been accessed; This represents the average reward per node. Indicates the exploration coefficient; This indicates the number of times the node's parent node has been visited. Represents heuristic weights; Represents heuristic fractions.

4. The UAV path planning method based on heuristic Monte Carlo tree search according to claim 3, characterized in that, The heuristic expression for calculating fractions is: ； in, ； ； ； ； In the formula, , These represent the heuristic score weights based on distance and direction, respectively. Represents the distance improvement heuristic function; This represents the orientation alignment heuristic function; This represents the Euclidean distance from the current location to the target point; Indicates the length of the map's diagonal; A unit vector representing the coordinates of the target point; A unit vector representing the current position coordinates; Represents the coordinate vector of the target point; The coordinate vector representing the current position; The coordinate vector representing the starting point.

5. The UAV path planning method based on heuristic Monte Carlo tree search according to any one of claims 1 to 4, characterized in that, Starting from the new leaf node, a goal-oriented and smoothness-constrained action selection strategy is used for path simulation, including: Starting from the new leaf node, determine whether there is an unobstructed path between it and the task target point; if so, directly select the unobstructed path as the current segment's feasible path; if not, calculate the new state obtained by the UAV after performing different actions, and then use the action selection strategy with goal orientation and smoothness constraints to calculate the comprehensive score from the new state to the task target point. Select the node corresponding to the new state with the highest comprehensive score as the leaf node for the next round of path expansion and continue the simulation until the termination condition is met.

6. The UAV path planning method based on heuristic Monte Carlo tree search according to claim 5, characterized in that, A goal-oriented and smoothness-constrained action selection strategy is used to calculate the comprehensive score from the new state to the task objective point, including: The goal-oriented score is calculated using the following expression: ； The smoothness score is calculated using the following expression: ； The obstacle penalty score is calculated using the following expression: ； The multi-step security assessment score is calculated using the following expression: ； The comprehensive score from the new state to the task objective point is calculated based on the goal orientation score, smoothness score, obstacle penalty score, and multi-step safety assessment score. The expression is as follows: ； In the formula, Represents the coordinate vector of the next new point; Represents the coordinate vector of the target point; This indicates the difference in turning angle between the current action and the previous action; This represents the maximum obstacle penalty score incurred upon a collision. This represents the baseline obstacle penalty score, indicating that no collision occurred but the distance to the obstacle was within the safe distance threshold. Indicates the safe distance threshold; Indicates the actual gap; Indicates a collision warning; Indicates the first The actual gap between the step and the nearest obstacle; This represents the safety margin penalty coefficient; This indicates a small amount of random perturbation; This indicates the maximum simulation process depth.

7. The UAV path planning method based on heuristic Monte Carlo tree search according to claim 6, characterized in that, When selecting the node corresponding to the new state with the highest overall score as the leaf node for the next round of path expansion and continuing the simulation, the following also applies: The drone executes actions based on the new state, updates its position, and then performs constraint checks. The constraints are as follows: ； In the formula, This represents the "OR" operation; Represents a coordinate vector; Indicates the first The coordinate vectors of the obstacles; Indicates the first The radius of the obstacle; This indicates the collision radius of the drone itself; This indicates that the drone is outside the map.

8. The UAV path planning method based on heuristic Monte Carlo tree search according to claim 7, characterized in that, The total reward value for each node is calculated during path simulation. The expression for calculating the total reward value is as follows: ； The total reward value is normalized using a pruning normalization method: ； ； In the formula, Indicates the number of steps in the simulation process; Indicates step-by-step reward; Indicates smoothness reward; Indicates terminal reward; Indicates a reward for efficiency; This represents the total reward value after trimming; This represents the normalized total reward value; For coefficients; Then, the normalized total reward value Propagate backward from the task objective point back to the root node, updating the visit count and total reward value of all nodes along the way, until the iteration terminates, resulting in the updated search tree.

9. The UAV path planning method based on heuristic Monte Carlo tree search according to claim 8, characterized in that, Also includes: After the iteration, each node obtains several cumulative normalized reward values. The Softmax function is used to normalize and average these cumulative normalized reward values ​​to obtain the normalized average reward value, expressed as: ； ； In the formula, Indicates the first Softmax weights for each simulation process; Indicates the first The cumulative normalized reward value obtained from each simulation process; Indicates the first The cumulative normalized reward value obtained from each simulation process; This represents the largest reward value among all cumulative normalized reward values; Indicates temperature parameter; Represents a node Number of times it was visited; This indicates the total number of simulations that have been completed. Represents a node The normalized average reward value; Based on the updated search tree, and according to the normalized average reward value Starting from the root node, select the sequence of child nodes with the most visits and extract the optimal path.

10. A UAV path planning device based on heuristic Monte Carlo tree search, characterized in that, The device includes: The selection module is used to acquire the UAV path planning task, model the UAV path planning task as a Markov decision problem, construct an initial search tree, and select the optimal leaf node from the root node of the initial search tree through a heuristic UCB strategy. An extension module is used to generate effective actions based on the kinematic model, calculate new positions and heading angles, and create new leaf nodes on the optimal leaf node; The simulation module is used to simulate the path from the new leaf node, using a goal-oriented and smoothness-constrained action selection strategy, until the task objective point is reached, and to calculate the total reward value of each node during the path simulation process. The backtracking module is used to prune and normalize the total reward value, then propagate it back to the root node along the task target point, updating the number of visits and the total reward value of all nodes along the way, until the iteration terminates, and obtains the updated search tree. The path extraction module is used to extract the optimal path based on the updated search tree, starting from the root node and selecting the sequence of child nodes with the most visits.

Images