Unmanned aerial vehicle navigation method

Abstract

The invention discloses an unmanned aerial vehicle navigation method, and relates to a system for controlling or adjusting non-electrical variables. The method comprises the following steps: establishing a global planning path, and generating a local target point on the global planning path; when the unmanned aerial vehicle flies, the first model generates an unmanned aerial vehicle action according to the position of the unmanned aerial vehicle at the current moment, the local target point position and the current speed information of the unmanned aerial vehicle; the second model performs reward evaluation on the unmanned aerial vehicle action according to the local target point and the unmanned aerial vehicle position, and the third model performs safety evaluation on the unmanned aerial vehicle action according to the unmanned aerial vehicle and the obstacle position; and adding the reward evaluation and the security evaluation into an experience pool for resampling, and updating and optimizing the first model. On the premise of ensuring zero collision of the unmanned aerial vehicle, the path decision-making efficiency is improved, and the flight stability of the unmanned aerial vehicle is improved.

Description

Technical Field

[0001] This invention relates to a system for controlling or regulating non-electrical variables, and more specifically to a method for navigating unmanned aerial vehicles (UAVs). Background Technology

[0002] Unmanned aerial vehicles (UAVs) have become a versatile and convenient tool with a wide range of applications, such as aerial photography, search and rescue, exploration, and monitoring. Regardless of the task, the key is to safely and autonomously complete path planning, which is also the guarantee for the efficient execution of tasks by UAVs. However, in actual flight, there may be some unknown factors in the environment. Ensuring zero collisions for UAVs in environments influenced by unknown factors is crucial. Some studies have introduced hard constraints (such as safety layers and safety barrier functions) to correct dangerous actions during the policy execution phase. While this can avoid collisions to some extent, relying entirely on hard constraints will prevent the policy training process from fully exploring high-reward paths, leading to a limited learning process, decreased reward performance, and ultimately difficulty in obtaining high-quality policies. Traditional reinforcement learning methods (such as SAC and DDPG) generally guide agents to avoid obstacles or maintain a safe flight state by constructing reward functions. While this soft constraint approach is flexible, it relies on the policy's own trial-and-error process to achieve safety. Once sudden obstacles, unknown areas, or perception delays occur in the environment, it can easily lead to unstable policy outputs, untimely obstacle avoidance, and serious consequences such as collisions. Especially in locally unknown environments, soft constraints cannot provide real-time and reliable safety guarantees.

[0003] Chinese Patent Publication No. CN 116301027 A discloses a path planning method for unmanned aerial vehicles (UAVs) in urban airspace based on safety reinforcement learning. This method combines a protection model (Shield) with the DDPG reinforcement learning algorithm, belonging to the field of UAV technology. When selecting actions, this method introduces the gravity of the target location and the safety shield as influence constraints. However, while this invention can avoid collisions to some extent, relying entirely on hard constraints will prevent the policy training process from fully exploring high-reward paths, thus limiting the learning process, reducing reward performance, and ultimately making it difficult to obtain a high-quality policy. Summary of the Invention

[0004] This invention aims to address the safety risks inherent in existing UAV path planning methods that rely solely on soft constraint mechanisms when facing complex or partially unknown environments. Furthermore, this invention aims to improve the decision-making efficiency of UAV path planning while ensuring zero collisions.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A drone navigation method, comprising: Establish a global planning path and generate local target points on the global planning path; When the drone is flying, the first model generates drone actions based on the drone's current position, the position of local target points, and the drone's current speed information; The second model evaluates the drone's actions based on the local target point and the drone's own position, while the third model evaluates the safety of the drone's actions based on the positions of the drone and obstacles. Re-sampling was performed by incorporating reward and safety evaluations into the experience pool, and the first model was updated and optimized. By evaluating the safety of drone actions based on the drone and obstacle positions, the safety of the drone's planned path was improved, the number of times the drone's safety shield was triggered was reduced, and the drone's flight was made smoother.

[0006] Preferably, the reward evaluation of the drone's actions includes: calculating the distance between the drone's current position and the positions of all local target points, and summing the distances in a weighted manner to obtain the reward value r1; The reward value r2 is obtained by calculating the distance between the drone's current position and the nearest local target point; The reward value for reaching the target point is denoted as r3; The reward value r4, which measures the number of steps taken by the drone, is calculated based on the number of steps taken by the drone. The reward evaluation is obtained by weighted summation of the above reward values.

[0007] By setting more reasonable reward evaluation criteria, the efficiency of path planning has been improved.

[0008] Preferably, the safety evaluation of drone actions includes: The safety value c1 is calculated based on the minimum distance between the drone and the obstacle; The safety value c2 is calculated based on the distance between the drone and all surrounding obstacles, which measures the number of nearby obstacles at the drone's current position. The safety value for a collision is denoted as c3; The safety evaluation is obtained by weighted summation of the above safety values. By setting the safety evaluation criteria, flight path safety is improved.

[0009] Preferably, when updating and optimizing the strategy for generating drone actions, the Lagrange multiplier is multiplied by the safety evaluation. By adjusting the Lagrange multiplier, the model's balance between performance and safety is adjusted; increasing the Lagrange multiplier improves model safety, while decreasing it improves model efficiency. Through the setting and iterative optimization of the Lagrange multiplier, the model can achieve optimal performance while ensuring that drone collisions do not occur.

[0010] Preferably, the safety shield mechanism is triggered after collecting the status information of the drone and the surrounding area. The safety shield consists of a perception layer, an obstacle avoidance layer, and an emergency obstacle avoidance layer. Whether to enter the obstacle avoidance layer is determined based on the current action and the relative distance between the drone's current position and surrounding obstacles.

[0011] Preferably, the perception layer processes and parses the sensor data in the input state, extracts key obstacle information in the environment, and further calculates the minimum distance between obstacles in a specific direction and the drone. Different safe distance thresholds are assigned to the distances to obstacles in front and to the side. Then, the current action is used to make a judgment. When the distance is less than the threshold, the drone enters the obstacle avoidance layer.

[0012] Preferably, the obstacle avoidance layer divides the fan-shaped area in front of the drone into multiple rays at an angle. If the distance between the nearest object on the ray and the drone is greater than a threshold, it is considered a gap point. If there are more than a certain number of consecutive gap points, it is considered a feasible gap. Each gap is screened, eliminating gaps that are too narrow or too close to obstacles, and then the deviation of its center angle from the current direction of travel is calculated as an indicator of the degree of proximity. If there are multiple close candidate gaps, then gap distance and width are further introduced to construct quality indicators, the gap with the highest quality is selected, and its center angle is extracted as the target direction. Adjust the drone's angular velocity according to the target direction.

[0013] Preferably, the emergency obstacle avoidance layer is activated when the minimum lateral distance is less than the emergency obstacle avoidance threshold, adjusting the angular velocity of the drone based on the position of the nearest obstacle, while reducing the linear velocity of the drone.

[0014] Preferably, the adjustment of the angular velocity of the UAV is obtained by a function to convert the target angle into angular velocity. The function has a larger slope at small angles and a slope that tends to 0 at large angles, so as to avoid sharp turns and ensure the smooth flight of the UAV.

[0015] Preferably, the perception layer also performs forward inference of the UAV's pose within a set prediction time window based on the UAV's current linear velocity and angular velocity at a certain time step. By projecting surrounding obstacles onto a Cartesian coordinate system, the minimum distance between each point on the predicted trajectory and the obstacles is calculated. If at any given moment this distance is less than the sum of the drone's radius and the safety margin, the trajectory is deemed to have a collision risk, and the current action is considered potentially dangerous, requiring entry into the obstacle avoidance layer. Predicting the drone's trajectory in advance further improves the reliability of the safety shield.

[0016] This invention proposes a drone navigation method that, while ensuring rapid response to drone mission requirements, minimizes the risk of collisions, achieves reliable verification of drone path planning actions, effectively guarantees the safety of output actions, and thus ensures the safety of the planned path. It also effectively addresses the uncertainty problem of solutions in general reinforcement learning algorithms. Specifically, during the entire algorithm training process, CMDP is incorporated to enable the trained policy to implement soft constraints, maximizing rewards while minimizing collisions. A safety shield constructed by a rule-based action correction mechanism serves as a final guarantee, effectively ensuring the safety of the solution by modifying unsafe actions output by the policy into safe actions to achieve hard constraints. Furthermore, this algorithm exhibits rapid convergence. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention.

[0018] Figure 2 This is a flowchart of the security shield of the present invention. Detailed Implementation

[0019] Example 1: The proposed UAV path planning method based on safety reinforcement learning in this invention comprises four functional modules in its SSAC algorithm architecture: environment, neural network module, safety shield, and experience replay pool. Unlike the original SAC algorithm, a third model and Lagrange multipliers are added to the neural network module to evaluate the safety of UAV flight, and an additional safety shield is also included. After acquiring and inputting the state from the environment through various sensors, the first model in the neural network module outputs actions according to the current strategy. The second and third models evaluate the actions and train the network based on the evaluation. The safety shield employs a rule-driven action correction mechanism, specifically including a perception layer, an obstacle avoidance layer, and an emergency obstacle avoidance layer. After the first model generates actions, it performs safety checks on the actions, modifying potentially dangerous actions into safe actions before outputting them for execution, thereby effectively avoiding the execution of dangerous actions.

[0020] The primary function of the first model is to output the drone's next action based on its current state (such as position, velocity, target point information, and obstacle environment). This network employs policy parameterization modeling and a stochastic policy to balance exploratory and conservative approaches, avoiding getting trapped in local optima. During training, the first model aims to maximize cumulative reward and minimize safety assessment, optimizing the objective function using a policy entropy regularization mechanism.

[0021] The second model evaluates the value of actions generated by the first model by calculating a soft Q-function. Its loss function is calculated based on the sum of squared errors between the predicted Q-value and the target Q-value. Parameters are updated using gradient descent to minimize the loss, and the target Q-value is determined by combining the reward function with the target Q-value of the next state.

[0022] The third model's main function is to assess the safety risks of actions (such as distance from obstacles, collision risk, etc.). Its training process constructs a loss function by calculating the sum of squared errors between the predicted risk value and the target risk value, and then updates the parameters using gradient descent to minimize the safety risk.

[0023] The second and third models work together to maximize action rewards while also ensuring the safety of actions. The strategy optimization of the first model provides reliable guidance to ensure that the drone can explore efficiently and maintain safe flight in complex environments.

[0024] like Figure 1 As shown, this embodiment includes the following steps: Initialize network weights and biases, Lagrange multipliers λ, temperature coefficient α, target entropy, experience pool, etc. Collect state information about the drone and its surrounding area, and define the state of the drone at any time t as s. t ,in, The perception layer in the safety shield module detects action a. t Perform security checks, where a t =[v t ,ω t Safety is determined based on the current action and the relative distance between the drone's current position and surrounding obstacles. If the action is dangerous, it enters the obstacle avoidance layer or emergency obstacle avoidance layer to modify the action, ultimately outputting a safe action 'a'. t Conversely, the original action is maintained. The final output safety action a obtained from the execution t And obtain the next state s according to the state transition function. t+1 The second and third models respectively obtain the reward r for the current state-action pair. t and safety evaluation c t ; The current state s t The final output is a safe action, and the reward is r. t Safety evaluation c t Next state s t+1 Training flag done t Store the experience in the experience replay pool, and sample a batch of experiences from the experience replay pool according to their importance for use in neural network updates; Calculate the loss functions for the second and third models respectively and update the network; After a certain number of rounds, the loss functions of the first model, policy entropy, and Lagrange multipliers are calculated and the network is updated. Soft update the objective functions of the second and third models; Record and save the training results, then begin the next training session.

[0025] This invention first establishes a global planning path using the RRT* algorithm based on known information, and then generates local target points along the path. Specifically, the environment module consists of the UAV's position information, overall flight environment information (mainly referring to the state information of the surrounding area), and the UAV's own information, mainly including: the UAV's current position state, the target point position of the UAV's flight mission, and the UAV's current speed information, where the speed information includes the UAV's linear velocity and angular velocity. In this invention, various static obstacles are equivalent to N cylindrical static obstacles, and the environment is locally unknown. A reference path is generated using global path planning based on partially known environmental information. Some previously unknown cylindrical static obstacles are generated around the reference path. These unknown obstacles require local path planning to adjust the actual flight path in real time. The algorithm of this invention focuses on solving the obstacle avoidance required for local path planning.

[0026] Since the next state of a drone depends only on its current state and the actions it has taken, we can express the decision-making process of a drone interacting with its environment under safety constraints as a constrained Markov decision process. Rewards and safety evaluations are used to measure whether an action is beneficial for achieving the task objectives and the degree of its negative impact, respectively. In this invention, rewards are used to measure whether an action is conducive to reaching the target point, and safety evaluations are used to measure whether an action will lead to a closer proximity to obstacles and a collision. Through a continuous sequence of actions, the drone transitions through multiple states from its starting position to finally reach the target location. Each action determines the drone's state at the next moment, thus forming a complete path.

[0027] Next, the final output safety action is executed, and the next state is obtained according to the state transition function. The second and third models obtain the reward and safety of the current state action, respectively. Specifically, upon entering the current state s... t At that time, the state information is transmitted to the first model and the action a is obtained. t The perception layer determines whether an action is dangerous. If the action may be dangerous, a safety action 'a' is generated through the safety shield. t ', execute a t 'Perform a state transition to obtain the next state s' t+1 and corresponding reward evaluation r t and safety evaluation c t .

[0028] Reward r tThe reward includes a reward value r1, which measures the distance between the drone's current position and the positions of all local target points; a reward value r2, which measures the drone's ability to track local target points; a reward value r3, which measures the number of steps taken by the drone; and a reward value r4, which measures the number of steps taken by the drone. r2 is calculated using the error between the drone's current position and the nearest local target point, r3 is a fixed value, and r4 is calculated using the number of steps taken by the drone.

[0029] Security c t This includes a safety value c1 that measures the distance between the drone's current position and the position of obstacles, a safety value c2 that measures the number of nearby obstacles around the drone at its current position, and a collision safety value c3. The collision safety value c3 is a fixed value.

[0030] The current state, the safe action output after security shield correction, the corresponding reward, security, the next state, and the training flag are stored in the experience replay pool. Then, a batch of experience data is selected from the experience replay pool according to importance sampling for subsequent neural network parameter updates. Specifically, the parameters of the second model in the Main network are updated by maximizing the reward, and the parameters of the third model in the Main network are updated by minimizing the error. Since the agent pays more attention to security changes after combining with CMDP, the output action becomes more conservative. If a deterministic policy is used, it may become overly conservative with iterations. Therefore, a stochastic policy is adopted to ensure a certain degree of exploration under security constraints and prevent getting trapped in local optima. For the stochastic policy, its objective function considers not only reward and security but also the stochasticity of the policy. Compared with the unconstrained soft actor-critic, an additional neural network model is learned to represent the security C. π (s t ,a t The changing state-action safety function. This invention involves learning five neural network models with parameters... The dual-Q network Q1, Q2, with parameters as follows: A dual-security network C1, C2, and a policy network π with parameter θ. θ .

[0031] Calculate the loss functions for the second and third models respectively and update the network. During training, the Q function Q... π (s t ,a tThe loss and gradient calculations for the Main network are identical to those in the soft actor-critic network. To avoid over-adjustment of the Actor network and to better evaluate early policies, a delayed update is applied to the first model. After a certain number of rounds, the first model in the Main network is updated again using gradient descent. The policy entropy term encourages policy randomness, thereby enhancing exploration capabilities and preventing the policy from getting stuck in local optima. By adjusting α, a larger α tends to encourage exploring more actions, while a smaller α makes the policy more deterministic, causing the agent to rely more on current experience, reducing exploration, and accelerating convergence. λ is a Lagrange multiplier used to balance performance (maximizing reward) and safety (minimizing safety evaluation). A larger λ means that during the learning process, the agent will pay more attention to safety, avoiding collisions or other risks; conversely, a smaller λ has a greater impact on the objective function, and the agent will prioritize efficiency, potentially sacrificing some safety for higher rewards.

[0032] Next, a multi-delay update mechanism was invoked, and updates were performed every m... π or m λ The policy entropy and Lagrange multipliers are updated step-by-step. This delayed mechanism helps stabilize the training process. The larger the constraint violation value, the smaller the Lagrange multiplier loss value. Then, the target network parameters are updated using soft updates based on the main network parameters. Specifically, in each training iteration, the parameters of the main network are slowly fused into the target network at a small proportion, causing the target network to gradually converge towards the main network. This update method effectively slows down the rate of change of the target value, thereby improving the stability of learning during policy training and avoiding training oscillations or policy divergence caused by drastic fluctuations in the target network.

[0033] To address the challenge of balancing performance and safety in path planning using traditional reinforcement learning methods, this invention proposes a Safe Soft Actor Commentator (SSAC) algorithm. This is a reinforcement learning method that integrates Constrained Markov Decision Process (CMDP) and a safety shield mechanism. While traditional SAC algorithms can learn policies by maximizing rewards, their reward mechanism cannot clearly distinguish whether performance degradation is caused by deviation from the target or by collisions, and it cannot directly constrain agent behavior, only adjusting through indirect "negative incentives," leading to unstable safety assurance. Therefore, SSAC introduces the CMDP framework to simultaneously optimize two objectives during policy training: first, maximizing task-related cumulative rewards to ensure the agent can efficiently complete target navigation; and second, minimizing the cumulative safety risk caused by collisions or high-risk actions, achieving "soft constraint" modeling for safety. Soft constraints refer to guiding policy learning to avoid risks through safety functions or penalty terms, but not directly preventing dangerous actions; it relies on the model spontaneously learning safe behaviors during training. While this method has significantly improved safety awareness, relying solely on safety evaluation constraints still carries the risk of policy-driven dangerous actions. Therefore, this invention further introduces a multi-layered safety shield mechanism, performing "hard constraint" corrections during the action output phase to achieve synergy between soft and hard constraints. Hard constraints, after policy output, directly shield or correct unsafe actions through rules or safety filtering mechanisms. Even if the policy itself has not learned to avoid these risks, it can still enforce the safety of the actions, thereby ensuring flight safety even during policy exploration or when perception is uncertain.

[0034] Example 2, as Figure 2 As shown, the present invention also includes a safety shield, wherein the sensing layer in the safety shield module detects action a. t Perform security checks, where a t =[v t ,ω t Safety is determined based on the current action and the relative distance between the drone's current position and surrounding obstacles. If the action is dangerous, it enters the obstacle avoidance layer or emergency obstacle avoidance layer to modify the action, ultimately outputting a safe action 'a'. t Conversely, if the action is reversed, the original action is maintained.

[0035] First, the perception layer processes and analyzes the sensor data from the input state, extracting key obstacle information from the environment and further calculating the minimum distance in specific directions (such as directly in front, left front, and right front), providing a basis for subsequent decision-making based on environmental information. Specifically, the perception layer is implemented as an environmental perception system based on 40 LiDAR scanning points, with a scanning range from 0.15m to 3.0m and an angle range of 240°. The perception layer divides the 40 scanning points into three key sectors: the front sector (indexes 14-25), the left sector (indexes 0-13), and the right sector (indexes 26-39), corresponding to different directional areas of the 240-degree LiDAR. The system then assesses the current movement by assigning different safety distance thresholds to obstacles in front and to the sides: if the drone is moving forward and the distance to the front is less than 0.8m, there is a risk of forward collision; if a sharp left or right turn is made and the corresponding distance to the side is less than 1.2m, there is a risk of turning collision. Furthermore, the system predicts the drone's trajectory over a short period based on the current movement and detects whether this trajectory may collide with surrounding obstacles in the future. Using the current linear velocity and angular velocity as input, the system performs forward inference on the drone's pose within a set prediction time window (e.g., 1 second) at certain time steps (e.g., 0.1 seconds), gradually generating its possible movement paths. Specifically, combining obstacle distance data measured by lidar, the system projects surrounding obstacles into a Cartesian coordinate system and calculates the minimum distance between each point on the predicted trajectory and the obstacles. If at any given time this distance is less than the sum of the drone's radius and the safety margin (e.g., 0.2m + 0.2m), the trajectory is deemed to have a collision risk, and the current movement is considered potentially dangerous. If this predicted risk is satisfied by any of the other directional collision risks (such as front, left, right), the system determines that the action output by the current strategy is unsafe, triggers the safety shield module to correct the action, and ensures that the actual action meets the safety requirements.

[0036] The emergency obstacle avoidance layer activates when the minimum lateral distance is ≤0.4m, prioritizing either "directional avoidance" or "emergency braking." "Directional avoidance" employs a "mirror image obstacle avoidance" strategy: selecting the avoidance direction by calculating the safety of symmetrical positions. For example, when a threat appears to the left front, the system calculates the index of the obstacle to the right relative to the nearest obstacle to the left front based on its position. If the right-side distance is sufficiently large (a difference of more than 0.2m), a right turn obstacle avoidance is executed with an angular velocity of ±0.3rad / s. "Emergency braking" forces a linear velocity of -1.0m / s (to stop) in emergency situations, including when the distance to the obstacle ahead is less than 0.5m or the distance to the lateral obstacle is less than 0.4m. This layer has higher priority than conventional planning and represents the system's safety baseline.

[0037] The obstacle avoidance layer implements a path planning algorithm based on gap detection. The main task of this layer is to identify passable areas (gaps) in real time based on LiDAR data and select an optimal direction to proceed from multiple gaps. The system first analyzes the LiDAR data to find sufficiently wide gaps between adjacent obstacles; these gaps must meet the requirements of the robot's body width and safety margin.

[0038] Specifically, the system sets a gap threshold of 2.0m and identifies passable areas through continuous detection. For detected gaps, the system requires a minimum width of 4 scan points and performs a lateral safety check: the minimum distance within ±7 points of the gap center must be ≥0.35m to ensure passage safety. A multi-gap optimization mechanism is implemented when multiple passable gaps exist. First, gaps are sorted by angle deviation, prioritizing those closest to the front. If the angle difference is within 5 degrees, a quality assessment is introduced, comprehensively considering both gap distance and width. Furthermore, when the distance to the i-th point is greater than a set value d... min ≥d gap If a gap is found when a certain number of consecutive gaps appear, it is considered a feasible gap. The system then filters each gap, eliminating those that are too narrow or too close to obstacles. It calculates the deviation of the gap's center angle from the current direction of travel (angle difference) as an indicator of "proximity." If multiple close candidate gaps exist, gap distance and width are further considered to construct a "quality index," comprehensively judging its safety and passability. The greater the distance from the obstacle and the wider the gap, the higher its overall quality. Finally, the system selects the gap with the highest quality and extracts its center angle as the target direction. This angle will be used to adjust the robot's angular velocity α. t [1], the further the target angle deviates from directly in front, the higher the angular velocity |a t [1] The larger the adjustment, the smoother and safer the obstacle avoidance strategy can be.

[0039] The system employs a progressive speed adjustment strategy for drone linear velocity, with a three-tiered mechanism. First, it assesses the foreground threat, setting a critical safe distance of 0.5 meters and a completely safe distance of 0.6 meters. When an obstacle is within this range, the system calculates a forward safety ratio, decreasing as the distance decreases. For example, at 0.55 meters, the ratio is 0.5, meaning the initial speed must be halved; at 0.52 meters, the ratio is 0.2, requiring a significant speed reduction.

[0040] Secondly, there's the lateral threat assessment, which uses a similar principle but with stricter thresholds—0.3 meters is the critical distance, and 0.4 meters is the safe distance. This is because drones have relatively poor lateral maneuverability, requiring a larger safety margin. Lateral threats are often more difficult to handle than frontal threats, so the system is more cautious about them.

[0041] The system compares the safety ratios of the front and sides, always choosing the stricter one as the final speed adjustment factor. This ensures that the drone does not ignore threats from other directions just because one direction is relatively safe.

[0042] The challenge of angular velocity control lies in responding quickly to obstacle avoidance requirements while avoiding oscillations caused by oversteering. Traditional linear proportional control often fails to respond adequately at small angles and over-responds at large angles. Therefore, the system uses the tanh function to handle the conversion from target angle to angular velocity. This function has a unique characteristic—a large slope at small angles provides sufficient response sensitivity; it gradually saturates at large angles to prevent excessive angular velocity. For example, when the target angle is very small (a few degrees), the tanh function provides an almost linear response, ensuring precise control; when the target angle is very large (tens of degrees), the output angular velocity tends to a reasonable maximum value of 1.0 (left turn) or -1.0 (right turn), minimizing sharp turns. This effectively simulates human driving intuition—slight adjustments to the direction for small deviations and decisive but not over-steering for large deviations.

[0043] After completing the SSAC algorithm model construction, test and output the model on rviz. Specifically, once the SSAC algorithm model meets the conditions for ending training and satisfies the convergence requirement, the model parameters can be exported for model testing in the next step; otherwise, retrain the model. The convergence requirement is that the output path meets the mission requirements specified by the UAV for 10 consecutive iterations, and the reward remains at a high value without significant jumps.

[0044] After the model meets the requirements of the above steps, its output performance needs to be tested. Specifically, the model parameters exported in the above steps are imported into a new Python environment, and the model is continuously trained for 1000 rounds in an environment with longer paths and more unknown obstacles. If the probability of collision reaches 0%, the model is considered to meet the requirements; otherwise, the model is retrained.

[0045] This invention ensures rapid response to UAV mission requirements while minimizing collision risks, enabling reliable verification of UAV path planning and execution actions, effectively guaranteeing the safety of output actions, and thus ensuring the safety of the planned path. It also effectively addresses the uncertainty problem of solutions in general reinforcement learning algorithms. Specifically, during the entire algorithm training process, CMDP is incorporated to implement soft constraints on the trained policy, maximizing rewards while minimizing collisions. A safety shield constructed by a rule-based action correction mechanism serves as a final guarantee, effectively ensuring the safety of the solution by modifying unsafe actions output by the policy into safe actions, thus implementing hard constraints. Furthermore, this algorithm exhibits rapid convergence.

[0046] Finally, it should be noted that the above preferred examples are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred examples, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A navigation method for unmanned aerial vehicles (UAVs), characterized in that, include: Establish a global planning path and generate local target points on the global planning path; When the drone is flying, the first model generates drone actions based on the drone's current position, the position of local target points, and the drone's current speed information; The second model evaluates the drone's actions based on the local target point and the drone's own position, while the third model evaluates the safety of the drone's actions based on the positions of the drone and obstacles. The reward and security evaluations were added to the experience pool for resampling, and the first model was updated and optimized.

2. The UAV navigation method according to claim 1, characterized in that, The reward evaluation of drone actions includes: Calculate the distance between the drone's current position and the positions of all local target points, and sum the distances in a weighted manner to obtain the reward value r1; The reward value r2 is obtained by calculating the distance between the drone's current position and the nearest local target point; The reward value for reaching the target point is denoted as r3; The reward value r4, which measures the number of steps taken by the drone, is calculated based on the number of steps taken by the drone. The reward evaluation is obtained by weighted summation of the above reward values.

3. The UAV navigation method according to claim 1, characterized in that, The safety evaluation of drone actions includes: The safety value c1 is calculated based on the minimum distance between the drone and the obstacle; The safety value c2 is calculated based on the distance between the drone and all surrounding obstacles, which measures the number of nearby obstacles at the drone's current position. The safety value for a collision is denoted as c3; The security evaluation is obtained by weighted summation of the above security values.

4. A UAV navigation method according to claim 1 or 3, characterized in that, When updating and optimizing the strategy for generating drone actions, the Lagrange multiplier is multiplied with the aforementioned safety evaluation. By adjusting the Lagrange multiplier, the model's balance between performance and safety is adjusted. Increasing the Lagrange multiplier improves the model's safety, while decreasing it improves the model's efficiency.

5. The UAV navigation method according to claim 1, characterized in that, After collecting status information about the drone and its surrounding area, the safety shield mechanism is triggered. The safety shield consists of a perception layer, an obstacle avoidance layer, and an emergency obstacle avoidance layer. The decision to enter the obstacle avoidance layer is based on the current action and the relative distance between the drone's current position and surrounding obstacles.

6. The UAV navigation method according to claim 5, characterized in that, The perception layer processes and parses the sensor data in the input state, extracts key obstacle information in the environment, and further calculates the minimum distance between obstacles in a specific direction and the drone. Different safe distance thresholds are assigned to the distances to obstacles in front and to the side. Then, the current action is combined to make a judgment. When the distance is less than the threshold, the drone enters the obstacle avoidance layer.

7. The UAV navigation method according to claim 5, characterized in that, include: The obstacle avoidance layer divides the fan-shaped area in front of the drone into multiple rays at an angle. If the distance between the nearest object on the ray and the drone is greater than a threshold, it is considered a gap point. If there are more than a certain number of consecutive gap points, it is considered a feasible gap. Each gap is screened, eliminating gaps that are too narrow or too close to obstacles, and then the deviation of its center angle from the current direction of travel is calculated as an indicator of the degree of proximity. If there are multiple close candidate gaps, then gap distance and width are further introduced to construct quality indicators, the gap with the highest quality is selected, and its center angle is extracted as the target direction. Adjust the drone's angular velocity according to the target direction.

8. The UAV navigation method according to claim 5, characterized in that, The emergency obstacle avoidance layer is activated when the minimum lateral distance is less than the emergency obstacle avoidance threshold. It adjusts the drone's angular velocity based on the position of the nearest obstacle while reducing the drone's linear velocity.

9. A drone navigation method according to claim 7 or 8, characterized in that, The angular velocity of the drone is adjusted by a function to convert the target angle into angular velocity. The slope of the function is larger at small angles and tends to be 0 at large angles.

10. A UAV navigation method according to claim 6, characterized in that, The perception layer also performs forward inference of the drone's pose within a set prediction time window based on the drone's current linear velocity and angular velocity at a certain time step. Project the surrounding obstacles into a Cartesian coordinate system and calculate the minimum distance between each point on the predicted trajectory and the obstacles; If at any given moment the distance is less than the sum of the drone's radius and the safety margin, the trajectory is deemed to have a collision risk, and the current action is considered to pose a potential danger, requiring entry into the obstacle avoidance layer.

Images