A dynamic smoke environment unmanned aerial vehicle double-layer path planning method based on reinforcement learning-gray wolf optimization fusion

Abstract

This invention relates to the field of UAV path planning technology, specifically to a two-layer path planning method for UAVs in dynamic smoke environments based on a fusion of reinforcement learning and gray wolf optimization, constructing a smoke field model and a UAV dynamics model. The upper layer employs a hybrid strategy combining Q-learning and gray wolf optimization, generating a global reference trajectory with a multi-objective fitness function as the optimization objective. Each gray wolf individual is configured with an independent Q-learning agent, adaptively selecting search actions based on fitness ranking, smoke exposure level, and iteration stage. Dynamic smoke threats are detected in real-time during flight. When local planning is triggered, a spectrum-guided tree search planner is activated to select the optimal locally feasible trajectory. Simultaneously, global path deviation, waypoint deviation distance, and smoke penetration risk are monitored. If any of these exceed a threshold and local obstacle avoidance is not in effect, global replanning is triggered. This invention balances long-range optimality with real-time local obstacle avoidance capabilities in dynamic smoke environments.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) path planning technology, specifically to a two-layer path planning method for UAVs in dynamic smoke environments based on reinforcement learning-gray wolf optimization fusion. Background Technology

[0002] Unmanned aerial vehicles (UAVs) are playing an increasingly important role in tasks such as fire rescue, environmental monitoring, and industrial inspection. In these mission scenarios, path planning is one of the core key technologies, aiming to generate a safe and efficient flight trajectory from the starting point to the destination while meeting various physical and mission constraints.

[0003] In practical applications, especially in disaster scenarios such as fires, UAV path planning faces multiple complex challenges: 1) Environmental threats: Smoke in fire scenes poses a threat to UAV sensors, image acquisition quality, and flight safety; 2) Multi-target conflicts: The mission requires UAVs to avoid areas with high concentrations of smoke to ensure their own safety and sensor clarity, while getting as close as possible to key ground targets (such as the fire point or trapped personnel) to obtain high-quality images or monitoring data. This is inherently conflicted with shortening the path length and avoiding fixed threats (such as radar or no-fly zones); 3) Real-time requirements: The algorithm needs to provide a feasible and high-quality path within a limited time.

[0004] Currently, traditional path planning methods, such as the A* algorithm, Rapid Randomized Tree Extension (RRT) and its variants, often struggle to achieve a good balance between path quality and computational efficiency when dealing with high-dimensional, nonlinear, and multi-objective optimization problems. Swarm intelligence optimization algorithms (such as Particle Swarm Optimization (PSO) and Genetic Algorithm (GA)) have been introduced to solve multi-objective path planning, but they are insufficient in adapting to complex environments and are prone to getting trapped in local optima. Reinforcement learning learns policies through trial-and-error interaction with the environment, and theoretically can adapt well to dynamic changes, but it faces problems of low exploration efficiency and slow convergence in continuous high-dimensional action spaces.

[0005] Therefore, existing technologies lack an intelligent path planning method that can effectively coordinate multiple objectives such as smoke avoidance, threat avoidance, and path economy, and the method must have good environmental adaptability, optimization efficiency, and solution quality. Summary of the Invention

[0006] The purpose of this invention is to provide a two-layer path planning method for UAVs in dynamic smoke environments based on reinforcement learning-gray wolf optimization fusion. The upper layer uses RL-GWO to generate a global trajectory, and the lower layer uses spectral guided tree search for real-time obstacle avoidance, combining dynamic evolution of the smoke field and overweight planning.

[0007] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution: A two-layer path planning method for UAVs in dynamic smoke environments based on reinforcement learning-gray wolf optimization fusion includes: A mission environment model is constructed that includes a dynamic smoke field, static threats, and flight boundaries, and a dynamic model of the UAV is established, wherein the dynamic smoke field is dynamically updated over time according to preset smoke source parameters. A hybrid optimization strategy combining Q-learning and gray wolf optimization algorithm is adopted, with a multi-objective fitness function as the optimization objective, to generate a global reference trajectory from the starting point to the ending point; During the flight of the UAV along the global reference trajectory, dynamic smoke threats are detected in real time. When local planning conditions are triggered, the spectrum-guided tree search planner is activated to perform online trajectory correction based on the dynamic model and generate a locally feasible trajectory to avoid dynamic smoke. It also detects the deviation between the local feasible trajectory and the global reference trajectory in real time, as well as the risk of smoke penetration. When any indicator exceeds a preset threshold, it triggers global replanning and regenerates the global reference trajectory with the current position as the new starting point.

[0008] Furthermore, the dynamic update method of the dynamic smoke field includes: The center of the smoke moves according to the wind speed vector: The smoke radius increases linearly according to the following formula: in, Let these be the initial center coordinates of the smoke source. The initial radius of influence. For diffusion rate, For wind speed vectors, For time steps, This is the wind direction influence coefficient, which is dynamically adjusted based on the angle between the wind direction and the smoke diffusion direction.

[0009] Furthermore, when the angle of sudden wind direction change is monitored in real time... or wind speed change Update immediately and value.

[0010] Furthermore, the multi-objective fitness function is expressed as: in, The cost is the path length. The cost of threatening to penetrate; This is the cost of path smoothness; Cost of feasibility constraints; Smoke-related hard constraint penalties; Cost of straightness; This is the actual length of the path. The straight-line distance from the starting point to the ending point. The preset path length cost weight; The first penalty coefficient is preset. For the first The path sampling point to the first The distance to the center of the threat Threat radius, Preset threat penalty weights; The second penalty coefficient is preset. The turning angle of the path at the control point. The preset smoothness weights; The third penalty coefficient is preset. This represents the number of sampling points along the path whose height exceeds the safety boundary. The preset feasibility weights; This is a preset maximum penalty value that is triggered when the path sampling point enters the smoke-safe zone, forcibly avoiding the smoke.

[0011] Preferred, , , , They are 100, 10, 20, and 1e6 respectively.

[0012] Furthermore, the hybrid optimization strategy that integrates Q-learning and the gray wolf optimization algorithm includes: The control point sequence of the path is used as the location code for individual gray wolves; Each individual gray wolf is equipped with an independent Q-learning agent; "The decision-making basis of the Q-learning agent includes the relative fitness level of the gray wolf population, smoke exposure risk information, and the current iteration stage; wherein the smoke exposure risk information is used to guide action selection and update rewards. The action space of the Q-learning agent includes global exploration, local exploration, path optimization, and fine-tuning." In each iteration of the Grey Wolf Optimization Algorithm, each individual selects and performs an action based on its current state to adjust its search behavior and receives a reward based on the improvement in fitness to update the Q-table. In the RL-GWO global planning process, to enhance the algorithm's adaptability to dynamic smoke environments, smoke exposure information is calculated for each individual gray wolf's path under the current smoke field, including the immediate hazard level Ld, exposure trend level Lt, and minimum smoke distance. Further risk indicators are constructed. , Where Ld represents the proximity of the path to the smoke region at the current moment, and Lt represents the trend of smoke exposure change between adjacent iterations. The risk index is not directly used as a Q-learning state variable, but rather as a risk-guided quantity participating in action selection and reward shaping: during the ε-greedy exploration phase, fine-tuning actions are prioritized when R is high, and when R is low... When the risk is low, global exploration actions are prioritized; during the utilization phase, the current optimal action is selected based on the Q-table. After completing the action, the smoke exposure risk of the new path is recalculated, and a trend reward is constructed based on the risk reduction rate. This reward, along with the base fitness reward, updates the Q-table, thereby achieving adaptive search adjustment for dynamic smoke environments. The position update rule of the Grey Wolf optimization algorithm is as follows: This represents the distance from the current gray wolf individual to the α wolf (the optimal individual); These are random coefficients used to control the scope of exploration. Let X be the position of wolf α, and let X be the current position of the individual gray wolf. This represents the distance from the current gray wolf individual to the β wolf (the second-best individual). For random coefficients, The position of the β wolf; This represents the distance from the current gray wolf individual to the delta wolf (the third best individual). For random coefficients, This represents the position of the δ wolf. , , For the current gray wolf individual affected by α, β, The wolf-guided update position; , , The correlation coefficient is the convergence factor, used to balance exploration and development. t represents the position of the current gray wolf individual in the next iteration; t represents the current iteration step.

[0013] Furthermore, the update formula for Q-learning is: Let Q be the value of action a performed in state s. The learning rate controls the magnitude of Q-value updates; r is the immediate reward gained from the current action. This serves as a discount factor, controlling the weight of future rewards. The new state transitioned to after performing action a; For the new state The largest Q value among all actions.

[0014] Furthermore, the guided search planner performs the following steps: linearizing the dynamic model based on the current state of the UAV to obtain the linearized state equation: in The drone state at step k+1; Let A be the state of the UAV at step k; let A be the 12×12 state Jacobian matrix. B is a 12×4 input Jacobian matrix. u(k) is the control input for k steps; c is the constant offset term; Calculate the prediction time domain length Internal controllable personality Ram matrix: The 12×12 controllable Glam matrix measures the controllable energy of the system within H steps. Let A be the k-th power of the state matrix; This is the transpose of the input matrix B; H is the k-th power of the transpose of the state matrix A; H is the prediction time domain length; k is the summation index, from 0 to H-1; right Perform spectral decomposition: Obtain the eigenvalue matrix and eigenvector matrix Select strongly controllable modes with eigenvalues ​​greater than a preset threshold, and generate multiple candidate reference trajectory branches along the positive and negative directions of their corresponding eigenvectors; use Monte Carlo tree search to evaluate each branch based on the reward function, and select the branch with the highest cumulative reward as the local planning trajectory; the node selection in the Monte Carlo tree search adopts the upper confidence bound formula: in To accumulate rewards, For the number of times a node is accessed. This represents the number of times the parent node has been visited. To explore constants.

[0015] Furthermore, the reward function is a single-step reward. Total Discount Rewards Combinations: Single-step reward: Total reward for a single step; To mitigate the impact of smoke; Weighting of waypoint tracking rewards; Dynamically smooth reward weights; Rewards for avoiding smoke; Rewards for waypoint tracking; in, The distance from the drone to the nearest center of the smoke. This is the preset maximum safe distance; Location of the drone. As a global reference waypoint, The preset Gaussian kernel width; , They are velocity and angular velocity, respectively. , These are the preset maximum permissible rate of change of velocity and the maximum permissible rate of change of angular velocity, respectively. Obstacle avoidance reward Defined as: in, The minimum safety margin from the drone's current position to all static threat safety boundaries is expressed as: in, These are the current position coordinates of the drone. It is the i-th static threat center. Let be the security boundary radius of the i-th static threat. The preset buffer distance can be set to 6m.

[0016] in Total discount reward As a discount factor, This is a single-step reward.

[0017] Furthermore, after generating the locally feasible trajectory, the process also includes smoothing the trajectory using a moving average filter: These are the coordinates of the k-th position after smoothing. To find the mean function; The coordinates of the i-th position in the original position sequence are given; the window interval is the range of position points centered at k with a length of L (odd number), and the average of the available points is taken at the boundary.

[0018] Furthermore, the real-time detection metrics include global path deviation, maximum waypoint deviation distance, and smoke penetration risk; when any of these metrics exceeds its corresponding preset threshold, and the drone is not currently in a local obstacle avoidance flight state, a global replanning is triggered.

[0019] On the other hand, this invention proposes a two-layer path planning system for UAVs in dynamic smoke environments based on reinforcement learning-gray wolf optimization fusion, used to execute the above-mentioned method, including: The environmental perception and modeling module is used to build and update environmental models that include dynamic smoke fields and static threats in real time. The upper-level global planning module is used to generate a global reference trajectory using a hybrid optimization strategy that combines Q-learning and the gray wolf optimization algorithm. The lower-level local planning module is used to generate local obstacle avoidance trajectories using a spectrum-guided tree search planner. The over-limit detection and replanning module is used to monitor trajectory deviation and smoke penetration risk, and triggers the upper-level global planning module to perform global replanning when the limit is exceeded; The beneficial effects of this invention are: This invention independently configures a Q-learning agent for each individual gray wolf, using fitness ranking, smoke exposure level, and iteration stage as states. It adaptively selects from four actions: global exploration, local exploration, path optimization, and fine-tuning, dynamically changing the individual's position update rules. This mechanism enables the population to automatically balance exploration and development during iteration, avoiding the local optima trap caused by fixed weighted average updates in traditional gray wolf algorithms. Combined with a multi-objective fitness function that includes a 1e6 penalty for smoke hard constraints and a cubed distance cost for threats, it ensures that the generated global reference trajectory is free from smoke intrusion, avoids static threats, and has a near-straight line length.

[0020] This invention constructs an evolutionary model where the smoke center moves with wind speed and its radius dynamically increases according to the diffusion rate and the wind advection term. Parameters are updated in real time under the condition of a wind direction change ≥30° or a wind speed change ≥0.5 m / s. Simultaneously, three indicators are monitored: global path deviation, waypoint deviation distance, and smoke penetration risk. When any one of these exceeds a threshold and the system is not in a local obstacle avoidance state, a global replanning is triggered, starting from the current position. This hierarchical mechanism allows local planning to handle short-term disturbances, while replanning addresses large-scale environmental changes, avoiding accumulated path deviations or wasted computational resources.

[0021] This invention linearizes the UAV dynamics model and calculates the controllable Gram matrix during local planning. Strongly controllable modes with eigenvalues ​​greater than a threshold are extracted through spectral decomposition, and candidate branches are generated along the positive and negative directions of the corresponding eigenvectors. This guidance method compresses the search space to a low-dimensional natural motion manifold, avoiding the blindness of random sampling. The optimal branch is selected by combining Monte Carlo tree search and continuous reward functions (smoke distance reward, Gaussian waypoint tracking reward, and velocity / angular velocity smoothing reward). After filtering, a dynamically feasible local trajectory without smoke collisions is output.

[0022] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart for the overall two-level path planning; Figure 2 Schematic diagram of the RL-GWO hybrid optimization iteration principle; Figure 3 Flowchart for SGT local programming spectrum expansion; Figure 4 Here is the logic diagram for determining the unsafe state of SGT; Figure 5 This is a schematic diagram of the GWO path; Figure 6 This is a schematic diagram of the path of the present invention; Figure 7 This is a 3D schematic diagram of the path of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1

[0027] This embodiment describes a two-layer path planning method for UAVs in dynamic smoke environments based on reinforcement learning-gray wolf optimization fusion. Taking a forest fire monitoring airspace with multiple dynamic smoke sources and several static threats as an example, such as... Figure 1 The overall process of the two-layer path planning shown includes: Environment modeling and parameter initialization First, input the drone's takeoff and landing points. In this embodiment, the drone's takeoff point is set. Coordinates are in meters, landing point The flight space boundary is: The direction value ranges from 0 meters to 100 meters. The direction value ranges from 0 meters to 80 meters. The direction ranges from 1 meter to 25 meters. The above values ​​are only specific choices for this embodiment.

[0028] The dynamic smoke field is constructed as follows: Two smoke sources are labeled. The initial center coordinates of source 1 are... Initial radius of influence The distance is 3 meters, and the diffusion speed is... The wind speed vector is 0.2 meters per second. for The initial center coordinates of source 2 are: Initial radius of influence The distance is 3 meters, and the diffusion speed is... The wind speed vector is 0.2 meters per second. for Set a wind speed change detection threshold: trigger an update when the wind direction change angle reaches or exceeds 30 degrees, or when the wind speed change reaches or exceeds 0.5 meters per second. Smoke safety factor. The value is 1.2, and the safety boundary is defined as follows: ,in The radius of the smoke at the current moment.

[0029] Static threats include radar and no-fly zones. This example labels three static threats. The center coordinates of the first threat are... The effective radius is 8 meters; the center coordinates of the second threat are... The effective radius is 6 meters; the center coordinates of the third threat are... Its effective radius is 6 meters.

[0030] The UAV adopts a 12-dimensional nonlinear quadrotor dynamics model, and its state vector is: The control input is the normalized speed of the four motors. The value of each rotational speed ranges from 0 to 1, where 0 indicates the motor is stopped and 1 indicates the motor's maximum speed. The simulation step size is set to 0.05 seconds, and the total number of simulation steps is 1000.

[0031] Global path planning The upper-level global planning adopts a hybrid optimization strategy that combines Q-learning and the gray wolf optimization algorithm. Figure 2 This demonstrates the iterative principle of RL-GWO hybrid optimization in the method described in this embodiment.

[0032] The path is encoded using control point coding, consisting of a starting point, several optimized control points, and an ending point connected sequentially. In this embodiment, the number of control points (excluding the starting and ending points) is set to 5, and the optimization dimension is 3 times the number of control points, i.e., 15 dimensions. The initial gray wolf population size is 30 individuals, and the maximum number of iterations is set to 100.

[0033] Each individual gray wolf is configured with an independent Q-learning agent. The state space of this Q-learning agent consists of three dimensions: the fitness ranking of the gray wolf population, the average smoke exposure level, and the current iteration stage. The action space of the Q-learning agent includes four types of actions: global exploration, local exploration, path optimization, and fine-tuning. During the main iteration, fitness evaluation, alpha wolf update, action selection and position update, boundary constraint handling, reward calculation, and Q-table update are performed sequentially. The fitness function adopts a multi-objective weighted sum, as follows: + The definitions of each cost term and the weight values ​​used in this embodiment are as follows: Path length cost In the formula This represents the actual total flight length of the path. This is the straight-line distance from the starting point to the ending point.

[0034] Threat Penetration Cost In the formula For the first The path sampling point to the first Euclidean distance of the threat center This represents the radius of effect of the threat.

[0035] Smoothness cost In the formula The turning angle formed by connecting the path to the adjacent control point at the control point is used to calculate the penalty when the turning angle exceeds 15 degrees. The maximum value among all turning angles that exceed this threshold is used in the calculation.

[0036] Feasibility constraint costs In the formula This represents the number of sampling points along the path whose altitude exceeds the flight safety boundary.

[0037] Smoke hard constraint penalty The value is: if any sampling point in the path enters the smoke area, then ;otherwise .

[0038] Straightness cost In the formula This represents the actual total flight length of the path. This is the straight-line distance from the starting point to the ending point.

[0039] After the iteration is completed, the optimal control point sequence is output, and the initial global path is generated by interpolation and smoothing.

[0040] Dynamic smoke field update In each simulation step, based on the current time And wind speed and direction data, update the coordinates of the center of each smoke source according to the smoke movement speed vector: Update the smoke radius according to the diffusion rate: ,in The wind direction influence coefficient is dynamically adjusted in this embodiment based on the angle between the wind direction and the smoke diffusion direction. The value ranges from 0 to 1. When the real-time monitored wind speed and direction data meet the conditions that the wind direction change angle reaches or exceeds 30 degrees, or the wind speed change reaches or exceeds 0.5 meters per second, the current wind speed vector is immediately updated. and wind direction influence coefficient Value. The safety boundary is defined as... This serves as the basis for subsequent crisis assessments.

[0041] It should be understood that the dynamic smoke field is updated in every planning cycle, i.e., every simulation step, and the updated smoke field information is passed to the global planner and local planner. This information is used for calculating the smoke exposure cost in the fitness assessment in global planning, and for determining unsafe states and calculating the reward function in local planning.

[0042] Local planning triggering and execution The drone flies along a global path, and in each simulation step, the distance from the current position to each smoke center is calculated in real time. When this distance is less than the warning distance, the spectral guided tree search local planning module is triggered. In this embodiment, the warning distance is set to 3 times the safety boundary. It should be noted that the warning distance can be dynamically set according to the smoke diffusion speed, drone flight speed, or mission requirements; for example, it can be set to 2 to 5 times the smoke safety boundary. Those skilled in the art will understand that any judgment based on the dynamic smoke threat distance can be used as the local planning condition. Figure 3 The spectral expansion process of SGT local planning in the method described in this embodiment is shown. Figure 4 The logic for determining unsafe states is demonstrated. The specific execution steps of the SGT module are as follows: The first step is dynamic linearization. Taking the current state of the UAV as the root node, the 12-dimensional nonlinear dynamic model is locally linearized to obtain the linearized state equations: in Let Jacobian be the state matrix. The input is the Jacobian matrix.

[0043] The second step is controllability analysis. This involves calculating the prediction time domain length. The controllable character Ram matrix within, in this embodiment The value is 10. The formula for calculating the controllable Galam matrix is: Then to Perform spectral decomposition: Obtain the eigenvalue matrix and eigenvector matrix .

[0044] The third step is to guide branch generation. A strongly controllable mode with eigenvalues ​​greater than a preset threshold is selected; in this embodiment, the preset threshold is set to 10% of the maximum eigenvalue. Multiple candidate reference trajectory branches are generated along the positive and negative directions of the eigenvector corresponding to the selected mode. Each branch generates a feasible trajectory through nonlinear dynamics tracking.

[0045] The fourth step is optimal branch selection. Monte Carlo tree search is used to evaluate each branch within a limited computational budget based on a reward function, selecting the branch with the highest cumulative reward as the local planning trajectory. In this embodiment, the maximum number of search nodes is set to 500. The node selection in Monte Carlo tree search uses the upper confidence bound formula: in For the first The cumulative reward for each child node, This represents the number of times the child node has been accessed. This represents the number of times the parent node has been visited. This is an exploration constant. In this embodiment, the exploration constant is... The value is 1.0. The reward function used in this embodiment is a combination of single-step reward and total discount reward. Single-step reward The calculation formula is: The weights are respectively taken as , , , Smoke avoidance bonus Defined as follows: If the drone enters the smoke area, the reward is 0; otherwise, the reward is... ,in The distance from the drone to the nearest center of the smoke. This is the preset maximum safe distance. In this embodiment, the maximum safe distance is set to 100 meters. Waypoint tracking reward. Defined as: in The coordinates of the drone's current position. For global reference waypoint coordinates, This represents the width of the Gaussian kernel. In this embodiment, the width of the Gaussian kernel is set to 10 meters.

[0046] Dynamic smoothing reward Defined as: in and These are the velocity vectors at two adjacent moments. and These are the angular velocity vectors at two adjacent moments. The preset maximum allowable rate of change of speed. This is the preset maximum allowable rate of change of angular velocity. In this embodiment, the maximum allowable rate of change of velocity is set to 10 meters per square second, and the maximum allowable rate of change of angular velocity is set to 5 radians per square second.

[0047] Obstacle avoidance reward Defined as: in, The minimum safety margin from the drone's current position to all static threat safety boundaries is expressed as: in, These are the current position coordinates of the drone. It is the i-th static threat center. Let be the security boundary radius of the i-th static threat. The preset buffer distance can be set to 6m.

[0048] Total Discount Rewards The calculation formula is: in This is the discount factor, which is set to 0.95 in this embodiment. This represents the total number of steps in the trajectory.

[0049] Step 5: Trajectory Smoothing. The generated local trajectories are smoothed using a moving average filter. The calculation formula is as follows: These are the coordinates of the k-th position after smoothing. To find the mean function; is the coordinate of the i-th position point in the original position sequence; the window interval is the range of position points centered at k with a length of L, and the average of available points is taken at the boundary. In this embodiment, the window length is set to 5.

[0050] The drone flies along the smoothed local obstacle avoidance trajectory until the threat is eliminated.

[0051] Over-limit detection and global replanning During flight, the following three indicators are continuously monitored: global path deviation, maximum waypoint deviation distance, and smoke penetration risk. When any one of these indicators exceeds its corresponding preset threshold, and the UAV is not currently in a local obstacle avoidance flight state, a global replanning is triggered. In this embodiment, the global path deviation threshold is set to 5 meters, the maximum waypoint deviation distance threshold is set to 10 meters, and the smoke penetration risk threshold is set to entering the smoke safety boundary. The replanning takes the current position as the new starting point, re-invokes the RL-GWO global planning module, and generates an updated global reference trajectory from the current point to the destination, enabling the UAV to adapt to environmental changes during subsequent flight.

[0052] Results Analysis Under the same dynamic smoke field and static threat environment, a control group was set up with the same simulation step size of 0.05 seconds and a total number of 1000 steps. The control group used the traditional GWO algorithm, which does not include Q-learning action adjustment and local planning modules. It generates a global path in a static environment, and the UAV flies along this path by pure tracking. The comparison results are as follows: the output path length of the control group is 100.59 meters, and the average smoke exposure value is 0.28; the method described in this embodiment outputs a path length of 83.94 meters, and the average smoke exposure value is 0.12.

[0053] Experimental results show that the method described in this embodiment, by introducing reinforcement learning dynamic policy adjustment and spectral guided tree search for local real-time obstacle avoidance, can more effectively avoid dynamic smoke regions. The path length is shortened by approximately 16.6%, and the average smoke exposure is reduced by approximately 57%. Figures 5 to 7 As shown, Figure 5 The traditional GWO path is shown, which is generally long, has large turning angles, poor smoothness, and is prone to penetrating smoke areas in some areas. Figure 6 The diagram illustrates the path generated by the method described in this embodiment. The green dashed circle represents the initial position of the dynamic smoke field, the arrows indicate the direction of smoke movement, the red empty circle represents the final position of the dynamic smoke, and the red solid circle represents radar threats or no-fly zones. It is evident that the path generated by the method described in this embodiment is shorter, smoother overall, has gentler turns, completely avoids smoke and static threats, can bypass sudden smoke diffusion in real time, and closely follows the flight boundary throughout the entire path. Figure 7 The 3D path diagram illustrates the local optimization effect of SGT: SGT calculates a controllable Gramma matrix and extracts natural motion modes based on a linearized model of the UAV's current state. Each mode corresponds to the motion direction with the minimum system energy consumption and the most sensitive response. When SGT generates branches along these mode directions, it tends to select the branch with the highest reward. The path adaptability and safety of the method described in this embodiment are superior to traditional methods, providing a reliable guarantee for the safe flight of UAVs in complex fire scenes and other scenarios.

[0054] In some embodiments, parameters such as the number of control points, the gray wolf population size, and the maximum number of iterations can be adjusted according to the actual task scenario and computing resources. For example, for more complex threat environments, the number of control points can be increased, such as by setting it to 8 or 10, to improve path compliance; for tasks with higher real-time requirements, the population size can be reduced to 20, and the maximum number of iterations can be reduced to 50.

[0055] In other embodiments, the smoke safety factor can be adjusted based on the smoke sensitivity of the UAV's sensors, and its value can range from 1.0 to 1.5. The maximum safe distance, Gaussian kernel width, maximum permissible rate of change of velocity, and maximum permissible rate of change of angular velocity can be calibrated according to the UAV model and flight performance. For example, for a small rotary-wing UAV, the maximum permissible rate of change of velocity can be set to 5 meters per square second, and the maximum permissible rate of change of angular velocity can be set to 3 radians per square second.

[0056] In some other embodiments, the warning distance multiplier in the local planning trigger condition can be dynamically adjusted according to the smoke diffusion speed. When the smoke diffusion speed is fast, a larger multiplier is used, such as 5 times; when the smoke diffusion speed is slow, a smaller multiplier is used, such as 2 times.

[0057] In summary, this invention proposes a two-layer path planning method for UAVs in dynamic smoke environments based on a fusion of reinforcement learning and gray wolf optimization. It constructs a smoke field model and a UAV dynamics model that dynamically evolve with wind speed and time. The upper layer employs a hybrid strategy integrating Q-learning and gray wolf optimization, generating a global reference trajectory with a multi-objective fitness function as the optimization objective. Each gray wolf individual is configured with an independent Q-learning agent, adaptively selecting search actions based on fitness ranking, smoke exposure level, and iteration stage. During flight, dynamic smoke threats are detected in real-time. When local planning is triggered, a spectral guided tree search planner is activated: after linearizing the dynamics model, a controllable Gram matrix is ​​calculated and spectral decomposed. Candidate branches are generated along the eigenvectors of strongly controllable modes, and the optimal locally feasible trajectory is selected by combining Monte Carlo tree search and a continuous reward function. Simultaneously, global path deviation, waypoint deviation distance, and smoke penetration risk are monitored. If any of these exceed a threshold and the UAV is not in a local obstacle avoidance state, a global replanning is triggered. This invention balances long-range optimality and real-time local obstacle avoidance capabilities in dynamic smoke environments.

[0058] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A two-layer path planning method for UAVs in dynamic smoke environments based on reinforcement learning-gray wolf optimization fusion, characterized in that, include: A mission environment model is constructed that includes a dynamic smoke field, static threats, and flight boundaries, and a dynamic model of the UAV is established, wherein the dynamic smoke field is dynamically updated over time according to preset smoke source parameters. A hybrid optimization strategy combining Q-learning and gray wolf optimization algorithm is adopted, with a multi-objective fitness function as the optimization objective, to generate a global reference trajectory from the starting point to the ending point; During the flight of the UAV along the global reference trajectory, the distance between the UAV and the dynamic smoke area is calculated in real time. When the distance is less than the preset warning distance, the spectrum-guided tree search planner is activated to perform online trajectory correction based on the dynamic model and generate a locally feasible trajectory to avoid the dynamic smoke. It also detects the deviation between the local feasible trajectory and the global reference trajectory in real time, as well as the risk of smoke penetration. When any indicator exceeds a preset threshold, it triggers global replanning and regenerates the global reference trajectory with the current position as the new starting point.

2. The method as described in claim 1, characterized in that, The dynamic update methods for the dynamic smoke field include: The center of the smoke moves according to the wind speed vector: The smoke radius increases linearly according to the following formula: in, Let the initial center coordinates of the smoke source be... The initial radius of influence. For diffusion rate, For wind speed vectors, For time steps, This is the wind direction influence coefficient, which is dynamically adjusted based on the angle between the wind direction and the smoke diffusion direction.

3. The method as described in claim 2, characterized in that, When the angle of sudden change in wind direction is monitored in real time or wind speed change Update immediately and value.

4. The method as described in claim 1, characterized in that, The multi-objective fitness function is expressed as follows: in, The cost is the path length. The cost of threatening to penetrate; This is the cost of path smoothness; The cost of feasibility constraints; Smoke-related hard constraint penalties; Cost of straightness; This is the actual length of the path. The straight-line distance from the starting point to the ending point. The preset path length cost weight; The first penalty coefficient is preset. For the first The path sampling point to the first The distance to the center of the threat Threat radius, Preset threat penalty weights; The second penalty coefficient is preset. The turning angle of the path at the control point. The preset smoothness weights; The third penalty coefficient is preset. This represents the number of sampling points in the path whose height exceeds the safety boundary. The preset feasibility weights; The maximum penalty value is preset and is triggered when the path sampling point enters the smoke safety zone, forcibly avoiding the smoke; This is the actual length of the path. This is the straight-line distance from the starting point to the ending point.

5. The method as described in claim 1, characterized in that, The hybrid optimization strategy that integrates Q-learning and the gray wolf optimization algorithm includes: The control point sequence of the path is used as the location code for individual gray wolves; Each individual gray wolf is equipped with an independent Q-learning agent; The state space of the Q-learning agent includes the fitness ranking of the gray wolf population, the average smoke exposure level, and the current iteration stage; The action space of the Q-learning agent includes global exploration, local exploration, path optimization, and fine-tuning. In each iteration of the Grey Wolf Optimization Algorithm, each individual selects and performs an action based on its current state to adjust its search behavior and receives a reward based on the improvement in fitness to update the Q-table. The position update rule of the Grey Wolf optimization algorithm is as follows: This represents the distance from the current gray wolf individual to the alpha wolf; These are random coefficients used to control the exploration range. Let X be the position of wolf α, and let X be the current position of the individual gray wolf. Let be the distance from the current gray wolf to the beta wolf. For random coefficients, The position of the β wolf; Let be the distance from the current gray wolf to the delta wolf. For random coefficients, The position of δ wolf; , , For the current gray wolf individual affected by α, β, The wolf-guided update position; , , The correlation coefficient is the convergence factor, used to balance exploration and development; The position of the current gray wolf in the next iteration; t is the current iteration number; Furthermore, the update formula for Q-learning is: Let Q be the value of the action a performed in state s. The learning rate controls the magnitude of Q-value updates; r is the immediate reward gained from the current action. This serves as a discount factor, controlling the weight of future rewards. The new state transitioned to after performing action a; For the new state The largest Q value among all actions.

6. The method as described in claim 1, characterized in that, The spectrum-guided tree search planner performs the following steps: linearizing the dynamic model based on the current state of the UAV to obtain the linearized state equation: in The drone state at step k+1; The drone state at step k; A is a 12×12 state Jacobian matrix. ; B is a 12×4 input Jacobian matrix. u(k) is the control input for k steps; c is the constant offset term; Calculate the prediction time domain length Internal controllable personality Ram matrix: The 12×12 controllable Gauram matrix measures the controllable energy of the system within H steps. Let A be the k-th power of the state matrix; This is the transpose of the input matrix B; H is the k-th power of the transpose of the state matrix A; H is the prediction time domain length; k is the summation index, from 0 to H-1; right Perform spectral decomposition: Obtain the eigenvalue matrix and eigenvector matrix Select a strongly controllable mode whose eigenvalues ​​are greater than a preset threshold, and generate multiple candidate reference trajectory branches along the positive and negative directions of its corresponding eigenvector; Monte Carlo tree search is used to evaluate each branch based on a reward function, and the branch with the highest cumulative reward is selected as the local programming trajectory. The node selection in the Monte Carlo tree search employs an upper confidence bound formula. in To accumulate rewards, For the number of times a node is accessed. This represents the number of visits to the parent node. To explore constants.

7. The method as described in claim 6, characterized in that, The reward function is a single-step reward. Total Discount Rewards Combinations: Single-step reward: Total reward for a single step; To mitigate the impact of smoke; Weighting of waypoint tracking rewards; Dynamically smooth reward weights; Rewards for avoiding smoke; Rewards for waypoint tracking; in, The distance from the drone to the nearest center of the smoke. This is the preset maximum safe distance; Location of the drone. As a global reference waypoint, The preset Gaussian kernel width; , They are velocity and angular velocity, respectively. , These are the preset maximum permissible rate of change of velocity and the maximum permissible rate of change of angular velocity, respectively. Obstacle avoidance reward Defined as: in, The minimum safety margin from the drone's current position to all static threat safety boundaries is expressed as: in, These are the current position coordinates of the drone. It is the i-th static threat center. Let be the security boundary radius of the i-th static threat. The preset buffer distance can be set to 6m; in Total discount reward As a discount factor, This is a single-step reward.

8. The method as described in claim 6, characterized in that, After generating the locally feasible trajectory, the process also includes smoothing the trajectory using a moving average filter: These are the coordinates of the k-th position after smoothing. To find the mean function; The coordinates of the i-th position in the original position sequence are given; the window interval is the range of position points centered at k with a length of L, and the average value of available points is taken at the boundary.

9. The method as described in claim 1, characterized in that, The real-time detection metrics include global path deviation, maximum waypoint deviation distance, and smoke penetration risk. When any metric exceeds its corresponding preset threshold and the drone is not currently in a local obstacle avoidance flight state, a global replanning is triggered.

10. A two-layer path planning system for UAVs in dynamic smoke environments based on reinforcement learning-gray wolf optimization fusion, used to execute the method according to any one of claims 1 to 9, characterized in that, include: The environmental perception and modeling module is used to build and update environmental models that include dynamic smoke fields and static threats in real time. The upper-level global planning module is used to generate a global reference trajectory using a hybrid optimization strategy that combines Q-learning and the gray wolf optimization algorithm. The lower-level local planning module is used to generate local obstacle avoidance trajectories using a spectrum-guided tree search planner. The over-limit detection and replanning module is used to monitor trajectory deviation and smoke penetration risk, and triggers the upper-level global planning module to perform global replanning when the limit is exceeded.

Images