An unmanned aerial vehicle city dynamic path planning method and system

Abstract

The application provides a kind of unmanned aerial vehicle city dynamic path planning method and system, method includes the action probability distribution of sub-target construction, and through the state quantity output of several moments of unmanned aerial vehicle and dynamic obstacle future multi-step prediction trajectory and space-time characteristics, to obtain space-time prediction characteristics;Output final action and execute;Through constraint filtering exception to obtain final dense reward and form LLM label reward data set;And with final dense reward as regression target and according to the confidence of final dense reward Weighted training MLP reward model.The application can alleviate the traditional reinforcement learning reward sparsity and artificial reward design subjective problem, improve convergence speed and enhance stability.

Description

Technical Field

[0001] This invention belongs to the field of UAV scheduling technology, specifically relating to a method and system for UAV urban dynamic path planning. Background Technology

[0002] With the development of the urban low-altitude economy and the widespread application of drones in intelligent inspection, logistics delivery, and emergency rescue, the mission environment is gradually expanding from relatively regular airspace to complex three-dimensional dynamic environments containing static buildings and various types of dynamic obstacles. This scenario requires drones to achieve efficient global reachability and real-time obstacle avoidance in local environments. While a single geometric planning method can provide a global path, it struggles to handle local conflicts in a timely manner under strong dynamic constraints. On the other hand, while a single reinforcement learning method has self-learning capabilities, it is susceptible to the influence of high-dimensional sparse rewards, resulting in slow convergence and unstable performance. Therefore, relying on only one technology is insufficient to meet the comprehensive optimization needs of complex low-altitude urban environments. It is necessary to construct a hierarchical path planning framework that structurally integrates global planning, local decision-making, and spatiotemporal prediction, and adaptively fuses them based on an attention mechanism, in order to balance global reachability and real-time obstacle avoidance in local environments.

[0003] In existing technologies, solutions to the problem of sparse rewards in reinforcement learning typically rely on manually designed reward terms and fixed weights, making them sensitive to reward parameters. Therefore, they fall short in terms of safety, efficiency, and stability for drone path planning in complex urban low-altitude multi-drone scenarios. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method and system for dynamic urban path planning by unmanned aerial vehicles (UAVs), thereby overcoming the shortcomings of the prior art.

[0005] In a first aspect, the present invention provides a method for urban dynamic path planning of unmanned aerial vehicles (UAVs), the method comprising:

[0006] A unified 3D grid map is constructed, and city buildings are discretized onto the grid of the 3D grid map through parameters, and dynamic obstacles are introduced into the 3D grid map;

[0007] Establish an unmanned aerial vehicle (UAV) system model, obtain the state observation vectors of the UAV on the three-dimensional grid map based on the UAV system model, and design the action space of the UAV;

[0008] Based on the motor neural planning network, the three-dimensional grid map, the state observation vector, and the action space, the action probability distribution of the sub-target is constructed. The predicted trajectory and spatiotemporal features for the next multiple steps are output through the state variables of the UAV and dynamic obstacles at several moments, so as to obtain the spatiotemporal prediction features.

[0009] Calculate the fusion coefficients of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector, output the final action, and execute it;

[0010] Record the state change information at each step of the execution process, calculate the reward for the next K steps and the future collision flag, and write the state change information, the reward and the future collision flag into a structured decision log.

[0011] Read the decision steps of the structured decision log and select key decision steps, construct the LLM cue words for each key decision step, and output candidate rewards and confidence scores. Filter out anomalies through constraints to obtain the final dense rewards and form an LLM labeled reward dataset.

[0012] In the reward dataset, a unified feature vector is constructed for each key decision step based on the state and action information used in online decision-making. The MLP reward model is trained with the final dense reward as the regression target and weighted according to the confidence of the final dense reward.

[0013] Compared with existing technologies, the beneficial effects of this invention are: by combining global planning of motor neural network, spatiotemporal prediction of reinforcement learning decision-making and self-attention hierarchical fusion framework, it can realize UAV autonomous path planning in complex urban dynamic low-altitude environment that takes into account global accessibility, local obstacle avoidance safety and path efficiency; combined with dense reward fine-tuning based on large language model, it can alleviate the problems of sparse rewards in traditional reinforcement learning and subjective design of manual rewards, improve convergence speed and enhance stability.

[0014] Furthermore, the step of constructing the action probability distribution of the sub-target based on the motor neural planning network, the three-dimensional grid map, the state observation vector, and the action space, and outputting the predicted trajectory and spatiotemporal features for future multiple steps through the state variables of the UAV and dynamic obstacles at several moments to obtain the spatiotemporal prediction features includes:

[0015] Offline training of the motor neural planning network generates correctly reachable sub-goals and guides the MPNet within the motor neural planning network. The A* module performs path search to obtain the optimal sequence of collision-free path points;

[0016] Calculate the three-dimensional displacement between the first path point in the optimal path point sequence and the current position of the UAV, and encode the three-dimensional displacement as a unique action index;

[0017] The main probability is assigned to the main action based on the unique action index, and the remaining probabilities are equally distributed among the other action indices to construct the action probability distribution of the sub-target.

[0018] Construct an STGNP predictor using a spatiotemporal graph neural network;

[0019] Based on the STGNP predictor, and by using the state variables of the UAV and dynamic obstacles at several moments, the predicted trajectory and spatiotemporal features for the next several steps are output, and spatiotemporal prediction features are generated based on the predicted trajectory and the spatiotemporal features.

[0020] Furthermore, the step of calculating the fusion coefficients of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector includes:

[0021] Output the Q-value distribution vector based on the action probability distribution;

[0022] The fusion coefficients of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector of the decision-maker are calculated based on the hierarchical attention fusion module and attention gating weights.

[0023] Furthermore, the steps of reading the decision steps of the structured decision log and selecting key decision steps, constructing LLM cue words for each key decision step, outputting candidate rewards and confidence scores, filtering anomalies through constraints to obtain the final dense reward, and forming an LLM-annotated reward dataset include:

[0024] Read the decision steps from the structured log, calculate the importance score, and filter key decision steps according to preset conditions;

[0025] Construct LLM cue words for each of the key decision steps, and output candidate rewards and confidence levels with multiple temperatures and multiple samples by evaluating the LLM;

[0026] Anomalies are filtered out by regulating LLM constraints to obtain the final dense reward and form an LLM-annotated reward dataset.

[0027] Furthermore, after the step of training the MLP reward model with the final dense reward as the regression target and weighted according to the confidence of the final dense reward, the method further includes:

[0028] The MLP reward model is weighted and fused with the original reward to form a unified reward, and the MLP reward model is trained in a closed loop.

[0029] The MLP reward model and the original reward are compared after closed-loop training to obtain the comparison results.

[0030] Secondly, the present invention also provides an unmanned aerial vehicle (UAV) urban dynamic path planning system, the system comprising:

[0031] The module is used to construct a unified 3D grid map, discretize city buildings onto the grid of the 3D grid map through parameters, and introduce dynamic obstacles onto the 3D grid map.

[0032] A design module is established to build an unmanned aerial vehicle (UAV) system model, obtain the state observation vectors of the UAV on the three-dimensional grid map based on the UAV system model, and design the action space of the UAV.

[0033] An output module is constructed to build the action probability distribution of the sub-target based on the motor neural planning network, the three-dimensional grid map, the state observation vector and the action space, and output the predicted trajectory and spatiotemporal features of the future multiple steps through the state variables of the UAV and dynamic obstacles at several moments, so as to obtain the spatiotemporal prediction features.

[0034] The calculation output module is used to calculate the fusion coefficient of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector, output the final action, and execute it.

[0035] The recording and calculation module is used to record the state change information of each step in the execution process, calculate the reward of the next K steps and the future collision flag, and write the state change information, the reward and the future collision flag into the structured decision log.

[0036] The read construction module is used to read the decision steps of the structured decision log and select key decision steps, construct the LLM cue words for each key decision step, and output candidate rewards and confidence scores. Anomalies are filtered through constraints to obtain the final dense rewards and form an LLM labeled reward dataset.

[0037] A training module is constructed to build a unified feature vector for each key decision step in the reward dataset based on the state and action information used in online decision-making, and to train an MLP reward model with the final dense reward as the regression target and weighted according to the confidence of the final dense reward.

[0038] Furthermore, the output module includes:

[0039] A search unit is generated to train the motor neural network offline, generating correctly reachable sub-objectives and guiding the MPNet within the motor neural network. The A* module performs path search to obtain the optimal sequence of collision-free path points;

[0040] The computational encoding unit is used to calculate the first path point in the optimal path point sequence and the current position of the UAV to obtain the three-dimensional displacement, and encode the three-dimensional displacement as a unique action index;

[0041] The construction unit is assigned a main probability based on the unique action index as the main action, and the remaining probabilities are equally distributed among other action indices to construct the action probability distribution of the sub-target.

[0042] Building units are used to construct STGNP predictors using spatiotemporal graph neural networks;

[0043] The first output unit is used to output the predicted trajectory and spatiotemporal features for the next multiple steps based on the STGNP predictor and the state variables of the UAV and dynamic obstacles at several moments, and to generate spatiotemporal prediction features based on the predicted trajectory and the spatiotemporal features.

[0044] Furthermore, the calculation output module includes:

[0045] The second output unit is used to output the Q-value distribution vector based on the action probability distribution;

[0046] The calculation unit is used to calculate the fusion coefficients of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector of the decision-maker based on the hierarchical attention fusion module and attention gating weights.

[0047] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described UAV urban dynamic path planning method.

[0048] Fourthly, the present invention also provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the above-described UAV urban dynamic path planning method. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of the UAV urban dynamic path planning method in the first embodiment of the present invention;

[0051] Figure 2 This is a flowchart of the STGNP predictor in the first embodiment of the present invention;

[0052] Figure 3 This is a flowchart of the hierarchical attention fusion module in the first embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the main steps of LLM intensive reward generation in the first embodiment of the present invention;

[0054] Figure 5This is a schematic diagram of the dual LLM core prompt design in the first embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the main steps of closed-loop training in the first embodiment of the present invention;

[0056] Figure 7 This is a structural block diagram of the UAV urban dynamic path planning system in the second embodiment of the present invention;

[0057] Figure 8 This is a schematic diagram of the structure of the electronic device in the third embodiment of the present invention.

[0058] Explanation of key component symbols:

[0059] 10. Construct the introductory module; 20. Establish the design module; 30. Construct the output module; 40. Calculate the output module; 50. Record the calculation module; 60. Read the constructed module; 70. Construct the training module;

[0060] 80. Bus; 81. Processor; 82. Memory; 83. Communication interface.

[0061] The embodiments of the present invention will be further described below with reference to the accompanying drawings. Detailed Implementation

[0062] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0063] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0065] Example 1

[0066] Please see Figure 1 The figure shows a UAV urban dynamic path planning method in the first embodiment of the present invention, the method comprising steps S1 to S7:

[0067] S1, construct a unified three-dimensional grid map, discretize city buildings onto the grid of the three-dimensional grid map through parameters, and introduce dynamic obstacles into the three-dimensional grid map;

[0068] It should be noted that a unified 3D grid map is constructed by discretizing static city buildings onto the grid through parameters and representing obstacle areas by occupying grids. At the same time, four types of dynamic obstacles are introduced: flocks of birds, construction machinery, mission-oriented drones, and patrol drones. Each type of dynamic obstacle includes state variables such as position, speed, and orientation, and the motion is determined by the mission-driven term and the obstacle avoidance force field term.

[0069] It is worth noting that: mission-oriented UAVs: follow preset mission waypoints along the path guided by the mission, calculate obstacle avoidance forces based on distance and relative speed with obstacles within the observation radius, and synthesize these forces with the desired speed to update the motion state; patrol UAVs: patrol along three predefined trajectories: straight line, figure-eight, and random, using PID error control; bird flocks: based on Reynolds Boids theory, superimpose obstacle avoidance forces and wind field forces on each individual bird in addition to the three types of group forces (separation, alignment, and aggregation), and update the motion; construction machinery: adopt a dual-state model of excavation and slow movement. In the excavation state, the radius of the danger zone is dynamically adjusted according to the progress of the operation. In the movement state, the machinery moves at low speed within the operation radius and applies enhanced obstacle avoidance forces. The updated dynamic obstacle motion results are mapped onto a three-dimensional grid to achieve a unified spatiotemporal occupancy representation of static and dynamic obstacles in the same coordinate system.

[0070] S2, Establish an unmanned aerial vehicle (UAV) system model, obtain the UAV's state observation vector on the three-dimensional grid map based on the UAV system model, and design the UAV's action space;

[0071] It should be noted that a single UAV moves between three-dimensional grids with a fixed step size in each decision step. Its state observation vector consists of three-dimensional grids within a local observation range centered on the UAV, relative position and distance information pointing to the current sub-target or endpoint, and several key dynamic features. The action space design consists of a discrete action set composed of 27 three-dimensional discrete displacement vectors, expressed as:

[0072] ;

[0073] ;

[0074] In the formula, This represents the discrete action space of the drone on a three-dimensional grid. , , They represent Unit grid displacement step size in the axial direction Unit grid displacement step size in the axial direction The unit grid displacement step size in the axis direction, where a value of -1 indicates movement along the negative direction of the corresponding coordinate axis, a value of 0 indicates no movement on the coordinate axis, and a value of 1 indicates movement along the positive direction of the corresponding coordinate axis. , They represent in The position vector of the drone at any time, The drone's position vector at any given time; the drone's original reward function uses environmental rewards. With termination of reward The combined form is expressed as:

[0075] ;

[0076] ;

[0077] ;

[0078] ;

[0079] In the formula, This represents the total reward obtained by the drone within a single time step, which is obtained by adding the environmental reward and the termination reward. , , , These represent climb bonus, target progress bonus, collision risk penalty, and waypoint bonus, respectively. , These represent the collision severity coefficient and the dynamic obstacle type coefficient, respectively. Indicates the actual distance at the time of the collision. This represents the effective collision radius; specific values ​​are shown in Table 1:

[0080] Table 1 Dynamic obstacle type coefficients and their effective collision radius

[0081]

[0082] S3, construct the action probability distribution of the sub-target based on the motor neural planning network, the three-dimensional grid map, the state observation vector and the action space, and output the predicted trajectory and spatiotemporal features of the future multiple steps through the state variables of the UAV and dynamic obstacles at several moments, so as to obtain the spatiotemporal prediction features.

[0083] Specifically, step S3 includes steps S31 to S35:

[0084] S31, Offline training of the motor neural planning network generates correctly reachable sub-goals and guides the MPNet in the motor neural planning network. The A* module performs path search to obtain the optimal sequence of collision-free path points;

[0085] S32, calculate the first path point in the optimal path point sequence and the current position of the UAV to obtain the three-dimensional displacement, and encode the three-dimensional displacement as a unique action index;

[0086] S33, assign a main probability to the main action with the unique action index, and make the remaining probabilities equally distributed among other action indices to construct the action probability distribution of the sub-target;

[0087] It should be noted that in complex 3D environments, the Motor Neural Planning Network (MPNet) learns expert trajectories for MPNet. The A* module provides high-quality sub-objectives, thereby reducing ineffective expansion and lowering average planning time while maintaining path feasibility. The motor neural network first receives normalized environmental location information, then extracts 128-dimensional environmental features through a three-layer 3D convolutional structure. After being concatenated with the environmental features, it is fed into a two-layer fully connected network with hidden layer dimensions of 128 and 64, respectively. Finally, it outputs normalized 3D sub-objective coordinates through linear layers and sigmoid activation, which serve as the MPNet output. The A* module searches for the endpoint, obtaining a collision-free optimal path point sequence. Then, the three-dimensional displacement is calculated by taking the first path point in this sequence and comparing it with the UAV's current position. It is then encoded into a unique action index in a 27-dimensional discrete action space, expressed as:

[0088] ;

[0089] ;

[0090] In the formula, , , They represent respectively to Integer encoding after linear shift, and... Integer encoding after linear shift, and... Integer encoding after linear shift, The unique action number is represented by the combination of three integer codes; the unique action index is assigned the main probability as the main action, and the remaining probabilities are equally distributed among the other actions to construct an action probability distribution, which is then further input into the hierarchical attention fusion (HAF) module.

[0091] S34, construct the STGNP predictor using a spatiotemporal graph neural network;

[0092] S35, based on the STGNP predictor and by using the state variables of the UAV and dynamic obstacles at several moments, output the predicted trajectory and spatiotemporal features for the next multiple steps, and generate spatiotemporal prediction features based on the predicted trajectory and the spatiotemporal features.

[0093] It should be noted that the STGNP predictor adopts a spatiotemporal graph neural network structure to jointly model UAVs and multiple types of dynamic obstacles, providing situational awareness and multi-step risk prediction for the hierarchical fusion framework.

[0094] The STGNP predictor first treats all drones and dynamic obstacles in the scene as graph nodes, and models them uniformly as a graph. , This represents the set of nodes consisting of all drones and dynamic obstacles. This represents the set of edges established based on spatial proximity, and adjacency relationships are constructed based on their three-dimensional Euclidean distances, resulting in an adjacency matrix that characterizes the spatial topology. , This represents the total number of nodes in the scene, and the state variables of each node are constructed into a 35-dimensional node feature vector to generate a node feature matrix. , express A 64-dimensional latent space is then generated. The node features are then mapped to a unified 64-dimensional latent space by the node encoder and displayed in the adjacency matrix. Under constraints, the relative motion and interactive threats between the drone and dynamic obstacles are explicitly modeled by passing the drone through a three-layer attention network.

[0095] Furthermore, to capture motion patterns in the temporal dimension, the STGNP predictor maintains a history buffer that stores up to 10 historical steps. When at least 3 historical steps have been accumulated, the graph feature sequences from several consecutive time points are rearranged into a 64-dimensional time series, which is then input into a two-layer LSTM with a hidden dimension of 64 for temporal modeling. When the historical steps are insufficient, prediction is made only based on the current graph feature output. Finally, the prediction head maps the spatiotemporal joint representation of each node to a node state prediction sequence for the next 5 steps and the corresponding 64-dimensional spatiotemporal features, which are then further input into the HAF module. The STGNP predictor flowchart is shown below. Figure 2 As shown.

[0096] S4, calculate the fusion coefficient of the action probability distribution, the spatiotemporal prediction feature, and the Q-value distribution vector, output the final action, and execute it;

[0097] Specifically, step S4 includes steps S41 to S42:

[0098] S41, outputs the Q-value distribution vector based on the action probability distribution;

[0099] S42, calculate the fusion coefficient of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector of the decision-maker based on the hierarchical attention fusion module and attention gating weight;

[0100] It should be noted that the decision-maker is a D3QN decision-maker, employing a Dueling DQN architecture. The state feature vector is input into a shared feature extraction network, passed through two fully connected layers, and Dropout is added to suppress overfitting. This extracts 64-dimensional hidden state features and constructs state value streams and action advantage streams, ultimately yielding Q-value vectors for 27 candidate actions. During training, a Double DQN mechanism is used to mitigate overestimation, and experience replay and soft updates are employed to ensure stability. The final result is input into a Hierarchical Attention Fusion (HAF) module for further processing.

[0101] It is worth noting that the state encoder and STGNP feature encoder in the multimodal attention layer of the hierarchical attention fusion module transform the current environment state and the 64-dimensional spatiotemporal features output by STGNP through a fully connected network, respectively, and then concatenate and compress them into a low-dimensional unified representation vector in the fusion encoder. This vector simultaneously contains task stage information and local threat level information. Based on this representation vector, the attention weight generator adaptively generates the weights for the three information streams of the D3QN decision-maker, hybrid planner, and STGNP predictor. and a fusion gating factor This is used to control the relative contributions of multi-source information in different scenarios. In the feature encoding layer, the Q-values ​​of the D3QN decision-maker are used to obtain the probability distribution via softmax. The action probability distribution of the MPNet-A* hybrid planner is mapped to a 27-dimensional vector. The STGNP multi-step trajectory prediction is flattened by time step and fed into the prediction encoder, where it is converted into 27-dimensional action-related features. The weighted fusion layer first utilizes The element-wise weighting of the three information streams is expressed as follows:

[0102] ;

[0103] In the formula, This is the original action probability vector output by the D3QN decision maker. For the feature vector of the hybrid planner, Predict the feature vector for STGNP; , , These represent the weights of the D3QN decision-maker information flow, the weights of the hybrid planner information flow, and the weights of the STGNP predictor information flow, respectively. This represents element-wise multiplication of vectors. , , These respectively represent the weight vector , , Under the influence of, , and The result obtained by element-wise weighting; the three weighted results are then compared with the original D3QN probability. The initial fusion results were obtained by inputting the fully connected fusion network together. Subsequently, a fusion gating factor was introduced. By generating residual interpolation, the final action selection probability is obtained, expressed as:

[0104] ;

[0105] In the formula, This represents the probability of the final action choice; when the scene is simple and obstacles are sparse, Reduce; when the environment is complex or high risks are predicted. This increases the robustness of decision-making while ensuring safety. The flowchart of the hierarchical attention fusion module is as follows: Figure 3 As shown.

[0106] S5, record the state change information of each step in the execution process, calculate the reward of the next K steps and the future collision flag, and write the state change information, the reward and the future collision flag into the structured decision log.

[0107] It should be noted that during the training process, the decision-making process is recorded to form a structured decision log. Each record in the log contains an index, action number, a state summary including the drone's position and estimated collision probability, event flags, and reward information. After each round, labels such as "cumulative reward for the next k steps" and "whether a collision will occur within the next few steps" are added. The generated decision log serves as the basis for LLM dense reward generation.

[0108] S6, read the decision steps of the structured decision log and select key decision steps, construct the LLM cue words for each key decision step, and output candidate rewards and confidence scores. Filter out anomalies through constraints to obtain the final dense rewards and form an LLM labeled reward dataset.

[0109] Specifically, step S6 includes steps S61 to S63:

[0110] S61, Read the decision steps in the structured log, calculate the importance score, and filter key decision steps through preset conditions;

[0111] S62, construct LLM cue words for each of the key decision steps, and output candidate rewards and confidence levels by evaluating the LLM with multiple temperatures and multiple samples;

[0112] S63, filtering out anomalies through supervised LLM constraints to obtain the final dense reward and form an LLM labeled reward dataset;

[0113] It should be noted that an importance score is calculated for each decision step. The trajectory of each round is ranked and filtered according to the importance score, taking into account physical quantities such as whether the current step is the termination step and the termination type, whether a collision will occur in the next few steps, the change in distance to the target, and the distance to the nearest obstacle.

[0114] The selection of key decision steps includes two modes: a risk mode selects steps near collisions or high-risk events, while a good sample mode selects steps that are safe and efficient in approaching the target, excluding collisions. The selected key steps are evaluated using a prompt constructed by integrating the environment, state summaries before and after the step, environmental rewards, and evaluation criteria. This prompt is repeatedly invoked to evaluate the LLM, with repeated sampling and evaluation at LLM temperatures of 0.1 and 0.7 to obtain candidate dense rewards, confidence scores, and brief textual explanations [reward, confidence, rationale]. This information is then fed into a supervisory LLM, where each candidate is evaluated for acceptability based on predefined filtering rules, such as prohibiting positive rewards in the event of a collision. Acceptable candidates are robustly aggregated using extreme value weighted averaging to generate the final dense reward and corresponding confidence score. The record information corresponding to the key steps and the dense reward confidence scores are used to construct the LLM-annotated dense reward dataset required for subsequent MLP training. The main steps are as follows: Figure 4 As shown, the core prompt design for evaluating and regulating LLM is as follows: Figure 5 As shown.

[0115] S7. Construct a unified feature vector for each key decision step in the reward dataset according to the state and action information used in online decision-making, and train an MLP reward model with the final dense reward as the regression target and weighted according to the confidence of the final dense reward.

[0116] Specifically, step S7 further includes steps S71 to S72:

[0117] S71, the MLP reward model is weighted and fused with the original reward to form a unified reward, and the MLP reward model is trained in a closed loop.

[0118] S72, compare the MLP reward model and the original reward after closed-loop training to obtain the comparison result;

[0119] It should be noted that, firstly, the LLM-annotated dense reward dataset generated in the previous stage is loaded offline. The physical information and action of each keystep are encoded into a fixed-length numerical feature vector, including the cropped and scaled distance to the target and its most recent step change, the probability of collisions before and after, the distances to the nearest obstacles before and after, the original environmental reward, and the one-hot encoding of the current 27-dimensional discrete action. Then, using these features as input and the dense reward as the supervision signal, a small feedforward neural network consisting of two fully connected layers and a tanh output layer is trained, and basic weights are generated. When the collected round trajectories are sample data from the early and middle stages of training, i.e., risk s... Under the premise that TEP (Time-Effective Prediction) dominates, a weighted mean square error (MSE) method with sample weight modulation is used during training. This involves multiplying the "good samples" with positive dense rewards by an adjustable amplification factor (here, 3.0) on top of the sample's base weights. The optimizer then iteratively updates the network parameters under this weighted MSE objective and evaluates the unweighted MSE once on the dataset, resulting in a dense reward MLP model. Finally, this MLP model is loaded for closed-loop training. During training, the dense reward is predicted in real-time at each time step based on the current physical features and actions, and the original environmental reward is linearly fused with the dense reward, expressed as:

[0120] ;

[0121] in, The actual rewards deposited into the experience pool. As a primitive environmental reward, For the predicted dense reward, This is an adjustable weight used to control the intensity of the impact of the large model's dense rewards on the overall training signal; it is set to 1.0 here. Throughout the process, the environment's termination condition and success determination logic remain unchanged, except when writing to the experience replay buffer and performing Q-network updates. This replaces standalone environmental rewards, enabling a reward-shaping mechanism of "collecting sample data - offline large model annotation - small model approximation - online closed-loop fusion," thus achieving a systematic improvement in both safety and efficiency while maintaining the original task objectives. Its main steps are as follows: Figure 6 As shown in Table 2, offline semantic evaluation of the drone trajectory and training of a dense reward model are performed to controllably correct and enhance the original environmental reward. This part mainly includes trajectory acquisition, LLM dense reward generation, and closed-loop training. The main parameters and settings are shown in Table 2.

[0122] Table 2 Key parameters and settings for fine-tuning the large model reward

[0123]

[0124] In addition, a self-attention hierarchical fusion framework combining MPNet-A* hybrid planner, D3QN (Dueling Double DQN) decision maker, and STGNP predictor is presented, with hyperparameters shown in Table 3.

[0125] Table 3. Hyperparameters of the framework algorithm

[0126]

[0127] The experimental map was set to 85×85×20 units, with three fixed drones. The environment included randomly generated static buildings of varying heights, sizes, and dimensions, as well as different types of dynamic obstacles. The results of comparing the cumulative reward and success rate of the hierarchical fusion framework (HAMS-D3QN-LLM-reward) fine-tuned based on the large model reward with four other baselines showed that the present invention significantly improved upon the baselines, with a win rate approaching 80%. The hierarchical fusion framework with LLM reward fine-tuning showed some improvement in reward and a slight improvement in success rate, which was more pronounced in the later stages of training. Furthermore, the training trajectory of the method was visualized. When buildings were densely packed or dynamic obstacles were detected, the drones could turn or ascend to avoid them. Simultaneously, the obstacle trajectories conformed to the pre-defined modeling dynamics.

[0128] In summary, the UAV urban dynamic path planning method in the above embodiments of the present invention combines global planning of motor neural network, spatiotemporal prediction of reinforcement learning decision-making, and self-attention hierarchical fusion framework to achieve autonomous UAV path planning in complex urban dynamic low-altitude environments that takes into account global accessibility, local obstacle avoidance safety, and path efficiency. Combined with dense reward fine-tuning based on a large language model, it can alleviate the problems of sparse rewards in traditional reinforcement learning and subjective design of manual rewards, improve convergence speed, and enhance stability.

[0129] Example 2

[0130] Please see Figure 7 The figure shows a UAV urban dynamic path planning system according to a second embodiment of the present invention, the system comprising:

[0131] The construction module 10 is used to construct a unified three-dimensional grid map, discretize city buildings onto the grid of the three-dimensional grid map through parameters, and introduce dynamic obstacles onto the three-dimensional grid map.

[0132] A design module 20 is established to build an unmanned aerial vehicle (UAV) system model, obtain the state observation vectors of the UAV on the three-dimensional grid map based on the UAV system model, and design the action space of the UAV.

[0133] The output module 30 is constructed to construct the action probability distribution of the sub-target based on the motor neural planning network, the three-dimensional grid map, the state observation vector and the action space, and output the predicted trajectory and spatiotemporal features of the future multiple steps through the state variables of the UAV and dynamic obstacles at several moments, so as to obtain the spatiotemporal prediction features.

[0134] The calculation output module 40 is used to calculate the fusion coefficient of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector, output the final action, and execute it.

[0135] The recording and calculation module 50 is used to record the state change information of each step in the execution process, calculate the reward of the next K steps and the future collision flag, and write the state change information, the reward and the future collision flag into the structured decision log.

[0136] The read construction module 60 is used to read the decision steps of the structured decision log and select key decision steps, construct the LLM prompt words for each key decision step, and output candidate rewards and confidence scores. Anomalies are filtered through constraints to obtain the final dense rewards and form an LLM labeled reward dataset.

[0137] A training module 70 is constructed to build a unified feature vector for each key decision step in the reward dataset based on the state and action information used in online decision-making, and to train an MLP reward model with the final dense reward as the regression target and weighted according to the confidence of the final dense reward.

[0138] In some alternative embodiments, the build output module 30 includes:

[0139] A search unit is generated to train the motor neural network offline, generating correctly reachable sub-objectives and guiding the MPNet within the motor neural network. The A* module performs path search to obtain the optimal sequence of collision-free path points;

[0140] The computational encoding unit is used to calculate the first path point in the optimal path point sequence and the current position of the UAV to obtain the three-dimensional displacement, and encode the three-dimensional displacement as a unique action index;

[0141] The construction unit is assigned a main probability based on the unique action index as the main action, and the remaining probabilities are equally distributed among other action indices to construct the action probability distribution of the sub-target.

[0142] Building units are used to construct STGNP predictors using spatiotemporal graph neural networks;

[0143] The first output unit is used to output the predicted trajectory and spatiotemporal features for the next multiple steps based on the STGNP predictor and the state variables of the UAV and dynamic obstacles at several moments, and to generate spatiotemporal prediction features based on the predicted trajectory and the spatiotemporal features.

[0144] In some alternative embodiments, the calculation output module 40 includes:

[0145] The second output unit is used to output the Q-value distribution vector based on the action probability distribution;

[0146] The calculation unit is used to calculate the fusion coefficients of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector of the decision-maker based on the hierarchical attention fusion module and attention gating weights.

[0147] In some alternative embodiments, the read construction module 60 includes:

[0148] The reading unit is used to read the decision steps in the structured log, calculate the importance score, and filter key decision steps according to preset conditions;

[0149] An output unit is constructed to build LLM cue words for each key decision step, and outputs candidate rewards and confidence scores with multiple temperatures and multiple samples by evaluating the LLM.

[0150] Forming units are used to filter out anomalies through supervised LLM constraints to obtain the final dense reward and form an LLM-annotated reward dataset.

[0151] In some alternative embodiments, the construction training module 70 includes:

[0152] A closed-loop training unit is used to weightedly fuse the MLP reward model with the original reward to form a unified reward, and to perform closed-loop training on the MLP reward model.

[0153] The comparison unit is used to compare the MLP reward model obtained after closed-loop training with the original reward to obtain the comparison result.

[0154] The functions or operation steps implemented by the above modules and units are largely the same as those in the above method embodiments, and will not be repeated here.

[0155] The UAV urban dynamic path planning system provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0156] Example 3

[0157] The third embodiment of the present invention also proposes an electronic device, please refer to [link / reference]. Figure 8 The image shows an electronic device according to a third embodiment of the present invention.

[0158] The electronic device may include a processor 81 and a memory 82 storing computer program instructions.

[0159] Specifically, the processor 81 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement this application.

[0160] The memory 82 may include a mass storage device for data or instructions. For example, and not limitingly, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include removable or non-removable (or fixed) media. Where appropriate, the memory 82 may be internal or external to a data processing device. In a particular embodiment, the memory 82 is non-volatile memory. In a particular embodiment, the memory 82 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0161] The memory 82 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 81.

[0162] The processor 81 reads and executes the computer program instructions stored in the memory 82 to implement the UAV urban dynamic path planning method of Embodiment 1 described above.

[0163] In some embodiments, the electronic device may further include a communication interface 83 and a bus 80. For example, Figure 8 As shown, the processor 81, memory 82, and communication interface 83 are connected through bus 80 and complete communication with each other.

[0164] The communication interface 83 is used to enable communication between the various modules, devices, units, and / or equipment in this application. The communication interface 83 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0165] Bus 80 includes hardware, software, or both, that couples the components of a device together. Bus 80 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although this application describes and illustrates a specific bus, this application considers any suitable bus or interconnection.

[0166] The electronic device can acquire the UAV urban dynamic path planning system and execute the UAV urban dynamic path planning method of this embodiment.

[0167] Furthermore, in conjunction with the UAV urban dynamic path planning method in Embodiment 1 above, this application can provide a storage medium for implementation. This storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement the UAV urban dynamic path planning method of Embodiment 1 above.

[0168] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for dynamic path planning of unmanned aerial vehicles (UAVs) in urban areas, characterized in that, The method includes: A unified 3D grid map is constructed, and city buildings are discretized onto the grid of the 3D grid map through parameters, and dynamic obstacles are introduced into the 3D grid map; Establish an unmanned aerial vehicle (UAV) system model, obtain the state observation vectors of the UAV on the three-dimensional grid map based on the UAV system model, and design the action space of the UAV; Based on the motor neural planning network, the 3D grid map, the state observation vector, and the action space, the action probability distribution of the sub-target is constructed. Then, by using the state variables of the UAV and dynamic obstacles at several moments, the predicted trajectory and spatiotemporal features for the next several steps are output to obtain the spatiotemporal prediction features. This step specifically includes: Offline training of the motor neural planning network generates correctly reachable sub-goals and guides the MPNet within the motor neural planning network. The A* module performs path search to obtain the optimal sequence of collision-free path points; Calculate the three-dimensional displacement between the first path point in the optimal path point sequence and the current position of the UAV, and encode the three-dimensional displacement as a unique action index; The main probability is assigned to the main action based on the unique action index, and the remaining probabilities are equally distributed among the other action indices to construct the action probability distribution of the sub-target. Construct an STGNP predictor using a spatiotemporal graph neural network; Based on the STGNP predictor, and by using the state variables of the UAV and dynamic obstacles at several moments, the predicted trajectory and spatiotemporal features for the next multiple steps are output, and spatiotemporal prediction features are generated based on the predicted trajectory and the spatiotemporal features. Calculate the fusion coefficients of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector, output the final action, and execute it; Record the state change information at each step of the execution process, calculate the reward for the next K steps and the future collision flag, and write the state change information, the reward and the future collision flag into a structured decision log. Read the decision steps of the structured decision log and select key decision steps, construct the LLM cue words for each key decision step, and output candidate rewards and confidence scores. Filter out anomalies through constraints to obtain the final dense rewards and form an LLM labeled reward dataset. In the reward dataset, a unified feature vector is constructed for each key decision step based on the state and action information used in online decision-making. The MLP reward model is trained with the final dense reward as the regression target and weighted according to the confidence of the final dense reward.

2. The UAV urban dynamic path planning method according to claim 1, characterized in that, The step of calculating the fusion coefficients of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector includes: Output the Q-value distribution vector based on the action probability distribution; The fusion coefficients of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector of the decision-maker are calculated based on the hierarchical attention fusion module and attention gating weights.

3. The UAV urban dynamic path planning method according to claim 1, characterized in that, The steps of reading the decision steps from the structured decision log and selecting key decision steps, constructing LLM cue words for each key decision step, outputting candidate rewards and confidence scores, filtering out anomalies through constraints to obtain the final dense reward, and forming an LLM-annotated reward dataset include: Read the decision steps from the structured decision log, calculate the importance score, and filter key decision steps using preset conditions; Construct LLM cue words for each of the key decision steps, and output candidate rewards and confidence levels with multiple temperatures and multiple samples by evaluating the LLM; Anomalies are filtered out by regulating LLM constraints to obtain the final dense reward and form an LLM-annotated reward dataset.

4. The UAV urban dynamic path planning method according to claim 1, characterized in that, After the steps of training the MLP reward model with the final dense reward as the regression target and weighted by the confidence of the final dense reward, the method further includes: The MLP reward model is weighted and fused with the original reward to form a unified reward, and the MLP reward model is trained in a closed loop. The MLP reward model and the original reward are compared after closed-loop training to obtain the comparison results.

5. A dynamic path planning system for urban unmanned aerial vehicles (UAVs), characterized in that, The system includes: The module is used to construct a unified 3D grid map, discretize city buildings onto the grid of the 3D grid map through parameters, and introduce dynamic obstacles onto the 3D grid map. A design module is established to build an unmanned aerial vehicle (UAV) system model, obtain the state observation vectors of the UAV on the three-dimensional grid map based on the UAV system model, and design the action space of the UAV. An output module is constructed to build the action probability distribution of the sub-target based on the motor neural planning network, the three-dimensional grid map, the state observation vector and the action space, and output the predicted trajectory and spatiotemporal features of the future multiple steps through the state variables of the UAV and dynamic obstacles at several moments, so as to obtain the spatiotemporal prediction features. The build output module includes: A search unit is generated to train the motor neural network offline, generating correctly reachable sub-objectives and guiding the MPNet within the motor neural network. The A* module performs path search to obtain the optimal sequence of collision-free path points; The computational encoding unit is used to calculate the first path point in the optimal path point sequence and the current position of the UAV to obtain the three-dimensional displacement, and encode the three-dimensional displacement as a unique action index; The construction unit is assigned a main probability based on the unique action index as the main action, and the remaining probabilities are equally distributed among other action indices to construct the action probability distribution of the sub-target. Building units are used to construct STGNP predictors using spatiotemporal graph neural networks; The first output unit is used to output the predicted trajectory and spatiotemporal features for the next multiple steps based on the STGNP predictor and the state quantities of the UAV and dynamic obstacles at several moments, and to generate spatiotemporal prediction features based on the predicted trajectory and the spatiotemporal features. The calculation output module is used to calculate the fusion coefficient of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector, output the final action, and execute it. The recording and calculation module is used to record the state change information of each step in the execution process, calculate the reward of the next K steps and the future collision flag, and write the state change information, the reward and the future collision flag into the structured decision log. The read construction module is used to read the decision steps of the structured decision log and select key decision steps, construct the LLM cue words for each key decision step, and output candidate rewards and confidence scores. Anomalies are filtered through constraints to obtain the final dense rewards and form an LLM labeled reward dataset. A training module is constructed to build a unified feature vector for each key decision step in the reward dataset based on the state and action information used in online decision-making, and to train an MLP reward model with the final dense reward as the regression target and weighted according to the confidence of the final dense reward.

6. The UAV urban dynamic path planning system according to claim 5, characterized in that, The calculation output module includes: The second output unit is used to output the Q-value distribution vector based on the action probability distribution; The calculation unit is used to calculate the fusion coefficients of the action probability distribution, the spatiotemporal prediction features, and the Q-value distribution vector of the decision-maker based on the hierarchical attention fusion module and attention gating weights.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the UAV urban dynamic path planning method as described in any one of claims 1 to 4.

8. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the UAV urban dynamic path planning method as described in any one of claims 1 to 4.

Images