Low-altitude unmanned aerial vehicle three-dimensional path planning method

Abstract

This invention discloses a three-dimensional path planning method for low-altitude unmanned aerial vehicles (UAVs) based on a deep Q-network guided by look-ahead planning. The method includes: constructing a three-dimensional state feature layer representing environmental obstacles, target points, and the UAV's position; calculating the value function of each action using a deep Q-network in parallel, and simulating future multi-step trajectories and generating trajectory evaluation functions through a look-ahead planning module; normalizing and weighting the two to form a comprehensive decision to select the optimal action; executing the action, updating network parameters, and repeating this process until the target is reached. This invention introduces a look-ahead planner based on explicit physical rules, providing stable guidance to the deep Q-network in the early stages of training, effectively suppressing training oscillations, and significantly improving the learning efficiency, convergence stability, and model interpretability of path planning.

Description

Technical Field

[0001] This invention relates to unmanned aerial vehicle (UAV) path planning technology, and in particular to a three-dimensional path planning method for low-altitude UAVs. Background Technology

[0002] With the widespread use of drones in low-altitude applications such as logistics, surveying, and inspection, path planning technology in complex urban environments has become a core key to their autonomous flight. Traditional path planning algorithms, such as Dijkstra's algorithm, A* algorithm, mixed integer programming, and Rapid Exploratory Random Tree* algorithm, aim to find the optimal or near-optimal path based on a search mechanism, given a known global or local obstacle map, drone position, and target address. However, they do not explicitly establish the data-driven relationships between feasible paths in the solution space. Therefore, the application of traditional planning algorithms in large-scale, uncertain environments still has certain limitations.

[0003] In recent years, deep reinforcement learning methods, through trial and error in interaction between agents and the environment, have been able to learn an end-to-end mapping from environmental states to action policies. By establishing connections between data through neural networks, they exhibit adaptive capabilities in large-scale, uncertain environments. Deep Q-Networks (DQNs) and their variants are typical methods for UAV path planning. However, deep Q-Networks are model-free methods, typically requiring a large number of interaction samples for training. In complex path planning problems, the training process is prone to oscillations, making it difficult to stably converge to the optimal policy. Furthermore, the decision logic of model-free methods is difficult to interpret, and the policies learned on specific training sets have poor generalization ability.

[0004] To address these challenges, some novel methods combining traditional planning with deep learning have emerged in existing technologies. For example, Value Iteration Networks (VINs) attempt to embed differentiable planning modules into neural networks to enhance the model's planning capabilities. VINs employ implicit planning modules for end-to-end learning, with a direct application scenario of two-dimensional path planning. In contrast, this invention uses explicit lookahead planning to guide the training of deep Q-networks, with a direct application scenario of three-dimensional path planning.

[0005] The learning mechanism of deep Q-networks relies on the direct interaction between the agent and the environment. Its model-free nature requires extensive trial-and-error data collection, often resulting in oscillations and slow convergence in training trajectories within high-dimensional state-action spaces such as 3D path planning. Furthermore, due to the lack of explicit physical logic, policies trained in specific environments have limited generalization ability.

[0006] Value iterative networks enhance the model's learning ability by embedding planning modules, but their internal planning computation components, such as reward mapping and state transition, are highly sensitive to the network structure and require end-to-end learning from scratch, thus not fully leveraging the guiding effect of planning on reinforcement learning models. In addition, the planning strategies they learn lack clear physical meaning, their interpretability has not been fundamentally improved, and their decision-making logic is relatively obscure.

[0007] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a three-dimensional path planning method for low-altitude unmanned aerial vehicles (UAVs).

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A three-dimensional path planning method for low-altitude unmanned aerial vehicles (UAVs) includes the following steps:

[0011] S1. Obtain raw observation data of the UAV flight environment and construct a three-dimensional state feature layer containing obstacle distribution, target point location and UAV's own position information;

[0012] S2. Based on the three-dimensional state feature layer at the current moment, use the deep Q-network module to calculate the first action value function of each action that can be performed in the current state;

[0013] S3. Based on the original observation data at the current moment, use the forward planning module to simulate the future multi-step trajectory caused by each action in the current state, and evaluate each future trajectory based on the predetermined evaluation criteria to generate the trajectory evaluation function for each action.

[0014] S4. After normalizing the first action value function and the trajectory evaluation function respectively, perform weighted fusion to obtain the comprehensive decision function of each actionable action, and select the action to be executed at the current moment based on the comprehensive decision function;

[0015] S5. Execute the selected action to transfer the drone to the next state and obtain an immediate reward. Update the parameters of the deep Q network based on the immediate reward and the next state, and repeat steps S1 to S5 until the drone reaches the target point.

[0016] A computer program product includes a computer program that, when executed by a processor, implements the low-altitude unmanned aerial vehicle (UAV) three-dimensional path planning method.

[0017] The present invention has the following beneficial effects:

[0018] This invention addresses the path planning problem faced by unmanned aerial vehicles (UAVs) when performing autonomous flight missions in urban low-altitude three-dimensional spaces containing obstacles. It proposes a three-dimensional path planning method for low-altitude UAVs based on a deep Q-network guided by look-ahead planning. This method achieves real-time, safe, and efficient trajectory planning from a given starting point to a target point. By dynamically responding to environmental information (including real-time obstacle maps, the current position of the UAV, and the position of the target point), the method dynamically adjusts the flight path, thereby simultaneously guiding the UAV toward the target point, effectively avoiding obstacles, shortening the overall flight distance, and ensuring flight stability and reliability.

[0019] This invention combines explicit planning with a clear physical meaning with a deep Q-network for 3D path planning of low-altitude unmanned aerial vehicles (UAVs). This fusion mechanism reduces ineffective trial and error during model training, enhancing training stability and convergence efficiency. Simultaneously, the explicit physical logic helps improve the generalization ability of the deep Q-network when facing heterogeneous scenarios. Specifically, this invention proposes a planning method that integrates a lookahead planner and a deep Q-network. By generating a forward-looking trajectory in the current state of the UAV and establishing a trajectory evaluation function, it guides the deep Q-network's action selection during training, thereby significantly improving the network's convergence speed and training stability.

[0020] In terms of specific design, this invention presents a forward-looking planner for low-altitude UAV 3D path planning tasks. This planner introduces a single-step evaluation function based on target incentives, obstacle penalties, and action stability at each action selection step. After completing the forward-looking planning, it quantifies and evaluates all possible actions in the current state based on the trajectory evaluation function. Furthermore, this invention designs a reward function suitable for low-altitude 3D path planning. This function includes a target incentive term to propel the UAV towards the target point, an obstacle penalty term to inhibit the UAV from approaching obstacles, and a single-step time penalty term to reduce overall flight time consumption. Regarding state representation, this invention proposes a method for constructing state features for low-altitude 3D path planning. By dividing the 3D space into horizontal layers, it constructs three types of state feature layers representing obstacle distribution, target position, and the UAV's current position, providing effective input for subsequent planning and decision-making.

[0021] Compared to traditional deep Q-network methods, this invention introduces explicit planning guidance based on physical meaning, providing effective and stable guidance signals in the early stages of training and suppressing the risk of oscillations during training. Compared to value iterative networks that also combine planning and learning, the planner of this invention does not rely on end-to-end learning from scratch to obtain planning functionality, and can efficiently guide deep Q-networks in the early stages of training. Furthermore, its decision-making mechanism is based on explicit physical rules for trajectory evaluation and guidance, with clear and transparent decision logic, exhibiting significant advantages in model interpretability.

[0022] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0023] Figure 1 The design framework diagram of the three-dimensional path planning method for low-altitude unmanned aerial vehicles of the present invention is shown.

[0024] Figure 2(a) is a schematic diagram of three-dimensional path planning based on the extraction of state features for low-altitude three-dimensional path planning according to an embodiment of the present invention.

[0025] Figure 2(b) is a schematic diagram of the obstacle state feature layer for low-altitude three-dimensional path planning state feature extraction according to an embodiment of the present invention.

[0026] Figure 2(c) is a schematic diagram of the target state feature layer for low-altitude three-dimensional path planning state feature extraction according to an embodiment of the present invention.

[0027] Figure 2(d) is a schematic diagram of the current position state feature layer of the UAV for low-altitude three-dimensional path planning state feature extraction according to an embodiment of the present invention.

[0028] Figure 3 A schematic diagram of low-altitude three-dimensional path planning action according to an embodiment of the present invention.

[0029] Figure 4 A schematic diagram of the look-ahead planner according to an embodiment of the present invention.

[0030] Figure 5 The deep Q-network graph guided by look-ahead planning in this embodiment of the invention.

[0031] Figure 6 A schematic diagram illustrating the implementation steps of a deep Q-network guided by look-ahead planning in an embodiment of the present invention.

[0032] Figure 7 This is a flowchart illustrating the overall process of the low-altitude UAV three-dimensional path planning method of the present invention. Detailed Implementation

[0033] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0034] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0035] This invention aims to address the problems of low training efficiency and poor policy interpretability in path planning for low-altitude UAVs in complex 3D environments. It proposes a planning method that integrates a look-ahead planner based on explicit physical rules with a deep Q-network. The planner provides stable physical guidance in the early stages of training, significantly improving the convergence speed and stability of deep reinforcement learning. At the same time, explicit decision logic enhances the interpretability and generalization ability of the model.

[0036] See Figures 1 to 7 This invention provides a three-dimensional path planning method for low-altitude unmanned aerial vehicles (UAVs), comprising the following steps:

[0037] Step S1: Obtain raw observation data of the UAV flight environment and construct a three-dimensional state feature layer containing obstacle distribution, target point location and UAV's own position information.

[0038] In some embodiments, step S1 specifically includes: discretizing the three-dimensional flight space into uniform cubic grid cells along three coordinate axes to form a state space; constructing an obstacle state feature layer based on the grid cells occupied by obstacles, wherein if a grid cell is fully or partially occupied by an obstacle, it is marked with a first value at the corresponding position in the feature layer, otherwise it is marked with a second value; constructing a target state feature layer based on the grid cells occupied by the target point, wherein only the unique grid cell position where the target point is located is marked with a first value, and the remaining positions are marked with a second value; constructing a drone position state feature layer based on the grid cells currently occupied by the drone, wherein only the unique grid cell position where the drone is currently located is marked with a first value, and the remaining positions are marked with a second value; dividing the discretized three-dimensional space into multiple horizontal layers along the height direction, and performing a two-dimensional projection of the obstacle state feature layer, target state feature layer, and drone position state feature layer in each horizontal layer;

[0039] The two-dimensional projection features corresponding to the multiple horizontal layers are stitched together to form the final three-dimensional state feature layer, which serves as the input to the depth Q-network module.

[0040] Step S2: Based on the three-dimensional state feature layer at the current moment, use the deep Q-network module to calculate the first action value function of each possible action in the current state.

[0041] In some embodiments, in step S2, the deep Q-network module is a combination of a convolutional neural network and a fully connected neural network. Its input is the three-dimensional state feature layer, and its output is the first action value function of each action that can be performed in the current state. The training of the deep Q-network module is based on a deep reinforcement learning framework that includes an experience replay mechanism and a target network mechanism. The network parameters are updated by minimizing the loss between the predicted value and the target value.

[0042] Step S3: Based on the original observation data at the current moment, use the look-ahead planning module to simulate the future multi-step trajectory caused by each action in the current state, and evaluate each future trajectory based on the predetermined evaluation criteria to generate the trajectory evaluation function for each action.

[0043] In some embodiments, the working mechanism of the forward planning module in step S3 specifically includes:

[0044] S31. Define a single-step evaluation function to assess the quality of the subsequent state reached after performing any action from any state; the single-step evaluation function is a weighted sum of at least three sub-terms, including: a target incentive term to encourage movement towards the target point, the value of which is related to the change in the Manhattan distance from the UAV to the target point before and after performing the action; and an obstacle approach penalty term to encourage moving away from obstacles, the value of which is related to the distance from the UAV's new position to the nearest obstacle in the environment. The sum of the squares of the reciprocals of the Euclidean distances to each obstacle; and a motion stability term used to encourage smooth flight, the value of which is the cosine of the angle between the current motion and the previous motion vector;

[0045] S32. For each action that can be performed in the current state, starting from the state reached after performing the action, perform a forward trajectory deduction of a fixed number of steps based on a random selection strategy derived from the single-step evaluation function; the random selection strategy is as follows: in each deduction state, normalize the single-step evaluation values ​​of all actions as the probability of selecting each action.

[0046] S33. For each generated forward trajectory, the single-step evaluation value corresponding to each action in the trajectory is accumulated according to a predetermined discount factor to obtain the trajectory evaluation value of the trajectory.

[0047] S34. For each action in the current state, repeat steps S32 and S33 to generate multiple independent forward trajectories, and calculate the arithmetic mean of the evaluation values ​​of these trajectories as the trajectory evaluation function corresponding to that action.

[0048] Step S4: Normalize the first action value function and the trajectory evaluation function respectively, and then perform weighted fusion to obtain a comprehensive decision function for each action, and select the action to be executed at the current time based on the comprehensive decision function.

[0049] In some embodiments, in step S4, the normalization of the first action value function and the trajectory evaluation function is specifically performed by: normalizing the set of first action value functions of all possible actions output by the deep Q network module and the set of trajectory evaluation functions of all possible actions output by the look-ahead planning module using the Softmax function to obtain the corresponding first normalized action value set and normalized trajectory evaluation set.

[0050] The weighted fusion specifically involves: weighting and summing each value in the first normalized action value set with the corresponding value in the normalized trajectory evaluation set according to a preset fusion weight to obtain the comprehensive decision function value of each action.

[0051] The action selection based on the comprehensive decision function is specifically as follows: the set of comprehensive decision function values ​​is normalized again using the Softmax function, the normalization result is used as the selection probability of each action in the current state, and the action to be executed is selected according to the probability distribution.

[0052] Step S5: Execute the selected action to transfer the drone to the next state and obtain an immediate reward. Update the parameters of the deep Q network based on the immediate reward and the next state, and repeat steps S1 to S5 until the drone reaches the target point.

[0053] In some embodiments, the calculation of the instant reward includes: if the drone reaches the target point, a positive high reward is given; if the drone moves to the grid cell where the obstacle is located, a negative high penalty is given; if neither of the above two situations occurs, the instant reward consists of three parts: the first part is a reward related to approaching the target, the value of which is a negative coefficient multiplied by the change in Manhattan distance from the drone to the target point before and after the action is performed; the second part is a penalty related to obstacle avoidance, the value of which is a negative coefficient multiplied by the sum of the squares of the Euclidean distances from the new position of the drone to all obstacles; the third part is a fixed negative value as a single-step time penalty.

[0054] In some embodiments, when updating the parameters of the deep Q-network in step S5, a transition tuple containing the current state, the executed action, the immediate reward obtained, and the next state is stored in an experience replay buffer. A batch of transition tuples is randomly sampled from the buffer. Using a target network with the same structure but slower parameter updates, the optimal action value estimate for the next state is calculated. A target Q-value is constructed by combining the immediate reward. The parameters of the online deep Q-network are updated using gradient descent by minimizing the mean squared error loss between the Q-value predicted by the current online deep Q-network and the target Q-value. The parameters of the target network are periodically copied from the online deep Q-network.

[0055] In some embodiments, the set of possible actions of a UAV is defined as: from its current grid cell, it can move to all grid cells within a cubic neighborhood of its surrounding cells, each represented by a displacement of -1, 0, or 1 in each of the three coordinate axes. For example, the set of possible actions of a UAV is defined as: from its current grid cell, it can move to all grid cells within a 3×3×3 cubic neighborhood centered on it; each action is represented by a triplet of displacements in the three coordinate axes, where each displacement is -1, 0, or 1.

[0056] In some embodiments, the low-altitude UAV three-dimensional path planning method is performed in rounds. After the UAV reaches the target point or meets the termination condition, the UAV is reset to the initial state and a new round of path planning and learning process begins.

[0057] This invention proposes a 3D path planning method for low-altitude unmanned aerial vehicles (UAVs). It deeply integrates a lookahead planner built with explicit physical meaning with a deep Q-network, effectively solving the key bottlenecks of traditional path planning methods and existing deep learning solutions in 3D path planning tasks for low-altitude UAVs. The explicit lookahead planning module introduced in this invention provides stable and efficient physical guidance to the deep Q-network from the early stages of training, significantly reducing the dependence of traditional deep Q-networks as model-free methods on large amounts of trial-and-error data. This effectively suppresses oscillations during training and greatly improves network convergence speed and training stability. Compared to traditional pure search algorithms or end-to-end deep reinforcement learning methods, this invention not only achieves safe and efficient path planning through multi-dimensional evaluation functions designed with target incentives, obstacle penalties, and action stability, but also possesses good model interpretability due to its explicit physical logic, enhancing policy generalization capabilities in heterogeneous scenarios. Furthermore, compared with existing technologies that combine planning and learning (such as value iterative networks), the planner of this invention does not rely on implicit embedding learning that is sensitive to network structure, nor does it need to learn planning functions end-to-end from scratch. Instead, it can directly and efficiently guide the deep Q-network based on physical rules from the beginning of training, thereby improving training efficiency while ensuring the clarity and transparency of decision logic.

[0058] The following further describes specific embodiments of the present invention and examples of its algorithm implementation.

[0059] A method for 3D path planning of low-altitude unmanned aerial vehicles (UAVs) is proposed, which designs a deep Q-network model guided by look-ahead planning for 3D path planning of low-altitude UAVs. The method mainly includes five steps: 1. Extraction of state features; 2. Definition of action variables; 3. Design of reward function; 4. Design of look-ahead planner; 5. Design of deep Q-network embedding look-ahead planner.

[0060] like Figure 1 As shown, the specific implementation process of the method is as follows:

[0061] (1) Extraction of state features

[0062] Figures 2(a) to 2(d) illustrate the state feature extraction process for low-altitude 3D path planning, demonstrating the process of constructing a state feature matrix based on the original 3D space. First, the original 3D space is divided into cubic grids of equal volume. If a grid is entirely or partially occupied by obstacles, this grid is called an obstacle grid. Let the set of obstacle grids be... Let any obstacle mesh within it And available express, The obstacle grid is in axis, axis, The position number on the axis. Then, assign the current grid position occupied by the drone. The grid position occupied by the target point .

[0063] At any time The complete state of the system It is a triple consisting of the drone's position, the target's position, and the obstacle's position:

[0064] (1)

[0065] The set of all possible states, i.e., the state space, is defined as:

[0066] (2)

[0067] in, It is the set of all meshes in the original three-dimensional space. This represents free space, which is the set of all unobstructed grids. This means that the drone and the target point do not overlap with obstacles.

[0068] As shown in Figure 2(a), the original three-dimensional space along The axis is discretized into eight horizontal layers. Figures 2(b), 2(c), and 2(d) show these eight horizontal layers along... The three state feature layers obtained by projection along the negative axis correspond to the obstacle, target, and current position of the UAV, respectively. Each feature layer uses a binary encoding rule, mathematically represented as follows:

[0069] 1) In the obstacle state feature layer, for any specific layer , ,

[0070] (3)

[0071] in This represents a set of obstacles.

[0072] 2) In the target state feature layer, for a specific layer , ,

[0073] (4)

[0074] 3) In the current position and state feature layer of the UAV, for a specific layer , ,

[0075] (5)

[0076] Based on the above rules, we can obtain , , This ultimately forms a three-dimensional feature layer:

[0077] (6)

[0078] (2) Definition of action variables

[0079] Figure 3 Demonstrates low-altitude 3D path planning; the drone's position is... Construct a The cube-shaped domain contains 27 grid cells, all of which are reachable positions for the next step. The set of actions can be represented as: (7)

[0080] Each action vector This represents the displacement components on the three coordinate axes.

[0081] When the drone performs actions Then, the state evolves as follows:

[0082] (8)

[0083] in, , It is the state transition function. Target point and obstacle set It is time-invariant under the static environment assumption.

[0084] (3) Design of reward function

[0085] Let the current location of the drone be... , After the action It is in a subsequent state. Target point location .

[0086] First, path planning encourages drones to move towards the target point, therefore, rewards are set based on goal approach. (9)

[0087] in, And it is a constant. This indicates the calculation of the Manhattan distance between two points. and The definition is as follows:

[0088] (10)

[0089] (11)

[0090] Then, let the obstacle mesh set be... Let any obstacle mesh within it And satisfy To encourage drones to stay away from obstacles, define obstacle avoidance-related rewards.

[0091] (12)

[0092] in, And it is a constant. And it is a constant. This indicates the calculation of the Euclidean distance between two points. The definition is as follows: (13)

[0093] Finally, a single-step penalty is added to each step before the drone reaches the finish line to prevent the drone from stopping.

[0094] (14)

[0095] in, And it is a constant.

[0096] The total reward for a single step of the drone, calculated by combining the goal-approaching reward (9), obstacle-approaching penalty (12), and single-step penalty (14), is:

[0097] (15)

[0098] Wherein, according to formula (1) , .

[0099] (4) Design of the look-ahead planner

[0100] like Figure 4 As shown, the forward-looking planner is used to perform forward-looking searches for the future paths of the drone and calculate the corresponding evaluation metrics for the forward-looking trajectory.

[0101] First, this invention defines a single-step evaluation function to quickly generate the look-ahead trajectory. The mathematical form of the single-step evaluation function is as follows:

[0102] (16)

[0103] in, For any position of the drone in the planner, For any of its subsequent positions, In a state The actions taken In a state The last action performed previously, viewed from a time series perspective, for The previous action.

[0104] In the single-step evaluation function, the target incentive term is:

[0105] (17)

[0106] in, And it is a constant. and The definitions are shown in formulas (10) and (11), respectively. This represents the Manhattan distance from the current state to the destination. This represents the Manhattan distance from the subsequent state to the endpoint.

[0107] Obstacle approach penalty items are (18)

[0108] in, And it is a constant. And it is a constant. The definition is shown in formula (13). This represents the Euclidean distance from the subsequent state to the endpoint. obstacle mesh set Mid-range position Recent One grid, It is a positive integer. , Represents a set The number of elements in the middle.

[0109] The motion smoothness item is

[0110] (19)

[0111] in, It is the angle between the forward and backward movements. and Each represents an action and The Euclidean norm (L2 norm) of the Euclidean system. Its mathematical form is:

[0112] (20)

[0113] (twenty one)

[0114] In summary, the specific form of the single-step evaluation function is: (twenty two)

[0115] Wherein, according to formula (1) , .

[0116] When the drone reaches a certain state At that time, assuming there are currently Optional actions , , ..., Each leads to a subsequent state. , , ... , For each subsequent state , Calculate the single-step evaluation function for all possible actions. , .Will After normalization using the softmax function, the generation strategy is... ,in For state Next action The probability of that, and its relation The mathematical relationship is as follows:

[0117] (twenty three)

[0118] If in state The following actions have already been performed. And transferred to The single-step evaluation that has already occurred is Similarly, marking Transferred to The subsequent single-step evaluation is The sequence has reached a state. Stop, generate , , ...., common A single-step evaluation. Because It is by Execute action The trajectory evaluation function is then defined: (twenty four)

[0119] in, , where is the discount factor.

[0120] because The subsequent trajectory is generated by a random strategy (23). To avoid the bias caused by a single trajectory generation, in The random strategy (23) will then be used multiple times to generate trajectories, and the average value of the trajectory evaluation function will be calculated: (25)

[0121] in Assign a trajectory number, , The number of trajectories generated.

[0122] Ultimately, the planner will obtain the state. Evaluation function for all possible actions , , ..., .

[0123] (5) Deep Q-network design guided by forward planning

[0124] A forward-looking planning-guided deep Q-network structure, such as Figure 5 As shown, it contains three modules: a deep Q-network module, a look-ahead planning module, and a fusion module. In the deep Q-network module, the input is the current time step. status The feature layer is constructed as shown in formulas (3), (4), and (5). The state features are further extracted through a convolutional neural network, and finally output by a fully connected neural network. Value functions for all possible actions , When the environmental state changes from Through action Transfer to And receive instant rewards. Construct transition tuples The deep Q-network is updated based on the loss function.

[0125] (26)

[0126] in For the weights of a deep Q-network, Here are the target network weights for the deep Q-network. The target network has the same structure as the deep Q-network, except that the deep Q-network updates its weights after each state transition, while the target network updates its weights every [period]. Update once per step.

[0127] In the forward planning module, the input is the current time. status and output Trajectory evaluation function for all possible actions , In the fusion module, the input is the current state. All corresponding motion value functions and trajectory evaluation functions, due to differences in their dimensions and scales, were normalized using the Softmax function to generate... and ,in Its mathematical form is

[0128] (27)

[0129] (28)

[0130] Construct the final decision function

[0131] (29)

[0132] Will , Normalization is performed using the Softmax function to obtain... Generating actions The probability is

[0133] (30)

[0134] exist Based on probability Generate actions .

[0135] The computational process of deep Q-networks guided by forward-looking planning is as follows: Figure 6 As shown, the calculation steps are as follows:

[0136] Initialization: Environment at its initial moment ,state

[0137] Deep Q network weights Target network weights ;

[0138] Experience storage pool ;

[0139] Step 1: The environment state module provides raw observation data to the state feature construction module. Based on this, a three-dimensional state feature layer is constructed. ;

[0140] Step 2: Input a deep Q-network, Input the lookahead planner;

[0141] Step 3: The lookahead planner is based on the current state. Generate all possible action prospective trajectories and calculate trajectory evaluation functions. , And obtained by normalization using the Softmax function. ;

[0142] Step 4: Deep Q-network based ,calculate Values ​​of all available actions in the state , And obtained by normalization using the Softmax function. ;

[0143] Step 5: Through The output signals of the deep Q-network and the lookahead planner are combined and normalized using the Softmax function to obtain the policy (action generation probability). , Subsequently, using Generate actions ;

[0144] Step 6: Execution of the action execution module , transition to a new state and generate instant rewards. The transferred tuples are stored in the experience replay buffer. A batch of tuples is randomly sampled from the experience replay buffer, and the mean squared error loss between the Q-network prediction and the target value is calculated based on formula (26). The online deep Q-network is then updated using the Adam method. Update the target network every step;

[0145] Step 7: Let Repeat steps 1-6 until the drone reaches the destination, then proceed to step 8;

[0146] Step 8: Let 0, state Repeat steps 1-7.

[0147] In summary, this invention proposes a three-dimensional path planning method for low-altitude unmanned aerial vehicles (UAVs) based on a deep Q-network guided by look-ahead planning. The key innovative contributions and prominent features of this invention include: First, it proposes a path planning framework that integrates a look-ahead planner with a deep Q-network, generating a forward-looking trajectory in the current state of the UAV and establishing a trajectory evaluation function to guide the action selection of the deep Q-network; Second, for the three-dimensional path planning task, it designs a single-step evaluation function based on multi-dimensional considerations such as target incentives, obstacle penalties, and action stability, and uses this function as the core to construct a look-ahead planner capable of quantifying and evaluating each action; Third, it designs a composite reward function that includes target approach, obstacle avoidance penalties, and single-step time consumption; Fourth, it proposes a state representation method that hierarchically divides the three-dimensional space and constructs three feature layers: obstacles, targets, and UAV positions.

[0148] Compared with existing technologies, the significant technical advantages of this invention are reflected in the following aspects: First, by introducing a look-ahead planner based on explicit physical rules, stable and effective guidance can be provided in the early stages of training, significantly suppressing the oscillation phenomenon commonly seen in deep Q-networks during training in complex 3D environments, and improving convergence speed and training stability. Second, compared with value iterative networks that also combine planning and learning, the planner of this invention does not rely on end-to-end learning from scratch to obtain planning capabilities, is insensitive to network structure, and can provide efficient guidance from the start of training; at the same time, its decision logic based on explicit physical rules has better interpretability and helps to improve the model's generalization ability in heterogeneous scenarios.

[0149] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0150] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0151] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0152] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc or CD-ROM; magnetic surface memory can be disk storage or magnetic tape storage. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0153] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0154] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0155] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0156] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0158] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0159] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0160] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0161] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A three-dimensional path planning method for low-altitude unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: S1. Obtain raw observation data of the UAV flight environment and construct a three-dimensional state feature layer containing obstacle distribution, target point location and UAV's own position information; S2. Based on the three-dimensional state feature layer at the current moment, use the deep Q-network module to calculate the first action value function of each action that can be performed in the current state; S3. Based on the original observation data at the current moment, use the forward planning module to simulate the future multi-step trajectory caused by each action in the current state, and evaluate each future trajectory based on the predetermined evaluation criteria to generate the trajectory evaluation function for each action. The working mechanism of the forward planning module specifically includes: defining a single-step evaluation function to evaluate the quality of the subsequent state reached after performing any action from any state; for each actionable action in the current state, taking the state reached after performing the action as the starting point, performing a fixed number of forward trajectory deductions based on a random selection strategy derived from the single-step evaluation function; the random selection strategy is as follows: in each deduction state, the single-step evaluation values ​​of all actionable actions are normalized as the probability of selecting each action; for each generated forward trajectory, the single-step evaluation values ​​corresponding to each action performed in the trajectory are accumulated according to a predetermined discount factor to obtain the trajectory evaluation value of the trajectory; for each actionable action in the current state, the above process is repeated to generate multiple independent forward trajectories, and the trajectory evaluation function is calculated; S4. After normalizing the first action value function and the trajectory evaluation function respectively, perform weighted fusion to obtain the comprehensive decision function of each actionable action, and select the action to be executed at the current time based on the comprehensive decision function; S5. Execute the selected action to transfer the drone to the next state and obtain an immediate reward. Update the parameters of the deep Q network based on the immediate reward and the next state, and repeat steps S1 to S5 until the drone reaches the target point.

2. The method as described in claim 1, characterized in that, Step S1 specifically includes: The three-dimensional flight space is discretized into uniform cubic grid cells along the three coordinate axes to form the state space; Based on the grid cells occupied by obstacles, an obstacle state feature layer is constructed. If a grid cell is fully or partially occupied by an obstacle, it is marked as the first value at the corresponding position in the feature layer; otherwise, it is marked as the second value. Based on the grid cells occupied by the target point, a target state feature layer is constructed, wherein only the unique grid cell where the target point is located is marked with the first value, and the other locations are marked with the second value; Based on the grid cell currently occupied by the drone, a drone position status feature layer is constructed, in which only the unique grid cell where the drone is currently located is marked with the first value, and the other locations are marked with the second value; The discretized three-dimensional space is divided into multiple horizontal layers along the height direction, and the obstacle state feature layer, target state feature layer and UAV position state feature layer in each horizontal layer are projected in two dimensions. The two-dimensional projection features corresponding to the multiple horizontal layers are stitched together to form the final three-dimensional state feature layer, which serves as the input to the depth Q-network module.

3. The method as described in claim 1 or 2, characterized in that, In step S2, the deep Q-network module is a combination of a convolutional neural network and a fully connected neural network. Its input is the three-dimensional state feature layer, and its output is the first action value function of each action that can be performed in the current state. The training of the deep Q-network module is based on a deep reinforcement learning framework that includes an experience replay mechanism and a target network mechanism. The network parameters are updated by minimizing the loss between the predicted value and the target value.

4. The method according to any one of claims 1 to 2, characterized in that, The working mechanism of the forward planning module described in step S3 also specifically includes: The single-step evaluation function is a weighted sum of at least three sub-items, including: a target incentive term to encourage movement towards the target point, the value of which is related to the change in the Manhattan distance from the UAV to the target point before and after the action; and an obstacle approach penalty term to encourage movement away from obstacles, the value of which is related to the distance from the UAV's new position to the nearest obstacle in the environment. The sum of the squares of the reciprocals of the Euclidean distances to each obstacle; and a motion stability term used to encourage smooth flight, the value of which is the cosine of the angle between the current motion and the previous motion vector; and Calculate the arithmetic mean of the trajectory evaluation values, and use it as the trajectory evaluation function corresponding to the action.

5. The method according to any one of claims 1 to 2, characterized in that, In step S4, the normalization of the first action value function and the trajectory evaluation function specifically includes: normalizing the set of first action value functions of all possible actions output by the deep Q network module and the set of trajectory evaluation functions of all possible actions output by the look-ahead planning module using the Softmax function to obtain the corresponding first normalized action value set and normalized trajectory evaluation set. The weighted fusion specifically includes: weighting and summing each value in the first normalized action value set with the corresponding value in the normalized trajectory evaluation set according to a preset fusion weight to obtain the comprehensive decision function value of each action. The action selection based on the comprehensive decision function specifically includes: normalizing the set of comprehensive decision function values ​​again using the Softmax function, using the normalization result as the selection probability of each action in the current state, and selecting the action to be executed based on the probability distribution.

6. The method according to any one of claims 1 to 2, characterized in that, In step S5, the calculation of the instant reward includes: If the drone reaches the target point, a large positive reward will be given; If the drone moves into the grid cell containing the obstacle, a high negative penalty will be imposed. If neither of the above two situations occurs, the immediate reward consists of three parts: the first part is a reward related to approaching the target, which is a negative coefficient multiplied by the change in Manhattan distance from the drone to the target point before and after the action is executed; the second part is a penalty related to obstacle avoidance, which is a negative coefficient multiplied by the sum of the squares of the Euclidean distances from the drone's new position to all obstacles; the third part is a fixed negative value as a single-step time penalty.

7. The method according to any one of claims 1 to 2, characterized in that, In step S5, when updating the parameters of the deep Q-network, a transition tuple containing the current state, the executed action, the immediate reward, and the next state is stored in an experience replay buffer. A batch of transition tuples is randomly sampled from the buffer. Using a target network with the same structure but slower parameter updates, the optimal action value estimate for the next state is calculated. A target Q-value is constructed by combining the immediate reward. The parameters of the online deep Q-network are updated using gradient descent by minimizing the mean squared error loss between the Q-value predicted by the current online deep Q-network and the target Q-value. The parameters of the target network are periodically copied from the online deep Q-network.

8. The method according to any one of claims 1 to 2, characterized in that, The set of possible actions for a drone is defined as: from its current grid cell, it can move to all adjacent grid cells within a cube neighborhood consisting of displacements of -1, 0, and 1 in each of the three coordinate axes.

9. The method according to any one of claims 1 to 2, characterized in that, The method is conducted in rounds. After the drone reaches the target point or meets the termination conditions, the drone is reset to the initial state and a new round of path planning and learning begins.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional path planning method for low-altitude unmanned aerial vehicles as described in any one of claims 1-9.

Images