Unmanned aerial vehicle path planning method and device based on reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle path planning method and device based on reinforcement learning, and relates to the technical field of unmanned aerial vehicles, and the method comprises the steps: setting the size of a three-dimensional flight environment of an unmanned aerial vehicle formation; acquiring a starting point position and a target point position of a long aircraft set in the unmanned aerial vehicle formation, and determining the number and the positions of obstacles at the same time; determining a simulated three-dimensional flight environment of the unmanned aerial vehicle according to the number and the positions of the obstacles, and discretizing the simulated three-dimensional flight environment of the unmanned aerial vehicle to establish a rasterized grid map; establishing an unmanned aerial vehicle state, a state space and a discretized unmanned aerial vehicle action space in a three-dimensional discretized space according to an unmanned aerial vehicle simulation three-dimensional flight environment; and based on the starting point position and the target point position of the long aircraft, an initial path of the long aircraft is obtained by adopting an IDDQN algorithm, and the initial path is optimized by adopting a Bezier curve to obtain final path information. The problem that in the prior art, the unmanned aerial vehicle path planning mode is low in efficiency is solved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a UAV path planning method and apparatus based on reinforcement learning. Background Technology

[0002] Low-altitude unmanned aerial vehicles (UAVs), as a type of unmanned aerial vehicle capable of highly maneuverable flight, high adaptability, precise navigation and guidance, and the ability to perform various types of missions, are characterized by low cost, low maintenance cost, high mobility, and convenient deployment, and have become one of the core carriers of low-altitude economic development. They have been widely used in typical low-altitude economic application scenarios such as low-altitude logistics delivery, urban air traffic, infrastructure inspection, precision agriculture plant protection, and emergency rescue. In urban areas, path planning remains one of the core challenges restricting the practical application of UAVs. Therefore, the obstacle avoidance problem in three-dimensional complex obstacle environments places further demands on the quality, real-time performance, and robustness of the global path algorithm.

[0003] Currently, commonly used UAV path planning algorithms include fast random search trees, dynamic window methods, artificial potential field methods, ant colony optimization, and particle swarm optimization. Faced with increasingly complex flight environments and high-dimensional spaces, traditional and intelligent algorithms suffer from potential problems such as getting trapped in local optima, long planning times, and low path smoothness. Some researchers have applied intelligent path planning algorithms, such as genetic algorithms, ant colony optimization, particle swarm optimization, and neural networks, to address the shortcomings of traditional methods. However, with the increasing complexity and uncontrollable factors in UAV flight environments at low altitudes, higher demands are placed on the active planning performance of UAVs. Traditional and intelligent UAV path planning algorithms are difficult to directly apply to unknown environments lacking prior experience, necessitating research into more autonomous UAV path planning methods that can adapt to conditions with minimal prior knowledge.

[0004] Reinforcement learning provides an effective framework for solving the aforementioned policy problems. It primarily involves interaction between the agent and the environment. The agent performs actions in the environment, and the environment provides feedback as a reward for those actions. The agent then adjusts its policy based on the reward value to maximize cumulative returns. By repeatedly interacting with the environment, the agent learns the optimal policy, making it an ideal candidate for solving complex low-altitude path planning problems. Through reinforcement learning, the agent can acquire the optimal policy with minimal prior knowledge.

[0005] For the path planning problem of unmanned aerial vehicles (UAVs), Q-Learning was one of the earliest reinforcement learning methods applied. However, as the complexity or dimensionality of the environment increases, the amount of data stored in Q-Learning increases exponentially, potentially leading to dimensionality explosion and making the path planning problem even more complex. To address the large state space issue, neural networks have been used to fit the Q-table in Q-Learning, and a Deep Q Network (DQN) method combining deep learning and reinforcement learning has been proposed. Compared to Q-Learning, DQN converges faster and effectively avoids the dimensionality explosion problem. However, as the density of environmental obstacles increases or the reward signal becomes sparse, the convergence speed of DQN slows down. Therefore, current UAV path planning methods suffer from low efficiency. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a method and apparatus for UAV path planning based on reinforcement learning, which aims to solve the problem of low efficiency in existing UAV path planning methods.

[0007] This invention proposes a path planning method for unmanned aerial vehicles (UAVs) based on reinforcement learning, the method comprising: Set the size of the three-dimensional flight environment for the drone formation; Obtain the starting position and target position of the lead drone in the drone formation, and simultaneously determine the number and location of obstacles; The three-dimensional flight environment of the UAV is determined based on the number and location of obstacles, and the three-dimensional flight environment of the UAV simulation is discretized to create a rasterized grid map. Based on the simulated 3D flight environment of the UAV, the UAV state, state space, and discretized UAV action space in the 3D discretized space are established. An action value function is established based on the mapping relationship between the state and action space. The action value function is used to calculate the time difference error of the UAV's empirical samples and to train and update the network parameters of the IDDQN algorithm. Determine the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space, as well as the reward function during training; The initial path of the lead aircraft is obtained using the IDDQN algorithm based on the starting position and target position of the lead aircraft, and the final path information is obtained by optimizing the initial path using Bézier curves.

[0008] Furthermore, in the aforementioned reinforcement learning-based UAV path planning method, the UAV state in a three-dimensional discretized space is established based on the simulated three-dimensional flight environment of the UAV. State space and the discrete action space of unmanned aerial vehicles The steps include: Pick This indicates the current location information of the lead aircraft. Indicates the target location of the lead aircraft. Obstacle information is represented, and the drone's status is represented as follows: The state space of the UAV is represented as ; With the drone as the center point, a regular hexahedron is constructed with an edge length of 2 units. The drone's action space is discretized into 26 directions that the drone can choose from, and the drone's action space is defined as follows: .

[0009] Furthermore, in the aforementioned reinforcement learning-based UAV path planning method, the steps of determining the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space and the reward function during training include: use The strategy determines the actions the primary machine chooses during learning, specifically:

[0010] in, Given the drone state, the parameters for the greedy strategy are... The lead aircraft Randomly select an action with a probability of [missing information]. The probability of choosing the action with the highest value; Establish the immediate reward function for IDDQN, which includes an action reward function and an environment reward function. The environment reward function includes a target point reward, an environment range reward, an obstacle reward, and a superstep reward. The action reward function and the final immediate reward function are as follows:

[0011]

[0012] in, This represents the Eulerian distance from the current state to the target point. This represents the Eulerian distance from the next state to the target point; The distance parameter is used to balance the distance reward. This represents the step reward value. The target location for the lead aircraft. For the lead aircraft in the current state of the drone Location, Next status for the lead aircraft Location, For the environmental reward function, Indicates the current number of steps being executed. Indicates environmental constraints. Indicates obstacle constraints. This indicates a range constraint for the number of steps.

[0013] Furthermore, in the aforementioned reinforcement learning-based UAV path planning method, the step of obtaining the initial path of the lead aircraft using the IDDQN algorithm based on the lead aircraft's starting position and target position includes: Define the empirical sample of the drone as The initialization sample priority and cumulative gradient are as follows: and At the same time, initialize the main network parameters and the target network parameters as follows: ; A Markov process with a given step size is performed to obtain a series of experience samples, which are stored in an initial experience pool. In the later stages of training the IDDQN algorithm, the experience sequences generated during training are combined into a temporary experience pool. ; in: ; in, Representing state After the action to state Markov processes, performing actions After Indicate whether this training session has ended; Define a range of parameters for a greedy strategy. ; In the temporary experience pool, a preset genetic algorithm is used to sequentially extract the experience sequences in the temporary experience pool as the parent, and at the same time extract the experience sequences different from the parent as the mother, decouple the action sets of the parent and the mother, decode the action sets and encode them in binary, and then perform crossover and mutation to obtain the action sets of the offspring. in, Binary encoding representing the action space; The obtained child action set is dynamically optimized into the action set of the target point, and the corresponding empirical sequence sample is obtained based on the continuous Markov process. For the obtained experience sequence sample set, after filtering according to fitness, the filtered target point experience sequences are updated to obtain a temporary experience pool with one optimization. Input the experience samples from the temporary experience pool into the initial experience pool; During the process of storing experience into the initial experience pool, experience samples are sampled from the initial experience pool in a given batch according to priority probability for network training. The probability of each sample being sampled in the experience pool can be expressed as:

[0014] in, Prioritize samples; Calculate the importance sampling weights of the sampled empirical samples to define the importance of the sample, which is used for subsequent gradient updates. The importance weights of the empirical samples are:

[0015] in, This represents the number of existing experience samples in the initial experience pool; The temporal difference error of the sampled empirical sample is calculated. An auxiliary importance weight is used to update the cumulative gradient and simultaneously to update the priority of the sample. The temporal difference error of the empirical sample is expressed as:

[0016] The priority of the empirical sample is updated using the temporal difference error of the empirical sample:

[0017] The cumulative gradient is updated by combining the importance weights of the samples and the temporal difference error:

[0018] Using formula , Update the main network parameters of DDQN to the learning rate; At regular time intervals, a soft update method is used to update the main network parameter values ​​to the target network:

[0019] in, For soft update parameters; The trained main network parameters As the training brain of the lead aircraft, it obtains discretized virtual path information through a Markov process. .

[0020] Furthermore, in the aforementioned reinforcement learning-based UAV path planning method, after the step of updating the temporary experience pool by filtering target points according to fitness to obtain an optimized temporary experience pool, the method further includes: Calculate the cumulative reward of an experience sequence in the experience pool once. ; according to Priority calculation method and Obtain the sampling priority of all empirical sequences ; according to Sampling probability calculation method and Obtain the sampling probability of the empirical sequence Finally, based on the sampling probability, a given number of empirical sequences are obtained, resulting in a temporary empirical pool after secondary optimization. in, Representing an empirical sequence Cumulative returns This represents a very small positive number, the purpose of which is to ensure that the priority of all empirical sequences is greater than 0. Weighting factor When it indicates degradation to uniform sampling, when The time indicates that the sampling probability is entirely based on the priority of the sample, that is, the higher the priority of the sample, the greater the probability of the sample being sampled.

[0021] Furthermore, in the aforementioned reinforcement learning-based UAV path planning method, the step of dynamically optimizing the obtained child action set into a target action set and obtaining the corresponding empirical sequence samples based on a continuous Markov process includes: When solving each subproblem, actions with a target tendency are randomly selected from the action space. By balancing directional guidance and random perturbation, a sequence that combines path approximation optimality and empirical diversity is generated. The solution of each subproblem is represented as an action solution space close to the target point. The problem is solved using the following formula for calculating the distance variation:

[0022] in, To change the distance, and These are the distances from the current state to the target point and the distances from the next state to the target point, respectively. When, it indicates that the action to be performed is an action that moves closer to the target point. When, it indicates that the action to be performed is an action that moves away from the target point; if performed in the current state. If none of the actions satisfy the constraints, then start from the action solution space far from the target point. Select action to .

[0023] Furthermore, in the aforementioned reinforcement learning-based UAV path planning method, the step of optimizing the initial path using Bézier curves to obtain the final path information includes: The optimized path information obtained by optimizing the initial path using Bézier curves can be represented as follows: The mathematical expression for a Bézier curve is:

[0024] The objective function of the Bézier curve is:

[0025] for The Bernstein basis functions are:

[0026] in, For the first Coordinates of control points Let be the order of the Bézier curve. To control the number of points, These are the normalized curve parameters, corresponding to the starting point of the curve. arrive , For discrete path points, For the first Parameter values ​​at discrete points These are the regularization weight parameters.

[0027] Another object of the present invention is to provide a path planning device for unmanned aerial vehicles based on reinforcement learning, the device comprising: The settings module is used to set the size of the three-dimensional flight environment for drone formations; The acquisition module is used to acquire the starting position and target position of the lead drone in the drone formation, as well as to determine the number and position of obstacles. The discrete module is used to determine the 3D flight environment of the UAV simulation based on the number and location of obstacles, and to discretize the 3D flight environment of the UAV simulation to create a rasterized grid map. A module is established to create the UAV state, state space, and discretized UAV action space in a three-dimensional discretized space based on the simulated three-dimensional flight environment of the UAV. The training module is used to establish the action value function based on the mapping relationship between the state and action space. The action value function is used to calculate the time difference error of the UAV's empirical samples and to train and update the network parameters of the IDDQN algorithm. The selection module is used to determine the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space, as well as the reward function during training. The planning module is used to obtain the initial path of the lead aircraft based on the starting position and target position of the lead aircraft using the IDDQN algorithm, and to optimize the initial path using Bézier curves to obtain the final path information.

[0028] Another object of the present invention is to provide a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0029] Another object of the present invention is to provide an electronic device including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0030] This invention obtains the starting position and target position of the lead drone in a drone formation, and simultaneously determines the number and position of obstacles. Based on the number and position of obstacles, it determines a simulated 3D flight environment for the drones, and discretizes this environment to create a gridded map. Based on the simulated 3D flight environment, it establishes the drone state, state space, and discretized drone action space in the 3D discretized space. An action value function is established based on the mapping relationship between the state and action spaces. This action value function is used to calculate the time difference error of the drone's experience samples and simultaneously train and update the network parameters of the IDDQN algorithm. It determines the strategy for selecting actions from the action space when the lead drone interacts with the flight environment space, as well as the reward function during training. Based on the lead drone's starting position and target position, the IDDQN algorithm is used to obtain the lead drone's initial path, and the initial path is optimized using Bézier curves to obtain the final path information. By dynamically introducing random path experience samples to the target during training, the effect of network parameter training is improved, thereby achieving efficient path planning for drones in a 3D environment. This solves the problems of insufficient feature interaction and large inter-task interference in existing technologies. Attached Figure Description Figure 1 This is a flowchart of the UAV path planning method based on reinforcement learning in the first embodiment of the present invention; Figure 2 This is a motion space diagram of 26 selectable motion directions for the UAV in the UAV path planning method based on reinforcement learning in the first embodiment of the present invention. Figure 3 This is a diagram of the IDDQN main network training framework in the UAV path planning method based on reinforcement learning in the first embodiment of the present invention. Figure 4 This is a structural block diagram of the UAV path planning device based on reinforcement learning in the third embodiment of the present invention.

[0031] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0032] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0033] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0035] Example 1 Please see Figure 1 The figure shows a UAV path planning method based on reinforcement learning in the first embodiment of the present invention, the method including steps S10 to S16.

[0036] Step S10: Set the size of the three-dimensional flight environment for the drone formation.

[0037] The flight environment of the drone formation is , specifically Represents all numbers belonging to the three-dimensional real space. point The following conditions must be met; express The range of coordinates is a closed interval ; express The range of coordinates is a closed interval ; express The range of coordinates is a closed interval In short, the flight environment of drone swarms is a... , , (All units are meters) is a three-dimensional cuboid space with a boundary.

[0038] Step S11: Obtain the starting point position and target point position of the lead drone in the drone formation, and simultaneously determine the number and position of obstacles; Among them, the lead aircraft is the aircraft that leads the drone formation. It plays a core role in command and decision-making within the formation and is responsible for guiding the path planning of the entire formation. First, it is necessary to clarify the starting position (where to start flying) and the target position (where to fly to). This is the core premise for planning the flight path. It is also necessary to determine the number of obstacles (how many obstacles there are) and the spatial position of each obstacle. At the same time, a cylindrical model is used to mathematically describe the shape and range of the obstacles.

[0039] Specifically, the mathematical model of the obstacle can be represented as:

[0040] in and The respective Coordinates of the edge points of the obstacle threat zone and the coordinates of the center of the bottom surface of the obstacle. ; Let be the radius of the base of the cylindrical obstacle; The height of the cylindrical obstacle.

[0041] Step S12: Determine the UAV simulation 3D flight environment based on the number and location of obstacles, and discretize the UAV simulation 3D flight environment to create a rasterized grid map.

[0042] The process involves defining the three-dimensional space within which the drone can fly based on the "number of obstacles" and "location" (where the obstacles are located). This space must avoid obstacles, therefore the flight environment is a passable space containing obstacles. The continuous three-dimensional flight space is then divided into numerous discrete "small cubic grids" (similar to "cutting" a three-dimensional space into countless small cubes). Each grid can be labeled with attributes such as "whether there are obstacles" and "whether it is flyable," allowing the computer to process spatial information in units of "grids."

[0043] Step S13: Based on the simulated three-dimensional flight environment of the UAV, establish the UAV state, state space, and discretized UAV action space in the three-dimensional discretized space.

[0044] Based on the discretized, gridded 3D flight environment, the UAV's state, state space, and action space are defined. Specifically, the UAV state is the specific situation of the UAV at a certain moment (e.g., current location, flight speed, orientation, etc.); the state space is the set of all possible UAV states (e.g., the positions of all flyable grids, combined with all combinations of dimensions such as speed and orientation); and the discretized action space is the set of discrete operations that the UAV can perform (e.g., moving one grid in the x / y / z direction, staying in place, etc.).

[0045] Specifically, take This indicates the current location information of the lead aircraft. Indicates the location of the target point. Representing obstacle information, the leader's state can therefore be represented as The state space of the lead aircraft can be represented as For the discretization of drone actions, a regular hexahedron is constructed with the drone as the center point and two unit lengths as the edge length, such as... Figure 2 As shown, the action space is discretized into 26 directions selectable by the UAV (including the normal directions of the 6 faces of the hexahedron, the directions of the 8 vertices, and the directions of the midpoints of the 12 edges), and the action space is defined as follows. .

[0046] Step S14: Establish an action value function based on the mapping relationship between state and action space. The action value function is used to calculate the time difference error of the UAV's empirical samples and to train and update the network parameters of the IDDQN algorithm.

[0047] Specifically, an action value function is constructed by mapping the state space to the action space (this function is used to quantify the expected benefits that the UAV can obtain when performing a specific action in a specific state). Then, this action value function is used to calculate the time difference error of the UAV's experience samples (i.e., the difference between the current estimate of the action value and the new estimate obtained after combining subsequent states). Based on this error, the parameters of the neural network used by the IDDQN (Dual Deep Q Network) algorithm are trained and updated, so that the network can learn the value of different state-action pairs more accurately, thereby optimizing the UAV's decision-making ability in path planning.

[0048] Step S15: Determine the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space, as well as the reward function during the training process.

[0049] The first part of the algorithm involves two main components: First, the lead aircraft interacts with the discretized 3D flight environment. The lead aircraft employs a "strategy" (i.e., decision rules, such as an ε-greedy strategy: selecting the action currently evaluated as most valuable by the network with a high probability to utilize existing experience, while randomly selecting actions with a lower probability to explore the unknown environment, balancing "utilization" and "exploration"). Second, a "reward function" (i.e., a feedback mechanism, whose core is to assign a quantified "reward" or "penalty" to each action of the lead aircraft during algorithm training; for example, giving a positive reward when approaching the target point, a large negative reward when colliding with obstacles, a small negative reward for prolonged stagnation, and the maximum positive reward for successfully reaching the target point). The design of these two components directly determines whether the lead aircraft can gradually learn the optimal decision-making logic for obstacle avoidance and efficient arrival at the target point in the flight environment through training.

[0050] Specifically, the action selection strategy adopted should ensure that the lead aircraft has a high degree of autonomous exploration capability, while also ensuring that the lead aircraft selects as many actions as possible, guiding it to explore the environment more comprehensively. Therefore, the strategy adopted... The strategy determines the actions the master machine chooses during learning; this strategy can be specifically described as follows:

[0051] in, These are the parameters for the greedy strategy. Given the UAV state. The lead aircraft Randomly select an action with a probability of [missing information]. The probability is used to select the action with the highest value.

[0052] Establish the immediate reward function for IDDQN, including the action reward function and the environment reward function. Define the environment reward as including the target point reward, the environment range reward, the obstacle reward, and the overstep reward. Therefore, the action reward function and the final reward function can be described as follows:

[0053]

[0054] in, This represents the Eulerian distance from the current state to the target point. This represents the Eulerian distance from the next state to the target point; The distance parameter is used to balance the distance reward. This represents the step reward value. The target location for the lead aircraft. For the lead aircraft in the current state of the drone Location, Next status for the lead aircraft Location, For the environmental reward function, Indicates the current number of steps being executed. Indicates environmental constraints. Indicates obstacle constraints. This indicates a range constraint for the number of steps.

[0055] Step S16: Based on the starting position and target position of the lead aircraft, the IDDQN algorithm is used to obtain the initial path of the lead aircraft, and the Bézier curve is used to optimize the initial path to obtain the final path information.

[0056] Given the starting point (flight start point) and target point (flight end point) of the lead aircraft, the IDDQN (Dual Deep Q-Network) algorithm—a reinforcement learning-based decision-making algorithm—is first applied. Leveraging its trained and optimized neural network model (which has learned environmental interaction rules and obstacle avoidance logic through action-value functions, reward functions, etc.), an initial path from the starting point to the target point is searched and generated in a discretized 3D flight environment containing obstacles. (This path is typically formed by connecting discrete nodes of a raster map, and may have abrupt transitions and lack of smoothness, only satisfying the basic feasibility requirement of "reaching the target and avoiding obstacles"). Subsequently, in order to make the path more in line with the actual flight requirements of the UAV (such as reducing drastic changes in fuselage attitude, reducing energy consumption, and improving flight stability), a parameterized curve tool, Bézier curve, which can generate a smooth and continuous trajectory through key control points, was used to fit and smooth the discrete nodes of the initial path, and finally obtained a continuous, smooth final flight path information that still meets the requirements of obstacle avoidance and accessibility.

[0057] For example, the mathematical expression for a Bézier curve is:

[0058] in, For the first Coordinates of control points Let be the order of the Bézier curve. To control the number of points. These are the normalized curve parameters, corresponding to the starting point of the curve. arrive ; for The Bernstein basis functions are:

[0059] Combining the above formulas, the objective function for the Bézier curve is:

[0060] in, For discrete path points, For the first Parameter values ​​at discrete points These are the regularization weight parameters.

[0061] In summary, the UAV path planning method based on reinforcement learning in the above embodiments of the present invention obtains the starting position and target position of the lead UAV in the UAV formation, and simultaneously determines the number and position of obstacles; determines the simulated 3D flight environment of the UAV based on the number and position of obstacles, and discretizes the simulated 3D flight environment to establish a rasterized grid map; establishes the UAV state, state space, and discretized UAV action space in the 3D discretized space based on the simulated 3D flight environment; establishes an action value function based on the mapping relationship between the state and action space, wherein the action value function is used to calculate the time difference error of the UAV's experience samples and simultaneously train and update the network parameters of the IDDQN algorithm; determines the strategy for selecting actions from the action space when the lead UAV interacts with the flight environment space and the reward function during the training process; obtains the initial path of the lead UAV based on the starting position and target position of the lead UAV using the IDDQN algorithm, and optimizes the initial path using Bézier curves to obtain the final path information; improves the effect of network parameter training by dynamically introducing random path experience samples to the target during training, thereby achieving efficient path planning for UAVs in a 3D environment. This solves the problems of insufficient feature interaction and large inter-task interference in the prior art.

[0062] Example 2 This embodiment also proposes a UAV path planning method based on reinforcement learning. The difference between the UAV path planning method based on reinforcement learning in this embodiment and the UAV path planning method based on reinforcement learning in Embodiment 1 is as follows: The steps for obtaining the initial path of the lead aircraft using the IDDQN algorithm based on the lead aircraft's starting position and target position include: Define the empirical sample of the drone as The initialization sample priority and cumulative gradient are as follows: and At the same time, initialize the main network parameters and the target network parameters as follows: ; like Figure 3 As shown, a Markov process with a given step size is performed to obtain a series of experience samples and store them in the initial experience pool. In the later stages of training the IDDQN algorithm, the experience sequences generated during training are combined into a temporary experience pool. ; in: ; in, Representing state After the action to state Markov processes, performing actions After Indicate whether this training session has ended; Define a range of parameters for a greedy strategy. ; In the temporary experience pool, a preset genetic algorithm is used to sequentially extract the experience sequences in the temporary experience pool as the parent, and at the same time extract the experience sequences different from the parent as the mother, decouple the action sets of the parent and the mother, decode the action sets and encode them in binary, and then perform crossover and mutation to obtain the action sets of the offspring. in, Binary encoding representing the action space; The obtained child action set is dynamically optimized into the action set of the target point, and the corresponding empirical sequence sample is obtained based on the continuous Markov process. For the obtained experience sequence sample set, the target point experience sequence is filtered according to fitness and finally updated to obtain an optimized temporary experience pool. Input the experience samples from the temporary experience pool into the initial experience pool; During the process of storing experience into the initial experience pool, experience samples are sampled from the initial experience pool in a given batch according to priority probability for network training. The probability of each sample being sampled in the experience pool can be expressed as:

[0063] in, Prioritize samples; Calculate the importance sampling weights of the sampled empirical samples to define the importance of the sample, which is used for subsequent gradient updates. The importance weights of the empirical samples are:

[0064] in, This represents the number of existing experience samples in the initial experience pool; The temporal difference error of the sampled empirical sample is calculated. An auxiliary importance weight is used to update the cumulative gradient and simultaneously to update the priority of the sample. The temporal difference error of the empirical sample is expressed as:

[0065] The priority of the empirical sample is updated using the temporal difference error of the empirical sample:

[0066] The cumulative gradient is updated by combining the importance weights of the samples and the temporal difference error:

[0067] Using formula , Update the main network parameters of DDQN to the learning rate; At regular time intervals, a soft update method is used to update the main network parameter values ​​to the target network:

[0068] in, For soft update parameters; The trained main network parameters As the training brain of the lead aircraft, it obtains discretized virtual path information through a Markov process. ; Furthermore, after the step of filtering according to fitness and finally updating the temporary experience pool with the filtered target point experience sequence to obtain an optimized temporary experience pool, the method further includes: Calculate the cumulative reward of an experience sequence in the experience pool once. ; according to Priority calculation method and Obtain the sampling priority of all empirical sequences ; according to Sampling probability calculation method and Obtain the sampling probability of the empirical sequence Finally, based on the sampling probability, a given number of empirical sequences are obtained, resulting in a temporary empirical pool after secondary optimization. in, Representing an empirical sequence Cumulative returns This represents a very small positive number, the purpose of which is to ensure that the priority of all empirical sequences is greater than 0. Weighting factor When it indicates degradation to uniform sampling, when This means that the sampling probability is entirely based on the priority of the sample, that is, the higher the priority of the sample, the greater the probability of it being sampled; Furthermore, the step of dynamically optimizing the obtained child action set into an action set for the target point using dynamic programming, and obtaining the corresponding empirical sequence samples based on a continuous Markov process, includes: When solving each subproblem, actions with a target tendency are randomly selected from the action space. By balancing directional guidance and random perturbation, a sequence that combines path approximation optimality and empirical diversity is generated. The solution of each subproblem is represented as an action solution space close to the target point. The problem is solved using the following formula for calculating the distance variation:

[0069] in, To change the distance, and These are the distances from the current state to the target point and the distances from the next state to the target point, respectively. When, it indicates that the action to be performed is an action that moves closer to the target point. When, it indicates that the action to be performed is an action that moves away from the target point; if performed in the current state. If none of the actions satisfy the constraints (obstacle constraints and environmental constraints), then the solution space for actions far from the target point is... Select action to .

[0070] In summary, the UAV path planning method based on reinforcement learning in the above embodiments of the present invention obtains the starting position and target position of the lead UAV in the UAV formation, and simultaneously determines the number and position of obstacles; determines the simulated 3D flight environment of the UAV based on the number and position of obstacles, and discretizes the simulated 3D flight environment to establish a rasterized grid map; establishes the UAV state, state space, and discretized UAV action space in the 3D discretized space based on the simulated 3D flight environment; establishes an action value function based on the mapping relationship between the state and action space, wherein the action value function is used to calculate the time difference error of the UAV's experience samples and simultaneously train and update the network parameters of the IDDQN algorithm; determines the strategy for selecting actions from the action space when the lead UAV interacts with the flight environment space and the reward function during the training process; obtains the initial path of the lead UAV based on the starting position and target position of the lead UAV using the IDDQN algorithm, and optimizes the initial path using Bézier curves to obtain the final path information; improves the effect of network parameter training by dynamically introducing random path experience samples to the target during training, thereby achieving efficient path planning for UAVs in a 3D environment. This solves the problems of insufficient feature interaction and large inter-task interference in the prior art.

[0071] Example 3 Please see Figure 4 The image shows a UAV path planning device based on reinforcement learning proposed in the third embodiment of the present invention. The device includes: The setting module 100 is used to set the size of the three-dimensional flight environment for drone formations. The acquisition module 200 is used to acquire the starting position of the lead drone and the target position in the drone formation, as well as to determine the number and position of obstacles. Discrete module 300 is used to determine the UAV simulation 3D flight environment based on the number and location of obstacles, and to discretize the UAV simulation 3D flight environment to create a rasterized grid map. Module 400 is established to create the UAV state, state space, and discretized UAV action space in a three-dimensional discretized space based on the UAV simulation three-dimensional flight environment. Training module 500 is used to establish action value functions from the mapping relationship between state and action space. The action value functions are used to calculate the time difference error of the UAV's empirical samples and to train and update the network parameters of the IDDQN algorithm. Selection module 600 is used to determine the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space, as well as the reward function during the training process; The planning module 700 is used to obtain the initial path of the lead aircraft based on the starting position and target position of the lead aircraft using the IDDQN algorithm, and to optimize the initial path using Bézier curves to obtain the final path information.

[0072] The functions or operation steps implemented by the above modules are largely the same as those in the above method embodiments, and will not be repeated here.

[0073] Example 4 In another aspect, the present invention provides a readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in any one of Embodiments 1 to 2 above.

[0074] Example 5 In another aspect, the present invention provides an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of any one of the methods described in Embodiments 1 to 2 above.

[0075] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequential list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0077] More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable storage media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0078] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0079] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0080] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for unmanned aerial vehicle (UAV) path planning based on reinforcement learning, characterized in that, The method includes: Set the size of the three-dimensional flight environment for the drone formation; Obtain the starting position and target position of the lead drone in the drone formation, and simultaneously determine the number and location of obstacles; The three-dimensional flight environment of the UAV is determined based on the number and location of obstacles, and the three-dimensional flight environment of the UAV simulation is discretized to create a rasterized grid map. Based on the simulated 3D flight environment of the UAV, the UAV state, state space, and discretized UAV action space in the 3D discretized space are established. An action value function is established based on the mapping relationship between the state and action space. The action value function is used to calculate the time difference error of the UAV's empirical samples and to train and update the network parameters of the IDDQN algorithm. Determine the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space, as well as the reward function during training; The initial path of the lead aircraft is obtained using the IDDQN algorithm based on the starting position and target position of the lead aircraft, and the final path information is obtained by optimizing the initial path using Bézier curves.

2. The UAV path planning method based on reinforcement learning according to claim 1, characterized in that, The process involves establishing the UAV state in a three-dimensional discretized space based on the simulated three-dimensional flight environment of the UAV. State space and the discretized drone action space The steps include: Pick This indicates the current location information of the lead aircraft. Indicates the target location of the lead aircraft. Obstacle information is represented, and the drone's status is represented as follows: The state space of the UAV is represented as ; With the drone as the center point, a regular hexahedron is constructed with an edge length of 2 units. The drone's action space is discretized into 26 directions that the drone can choose from, and the drone's action space is defined as follows: .

3. The UAV path planning method based on reinforcement learning according to claim 2, characterized in that, The steps of determining the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space and the reward function during training include: use The strategy determines the actions the primary machine chooses during learning, specifically: in, Given the drone state, the parameters for the greedy strategy are... The lead aircraft Randomly select an action with a probability of [missing information]. The probability of choosing the action with the highest value; Establish the immediate reward function for IDDQN, which includes an action reward function and an environment reward function. The environment reward function includes a target point reward, an environment range reward, an obstacle reward, and a superstep reward. The action reward function and the final immediate reward function are as follows: in, This represents the Eulerian distance from the current state to the target point. This represents the Eulerian distance from the next state to the target point; The distance parameter is used to balance the distance reward. This represents the step reward value. The target location for the lead aircraft. For the lead aircraft in the current state of the drone Location, Next status for the lead aircraft Location, For the environmental reward function, Indicates the current number of steps being executed. Indicates environmental constraints. Indicates obstacle constraints. This indicates a range constraint for the number of steps.

4. The UAV path planning method based on reinforcement learning according to claim 1, characterized in that, The steps for obtaining the initial path of the lead aircraft using the IDDQN algorithm based on the lead aircraft's starting position and target position include: Define the empirical sample of the drone as The initialization sample priority and cumulative gradient are as follows: and At the same time, initialize the main network parameters and the target network parameters as follows: ; A Markov process with a given step size is performed to obtain a series of experience samples, which are stored in an initial experience pool. In the later stages of training the IDDQN algorithm, the experience sequences generated during training are combined into a temporary experience pool. ; in: ； in, Representing state After the action to state Markov processes, performing actions After Indicate whether this training session has ended; Define a range of parameters for a greedy strategy. ; In the temporary experience pool, a preset genetic algorithm is used to sequentially extract the experience sequences in the temporary experience pool as the parent, and at the same time extract the experience sequences different from the parent as the mother, decouple the action sets of the parent and the mother, decode the action sets and encode them in binary, and then perform crossover and mutation to obtain the action sets of the offspring. in, Binary encoding representing the action space; The obtained child action set is dynamically optimized into the action set of the target point, and the corresponding empirical sequence sample is obtained based on the continuous Markov process. For the obtained experience sequence sample set, after filtering according to fitness, the filtered target point experience sequences are updated to obtain a temporary experience pool with one optimization. Input the experience samples from the temporary experience pool into the initial experience pool; During the process of storing experience into the initial experience pool, experience samples are sampled from the initial experience pool in a given batch according to priority probability for network training. The probability of each sample being sampled in the experience pool can be expressed as: in, Prioritize samples; Calculate the importance sampling weights of the sampled empirical samples to define the importance of the sample, which is used for subsequent gradient updates. The importance weights of the empirical samples are: in, This represents the number of existing experience samples in the initial experience pool; The temporal difference error of the sampled empirical sample is calculated. An auxiliary importance weight is used to update the cumulative gradient and simultaneously to update the priority of the sample. The temporal difference error of the empirical sample is expressed as: The priority of the empirical sample is updated using the temporal difference error of the empirical sample: The cumulative gradient is updated by combining the importance weights of the samples and the temporal difference error: Using formula , Update the main network parameters of DDQN to the learning rate; At regular time intervals, a soft update method is used to update the main network parameter values ​​to the target network: in, For soft update parameters; The trained main network parameters As the training brain of the lead aircraft, it obtains discretized virtual path information through a Markov process. .

5. The UAV path planning method based on reinforcement learning according to claim 4, characterized in that, The step of updating the temporary experience pool by filtering the target points according to fitness and obtaining an optimized temporary experience pool also includes: Calculate the cumulative reward of an experience sequence in the experience pool once. ; according to Priority calculation method and Obtain the sampling priority of all empirical sequences ; according to Sampling probability calculation method and Obtain the sampling probability of the empirical sequence Finally, based on the sampling probability, a given number of empirical sequences are obtained, resulting in a temporary empirical pool after secondary optimization. in, Representing an empirical sequence Cumulative returns This represents a very small positive number, the purpose of which is to ensure that the priority of all empirical sequences is greater than 0. Weighting factor When it indicates degradation to uniform sampling, when The time indicates that the sampling probability is entirely based on the priority of the sample, that is, the higher the priority of the sample, the greater the probability of the sample being sampled.

6. The UAV path planning method based on reinforcement learning according to claim 5, characterized in that, The steps of dynamically optimizing the obtained child action set into the action set of the target point using dynamic programming, and obtaining the corresponding empirical sequence samples based on a continuous Markov process include: When solving each subproblem, actions with a target tendency are randomly selected from the action space. By balancing directional guidance and random perturbation, a sequence that combines path approximation optimality and empirical diversity is generated. The solution of each subproblem is represented as an action solution space close to the target point. The problem is solved using the following formula for calculating the distance variation: in, To change the distance, and These are the distances from the current state to the target point and the distances from the next state to the target point, respectively. When, it indicates that the action to be performed is an action that moves closer to the target point. When this occurs, it indicates that the action being performed is an action that moves away from the target point; If executed in the current state If none of the actions satisfy the constraints, then start from the action solution space far from the target point. Select action to .

7. The UAV path planning method based on reinforcement learning according to claim 6, characterized in that, The step of optimizing the initial path using Bézier curves to obtain the final path information includes: The optimized path information obtained by optimizing the initial path using Bézier curves can be represented as follows: The mathematical expression for a Bézier curve is: The objective function of the Bézier curve is: for The Bernstein basis functions are: in, For the first Coordinates of control points Let be the order of the Bézier curve. To control the number of points, These are the normalized curve parameters, corresponding to the starting point of the curve. arrive , For discrete path points, For the first Parameter values ​​at discrete points These are the regularization weight parameters.

8. A path planning device for unmanned aerial vehicles based on reinforcement learning, characterized in that, The device includes: The settings module is used to set the size of the three-dimensional flight environment for drone formations; The acquisition module is used to acquire the starting position and target position of the lead drone in the drone formation, as well as to determine the number and position of obstacles. The discrete module is used to determine the 3D flight environment of the UAV simulation based on the number and location of obstacles, and to discretize the 3D flight environment of the UAV simulation to create a rasterized grid map. A module is established to create the UAV state, state space, and discretized UAV action space in a three-dimensional discretized space based on the simulated three-dimensional flight environment of the UAV. The training module is used to establish the action value function based on the mapping relationship between the state and action space. The action value function is used to calculate the time difference error of the UAV's empirical samples and to train and update the network parameters of the IDDQN algorithm. The selection module is used to determine the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space, as well as the reward function during training. The planning module is used to obtain the initial path of the lead aircraft based on the starting position and target position of the lead aircraft using the IDDQN algorithm, and to optimize the initial path using Bézier curves to obtain the final path information.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the program, implements the steps of the method as described in any one of claims 1 to 7.

Images