A method and device for unmanned aerial vehicle path planning based on reinforcement learning

Abstract

The application discloses a kind of unmanned plane path planning method and device based on reinforcement learning, it is related to unmanned plane technical field, the method includes: setting unmanned plane formation three-dimensional flight environment size;Obtain the position of the starting point of the long machine in the unmanned plane formation, target point position, and simultaneously determine the number and position of obstacles;According to the number and position of obstacles, determine the unmanned plane simulation three-dimensional flight environment, the grid map of the unmanned plane simulation three-dimensional flight environment is discretized;According to the unmanned plane simulation three-dimensional flight environment, establish the unmanned plane state in three-dimensional discretization space, state space and discretized unmanned plane action space;Based on the starting point of the long machine, target point position uses IDDQN algorithm to obtain the initial path of long machine, and adopts Bezier curve to optimize initial path and obtains final path information.The application solves the problem of low efficiency of the existing unmanned plane path planning mode.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicles, and particularly relates to an unmanned aerial vehicle path planning method and device based on reinforcement learning. BACKGROUND

[0002] As a kind of unmanned aerial vehicle that can fly high maneuverability, high adaptability, accurate navigation and guidance and can perform multiple types of tasks, low-altitude unmanned aerial vehicle has the characteristics of low cost, low maintenance cost, flexible and convenient deployment, and has become one of the core carriers of low-altitude economic development. In low-altitude logistics distribution, urban air traffic, infrastructure inspection, precision agriculture plant protection, emergency rescue and other typical application scenarios of low-altitude economy, it has been widely used. In urban areas, path planning problem is still one of the core challenges of the practical application of unmanned aerial vehicles. Therefore, the obstacle avoidance problem of three-dimensional complex obstacle environment further improves the quality, real-time performance and robustness of the global path of the algorithm.

[0003] Currently, common unmanned aerial vehicle path planning algorithms include fast random search tree, dynamic window method, artificial potential field method, ant colony algorithm and particle swarm algorithm. In the face of increasingly complex flight environment and high-dimensional space, traditional algorithms and intelligent algorithms may have potential problems such as falling into local optimum, long planning time and low path smoothness. Related scholars have applied intelligent path planning algorithms such as genetic algorithm, ant colony algorithm, particle swarm algorithm and neural network to path planning to solve the shortcomings of traditional methods. However, with the increasing complexity and uncontrollable factors of unmanned aerial vehicle flight environment in low-altitude application scenarios, higher requirements are put forward for the active planning performance of unmanned aerial vehicles. Traditional and intelligent unmanned aerial vehicle path planning algorithms are difficult to be directly applied to unknown environments lacking prior experience, and it is urgent to research more autonomous unmanned aerial vehicle path planning methods that can adapt to little prior knowledge.

[0004] Reinforcement learning provides an effective framework for solving the above problems. It mainly interacts with the environment and the agent. The agent performs actions in the environment, the environment feedbacks the reward of the agent performing the action, and adjusts the strategy according to the reward value to maximize the cumulative return. The agent learns the optimal strategy through repeated interaction with the environment, which makes it an ideal candidate for solving complex low-altitude path planning problems. The agent can learn the optimal strategy under the condition of little prior knowledge through reinforcement learning.

[0005] Q-Learning is an early applied reinforcement learning method for the path planning problem of unmanned aerial vehicles. However, when the complexity of the environment rises or the dimension of the environment increases, the storage data of Q-Learning will increase by a geometric multiple, and the dimension explosion problem may occur, which makes the path planning problem more complex. A neural network is used to fit the Q-table in Q-Learning to solve the problem of large state space, and a deep Q network (DQN) method combining deep learning and reinforcement learning is proposed. Compared with Q-Learning, DQN has faster convergence speed and effectively avoids the dimension explosion problem. However, when the density of environmental obstacles increases or the reward signal is sparse, the convergence speed of DQN will also slow down. Therefore, the current unmanned aerial vehicle path planning method has the problem of low efficiency. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a reinforcement learning-based unmanned aerial vehicle path planning method and device, aiming to solve the problem of low efficiency of the current unmanned aerial vehicle path planning method.

[0007] In one aspect, the present application provides a reinforcement learning-based unmanned aerial vehicle path planning method, which comprises:

[0008] Setting the size of the three-dimensional flight environment of the unmanned aerial vehicle formation;

[0009] Obtaining the set starting position and target position of the lead aircraft in the unmanned aerial vehicle formation, and simultaneously determining the number and position of obstacles;

[0010] Determining the unmanned aerial vehicle simulation three-dimensional flight environment according to the number and position of obstacles, and discretizing the unmanned aerial vehicle simulation three-dimensional flight environment to establish a grid map;

[0011] According to the unmanned aerial vehicle simulation three-dimensional flight environment, establishing the state, state space and discretized action space of the unmanned aerial vehicle in the three-dimensional discretized space;

[0012] Mapping the state and action space to establish an action value function, wherein the action value function is used to calculate the time difference error of the experience sample of the unmanned aerial vehicle and simultaneously train and update the network parameters of the IDDQN algorithm;

[0013] Determining the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space and the reward function in the training process;

[0014] Based on the starting position and target position of the lead aircraft, the IDDQN algorithm is used to obtain the initial path of the lead aircraft, and the Bezier curve is used to optimize the initial path to obtain the final path information.

[0015] Further, the unmanned aerial vehicle path planning method based on reinforcement learning, wherein the step of establishing the unmanned aerial vehicle state in the three-dimensional discrete space according to the three-dimensional flight environment simulation of the unmanned aerial vehicle comprises: a state space and a discrete unmanned aerial vehicle action space .

[0016] Taking to represent the current position information of the long machine, to represent the target point position of the long machine, to represent the obstacle information, the unmanned aerial vehicle state is represented as , and the unmanned aerial vehicle state space is represented as .

[0017] A regular hexahedron is constructed with the unmanned aerial vehicle as the center point and 2 unit lengths as the edge length, the unmanned aerial vehicle action space is discretized into 26 directions available for the unmanned aerial vehicle to select, and the unmanned aerial vehicle action space is defined as .

[0018] Further, the unmanned aerial vehicle path planning method based on reinforcement learning, wherein the step of determining the strategy of selecting an action from the action space when the long machine interacts with the flight environment space and the reward function in the training process comprises:

[0019] The strategy is adopted to determine the action selection of the long machine in learning, and specifically:

[0020]

[0021] wherein is a greedy strategy parameter, given the unmanned aerial vehicle state , the long machine randomly selects an action with a probability of , and selects an action with the maximum action value with a probability of .

[0022] The immediate reward function of the IDDQN is established, and the immediate reward function comprises an action reward function and an environment reward function, and the environment reward function comprises a target point reward, an environment range reward, an obstacle reward and a step number reward, wherein the action reward function and the final immediate reward function are as follows:

[0023]

[0024]

[0025] wherein represents the Euler distance from the current state to the target point, represents the Euler distance from the next state to the target point. a distance parameter for balancing the distance reward, a step number reward value, a target position of the long machine, a position of the long machine in a current UAV state , a position of the long machine in a next state , an environment reward function, a current step number, an environment range constraint, an obstacle constraint, a step number range constraint.

[0026] Further, the above-mentioned UAV path planning method based on reinforcement learning, wherein the step of obtaining an initial path of the long machine based on the starting position of the long machine and the target position by using the IDDQN algorithm comprises:

[0027] defining an experience sample of the UAV as , initializing the priority corresponding to the sample and the cumulative gradient as and , and initializing the main network parameters and the target network parameters as ;

[0028] a series of experience samples are obtained by performing a Markov process under a given step length and are saved in an initial experience pool, and in the later stage of IDDQN algorithm training, experience sequences generated by training are combined into a temporary experience pool ;

[0029] wherein:

[0030] ;

[0031] wherein, represents a state , an action , a state , a Markov process, an action , a reward indicating whether the training is ended or not;

[0032] defining a range of a greedy strategy parameter ;

[0033] In the temporary experience pool, a preset genetic algorithm is used to sequentially extract experience sequences in the temporary experience pool as parents, and extract experience sequences different from the parents as mothers, decouple the action sets of the parents and the mothers, decode the action sets and perform binary coding, and then perform crossing and mutation to obtain the action sets of the offspring;

[0034] wherein, denotes a binary encoding of the action space;

[0035] performing dynamic programming optimization on the obtained offspring action set to an action set of the target point, and obtaining a corresponding experience sequence sample based on a continuous Markov process;

[0036] For the obtained experience sequence sample set, after screening according to the fitness, the screened target point experience sequence is updated to obtain a temporary experience pool after one optimization;

[0037] The experience samples in the temporary experience pool are input into the initial experience pool;

[0038] In the process of storing experience into the initial experience pool, according to the priority probability, a given batch of experience samples is sampled in the initial experience pool for network training, and the probability of sampling each sample in the experience pool can be expressed as:

[0039]

[0040] wherein, is the sample priority;

[0041] The importance sampling weight of the sampled experience sample is calculated, so as to define the importance of the sample, which is used for subsequent update of the accumulated gradient, and the importance weight of the experience sample is:

[0042]

[0043] wherein, denotes the number of existing experience samples in the initial experience pool;

[0044] The time difference error of the sampled experience sample is calculated, which is used for the update of the accumulated gradient with the importance weight, and is also used for updating the priority of the sample, and the time difference error of the experience sample is expressed as:

[0045]

[0046] The time difference error of the experience sample is used to update the priority of the experience sample:

[0047]

[0048] The accumulated gradient is updated in combination with the importance weight of the sample and the time difference error:

[0049]

[0050] The formula , is the learning rate, and the main network parameters of the DDQN are updated;

[0051] At certain time steps, the main network parameter values are updated to the target network using a soft update method:

[0052]

[0053] wherein, is a soft update parameter;

[0054] The trained main network parameters are used as the training brain of the long machine to obtain discretized virtual path information through a Markov process.

[0055] Further, the above-mentioned unmanned aerial vehicle path planning method based on reinforcement learning, wherein the step of updating the filtered target point experience sequence to obtain a temporarily optimized temporary experience pool after screening according to the fitness further comprises:

[0056] calculating the cumulative return of the experience sequence in the once-optimized experience pool ;

[0057] According to the priority calculation method and the sampling priority of all experience sequences is obtained ;

[0058] According to the sampling probability calculation method and the sampling probability of the experience sequence is obtained , and finally a given number of experience sequences are obtained according to the sampling probability to obtain a twice-optimized temporary experience pool;

[0059] wherein, represents the cumulative return of the experience sequence , represents a very small positive number, which is to ensure that the priority of all experience sequences is greater than 0, a weight factor, when it represents degeneration to uniform sampling, when it represents that the sampling probability is completely based on the priority of the sample, that is, the higher the priority of the sample, the greater the probability of being sampled.

[0060] Further, the above-mentioned unmanned aerial vehicle path planning method based on reinforcement learning, wherein the step of obtaining the experience sequence sample corresponding to the action set of the target point by dynamically planning the obtained child action set comprises:

[0061] In solving each sub-problem, an action with target tendency is randomly selected in the action space, a sequence with approximate optimal path and experience diversity is generated by balancing direction guidance and random disturbance, and each sub-problem is solved as an action solution space close to the target point The solving problem is specifically solved by using the following distance change difference formula:

[0062]

[0063] Wherein, is the distance change difference, is the distance from the current state to the target point, and the distance from the next state to the target point is , when , it indicates that the executed action is a close-to-target action, and when , it indicates that the executed action is a far-from-target action; if all actions performed in the current state do not satisfy the constraint condition, an action is selected from the action solution space far from the target point to make .

[0064] Further, the unmanned aerial vehicle path planning method based on reinforcement learning, wherein the step of optimizing the initial path by using the Bezier curve to obtain the final path information comprises:

[0065] The optimized path information obtained by optimizing the initial path by using the Bezier curve can be represented as , and the mathematical expression of the Bezier curve is:

[0066]

[0067] The objective function of the Bezier curve is:

[0068]

[0069] is the order Bernstein basis function, and the mathematical expression of the Bernstein basis function is:

[0070]

[0071] Wherein, is the coordinate of the control point, is the order of the Bezier curve, is the number of control points, is the normalized curve parameter, and corresponds to the starting point of the curve to , is the discrete path point, is the​ a parameter value of a discrete point, is a regularization weight parameter.

[0072] Another object of the present application is to provide a UAV path planning device based on reinforcement learning, the device comprising:

[0073] a setting module configured to set a size of a three-dimensional flight environment of a UAV formation;

[0074] an acquisition module configured to acquire a starting position of a lead aircraft, a target position, and a number and a position of obstacles in the UAV formation;

[0075] a discretization module configured to determine a UAV simulation three-dimensional flight environment according to the number and the position of the obstacles, discretize the UAV simulation three-dimensional flight environment, and establish a grid map;

[0076] an establishment module configured to establish a UAV state, a state space, and a discretized UAV action space in a three-dimensional discretized space according to the UAV simulation three-dimensional flight environment;

[0077] a training module configured to establish an action value function from a mapping relationship between the state and the action space, wherein the action value function is configured to calculate a time difference error of an experience sample of the UAV and simultaneously train network parameters of an IDDQN algorithm;

[0078] a selection module configured to determine a strategy for selecting an action from the action space when the lead aircraft interacts with the flight environment space and a reward function in a training process;

[0079] a planning module configured to acquire an initial path of the lead aircraft based on the starting position and the target position of the lead aircraft using the IDDQN algorithm, and optimize the initial path using a Bezier curve to obtain final path information.

[0080] Another object of the present application is to provide a readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the above method.

[0081] Another object of the present application is to provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the processor, when executing the program, implements the steps of the above method.

[0082] The application obtains the set leader starting position and target point position in the unmanned aerial vehicle formation, and simultaneously determines the number and position of obstacles; determines the unmanned aerial vehicle simulation three-dimensional flight environment according to the number and position of obstacles, discretizes the unmanned aerial vehicle simulation three-dimensional flight environment to establish a grid map; establishes the unmanned aerial vehicle state, state space and discretized unmanned aerial vehicle action space in the three-dimensional discretized space according to the unmanned aerial vehicle simulation three-dimensional flight environment; establishes an action value function from the state and action space mapping relationship, wherein the action value function is used to calculate the time difference error of the experience sample of the unmanned aerial vehicle and simultaneously train and update the network parameters of the IDDQN algorithm; determines the strategy of selecting actions from the action space when the leader interacts with the flight environment space and the reward function in the training process; obtains the initial path of the leader based on the leader starting position and target point position by using the IDDQN algorithm, optimizes the initial path by using the Bezier curve to obtain the final path information, dynamically introduces the random path experience sample to the target in the training, and thus improves the training effect of the network parameters, so as to realize efficient path planning of the unmanned aerial vehicle in the three-dimensional environment. The problems of insufficient feature interaction and large task interference in the prior art are solved. BRIEF DESCRIPTION OF DRAWINGS

[0083] Figure 1 Flow chart of the unmanned aerial vehicle path planning method based on reinforcement learning in the first embodiment of the application;

[0084] Figure 2 Action space diagram of 26 motion directions selectable by the unmanned aerial vehicle in the unmanned aerial vehicle path planning method based on reinforcement learning in the first embodiment of the application;

[0085] Figure 3 IDDQN main network training framework diagram in the unmanned aerial vehicle path planning method based on reinforcement learning in the first embodiment of the application;

[0086] Figure 4 Structure block diagram of the unmanned aerial vehicle path planning device based on reinforcement learning in the third embodiment of the application.

[0087] The following specific embodiments will further illustrate the application in combination with the above-mentioned drawings. DETAILED DESCRIPTION

[0088] In order to facilitate the understanding of the application, the application will be described more fully below with reference to the related drawings. Several embodiments of the application are shown in the drawings. However, the application can be realized in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the application more thorough and comprehensive.

[0089] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0090] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0091] Example 1

[0092] Please see Figure 1 The figure shows a UAV path planning method based on reinforcement learning in the first embodiment of the present invention, the method including steps S10 to S16.

[0093] Step S10: Set the size of the three-dimensional flight environment for the drone formation.

[0094] The flight environment of the drone formation is , specifically Represents all numbers belonging to the three-dimensional real space. point The following conditions must be met; express The range of coordinates is a closed interval ; express The range of coordinates is a closed interval ; express The range of coordinates is a closed interval In short, the flight environment of drone swarms is a... , , (All units are meters) is a three-dimensional cuboid space with a boundary.

[0095] Step S11: Obtain the starting point position and target point position of the lead drone in the drone formation, and simultaneously determine the number and position of obstacles;

[0096] Wherein, the long machine is the leading aircraft in the unmanned aerial vehicle formation, which undertakes the core role of commanding and decision-making in the formation, and is responsible for leading the path planning of the whole formation; first, the starting point position (from where to start flying) and the target point position (where to fly to) of the long machine need to be determined, which is the core premise of planning the flight path, and the number of obstacles (how many obstacles) and the spatial position of each obstacle need to be determined; at the same time, the shape and range of the obstacle are mathematically described by a cylindrical model.

[0097] Specifically, the mathematical model of the obstacle can be expressed as:

[0098]

[0099] Wherein And The coordinates of the edge points of the first Obstacle threat area and the coordinates of the bottom center of the obstacle, . The radius of the bottom of the cylindrical obstacle; The height of the cylindrical obstacle.

[0100] Step S12, according to the number and position of the obstacles, a three-dimensional flight environment for unmanned aerial vehicle simulation is determined, and the three-dimensional flight environment for unmanned aerial vehicle simulation is discretized to establish a grid map.

[0101] Wherein, according to the number of obstacles (how many obstacles) and the position (where the obstacle is), the three-dimensional space range in which the unmanned aerial vehicle can fly (which needs to avoid obstacles, so the flight environment is the passable space containing obstacles) is determined, and then the continuous three-dimensional flight space is divided into a large number of discrete "small cube grids" (similar to "cutting" the three-dimensional space into a large number of small cubes). Each grid can be marked with attributes such as "whether there is an obstacle" and "whether it can fly", so that the computer can process spatial information in units of "grids".

[0102] Step S13, according to the three-dimensional flight environment for unmanned aerial vehicle simulation, the state of the unmanned aerial vehicle in the three-dimensional discretized space, the state space and the discretized action space of the unmanned aerial vehicle are established.

[0103] Wherein, based on the discretized three-dimensional flight environment, the state, state space and action space of the unmanned aerial vehicle are defined. Specifically, the state of the unmanned aerial vehicle is the specific situation of the unmanned aerial vehicle at a certain time (such as: the current position, flight speed, direction, etc.); the state space is the set of all possible states of the unmanned aerial vehicle (such as: all the positions of the flyable grids, combined with the speed, direction, etc. dimensions of all combinations); the discretized action space is the set of discrete operations that the unmanned aerial vehicle can perform (such as: moving 1 grid in x / y / z direction, staying in place, etc.).

[0104] Specifically, take represents the current position information of the long machine, represents the target point position, represents the obstacle information, so the state of the long machine can be represented as , the state space of the long machine can be represented as , the action of the unmanned aerial vehicle is discretized, and a regular hexahedron is constructed with the unmanned aerial vehicle as the center point and 2 unit lengths as the edge length, as shown in Figure 2 , the action space is discretized into 26 directions available for the unmanned aerial vehicle to choose (including the normal direction of the 6 faces of the regular hexahedron, the direction of the 8 vertices, and the direction of the 12 midpoint of the edges), and the action space is defined as .

[0105] Step S14, the action value function is established by the mapping relationship between the state and the action space, wherein the action value function is used to calculate the time difference error of the experience sample of the unmanned aerial vehicle and simultaneously train and update the network parameters of the IDDQN algorithm.

[0106] , the action value function (which is used to quantify the expected return that the unmanned aerial vehicle can obtain when performing a specific action in a specific state) is constructed by the mapping relationship between the state space and the action space, then the time difference error of the experience sample of the unmanned aerial vehicle (i.e. the difference between the current estimate of the action value and the new estimate obtained by combining the subsequent state) is calculated using this action value function, and the parameters of the neural network used by the IDDQN (Double Deep Q Network) algorithm are trained and updated based on this error, so that the network can more accurately learn the value of different state-action pairs, and further optimize the decision-making ability of the unmanned aerial vehicle in path planning.

[0107] Step S15, determine the strategy for selecting actions from the action space when the long machine interacts with the flight environment space and the reward function in the training process.

[0108] Wherein, the long machine selects the specific action "strategy" (i.e. decision rule, such as adopting an epsilon-greedy strategy: with a higher probability to select the action currently evaluated by the network as the highest value to utilize the existing experience, and with a lower probability to randomly select an action to explore the unknown environment, balancing "utilization" and "exploration") from the preset discretized action space (such as moving 1 grid in x / y / z axis direction, staying, etc. Enumerated actions) when interacting with the discretized three-dimensional flight environment, and the "reward function" (i.e. feedback mechanism, the core of which is to assign a quantitative "reward" or "punishment" to each step of the long machine's action, for example: giving a positive reward when approaching the target point, giving a large negative reward when colliding with obstacles, giving a small negative reward when staying still for a long time, giving the maximum positive reward when successfully reaching the target point, etc.) used to evaluate the long machine's action in the algorithm training process, and the design of the two directly determines whether the long machine can gradually learn the optimal decision logic for obstacle avoidance and efficient arrival at the target point in the flight environment through training.

[0109] Specifically, the action selection strategy adopted ensures that the long machine has a high autonomous exploration ability, and at the same time ensures that the long machine selects as many actions as possible, guiding the long machine to explore the environment more comprehensively, therefore, the action selection strategy adopted is The strategy determines the action selection of the long machine in learning, which can be specifically described as:

[0110]

[0111] Wherein, is the greedy strategy parameter. Given the state of the unmanned aerial vehicle , the long machine randomly selects an action with a probability of , and selects the action with the maximum value with a probability of .

[0112] The immediate reward function of IDDQN is established, including the action reward function and the environment reward function, and the environment reward is defined to include the target point reward, the environment range reward, the obstacle reward, and the step number reward, so that the action reward function and the final reward function can be described as:

[0113]

[0114]

[0115] Wherein, represents the Euler distance from the current state to the target point, represents the Euler distance from the next state to the target point; is the distance parameter for balancing the distance reward, represents the step number reward value, is the target position of the long machine, For the lead aircraft in the current state of the drone Location, Next status for the lead aircraft Location, For the environmental reward function, Indicates the current number of steps being executed. Indicates environmental constraints. Indicates obstacle constraints. This indicates a range constraint for the number of steps.

[0116] Step S16: Based on the starting position and target position of the lead aircraft, the IDDQN algorithm is used to obtain the initial path of the lead aircraft, and the Bézier curve is used to optimize the initial path to obtain the final path information.

[0117] Given the starting point (flight start point) and target point (flight end point) of the lead aircraft, the IDDQN (Dual Deep Q-Network) algorithm—a reinforcement learning-based decision-making algorithm—is first applied. Leveraging its trained and optimized neural network model (which has learned environmental interaction rules and obstacle avoidance logic through action-value functions, reward functions, etc.), an initial path from the starting point to the target point is searched and generated in a discretized 3D flight environment containing obstacles. (This path is typically formed by connecting discrete nodes of a raster map, and may have abrupt transitions and lack of smoothness, only satisfying the basic feasibility requirement of "reaching the target and avoiding obstacles").

[0118] Subsequently, in order to make the path more in line with the actual flight requirements of the UAV (such as reducing drastic changes in fuselage attitude, reducing energy consumption, and improving flight stability), a parameterized curve tool, Bézier curve, which can generate a smooth and continuous trajectory through key control points, was used to fit and smooth the discrete nodes of the initial path, and finally obtained a continuous, smooth final flight path information that still meets the requirements of obstacle avoidance and accessibility.

[0119] For example, the mathematical expression for a Bézier curve is:

[0120]

[0121] in, For the first Coordinates of control points Let be the order of the Bézier curve. To control the number of points. These are the normalized curve parameters, corresponding to the starting point of the curve. arrive ; for The Bernstein basis functions are:

[0122]

[0123] The objective function of the Bezier curve is obtained by combining the above formulae:

[0124]

[0125] wherein, is a discrete path point, is a parameter value of the i-th discrete point, is a regularization weight parameter.

[0126] In summary, the unmanned aerial vehicle path planning method based on reinforcement learning in the above embodiments of the present application, by acquiring the long machine starting point position, target point position set in the unmanned aerial vehicle formation, and simultaneously determining the number and position of obstacles; determining the unmanned aerial vehicle simulation three-dimensional flight environment according to the number and position of obstacles, discretizing the unmanned aerial vehicle simulation three-dimensional flight environment to establish a grid map; according to the unmanned aerial vehicle simulation three-dimensional flight environment, establishing the unmanned aerial vehicle state, state space and discretized unmanned aerial vehicle action space in the three-dimensional discretized space; the action value function is established by the state and action space mapping relationship, wherein the action value function is used to calculate the time difference error of the experience sample of the unmanned aerial vehicle and simultaneously train and update the network parameters of the IDDQN algorithm; determining the strategy of selecting actions from the action space when the long machine interacts with the flight environment space and the reward function in the training process; based on the long machine starting point position and target point position, the IDDQN algorithm is used to obtain the initial path of the long machine, and the initial path is optimized by using the Bezier curve to obtain the final path information, by dynamically introducing the random path experience sample to the target in the training, and then improving the effect of network parameter training, so as to realize the efficient path planning of the unmanned aerial vehicle in the three-dimensional environment. The problems of insufficient feature interaction and large task interference in the prior art are solved.

[0127] Embodiment Two

[0128] The embodiment also proposes an unmanned aerial vehicle path planning method based on reinforcement learning, which is different from the unmanned aerial vehicle path planning method based on reinforcement learning in embodiment one in that:

[0129] The step of obtaining the initial path of the long machine by using the IDDQN algorithm based on the long machine starting point position and target point position comprises:

[0130] The experience sample of the unmanned aerial vehicle is defined as , the priority corresponding to the sample is initialized as and , and the main network parameter and the target network parameter are initialized as ;

[0131] As​Figure 3 As shown, a Markov process under a given step is performed to obtain a series of experience samples and save them in the initial experience pool. In the later stage of the IDDQN algorithm training, the experience sequence generated by the training is composed into a temporary experience pool

[0132] wherein:

[0133]

[0134] wherein, represents the state after the action to the state of the Markov process, the action after the action indicates whether the current training is finished;

[0135] Define a range of parameters for a greedy strategy

[0136] In the temporary experience pool, a preset genetic algorithm is used to sequentially extract experience sequences in the temporary experience pool as parents, and extract experience sequences different from the parents as mothers. The action set of the parents and the mothers is decoupled, the action set is decoded and binary coded, and then crossover and mutation are performed to obtain the action set of the offspring;

[0137] wherein, represents the binary coding of the action space;

[0138] The obtained offspring action set is optimized by dynamic programming to the action set of the target point, and the corresponding experience sequence sample is obtained based on the continuous Markov process;

[0139] For the obtained experience sequence sample set, the fitness is screened according to the fitness, and the target point experience sequence after screening is updated to obtain a temporary experience pool after one optimization in the temporary experience pool;

[0140] The experience samples in the temporary experience pool are input into the initial experience pool;

[0141] In the process of storing experience into the initial experience pool, according to the priority probability, a given batch of experience samples is sampled in the initial experience pool for network training. The probability of sampling each sample in the experience pool can be represented as:

[0142]

[0143] wherein, is the sample priority;

[0144] ​​​​The importance sampling weight of the sampled experience sample is calculated, so as to define the importance of the sample for subsequent update of the accumulated gradient, and the importance weight of the experience sample is:

[0145]

[0146] wherein, represents the number of existing experience samples in the initial experience pool;

[0147] The time difference error of the sampled experience sample is calculated, which is used for the update of the accumulated gradient with the importance weight and is used for updating the priority of the sample, and the time difference error of the experience sample is represented as:

[0148]

[0149] The priority of the experience sample is updated by using the time difference error of the experience sample:

[0150]

[0151] The accumulated gradient is updated by combining the importance weight of the sample and the time difference error:

[0152]

[0153] The formula , is used to update the main network parameters of the DDQN;

[0154] At intervals of a certain time step, the main network parameter value is updated to the target network by using a soft update method:

[0155]

[0156] wherein, is a soft update parameter;

[0157] The trained main network parameters are used as the training brain of the long machine, and the discretized virtual path information is obtained through a Markov process.

[0158] Further, the step of updating the filtered target point experience sequence to obtain a once-optimized temporary experience pool in the temporary experience pool according to the fitness screening further includes:

[0159] The accumulated return of the experience sequence in the once-optimized experience pool is calculated .

[0160] The priority calculation method is in accordance with the priority calculation method and Obtain the sampling priority of all empirical sequences ;

[0161] according to Sampling probability calculation method and Obtain the sampling probability of the empirical sequence Finally, based on the sampling probability, a given number of empirical sequences are obtained, resulting in a temporary empirical pool after secondary optimization.

[0162] in, Representing an empirical sequence Cumulative returns This represents a very small positive number, the purpose of which is to ensure that the priority of all empirical sequences is greater than 0. Weighting factor When it indicates degradation to uniform sampling, when This means that the sampling probability is entirely based on the priority of the sample, that is, the higher the priority of the sample, the greater the probability of it being sampled;

[0163] Furthermore, the step of dynamically optimizing the obtained child action set into an action set for the target point using dynamic programming, and obtaining the corresponding empirical sequence samples based on a continuous Markov process, includes:

[0164] When solving each subproblem, actions with a target tendency are randomly selected from the action space. By balancing directional guidance and random perturbation, a sequence that combines path approximation optimality and empirical diversity is generated. The solution of each subproblem is represented as an action solution space close to the target point. The problem is solved using the following formula for calculating the distance variation:

[0165]

[0166] in, To change the distance, and These are the distances from the current state to the target point and the distances from the next state to the target point, respectively. When, it indicates that the action to be performed is an action that moves closer to the target point. When, it indicates that the action to be performed is an action that moves away from the target point; if performed in the current state. If none of the actions satisfy the constraints (obstacle constraints and environmental constraints), then the solution space for actions far from the target point is... Select action to .

[0167] In summary, the unmanned aerial vehicle path planning method based on reinforcement learning in the above embodiments of the application, by acquiring the long machine starting point position, target point position set in the unmanned aerial vehicle formation, and simultaneously determining the number and position of obstacles; according to the number and position of obstacles, determine the unmanned aerial vehicle simulation three-dimensional flight environment, discretize the unmanned aerial vehicle simulation three-dimensional flight environment to establish a grid map; according to the unmanned aerial vehicle simulation three-dimensional flight environment, establish the unmanned aerial vehicle state, state space and discretized unmanned aerial vehicle action space in the three-dimensional discretized space; the action value function is established by the state and action space mapping relationship, wherein the action value function is used to calculate the time difference error of the experience sample of the unmanned aerial vehicle and simultaneously train the network parameters of the IDDQN algorithm; determine the strategy of selecting actions from the action space when the long machine interacts with the flight environment space and the reward function in the training process; based on the long machine starting point position and the target point position, the IDDQN algorithm is used to acquire the initial path of the long machine, and the initial path is optimized by using the Bezier curve to obtain the final path information, by dynamically introducing the random path experience sample to the target in the training, and then improving the effect of network parameter training, so as to realize the efficient path planning of the unmanned aerial vehicle in the three-dimensional environment. The problems of insufficient feature interaction and large task interference in the prior art are solved.

[0168] Embodiment three

[0169] Please refer to Figure 4 , which is the unmanned aerial vehicle path planning device based on reinforcement learning proposed in the third embodiment of the application, the device comprises:

[0170] The setting module 100 is used for setting the size of the three-dimensional flight environment of the unmanned aerial vehicle formation;

[0171] The acquisition module 200 is used for acquiring the long machine starting point position, target point position set in the unmanned aerial vehicle formation, and simultaneously determining the number and position of obstacles;

[0172] The discretization module 300 is used for determining the unmanned aerial vehicle simulation three-dimensional flight environment according to the number and position of obstacles, and discretizing the unmanned aerial vehicle simulation three-dimensional flight environment to establish a grid map;

[0173] The establishment module 400 is used for establishing the unmanned aerial vehicle state, state space and discretized unmanned aerial vehicle action space in the three-dimensional discretized space according to the unmanned aerial vehicle simulation three-dimensional flight environment;

[0174] The training module 500 is used for establishing the action value function by the state and action space mapping relationship, wherein the action value function is used to calculate the time difference error of the experience sample of the unmanned aerial vehicle and simultaneously train the network parameters of the IDDQN algorithm;

[0175] The selection module 600 is configured to determine a strategy for selecting an action from the action space when the long machine interacts with the flight environment space and a reward function in the training process.

[0176] The planning module 700 is configured to obtain an initial path of the long machine based on a starting position and a target position of the long machine by using an IDDQN algorithm, and optimize the initial path by using a Bezier curve to obtain final path information.

[0177] The functions or operation steps realized when the above modules are executed are substantially the same as those of the above method embodiments, and thus will not be described here again.

[0178] Embodiment four

[0179] In another aspect, the present application also provides a readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method according to any one of the above embodiments one to two.

[0180] Embodiment five

[0181] In another aspect, the present application also provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of the above embodiments one to two when executing the program.

[0182] The technical features of the above embodiments can be combined in any manner, and for the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0183] Those skilled in the art can understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logical functions, which can be specifically implemented in any computer readable storage medium for use by or in conjunction with an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from an instruction execution system, device or apparatus. For the present specification, the "computer readable storage medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in conjunction with an instruction execution system, device or apparatus, or in conjunction with these instruction execution systems, devices or apparatus.

[0184] More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (electrical, optical, and the like) having a program stored thereon as a signal; a portable computer diskette (magnetic, optical, and the like); a random access memory (RAM), a read-only memory (ROM); an erasable programmable read-only memory (EPROM or Flash memory); optical fiber devices; and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable storage medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via an optical scanner, then compiled, interpreted, or otherwise processed, as necessary, and stored in a computer memory.

[0185] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example, by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques, which are well known in the art, can be used to implement the application: a hybrid of the techniques mentioned above; a combination of one or more of the techniques mentioned above; or one or more other techniques suitable for use in the computer hardware devices described above.

[0186] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples.

[0187] The above-described embodiments only express several implementation manners of the application, which are described in a more specific and detailed manner, but cannot be understood as a limitation on the patent scope of the application. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the application, which are all within the protection scope of the application. Therefore, the patent protection scope of the application should be subject to the appended claims.

Claims

1. A method for unmanned aerial vehicle (UAV) path planning based on reinforcement learning, characterized in that, The method includes: Set the size of the three-dimensional flight environment for the drone formation; Obtain the starting position and target position of the lead drone in the drone formation, and simultaneously determine the number and location of obstacles; The three-dimensional flight environment of the UAV is determined based on the number and location of obstacles, and the three-dimensional flight environment of the UAV simulation is discretized to create a rasterized grid map. Based on the simulated 3D flight environment of the UAV, the UAV state, state space, and discretized UAV action space in the 3D discretized space are established. An action value function is established based on the mapping relationship between the state and action space. The action value function is used to calculate the time difference error of the UAV's empirical samples and to train and update the network parameters of the IDDQN algorithm. Determine the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space, as well as the reward function during training; The initial path of the lead aircraft is obtained by using the IDDQN algorithm based on the starting position and target position of the lead aircraft, and the final path information is obtained by optimizing the initial path using Bézier curves. The process involves establishing the UAV state in a three-dimensional discretized space based on the simulated three-dimensional flight environment of the UAV. s State space S and the discrete action space of unmanned aerial vehicles The steps include: Pick This indicates the current location information of the lead aircraft. Indicates the target location of the lead aircraft. Obstacle information is represented, and the drone's status is represented as follows: The state space of the UAV is represented as ; With the drone as the center point, a regular hexahedron is constructed with an edge length of 2 units. The drone's action space is discretized into 26 directions that the drone can choose from, and the drone's action space is defined as follows: .

2. The UAV path planning method based on reinforcement learning according to claim 1, characterized in that, The steps of determining the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space and the reward function during training include: use The strategy determines the actions the primary machine chooses during learning, specifically: in, Given the drone state, the parameters for the greedy strategy are... The lead aircraft Randomly select an action with a probability of [missing information]. The probability of choosing the action with the highest value; Establish the immediate reward function for IDDQN, which includes an action reward function and an environment reward function. The environment reward function includes a target point reward, an environment range reward, an obstacle reward, and a superstep reward. The action reward function and the final immediate reward function are as follows: in, This represents the Eulerian distance from the current state to the target point. This represents the Eulerian distance from the next state to the target point; The distance parameter is used to balance the distance reward. This represents the step reward value. The target location for the lead aircraft. For the lead aircraft in the current state of the drone Location, Next status for the lead aircraft Location, For the environmental reward function, Indicates the current number of steps being executed. Indicates environmental constraints. Indicates obstacle constraints. This indicates a range constraint for the number of steps.

3. The UAV path planning method based on reinforcement learning according to claim 1, characterized in that, The steps for obtaining the initial path of the lead aircraft using the IDDQN algorithm based on the lead aircraft's starting position and target position include: Define the empirical sample of the drone as The initialization sample priority and cumulative gradient are as follows: and At the same time, initialize the main network parameters and the target network parameters as follows: ; A Markov process with a given step size is performed to obtain a series of experience samples, which are stored in an initial experience pool. In the later stages of training the IDDQN algorithm, the experience sequences generated during training are combined into a temporary experience pool. ; in: ； in, Representing state After the action to state Markov processes, performing actions After Indicate whether this training session has ended; Define a range of parameters for a greedy strategy. ; In the temporary experience pool, a preset genetic algorithm is used to sequentially extract the experience sequences in the temporary experience pool as the parent, and at the same time extract the experience sequences different from the parent as the mother, decouple the action sets of the parent and the mother, decode the action sets and encode them in binary, and then perform crossover and mutation to obtain the action sets of the offspring. in, Binary encoding representing the action space; The obtained child action set is dynamically optimized into the action set of the target point, and the corresponding empirical sequence sample is obtained based on the continuous Markov process. For the obtained experience sequence sample set, after filtering according to fitness, the filtered target point experience sequences are updated to obtain a temporary experience pool with one optimization. Input the experience samples from the temporary experience pool into the initial experience pool; During the process of storing experience into the initial experience pool, experience samples in a given batch are sampled from the initial experience pool according to priority probabilities for network training. The probability of each sample being sampled in the experience pool is expressed as: in, Prioritize samples; Calculate the importance sampling weights of the sampled empirical samples to define the importance of the sample, which is used for subsequent gradient updates. The importance weights of the empirical samples are: in, This represents the number of existing experience samples in the initial experience pool; The temporal difference error of the sampled empirical sample is calculated. An auxiliary importance weight is used to update the cumulative gradient and simultaneously to update the priority of the sample. The temporal difference error of the empirical sample is expressed as: The priority of the empirical sample is updated using the temporal difference error of the empirical sample: The cumulative gradient is updated by combining the importance weights of the samples and the temporal difference error: Using formula , Update the main network parameters of DDQN to the learning rate; At regular time intervals, a soft update method is used to update the main network parameter values ​​to the target network: in, For soft update parameters; The trained main network parameters As the training brain of the lead aircraft, it obtains discretized virtual path information through a Markov process. .

4. The UAV path planning method based on reinforcement learning according to claim 3, characterized in that, The step of updating the temporary experience pool by filtering the target points according to fitness and obtaining an optimized temporary experience pool also includes: Calculate the cumulative reward of an experience sequence in the experience pool once. ; according to Priority calculation method and Obtain the sampling priority of all empirical sequences ; according to Sampling probability calculation method and Obtain the sampling probability of the empirical sequence Finally, based on the sampling probability, a given number of empirical sequences are obtained, resulting in a temporary empirical pool after secondary optimization. in, Representing an empirical sequence Cumulative returns This represents a positive number, the purpose of which is to ensure that the priority of all empirical sequences is greater than 0. Weighting factor When it indicates degradation to uniform sampling, when The time indicates that the sampling probability is entirely based on the priority of the sample, that is, the higher the priority of the sample, the greater the probability of the sample being sampled.

5. The UAV path planning method based on reinforcement learning according to claim 4, characterized in that, The steps of dynamically optimizing the obtained child action set into the action set of the target point using dynamic programming, and obtaining the corresponding empirical sequence samples based on a continuous Markov process include: When solving each subproblem, actions with a target tendency are randomly selected from the action space. By balancing directional guidance and random perturbation, a sequence that combines path approximation optimality and empirical diversity is generated. The solution of each subproblem is represented as an action solution space close to the target point. The problem is solved using the following formula for calculating the distance variation: in, To change the distance, and These are the distances from the current state to the target point and the distances from the next state to the target point, respectively. When, it indicates that the action to be performed is an action that moves closer to the target point. When this occurs, it indicates that the action being performed is an action that moves away from the target point; If executed in the current state If none of the actions satisfy the constraints, then start from the action solution space far from the target point. Select action to .

6. The UAV path planning method based on reinforcement learning according to claim 5, characterized in that, The step of optimizing the initial path using Bézier curves to obtain the final path information includes: The initial path is optimized using Bézier curves, and the optimized path information is represented as follows: The mathematical expression for a Bézier curve is: The objective function of the Bézier curve is: for The Bernstein basis functions are: in, For the first Coordinates of control points Let be the order of the Bézier curve. To control the number of points, These are the normalized curve parameters, corresponding to the starting point of the curve. arrive , For discrete path points, For the first Parameter values ​​at discrete points These are the regularization weight parameters.

7. A UAV path planning device based on reinforcement learning, characterized in that, The apparatus for implementing the UAV path planning method based on reinforcement learning as described in any one of claims 1 to 6, the apparatus comprising: The settings module is used to set the size of the three-dimensional flight environment for drone formations; The acquisition module is used to acquire the starting position and target position of the lead drone in the drone formation, as well as to determine the number and position of obstacles. The discrete module is used to determine the 3D flight environment of the UAV simulation based on the number and location of obstacles, and to discretize the 3D flight environment of the UAV simulation to create a rasterized grid map. A module is established to create the UAV state, state space, and discretized UAV action space in a three-dimensional discretized space based on the simulated three-dimensional flight environment of the UAV. The training module is used to establish the action value function based on the mapping relationship between the state and action space. The action value function is used to calculate the time difference error of the UAV's empirical samples and to train and update the network parameters of the IDDQN algorithm. The selection module is used to determine the strategy for selecting actions from the action space when the lead aircraft interacts with the flight environment space, as well as the reward function during training. The planning module is used to obtain the initial path of the lead aircraft based on the starting position and target position of the lead aircraft using the IDDQN algorithm, and to optimize the initial path using Bézier curves to obtain the final path information.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the program, implements the steps of the method as described in any one of claims 1 to 6.

Images