An improved DQN algorithm-based item delivery unmanned aerial vehicle cluster path planning method

Abstract

The application discloses an improved DQN algorithm-based item delivery unmanned aerial vehicle cluster path planning method, which comprises the following steps: step 1, task allocation is performed on the unmanned aerial vehicle cluster and a delivery destination; step 2, an initial state of an environment of the improved DQN algorithm is determined, and a representation mode of state, action and policy elements of a path search problem of the unmanned aerial vehicle is set; step 3, a neural network structure of the improved DQN algorithm is set according to state and action dimensions, and an improved measure of monotonically decreasing steps is proposed; step 4, an initial path is planned by using the improved DQN algorithm, the starting point of the unmanned aerial vehicle and the delivery destination are made to be one-to-one corresponding according to a task allocation result, and path planning is performed; step 5, dangerous points in the path are adjusted; and step 6, unmanned aerial vehicle path optimization processing is performed; and the application can obtain a path planning method with simple model and rapid planning.

Description

Technical Field

[0001] The present invention belongs to the technical field of drone swarm decision-making and planning, and specifically relates to a method for item delivery drone swarm trajectory planning based on an improved DQN algorithm. Background Art

[0002] Food delivery has always been a major concern for food delivery platforms. Using drones offers advantages for faster delivery. However, drone trajectory planning typically targets unmanned scenarios with sparse obstacles, minimal interference, and the absence of large buildings. However, trajectory planning for drone swarms in densely populated urban areas remains challenging.

[0003] Patent CN201910437021.2 discloses a potential function reward DQN (DQN)-based drone path planning method in a continuous state with unknown environmental information. The method first establishes the drone's state space in the environment. This state space is a continuous state space containing an infinite number of drone states. Secondly, 360 degrees is divided into several angles n as the drone's heading angles to establish the drone's action space. Next, the potential function reward of the target on the drone and the potential function reward of the obstacle on the drone are calculated and superimposed as the drone's total potential function reward. The total potential function reward of the drone is then used to train the Q estimation network for path planning. Finally, the trained Q estimation network is used to plan the drone's path in a continuous state with unknown environmental information. This method mainly solves the drone's path planning problem in the absence of an environmental model and meets the drone's requirement for the continuity of the environmental state during mission execution. The potential function reward speeds up the drone's path planning and has good applicability.

[0004] Patent CN202110591320.9 discloses a DQN-based drone path planning method. The method first generates a training set using a grid method. A reward function for obstacles is set, and the Q value for reinforcement learning is initialized. A convolutional neural network within the DQN is constructed and trained. An image segmentation network, Mask R-NN, is built within the DQN and tested using remote sensing images. The grid map output by the Mask R-CNN network is input into the trained model, ultimately outputting a remote sensing image of the vehicle successfully navigating the obstacles and reaching the destination. This invention implements path planning using the DQN algorithm within real-world images.

[0005] The aforementioned inventions are all based on the application of DQN in path planning. However, the network architecture they employ uses convolutional layers for feature extraction before using fully connected layers to calculate the action strategy. This increases DQN network training time. Furthermore, the reward function settings in these inventions are relatively complex. Summary of the Invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a trajectory planning method for a swarm of drones delivering items based on an improved DQN algorithm, in order to simplify the planning model, reduce the planning time, and ensure better performance while improving planning efficiency.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A method for swarm trajectory planning of item delivery drones based on an improved DQN algorithm includes the following steps:

[0009] Step 1: Assign tasks to the drone cluster and delivery destinations. When the number of delivery destinations is equal to the number of drones, an auction algorithm based on drones is used. When the number of delivery destinations is greater than the number of drones, an auction algorithm based on delivery destinations is used.

[0010] Step 2: After the task assignment in step 1, the correspondence between the drone’s starting point and the delivery destination is obtained, the initial state of the environment for the improved DQN algorithm is determined, and the representation of the state, action, and strategy elements of the drone’s pathfinding problem is set;

[0011] Step 3: Set reward functions for different drone states and improve the neural network structure used by the DQN algorithm. After obtaining the state and action representation, the possible states of the drone are known. The reward function is set for the possible states. The neural network structure of the improved DQN algorithm is set according to the state and action dimensions. An improvement measure of monotonically decreasing step size is proposed to speed up training.

[0012] Step 4: Use the improved DQN algorithm to plan the initial trajectory. According to the task allocation results, the starting point of the drone and the delivery destination are matched one by one to perform trajectory planning.

[0013] Step 5: Adjust the dangerous points in the track. Traverse the drone's track points and adjust the positions of the points whose distance to obstacles is less than the safe distance so that the distance to the surrounding obstacles is greater than the safe distance.

[0014] Step 6: Optimize the drone trajectory. If there are no dynamic obstacles or conflict points in the scene, use the Bezier curve for smoothing to obtain the final trajectory. If there are dynamic obstacles or conflict points in the scene, use the particle swarm algorithm for local trajectory planning.

[0015] Furthermore, in step 1, tasks are assigned to the starting points and delivery destinations of each drone. The task assignment adopts an auction algorithm, and the bidding is based on the straight-line distance between the starting point and the delivery destination of the drone.

[0016] Different auction algorithms are used for different situations:

[0017] When the number of drones and delivery destinations is the same, an auction algorithm based on drones is used. That is, drones are auctioned as items and delivery destinations are bidders. It is assumed that the priority of drones decreases as the number increases. That is, when there are N drones, drone No. 1 has the highest priority and drone No. N has the lowest priority. The bidding starts from drone No. 1 to drone No. N in turn. This ensures that drones with higher priorities are delivered to the delivery destinations that are closest in a straight line.

[0018] For the case where the number of delivery destinations is greater than the number of drones, an auction algorithm based on delivery destinations is adopted. That is, the delivery destinations are used as auction items and drones are bidders. At the same time, the number of delivery destinations that a drone can auction is limited to L. The value of L is determined by the formula LN>M, where N represents the number of drones and M represents the number of delivery destinations.

[0019] min

[0020] Auctioning based on delivery purpose can ensure that there is a drone going to each delivery destination. For the case where a drone has to go to multiple destinations, the problem is further modeled as a traveling salesman problem, and then the particle swarm algorithm is used to determine the order of visiting each delivery destination.

[0021] The traveling salesman problem model is that a salesman needs to go to different locations to sell goods and determine the order of visiting each location so that the salesman's path is the shortest. That is, a drone needs to determine the order of visiting each delivery destination to reach multiple delivery points. The particle swarm algorithm is used to determine the order of visiting each delivery destination. The optimal solution to this problem is a numbering sequence of starting points and task points. Corresponding to the formula of the particle swarm algorithm, the position of the particle is this sequence, and the speed of the particle is an ordered exchange sequence, which can change the order of traversing task points. The exchange sequence refers to the previous exchange sequence, the individual's historical optimal numbering sequence and the optimal numbering sequence of the current generation population are updated, and the numbering sequence is updated according to the exchange sequence. The fitness function is positively correlated with the sum of the straight-line distances.

[0022] Furthermore, in step 2, the trajectory planning problem is modeled as a Markov decision process. The Markov decision process is a typical problem model in reinforcement learning. The model includes state, action, strategy, reward and return. The state refers to the state of the agent's current environment, represented by S. The Markov decision process has an important property, namely the Markov property, that is, the state at the next moment is only related to the state at the current moment, which is expressed as:

[0023] P[S t+1 |S t ]=P[St+1 |S t ,...,S1]

[0024] The above formula S t+1 represents the state at time t+1, S t Represents the state at time t. The left side of the formula represents the probability of the state at time t+1 based on the condition that the state at time t occurs. The right side of the formula represents the probability of the state at time t+1 based on the condition that all past states occur.

[0025] The set of all states of the environment corresponds to the state space of the model. The state space is discrete or continuous. The agent action is represented by a, which describes the agent's behavior. The set of agent actions constitutes the action space A. The action space is continuous or discrete. The strategy π is the probability function of the agent taking the corresponding action in the corresponding state. The reward r is the immediate feedback from the environment after the agent performs a certain action. The reward is positive or negative.

[0026] The reward G is the cumulative value of the reward in a round, expressed as:

[0027]

[0028] γ is a discount factor, which indicates the importance of future rewards to the current state's cumulative rewards. It ranges from 0 to 1. A larger γ value indicates more attention to future rewards. k is the increment at the time level.

[0029] There are two value functions used to optimize strategies in reinforcement learning. The state value function is used to evaluate the expected future rewards under the current state. The expression is:

[0030] v π (s)=E[G t |S t =s] = E[r t +γv(S t+1 )|S t =s]

[0031] In the formula, v π (s t ) represents the state value under the random strategy π, s t Represents the state at time t, that is, the current state, G t Represents the cumulative return at time t. The state value is equal to the reward of the currently executed action plus the discount rate multiplied by the state value at the next moment. Under the current state, the average value is the average of these returns based on the probability of selecting different actions according to the random strategy, which produces different returns. The state value reflects whether the current state has high value for the future;

[0032] The action-state value function is used to evaluate the t The expected future reward of taking this action is expressed as:

[0033] Q π (s t ,a t )=E[r t +γQ π (s t+1 ,a t+1 )|s,a];

[0034] where Q π (s t ,a t ) means that under the strategy π, the state s t , action a t The corresponding action-state value function, r t is the reward at the current moment, γ is the reward discount factor, and the strategy π is the probability function of the agent taking the corresponding action in the corresponding state. The strategy is usually optimized to maximize the return, and is optimized to maximize the action-state value function or state function;

[0035] For the UAV trajectory planning problem, the UAV corresponds to a single agent in the Markov decision process. The environment is modeled based on real terrain images, and the target scene is modeled as a k*k grid map. The white area represents the feasible area of ​​the UAV cluster, and the black area represents the obstacle area. The scene information is represented by a k*k two-dimensional matrix, where 0 represents the feasible area, 1 represents the obstacle area, 2 represents the current location of the UAV, 3 represents the starting point of the UAV, and 4 represents the end point of the UAV, that is, the delivery destination. The state is the two-dimensional matrix corresponding to the entire map. The UAV has 8 actions, which are the action of walking to the nearest discrete point in 8 equally divided directions of 360 degrees. The strategy uses the ε-greedy strategy.

[0036]

[0037] Compared with the greedy strategy, the ε-greedy strategy has a certain probability of randomly selecting actions, which can avoid overfitting and increase the exploratory nature of the drone's pathfinding. ε is a linearly increasing quantity, and the formula is as follows:

[0038]

[0039] Furthermore, in step 3, the reward function is set up so that a large positive reward is provided when the drone reaches the target point, and a negative reward is provided when the drone hits an obstacle, which is called a penalty. However, since there are only rewards for reaching special points, the drone's pathfinding is still slow. A two-way reward is also provided for the intermediate pathfinding process. The specific expression of the entire reward function is as follows:

[0040]

[0041] The expression shows that hitting an obstacle gives -2, exceeding the boundary gives -5, and reaching the target point gives 10. The remaining positions are calculated in a bidirectional manner. d1 represents the distance from the current point to the target point, d2 represents the distance from the next point to the target point, d3 represents the distance from the current point to the starting point, d4 represents the distance from the current point to the next point, and d5 represents the distance from the next point to the starting point. Therefore, d1-d2 indicates whether the drone is approaching the target point, and d5-d4-d3 indicates whether the drone is moving away from the starting point.

[0042] In step 3, after determining the reward function, the reward values ​​for the drone in different states in the scene are obtained. To complete the training of the improved DQN pathfinding model, it is necessary to design an improved DQN neural network structure based on the state and action dimensions determined in the previous step, and then begin pathfinding training. The Q network corresponding to the improved DQN algorithm consists of two fully connected layers. The input layer is the vector corresponding to the state matrix. The first fully connected layer uses 50 neurons, the second fully connected layer uses 20 neurons, and the output layer contains the evaluation values ​​of the eight actions. Compared with convolutional neural networks, this fully connected neural network has fewer parameters and converges faster, making it more advantageous for solving pathfinding problems. A monotonically decreasing step size improvement is used to improve the convergence speed of pathfinding. The traditional DQN algorithm converges slowly during trajectory planning and cannot quickly obtain a trajectory. With the monotonically decreasing step size of the drone, each step can only reach the nearest integer point in any of the eight directions. After the improvement, the drone's initial step size is maxstep, which decreases with the number of iterations. When it reaches maxstep, the step size decreases. During a round, the step size is reduced to one unit.

[0043]

[0044] This improvement can help drones be more exploratory in the early stages, enrich the sample types in the experience pool, help drones understand their surroundings more quickly, and find the shortest path more quickly.

[0045] Furthermore, in step 4, the starting point of the drone is randomly set in the scene, the delivery destination is set, and the track planning is performed. During the iteration process, the drone selects an action and completes the position adjustment. When it encounters an obstacle, hits a boundary, or reaches the destination, this round ends; the relevant information of this round is saved in the experience pool, and a new round begins; through continuous iteration and learning from the experience of the experience pool, the drone can find discrete track points from the starting point to the delivery destination; the track obtained at this time is composed of some discrete track points and needs to be further optimized.

[0046] Furthermore, in step 5, the position of the dangerous waypoint is adjusted, and the track point whose distance to the surrounding obstacles is less than the safe distance is adjusted so that it is in a safe position, and the safe distance is set to 1.5 distance units.

[0047] Furthermore, in step 6, the track obtained in step 5 is optimized; it is determined whether there are dynamic obstacles in the scene or whether there are conflict points in the planned drone tracks. If neither of the two situations exists, the Bezier curve is used for smoothing, and finally a smooth, collision-free drone cluster track is obtained; if any of the above situations exists, a local target point is selected from the track generated in step 5, the particle swarm algorithm is used for local track planning, and the local track is smoothed with the Bezier curve.

[0048] The corresponding formula of the Bezier curve is as follows:

[0049]

[0050] P i Represents the control point of a Bezier curve. B(t) is the corresponding point on the Bezier curve, t is the time parameter, and n is the order of the Bezier curve.

[0051] Beneficial effects of the present invention:

[0052] The fully connected network structure used in the improved DQN algorithm simplifies the model while ensuring better performance of the trajectory planning algorithm.

[0053] The proposed monotonically decreasing step size can enrich the sample types in the experience pool, accelerate drone learning, improve convergence speed, and reduce time.

[0054] The bidirectional intermediate reward effectively solves the slow convergence problem of the DQN algorithm in trajectory planning caused by sparse rewards.

[0055] Track optimization processing can effectively improve the safety of UAV tracks and ensure the safety of UAV flight. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of a method for trajectory planning of a cluster of drones delivering items based on an improved DQN algorithm of the present invention.

[0057] Figure 2 This is an action distribution diagram of the trajectory planning method for a cluster of drones delivering items based on an improved DQN algorithm of the present invention.

[0058] Figure 3 This is a network structure diagram of an improved DQN algorithm for a method for cluster trajectory planning of item delivery drones based on the improved DQN algorithm of the present invention.

[0059] Figure 4 This is a schematic diagram of danger point adjustment of a method for swarm trajectory planning of item delivery drones based on an improved DQN algorithm of the present invention.

[0060] Figure 5 This is a trajectory planning result diagram of a cluster trajectory planning method for item delivery drones based on an improved DQN algorithm in a static dense obstacle scenario.

[0061] Figure 6 This is a trajectory planning result diagram of a cluster trajectory planning method for item delivery drones based on an improved DQN algorithm in a dynamic and dense obstacle scenario. DETAILED DESCRIPTION

[0062] The present invention will be described in further detail below with reference to the accompanying drawings.

[0063] like Figure 1 As shown: Step 1: Assign tasks to the starting point and delivery destination of each drone. Here, the task assignment adopts an auction algorithm, and the basis for bidding is determined according to the straight-line distance between the starting point of the drone and the delivery destination. Different auction algorithms are adopted for different situations. For the case where the number of drones and the number of delivery destinations are the same, an auction algorithm with drones as the main body is adopted, that is, drones are used as auction items, and delivery destinations are used as bidders. It is assumed that the priority of drones decreases as the number increases, that is, when there are five drones, drone No. 1 has the highest priority and drone No. 5 has the lowest priority. Start bidding from drones No. 1 to 5 in sequence. This ensures that drones with high priorities can obtain delivery destinations with a short straight-line distance. For the case where the number of delivery destinations is greater than the number of drones, an auction algorithm with delivery destinations as the main body is adopted, that is, delivery destinations are used as auction items, drones are used as bidders, and at the same time, the drone is limited to auctioning a maximum of L delivery destinations, where the value of L is based on the formula Determine, where N represents the number of drones and M represents the number of delivery destinations. This ensures that there is a drone for each delivery destination. In the allocation result, some drones are responsible for multiple task points. It is necessary to consider the order in which tasks are executed to minimize the flight path and save fuel. At this time, for drones with multiple delivery destinations to visit, the order in which the delivery destinations are traversed can be modeled as a traveling salesman problem. The traveling salesman problem model is that a product salesman needs to go to different locations to sell products, and the order of visiting each location is determined so that the salesman's path is the shortest. Applied to the present invention, if a drone needs to reach multiple delivery points, it is necessary to determine the order in which to visit each delivery destination. Here, a particle swarm algorithm is used to determine the order in which to visit each delivery destination. The optimal solution to this problem is a numbered sequence of starting points and task points. Corresponding to the formula of the particle swarm algorithm, the position of the particle is this sequence, and the speed of the particle is an ordered exchange sequence, which can change the order of traversing the task points. The exchange sequence refers to the previous exchange sequence, the individual historical optimal numbering sequence and the optimal numbering sequence of the current generation population for update, and the numbering sequence is updated according to the exchange sequence. The fitness function is positively correlated with the sum of the straight-line distances.

[0064] In the embodiment of the present invention, the number of drones N is set to 5, and the number of delivery destinations is 5 or 10, and tasks are assigned in different ways respectively. The starting position of the drones and the delivery destination positions are determined by random sampling.

[0065] Step 2: Model the trajectory planning problem as a Markov decision process. After the task assignment in step 1, the correspondence between the drone's starting point and the delivery destination can be obtained. This allows us to determine the initial state of the environment for the improved DQN algorithm and set the representation of the state, action, strategy, and other elements of the drone's pathfinding problem.

[0066] State refers to the state of the agent's current environment. It is usually represented by S. The Markov decision process has an important property, namely the Markov property, which means that the state at the next moment is only related to the state at the current moment. The formula is expressed as:

[0067] P[S t+1 |S t ]=P[S t+1 |S t ,...,S1]

[0068] The above formula S t+1 represents the state at time t+1, S t Represents the state at time t. The left side of the formula represents the probability of the state at time t+1 based on the condition that the state at time t occurs. The right side of the formula represents the probability of the state at time t+1 based on the condition that all past states occur.

[0069] The set of all states of the environment corresponds to the state space of the model. The state space can be discrete or continuous. The agent action is represented by a, which is the description of the agent's behavior. The set of all possible actions of the agent constitutes the action space A. The action space can be continuous or discrete. The strategy π is the probability function of the agent taking the corresponding action in the corresponding state. The reward r is the immediate feedback from the environment after the agent performs a certain action. The reward can be positive or negative.

[0070] The reward G is the cumulative value of the reward in a round, expressed as:

[0071]

[0072] γ is a discount factor, which indicates the importance of future rewards to the current state's accumulated rewards. It ranges from 0 to 1. A larger γ value indicates greater emphasis on future rewards. k is the time increment.

[0073] There are two value functions used to optimize strategies in reinforcement learning. The state value function is used to evaluate the expected future rewards under the current state. The expression is:

[0074] v π (s t )=E[G t |S=s t ]=E[r t +γv(s t+1 )|S=s t ]

[0075] In the formula, v π (s t ) represents the state value under the random strategy π, s t Represents the state at time t, that is, the current state, G t Represents the cumulative reward at time t. The state value is equal to the reward for the currently executed action plus the discount rate multiplied by the state value at the next moment. In the current state, the average value is the average of the rewards generated by the probabilities of selecting different actions based on the stochastic strategy. The state value reflects whether the current state has high value in the future.

[0076] The action-state value function is used to evaluate the t The expected future reward of taking this action is expressed as:

[0077] Q π (s t ,a t )=E[r t +γQ π (s t+1a t+1 )|s,a]

[0078] where Q π (s t ,a t ) represents the action state value function of state s t , action a t under policy π. r t is the reward at the current time, and γ is the reward discount factor. The policy π is the probability function of the agent taking the corresponding action under the corresponding state. The policy is usually optimized for maximum return, which can be optimized for maximum action state value function or state function.

[0079] Therefore, for the unmanned aerial vehicle path planning problem, the unmanned aerial vehicle corresponds to a single agent in the Markov decision process. The environment is modeled according to the real terrain picture, and the target scene is modeled as a k*k grid picture, and the white area represents the feasible area of the unmanned aerial vehicle cluster, and the black area represents the obstacle area; so as to represent the scene information with a k*k two-dimensional matrix, and different numbers distinguish the starting point, the delivery destination, the obstacle, the unmanned aerial vehicle and the like; the state is the two-dimensional matrix corresponding to the entire map. The unmanned aerial vehicle has 8 actions, and the policy uses the ε-greedy strategy.

[0080]

[0081] Compared with the greedy strategy, the ε-greedy strategy has a certain probability of randomly selecting actions, which can effectively avoid overfitting and increase the exploratory nature of the unmanned aerial vehicle pathfinding. Wherein ε is a linearly increasing quantity, and the formula is as follows:

[0082]

[0083] In this way, the unmanned aerial vehicle can have good exploratory nature in the early stage and good utilization in the later stage.

[0084] In the embodiment of the application, the size of the scene is 50*50, the state is set as a one-dimensional vector corresponding to a two-dimensional matrix, and the size is 1*2500. Among them, the numbers 0 in the vector represent the feasible area, 1 represents the obstacle area, 2 represents the current position of the unmanned aerial vehicle, 3 represents the starting point of the unmanned aerial vehicle, and 4 represents the delivery destination; the action is mainly divided into 8 actions from the direction angle, and the 8 actions are distributed as shown in Figure 2 , respectively, west, east, north, south, southeast, northeast, northwest, and southwest, and the strategy adopts the ε-greedy strategy, and εmax Set to 0.99, ε min Set to 0 and δ to 0.001.

[0085] Step 3: Set the reward function and network structure. In reinforcement learning, the goal of the agent is to achieve a high reward in each round of interaction with the environment. In other words, the cumulative reward per round must be maximized. Therefore, it is necessary to effectively integrate the problem being solved with the agent's interaction mechanism. The maximum cumulative reward for the agent corresponds to the drone finding a short, collision-free path in this invention. Therefore, to incentivize the drone to find the delivery destination, we provide a large positive reward for reaching the destination. A negative reward is given for colliding with an obstacle, helping the drone learn to avoid obstacles and reducing the number of actions that receive negative rewards. Many methods for finding paths at discrete points only consider actions in these special states. They fail to consider actions in other states where the drone has not reached the destination or encountered obstacles. This results in a high degree of randomness in the drone's search for the target point, increases the number of steps required to find the target point, and increases the time required for each round. This invention incorporates a two-way intermediate reward to help the drone achieve better guidance during pathfinding and find the target point more quickly. This two-way intermediate reward consists of two parts. The first part is the distance from the current position to the target point minus the distance from the next moment to the target point. If it is positive, it means the drone is approaching the target point, and if it is negative, it means the drone is moving away from the target point. The second part is the distance from the next moment to the starting point minus the distance from the current position to the starting point, minus the distance from the current position to the next moment. This indicates that the drone is moving away from the starting point. If it is negative, it indicates that the drone is moving towards the starting point. The specific formula is as follows:

[0086]

[0087] The reward function expression shows that hitting an obstacle gives -2, going beyond the boundary gives -5, reaching the target point gives 10, and the remaining positions are calculated in a bidirectional manner.

[0088] Where d1 represents the distance from the current point to the target point, and the expression is:

[0089]

[0090] Where d2 represents the distance from the next moment to the target point, and the expression is:

[0091]

[0092] Where d3 represents the distance from the current point to the starting point, and the expression is:

[0093]

[0094] Where d4 represents the distance from the current point to the next point in time, and the expression is:

[0095]

[0096] Where d5 represents the distance from the next moment to the starting point, and the expression is:

[0097]

[0098] In general, d1-d2 indicates whether the drone is approaching the target point, and d5-d4-d3 indicates whether the drone is moving away from the starting point.

[0099] Set the neural network structure. After obtaining the representation of state, action, etc., the possible states of the drone can be known, and reward functions can be set for these states. The neural network structure of the improved DQN algorithm is set according to the state and action dimensions. The network used by the DQN algorithm is usually a convolutional neural network, which requires the previous convolution layer to specify the uniform specifications of the input image. At the same time, using the convolution layer to extract features will also take a certain amount of training time and lose some features. In order to simplify the model and increase the training speed in the present invention, a fully connected neural network is used. This network has only fully connected layers. Compared with convolutional neural networks, fully connected networks have fewer parameters and faster convergence, and are more advantageous in pathfinding problems.

[0100] In the embodiment of the present invention, the proportional factor α in the bidirectional reward in the reward function is set to 0.6. For the pathfinding problem in this paper, two layers of fully connected layers are used. First, the input layer is the vector 1*2500 corresponding to the state matrix. The first fully connected layer uses 50 neurons, the second fully connected layer uses 20 neurons, and the third layer is the action output layer, which outputs the evaluation values ​​corresponding to 8 actions. The network structure used in this invention can be seen in detail. Figure 3 .

[0101] Improvement measures for monotonically decreasing step size. The traditional DQN algorithm converges slowly during trajectory planning and cannot quickly obtain a trajectory. This invention proposes a monotonically decreasing step size improvement measure for drones to increase the convergence speed and reduce the time spent. That is, each step of the original one unit is changed to the initial maxstep of the drone, which decreases with the increase of the number of iterations. When it reaches During a round, the step size is reduced to one unit.

[0102]

[0103] In the embodiment of the application, the maximum number of episodes max episode is set to 50000, and the starting point of the unmanned aerial vehicle and the delivery destination are sampled from the points with a value of 0 in the two-dimensional matrix, i.e. the non-obstacle points. The maxstep in the improvement measure is set to 3.

[0104] Step 4: Randomly set the starting point of the unmanned aerial vehicle and the delivery destination in the scene, and perform path planning. In the iteration process, the unmanned aerial vehicle selects an action, completes position adjustment, and ends the episode when it encounters an obstacle, a boundary or reaches the destination. When the delivery destination is reached, the path points passed through in this episode are saved as the path. The relevant information of this episode is saved to the experience pool, and a new episode is started. Through continuous iteration and learning from the experience in the experience pool, the cycle is ended when the maximum number of episodes is reached. The shortest path is found from the previously saved multiple paths and output. The path obtained at this time is composed of discrete path points and needs to be further optimized.

[0105] Step 5: Adjust the dangerous points in the path. The initial path planned by the improved DQN algorithm in the above step is composed of countless integer points. The path is traversed, and the position of the path point is adjusted if the distance between the path point and the surrounding obstacle is less than the safety distance. The position is adjusted according to the inverse vector of the vector from the current point to the centroid of the surrounding obstacle. For details, see Figure 4 .

[0106] In the embodiment of the application, the safety distance is set to 1.5 distance units.

[0107] Step 6: Optimize the path obtained in step 5. Determine whether there are dynamic obstacles in the scene or whether there are conflict points in the planned paths of the unmanned aerial vehicles. The conflict point is a point at which multiple unmanned aerial vehicles arrive at the same location at the same time. If neither of the two conditions is met, a Bezier curve is used for smoothing processing, and finally a smooth and collision-free path of the unmanned aerial vehicle cluster is obtained. The corresponding formula of the Bezier curve is as follows:

[0108]

[0109] P i P represents the control points of the Bezier curve. B(t) is a point on the corresponding Bezier curve, t is a time parameter, and n represents the order of the Bezier curve. represents the number of combinations of i numbers in n numbers.

[0110] According to whether the number of delivery destinations is the same as the number of unmanned aerial vehicles, two results can be obtained, Figure 5 (a) represents the path planning condition when the number of unmanned aerial vehicles is equal to the number of delivery destinations. Figure 5 (b) represents the path planning condition when the number of unmanned aerial vehicles is less than the number of delivery destinations.

[0111] If any of the above situations exist, a local target point is selected from the track generated in step 5, and the particle swarm algorithm is used to plan and smooth the local track. The selection of the local target point is based on the distance from the track point to the dynamic obstacle. When the distance between the track point and the dynamic obstacle is less than the threat distance, the particle swarm algorithm is used to replan from this point to enable the drone to complete dynamic obstacle avoidance.

[0112] In an embodiment of the present invention, the control point P of the Bezier curve i The trained model is used for multi-UAV trajectory planning. In the case of dynamic obstacles, the particle swarm algorithm is used for local trajectory planning, where the threat distance is set to 5 distance units. Figure 6 shown. Figure 6 The dotted lines around the single square in (b) represent dynamic obstacles and their activity areas. It can be seen from the figure that this method can successfully avoid the activity range of dynamic obstacles, and the UAV flight safety is relatively high.

Claims

1. A method for swarm trajectory planning of drones for item delivery based on an improved DQN algorithm, characterized by: The following steps are included: Step 1: Assign tasks to drone clusters and delivery destinations; When the number of delivery destinations is equal to the number of drones, the auction algorithm with drones as the main body is adopted. When the number of delivery destinations is greater than the number of drones, the auction algorithm with delivery destinations as the main body is adopted. Step 2: After the task assignment in step 1, the correspondence between the drone’s starting point and the delivery destination is obtained, the initial state of the environment for the improved DQN algorithm is determined, and the representation of the state, action, and strategy elements of the drone’s pathfinding problem is set; Step 3: Set reward functions for different states of the drone and improve the neural network structure used by the DQN algorithm. After obtaining the state and action representation, the existing state of the drone is known. The reward function is set for the existing state. The neural network structure of the improved DQN algorithm is set according to the state and action dimensions. An improvement measure of monotonically decreasing step size is proposed to speed up training. Step 4: Use the improved DQN algorithm to plan the initial trajectory. According to the task allocation results, the starting point of the drone and the delivery destination are matched one by one to perform trajectory planning. Step 5: Adjust the dangerous points in the track. Traverse the drone's track points and adjust the positions of the points whose distance to obstacles is less than the safe distance so that the distance to the surrounding obstacles is greater than the safe distance. Step 6: Optimize the drone's trajectory. If there are no dynamic obstacles or conflict points in the scene, use the Bezier curve for smoothing to obtain the final trajectory. If there are dynamic obstacles or conflict points in the scene, use the particle swarm algorithm for local trajectory planning. In step 3, after determining the reward function, the reward value of the drone in different states in the scene is obtained. The Q network corresponding to the improved DQN algorithm consists of two fully connected layers. First, the input layer is the vector corresponding to the state matrix. The first fully connected layer uses 50 neurons, the second fully connected layer uses 20 neurons, and the output layer is the evaluation value of 8 actions. A monotonically decreasing drone step size improvement measure is proposed, which can effectively accelerate the convergence of DQN. The original DQN algorithm action stipulates that each step can only go to the nearest integer point along any direction of the 8 directions. After the improvement, the initial step size of the drone is maxstep, which decreases as the number of iterations increases. When it reaches During the round, the step size is reduced to one unit; 2. The method for swarm trajectory planning of drones for item delivery based on the improved DQN algorithm according to claim 1 is characterized in that: In step 1, tasks are assigned to the starting points and delivery destinations of each drone. The task assignment adopts an auction algorithm, and the basis for bidding is determined according to the straight-line distance between the starting point and the delivery destination of the drone; Different auction algorithms are used for different situations: When the number of drones and delivery destinations is the same, an auction algorithm based on drones is used. That is, drones are auctioned items and delivery destinations are bidders. The priority of drones is set to decrease as the number increases. That is, when there are N drones, drone No. 1 has the highest priority and drone No. N has the lowest priority. The bidding starts from drone No. 1 to drone No. N in turn. This ensures that drones with high priority get delivery destinations that are close in a straight line. For the case where the number of delivery destinations is greater than the number of drones, an auction algorithm based on delivery destinations is adopted, that is, the delivery destinations are auctioned as items and drones are bidders. At the same time, the number of delivery destinations that drones can auction is limited to L, where the value of L is determined according to the formula where N represents the number of drones and M represents the number of delivery destinations. Auctioning based on the delivery purpose can ensure that there is a drone at each delivery destination. For the case where a drone has to go to multiple destinations, the problem is modeled as a traveling salesman problem, and then the particle swarm algorithm is used to determine the order of visiting each delivery destination.

3. The method for swarm trajectory planning of drones for item delivery based on the improved DQN algorithm according to claim 2 is characterized in that: The problem is modeled as a traveling salesman problem, that is, a salesman has to go to different locations to sell goods, and determine the order of visiting each location so that the salesman's path is the shortest. That is, the drone needs to determine the order of visiting each delivery destination to reach multiple delivery points. The particle swarm algorithm is used to determine the order of visiting each delivery destination. The optimal solution to this problem is a numbering sequence of the starting point and the task point. Corresponding to the formula of the particle swarm algorithm, the position of the particle is this sequence, and the speed of the particle is an ordered exchange sequence, which can change the order of traversing the task points. The exchange sequence refers to the previous exchange sequence, the individual's historical optimal numbering sequence and the current generation population's optimal numbering sequence are updated, and the numbering sequence is updated according to the exchange sequence. The fitness function is positively correlated with the sum of the straight-line distances.

4. The method for trajectory planning of a swarm of drones delivering goods based on an improved DQN algorithm according to claim 1, characterized in that: In step 2, the trajectory planning problem is modeled as a Markov decision process, which is a typical problem model in reinforcement learning. The model includes state, action, strategy, reward and return elements; For the UAV trajectory planning problem, the UAV corresponds to a single agent in the Markov decision process. The environment is modeled based on real terrain images, and the target scene is modeled as a k*k grid map. The white area represents the feasible area of ​​the UAV cluster, and the black area represents the obstacle area. The target scene information is represented by a k*k two-dimensional matrix, where 0 represents the feasible area, 1 represents the obstacle area, 2 represents the current location of the UAV, 3 represents the starting point of the UAV, and 4 represents the end point of the UAV, that is, the delivery destination. The state is the two-dimensional matrix corresponding to the entire map. The UAV has 8 actions, which are the action of walking to the nearest discrete point in 8 equally divided directions of 360 degrees. The strategy uses the ε-greedy strategy. Where ε is a linearly increasing quantity, the formula is as follows:

5. The method for swarm trajectory planning of drones for item delivery based on the improved DQN algorithm according to claim 1 is characterized in that: In step 3, the reward function is set up as follows: a positive reward is provided when the drone reaches the target point, and a negative reward is provided when the drone hits an obstacle, which is called a penalty. However, with only these rewards for reaching special points, the drone's pathfinding is still slow. A two-way reward is also provided for the intermediate pathfinding process. The specific expression of the entire reward function is as follows: The expression shows that hitting an obstacle gives -2, exceeding the boundary gives -5, and reaching the target point gives 10. The remaining positions are calculated in a bidirectional manner. d1 represents the distance from the current point to the target point, d2 represents the distance from the next point to the target point, d3 represents the distance from the current point to the starting point, d4 represents the distance from the current point to the next point, and d5 represents the distance from the next point to the starting point. Therefore, d1-d2 indicates whether the drone is approaching the target point, and d5-d4-d3 indicates whether the drone is moving away from the starting point.

6. The method for trajectory planning of a swarm of drones delivering goods based on an improved DQN algorithm according to claim 1, characterized in that: In step 4, the starting point of the drone is randomly set in the scene, the delivery destination is set, and the track planning is carried out. During the iteration process, the drone selects an action and completes the position adjustment. When it encounters an obstacle, hits a boundary, or reaches the destination, this round ends; the relevant information of this round is saved in the experience pool, and a new round begins; through continuous iteration and learning from the experience of the experience pool, the drone can find discrete track points from the starting point to the delivery destination; the track obtained at this time is composed of some discrete track points.

7. The method for trajectory planning of a swarm of drones delivering goods based on an improved DQN algorithm according to claim 1, characterized in that: In step 5, the position of the dangerous waypoint is adjusted, and the track point whose distance to the surrounding obstacles is less than the safe distance is adjusted so that it is in a safe position, and the safe distance is set to 1.5 distance units.

8. The method for trajectory planning of a swarm of drones delivering goods based on an improved DQN algorithm according to claim 1, characterized in that: In step 6, the track obtained in step 5 is optimized; it is determined whether there are dynamic obstacles in the scene or whether there are conflicting points in the planned drone tracks. If neither of the two situations exists, Bezier curves are used for smoothing, and finally a smooth, collision-free drone cluster track is obtained; if any of the above situations exists, a local target point is selected from the track generated in step 5, a particle swarm algorithm is used for local track planning, and the local track is smoothed using Bezier curves.

9. The method for trajectory planning of a swarm of drones delivering goods based on an improved DQN algorithm according to claim 8, characterized in that: The corresponding formula of the Bezier curve is as follows: P i Represents the control point of the Bezier curve, B(t) is the corresponding point on the Bezier curve, t is the time parameter, and n represents the order of the Bezier curve.

Images