A UAV-assisted delivery system and method based on reinforcement learning
By constructing Markov decision-making process model and PPO-PSO algorithm, combined with LSTM-CNN neural network, we design drone task allocation and path planning algorithms, and solve the problem of low path planning and task allocation efficiency in drone logistics systems. The effect of reducing operating costs and improving transportation efficiency has been achieved.
Patent Information
- Application Number
- CN202410207143.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-26
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-02-26
AI Technical Summary
The prior art is difficult to effectively reduce the operating costs of logistics companies and improve the transportation efficiency of drone logistics systems, especially inefficient problems in path planning and task allocation.
Adopting a drone-assisted distribution system based on reinforcement learning, by constructing Markov decision-making process model and PPO-PSO algorithm, combining the LSTM-CNN neural network architecture, task allocation and path planning algorithms are designed to meet optimization goals and constraints, and achieve The autonomous guidance and obstacle avoidance capabilities of drones.
The optimal path planning is achieved under the satisfaction of safety and feasibility constraints, which reduces operating costs and improves transportation efficiency, which is about 15% lower than traditional methods, and enhances the robustness of the system.
Smart Images

Figure CN118170013B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) tracks, and in particular to a UAV-assisted delivery system and method based on reinforcement learning. Background Art
[0002] In recent years, there have been many research results in the fields of logistics and drones. However, there are few research results that combine these two fields. It is of theoretical significance to study the combination of drones and logistics, as follows:
[0003] (1)Due to their fast flight speed and light weight and agility, drones are very suitable for the last mile delivery of logistics and distribution services.
[0004] (2) Because drones are not restricted by road infrastructure, they can travel through mountains, jungles, rivers, and other areas with relative ease, taking shorter routes and having greater efficiency than other modes of transportation.
[0005] (3) Ground vehicles encounter many obstacles during delivery and sometimes require support to cross other inaccessible areas such as the sea, but drones are not hindered on their way to their destination.
[0006] (4) Drones are used to provide fast and accurate delivery services in urban areas because they can effectively avoid traffic jams. Drones can also be used to deliver relief supplies to disaster areas, many of which are inaccessible to ground vehicles or even rescue workers on foot. In addition, there is no concern about secondary injuries during relief delivery missions in disaster areas.
[0007] Path planning is essential for UAVs to perform their missions and avoid threats in their operating environment. The planned path should be optimal under specific criteria defined by the application. For most applications such as aerial photography, mapping, and ground inspection, the criterion is usually to minimize the flight distance between locations visited by the UAV to reduce the time and fuel required. The criterion can also be to maximize the probability of detection as in dynamic target search, minimize the flight time as in surveillance and rescue, or find a Pareto solution for multi-target navigation. In addition, the planned path needs to meet the safety constraints and feasibility constraints imposed by the UAV on the operating environment.
[0008] The UAV logistics system has some basic characteristics that are different from general ground logistics, such as limited flight time, strong load capacity, and the impact of cargo weight on flight capacity. Therefore, an effective logistics system and task allocation strategy play an important role in reducing the operating costs of logistics companies and improving transportation efficiency. Summary of the invention
[0009] Purpose of the invention: In order to reduce the operating costs of logistics companies and improve transportation efficiency, the present invention provides a drone-assisted distribution system and method based on reinforcement learning for the application of drones in delivery logistics.
[0010] Technical solution: A drone-assisted delivery method based on reinforcement learning, including the following steps:
[0011] Step 1: construct a mathematical model for drone-assisted delivery, namely, a drone delivery scheduling problem model; the drone delivery scheduling problem model is constructed using a Markov decision process model as a basic model, including a drone task allocation model and a drone path planning model;
[0012] The drone-assisted delivery process is as follows: the vehicle loads the items to be delivered into the drone loading bin, and the drone delivers the items to be delivered in the drone loading bin to the express cabinet closest to the user according to the planned optimal path. When the drone completes the delivery to the nearest express cabinet, it will immediately fly to another express cabinet, and return to the drone loading bin after all the deliveries are completed.
[0013] Step 2: Based on the PPO-PSO algorithm, the LSTM-CNN neural network architecture is used to design a task allocation algorithm for the UAV task allocation model and a route planning algorithm for the UAV path planning model;
[0014] Step 3: Build an autonomous guidance and tracking obstacle avoidance model to enable the UAV to adapt to the uncertainty of the object and environment, have the ability to change parameters and structures, and achieve continuous tracking and reasonable obstacle avoidance of random moving targets on the ground;
[0015] Step 4: Train the autonomous guidance and tracking obstacle avoidance model on the pybullet platform, deploy the trained neural network architecture to the designed experimental environment, and use the ROS system for simulation verification.
[0016] Furthermore, in step 1, the drone delivery scheduling problem model should satisfy the following optimization objectives and constraints:
[0017] (1) Define the optimization goal A as the user's satisfaction with the delivery service:
[0018]
[0019] Where, t i For package p i The actual delivery time, For package p i The best delivery time; i represents the i-th package; n represents the number of packages;
[0020] (2) Define optimization target B as the total delivery time of all packages:
[0021]
[0022] In the formula, For drone u k The time it takes to complete all its tasks, u k is the kth UAV;
[0023] (3) Convert the multi-objective optimization problem into a single-objective optimization problem, and convert the optimization objectives of total delivery time B and user satisfaction with delivery service A into the optimization of objective C:
[0024] C=B max ×A+B (3)
[0025] In the formula, B max The maximum delivery completion time of all packages;
[0026] (4) Constraints, including the maximum weight limit of a single package, the total load limit of a single drone flight, and the transportation distance limit of a single delivery;
[0027] (4.1) The maximum weight limit for a single package is:
[0028]
[0029] In the formula, w i Indicates package p i The weight, represents any UAV and the number of flights, and c represents the upper limit of the UAV load;
[0030] (4.2) The total load limit for a single flight of a UAV is:
[0031]
[0032] Where, T k,f For drone u k The total number of packages that need to be delivered in the fth flight, p k,f,b For drone u k The bth package delivered on the fth flight, Indicates package p k,f,b weight;
[0033] (4.2) The transportation distance limit for a single delivery is:
[0034]
[0035] In the formula, Indicates drone uk The destination of all packages in the fth flight, s0 represents the initial station; v represents the average flight power of the drone, e represents the maximum flight time of the drone; l k,f,b Indicates drone u k The destination of the bth package on the fth flight;
[0036] (5) It is expected that C will be minimized while satisfying the constraints in step (4). The number of delivered packages can be calculated using formulas (7)-(10):
[0037]
[0038] In the formula, UAV k The time it takes to complete all its tasks, Indicates drone u k The start time of the flight, Indicates drone u k The total time of the flight;
[0039]
[0040] In the formula, t k,f Indicates drone u k The start time of the fth flight; t k,f-1 Indicates drone u k The start time of the f-1 flight; q k,f-1 Indicates drone u k The total time of the f-1 trip;
[0041]
[0042] In the formula, q k,f Indicates drone u k The total time of the fth trip; t d represents the time required for the drone to load the package at s0; t c Indicates that the drone is in s1,...,s m The time required to deliver the package; s1,...,s m represents the 1st,…,mth sites;
[0043]
[0044] In the formula, p k,f,b Indicates drone u k The bth package delivered on the fth flight; x k,f,b,i is the decision variable, b represents the bth package;
[0045] (6) The decision formula and constraints of the drone delivery scheduling problem model are (11) and (12):
[0046]
[0047]
[0048] where x k,f,b,i is a decision variable, which takes 0 or 1, representing execution of the action or non-execution of the action, respectively; represents any i.
[0049] Furthermore, in the step 1, the Markov decision MDP of the Markov decision process model is described by a quintuple (S, A, R, P, γ); wherein S is the state space, A is the action space, R is the reward, P is the state transition probability, γ is the attenuation coefficient, and γ is in the interval [0,1];
[0050] In the nth cycle, the state s∈S is recorded as Sn, the drone control signal a∈A is recorded as An, and the cumulative reward corresponding to the drone action is recorded as R n+1 (S n ,A n ); After the drone takes action a in state s, the probability P of the environment state transitioning to s' is recorded as P ass′ ;
[0051] The reward of the Markov decision process is shown in formula (13), and the training goal is to maximize the reward:
[0052] Reward=r1+γr2+γ 2 r3+…+γ n-2 r n-1 +γ n-1 r n (13)
[0053] Based on the Markov decision MDP, the state observation function O and the confidence b are introduced; the update formula of the confidence b is as follows:
[0054]
[0055] Where η is a constant expressed as: η = 1 / Pr(o|b,a), and Pr(o|b,a) can be expressed as follows:
[0056]
[0057] The optimization objective of POMDP is the value function. When the confidence is b0 and the strategy is π, the value function can be expressed as:
[0058]
[0059] Therefore, the optimal strategy can be obtained as:
[0060]
[0061] Furthermore, in step 2:
[0062] The task allocation algorithm is used to generate a delivery plan; the route planning algorithm is used for route planning and scheduling of drones;
[0063] The PPO-PSO algorithm is an algorithm that improves the PPO algorithm by integrating the particle swarm algorithm into the PPO algorithm based on the characteristics of the standard PPO algorithm and the particle swarm algorithm.
[0064] The task allocation algorithm and route planning algorithm adopt independent LSTM-CNN neural network architectures, modify the agent iteration method based on the PPO-PSO algorithm using particle optimization, parameterize the agent using the CNN network, calculate the reward value using the long short-term memory network, update the network parameters through state information and perform adaptive optimization iterations, while considering the time series data and environmental context information received by the drone.
[0065] Furthermore, in step three:
[0066] The autonomous guidance and tracking obstacle avoidance model is used for unmanned aerial vehicle target tracking, has two stages: long-distance autonomous guidance and short-distance autonomous tracking obstacle avoidance, and is constructed using a Markov decision process model as a basic model.
[0067] The present invention also discloses a UAV-assisted distribution system based on reinforcement learning, including a distribution center, a UAV loading bin, a distribution terminal and a dispatching center;
[0068] The items to be delivered in the distribution center are transported by vehicles to the drone loading bin; the drone loading bin is used to load the items to be delivered, and the drone delivers the items to be delivered in the drone loading bin to the delivery destination; the dispatch center is set up in the drone loading bin, and is used to dispatch drones. The dispatch software updates the logistics status and drone status in real time, and generates drone delivery path planning results for staff reference; when performing complex scenario tasks, multiple drones can be assigned to perform tasks collaboratively.
[0069] Beneficial effects:
[0070] 1) In order to make the planned path meet the safety constraints and feasibility constraints imposed by the UAV on the operating environment, the present invention constructs a UAV delivery scheduling problem model;
[0071] The present invention adopts an improved proximal policy optimization algorithm PPO-PSO, uses a long short-term memory (LSTM) network to calculate the reward value, updates the network parameters through state information (such as the real-time position relationship between the UAV and the target) and performs adaptive optimization iterations. At the same time, the time series data and environmental context information received by the UAV are considered, and the time required for obtaining the optimal solution is about 15% lower than that of PPO, and has better robustness, which can reduce the operating costs of logistics companies and improve transportation efficiency.
[0072] 2) In response to the performance issues of UAVs against ground dynamic targets, a model with long-range autonomous guidance and short-range autonomous tracking and obstacle avoidance was developed, which can achieve continuous tracking and reasonable obstacle avoidance of randomly moving targets on the ground. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] Figure 1 It is the computational graph of each unit in LSTM;
[0074] Figure 2 A flow chart of the two-stage method for drone-assisted delivery;
[0075] Figure 3 The training process of each module of the drone-assisted delivery system;
[0076] Figure 4 A training environment scene diagram for the drone-assisted delivery system;
[0077] Figure 5 Node computation graph for drone-assisted delivery system;
[0078] Figure 6 This is the flight trajectory diagram observed by the RViz tool in this embodiment. DETAILED DESCRIPTION
[0079] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.
[0080] The present invention discloses a UAV-assisted distribution system based on reinforcement learning, comprising a distribution center, a UAV loading bin, a distribution destination and a dispatching center; the items to be distributed in the distribution center are transported to the UAV loading bin by vehicles; the UAV loading bin is used to load the items to be distributed, and the UAV distributes the items to be distributed in the UAV loading bin to the distribution destination; relevant personnel can replace batteries for the UAV in the UAV loading bin; the dispatching center is set up in the UAV loading bin, and is used to dispatch UAVs, and the dispatching software updates the logistics status and UAV status in real time, and generates UAV distribution path planning results for reference by the staff; when performing complex scene tasks, multiple UAVs can be assigned to perform tasks collaboratively, and when a system failure occurs, the failure is repaired by relevant personnel; the distribution destination may be an express cabinet.
[0081] The drone-assisted delivery planning problem can be viewed as a combinatorial optimization problem, and the solution to the problem is similar to the classic combinatorial optimization problem, the Traveling Salesman Problem (TSP). Assumptions: (1) The drones are isomorphic, and the basic parameters of each drone are the same; (2) After completing the delivery to a customer, the drone will immediately fly to another customer or return to the drone loading warehouse; (3) In this problem, the drone is regarded as a traveling salesman, and the customer is regarded as the city nodes that need to be passed; (4) The drone can complete the delivery and return to the drone loading warehouse before the battery is exhausted. The drone-assisted delivery process is as follows: the vehicle loads the items to be delivered into the drone loading warehouse, and the drone delivers the items to be delivered in the drone loading warehouse to the express cabinet closest to the user according to the planned optimal path, and the user picks up the items by himself; the drone will immediately fly to another express cabinet after completing the delivery to the nearest express cabinet; the drone returns to the drone loading warehouse after all deliveries are completed.
[0082] In order to better perform drone-assisted delivery, the present invention proposes a drone-assisted delivery method based on reinforcement learning, which includes the following steps:
[0083] Step 1: construct a mathematical model for drone-assisted delivery, namely, a drone delivery scheduling problem model; the drone delivery scheduling problem model is constructed using a Markov decision process model as a basic model, including a drone task allocation model and a drone path planning model;
[0084] (I) The drone delivery scheduling problem model should meet the following optimization objectives and constraints:
[0085] The drone loading bin has two indicators, which are used to distinguish the time when the drone takes off from the drone loading bin and the time when the takeoff is completed (flight). It is assumed that the replenishment process when the drone arrives at the drone loading bin has nothing to do with the remaining consumables. The delivery plans of drones and ground vehicles are independently derived based on the same (drone loading bin) warehouse specifications and the same customer information. The method proposed in the present invention uses different drone and ground vehicle information, such as vehicle speed and load capacity, as input data to independently obtain the delivery plans of drones and ground vehicles. There is a drone loading bin (number: N d ), vehicle (UAV or ground vehicle, No.: N k ) and customer delivery tasks (No. j The attribute of the delivery vehicle k is its initial position (drone loading bin number: N d ), moving speed (TS k )、Maximum moving distance(Q k ) and loading capacity (C k). Customer i (express locker) is randomly distributed in the service area, and its demand is determined by its location and demand quantity (A i ), service process time (P i ) and service time window (E i and L i ). Therefore, the UAV delivery scheduling problem model can be classified as a small-scale scheduling problem, which is solved by a mixed integer linear programming model. The relevant symbols are defined as follows:
[0086]
[0087]
[0088]
[0089] (1) Define the optimization target A as the user's satisfaction with the delivery service. The optimization target A is a quantity related to the delivery time:
[0090]
[0091] Where, t i For package p i The actual delivery time, For package p i The best delivery time; i represents the i-th package; n represents the number of packages.
[0092] (2) Define optimization target B as the total delivery time of all packages:
[0093]
[0094] In the formula, For drone u k The time it takes to complete all its tasks, u k is the kth drone.
[0095] (3) Convert the multi-objective optimization problem into a single-objective optimization problem, and convert the optimization objectives of the total delivery time B and the user's satisfaction with the delivery service A into the optimization of objective C:
[0096] C=B max ×A+B (3)
[0097] In the formula, B max The maximum delivery completion time of all packages.
[0098] (4) Constraints, including the maximum weight limit of a single package, the total load limit of a single drone flight, and the transportation distance limit of a single delivery;
[0099] (4.1) The maximum weight limit for a single package is:
[0100]
[0101] In the formula, w i Indicates package p i The weight, represents any drone and the number of flights, c
[0102] Indicates the upper limit of the drone's payload;
[0103] (4.2) The total load limit for a single flight of a UAV is:
[0104]
[0105] Where, T k,f For drone u k The total number of packages that need to be delivered in the fth flight, p k,f,b For drone u k The bth package delivered on the fth flight, Indicates package p k,f,b weight.
[0106] (4.2) The transportation distance limit for a single delivery is:
[0107]
[0108] In the formula, Indicates drone u k The destination of all packages in the fth flight, s0 represents the initial station; v represents the average flight power of the drone, e represents the maximum flight time of the drone; l k,f,b Indicates drone u k The destination of the bth package on the fth flight;
[0109] (5) It is expected that C will be minimized while satisfying the above three system constraints. The number of delivered packages can be calculated using formulas (7)-(10):
[0110]
[0111] In the formula, UAV k The time it takes to complete all its tasks, Indicates drone u k The start time of the flight, Indicates drone u k The total time of the flight;
[0112]
[0113] Where, t k,f Indicates drone u k The start time of the fth flight; t k,f-1 Indicates drone u k The start time of the f-1 flight; q k,f-1 Indicates drone u k The total time of the f-1 trip;
[0114]
[0115] In the formula, q k,f Indicates drone u k The total time of the fth trip; t d represents the time required for the drone to load the package at s0; t c Indicates that the drone is in s1,...,s m The time required to deliver the package; s1,...,s m represents the 1st,…,mth sites.
[0116]
[0117] In the formula, p k,f,b Indicates drone u k The bth package delivered on the fth flight; x k,f,b,i is the decision variable, and b represents the bth package.
[0118] (6) The decision formula of the drone delivery scheduling problem model is (11)-(12):
[0119]
[0120]
[0121] where x k,f,b,i is a decision variable, which can only take the value of 0 or 1, representing execution or non-execution of an action respectively; represents any i.
[0122] (II) The drone delivery scheduling problem model constructed by the present invention is constructed using the Markov decision process model as the basic model. The Markov decision MDP of the Markov decision process model can be described by a five-tuple (S, A, R, P, γ). Where S is the state space, A is the action space, R is the reward, P is the state transition probability, γ is the attenuation coefficient, and γ is in the interval [0,1]. In the nth cycle, the state s∈S is denoted as Sn, the drone control signal a∈A is denoted as An, and the cumulative reward corresponding to the drone action is denoted as R n+1 (Sn ,A n ). After the drone takes action a in state s, the probability P of the environment state transitioning to s' is recorded as P ass′ .
[0123] The reward of the Markov decision process is shown in formula (13), and the training goal is to maximize the reward.
[0124] Reward=r1+γr2+γ 2 r3+…+γ n-2 r n-1 +γ n-1 r n (13)
[0125] Considering that the state of the UAV-assisted delivery system is not completely observable, the decision process is modeled as a partially observable Markov decision (POMDP). POMDP introduces some additional system information on the basis of MDP, such as the state observation function O, confidence b, etc. The update formula of confidence b is as follows:
[0126]
[0127] Where η is a constant expressed as: η = 1 / Pr(o|b,a), and Pr(o|b,a) can be expressed as follows:
[0128]
[0129] The optimization objective of POMDP is the value function. When the confidence is b0 and the strategy is π, the value function can be expressed as:
[0130]
[0131] Therefore, the optimal strategy can be obtained as:
[0132]
[0133] Step 2: Based on the PPO-PSO algorithm, the LSTM-CNN neural network architecture is used to design a task allocation algorithm for the UAV task allocation model and a route planning algorithm for the UAV path planning model;
[0134] The task allocation algorithm is used to generate a distribution plan; the route planning algorithm is used for route planning and scheduling of drones.
[0135] Based on the characteristics of the standard PPO algorithm and the particle swarm algorithm (PSO), the present invention integrates the particle swarm algorithm into the PPO algorithm to improve the PPO algorithm, that is, improves the proximal policy gradient algorithm and integrates it into the PSO algorithm, referred to as the PPO-PSO algorithm.
[0136] The task allocation algorithm and route planning algorithm adopt independent LSTM-CNN neural network architectures, modify the agent iteration method based on the PPO-PSO algorithm using particle optimization, parameterize the agent using a CNN network, calculate the reward value using a long short-term memory (LSTM) network, update the network parameters through state information (such as the real-time position relationship between the UAV and the target) and perform adaptive optimization iterations, while considering the time series data and environmental context information received by the UAV, thereby solving the problem of poor local search ability of the neural network.
[0137] (1) Reinforcement Learning: PPO Algorithm
[0138] The PPO algorithm uses a CNN network to parameterize the agent. In the PPO algorithm, the updating mechanism of the critic network parameters is not much different from the traditional AC algorithm, and corresponding improvements are made to the updating mechanism of the performer network.
[0139] First, PPO introduces a new parameter as shown below, where π θ (a t ∣s t ) is the state s t When the policy θ network takes action a t The probability of θ. This parameter reflects the strategy θ and strategy θ k degree of difference.
[0140]
[0141] According to this parameter, the objective function can be set as:
[0142]
[0143] in is the advantage function value at time t, the hyperparameter ε represents the greediness coefficient, and clip(·) is the clipping function.
[0144] The advantage function of the PPO algorithm uses the algorithm idea of temporal difference and the general estimation of the advantage function. The calculation formula can be expressed as follows:
[0145]
[0146] in, is the timing difference error, which can be expressed as:
[0147]
[0148] Where V(s t ) is the value function, r t is the reward function.
[0149] The PPO algorithm process is as follows:
[0150]
[0151]
[0152] (2) Neural network architecture: LSTM-CNN architecture
[0153] The neural network architecture used in the algorithm of the present invention is the LSTM-CNN architecture. The performer selects actions based on probability distribution, the critic judges the value according to the actions generated by the performer, and the performer modifies the probability of the selected behavior according to the value of the critic. It can be seen that the convergence quality of the value network and the decision-making performance of the policy network affect each other. A suitable network can accelerate the convergence efficiency of the PPO algorithm and improve the convergence quality.
[0154] Long short-term memory (LSTM) neural networks are used in scenarios where input data has time sequence and dependency. In logistics and delivery tasks, the data received by drones is time-series, and the environmental state has contextual relevance, making the LSTM network suitable for task requirements. LSTM is combined with deep reinforcement learning algorithms and applied to drone logistics and delivery tasks. Its essence is to obtain state information such as the target motion trajectory as input, and use the maneuver control signal corresponding to the drone at the current moment as output to calculate the reward value. The network parameters are updated and retransmitted through an adaptive optimization algorithm to obtain the final model. The recurrent unit of the LSTM framework consists of four parts, namely the forget gate f, the input gate i, the output gate o, and the memory unit C. The weight matrix corresponding to the forget gate is w f and b f , the weight matrix corresponding to the input gate is w i and b i The weight matrix corresponding to the output gate is W o and b o , the weight matrix corresponding to the memory unit is W C and b C The calculation of each gate unit includes the current input X n , the state h generated at the previous moment n-1 , output gate O n And calculate the hidden layer state h under the action of tanh(·) function n Storage unit C n The calculation details of each unit in LSTM can be found in Figure 1 The calculation formula is as follows:
[0155] f n =σ(W f [h n-1 ,X n ]+bf )
[0156] i n =σ(W i [h n-1 ,X n ]+b i )
[0157]
[0158]
[0159] o n =σ(W o [h n-1 ,X n ]+b o )
[0160] h n =o n *tanh(C n )
[0161] In order to reduce the model error caused by the large order of magnitude difference between the input feature data and make the features of different dimensions comparable, the input feature parameters need to be normalized.
[0162] (2) Complete method flow and training process of each model:
[0163] Drone-assisted delivery is divided into two stages: the first stage is drone task allocation and the second stage is path planning. They correspond to the drone task allocation model and the second stage path planning model respectively. Both are trained by PPO, and the LSTM-CNN networks of the two are independent of each other. For the PPO algorithm, the training process and the testing process are basically the same in terms of process. The difference is that the latter does not require updating network parameters. For the complete method flow, refer to Figure 2 .
[0164] The end state of the first stage is the initial state of the second stage. The PPO training process of the first stage and the second stage are independent of each other. Independent environments are established for different stages to train the internal network of each model respectively. The test process only connects the models of the two stages. The training process of each model refers to Figure 3 .
[0165] The performance of the model was evaluated, and a variety of task scenarios were selected. The PSO algorithm was used as the control group of the present invention to compare the convergence speed and stability of the results.
[0166] Step 3: Build an autonomous guidance and tracking obstacle avoidance model to enable the UAV to adapt to the uncertainty of the object and environment, have the ability to change parameters and structures, and achieve continuous tracking and reasonable obstacle avoidance of random moving targets on the ground;
[0167] The autonomous guidance and tracking obstacle avoidance model constructed by the present invention has two stages: long-distance autonomous guidance and short-distance autonomous tracking and obstacle avoidance. It is constructed using a Markov decision process model as a basic model. The autonomous guidance and tracking obstacle avoidance model is used for UAV target tracking and can achieve continuous tracking and reasonable obstacle avoidance of random moving targets on the ground.
[0168] There are two stages in drone target tracking:
[0169] (1) A conversion function is used to map the original feature data into data with a value range of [0,1], and a nonlinear transformation is performed on the input layer parameters of the autonomously guided actor network. The sin(·) and cos(·) functions are used to transform σ n Decompose, the parameters υ(n) and L are restricted by the tanh(·) function, and the parameter τ n Remain unchanged.
[0170] (2) The input layer parameters of the companion flight obstacle avoidance actor network are linearly transformed using the following mapping:
[0171]
[0172] The hidden layer can map the features of the input data to a higher dimensional space to achieve better linear partitioning. The number of hidden layer neurons greatly affects the fitting ability of the network. Insufficient number of neurons will reduce the fitting ability of the network and fail to achieve the expected prediction accuracy, while too many neurons will lead to overfitting and increased training time. Therefore, selecting the appropriate number of hidden layer neurons is an important part of the modeling process. In this embodiment, 256 hidden layer neurons are used.
[0173] Step 4: Train the autonomous guidance and tracking obstacle avoidance model on the pybullet platform, deploy the trained neural network architecture to the designed experimental environment, and use the ROS system for simulation verification.
[0174] The experimental software and hardware configuration used in this embodiment can be seen in the following table.
[0175]
[0176] PyBullet provides forward dynamics simulation, inverse dynamics calculation, forward and inverse kinematics, collision detection and light calculation. In addition to physical simulation, PyBullet also supports image rendering, pybullet comes with a CPU renderer and OpenGL visualization capabilities.
[0177] 1. Autonomous guidance and tracking obstacle avoidance model Simulation training
[0178] Autonomous guidance and tracking obstacle avoidance model Training environment scene diagram reference Figure 4 This embodiment also uses the RViz tool, which is the official 3D visualization tool of ROS. Almost all the data from the sensor can be viewed through this tool. Figure 5 It is the node calculation graph of the system of the embodiment of the present invention. There are four communication mechanisms in ROS: topic, message, service, and action. Topic communication is similar to the UDP communication protocol. The publisher publishes the topic, and the subscriber can receive the corresponding information after choosing to subscribe. The message is a data structure in ROS. Each message is a data structure, which can be a standard data type (such as integer, floating point, Boolean, etc.), or a composite type (similar to struct in C language), or a developer-defined data type. Service is a communication model based on the client and the server, similar to the TCP protocol. The advantage of this communication over topic communication is real-time. The action communication mechanism is usually used for scenarios that do not require real-time updates and consume a lot of data resources, such as map synchronization. The flight trajectory diagram in this embodiment can also be synchronized to the ground end in real time for observation through the RViz tool, refer to Figure 6 .
[0179] On this basis, the traditional PPO algorithm is improved by using the long short-term memory (LSTM) neural network, and experimental verification is carried out on a UAV simulation test platform based on the Robot Operating System (ROS). The simulation results show that the autonomous guidance and tracking obstacle avoidance model proposed in this paper can achieve continuous tracking and reasonable obstacle avoidance of random moving targets on the ground. And compared with the PSO algorithm, the PPO algorithm control strategy based on the LSTM network can effectively suppress the influence of environmental disturbances and increase the robustness of the system.
[0180] (II) Simulation training of drone delivery scheduling problem model (including drone task allocation model and drone path planning model)
[0181] Three types of mission scenarios were selected to evaluate the model performance: Scenario 1 used the drone task allocation model, and Scenario 2 used the drone path planning model. In Scenario 3, the two models were organized into the same mission process. By setting the drone height threshold to 10m and the relative distance between the two to less than 20m, the model training networks of Scenario 1 and Scenario 2 were applied to the entire process after convergence. The model training parameters of the three scenarios are shown in the following table.
[0182]
[0183]
[0184] Based on the above scenario, the PSO algorithm and the task allocation algorithm and route planning algorithm based on the PPO-PSO algorithm proposed in the present invention, which are constructed using the LSTM-CNN neural network architecture, are set as the control group, the number of hidden layer neurons is set to 256, and the same parameter optimizer is used to update the model training parameters.
[0185] In the experimental result curve, the horizontal axis represents the number of training rounds, and the vertical axis represents the reward function value. The results show that the autonomous decision-making and planning process of the drone is reasonable. When the target moves along a straight line or a circular trajectory, the initial value and final convergence value of the single-step average reward of the drone trained by the PSO algorithm and the task allocation algorithm and route planning algorithm based on the PPO-PSO algorithm proposed in the present invention, which are constructed using the LSTM-CNN neural network architecture, fluctuate within a smaller range, and the latter converges faster. The evaluation data is shown in the table below.
[0186]
[0187] It can be seen from the curve graph and table that the deep reinforcement learning algorithm is better than the particle swarm algorithm in convergence speed, and the reward variance of the deep reinforcement learning algorithm is smaller than that of the particle swarm algorithm, that is, the deep reinforcement learning algorithm is more stable than the particle swarm algorithm.
[0188] The present invention aims at the performance problem of multi-rotor unmanned aerial vehicles (UAVs) against ground dynamic targets. The present invention develops long-distance autonomous guidance and short-distance autonomous tracking and obstacle avoidance models, and uses an improved PPO-PSO algorithm to design the task allocation and path planning algorithm of the UAV to obtain the optimal convergence and solution speed. Finally, experiments were carried out on a UAV simulation test platform based on the Robot Operating System (ROS). The decision-making and planning capabilities of reinforcement learning can realize the automatic execution of tasks of the UAV, and compared with the traditional particle swarm algorithm, reinforcement learning is better in convergence speed and solution speed. The improved PPO algorithm control strategy can effectively suppress the influence of environmental disturbances, increase the robustness of the system, and has better generalization. A UAV-assisted distribution system that can effectively suppress environmental disturbances, has high robustness and good generalization.
[0189] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.
Claims
1. A drone-assisted delivery method based on reinforcement learning, characterized in that: The steps include: Step 1: construct a mathematical model for drone-assisted delivery, namely, a drone delivery scheduling problem model; the drone delivery scheduling problem model is constructed using a Markov decision process model as a basic model, including a drone task allocation model and a drone path planning model; The drone-assisted delivery process is as follows: the vehicle loads the items to be delivered into the drone loading bin, and the drone delivers the items to be delivered in the drone loading bin to the express cabinet closest to the user according to the planned optimal path. When the drone completes the delivery to the nearest express cabinet, it will immediately fly to another express cabinet, and return to the drone loading bin after all the deliveries are completed. Step 2: Based on the PPO-PSO algorithm, the LSTM-CNN neural network architecture is used to design a task allocation algorithm for the UAV task allocation model and a route planning algorithm for the UAV path planning model; The task allocation algorithm and route planning algorithm adopt independent LSTM-CNN neural network architectures, modify the agent iteration method based on the PPO-PSO algorithm using particle optimization, parameterize the agent using the CNN network, calculate the reward value using the long short-term memory network, update the network parameters through the state information and perform adaptive optimization iterations, while considering the time series data and environmental context information received by the drone; Step 3: Build an autonomous guidance and tracking obstacle avoidance model to enable the UAV to adapt to the uncertainty of the object and environment, have the ability to change parameters and structures, and achieve continuous tracking and reasonable obstacle avoidance of random moving targets on the ground; Step 4: Train the autonomous guidance and tracking obstacle avoidance model on the pybullet platform, deploy the trained neural network architecture to the designed experimental environment, and use the ROS system for simulation verification.
2. The method for unmanned aerial vehicle-assisted delivery based on reinforcement learning according to claim 1, characterized in that: In step 1, the drone delivery scheduling problem model should meet the following optimization objectives and constraints: (1) Define the optimization goal A as the user’s satisfaction with the delivery service: (1) In the formula, For Package The actual delivery time, For Package The best delivery time; Indicates packages; n Indicates the number of packages; (2) Define optimization target B as the total delivery time of all packages: (2) In the formula, For drones The time it takes to complete all its tasks, For the drones; (3) Convert the multi-objective optimization problem into a single-objective optimization problem, and convert the optimization objectives of total delivery time B and user satisfaction with delivery service A into the optimization of objective C: (3) In the formula, The maximum delivery completion time of all packages; (4) Constraints, including the maximum weight limit of a single package, the total load limit of a single drone flight, and the transportation distance limit of a single delivery; (4.1) The maximum weight limit for a single package is: (4) In the formula, Indicates package The weight, Represents any drone and the number of flights, Indicates the upper limit of the drone's payload; (4.2) The total load limit for a single flight of a UAV is: (5) In the formula, For drones No. The total number of packages that need to be delivered on this flight, For drones No. Flight Packages delivered, Indicates package weight; (4.2) The transportation distance limit for a single delivery is: (6) In the formula, Indicates drone No. The destination of all packages on the flight, represents the initial site; represents the average flight power of the drone, Indicates the maximum flight time of the drone; Indicates drone No. Flight The destination of the package; (5) It is expected that C is minimized while satisfying the constraints in step (4). The number of delivered packages can be calculated using formulas (7)-(10): (7) In the formula, Drones The time it takes to complete all its tasks, Indicates drone The start time of the flight, Indicates drone The total time of the flight; (8) In the formula, Indicates drone No. The start time of the flight; Indicates drone No. The start time of the flight; Indicates drone No. Total time of the trip; (9) In the formula, Indicates drone No. Total time of the trip; Indicates that the drone is The time required to load the package; Indicates that the drone is The time required to deliver the package; represents the 1st,…,mth sites; (10) In the formula, Indicates drone No. Flight parcels delivered; is the decision variable, represents the bth package; (6) The decision formula and constraints of the drone delivery scheduling problem model are (11) and (12): (11) (12) in is a decision variable, which takes 0 or 1, representing execution of the action or non-execution of the action, respectively; Indicates any i , h Refers to the total number of drones.
3. The method for drone-assisted delivery based on reinforcement learning according to claim 1, characterized in that: In the step 1, the Markov decision process model of the Markov decision process MDP is represented by a five-tuple to describe; is the state space, For the action space, For reward, is the state transition probability, is the attenuation coefficient, and is in the interval [0,1]; In the In a cycle, the state Sn, the drone control signal Recorded as , the cumulative reward corresponding to the drone action is recorded as ; Drone in status Medium Action After that, the environmental state changes to Probability of transition Recorded as ; The reward of the Markov decision process is shown in formula (13), and the training goal is to maximize the reward: (13) Based on the Markov decision MDP, the state observation function is introduced. O and confidence b; the update formula of confidence b is as follows: ; In the formula, Is a constant expressed as: , It can be expressed as follows: ; The optimization objective of POMDP is the value function. , the strategy is The time-valued function can be expressed as: ; Therefore, the optimal strategy can be obtained as: 。 4. The method for drone-assisted delivery based on reinforcement learning according to claim 1, characterized in that: In the step 2: The task allocation algorithm is used to generate a delivery plan; the route planning algorithm is used for route planning and scheduling of drones; The PPO-PSO algorithm refers to an algorithm that improves the PPO algorithm by integrating the particle swarm algorithm into the PPO algorithm based on the characteristics of the standard PPO algorithm and the particle swarm algorithm.
5. The method for drone-assisted delivery based on reinforcement learning according to claim 1, characterized in that: In the step three: The autonomous guidance and tracking obstacle avoidance model is used for UAV target tracking, has two stages: long-distance autonomous guidance and short-distance autonomous tracking obstacle avoidance, and is constructed using a Markov decision process model as a basic model.
6. A UAV-assisted delivery system based on reinforcement learning, characterized in that: The drone-assisted delivery system adopts the drone-assisted delivery method described in any one of claims 1 to 5, and the drone-assisted delivery system includes a delivery center, a drone loading warehouse, a delivery terminal, and a dispatch center; The items to be delivered in the distribution center are transported by vehicles to the drone loading bin; the drone loading bin is used to load the items to be delivered, and the drone delivers the items to be delivered in the drone loading bin to the delivery destination; The dispatch center is set up in the drone loading warehouse and is used to dispatch drones. The dispatch software updates the logistics status and drone status in real time and generates drone delivery path planning results for reference by staff. When performing tasks in complex scenarios, multiple drones can be assigned to perform tasks collaboratively.
