A method for controlling the path of UAV water logistics distribution considering real-time obstacle avoidance

By applying an improved deep deterministic strategy gradient algorithm in water logistics distribution scenarios, a real-time path regulation model for drones is constructed, which solves the problem of collision risk between logistics drones and invading drones, and achieves better path adjustment effect , enhances the safety of drones and mission execution efficiency.

CN118034283BActive Publication Date: 2024-11-22NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410144737.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-01
Publication Date
2024-11-22
Estimated Expiration
2044-02-01

AI Technical Summary

Technical Problem

In the water logistics distribution scenario, the flight trajectory of non-cooperative drones is unknown, and there is a risk of collision with logistics drones. The original path planning cannot ensure the safe operation of logistics drones, and real-time path adjustment is required to avoid obstacles.

Method used

A real-time path control method for drone based on deep deterministic strategy gradient algorithm (DDPG) was designed. By constructing state space, action space and reward functions, the improved DDPG algorithm network structure is generated, and the training model is used to realize logistics drone. real-time path control can be used to adjust paths when invading drones.

Benefits of technology

This method improves the path control performance of logistics drones in water logistics distribution scenarios, can effectively avoid invasive drones, achieve better path adjustment effects, and enhances the safety and task execution efficiency of logistics drones.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118034283B_ABST
    Figure CN118034283B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for controlling the path of UAV water logistics distribution considering real-time obstacle avoidance, including: generating a pre-planning scheme for logistics distribution; establishing a kinematic model of a logistics UAV in two-dimensional space; constructing a real-time path control model for a logistics UAV; designing the actor-critic network structure in a deep deterministic policy gradient algorithm, and improving the algorithm to obtain an improved deep deterministic policy gradient algorithm; training the real-time path control model for the logistics UAV based on the improved deep deterministic policy gradient algorithm, and using the trained model to realize real-time path control of the logistics UAV. The method of the present invention can be used for the logistics UAV to adjust the path in real time and realize autonomous obstacle avoidance when other non-cooperative UAVs invade during the distribution process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle traffic management, and specifically relates to a method for controlling the path of unmanned aerial vehicle water logistics distribution taking into account real-time obstacle avoidance. Background Art

[0002] In the field of logistics and distribution, compared with traditional distribution methods, drone delivery has the characteristics of fast speed, low cost and strong flexibility. Path planning is an important technology to achieve autonomous control of drone flight and complete tasks safely and smoothly, which directly affects the efficiency of drones in performing tasks in dynamic, unknown and complex environments.

[0003] In the water logistics distribution scenario, in addition to logistics drones that operate according to the planned path, there are also non-cooperative drones that perform other tasks, such as river inspections, emergency rescue and other tasks. The addition of other drones in the airspace increases the complexity of the flight airspace. At the same time, the flight trajectory plan of non-cooperative drones is unknown. When such drones appear within the delivery range of logistics drones, there is a certain risk of collision. Therefore, the original planned flight path cannot guarantee the safe operation of logistics drones. It is necessary to enable logistics drones to have real-time path adjustment capabilities and to achieve autonomous obstacle avoidance when encountering other non-cooperative drones. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method for controlling the path of UAV water logistics distribution taking into account real-time obstacle avoidance. The method of the present invention can be used to deal with the invasion of other non-cooperative UAVs during the distribution process of the logistics UAV. The logistics UAV can adjust the path in real time and achieve autonomous obstacle avoidance.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] The present invention provides a method for controlling the path of UAV water logistics distribution taking into account real-time obstacle avoidance, and the steps are as follows:

[0007] 1) Generate a logistics distribution pre-planning plan based on the target ship distribution mission requirements and target ship planned trajectory;

[0008] 2) Establish a kinematic model of logistics UAV in two-dimensional space;

[0009] 3) Based on the kinematic model in step 2), the state space, action space and reward function of the logistics UAV are designed to build a real-time path control model for the logistics UAV;

[0010] 4) According to the drone state space information and action space information designed in step 3), the actor-critic network structure in the deep deterministic policy gradient algorithm is designed, and the algorithm is improved to obtain an improved deep deterministic policy gradient algorithm;

[0011] 5) Based on the improved deep deterministic policy gradient algorithm in step 4), the real-time path control model of the logistics drone in step 3) is trained, and the trained model is used to realize the real-time path control of the logistics drone.

[0012] Further, the step 1) specifically includes: selecting a meeting location close to the distribution center as the distribution point according to the principle of proximity based on the collected target ship's planned track and distribution task requirements, and calculating the flight speed required for the logistics drone's distribution according to the distance between the logistics drone's location and the meeting location and the time it takes for the target ship to sail to the meeting location;

[0013] Set the starting position of the logistics drone to (x 0 ,y 0 ), the encounter position is (i, j), the time required for the target ship to sail to (i, j) according to its planned track is ΔT, and the flight speed required for logistics drone delivery is calculated as follows:

[0014]

[0015] Where V represents the required flight speed for logistics drone delivery.

[0016] Furthermore, the step 2) specifically includes:

[0017] Set the coordinates of the logistics drone at time t to (x t ,y t ), v x,t 、v y,t are the speeds of the logistics drone in the x and y axes at time t, and the acceleration is n t , the heading angle is ω, and the calculation expression of the kinematic model of the logistics UAV is:

[0018]

[0019]

[0020] In the formula, v x,t+1 、v y,t+1 are the speeds of the logistics drone in the x and y axes at time t+1, respectively. t+1 ,y t+1 ) is the coordinate of the UAV at time t+1.

[0021] Furthermore, the specific process of step 3) is as follows:

[0022] 31) Design the state space of logistics drones;

[0023] The design status information includes the internal status information of the logistics drone, the relevant information of the detected intrusion drone, and the maximum trajectory reachable space information of the intrusion drone, which is specifically expressed as follows:

[0024]

[0025] In the formula, S represents the state space set of logistics drones, v′ t represents the flight speed of the logistics drone at time t after normalization, ω t represents the heading angle of the logistics drone at time t, u′ t represents the normalized flight speed of the invading drone at time t, α t represents the heading angle between the logistics drone and the intrusion drone at time t, d′ t represents the relative distance between the logistics drone and the intrusion drone at time t after normalization, e′ t represents the distance between the logistics drone and the maximum trajectory reachable space at time t after normalization, DE t Indicates whether the logistics drone is within the maximum trajectory reachable space at time t;

[0026] 32) Design the action space of logistics drones;

[0027] Based on the kinematic model of the drone in step 2), the action space of the drone is set to be related to the acceleration n and the heading angle ω; at the same time, considering the flight safety of the drone, the acceleration and heading angle should meet the corresponding constraints, and the specific settings are as follows:

[0028] A=[n,ω] T , n∈[-n max ,n max ],ω∈[-ω max ,ω max ] (5)

[0029] In the formula, A represents the action space set of logistics drones, n max represents the maximum acceleration of the logistics drone, ω max Indicates the maximum heading angle of the logistics drone;

[0030] 33) Design a reward function for logistics drones;

[0031] The reward function consists of a continuous reward function and a sparse reward function. The specific expression is as follows:

[0032] R=R 1 +R2 (6)

[0033] In the formula, R represents the reward function, R 1 Represents the continuous reward part, R 2 represents the sparse reward part;

[0034] The state space of the logistics UAV in the above step 31) is used as the input information of the real-time path control model of the logistics UAV, the action space of the logistics UAV in the above step 32) is used as the output information of the real-time path control model of the logistics UAV, and the reward function of the logistics UAV in the above step 33) is used as the calculation method of the reward obtained by the interaction between the logistics UAV and the environment during model training, thereby constructing the real-time path control model of the logistics UAV.

[0035] Furthermore, the step 31) specifically includes:

[0036] 311) Determine the internal status information of the logistics drone;

[0037] The flight speed and heading angle of the logistics drone are selected as part of the state information, and the flight speed is normalized. The specific calculation formula is as follows:

[0038]

[0039] In the formula, v′ t represents the flight speed of the logistics drone at time t after normalization, v t represents the flight speed of the logistics drone at time t, v max Indicates the maximum flight speed of the logistics drone;

[0040] 312) Determine the information related to the detected intruding drone;

[0041] The flight speed, heading angle, and relative distance between the invading drone and the logistics drone are selected as the invading drone related information, and the flight speed of the invading drone is normalized. The specific calculation formula is as follows:

[0042]

[0043] Where u′ t represents the normalized flight speed of the invading drone at time t, u t represents the flight speed of the invading drone at time t, u max Indicates the maximum flight speed of the invading drone;

[0044] Assume the coordinates of the invading drone at time t to be (p t ,q t ), the relative distance calculation formula and normalization formula between the logistics drone and the intrusion drone are as follows:

[0045]

[0046]

[0047] Where, d t represents the relative distance between the logistics drone and the intrusion drone at time t, d′ t represents the relative distance between the logistics drone and the intrusion drone at time t after normalization, d max Indicates the maximum warning range radius of the logistics drone;

[0048] 313) Calculate the maximum reachable space of the invading drone’s trajectory;

[0049] According to the flight performance of the UAV, the trajectory of the invading UAV is analyzed according to the three flight stages of turning, acceleration and constant speed; the initial coordinates of the invading UAV are set as (p 0 ,q 0 ), with initial velocity u 0 , and take the position of the invading drone as the coordinate origin O, the positive direction of the Y axis is the speed direction, and the positive direction of the X axis is the right side of the invading drone body, and establish the Cartesian coordinate system OXY;

[0050] Turning flight phase:

[0051]

[0052]

[0053] In the formula, (p T ,q T ) represents the coordinates of the invading drone after turning, r represents the turning radius of the invading drone, Indicates the turning angle of the invading drone;

[0054] Accelerated flight phase:

[0055]

[0056]

[0057] In the formula, (p A ,q A ) represents the coordinates of the invading drone after accelerating its flight, t A Represents the flight time of the invading drone during the acceleration phase, m max Indicates the maximum acceleration of the invading drone;

[0058] Constant speed flight phase:

[0059]

[0060]

[0061] In the formula, (p U ,q U ) represents the coordinates of the invading drone after uniform flight, u max represents the maximum flight speed of the invading drone, t U Indicates the uniform flight time of the invading drone;

[0062] The coordinates of the invading drone after uniform flight are input and fitted through the arc model Circle in python to obtain the coordinates of the center of the circle (p c ,q c ) and radius f c , and then get the corresponding state input information, the calculation formula is as follows:

[0063]

[0064] e t =h t -f c (18)

[0065]

[0066] In the formula, h t represents the distance between the logistics drone and the center of the maximum trajectory space at time t, e t represents the distance between the logistics drone and the maximum trajectory reachable space at time t, DE t Indicates whether the logistics drone is within the maximum trajectory reachable space at time t; t Normalize to get e′ t , the calculation formula is as follows:

[0067]

[0068] Furthermore, the step 33) specifically includes:

[0069] 331) Design a continuous reward function R for logistics drones 1 ;

[0070] Continuous reward function R 1 Including obstacle avoidance reward, path restoration reward and step length penalty, the specific calculation formula is as follows:

[0071]

[0072]

[0073] In the formula, re avoidrepresents the obstacle avoidance reward, re origin represents the path restoration reward, (x′ t ,y′ t ) represents the position of the logistics drone after avoiding obstacles at time t, (p t ,q t ) represents the position of the invading drone at time t, (x t ,y t ) represents the position of the logistics drone on the original path at time t, and the continuous reward function R 1 The specific calculation formula is as follows:

[0074] R 1 =λre avoid +(1-λ)re origin -1 (23)

[0075] In the formula, -1 represents the set logistics drone step size penalty, and λ represents the reward weight;

[0076] 332) Design the coefficient reward function R of the logistics drone 2 ;

[0077] Set the collision penalty between the logistics drone and the intrusion drone. The specific sparse reward settings are as follows:

[0078]

[0079] Furthermore, the deep deterministic policy gradient algorithm designed in the step 4) adopts an actor-critic network structure, which includes an online network and a target network corresponding thereto; the input information of the actor online network is the content contained in the state space set S, 7 input neurons are set, 2 hidden layers are set in the middle layer, and they are connected in a fully connected manner. The number of neurons in each hidden layer is 256, and the ReLU activation function is used; the output layer is the acceleration and heading angle of the drone, 2 neurons are set, and the tanh activation function is used, and the actor target network structure design is the same as its online network; the input information of the critic online network is state information and action information, the input layer has 9 neurons, 2 hidden layers are set in the middle layer, and they are connected in a fully connected manner. The number of neurons in each hidden layer is 256, and the ReLU activation function is used; the output layer is the action value, 1 neuron is set, and the critic target network structure design is the same as its online network.

[0080] Furthermore, the step 4) specifically includes:

[0081] 41) Design hybrid exploration strategies;

[0082] In the early stage of training, the dynamic ε-greedy method is used to select actions, and noise interference is added in the later stage of training. The specific formula is as follows:

[0083]

[0084] In the formula, a t represents the action of the logistics drone at time t, uniform() represents a random number that obeys a uniform distribution, and a low and a high Indicates the upper and lower limits of the action, clip() means to change a t Restricted to a low and a high In the range, μ represents the current policy function of the actor online network, θ represents the parameters of the actor online network, and μ(s t |θ μ ) indicates that the actor online network is in state s t The action generated by t represents the noise sampled from the Ornstein-Uhlenbeck process (OU noise for short), rand() represents a random number between (0,1), and ε is a decreasing greedy factor. The specific formula is as follows:

[0085]

[0086] In the formula, ε max , ε min Respectively represent the maximum and minimum values ​​of the greedy factor ε; t total represents the cumulative time step; Represents the value of the greedy factor ε which decreases after each time step;

[0087] 42) Design a dual experience playback mechanism;

[0088] Two experience replay pools EXP1 and EXP2 are set up to store all training samples and training samples with reward values ​​greater than 1 in a single time step respectively. During the sampling of small batch samples, 90% of the total number of samples are randomly selected from EXP1, and the remaining required number of samples are randomly selected from EXP2.

[0089] Furthermore, the specific process of step 5) is as follows:

[0090] 51) Initialize actor and critic online network parameters, target network parameters, total number of training rounds, OU noise, reward discount factor, greedy factor, and experience replay pools EXP1 and EXP2;

[0091] 52) According to the pre-planned logistics distribution plan, the logistics drone flies according to the original plan and randomly selects a preset intrusion drone trajectory;

[0092] 53) When the intruding drone enters the warning range of the logistics drone, the maximum reachable range of the intruding drone’s trajectory at the future moment is calculated, and the distance relationship between the current position of the logistics drone and the maximum reachable range of the trajectory is determined;

[0093] 54) Combine the flight information of the logistics drone and the flight information of the intrusion drone it detects to obtain the status information s t , according to the hybrid exploration mechanism in the network, action a is selected t ;

[0094] 55) Logistics drone performs action a t , interacting with the environment is rewarded t and the next state s t+1 , the invading drone moves to the next location;

[0095] 56) Determine whether the number of samples in the experience replay pools EXP1 and EXP2 is greater than the minimum sample batch number N. If less than N, return to step 53); if greater than N, extract 90% of the minimum sample batch number N from the experience replay pool EXP1, and extract 10% of the minimum sample batch number N from the experience replay pool EXP2;

[0096] 57) Input the small batch of samples into the critic target network to obtain the target reward value w t , the policy gradient obtained by sampling is used to update the actor online network, and the critic online network is updated by minimizing the loss function. The specific formula is as follows:

[0097]

[0098] In the formula, w t represents the reward value obtained in the target network, Q(s t ,a t |θ Q ) means to convert s t and a t The actual value Q, θ obtained by jointly inputting the critic online network Q represents the critic online network parameters, w t The specific calculation formula is as follows:

[0099]

[0100] In the formula, re t represents the immediate reward obtained at time t, γ represents the reward discount factor, Q′(st+1 ,μ'(s t+1 |θ μ' )|θ Q' ) represents the target action and the next state s t+1 The Q value obtained after inputting the target network together, μ'(s t+1 |θ μ' ) The next action value output by the actor target network;

[0101] 58) Update the network target parameters. The specific formula is as follows:

[0102]

[0103] Where ξ represents the coefficient used to adjust the update, ξ∈[0,1], θ Q' and θ Q Represent the critic online network and target network parameters, θ μ' and θ μ Represent the critic online network and target network parameters respectively;

[0104] 59) Determine the status s t+1 Is it a termination state? If not, return to step 53); if yes, determine whether the number of training rounds reaches the total number of rounds. If not, return to step 54). If reached, it means that the training is completed and the corresponding network parameters are saved;

[0105] 510) uses the model trained in step 59) to achieve real-time path control of the logistics drone.

[0106] Beneficial effects of the present invention:

[0107] Based on the water ship logistics distribution scenario, the present invention generates a preliminary water logistics drone distribution pre-planning scheme. Considering the possible conflict between logistics drones and invading drones during the distribution process, a real-time path control model for logistics drones based on the improved DDPG algorithm is proposed. By comparing and testing the models trained before and after the improvement, it is proved that the improved algorithm has better performance and stronger generalization ability, and the logistics drone can achieve real-time adjustment of the path. The method of the present invention provides a reference for the real-time path control method of logistics drones in the water ship logistics distribution scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Figure 1 Flow chart of the method of the present invention.

[0109] Figure 2 This is a trend diagram of the error change in each round of an embodiment of the present invention.

[0110] Figure 3This is a graph showing changes in average reward value according to an embodiment of the present invention.

[0111] Figure 4 Schematic comparison diagram of obstacle avoidance of a logistics drone according to an embodiment of the present invention. DETAILED DESCRIPTION

[0112] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0113] Reference Figure 1 As shown, a method for controlling the path of UAV water logistics distribution taking into account real-time obstacle avoidance of the present invention comprises the following steps:

[0114] 1) Generate a logistics distribution pre-planning plan based on the target ship distribution mission requirements and target ship planned trajectory;

[0115] Based on the collected target ship's planned track and delivery mission requirements, a meeting location close to the distribution center is selected as the delivery point according to the principle of proximity. The flight speed required for the logistics drone delivery is calculated based on the distance between the logistics drone's location and the meeting location and the time it takes for the target ship to sail to the meeting location.

[0116] Set the starting position of the logistics drone to (x 0 ,y 0 ), the encounter position is (i, j), the time required for the target ship to sail to (i, j) according to its planned track is ΔT, and the flight speed required for logistics drone delivery is calculated as follows:

[0117]

[0118] Where V represents the required flight speed for logistics drone delivery.

[0119] In this example, the generated logistics drone pre-planning plan is as shown in Table 1:

[0120] Table 1

[0121]

[0122] 2) Establish a kinematic model of logistics UAV in two-dimensional space;

[0123] Set the coordinates of the logistics drone at time t to (x t ,y t ), v x,t 、v y,t are the speeds of the logistics drone in the x and y axes at time t, and the acceleration is n t , the heading angle is ω, and the calculation expression of the kinematic model of the logistics UAV is:

[0124]

[0125]

[0126] In the formula, v x,t+1 、v y,t+1 are the speeds of the logistics drone in the x and y axes at time t+1, respectively. t+1 ,y t+1 ) is the coordinate of the UAV at time t+1.

[0127] 3) Based on the kinematic model in step 2), the state space, action space and reward function of the logistics UAV are designed to build a real-time path control model for the logistics UAV;

[0128] 31) Design the state space of logistics drones;

[0129] The design status information includes the internal status information of the logistics drone, the relevant information of the detected intrusion drone, and the maximum trajectory reachable space information of the intrusion drone, which is specifically expressed as follows:

[0130] S=(v′ t ,ω t ,u′ t ,α t ,d′ t ,e′ t ,DE t ) (4)

[0131] In the formula, S represents the state space set of logistics drones, v′ t represents the flight speed of the logistics drone at time t after normalization, ω t represents the heading angle of the logistics drone at time t, u′ t represents the normalized flight speed of the invading drone at time t, α t represents the heading angle between the logistics drone and the intrusion drone at time t, d′ t represents the relative distance between the logistics drone and the intrusion drone at time t after normalization, e′ t represents the distance between the logistics drone and the maximum trajectory reachable space at time t after normalization, DE t Indicates whether the logistics drone is within the maximum trajectory reachable space at time t;

[0132] 32) Design the action space of logistics drones;

[0133] Based on the kinematic model of the drone in step 2), the action space of the drone is set to be related to the acceleration n and the heading angle ω; at the same time, considering the flight safety of the drone, the acceleration and heading angle should meet the corresponding constraints, and the specific settings are as follows:

[0134] A=[n,ω] T , n∈[-n max ,n max ],ω∈[-ω max ,ω max ] (5)

[0135] In the formula, A represents the action space set of logistics drones, n max represents the maximum acceleration of the logistics drone, ω max Indicates the maximum heading angle of the logistics drone;

[0136] In this example, the maximum acceleration of the logistics drone used is 6m / s 2 , the maximum heading angle is π.

[0137] 33) Design a reward function for logistics drones;

[0138] The reward function consists of a continuous reward function and a sparse reward function. The specific expression is as follows:

[0139] R=R 1 +R 2 (6)

[0140] In the formula, R represents the reward function, R 1 Represents the continuous reward part, R 2 represents the sparse reward part;

[0141] The state space of the logistics UAV in the above step 31) is used as the input information of the real-time path control model of the logistics UAV, the action space of the logistics UAV in the above step 32) is used as the output information of the real-time path control model of the logistics UAV, and the reward function of the logistics UAV in the above step 33) is used as the calculation method of the reward obtained by the interaction between the logistics UAV and the environment during model training, thereby constructing the real-time path control model of the logistics UAV.

[0142] Wherein, the step 31) specifically includes:

[0143] 311) Determine the internal status information of the logistics drone;

[0144] The flight speed and heading angle of the logistics drone are selected as part of the state information, and the flight speed is normalized. The specific calculation formula is as follows:

[0145]

[0146] In the formula, v′ t represents the flight speed of the logistics drone at time t after normalization, v t represents the flight speed of the logistics drone at time t, v maxIndicates the maximum flight speed of the logistics drone;

[0147] In this example, the maximum flight speed of the logistics drone used is 25m / s.

[0148] 312) Determine the information related to the detected intrusion drone;

[0149] The flight speed, heading angle, and relative distance between the invading drone and the logistics drone are selected as the invading drone related information, and the flight speed of the invading drone is normalized. The specific calculation formula is as follows:

[0150]

[0151] In the formula, u′ t represents the normalized flight speed of the invading drone at time t, u t represents the flight speed of the invading drone at time t, u max Indicates the maximum flight speed of the invading drone;

[0152] Assume the coordinates of the invading drone at time t to be (p t ,q t ), the relative distance calculation formula and normalization formula between the logistics drone and the intrusion drone are as follows:

[0153]

[0154]

[0155] Where, d t represents the relative distance between the logistics drone and the intrusion drone at time t, d′ t represents the relative distance between the logistics drone and the intrusion drone at time t after normalization, d max Indicates the maximum warning range radius of the logistics drone;

[0156] In this example, the maximum flight speed of the intrusion drone is set to 25m / s, and the maximum warning range radius of the logistics drone is set to 100m.

[0157] 313) Calculate the maximum reachable space of the invading drone’s trajectory;

[0158] According to the flight performance of the UAV, the trajectory of the invading UAV is analyzed according to the three flight stages of turning, acceleration and constant speed; the initial coordinates of the invading UAV are set as (p 0 ,q 0 ), with initial velocity u 0 , and take the position of the invading drone as the coordinate origin O, the positive direction of the Y axis is the speed direction, and the positive direction of the X axis is the right side of the invading drone body, and establish the Cartesian coordinate system OXY;

[0159] Turning flight phase:

[0160]

[0161]

[0162] In the formula, (p T ,q T ) represents the coordinates of the invading drone after turning, r represents the turning radius of the invading drone, Indicates the turning angle of the invading drone;

[0163] Accelerated flight phase:

[0164]

[0165]

[0166] In the formula, (p A ,q A ) represents the coordinates of the invading drone after accelerating its flight, t A Represents the flight time of the invading drone during the acceleration phase, m max Indicates the maximum acceleration of the invading drone;

[0167] Constant speed flight phase:

[0168]

[0169]

[0170] In the formula, (p U ,q U ) represents the coordinates of the invading drone after it flies at a constant speed, umax represents the maximum flight speed of the invading drone, and t U Indicates the uniform flight time of the invading drone;

[0171] The coordinates of the invading drone after uniform flight are input and fitted through the arc model Circle in python to obtain the coordinates of the center of the circle (p c ,q c ) and radius fc, and then obtain the corresponding state input information, the calculation formula is as follows:

[0172]

[0173] e t =h t -f c (18)

[0174]

[0175] In the formula, h t represents the distance between the logistics drone and the center of the maximum trajectory space at time t, e t represents the distance between the logistics drone and the maximum trajectory reachable space at time t, DE t Indicates whether the logistics drone is within the maximum trajectory reachable space at time t; t Normalize to get e′ t , the calculation formula is as follows:

[0176]

[0177] In this example, the maximum flight speed of the invading drone is 25m / s and the maximum acceleration is 6m / s 2 , prediction time interval is 2s.

[0178] Wherein, the step 33) specifically includes:

[0179] 331) Design a continuous reward function R for logistics drones 1 ;

[0180] Continuous reward function R 1 Including obstacle avoidance reward, path restoration reward and step length penalty, the specific calculation formula is as follows:

[0181]

[0182]

[0183] In the formula, re avoid represents the obstacle avoidance reward, re origin represents the path restoration reward, (x′ t ,y′ t ) represents the position of the logistics drone after avoiding obstacles at time t, (p t ,q t ) represents the position of the invading drone at time t, (x t ,y t ) represents the position of the logistics drone on the original path at time t, and the continuous reward function R 1 The specific calculation formula is as follows:

[0184] R 1 =λre avoid +(1-λ)re origin -1 (23)

[0185] In the formula, -1 represents the set logistics drone step size penalty, and λ represents the reward weight;

[0186] In this example, the reward weight is 0.3;

[0187] 332) Design the coefficient reward function R of the logistics drone 2 ;

[0188] Set the collision penalty between the logistics drone and the intrusion drone. The specific sparse reward settings are as follows:

[0189]

[0190] 4) Based on the drone state space information and action space information designed in step 3), the actor-critic network structure in the Deep Deterministic Policy Gradient (DDPG) algorithm is designed, and the algorithm is improved to obtain an improved Deep Deterministic Policy Gradient algorithm;

[0191] The deep deterministic policy gradient algorithm adopts the actor-critic network structure, which includes the corresponding online network and target network. The input information of the actor online network is the content contained in the state space set S, with 7 input neurons, 2 hidden layers in the middle layer, and fully connected. The number of neurons in each hidden layer is 256, and the ReLU activation function is used. The output layer is the acceleration and heading angle of the drone, with 2 neurons, and the tanh activation function is used. The structure design of the actor target network is the same as that of its online network. The input information of the critic online network is state information and action information. The input layer has 9 neurons, and 2 hidden layers in the middle layer are connected in a fully connected manner. The number of neurons in each hidden layer is 256, and the ReLU activation function is used. The output layer is the action value, with 1 neuron, and the structure design of the critic target network is the same as that of its online network.

[0192] 41) Design hybrid exploration strategies;

[0193] In order to enhance the early action exploration capability of the drone and maintain sample diversity, the dynamic ε-greedy method is used to select actions in the early stage of training, and noise interference is added in the later stage of training. The specific formula is as follows:

[0194]

[0195] In the formula, a t represents the action of the logistics drone at time t, uniform() represents a random number that obeys a uniform distribution, and a low and a high Indicates the upper and lower limits of the action, clip() means to change a t Restricted to a low and a highIn the range, μ represents the current policy function of the actor online network, θ represents the parameters of the actor online network, and μ(s t |θ μ ) indicates that the actor online network is in state s t The action generated by t represents the noise sampled from the Ornstein-Uhlenbeck process (OU noise for short), rand() represents a random number between (0,1), and ε is a decreasing greedy factor. The specific formula is as follows:

[0196]

[0197] In the formula, ε max , ε min Respectively represent the maximum and minimum values ​​of the greedy factor ε; t total represents the cumulative time step; Represents the value of the greedy factor ε which decreases after each time step;

[0198] 42) Design a dual experience playback mechanism;

[0199] In order to solve the problems of slow learning in the initial stage of network training and failed samples reducing the efficiency of network training, two experience replay pools EXP1 and EXP2 are set to store all training samples and training samples with reward values ​​greater than 1 in a single time step respectively. During the sampling of small batch samples, 90% of the total number of samples are randomly selected from EXP1, and the remaining required number of samples are randomly selected from EXP2.

[0200] 5) Based on the improved deep deterministic policy gradient algorithm in step 4), the real-time path control model of the logistics drone in step 3) is trained, and the trained model is used to realize the real-time path control of the logistics drone;

[0201] 51) Initialize actor and critic online network parameters, target network parameters, total number of training rounds, OU noise, reward discount factor, greedy factor, and experience replay pools EXP1 and EXP2;

[0202] 52) According to the pre-planned logistics distribution plan, the logistics drone flies according to the original plan and randomly selects a preset intrusion drone trajectory;

[0203] 53) When the intruding drone enters the warning range of the logistics drone, the maximum reachable range of the intruding drone’s trajectory at the future moment is calculated, and the distance relationship between the current position of the logistics drone and the maximum reachable range of the trajectory is determined;

[0204] 54) Combine the flight information of the logistics drone and the flight information of the intrusion drone it detects to obtain the status information s t , according to the hybrid exploration mechanism in the network, action a is selected t ;

[0205] 55) Logistics drone performs action a t , interacting with the environment is rewarded t and the next state s t+1 , the invading drone moves to the next location;

[0206] 56) Determine whether the number of samples in the experience replay pools EXP1 and EXP2 is greater than the minimum sample batch number N. If less than N, return to step 53); if greater than N, extract 90% of the minimum sample batch number N from the experience replay pool EXP1, and extract 10% of the minimum sample batch number N from the experience replay pool EXP2;

[0207] 57) Input the small batch of samples into the critic target network to obtain the target reward value w t , the policy gradient obtained by sampling is used to update the actor online network, and the critic online network is updated by minimizing the loss function. The specific formula is as follows:

[0208]

[0209] In the formula, w t represents the reward value obtained in the target network, Q(s t ,a t |θ Q ) means to convert s t and a t The actual value Q, θ obtained by jointly inputting the critic online network Q represents the critic online network parameters, w t The specific calculation formula is as follows:

[0210] w t =re t +γQ'(s t+1 ,μ'(s t+1 |θ μ' )|θ Q' ) (28)

[0211] In the formula, re t represents the immediate reward obtained at time t, γ represents the reward discount factor, Q'(s t+1 ,μ'(s t+1 |θ μ' )|θ Q' ) represents the target action and the next state s t+1The Q value obtained after inputting the target network together, μ'(s t+1 |θ μ' ) The next action value output by the actor target network;

[0212] 58) Update the network target parameters. The specific formula is as follows:

[0213]

[0214] Where ξ represents the coefficient used to adjust the update, ξ∈[0,1], θ Q' and θ Q Represent the critic online network and target network parameters, θ μ' and θ μ Represent the critic online network and target network parameters respectively;

[0215] 59) Determine the status s t+1 Is it a termination state? If not, return to step 53); if yes, determine whether the number of training rounds reaches the total number of rounds. If not, return to step 54). If reached, it means that the training is completed and the corresponding network parameters are saved;

[0216] 510) uses the model trained in step 59) to achieve real-time path control of the logistics drone.

[0217] In the example, the relevant parameters of the DDPG algorithm before and after the improvement are as shown in Table 2;

[0218] Table 2

[0219] parameter Value parameter Value Maximum number of training rounds exposures 4000 The number of samples in a small batch N 128 Experience replay pool EXP1, EXP2 capacity 30000 Maximum number of training steps per round 300 Reward discount factor γ 0.9 <![CDATA[Maximum Greedy Value ε max > 0.5 <![CDATA[Minimum greedy value ε min > 0.001 Decrement the greedy value ▽ε 0.0005 actor network learning rate 0.0001 Critics network learning rate 0.0005 Soft update adjustment coefficient ξ 0.001 Optimizer Adam

[0220] The logistics drone is trained according to the method proposed in the above example. The error change trend of each round is as follows: Figure 2 As shown, the average reward value changes as Figure 3 As shown in the figure, the error gradually decreases with the increase of iteration number until convergence, indicating that the network and reward value of DDPG and improved DDPG algorithms can converge after training, and the improved DDPG algorithm has smaller error and higher average reward value, indicating that the improved DDPG algorithm has better performance.

[0221] Figure 4After the two algorithms were trained, they were tested using a logistics drone. The figure shows that if the logistics drone flies along the original path, there is a risk of collision between the two drones from t+3 to t+4; the obstacle avoidance route generated by the DDPG algorithm can quickly distance itself from the invading drone, but the path changes greatly and the overall effect is poor; the obstacle avoidance path generated by the improved DDPG algorithm takes a long time to complete obstacle avoidance in the early stage, but the overall path is relatively smooth and closer to the actual flight situation. This shows that the local obstacle avoidance path generated by the model trained by the improved DDPG algorithm is better and has better results.

[0222] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims

1. A method for controlling the path of UAV water logistics distribution considering real-time obstacle avoidance, characterized in that: Here are the steps: 1) Generate a logistics distribution pre-planning plan based on the target ship distribution mission requirements and target ship planned trajectory; 2) Establish a kinematic model of logistics UAV in two-dimensional space; 3) Based on the kinematic model in step 2), the state space, action space and reward function of the logistics UAV are designed to build a real-time path control model for the logistics UAV; 4) According to the drone state space information and action space information designed in step 3), the actor-critic network structure in the deep deterministic policy gradient algorithm is designed, and the algorithm is improved to obtain an improved deep deterministic policy gradient algorithm; 5) Based on the improved deep deterministic policy gradient algorithm in step 4), the real-time path control model of the logistics drone in step 3) is trained, and the trained model is used to realize the real-time path control of the logistics drone; The deep deterministic policy gradient algorithm designed in the step 4) adopts an actor-critic network structure, which includes an online network and a target network corresponding thereto; the input information of the actor online network is the content contained in the state space set S, 7 input neurons are set, 2 hidden layers are set in the middle layer, and they are connected in a fully connected manner. The number of neurons in each hidden layer is 256, and the ReLU activation function is used; the output layer is the acceleration and heading angle of the drone, 2 neurons are set, and the tanh activation function is used, and the actor target network structure design is the same as its online network; the input information of the critic online network is state information and action information, the input layer has 9 neurons, 2 hidden layers are set in the middle layer, and they are connected in a fully connected manner. The number of neurons in each hidden layer is 256, and the ReLU activation function is used; the output layer is the action value, 1 neuron is set, and the critic target network structure design is the same as its online network; The step 4) specifically includes: 41) Design hybrid exploration strategies; In the early stage of training, the dynamic ε-greedy method is used to select actions, and noise interference is added in the later stage of training. The specific formula is as follows: In the formula, a t represents the action of the logistics drone at time t, uniform() represents a random number that obeys a uniform distribution, and a low and a high Indicates the upper and lower limits of the action, clip() means to change a t Restricted to a low and a high In the range, μ represents the current policy function of the actor online network, θ represents the parameters of the actor online network, and μ(s t θ μ ) indicates that the actor online network is in state s t The action generated by t represents the noise sampled from the Ornstein-Uhlenbeck process, rand() represents a random number between (0,1), and ε is a decreasing greedy factor. The specific formula is as follows: ε=max(ε max -t total ×▽e,e min ) (26) In the formula, ε max , ε min Respectively represent the maximum and minimum values ​​of the greedy factor ε; t total Represents the cumulative time step; ▽ε represents the value of the greedy factor ε which decreases after each time step; 42) Design a dual experience playback mechanism; Two experience replay pools EXP1 and EXP2 are set up to store all training samples and training samples with reward values ​​greater than 1 in a single time step respectively. During the sampling of small batch samples, 90% of the total number of samples are randomly selected from EXP1, and the remaining required number of samples are randomly selected from EXP2.

2. The method for controlling the path of UAV water logistics distribution considering real-time obstacle avoidance according to claim 1 is characterized in that: The step 1) specifically includes: selecting a meeting location close to the distribution center as the distribution point based on the collected target ship's planned track and distribution task requirements according to the principle of proximity, and calculating the flight speed required for the logistics drone's distribution according to the distance between the logistics drone's location and the meeting location and the time it takes for the target ship to sail to the meeting location; Assume that the starting position of the logistics drone is (x0, y0), the encounter position is (i, j), and the time required for the target ship to sail to (i, j) according to its planned track is ΔT. The calculation formula for the flight speed required for logistics drone delivery is as follows: Where V represents the required flight speed for logistics drone delivery.

3. The method for controlling the path of UAV water logistics distribution considering real-time obstacle avoidance according to claim 1 is characterized in that: The step 2) specifically includes: Set the coordinates of the logistics drone at time t to (x t ,y t ), v x,t 、v y,t are the speeds of the logistics drone in the x and y axes at time t, and the acceleration is n t , the heading angle is ω, and the calculation expression of the kinematic model of the logistics UAV is: In the formula, v x,t+1 、v y,t+1 are the speeds of the logistics drone in the x and y axes at time t+1, respectively. t+1 ,y t+1 ) is the coordinate of the UAV at time t+1.

4. The method for controlling the path of UAV water logistics distribution considering real-time obstacle avoidance according to claim 3 is characterized in that: The specific process of step 3) is as follows: 31) Design the state space of logistics drones; The design status information includes the internal status information of the logistics drone, the relevant information of the detected intrusion drone, and the maximum trajectory reachable space information of the intrusion drone, which is specifically expressed as follows: S=(v′ t ,ω t ,u′ t ,α t ,d′ t ,e′ t ,DE t ) (4) In the formula, S represents the state space set of logistics drones, v t ' represents the flight speed of the logistics drone at time t after normalization, ω t represents the heading angle of the logistics drone at time t, u t ' represents the flight speed of the invading drone at time t after normalization, α t represents the heading angle between the logistics drone and the intrusion drone at time t, d t ' represents the relative distance between the logistics drone and the intrusion drone at time t after normalization, e t ' represents the distance between the logistics drone and the maximum trajectory reachable space at time t after normalization, DE t Indicates whether the logistics drone is within the maximum trajectory reachable space at time t; 32) Design the action space of logistics drones; Based on the kinematic model of the drone in step 2), the action space of the drone is set to be related to the acceleration n and the heading angle ω; at the same time, considering the flight safety of the drone, the acceleration and heading angle should meet the corresponding constraints, and the specific settings are as follows: A=[n,ω] T ,n∈[-n max ,n max ],ω∈[-ω max ,oh max ] (5) In the formula, A represents the action space set of logistics drones, n max represents the maximum acceleration of the logistics drone, ω max Indicates the maximum heading angle of the logistics drone; 33) Design a reward function for logistics drones; The reward function consists of a continuous reward function and a sparse reward function. The specific expression is as follows: R=R1+R2 (6) In the formula, R represents the reward function, R1 represents the continuous reward part, and R2 represents the sparse reward part; The state space of the logistics UAV in the above step 31) is used as the input information of the real-time path control model of the logistics UAV, the action space of the logistics UAV in the above step 32) is used as the output information of the real-time path control model of the logistics UAV, and the reward function of the logistics UAV in the above step 33) is used as the calculation method of the reward obtained by the interaction between the logistics UAV and the environment during model training, thereby constructing the real-time path control model of the logistics UAV.

5. The method for controlling the path of UAV water logistics distribution considering real-time obstacle avoidance according to claim 4 is characterized in that: The step 31) specifically includes: 311) Determine the internal status information of the logistics drone; The flight speed and heading angle of the logistics drone are selected as part of the state information, and the flight speed is normalized. The specific calculation formula is as follows: In the formula, v′ t represents the flight speed of the logistics drone at time t after normalization, v t represents the flight speed of the logistics drone at time t, v max Indicates the maximum flight speed of the logistics drone; 312) Determine the information related to the detected intrusion drone; The flight speed, heading angle, and relative distance between the invading drone and the logistics drone are selected as the invading drone related information, and the flight speed of the invading drone is normalized. The specific calculation formula is as follows: In the formula, u t ' represents the normalized flight speed of the invading drone at time t, u t represents the flight speed of the invading drone at time t, u max Indicates the maximum flight speed of the invading drone; Assume the coordinates of the invading drone at time t to be (p t ,q t ), the relative distance calculation formula and normalization formula between the logistics drone and the intrusion drone are as follows: Where, d t represents the relative distance between the logistics drone and the intrusion drone at time t, d t ' represents the relative distance between the logistics drone and the intrusion drone at time t after normalization, d max Indicates the maximum warning range radius of the logistics drone; 313) Calculate the maximum reachable space of the invading drone’s trajectory; According to the flight performance of the UAV, the trajectory of the invading UAV is analyzed according to the three flight stages of turning, acceleration and constant speed; the initial coordinates of the invading UAV are set to (p0, q0), the initial speed is u0, and the position of the invading UAV is taken as the coordinate origin O, the positive direction of the Y axis is the speed direction, and the positive direction of the X axis is the right side of the invading UAV fuselage, and the Cartesian coordinate system OXY is established; Turning flight phase: In the formula, (p T ,q T ) represents the coordinates of the invading drone after turning, r represents the turning radius of the invading drone, Indicates the turning angle of the invading drone; Accelerated flight phase: In the formula, (p A ,q A ) represents the coordinates of the invading drone after accelerating its flight, t A Represents the flight time of the invading drone during the acceleration phase, m max Indicates the maximum acceleration of the invading drone; Constant speed flight phase: In the formula, (p U ,q U ) represents the coordinates of the invading drone after uniform flight, u max represents the maximum flight speed of the invading drone, t U Indicates the uniform flight time of the invading drone; The coordinates of the invading drone after uniform flight are input and fitted through the arc model Circle in python to obtain the coordinates of the center of the circle (p c ,q c ) and radius f c , and then get the corresponding state input information, the calculation formula is as follows: e t =h t -f c (18) In the formula, h t represents the distance between the logistics drone and the center of the maximum trajectory space at time t, e t represents the distance between the logistics drone and the maximum trajectory reachable space at time t, DE t Indicates whether the logistics drone is within the maximum trajectory reachable space at time t; t Normalize to get e′ t , the calculation formula is as follows:

6. The method for controlling the path of UAV water logistics distribution considering real-time obstacle avoidance according to claim 4 is characterized in that: The step 33) specifically includes: 331) Design the continuous reward function R1 for the logistics drone; The continuous reward function R1 includes obstacle avoidance reward, path restoration reward and step length penalty. The specific calculation formula is as follows: In the formula, re avoid represents the obstacle avoidance reward, re origin represents the path restoration reward, (x t ',y t ') represents the position of the logistics drone after avoiding obstacles at time t, (p t ,q t ) represents the position of the invading drone at time t, (x t ,y t ) represents the position of the logistics drone on the original path at time t, and the specific calculation formula of the continuous reward function R1 is as follows: R1=λre avoid +(1-λ)re origin -1 (23) In the formula, -1 represents the set logistics drone step size penalty, and λ represents the reward weight; 332) Design the logistics drone coefficient reward function R2; Set the collision penalty between the logistics drone and the intrusion drone. The specific sparse reward settings are as follows:

7. The method for controlling the path of UAV water logistics distribution considering real-time obstacle avoidance according to claim 1 is characterized in that: The specific process of step 5) is as follows: 51) Initialize actor and critic online network parameters, target network parameters, total number of training rounds, OU noise, reward discount factor, greedy factor, and experience replay pools EXP1 and EXP2; 52) According to the pre-planned logistics distribution plan, the logistics drone flies according to the original plan and randomly selects a preset intrusion drone trajectory; 53) When the intruding drone enters the warning range of the logistics drone, the maximum reachable range of the intruding drone's trajectory at the future moment is calculated, and the distance relationship between the current position of the logistics drone and the maximum reachable range of the trajectory is determined; 54) Combine the flight information of the logistics drone and the flight information of the intrusion drone it detects to obtain the status information s t , according to the hybrid exploration mechanism in the network, action a is selected t ; 55) Logistics drone performs action a t , interacting with the environment is rewarded t and the next state s t+1 , the invading drone moves to the next location; 56) Determine whether the number of samples in the experience replay pools EXP1 and EXP2 is greater than the minimum sample batch number N. If less than N, return to step 53); if greater than N, extract 90% of the minimum sample batch number N from the experience replay pool EXP1, and extract 10% of the minimum sample batch number N from the experience replay pool EXP2; 57) Input the small batch of samples into the critic target network to obtain the target reward value w t , the policy gradient obtained by sampling is used to update the actor online network, and the critic online network is updated by minimizing the loss function. The specific formula is as follows: In the formula, w t represents the reward value obtained in the target network, Q(s t ,a t θ Q ) means to convert s t and a t The actual value Q, θ obtained by jointly inputting the critic online network Q represents the critic online network parameters, w t The specific calculation formula is as follows: w t =re t +γQ'(s t+1 ,m'(s t+1 |θ μ' )i Q' ) (28) In the formula, re t represents the immediate reward obtained at time t, γ represents the reward discount factor, Q'(s t+1 ,μ'(s t+1 θ μ ')θ Q ') represents the target action and the next state s t+1 The Q value obtained after inputting the target network together, μ'(s t+1 θ μ ')The next action value output by the actor target network; 58) Update the network target parameters. The specific formula is as follows: Where ξ represents the coefficient used to adjust the update, ξ∈[0,1], θ Q' and θ Q Represent the critic online network and target network parameters, θ μ' and θ μ Represent the critic online network and target network parameters respectively; 59) Determine the status s t+1 Is it a termination state? If not, return to step 53); if yes, determine whether the number of training rounds reaches the total number of rounds. If not, return to step 54). If reached, it means that the training is completed and the corresponding network parameters are saved; 510) uses the model trained in step 59) to achieve real-time path control of the logistics drone.