A scheduling method for charging a UAV to charge a mission UAV in the air

Abstract

The present invention discloses a scheduling method for a charging drone to charge a mission drone in the air, which is used to schedule the charging drone to charge the mission drone that is performing a mission in the air. The method determines the remote charging or near-field charging mode according to the charging demand of the mission drone, and then performs classification modeling and calculation according to the number of participating charging drones, including establishing a model for the charging scheduling problem of a single charging drone and establishing a model for solving the charging scheduling problem of multiple charging drones; finally, the charging scheduling strategy of the charging drone is optimized based on the algorithm of deep reinforcement learning or multi-agent reinforcement learning, and the charging drone makes a decision based on the optimized strategy and the state of the current environment, and completes the charging of the mission drone according to the charging scheduling instruction. The present invention realizes the shortest task completion time of the mission drone, the fair charging of the mission drone, and the optimization of the charging scheduling strategy of the charging drone.

Description

Technical Field

[0001] The present invention belongs to flight scheduling technology, and in particular relates to a scheduling method for a charging UAV to charge a mission UAV in the air. Background Art

[0002] In recent years, unmanned aerial vehicles (UAVs) have been widely used in many fields due to their advantages such as low cost, convenient remote control, flexible deployment, and strong maneuverability. UAVs can carry various equipment and complete various tasks for many civilian applications, such as data collection, environmental monitoring, regional detection, communication, logistics, etc. Most small UAVs are powered by batteries. However, the battery capacity of such UAVs is limited and can only support short working and flight times. The operating time of most battery-powered UAVs ranges from tens of minutes to one hour. The range of UAV activities is therefore also limited. If the mission time exceeds the endurance of the UAV or the UAV needs to perform the mission in a larger range, the UAV must land to recharge or dispatch more UAVs to perform the mission. If the UAV needs to land on the ground to recharge, the mission being performed will be interrupted and the completion time of the mission will be delayed.

[0003] To significantly increase the working time of drones, an intuitive way is to equip drones with larger capacity batteries, but this will also significantly increase the weight and volume of drones. To overcome the above problems, online energy replenishment for a group of selected drones has become a promising option. In order to charge drones during missions, wireless charging solutions based on ground stations have been proposed, which can be divided into two types: near-field charging and far-field charging. With near-field charging, the drone needs to land on a ground charging station and charge through inductive coupling or magnetic resonance coupling. Far-field charging is usually performed by a laser beam, in which the drone hovers around the ground charging station and can be charged by an aligned laser beam from the ground station. Although existing ground charging station-based solutions can effectively extend the working time of drones, there are still some disadvantages. First, the ground station needs to be pre-deployed in certain fixed locations, which incurs additional costs when the drone application scenario or environment is dynamic. Second, the drone being charged must pause the ongoing mission and physically move to the charging station for energy replenishment. This will undoubtedly increase the mission time and energy consumption, and will also interrupt the mission being performed by the drone. Summary of the invention

[0004] Purpose of the invention: In order to solve the problem of insufficient power in UAV application scenarios and considering the rationality of wireless charging applications, the present invention provides a scheduling method for a charging UAV to charge a mission UAV in the air.

[0005] Technical solution: A method for scheduling a charging drone to charge a mission drone in the air, the method is used to schedule a charging drone to charge a mission drone in the process of performing a mission, and the scheduling method comprises the following steps:

[0006] (1) Determine the charging method based on the charging requirement information of the mission UAV, which includes the environmental obstacles, location, action and flight data of the charging UAV during the mission;

[0007] (2) A model for the charging scheduling problem of a single charging UAV is established based on the charging requirements and mission type of the mission UAV, and the charging scheduling strategy of a single charging UAV is optimized based on the DDPG algorithm;

[0008] (3) A model for the charging scheduling problem of multiple charging drones is established based on the charging requirements and mission types of the mission drones, and the charging scheduling strategies of multiple charging drones are optimized based on the MADDPG algorithm;

[0009] (4) With the goal of minimizing the time it takes for the task drone to complete the task, the charging scheduling strategy of the charging drone is optimized based on deep reinforcement learning or multi-agent reinforcement learning. The charging drone inputs the current state into the actor network obtained through training to make decisions, and completes the charging of the task drone according to the charging scheduling instructions output by the actor network.

[0010] Furthermore, the drone charging method described in step (1) includes near-field charging and far-field charging. The charging drone and the mission drone are equipped with corresponding wireless charging equipment. In near-field charging, the charging drone performs inductive coupling or magnetic resonance coupling wireless charging based on the wireless charging platform carried by the mission drone. In far-field charging, the charging drone performs energy conversion charging through RF radio frequency or laser.

[0011] In step (1) of the method, when there are n mission drones performing a mission, M = {M i |i=1,2,…,n}, the obstacles in the environment where the mission drone performs the mission are represented by B={B i |i=1,2,…,n}, the location where the drone performs its mission is represented by P={P i |i=1,2,…,n}, mission drone M i The action at time t is Actions include hovering or flying, and the flight direction during the flight is The flight speed is v;

[0012] The charging drone that performs the charging task is represented by C = {C i |i=1,2,…,m}, charging drone C i The action at time t is Actions include hovering or flying, and the flight direction during the flight is The flight speed is v.

[0013] The charging decision process of a single charging UAV to a task UAV in step (2) can be modeled as a Markov decision process, and its model is expressed as M = (S, A, P, R, γ). Its expression and parameter meanings are as follows:

[0014] S represents the state space of the environment. The state space is the set of actions that the charging drone takes based on the observed state. It is an infinite set, including all possible states during the charging task. The state space at time t is represented by S t ,Each state is composed of the current position coordinates of the charging drone, the remaining energy of the mission drone, the accumulated energy of the charging drone, and the working state of the mission drone, including the charging state and the moving state;

[0015] A represents the action space of the mission drone, which contains all possible actions that the charging drone may take during the charging mission. The action taken by the charging drone at time t is given by It means that each action of the charging drone contains a moving direction and a charging target. The action space of the moving direction is continuous, and the charging target is the object that the charging drone is currently or will provide charging for.

[0016] P represents the state transition probability, which is defined as the state transition probability of the given charging drone in the current state S at time t. t Take a specific action A t , the charging drone reaches the next possible state S t+1 The probability of P:S t ×A t →S t+1 , and different actions made by the charging drone will change the state of the charging drone in different ways;

[0017] R represents the reward function, and the reward function S t ×A t →R t+1 , the calculation is the charging execution action A t After that, its status changes from S t Convert to S t+1 The value of the reward received;

[0018] γ represents the discount factor, γ∈[0,1], which determines the impact of future rewards on current immediate rewards.

[0019] In step (2) and the above-mentioned model for the charging scheduling problem of a single charging drone, when the task drone completes the task and returns to the starting point, the task is completed, and the task drone M i The task completion time is T i , T i =Tm i +Tc i +Tw i , where Tm i Indicates the total time spent on the move, Tc i is the time spent on executing the task, Tw i Indicates the total time the mission drone spends waiting for the charging drone to charge;

[0020] The goal of charging scheduling strategy optimization is to minimize the task completion time. In order to achieve this goal, the charging drone takes action A. t After that, its status changes from S t Convert to S t+1 The calculation method of the reward value obtained includes the following process:

[0021] According to the pre-planned travel path and mission of the mission drone, it is set that the mission drone can be charged by the charging drone on time to keep working, and the reward function is defined to make the charging drone choose a reasonable charging target and moving direction; the reward value obtained by the charging drone at the execution action at time t is designed as follows:

[0022] r t =Rc t +Rm t -P l -P b

[0023] Rc t It represents the reward obtained by successfully charging the mission drone within the t period, and is defined as follows:

[0024]

[0025] If the charging drone does not charge the mission drone, the reward is zero. If it charges, a positive reward will be generated. Greater than Em i In the case of giving the task drone M i If charging, Rc will be rewarded t Zero; Rc t E in t is the charge at time t, Rc is a positive offset used to encourage the charging drone h to spend more time charging, f t Indicates the fairness of charging UAV to each mission UAV;

[0026] is the remaining energy of the mission UAV at time t, Em = {Em 1 ,Em 2 …Em n} represents the set of estimated minimum charging amounts required for the mission drone to complete the mission, represents the cumulative charge of the mission drone from time 1 to t, and the fairness factor f t Calculated by Jain fairness index, the calculation expression is as follows:

[0027] f t =w f fc t +(1-w f )fr t

[0028]

[0029]

[0030] w f is the weight value, Rc t Rm t Used to encourage the charging drone to move towards the mission drone's location;

[0031] Due to Rm t It is negatively correlated with the remaining energy level of the target mission drone, and the mission drone with less remaining power is assigned a high priority for charging. d and w e is a negative value; when the relative distance between the charging drone and the mission drone becomes smaller or the power of the charging target becomes smaller, Rm t The larger the Rm, the more it can guide the charging drone to approach the mission drone. t The calculation formula is as follows:

[0032]

[0033] Where D(i) t is the Euclidean distance M between the charging drone and the mission drone i ,and It is M i The remaining energy at time t, when the charging drone charges the mission drone, Rm t Set to zero;

[0034] P l If the charging drone fails to keep the remaining energy of any mission drone within the safety threshold E th Above, the penalty for charging drones, Eat is the average remaining power of the mission drone, p is a fixed positive value, and its expression is as follows:

[0035]

[0036] P b It is a penalty for the charging drone when it flies out of the boundary or hits an obstacle in the environment.

[0037] Step (3) in establishing a model for the charging scheduling problem of multiple charging drones, the process of simultaneously scheduling multiple charging drones is modeled as a Markov game M = (S, A, T, R, O, γ);

[0038] S represents the state space, and the state at time t is S t , including the status of all drones in the environment;

[0039] A represents the action space, and the actions taken by multiple charging drones at time t are given by express;

[0040] T represents the state transfer function, state transfer function T: In the environment, all charging drones take actions at the same time, which affects the environment;

[0041] R represents the reward function, the reward function The calculation is for charging drone C i Execute an action After that, its status changed from Convert to The value of the reward received;

[0042] O represents the observation set: Observation refers to the charging drone's perception of the environment from its own perspective. The charging drone makes decisions based on its own observation of the environment. is the set of observations of all charging drones at time t;

[0043] γ is the discount factor, γ∈[0,1], which determines the impact of future rewards on current immediate rewards.

[0044] In step (3) and the above-mentioned model of charging scheduling problem for multiple charging drones, the charging drone C i The reward design at time t is as follows:

[0045]

[0046] The definition is as follows:

[0047]

[0048] f t represents the fairness factor;

[0049] The calculation formula is as follows:

[0050]

[0051] Where D(i,j) t It is a charging drone C i and Mission Drone M j The Euclidean distance between It is a mission drone M j The remaining power at time t, w d and w e Yes Adjustment Negative coefficient of ;

[0052] P l If the charging drone fails to keep the remaining energy of any mission drone within the safety threshold E th Above, the penalty for charging drones; P b It is the penalty for charging the drone when it flies out of bounds or hits an obstacle in the environment.

[0053] Furthermore, step (4) optimizes the charging UAV scheduling strategy by training the actor network, optimizes the charging scheduling strategy of the charging UAV based on deep reinforcement learning or multi-agent reinforcement learning, optimizes the charging scheduling strategy of a single charging UAV based on deep reinforcement learning, and optimizes the charging scheduling strategy of multiple charging UAVs based on multi-agent reinforcement learning.

[0054] Beneficial effect: Compared with the traditional ground static charging method, the scheduling method for aerial charging of mission drones by the charging drone described in the present invention saves the time of the mission drone interrupting the mission for charging, and also expands and extends the coverage of the mission drone through aerial charging. It can be flexibly deployed and achieves the goal of minimizing the time to complete the mission and minimizing the energy consumption. It has the effect of flexibly deploying and allocating the charging time and charging location. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a scheduling flow chart of the method of the present invention;

[0056] Figure 2 This is a schematic diagram of the remote field charging method of the drone;

[0057] Figure 3 This is a schematic diagram of the near-field charging method of the drone;

[0058] Figure 4This is a scenario of a single-charger drone charging task in the embodiment;

[0059] Figure 5 This is a scenario of a multi-charging drone charging task in the embodiment;

[0060] FIG6( a ) is a comparison chart of the average task completion time of charging a single charging drone according to the present invention and a ground-based charging method in an embodiment;

[0061] FIG6( b ) is a comparison chart of average task completion time of charging multiple charging drones according to the present invention and a ground-based charging method in an embodiment;

[0062] FIG. 7( a ) is a diagram showing the flight trajectory of a single charging drone when performing a mission in an embodiment;

[0063] FIG. 7( b ) is a diagram showing the flight trajectories of multiple charging drones when performing a mission in an embodiment. DETAILED DESCRIPTION

[0064] In order to explain the technical solution disclosed by the present invention in detail, further description is made below in conjunction with the accompanying drawings.

[0065] The present invention provides a scheduling method for charging a mission drone in the air, which is intended to charge the drone without interrupting the mission. In the present invention, a drone with a mission is referred to as a mission drone, where the mission generally involves data communication and computing (e.g., drone edge computing). The mission drone can be wirelessly charged by another drone, i.e., a charging drone. The latest developments in wireless charging technology have made our design possible. For example, existing charging platforms based on magnetic resonance coupling provide charging for portable devices, radio frequency antenna array patches, and distributed laser charging (DLC) systems. Several benefits can be achieved by charging drones in the air. First, using charging drones instead of (or in addition to) ground charging stations can significantly improve flexibility and reduce the costs incurred by charging station deployment. Second, with the advantages of wireless charging and the high maneuverability of drones, the mission drones in the system can be charged on demand while moving or hovering. Third, since the mission drone can replenish energy online through the charging drone, the mission drone can continue to perform the mission without interrupting charging by landing for charging.

[0066] In order to charge the mission drones in the environment, the charging drones need to be scheduled. The charging drone needs to make decisions based on the environment and the power status of the charging drone and select a reasonable charging object. Reinforcement learning is often used to describe and solve the problem of how an intelligent agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment. Reinforcement learning can be used to solve the problem of drones making charging decisions. For a single charging drone, an algorithm based on deep deterministic policy gradient (DDPG) is used to solve this problem. In the charging scenario of multiple charging drones, an algorithm based on multi-agent deep deterministic policy gradient (MADDPG) can solve the scheduling strategy optimization problem of multiple charging drones.

[0067] The scheduling process of the method described in the present invention is as follows: Figure 1 As shown, the method is used to schedule a charging drone to charge a task drone that is performing a task. Both the charging drone and the task drone are equipped with wireless charging equipment. The scheduling method includes the following steps:

[0068] (1) determining a charging method according to the charging requirements of the mission UAV, wherein the charging requirements information includes environmental obstacles, location, action and flight information provided by the charging UAV for performing the mission;

[0069] (2) A model for the charging scheduling problem of a single charging UAV is established based on the charging requirements and mission type of the mission UAV, and the charging scheduling strategy of a single charging UAV is optimized based on the DDPG algorithm;

[0070] (3) A model for the charging scheduling problem of multiple charging drones is established based on the charging requirements and mission types of the mission drones, and the charging scheduling strategies of multiple charging drones are optimized based on the MADDPG algorithm;

[0071] (4) Based on deep reinforcement learning or multi-agent reinforcement learning, the charging scheduling strategy of the charging drone is optimized (training the actor network). The charging drone inputs the current state into the actor network obtained through training to make decisions, and completes the charging of the mission drone according to the charging scheduling instructions output by the actor network.

[0072] The specific process takes the dispatching of charging drones to charge mission drones that perform data collection tasks as an example.

[0073] (1) It is necessary to determine the wireless charging method for charging the mission drone.

[0074] The charging drone can replenish the power of the drone that is performing a mission: Use a charging drone with wireless charging function to charge the mission drone that needs to be charged. There are two charging methods according to the type of wireless charging: far-field charging and near-field charging. Figure 2 As shown in the figure, it means that the mission drone is flying near the charging drone, and the charging drone charges the mission drone through RF radio frequency charging or laser wireless charging. Figure 3 As shown, it means that the mission drone lands on the wireless charging platform of the charging drone for charging, mainly using inductive coupling or magnetic resonance coupling charging.

[0075] The choice of charging method should be determined by the size and energy consumption of the mission drone. The charging power of RF radio frequency in near-field charging and far-field charging is relatively small, and is generally suitable for charging small drones. The charging efficiency of laser charging can reach 30%, which can provide greater charging power. Due to the use of wireless charging, charging can also be performed while the drone is moving.

[0076] The two charging methods proposed in the present invention can be used to charge the data collection mission drone. In this embodiment, a far-field charging solution can be used to charge the mission drone using laser. The charging power is set to 90w, and the working power of the mission drone is 60w.

[0077] (2) It is necessary to determine the mission drone, the relevant parameters of the charging drone and the environment in which the mission is to be performed. Figure 4 and Figure 5 The working scenarios of a single charging drone and multiple charging drones are shown respectively, where MUAV refers to the data collection mission drone, CUAV refers to the charging drone, and PoI refers to the location where the mission drone performs the data collection mission. The mission drone sets off to perform the data collection mission and returns to the starting point after the mission is completed. The black dotted line is the trajectory of the mission drone performing the data collection mission, and the gray dotted line is the trajectory of the mission drone returning. However, due to insufficient mission power, if it is not charged, the mission of the mission drone will be interrupted and land early. It is necessary to dispatch a charging drone to charge it to ensure that it can complete the collection mission and return to the starting point. All environmental factors, including drones, obstacles, PoI, etc., can be defined according to the technology of the aforementioned technical solution. The location of PoI and obstacles, and the flight path of the mission drone to perform the collection mission need to be defined in advance. There are n mission drones performing the mission, expressed as M = {M i |i=1,2,…,n}. The obstacles in the environment where the mission drone performs the mission are represented by B={B i |i=1,2,…,n}, the location where the drone performs its mission is represented by P={P i |i=1,2,…,n}. Mission drone M i The action at time t is The action can be hovering to perform tasks, or flying. The flight direction during the flight is The flying speed is v. The charging drone that performs the charging task is represented by C = {Ci |i=1,2,…,m}. Charging drone C i The action at time t is The action can be hovering or flying. The flight direction during the flight is The flight speed is v.

[0078] (3) For the charging problem of a single charging drone, it is necessary to establish a model for the charging scheduling problem of a single charging drone. According to the modeling method of the aforementioned technical solution, a model of the Markov decision process is established. The specific state, action, reward and other designs are set according to the aforementioned technical solution. This process can be modeled as a Markov process M = (S, A, P, R, γ).

[0079] a) State space: The state space S of the environment is an infinite set, including all possible states during the charging task. The state at time t is S t Each state consists of the current position of the drone (two-dimensional coordinates), the remaining energy of the mission drone, the accumulated energy of the charging drone, and the working state of the mission drone, including the charging state and the moving state.

[0080] b) Action space: The action space A contains all possible actions that the charging drone may take during the charging mission. The action taken by the charging drone at time t is given by Each action of the charging drone contains a moving direction (a two-dimensional vector) and a charging target. The action space of the moving direction is continuous. The charging target is the object (task drone) that the charging drone is currently or will provide charging to.

[0081] c) State transition probability: The state transition probability is defined as: at time t, given the charging drone in the current state S t Take a specific action A t , the charging drone reaches the next possible state S t+1 The probability of P:S t ×A t →S t+1 Different actions taken by the charging drone will change the state of the charging drone in different ways, for example, when taking the action of moving to another location, all drone positions and the remaining energy of the mission drone will change.

[0082] d) Reward function: S t ×A t →R t+1 , the calculation is to charge the drone to take action C t After that, its state changes from S to S t+1The reward obtained. In the problem of the present invention, the goal is to minimize the task completion time. The travel path and mission of the task drone are pre-planned. If the task drone can be charged on time by the charging drone (keeping the energy state above the safety threshold), the reward of the charging drone at time t is designed as follows:

[0083] r t =Rc t +Rm t -P l -P b

[0084] Rc t It represents the reward obtained by successfully charging the mission drone within the t period, and is defined as follows:

[0085]

[0086] If the charging drone does not charge the mission drone, the reward is zero. If it charges, it will generate a positive reward. Greater than Em i In the case of giving the task drone M i If charging, Rc will be rewarded t Rc is zero. t E in t is the charge at time t. Rc is a positive offset used to encourage the charging drone h to spend more time charging. t Indicates the fairness of the charging drone charging each mission drone. If each mission drone is charged fairly, the charging drone will get a larger reward value. represents the remaining energy of the mission UAV at time t, Em = {Em 1 ,Em 2 …Em n} is the set of estimated minimum charge required for the mission drone to complete the mission. represents the cumulative charge of the mission drone from time 1 to t. Fairness factor f t It is calculated using the Jain Fairness Index as follows:

[0087] f t =w f fc t +(1-w f )fr t

[0088]

[0089]

[0090] w fIt is a weight that can be adjusted to improve the algorithm performance.

[0091] Rc t Rm t Used to encourage the charging drone to move towards the mission drone’s location. We believe that Rm t It is negatively correlated with the remaining energy level of the target mission UAV, that is, the mission UAV with less remaining power has a high priority for charging. d and w e When the relative distance between the charging drone and the charging target (task drone) becomes smaller or the power of the charging target becomes smaller, Rm t The larger the value, the more it can guide the charging drone closer to the mission drone. t The calculation formula is as follows:

[0092]

[0093] Where D(i) t is the Euclidean distance M between the charging drone and the mission drone i ,and It is M i The remaining energy at time t. When the charging drone charges the mission drone, Rm t Set to zero.

[0094] P l If the charging drone fails to keep the remaining energy of any mission drone within the safety threshold E th Above, the penalty for charging drones, Ea t is the average remaining power of the mission drone, p is a fixed positive value, and its expression is as follows:

[0095]

[0096] P b It is a penalty for the charging drone when it flies out of the boundary or hits obstacles in the environment (such as trees, buildings, no-fly zones).

[0097] e) Discount factor: γ∈[0,1] is defined as a discount factor that determines the extent to which future rewards affect current immediate rewards.

[0098] (4) For the charging problem of multiple charging drones, it is necessary to establish a model for the charging scheduling problem of multiple charging drones. According to the modeling method of the aforementioned technical solution, a Markov game model is established. The specific state, action, reward and other designs are set according to the aforementioned technical solution. The process of simultaneously scheduling multiple charging drones can be modeled as a Markov game M = (S, A, T, R, O, γ).

[0099] A) State space: The state at time t is S t . Includes the status of all drones in the environment.

[0100] B) Action Space: The actions taken by multiple charging drones at time t are given by express..

[0101] C) State transfer function: State transfer function T: In the environment, all charging drones take actions at the same time, affecting the environment.

[0102] D) Observation: Observation refers to the drone’s perception of the environment from its own perspective. The drone makes decisions based on its observation of the environment. is the set of observations of all charging drones at time t.

[0103] E) Reward function:

[0104] The reward function is What is obtained is the set of rewards obtained by all charging drones after taking action. The reward function under multi-drone scheduling is similar to that under single-drone scheduling. i The reward design at time t is as follows:

[0105]

[0106] The definition is as follows:

[0107]

[0108] f t The calculation method is the same as the calculation method of the model for the charging scheduling problem of a single charging drone mentioned above.

[0109] The calculation formula is as follows:

[0110]

[0111] Where D(i,j) t It is a charging drone C i and Mission Drone M j The Euclidean distance between It is a mission drone M j The remaining power at time t. w d and w e Yes Adjustment Negative coefficient of .

[0112] P lIf the charging drone fails to keep the remaining energy of any mission drone within the safety threshold E th The above calculation method for the penalty (i.e., negative reward) for the charging drone is the same as the calculation process for solving the charging model of the above single charging drone.

[0113] P b It is the penalty for charging the drone when it flies out of bounds or hits a block in the environment (e.g. a tree, building, no-fly zone).

[0114] F) Discount factor: γ∈[0,1] is defined as a discount factor that determines the extent to which future rewards affect current immediate rewards.

[0115] (5) The actor network is trained using deep reinforcement learning or multi-agent reinforcement learning, and the charging drone inputs the current state to the actor network obtained through training to make decisions. The working environment of the task drone is set as a two-dimensional square plane. When the charging drone and the task drone set off, the task and charging scheduling begins. Each training set of the reinforcement learning algorithm is the historical experience of performing the charging task. Each training segment of the reinforcement learning algorithm starts when the charging drone and the task drone (full energy) start working and ends when all the task drones return to the station (after completing the task) or the step size reaches the preset limit value. In the setting of the reinforcement learning algorithm, the actor and critic neural networks have 2 hidden layers, each with 64 neurons. The actor network uses Softmax as the activation function to generate actions. The experience replay buffer size can accommodate up to 600,000 records for updating weights, and 1024 records are sampled from the buffer for each round of update. The learning rate and discount factor are set to α = 0.01 and γ = 0.95, respectively. The algorithm ends after training 60,000 tasks. In the reward function, w f Set to 0.4, w d =-1,w e =-1.33.

[0116] In the method described in the present invention, the DDPG method is used to solve the single-charged drone, and the MADDPG algorithm is used to solve the multi-drone. Through the actor network obtained after training, the charging drone inputs its own state or observation of the environment, thereby obtaining the action to be taken. The algorithm is as follows, where MUAV is the task drone and CUAV is the charging drone. Code can also be written according to the relevant model modeling and algorithm to build a data collection environment. The algorithm is used to process the charging drone strategy optimization problem in the single-charged drone charging scenario and the multi-charged drone charging scenario. During the training process, the strategy gradually converges. Figures 6(a) and 6(b) show the comparison between the charging of a single-charged drone and the charging of a multi-charged drone in shortening the task drone execution time and the ground-based charging method (ground charging pile, charging car). Using a charging drone to charge has an advantage in the task completion time. Figures 7(a) and 7(b) respectively show the flight trajectory of the drone during the charging scheduling process in the single-charged drone charging and multi-charged drone environments. The charging drone can charge the task drone that performs the data collection task in a timely manner according to the learned strategy.

Claims

1. A method for scheduling a charging drone to charge a mission drone in the air, the method being used to schedule a charging drone to charge a mission drone in the process of performing a mission, characterized in that: The scheduling method comprises the following steps: (1) Determine the charging method based on the charging demand information of the mission UAV, which includes the environmental obstacles, location, action and flight data of the charging UAV during the mission. The charging methods include near-field charging and far-field charging. In near-field charging, the charging UAV is charged wirelessly by inductive coupling or magnetic resonance coupling based on the wireless charging platform carried by the mission UAV. In far-field charging, the charging UAV is charged by energy conversion through RF radio frequency or laser. When there are n mission drones performing the mission, this step is expressed as M = {M i |t=1,2,...,n},the obstacles in the environment where the mission drone performs the mission are represented by B={B i |i=1,2,...,n},the location where the drone performs its mission is represented by P={P i |i=1,2,...,n},mission drone M i The action at time t is Actions include hovering or flying, and the flight direction during the flight is The flight speed is v; The charging drone that performs the charging task is represented by C = {C i |i=1,2,...,m},charging drone C i The action at time t is Actions include hovering or flying, and the flight direction during the flight is The flight speed is v; (2) A model for the charging scheduling problem of a single charging UAV is established based on the charging requirements and mission type of the mission UAV, and the charging scheduling strategy of a single charging UAV is optimized based on the DDPG algorithm; The charging decision process of a single charging UAV to a mission UAV can be modeled as a Markov decision process, and its model is expressed as M = (S, A, P, R, γ). Its expression and parameter meanings are as follows: S represents the state space of the environment. The state space is the set of actions that the charging drone takes based on the observed state. It is an infinite set, including all possible states during the charging task. The state space at time t is represented by S t ,Each state is composed of the current position coordinates of the charging drone, the remaining energy of the mission drone, the accumulated energy of the charging drone, and the working state of the mission drone, including the charging state and the moving state; A represents the action space of the mission drone, which contains all possible actions that the charging drone may take during the charging mission. The action taken by the charging drone at time t is given by It means that each action of the charging drone contains a moving direction and a charging target. The action space of the moving direction is continuous, and the charging target is the object that the charging drone is currently or will provide charging for. P represents the state transition probability, which is defined as the state transition probability of the given charging drone in the current state S at time t. t Take a specific action A t , the charging drone reaches the next possible state S t+1 The probability of P:S t ×A t →S t+1 , and different actions made by the charging drone will change the state of the charging drone in different ways; R represents the reward function, and the reward function S t ×A t →R t+1 , the calculation is the charging execution action A t After that, its status changes from S t Convert to S t+1 The value of the reward received; γ represents the discount factor, γ∈[0,1], which determines the impact of future rewards on current immediate rewards; In this step, for the model of charging scheduling problem for a single charging drone, when the mission drone completes the mission and returns to the starting point, the mission is completed. i The task completion time is T i , T i =Tm i +Tc i +Tw i , where Tm i Indicates the total time spent on the move, Tc i is the time spent on executing the task, Tw i Indicates the total time the mission drone spends waiting for the charging drone to charge; Charging drone takes action A t After that, its status changes from S t Convert to S t+1 The calculation method of the reward value obtained includes the following process: According to the pre-planned travel path and mission of the mission drone, it is set that the mission drone can be charged by the charging drone on time to keep working, and the reward function is defined to make the charging drone choose a reasonable charging target and moving direction; the reward value obtained by the charging drone at the execution action at time t is designed as follows: r t =Rc t +Rm t -P l -P b Rc t It represents the reward obtained by successfully charging the mission drone within the t period, and is defined as follows: If the charging drone does not charge the mission drone, the reward is zero. If it charges, a positive reward will be generated. Greater than Em i In the case of giving the task drone M i If charging, Rc will be rewarded t Zero; Rc t E in t is the charge amount at time t, Rc is a positive offset used to encourage the charging drone to spend more time charging, f t Indicates the fairness of charging UAV to each mission UAV; is the remaining energy of the mission drone at time t, Em = {Em1, Em2...Em n } represents the set of estimated minimum charging amounts required for the mission drone to complete the mission, represents the cumulative charge of the mission drone from time 1 to t, and the fairness factor f t Calculated by Jain fairness index, the calculation expression is as follows: f t =w f fc t +(1-w f )fr t wf is the weight value, Rc t Rm t Used to encourage the charging drone to move towards the mission drone's location; Due to Rm t It is negatively correlated with the remaining energy level of the target mission drone, and the mission drone with less remaining power is assigned a high priority for charging. d and w e is a negative value; when the relative distance between the charging drone and the mission drone becomes smaller or the power of the charging target becomes smaller, Rm t The larger the Rm, the more it can guide the charging drone to approach the mission drone. t The calculation formula is as follows: Where D(i) t is the Euclidean distance M between the charging drone and the mission drone i ,and It is M i The remaining energy at time t, when the charging drone charges the mission drone, Rm t Set to zero; P l If the charging drone fails to keep the remaining energy of any mission drone within the safety threshold E th Above, the penalty for charging drones, Ea t is the average remaining power of the mission drone, p is a fixed positive value, and its expression is as follows: P b It is a penalty for charging drones when they fly out of bounds or hit obstacles in the environment; (3) A model for the charging scheduling problem of multiple charging drones is established based on the charging requirements and mission types of the mission drones, and the charging scheduling strategies of multiple charging drones are optimized based on the MADDPG algorithm; In this step, in the model of establishing the charging scheduling problem of multiple charging drones, the process of simultaneously scheduling multiple charging drones is modeled as a Markov game M = (S, A, T, R, O, γ); S represents the state space, and the state at time t is S t , including the status of all drones in the environment; A represents the action space, and the actions taken by multiple charging drones at time t are given by express; T represents the state transfer function, state transfer function T: In the environment, all charging drones take actions at the same time, which affects the environment; R represents the reward function, the reward function The calculation is for charging drone C i Execute an action After that, its status changed from Convert to The value of the reward received; O represents the observation set: Observation refers to the charging drone's perception of the environment from its own perspective. The charging drone makes decisions based on its own observation of the environment. represents the observation set of all charging drones at time t; γ is the discount factor, γ∈[0,1], which determines the impact of future rewards on current immediate rewards; In this step, in the model of charging scheduling problem for multiple charging drones, the charging drone C i The reward design at time t is as follows: The definition is as follows: f t represents the fairness factor; The calculation formula is as follows: Where D(i, j) t It is a charging drone C i and the Euclidean distance between the mission drone Mj, It is a mission drone M j The remaining power at time t, w d and w e Yes Adjustment Negative coefficient of ; P l If the charging drone fails to keep the remaining energy of any mission drone within the safety threshold E th Above, the penalty for charging drones; P b It is the penalty for charging the drone when it flies out of bounds or hits an obstacle in the environment; (4) With the goal of minimizing the time it takes for the task drone to complete the task, the charging scheduling strategy of the charging drone is optimized based on deep reinforcement learning or multi-agent reinforcement learning. The charging drone inputs the current state into the actor network obtained through training to make decisions, and completes the charging of the task drone according to the charging scheduling instructions output by the actor network.

2. The method for scheduling a charging drone to charge a mission drone in the air according to claim 1, characterized in that: Step (4) optimizes the charging UAV scheduling strategy by training the actor network, optimizes the charging scheduling strategy of the charging UAV based on deep reinforcement learning or multi-agent reinforcement learning, optimizes the charging scheduling strategy of a single charging UAV based on deep reinforcement learning, and optimizes the charging scheduling strategy of multiple charging UAVs based on multi-agent reinforcement learning.

Images