Unmanned aerial vehicle charging coordinated scheduling method and system based on reinforcement learning

Abstract

The invention provides an unmanned aerial vehicle charging coordinated scheduling method and system based on reinforcement learning, and belongs to the technical field of unmanned aerial vehicle energy management and scheduling optimization. Comprising the steps of defining an unmanned aerial vehicle charging and battery swapping hybrid power supply system model and an unmanned aerial vehicle network scene based on the unmanned aerial vehicle network scene; the method comprises the steps of solving a charging decision of each unmanned aerial vehicle based on an unmanned aerial vehicle charging hybrid power supply system model, establishing a Markov decision process framework, and modeling a charging coordinated scheduling problem into a reinforcement learning problem; performing decision optimization by adopting a double-delay depth deterministic strategy gradient TD3PG algorithm; and with the goal of minimizing the time of reaching the income threshold, each unmanned aerial vehicle intelligent agent learns an optimal charging decision strategy according to global state information, and outputs a charging decision of each unmanned aerial vehicle in real time through an actor-commentator Actor-Critic network architecture, thereby realizing dynamic balance of service quality, energy cost and time efficiency, and improving the efficiency of the unmanned aerial vehicle. And collaborative optimization is carried out among the unmanned aerial vehicles through sharing rewards.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) energy management and scheduling optimization technology, and in particular to a UAV charging and swapping coordination scheduling method and system based on reinforcement learning. Background Technology

[0002] With the rapid development of drone technology, drones have demonstrated enormous potential as aerial base stations in scenarios such as emergency communication, remote area coverage, and temporary hotspot deployment. However, the limited endurance of drones has become a key bottleneck restricting their large-scale application. Traditional single charging modes suffer from long charging times and prolonged service interruptions, while single battery swapping modes, although fast, are costly and have low resource utilization. Therefore, a hybrid power supply strategy that combines charging and battery swapping has become an important direction for improving the operational efficiency of drone networks.

[0003] Hybrid power supply scenarios combine charging stations and battery swapping stations. When the drone has sufficient power and time permits, it chooses to charge at a lower-cost charging station; when the mission is urgent or the power is severely depleted, it chooses a battery swapping station to quickly replace the battery and restore service. This model requires comprehensive consideration of multiple factors, including the drone's power status, the capacity of charging and swapping facilities, and service quality requirements.

[0004] Currently, drone energy management technology has a certain research foundation, but existing solutions mainly employ rule-based heuristic algorithms or traditional optimization methods, which are insufficient to handle dynamic and complex real-world scenarios. When multiple drones operate collaboratively, their charging and swapping decisions influence each other, requiring consideration of multiple constraints such as infrastructure resource competition and network coverage continuity. If the decision-making strategy is too conservative, frequent charging and swapping will lead to low service efficiency; if the decision is too aggressive, service interruptions or even safety incidents may occur due to depleted power. Furthermore, achieving a dynamic balance between multiple objectives such as service quality, energy cost, and time efficiency remains a core problem that urgently needs to be solved in this field. Summary of the Invention

[0005] In view of this, the present invention provides a method and system for coordinating and scheduling charging and swapping of unmanned aerial vehicles (UAVs) based on reinforcement learning. By utilizing dynamic environmental information such as the real-time status of UAVs and the distribution of charging and swapping facilities, and through reinforcement learning algorithms, the charging and swapping strategy is optimized from the perspective of UAV energy management and collaborative decision-making, taking into account multiple constraints such as service quality assurance, energy consumption, and time cost. This improves service quality and energy utilization efficiency, and reduces operating costs and service interruption time.

[0006] The technical solution adopted by the embodiments of the present invention to solve its technical problem is as follows:

[0007] A reinforcement learning-based method for coordinating and scheduling charging and swapping of unmanned aerial vehicles (UAVs) includes:

[0008] Step S1: Define a hybrid power supply system model for drone charging and swapping based on the drone network scenario. The hybrid power supply system model includes a drone energy consumption model, a wireless charging service model, a battery swapping service model, a backup battery rotation model, and a revenue model. The constraints of the hybrid power supply system model include drone power safety constraints and charging and swapping facility capacity constraints. The drone network scenario includes an airborne base station, an integrated power replenishment station, and a battery monitoring module for each drone.

[0009] Step S2, based on the UAV charging and swapping hybrid power supply system model, solve for the charging and swapping decisions of each UAV, including:

[0010] A Markov decision process framework is established to model the charging and swapping coordination and scheduling problem as a reinforcement learning problem;

[0011] A dual-delay deep deterministic policy gradient (TD3PG) algorithm is employed for decision optimization. With the objective of minimizing the time to reach the reward threshold, each UAV agent learns the optimal charging / swapping decision strategy based on its state information. An Actor-Critic architecture is used to output the charging / swapping decisions of each UAV in real time. During training, an experience replay mechanism is used to store historical interaction data between the UAVs and the integrated charging station, and network parameters are updated through batch sampling. The Actor-Critic architecture introduces a local Actor-Critic network and a target Actor-Critic network employing a dual-Q network mechanism. The target Actor-Critic network performs soft updates based on the local Actor-Critic network. UAVs achieve collaborative optimization by sharing reward signals. The local Actor-Critic network is inversely optimized by calculating the Q-value loss between the local and target Critic networks, and the local Actor-Critic network outputs the actions of each UAV.

[0012] Drones can collaborate and optimize by sharing rewards.

[0013] Preferably, in step S1, let This indicates the number of drones in the airborne base station, and the integrated power replenishment station. N wl N w N B These represent the number of wireless charging piles, the number of wired charging ports, and the total number of backup batteries in the integrated power replenishment station, respectively.

[0014] ;

[0015] In the formula, This indicates that the drone is in working state, wireless charging state, and battery replacement state, respectively; II(·) is the indicator function;

[0016] ; ;

[0017] In the formula, This represents the number of wireless charging stations in use at time t. This represents the number of drones that used wireless charging to replenish power at time t;

[0018] ;

[0019] In the formula, This represents the number of spare batteries in use at time t. This represents the number of drones that use battery swapping for power replenishment at time t.

[0020] Preferably, the UAV charging and swapping hybrid power supply system model in step S1 is as follows:

[0021] In the drone energy consumption model, the energy consumption at time t+1 under operating conditions is represented as:

[0022] ;

[0023] In the formula, This represents the battery status of the i-th drone at time t. This indicates the power consumption of the drone per unit time.

[0024] In the wireless charging service model, the drone's battery level update is represented as:

[0025] ;

[0026] In the formula, E wl Indicates wireless charging rate; SoC max Indicates the maximum battery capacity;

[0027] In the battery swapping service model, the drone's battery level update is represented as follows:

[0028] ;

[0029] Add dynamic update rules for system state in the backup battery rotation model:

[0030] For wireless charging piles in integrated power replenishment stations:

[0031] ;

[0032] In the formula, This represents the total number of wireless charging pads at time t; This indicates the number of wireless charging pads currently in use at time t. Indicates the number of wireless charging pads available at time t;

[0033] For batteries in the environment:

[0034] ;

[0035] In the formula, This represents the total number of backup batteries at time t; This indicates the number of depleted batteries that have been replaced from the drone at time t, and the number that enters the wired charging replenishment sequence. This indicates the number of fully charged batteries that have been fully charged and ready for use through a wired charging power replenishment sequence.

[0036] Revenue Model: The cost of SoC from 0 to 1 consists of three types of expenses, denoted as the cost set. , This indicates the unit electricity cost payment for wireless charging. This indicates the cost per unit of electricity required for wired charging of the battery. This indicates the labor cost required for each battery replacement.

[0037] When the drone is in operation, the benefit generated per time slot t is: It also consumes electricity, and after a certain period of time... The battery level dropped to the charging threshold. hour, The drone sends a charging request to the system.

[0038] The system assigns a power-up method to drones that need power-up based on their battery level and the availability of equipment at the power-up station, including wireless charging and battery swapping.

[0039] Define a revenue model for the drone, setting a total revenue target P in the environment, based on the drone's current state. Define the benefit of a single drone at time t. :

[0040] ;

[0041] Further define the cumulative revenue of a single drone up to time t as:

[0042] ;

[0043] The total cumulative revenue of all drones in the cluster is:

[0044] ;

[0045] Define the benefit attainment indicator function:

[0046] ;

[0047] In the formula, P is the target revenue threshold, and T is the maximum working time step; when When t represents the time when the drone swarm has achieved the target benefit; otherwise, the swarm is still in the benefit accumulation stage; the goal is to minimize the time to reach the target benefit threshold of P.

[0048] Preferably, the constraints in step S1 include:

[0049] Drone power safety constraints: ;

[0050] Charging and battery swapping facility capacity constraints:

[0051] st: , .

[0052] Preferably, step S2 includes:

[0053] Using the Markov Decision Process (MDP) framework, the interaction between the state space, action space, and reward is modeled as a mathematical closed loop of state → policy → action → state transition → reward → policy optimization.

[0054] Define action space ;

[0055] Define state space :

[0056] ;

[0057] in, This represents the average battery level of all drones at time t. This represents the standard deviation of the battery capacity of all drones. Indicates the percentage of drones in operation. This indicates the occupancy rate of wireless charging stations. Indicates the battery swapping queue occupancy rate. Indicates the revenue completion rate. It represents the time progress and is the ratio of the current working time of the system to the working time threshold T;

[0058] Define a compound reward function R t The expression is:

[0059] ;

[0060] In the formula: work rewards α1 is the work reward coefficient; a penalty is imposed when the battery level of all drones falls below a safety threshold. Where α2 is the safety penalty coefficient; It is the final reward for the drone swarm to achieve ultimate benefit, where α3 is the task completion reward coefficient; α4 is the time penalty coefficient;

[0061] α1, α2, α3, and α4 are obtained through parameter optimization;

[0062] In the Actor-Critic architecture, the local Actor network is based on a state space. Output Action :

[0063] ;

[0064] ;

[0065] action It acts on the environment, directly changing the state space of the environment through the state transition function T. ,get :

[0066] ;

[0067] action The state generated after inputting the environment ,action Reward R t and state space The data is stored in the experience pool; the calculations in the experience pool are input into two local Critic networks, which are then processed based on their current states. Reward R t and actions Calculate the current Q-value reward ( );

[0068] Target Actor Networks by providing actions Add noise, output action The two-target Critic network is based on state space. Reward R t+1 and actions Calculate Q-value reward And further calculate the target Q value. , ,in It is a discount factor;

[0069] Calculate the loss criteria loss :

[0070] ;

[0071] In the formula, F mse () mean square error;

[0072] Based on the loss result, the local Actor network is optimized backwards using policy gradient, and the local Actor network resets its actions in the direction that yields higher rewards. After the loss converges, the local Actor network outputs the action corresponding to the minimum loss result. .

[0073] A reinforcement learning-based UAV charging and battery swapping coordination scheduling system, implementing the aforementioned reinforcement learning-based UAV charging and battery swapping coordination scheduling method, the system includes:

[0074] A modeling unit is established to define a hybrid power supply system model for drones based on a drone network scenario. This model includes a drone energy consumption model, a wireless charging service model, a battery swapping service model, a backup battery rotation model, and a revenue model. The constraints of this model include drone power safety constraints and charging / swapping facility capacity constraints. The drone network scenario includes an aerial base station, an integrated power replenishment station, and battery monitoring modules for each drone.

[0075] The solution unit is used to solve the charging and swapping decisions of each UAV based on the UAV charging and swapping hybrid power supply system model. This includes: establishing a Markov decision process framework to model the charging and swapping coordination and scheduling problem as a reinforcement learning problem; using the dual-delay deep deterministic policy gradient (TD3PG) algorithm for decision optimization; with the objective of minimizing the time to reach the reward threshold, each UAV agent learns the optimal charging and swapping decision strategy based on its state information, and outputs the charging and swapping decisions of each UAV in real time through an actor-critic architecture; during training, historical interaction data between UAVs and integrated power replenishment stations is stored using an experience replay mechanism, and network parameters are updated through batch sampling; the actor-critic architecture introduces a local actor-critic network and a target actor-critic network using a dual-Q network mechanism. The target actor-critic network performs soft updates based on the local actor-critic network, and UAVs achieve collaborative optimization by sharing reward signals. The local actor network is inversely optimized by calculating the Q-value loss between the local and target actor networks, and the local actor network outputs the actions of each UAV.

[0076] As can be seen from the above technical solution, the present invention provides a method and system for coordinated scheduling of UAV charging and swapping based on reinforcement learning. Firstly, a hybrid power supply system model for UAV charging and swapping is defined based on the UAV network scenario. This model includes a UAV energy consumption model, a wireless charging service model, a swapping service model, a backup battery rotation model, and a revenue model. Constraints on the hybrid power supply system model include UAV power safety constraints and charging / swapping facility capacity constraints. The UAV network scenario includes an airborne base station, an integrated power replenishment station, and battery monitoring modules for each UAV. Based on the hybrid power supply system model, the charging and swapping decisions of each UAV are solved, establishing a Markov decision process framework and modeling the coordinated scheduling problem as a reinforcement learning problem. The decision optimization is performed using the dual-delay deep deterministic policy gradient (TD3PG) algorithm. With the objective of minimizing the time to reach the revenue threshold, each UAV agent learns the optimal charging and swapping decision strategy based on its state information. Through an actor-critic network architecture, the charging and swapping decisions of each UAV are output in real time, achieving a dynamic balance between service quality, energy cost, and time efficiency. UAVs collaborate to optimize by sharing rewards. This invention utilizes dynamic environmental information such as the real-time status of drones and the distribution of charging and battery swapping facilities. Through reinforcement learning algorithms, it achieves intelligent collaborative scheduling of charging at charging stations and battery swapping at battery swapping stations, taking into account multiple constraints such as service quality assurance, energy consumption, and time cost. It explores the most suitable hybrid power supply decision strategy for drones in dynamic and complex scenarios. Attached Figure Description

[0077] Figure 1 This is a flowchart of a reinforcement learning-based UAV charging and swapping coordination scheduling method according to the present invention.

[0078] Figure 2 A scene depicting the charging and battery swapping of a drone.

[0079] Figure 3 Update the status graph for drones and charging stations.

[0080] Figure 4 This is a schematic diagram illustrating the reward convergence of the DDPG and TD3PG algorithms.

[0081] Figure 5 This diagram illustrates the number of steps required for a drone swarm to achieve the target using the TD3PG and DDPG optimization algorithms.

[0082] Figure 6 This is a schematic diagram of the framework principle of the TD3PG reinforcement learning algorithm. Detailed Implementation

[0083] The technical solution and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0084] This invention optimizes charging and battery swapping strategies from the perspective of UAV energy management and collaborative decision-making. Utilizing dynamic environmental information such as the real-time status of the UAV and the distribution of charging and battery swapping facilities, it autonomously learns the optimal strategy through a deep reinforcement learning algorithm. While ensuring service quality, it minimizes energy costs and explores the most suitable hybrid power supply decision-making mechanism for UAVs in dynamic and complex scenarios, improving the system's intelligence, adaptability, and overall operational efficiency. (Reference) Figure 1 As shown, this invention provides a reinforcement learning-based method for coordinating and scheduling charging and swapping of unmanned aerial vehicles (UAVs). The implementation steps include:

[0085] Step S1: Within the drone network service area, deploy N drones as aerial base stations and configure a battery swapping station (BCSS) integrating wireless charging and battery swapping stations; define a drone charging / swapping hybrid power supply system model based on the drone network scenario, including a drone energy consumption model, a wireless charging service model, a battery swapping service model, a backup battery rotation model, and a revenue model; the constraints of the drone charging / swapping hybrid power supply system model include drone power safety constraints and charging / swapping facility capacity constraints; refer to... Figure 2 and Figure 3 As shown, the drone network scenario includes an airborne base station, an integrated power replenishment station, and a battery monitoring module for each drone; the battery monitoring module is used to monitor status information such as remaining power, location, and service load in real time.

[0086] Step S2, based on the UAV charging and swapping hybrid power supply system model, solve for the charging and swapping decisions of each UAV, including:

[0087] Step S21: Establish a Markov decision process framework and model the charging and swapping coordination scheduling problem as a reinforcement learning problem;

[0088] Step S22: The decision optimization is performed using the dual-delay deep deterministic strategy gradient TD3PG algorithm. With the goal of minimizing the time to reach the revenue threshold, each UAV agent learns the optimal charging and swapping decision strategy based on its state information. Through the actor-critic network architecture, the charging and swapping decisions of each UAV are output in real time to achieve a dynamic balance between service quality, energy cost and time efficiency.

[0089] Step S23: During training, historical interaction data between the UAV and the integrated power station is stored using an experience replay mechanism, and network parameters are updated through batch sampling. A target network and dual-Q network mechanism are introduced into the Actor-Critic architecture (i.e., two independent Critic networks are used, and their minimum value is taken as the target Q value. At the same time, the target network parameters are slowly updated through a soft update mechanism to stabilize the training process and reduce Q value overestimation). That is, the network framework has one local Actor, one target Actor, two local Critic networks, and two target Critic networks. The UAVs achieve collaborative optimization by sharing reward signals.

[0090] In step S1, let This indicates the number of drones in the airborne base station, and the integrated power replenishment station. N wl N w N B These represent the number of wireless charging piles, the number of wired charging ports, and the total number of backup batteries in the integrated power replenishment station, respectively.

[0091] ;

[0092] In the formula, This indicates that the drone is currently in one of three states: working, wireless charging, or battery replacement; II(·) is an indicator function (meaning that the expression = 1 if the requirements in parentheses are met, otherwise it is 0).

[0093] ;

[0094] In the formula, This represents the number of wireless charging stations in use at time t. This represents the number of drones that used wireless charging to replenish power at time t;

[0095] ;

[0096] In the formula, This represents the number of spare batteries in use at time t. This represents the number of drones that use battery swapping for power replenishment at time t.

[0097] Model of a hybrid power supply system for drones that combines charging and battery swapping:

[0098] In the drone energy consumption model, the energy consumption at time t+1 under operating conditions is represented as:

[0099] ;

[0100] In the formula, This represents the battery status of the i-th drone at time t. This indicates the power consumption of the drone per unit time.

[0101] In the wireless charging service model, the drone's battery level update is represented as:

[0102] ;

[0103] In the formula, E wl Indicates the wireless charging rate (the maximum charging power that a single dock can provide per unit time under wireless charging); SoC max Indicates the maximum battery capacity;

[0104] In the battery swapping service model, the drone's battery level update is represented as follows:

[0105] ;

[0106] Add dynamic update rules for system state in the backup battery rotation model:

[0107] For wireless charging piles in integrated power replenishment stations:

[0108] ;

[0109] In the formula, This represents the total number of wireless charging pads at time t; This indicates the number of wireless charging pads currently in use at time t. Indicates the number of wireless charging pads available at time t;

[0110] For batteries in the environment:

[0111] ;

[0112] In the formula, This represents the total number of backup batteries at time t; This indicates the number of depleted batteries that have been replaced from the drone at time t, and the number that enters the wired charging replenishment sequence. This indicates the number of fully charged batteries that have been fully charged and ready for use through a wired charging power replenishment sequence.

[0113] Revenue Model: The cost of SoC from 0 to 1 consists of three types of expenses, denoted as the cost set. , This indicates the unit electricity cost payment for wireless charging. This indicates the cost per unit of electricity required for wired charging of the battery. This indicates the labor cost required for each battery replacement.

[0114] When the drone is in operation, the benefit generated per time slot t is: It also consumes electricity, and after a certain period of time... The battery level dropped to the charging threshold. hour, The drone triggers a charging request to the system; the system allocates a charging method (wireless charging or battery swapping) to the drone that needs charging based on the drone's battery level and the availability of equipment in the charging station.

[0115] The system assigns a power-up method to drones that need power-up based on their battery level and the availability of equipment at the power-up station, including wireless charging and battery swapping.

[0116] Define a revenue model for the drone, setting a total revenue target P in the environment, based on the drone's current state. Define the benefit of a single drone at time t. :

[0117] ;

[0118] Further define the cumulative revenue of a single drone up to time t as:

[0119] ;

[0120] The total cumulative revenue of all drones in the cluster is:

[0121] ;

[0122] Define the benefit attainment indicator function:

[0123] ;

[0124] In the formula, P is the target revenue threshold, and T is the maximum working time step; when When t represents the time when the drone swarm has achieved the target benefit; otherwise, the swarm is still in the benefit accumulation stage; the goal is to minimize the time to reach the target benefit threshold of P.

[0125] The constraints include:

[0126] Drone power safety constraints: ;

[0127] Charging and battery swapping facility capacity constraints:

[0128] st: , .

[0129] The specific implementation of step S2 includes:

[0130] Using the Markov Decision Process (MDP) framework, a core mathematical framework for reinforcement learning, we essentially make decisions based on the current state, and the future state depends only on the current state and actions, independent of history. Based on this, the interaction between the state space, action space, and reward is modeled as a mathematical closed loop: "State → Policy → Action → State Transition → Reward → Policy Optimization".

[0131] Define the action space as , is the decision space of the intelligent agent;

[0132] State space is the perception space of an intelligent agent; it is the "set of current states" of the environment.

[0133] ;

[0134] In the formula, This represents the average battery level of all drones at time t. This represents the standard deviation of the battery capacity of all drones. Indicates the percentage of drones in operation. This indicates the occupancy rate of wireless charging stations. Indicates the battery swapping queue occupancy rate. This represents the revenue completion rate (the ratio of the current revenue to the target revenue threshold). It represents the time progress (the ratio of the current working time of the system to the working time threshold).

[0135] Composite reward function R t This function comprehensively considers service revenue, safety constraints, charging costs, and time efficiency; its mathematical expression is:

[0136] ;

[0137] In the formula: α1 is the work reward coefficient, which incentivizes more drones to participate in mission execution; a penalty is imposed when the battery level of all drones falls below a safety threshold. α2 is a safety penalty coefficient used to suppress low-power operation and reduce the risk of task interruption. It is the final reward for the drone swarm to achieve the ultimate benefit, where α3 is the task completion reward coefficient, used to incentivize drones to achieve the global benefit goal. ,in These are time penalty coefficients, which constrain the time delay of task execution by fixing the penalty term, thus pushing the system to complete the objective within a preset time. Specifically, the optimal parameters for the above four coefficients were found through a grid search during the experiment.

[0138] Actor networks are decision-making modules in reinforcement learning, based on state space. Output Action The Critic network is a value evaluation module for reinforcement learning agents, based on state. and actions Evaluate the merits of the decision.

[0139] The local Actor network will be based on the current state. Output a collection of specific actions Different drones have different recharging thresholds. This parameter is used to guide the drone's charging decisions:

[0140]

[0141] action It acts on the environment, directly changing the state space of the environment through the state transition function T. :

[0142] ;

[0143] For example, an action triggers charging for the drone, increasing its battery level, while simultaneously occupying the wireless charger at the charging station; the environment outputs an instant reward R based on changes in action and state. t If the action allows the drone to charge in time without consuming too many resources, the environment will give a positive reward; if the action causes congestion at the charging station, the environment will give a negative reward.

[0144] action The state generated after inputting the environment ,action Reward R t and state space The data is stored in the experience pool; the calculations in the experience pool are input into two local Critic networks, which are then processed based on their current states. Reward R t and actions Calculate the current Q-value reward ( The calculation of the Q value refers to existing technical solutions, such as reference [1];

[0145] Target Actor Networks by providing actions Add noise, output action The two-target Critic network is based on state space. Reward R t+1 and actions Calculate Q-value reward And further calculate the target Q value. , ,in It is a discount factor;

[0146] Calculate the loss criteria loss :

[0147] ;

[0148] In the formula, F mse () mean square error;

[0149] Based on the loss result, the local Actor network is optimized backwards using policy gradient, and the local Actor network resets its actions in the direction that yields higher rewards. After the loss converges, the local Actor network outputs the action corresponding to the minimum loss result. .

[0150] This invention provides a reinforcement learning-based UAV charging and swapping coordination scheduling system for implementation. Figure 1 The method includes:

[0151] A modeling unit is established to define a hybrid power supply system model for drones based on a drone network scenario. The model includes a drone energy consumption model, a wireless charging service model, a battery swapping service model, a backup battery rotation model, and a revenue model. The constraints of the model include drone power safety constraints and charging / swapping facility capacity constraints. The drone network scenario includes an aerial base station, an integrated power replenishment station, and a battery monitoring module for each drone.

[0152] The solution unit is used to solve the charging and swapping decisions of each UAV based on the UAV charging and swapping hybrid power supply system model. This includes: establishing a Markov decision process framework to model the charging and swapping coordination and scheduling problem as a reinforcement learning problem; using the dual-delay deep deterministic policy gradient (TD3PG) algorithm for decision optimization; with the objective of minimizing the time to reach the reward threshold, each UAV agent learns the optimal charging and swapping decision strategy based on its state information, and outputs the charging and swapping decisions of each UAV in real time through an actor-critic architecture; during training, historical interaction data between UAVs and integrated power replenishment stations is stored using an experience replay mechanism, and network parameters are updated through batch sampling; the actor-critic architecture introduces a local actor-critic network and a target actor-critic network using a dual-Q network mechanism. The target actor-critic network performs soft updates based on the local actor-critic network, and UAVs achieve collaborative optimization by sharing reward signals. The local actor network is inversely optimized by calculating the Q-value loss between the local and target actor networks, and the local actor network outputs the actions of each UAV.

[0153] To investigate the effectiveness of the method of this invention, the proposed TD3PG algorithm was simulated and analyzed using Python 3.8.8 and TensorFlow 2.6.0. The simulation was conducted on a 64-bit Windows 10 computer equipped with an Intel(R) i5-9400 CPU. The simulation was based on the power supply strategy recommended above, and other simulation parameter settings are shown in Table 1.

[0154] Table 1 Simulation Parameter Settings for TD3PG Algorithm

[0155]

[0156] Comparative analysis of TD3PG algorithms:

[0157] To verify the superiority of the selected TD3PG algorithm, the DDPG algorithm was used for comparison. Figure 4 The convergence of the DDPG and TD3PG algorithms is compared. Under the same reward pattern, TD3PG converged in 250 time steps, while DDPG converged in 300 steps, lagging by 50 steps, demonstrating a clear advantage in convergence performance for TD3PG. Figure 5 As shown, under the same environment, the time steps to complete the total reward, TD3PG, are around 240 steps, which is significantly lower than DDPG's 300 steps, and converge faster, proving the superiority of this algorithm. Figure 6 This is a schematic diagram of the framework principle of the TD3PG reinforcement learning algorithm.

[0158] This invention provides a coordinated scheduling optimization strategy for charging and battery swapping in unmanned aerial vehicle (UAV) networks. Considering factors such as energy cost and time efficiency, a multi-objective optimization model based on Markov decision process is constructed. The intelligent coordinated scheduling of charging and battery swapping is realized through the TD3PG deep reinforcement learning algorithm, maximizing the advantages of the hybrid power supply mode and reducing the limitations of a single energy replenishment method. In addition, the adaptive adjustment mechanism of the dynamic charging and battery swapping threshold X0 can flexibly select the optimal energy replenishment method according to the real-time environmental conditions (power distribution, facility occupancy rate), effectively improving energy utilization efficiency, reducing operating costs, and shortening task completion time.

[0159] The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the invention. Those skilled in the art will understand that implementing all or part of the above-described embodiments and making equivalent changes in accordance with the claims of the present invention are still within the scope of the invention.

[0160] [1]Wu J, Wu QMJ, Chen S, et al. A-TD3: An adaptive asynchronoustwin delayed deep deterministic for continuous action spaces[J]. IEEE Access, 2022, 10: 128077-128089.

Claims

1. A method for coordinated scheduling of charging and swapping of unmanned aerial vehicles (UAVs) based on reinforcement learning, characterized in that, include: Step S1: Define a hybrid power supply system model for drone charging and swapping based on the drone network scenario. The hybrid power supply system model includes a drone energy consumption model, a wireless charging service model, a battery swapping service model, a backup battery rotation model, and a revenue model. The constraints of the hybrid power supply system model include drone power safety constraints and charging and swapping facility capacity constraints. The drone network scenario includes an airborne base station, an integrated power replenishment station, and a battery monitoring module for each drone. Step S2, based on the UAV charging and swapping hybrid power supply system model, solve for the charging and swapping decisions of each UAV, including: A Markov decision process framework is established to model the charging and swapping coordination and scheduling problem as a reinforcement learning problem; A dual-delay deep deterministic policy gradient (TD3PG) algorithm is employed for decision optimization. With the objective of minimizing the time to reach the reward threshold, each UAV agent learns the optimal charging / swapping decision strategy based on its state information. An Actor-Critic architecture is used to output the charging / swapping decisions of each UAV in real time. During training, an experience replay mechanism is used to store historical interaction data between the UAV and the integrated charging station, and network parameters are updated through batch sampling. The Actor-Critic architecture introduces a local Actor-Critic network and a target Actor-Critic network employing a dual-Q network mechanism. The target Actor-Critic network performs soft updates based on the local Actor-Critic network. UAVs collaborate to optimize by sharing reward signals. The local Actor network is inversely optimized by calculating the Q-value loss between the local and target Critic networks, and the local Actor network outputs the actions of each UAV.

2. The UAV charging and swapping coordination scheduling method based on reinforcement learning as described in claim 1, characterized in that, In step S1, let This indicates the number of drones in the airborne base station, and the integrated power replenishment station. N wl N w N B These represent the number of wireless charging piles, the number of wired charging ports, and the total number of backup batteries in the integrated power replenishment station, respectively. ; In the formula, This indicates that the drone is in working state, wireless charging state, and battery replacement state, respectively; II(·) is the indicator function; ; ; In the formula, This represents the number of wireless charging stations in use at time t. This represents the number of drones that used wireless charging to replenish power at time t. ; In the formula, This represents the number of spare batteries in use at time t. This represents the number of drones that use battery swapping for power replenishment at time t.

3. The UAV charging and swapping coordination scheduling method based on reinforcement learning as described in claim 2, characterized in that, The model of the UAV charging and swapping hybrid power supply system in step S1: In the drone energy consumption model, the energy consumption at time t+1 under operating conditions is represented as: ; In the formula, This represents the battery status of the i-th drone at time t. This indicates the power consumption of the drone per unit time. In the wireless charging service model, the drone's battery level update is represented as follows: ; In the formula, E wl Indicates wireless charging rate; SoC max Indicates the maximum battery capacity; In the battery swapping service model, the drone's battery level update is represented as follows: ; Add dynamic update rules for system state in the backup battery rotation model: For wireless charging piles in integrated power replenishment stations: ; In the formula, This represents the total number of wireless charging pads at time t; This indicates the number of wireless charging pads currently in use at time t. Indicates the number of wireless charging pads available at time t; For batteries in the environment: ； In the formula, This represents the total number of backup batteries at time t; This indicates the number of depleted batteries that have been replaced from the drone at time t, and the number that enters the wired charging replenishment sequence. This indicates the number of fully charged batteries that have been fully charged and ready for use through a wired charging power replenishment sequence. Revenue Model: The cost of SoC from 0 to 1 consists of three types of expenses, denoted as the cost set. , This indicates the unit electricity cost payment for wireless charging. This indicates the cost per unit of electricity required for wired charging of the battery. This indicates the labor cost required for each battery replacement. When the drone is in operation, the benefit generated per time slot t is: It also consumes electricity, and after a certain period of time... The battery level dropped to the charging threshold. hour, The drone sends a charging request to the system. The system assigns a power-up method to drones that need power-up based on their battery level and the availability of equipment at the power-up station, including wireless charging and battery swapping. Define a revenue model for the drone, setting a total revenue target P in the environment, based on the drone's current state. Define the benefit of a single drone at time t. : ; Further define the cumulative revenue of a single drone up to time t as: ; The total cumulative revenue of all drones in the cluster is: ; Define the benefit attainment indicator function: ; In the formula, P is the target revenue threshold, and T is the maximum working time step; when When t represents the time when the drone swarm has achieved the target benefit; otherwise, the swarm is still in the benefit accumulation stage; the goal is to minimize the time to reach the target benefit threshold of P.

4. The UAV charging and swapping coordination scheduling method based on reinforcement learning as described in claim 3, characterized in that, The constraints in step S1 include: Drone power safety constraints: ; Charging and battery swapping facility capacity constraints: st: , 。 5. The UAV charging and swapping coordination scheduling method based on reinforcement learning as described in claim 4, characterized in that, Step S2 includes: Using the Markov Decision Process (MDP) framework, the interaction between the state space, action space, and reward is modeled as a mathematical closed loop of state → policy → action → state transition → reward → policy optimization. Define action space ; Define state space : ; in, This represents the average battery level of all drones at time t. This represents the standard deviation of the battery capacity of all drones. Indicates the percentage of drones in operation. This indicates the occupancy rate of wireless charging stations. Indicates the battery swapping queue occupancy rate. Indicates the revenue completion rate. It represents the time progress and is the ratio of the current working time of the system to the working time threshold T; Define a compound reward function R t The expression is: ; In the formula: work rewards α1 is the work reward coefficient; a penalty is imposed when the battery level of all drones falls below a safety threshold. Where α2 is the safety penalty coefficient; It is the final reward for the drone swarm to achieve ultimate benefit, where α3 is the task completion reward coefficient; α4 is the time penalty coefficient; α1, α2, α3, and α4 are obtained through parameter optimization; In the Actor-Critic architecture, the local Actor network is based on state space. Output Action : ; ; action It acts on the environment, directly changing the state space of the environment through the state transition function T. ,get : ; action The state generated after inputting the environment ,action Reward R t and state space The data is stored in the experience pool; the calculations in the experience pool are input into two local Critic networks, which are then processed based on their current states. Reward R t and actions Calculate the current Q-value reward ( ); Target Actor Networks by providing actions Add noise, output action The two-target Critic network is based on state space. Reward R t+1 and actions Calculate Q-value reward And further calculate the target Q value. , ,in It is a discount factor; Calculate the loss criteria loss : ; In the formula, F mse () mean square error; Based on the loss result, the local Actor network is optimized backwards using policy gradient, and the local Actor network resets its actions in the direction that yields higher rewards. After the loss converges, the local Actor network outputs the action corresponding to the minimum loss result. .

6. A UAV charging and swapping coordination scheduling system based on reinforcement learning, characterized in that, The system for performing the method according to any one of claims 1-5 comprises: A module is established to define a hybrid power supply system model for drones based on a drone network scenario. The hybrid power supply system model includes a drone energy consumption model, a wireless charging service model, a battery swapping service model, a backup battery rotation model, and a revenue model. The constraints of the hybrid power supply system model include drone power safety constraints and charging / swapping facility capacity constraints. The drone network scenario includes an aerial base station, an integrated power replenishment station, and a battery monitoring module for each drone. The solution module is used to solve the charging and swapping decisions of each UAV based on the UAV charging and swapping hybrid power supply system model. This includes: establishing a Markov decision process framework to model the charging and swapping coordination and scheduling problem as a reinforcement learning problem; using the dual-delay deep deterministic policy gradient (TD3PG) algorithm for decision optimization; with the objective of minimizing the time to reach the reward threshold, each UAV agent learns the optimal charging and swapping decision strategy based on its state information, and outputs the charging and swapping decisions of each UAV in real time through an actor-critic architecture; during training, historical interaction data between UAVs and integrated power replenishment stations is stored using an experience replay mechanism, and network parameters are updated through batch sampling; the actor-critic architecture introduces a local actor-critic network and a target actor-critic network using a dual-Q network mechanism. The target actor-critic network performs soft updates based on the local actor-critic network, and UAVs achieve collaborative optimization by sharing reward signals. The local actor network is inversely optimized by calculating the Q-value loss between the local and target actor networks, and the local actor network outputs the actions of each UAV.

Images