A DRL-based method and system for base station-UAV energy scheduling in master-slave game dynamics.

By using the DRL (Digital Relationship Management) master-slave game method, a base station-UAV energy scheduling system was established, realizing global collaborative optimization and multi-objective equilibrium between base stations and UAV swarms. This solves the problems of decision-making fragmentation and insufficient adaptability in air-ground collaborative scenarios in existing technologies, and has good engineering applicability and privacy protection capabilities. It is suitable for hydrogen-electric hybrid UAV applications with multiple base stations, multiple models, and multiple tasks.

CN123117741APending Publication Date: 2026-11-17HARBIN INSTITUTE OF TECHNOLOGY (SHENZHEN) (INSTITUTE OF SCIENCE AND TECHNOLOGY INNOVATION HARBIN INSTITUTE OF TECHNOLOGY SHENZHEN)
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611627773.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-21
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing energy management technologies do not incorporate the entire process of energy production, conversion, and scheduling of drone base stations into the optimization system. This results in insufficient global collaborative optimization capabilities for hydrogen-electric hybrid drones in air-ground collaborative scenarios, insufficient collaborative optimization capabilities under multi-objective conflicts, and insufficient adaptability of reinforcement learning algorithms in complex scenarios, making it difficult to meet the engineering application requirements of real-time performance, stability, and global optimality.

Method used

The method of solving master-slave game using DRL is adopted to establish a two-layer architecture of physical layer and information layer for master-slave game. The energy management of base station and UAV swarm is coordinated by guiding incentive signals. The follower energy management agent and base station scheduling agent based on deep reinforcement learning are constructed to achieve Stackelberg equilibrium. The training is carried out using a two-stage deep reinforcement learning paradigm, and the optimization is carried out by combining a multi-objective Actor-Critic architecture and a nonlinear Chebyshev scalarization module.

Benefits of technology

It achieves global collaborative optimization of energy scheduling between base stations and UAVs, solves the problems of fragmented decision-making dimensions and multi-objective conflicts, has good algorithm adaptability and engineering applicability, protects the flight plan privacy of UAV users, and is suitable for wide-area collaborative operation scenarios with multiple base stations, multiple UAV models, and multiple tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117741A_ABST
    Figure CN123117741A_ABST
Patent Text Reader

Abstract

This invention provides a base station-UAV energy scheduling method and system for solving master-slave game using DRL, comprising: establishing a physical and informational two-layer architecture for master-slave game, with the ground base station as the leader and the UAV swarm as the followers, and defining an interaction protocol centered on the electric energy guidance incentive coefficient, the hydrogen energy guidance incentive coefficient, and the expected energy replenishment demand package; constructing an energy management agent for the followers to respond to the incentive signal; and constructing a multi-objective Actor-Critic architecture for the leader, embedding a nonlinear Chebyshev scalarization module, which generates updated guidance incentive signals and day-ahead production scheduling schemes for the base station through its policy network and value network. Through day-ahead offline master-slave game iterative deduction, a Stackelberg equilibrium is sought, and the day-ahead production scheduling scheme for the base station, the settlement guidance incentive coefficients for each time period, and the UAV reference energy management strategy are output for actual execution the following day, realizing air-ground collaborative cross-timescale scheduling of the base station-UAV energy system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of base station-UAV energy scheduling, UAV energy management, and multi-agent deep reinforcement learning. Specifically, it relates to a base station-UAV energy scheduling method and system for solving master-slave game using DRL. More specifically, it relates to a base station-UAV energy scheduling method and system based on master-slave game and multi-objective deep reinforcement learning. Background Technology

[0002] Hydrogen fuel cell hybrid drones combine the core advantages of hydrogen energy (high energy density, long endurance, and zero carbon emissions) with the characteristics of lithium batteries (high power response and strong adaptability to dynamic operating conditions). They can effectively solve industry pain points such as the short endurance of pure electric drones and the poor environmental performance of fuel-powered drones, and have become the core development direction of drone equipment in the low-altitude economy.

[0003] As the core infrastructure for the air-to-ground collaborative operation of hydrogen-electric hybrid drones, drone base stations are crucial for overcoming the bottleneck of drone operating radius, extending continuous working time, and achieving flexible collaborative scheduling of multiple drones. Currently, drone base stations in the industry are gradually evolving from simple hydrogen refueling / charging terminals into comprehensive energy nodes integrating photovoltaic power generation, hydrogen production and storage, water source preparation, and task scheduling. Especially in remote areas without grid coverage, drone base stations with integrated "photovoltaic-electricity-water-hydrogen" capabilities are a core prerequisite for realizing autonomous operation of hydrogen-electric hybrid drones across all scenarios. However, existing energy management technologies have not yet incorporated the entire process of energy production, conversion, and scheduling of drone base stations into their optimization systems, becoming a core technological bottleneck restricting the large-scale implementation of hydrogen-electric hybrid drone projects.

[0004] Extensive research has been conducted on energy management strategies for hydrogen-powered drones. Based on the degree to which these strategies rely on system mathematical models and prior information about operating conditions, as well as their adaptability to dynamic and stochastic scenarios, existing mainstream technical solutions can be divided into two main categories. The specific classification and core characteristics are as follows: Energy management strategies based on models and preset operating conditions For example, patent CN202510158345 proposes an OPSA-ECMS-based energy management method for hydrogen fuel cell hybrid unmanned aerial vehicles (UAVs). The UAV uses a hybrid power system with a hydrogen fuel cell as the main power source and a lithium-ion battery as an auxiliary power source. The energy management system, based on a pre-set UAV cruise path and control strategy, uses OPSA for offline calculations to plan the most energy-efficient energy allocation scheme, collects real-time data from the system's physical layer, uses ECMS to track the energy allocation scheme, and optimizes the energy allocation online.

[0005] The core commonality of this type of strategy is that they all rely on fixed operating condition boundaries, complete prior information and accurate system mathematical models, and achieve energy allocation through preset rules, numerical solutions or parameter tuning. The principle is clear and stable and controllable power scheduling can be achieved under deterministic steady-state operating conditions. However, their operating condition generalization ability and anti-interference ability are weak, making it difficult to adapt to the multi-variable coupling, strong randomness and non-steady-state operation requirements in UAV-base station collaborative scenarios.

[0006] (2) Energy management strategy adapted to random and dynamic operating conditions For example, patent CN202411571487 proposes an energy adaptive optimization method and system for a hydrogen fuel cell hybrid unmanned aerial vehicle (UAV). By employing a pre-set PPO algorithm and the aforementioned basic equipment parameters, the method constructs the actor neural network and commentator neural network of the hydrogen fuel cell hybrid UAV. Firstly, this enables more accurate prediction and response to various changes during flight, improving energy utilization efficiency and stability. Secondly, through the self-optimization capability of the near-end strategy optimization algorithm, it continuously learns and adapts to new flight environments and mission requirements, enhancing the UAV's intelligence and adaptability. Thirdly, it achieves the hybrid use of hydrogen fuel cells and lithium batteries, further improving the overall performance and reliability of the UAV through adaptive intelligent optimal allocation.

[0007] This type of strategy is currently a research hotspot in this field. Its core is based on data-driven deep reinforcement learning methods, fundamentally different from model-based and pre-defined operating conditions strategies. It possesses strong self-learning and adaptive capabilities, and can achieve autonomous learning and online optimization through real-time interaction with the environment without relying on precise system mathematical models or complete prior information about operating conditions. This effectively addresses random fluctuations and dynamic changes within the scenario. This approach has demonstrated good application potential in single-unit UAV energy management and is currently the optimal technical path for adapting to complex UAV-base station collaborative scenarios.

[0008] However, existing research and technical solutions still have three major technical shortcomings, which cannot meet the actual engineering needs of hydrogen-electric hybrid drones and base stations working together: First, the optimization dimensions are severely fragmented, lacking global collaborative optimization capabilities. Existing energy management strategies generally focus on optimizing the airborne energy system of individual drones, only optimizing the power allocation of fuel cells and power batteries, and the energy consumption and lifespan degradation of individual drones. They completely fail to incorporate the drone base station, a core energy production and scheduling node, into the optimization system, and do not consider the energy efficiency optimization of the entire process, including base station photovoltaic power generation, air water intake, water pipeline distribution, water electrolysis hydrogen production, and hydrogen storage and refueling. They also fail to incorporate key variables such as base station hydrogen storage capacity, hydrogen production capacity, water reserves, and the timing of energy demand from multiple drones into the decision-making system. This results in a serious disconnect between the optimization results and actual air-ground collaborative operation scenarios, making it impossible to achieve global integrated optimization of "airborne energy management - ground base station full-chain energy scheduling". The practical applicability and system integrity of the strategies are seriously insufficient. Second, the collaborative optimization capability under multiple conflicting objectives is insufficient and cannot adapt to multi-dimensional engineering needs. In the scenario of drone-base station collaboration, it is necessary to simultaneously consider multiple coupled and conflicting optimization objectives, such as minimizing drone flight energy consumption, maximizing the lifespan of airborne equipment, maximizing the renewable energy absorption rate of base stations, optimizing the energy efficiency of the entire hydrogen production and storage process, ensuring low-consumption and safe operation of water pipeline networks, and optimizing the efficiency of multi-drone energy replenishment scheduling. However, most existing technologies only optimize a single energy consumption objective, and the few multi-objective optimization schemes also have problems such as fixed weights, inability to adaptively balance, and easy getting trapped in local optima, which cannot achieve global collaborative optimization of multiple objectives. Third, existing reinforcement learning algorithms are not adaptable to complex scenarios and have obvious performance defects. Even existing reinforcement learning-based optimization schemes still suffer from problems such as Q-value overestimation, poor training stability, slow convergence speed, and insufficient environment exploration ability when dealing with high-dimensional continuous action spaces and multi-objective, multi-constraint coupled optimization problems in UAV-base station cooperative scenarios. They are difficult to adapt to complex dynamic scenarios involving multiple UAVs and multiple base stations, and cannot simultaneously meet the engineering application requirements of real-time performance, stability, and global optimality.

[0009] To address the aforementioned issues, there is an urgent need for an energy management optimization method that can achieve global collaborative optimization of the "airborne energy system-ground base station system," balance optimization of multiple conflicting objectives, and possess engineering feasibility. This invention constructs a master-slave game architecture, using guiding incentive signals as a bridge for policy coordination between the leader base station and the follower drone swarm. It employs a two-stage deep reinforcement learning paradigm of "independent pre-training of followers and game deduction between the leader and followers," fundamentally solving the core technical challenges of fragmented decision-making dimensions, difficulty in balancing multiple conflicting objectives, and insufficient adaptability to complex dynamic scenarios in base station-drone air-ground collaborative scenarios. Summary of the Invention

[0010] To address the shortcomings of existing technologies, the purpose of this invention is to provide a base station-UAV energy scheduling method and system for solving master-slave game using DRL.

[0011] A base station-UAV energy scheduling method for solving master-slave game using DRL provided by the present invention includes: Step S1: Establish a two-layer architecture of physical and information layers for master-slave game theory; wherein, the physical layer is used to clarify the equipment composition and energy flow relationship between the ground UAV base station subsystem as the energy supply leader and the airborne hydrogen-electric hybrid power subsystem as the energy consumer follower; the information layer is used to clarify the information interaction protocol between the leader and the follower; the leader issues guidance incentive signals to the follower, and the follower feeds back to the leader a desired energy replenishment demand package including landing time, expected hydrogen replenishment amount, and final battery charge state of flight; Step S2: Construct follower energy management agent and base station scheduling agent based on deep reinforcement learning for the leader and follower in the master-slave game, respectively, and train the follower energy management agent and base station scheduling agent separately to obtain the trained follower energy management agent and the trained base station scheduling agent. Step S3: Based on the trained follower energy management agent and the trained base station scheduling agent, trigger the offline master-slave game deduction before the execution date, seek the Stackelberg equilibrium, and obtain the base station production scheduling scheme before the execution date, the settlement guidance incentive coefficient for each time period, and the output of the reference energy management strategy for each UAV. Step S4: Use the base station daytime production scheduling scheme, settlement guidance incentive coefficients for each time period, and reference energy management strategies for each UAV obtained from the game equilibrium for actual execution the next day.

[0012] Preferably, the guiding excitation signal includes: an electrical energy guiding excitation coefficient and a hydrogen energy guiding excitation coefficient; The electrical energy incentive coefficient and the hydrogen energy incentive coefficient are dimensionless strategy variables formulated by the leader to guide the followers' electricity consumption behavior and hydrogen consumption behavior, respectively. The higher the value, the higher the unit cost of the corresponding energy, thereby incentivizing the followers to reduce the consumption of that energy or adjust their energy consumption time.

[0013] Preferably, the base station scheduling agent includes a state space S, an action space A, and a multi-objective sub-reward vector. ; Wherein, the state space S is an observation vector constructed based on hourly scheduling compensation, including: the base station's own state and the equivalent state of the UAV swarm; The base station's own status includes: the charging status of the energy storage device s1, the remaining water storage ratio of the water storage device s2, the remaining hydrogen storage ratio of the hydrogen storage device s3, and the real-time output power of the photovoltaic power generation device s4. The equivalent state of the drone swarm is a macroscopic time-series variable generated by the base station scheduling agent after performing aggregation and continuity evaluation on the expected power replenishment demand packets fed back by each follower; Wherein, the scheduling period t=1,2,…,T, where T is the total number of time periods divided by the daily scheduling cycle; for each time period t, the equivalent state of the UAV swarm includes: the equivalent number of UAVs that need additional power replenishment in each time period s5, the total power replenishment requirement in each time period s6, and the total hydrogen replenishment requirement in each time period s7.

[0014]

[0015]

[0016] The summation iterates through all drones i=1,2,…,N; The energy allocation coefficient is defined as the proportion of the intersection of the resupply time window and the scheduling period t of the i-th UAV to its total resupply time, and satisfies the following condition: The refueling time window is determined jointly by the landing period in the expected energy demand package and the refueling duration based on the expected hydrogen replenishment amount; when the refueling time window does not cover the period t, ; As a status indicator, if the final state of battery charge during flight... Below the ideal continuation threshold ,but ,otherwise ; The additional power required is determined by the base station scheduling agent after performing a continuity assessment on the i-th drone; when hour, ;when hour, ,in, The rated capacity of the airborne power lithium battery pack; The expected amount of hydrogen replenishment in the expected energy replenishment demand package reported by the i-th UAV; The action space A is a continuous action vector that includes the operating power instructions and incentive strategy instructions of each controllable energy conversion and storage device of the base station within a scheduling step. The action space A includes: the charging and discharging power coefficient of the energy storage device a1, a1∈[-1,1], a1>0 indicates discharging, a1<0 indicates charging, and its absolute value is multiplied by the rated charging and discharging power of the energy storage device to obtain the actual charging and discharging power. The water discharge flow coefficient of the water storage device is a2, a2∈[0,1], which is multiplied by the rated water discharge flow of the water storage device to obtain the actual water discharge flow. The hydrogen release flow rate coefficient of the hydrogen storage device is a3, where a3∈[0,1]. Multiplying it by the rated hydrogen release flow rate of the hydrogen storage device yields the actual hydrogen release flow rate. The operating power coefficient a4 of the water electrolysis hydrogen production unit, a4∈[0,1], is multiplied by the rated power of the water electrolysis hydrogen production unit to obtain the actual operating power; The operating power coefficient of the air-water intake device is a5, where a5∈[0,1]. Multiplying it by the rated power of the air-water intake device yields the actual operating power. The electrical energy guiding excitation coefficient is a6, where a6∈[0,1]; The hydrogen energy guiding excitation coefficient is a7, where a7∈[0,1]; Wherein, the electrical energy guiding excitation coefficient a6 and the actual electrical energy guiding excitation value The mapping relationship between them is as follows:

[0017] The hydrogen energy guiding excitation coefficient a7 and the actual hydrogen energy guiding excitation value The mapping relationship between them is as follows:

[0018] In the formula, and These are the preset lower and upper limits of the electrical energy guiding excitation value, respectively. and These are the preset lower and upper limits of the hydrogen energy guiding excitation value, respectively; The multi-objective sub-reward vector There are three optimization objectives, all of which are transformed into maximization directions, including: r1: The objective of minimizing economic operating costs is transformed into a reward component in the maximization direction, defined as the negative value of the total economic operating cost within the scheduling period:

[0019] Among them, the cost of purchasing electricity from the power grid :

[0020] In the formula: The duration of a single scheduling period; The external grid time-of-use electricity price for time period t; The power exchanged between the base station and the upstream power grid is determined by the power balance equation:

[0021] in: The total power load inside the base station during time period t. ; , These are the rated powers of the water electrolysis hydrogen production unit and the air-to-water extraction unit, respectively. Other auxiliary load constants; , These are the charging power and discharging power of the energy storage device, respectively. , , The rated charge and discharge power of the energy storage device; Equipment operation and maintenance costs :

[0022] In the formula: 、 、 、 、 These are the unit power or unit flow rate operation and maintenance cost coefficients for energy storage devices, water electrolysis hydrogen production devices, air water intake devices, water storage devices, and hydrogen storage devices, respectively. The rated discharge flow rate of the water storage device; The rated hydrogen release flow rate for the hydrogen storage device; Water network replenishment cost :

[0023] in, The water supply rate of the water transmission network during time period t is determined by the water balance relationship:

[0024] In the formula: The water consumption rate per unit power of the water electrolysis hydrogen production unit; The water production rate per unit power of the air-to-water intake device; Water price for supplementing the water supply network; Safety boundary penalty items :

[0025] In the formula: s1(t), s2(t), and s3(t) represent the state of charge of the energy storage device, the remaining water storage ratio of the water storage device, and the remaining hydrogen storage ratio of the hydrogen storage device at the end of the current time period, respectively. 、 These are the safety lower limits for water storage devices and hydrogen storage devices, respectively. 、 、 Preset penalty weight coefficient; r2: Power supply reliability maximization objective:

[0026] Where: Total load requirement of base stations Local power supply capability The local power supply capacity is the smaller of the sum of photovoltaic and energy storage discharge and the total load demand; the power supply capacity exceeding the load demand is not included. To prevent the denominator from being zero, a positive decimal is preset. This target adopts the complementary value of the standard load failure probability LOLP. The numerator is the cumulative power shortage that local resources cannot meet the total load demand within the scheduling period, and the denominator is the cumulative power demand of the total load. The value of r2 is in the range of [0,1]. The larger the value, the higher the power supply reliability of the base station when it relies only on local photovoltaic and energy storage devices. r3: Maximize the success rate of drone missions

[0027] In the formula: N is the total number of UAVs that perform flight missions within the scheduling period; , These are the required power and hydrogen refueling amounts for the i-th drone, respectively, defined in the same state space. , These represent the total power replenishment demand and total hydrogen replenishment demand for drone swarms, respectively. , s6(t), s7(t), and s3(t) represent the total charging capacity and total hydrogen refueling capacity that the base station can actually provide during the scheduling cycle, respectively; s6(t), s7(t), and s3(t) represent the total energy demand, total hydrogen demand, and remaining hydrogen storage ratio of the hydrogen storage device for time period t, respectively, as defined in the state space; a3 is the hydrogen release flow coefficient of the hydrogen storage device, as defined in the action space. The rated hydrogen release flow rate for the hydrogen storage device; The rated hydrogen storage capacity of the hydrogen storage device, in units of... Consistent; The total charging power that the base station can actually provide during time period t is constrained by photovoltaic, energy storage, grid power purchase and charging facility capacity, and is given by the simulation environment after the action is executed. The duration of a single scheduling period; To prevent the denominator from being zero, a positive decimal is preset; r3∈[0,1], the larger the value, the more sufficient the base station's power supply guarantee for the drone swarm. When the total demand is zero, the corresponding ratio is 1.

[0028] Preferably, the base station scheduling agent adopts a multi-objective Actor-Critic architecture with multi-objective sub-item reward vectors and nonlinear scalarization modules, which is used to generate updated guidance incentive signals and base station day-ahead production scheduling schemes through its policy network and value network after integrating the expected energy replenishment demand package fed back by followers. The base station scheduling agent performs the following single-step interaction and update in each training round, including: Obtain state s and input state s into the Actor main network to obtain deterministic actions. After being corrected by a preset safety constraint layer, a safe action is output. ; Perform safety actions The environment interaction returns to the next state. Multi-objective component reward vector and the round-end flag "done"; generate a discount mask. , Discount factor; Sample Store the sample in the priority experience replay pool and assign it the highest priority in the current replay pool. When the number of samples in the replay pool exceeds the preset batch size, the samples are prioritized. The sampling probability distribution is generated by exponentiation, and the importance sampling weights are calculated. The following sub-steps are performed on the sampled samples: Calculate the next action using the target network And add truncated noise; Obtain the multi-objective Q-value vector of the dual-Critic target network by taking the minimum value in each dimension. ; Constructing multi-objective target value vectors The multi-objective target value vector y is scalarized using a nonlinear Chebyshev scalarization function to obtain... If done is true, then mask = 0, and y = r. ; Calculate the scalar Q-value of the current state-action pair. , The parameters of the Critic main network are updated using the mean square error weighted by importance sampling weights as the loss function.

[0029] in, For importance sampling weights, To scalarize the Q value, To quantify the TD target value; The absolute value of the deviation between the mean of the dual Critic scalar Q values ​​and the scalar TD target value As the TD error, the absolute value of the TD error is written back to the corresponding sample position as the new priority. After every two updates to the Critic main network, minimize the scalar Q value output by the first Critic main network. Update the Actor's main network parameters with this as the goal. ; After each update of the Actor main network, the soft update factor is applied. Update the target network parameters; make If done is true, the current round ends; otherwise, the execution is repeated until the current round ends.

[0030] Preferably, the experience replay pool is a priority experience replay pool, comprising: Interaction samples are stored in a fixed-capacity circular buffer. Each sample contains a state, a multi-objective sub-reward vector, a discount mask, and an action. Newly stored samples are assigned the highest priority in the current buffer to ensure that new samples can be sampled and learned preferentially. Based on the priority of each sample The sampling probability distribution is generated by exponentiation, and random sampling is performed according to this distribution. Importance sampling weights are then calculated to correct for distribution bias introduced by priority sampling. To preset hyperparameters, It degenerates into uniform sampling; The absolute value of the TD error obtained after the sampled sample is evaluated by the network is used as the new priority, and it is written back to the corresponding sample position to realize the dynamic update of the priority.

[0031] Preferably, the follower energy management agent adopts an Actor-Critic architecture of policy network and value network to generate a flight power allocation scheme based on the guidance and incentive signals received from the leader, and thereby obtain a desired energy replenishment demand package including landing time, desired hydrogen replenishment amount and final battery state of charge.

[0032] Preferably, step S3 includes: Step S3.1: Generate initial guidance stimulus signal: The base station scheduling agent constructs an initial state vector in the state space S, inputs it into its Actor main network, and generates and publishes the initial electric energy guiding excitation coefficient curve and hydrogen energy guiding excitation coefficient curve. Step S3.2: Follower Response: After receiving the guidance stimulus coefficient curve, each of the follower energy management agents uses the guidance stimulus coefficient curve and the predetermined flight mission power profile as inputs to perform full-cycle power allocation through its policy network, generating its own data including the landing period and the desired hydrogen refueling amount. and the final state of battery charge during flight The expected power replenishment demand package is fed back to the base station scheduling agent; Step S3.3; Leader Re-optimization: After collecting all expected power replenishment request packets, the base station scheduling agent performs the following sub-steps: Step S3.3.1: Requirement Aggregation: Using the energy distribution coefficient The expected hydrogen replenishment amount of each follower Aggregate the total hydrogen demand for energy replenishment in each time period ; S3.3.2: Succession Assessment: The final state of battery charge of each follower during flight Conduct a connection assessment and determine the connection status indicators. And the additional power required This is aggregated into the total replenishment power demand for each time period. And the equivalent number of drones that need additional charging at different times. ; S3.3.3: Strategy Update: The base station scheduling agent will include the aggregated... , , The complete state vector of the state space S, including the Actor main network, is input into its Actor main network. The network outputs the action vector, and after decoding, the updated base station day-ahead production scheduling scheme and the guiding incentive coefficient curve are obtained. Repeatedly trigger steps S3.2 to S3.3 until the change amplitude of the guiding incentive coefficient curve in two adjacent iterations is less than the preset threshold ε, reaching Stackelberg equilibrium; at equilibrium, the base station scheduling agent decodes and outputs a day-ahead production scheduling scheme containing the next day's power purchase plan, electrolysis hydrogen production plan, and energy storage charging and discharging plan from its final action vector, determines the settlement guiding incentive coefficient for each time period, and records the reference energy management strategy generated by each follower energy management agent under the equilibrium guiding incentive coefficient.

[0033] Preferably, the training of the base station scheduling agent adopts a policy delayed update mechanism, including: updating the Actor main network once after updating the Critic main network twice; during the training process, the target network parameters gradually approach the main network parameters through a soft update method. , ,in , These are the parameters for the Actor's main network and target network, respectively. , These are the parameters of the k-th Critic main network and the target network, respectively, where k = 1, 2, This is the soft update coefficient.

[0034] Preferably, during the training process of the base station scheduling agent, a global parameter update mechanism is triggered after each round, including: Update Utopia Reference Points ,include: After each training round, calculate the cumulative reward for each objective in that round. For each objective i, update the Utopia reference point only when a better cumulative reward is obtained: ,in, It is a preset positive decimal to avoid numerical instability caused by the utopian reference point coinciding with the current optimal value; Adaptive update of preference weights : After each preset K training rounds, the convergence of each objective is calculated. :

[0035] in, The cumulative reward for the i-th objective in the most recent round. The cumulative reward reference value for this objective recorded when the Utopia reference point is first updated during the initial training phase; Calculate the original preference weights based on the convergence and introduce a preset lower bound for the weights. Perform cropping and normalization:

[0036] This causes the weight allocation to be tilted toward targets that have not yet been fully optimized, and prevents the weight of any target from degenerating to zero.

[0037] A base station-UAV energy scheduling system for solving master-slave game using DRL, provided by the present invention, includes: Module M1: Establishes a two-layer architecture of physical and information layers for master-slave game theory; wherein, the physical layer is used to clarify the equipment composition and energy flow relationship between the ground UAV base station subsystem as the energy supply leader and the airborne hydrogen-electric hybrid power subsystem as the energy consumer follower; the information layer is used to clarify the information interaction protocol between the leader and the follower; the leader issues guidance and incentive signals to the follower, and the follower feeds back to the leader a desired energy replenishment demand package including landing time, expected hydrogen replenishment amount, and final battery charge state of flight; Module M2: Construct follower energy management agents and base station scheduling agents based on deep reinforcement learning for the leader and follower in the master-slave game, respectively, and train the follower energy management agents and base station scheduling agents separately to obtain the trained follower energy management agents and trained base station scheduling agents. Module M3: Based on the trained follower energy management agent and the trained base station scheduling agent, the module performs offline master-slave game deduction before the execution date to seek Stackelberg equilibrium and obtain the base station day-ahead production scheduling scheme, the settlement guidance incentive coefficient for each time period, and the output of the reference energy management strategy for each UAV. Module M4: The base station daytime production scheduling scheme obtained from the game equilibrium, the settlement guidance incentive coefficients for each time period, and the reference energy management strategies for each UAV are used for actual execution the next day.

[0038] Compared with the prior art, the present invention has the following beneficial effects: 1. Global Coordination at the Architectural Level: This invention models the base station-UAV air-to-ground coordinated energy scheduling as a master-slave game problem, replacing the traditional fixed electricity or hydrogen price with a guiding incentive signal as a bridge to coordinate the heterogeneous decision-making of the base station and the UAV swarm. This guiding incentive signal is dynamically generated by the leader base station through its multi-objective Actor-Critic network and directly embedded into the energy cost function of the follower UAVs, forming an incentive-compatible mechanism (i.e., in the process of minimizing its own single flight energy cost, the UAV's electricity or hydrogen consumption behavior naturally responds to the control intention of the base station), solving the problem of the time dimension separation in base station-UAV air-to-ground decision-making and achieving global coordination optimization at the system level; 2. Multi-objective equilibrium optimization capability at the algorithm level: This invention designs a multi-objective Actor-Critic architecture based on a nonlinear Chebyshev scalarization module for the base station scheduling agent. By applying the Chebyshev scalarization criterion throughout the entire process of action selection and network update, it fundamentally avoids the inherent defect of traditional linear weighting methods in failing to cover non-convex Pareto fronts. Combined with dynamic utopian reference points and adaptive preference weight updates, it achieves balanced optimization of three conflicting objectives: economic operating cost, power supply reliability, and UAV mission success rate.

[0039] 3. Two-Stage Training Paradigm and Privacy Protection: This invention employs a two-stage deep reinforcement learning paradigm of "independent pre-training of followers and master-slave game theory deduction." Follower agents complete independent training before participating in the game, learning rational response patterns to arbitrary guiding stimuli. During game theory deduction, the trained follower only needs to receive guiding stimuli to output a power allocation scheme through its policy network, and only provides feedback to the leader on the macroscopic level of expected energy replenishment needs, without revealing the complete flight mission profile. This mechanism achieves global optimization while maximizing the protection of the drone user's flight plans and other privacy information.

[0040] 4. Broad Engineering Adaptability and Application Prospects: The deep reinforcement learning framework for followers in this invention can be implemented using policy gradient algorithms such as TD3, DDPG, and SAC, exhibiting excellent algorithmic flexibility. This invention can be smoothly extended to wide-area collaborative operation scenarios involving multiple base stations, multiple drone models, and multiple tasks. It possesses extremely high application value and promising prospects in core application areas of hydrogen-electric hybrid drones, such as low-altitude inspection, emergency logistics, and smart city monitoring. Attached Figure Description

[0041] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the base station-UAV energy scheduling method for solving master-slave game using DRL, as described in this invention.

[0042] Figure 2 This is a topology diagram of the base station-UAV energy system in this invention.

[0043] Figure 3 This is a schematic diagram of the optimal control principle of the base station-UAV energy system in this invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0045] Example 1 The present invention provides a base station-UAV energy scheduling method and system for solving master-slave game using DRL. By constructing a master-slave game architecture, the base station is regarded as the leader and the UAV swarm is regarded as the follower. The guiding incentive signal is used as the game link. The two-stage deep reinforcement learning paradigm of "independent pre-training of followers and master-slave game deduction" is adopted to realize the global collaborative optimization of multi-objective energy scheduling of base stations and energy consumption behavior of UAV swarm. It solves the core problems of poor air-ground collaboration adaptability of base station-UAV in the prior art, single optimization objective, and lack of global collaboration capability.

[0046] The DRL-based base station-UAV energy scheduling method for solving master-slave game theory is as follows: Figure 1 As shown, it includes the following steps: Step S1: Establish a two-tiered physical and information architecture for master-slave game theory; such as... Figure 2 As shown, the base station-UAV hydrogen-electric energy system topology of this embodiment consists of two parts: a physical layer and an information layer. It includes a photovoltaic power generation module 1, a grid interaction module 2, an energy storage device 3, an air-to-water extraction device 4, a water supply network 5, a water storage device 6, an electrolytic hydrogen production device 7, a hydrogen storage device 8, a ground data acquisition and communication unit 9, an airborne status detection unit 10, an airborne power lithium battery pack 11, an airborne hydrogen storage tank 12, an airborne fuel cell stack 13, a DC / DC converter 14, an electronic speed controller 15, a motor drive module 16, and a propeller 17. The physical layer clearly defines the ground UAV base station subsystem as the energy supply leader and the airborne hydrogen-electric hybrid system as the energy consumer follower. The composition and energy flow relationship of the force system; the information layer clarifies the information interaction protocol between the leader and the follower, which is different from the fixed electricity price or hydrogen price in traditional methods. The leader is used to issue a guidance and incentive signal containing the electric energy guidance incentive coefficient and the hydrogen energy guidance incentive coefficient to the follower. The follower is used to feed back to the leader a expected energy replenishment demand package containing the landing time, expected hydrogen replenishment amount, and battery charge state at the end of flight. The design of this information interaction protocol protects the privacy information such as the UAV flight plan (the UAV does not need to report the complete mission profile) while providing the leader with the necessary macro data for global optimization, thus achieving a balance between information sharing and privacy protection.

[0047] The electrical energy guidance incentive coefficient and the hydrogen energy guidance incentive coefficient are dimensionless strategy variables formulated by the leader to guide the electricity consumption behavior and hydrogen consumption behavior of the followers, respectively. The higher the value, the higher the unit cost of the corresponding energy, thereby incentivizing the followers to reduce the consumption of the energy or adjust their energy consumption time. Through this design, the guidance incentive signal has the dual functions of cost accounting and strategy control.

[0048] Step S2, construct deep reinforcement learning-based agents for the leader and followers in the master-slave game respectively; such as Figure 3As shown, the core of this step is to endow both sides of the game with "intelligence." Specifically, the follower energy management agent employs an Actor-Critic architecture with a policy network and a value network. It generates a flight power allocation scheme based on the guidance incentive signals received from the leader, and accordingly obtains the expected energy replenishment demand package. The base station scheduling agent employs a multi-objective Actor-Critic architecture with multi-objective sub-item reward vectors and a nonlinear scalarization module. After integrating the expected energy replenishment demand package fed back by the followers, it generates updated guidance incentive signals and a day-ahead production scheduling scheme for the base station through its policy network and value network.

[0049] Step S3: Perform day-ahead offline master-slave game deduction to seek Stackelberg equilibrium; the base station scheduling agent generates and publishes the initial power guidance incentive coefficient curve and hydrogen energy guidance incentive coefficient curve; each follower energy management agent generates its own expected energy replenishment demand package based on the guidance incentive coefficient curve, with the goal of minimizing the single flight energy cost weighted by the guidance incentive coefficient; the base station scheduling agent aggregates and evaluates all expected energy replenishment demand packages, generates the total energy replenishment demand for each time period, updates the status accordingly, and generates a new day-ahead base station production scheduling scheme and guidance incentive coefficient curve; repeat the above process until the guidance incentive coefficient curve converges.

[0050] Step S4: Output the base station's day-ahead production scheduling scheme, settlement guidance incentive coefficients for each time period, and reference energy management strategies for each UAV obtained from the game equilibrium, for actual execution the next day: The base station performs energy production and storage operations for each time period according to the day-ahead production scheduling scheme; During actual flight, the UAV uses the equilibrium guidance incentive coefficient curve as input, performs power allocation online through its onboard energy management agent, and performs energy replenishment settlement according to the settlement guidance incentive coefficient of the landing time period.

[0051] Specifically, step S1 includes: The ground-based UAV base station subsystem includes a photovoltaic power generation device, an energy storage device, an air-water extraction device, a water storage device, an electrolytic water hydrogen production device, a hydrogen storage device, and a ground communication device. The photovoltaic power generation device is used to charge the energy storage device; The energy storage device is used to store electrical energy from photovoltaic power generation devices and the upstream power grid, as well as to supply power to other electrical equipment in the base station; The air-to-water extraction device is used to extract liquid water from the air and supply water to the water storage device; The water storage device is used to store water from the air water intake device and the water supply network, as well as to provide the water source required for hydrogen production by the water electrolysis hydrogen production equipment; The upstream power grid and water pipeline network serve as supplementary sources of external power and water, respectively, ensuring the reliability of power and water supply to the base station when photovoltaic output or air-source water is insufficient. The water electrolysis hydrogen production device is used to electrolyze water to produce hydrogen gas. The hydrogen storage device is used to store hydrogen produced by the water electrolysis hydrogen production device and to replenish hydrogen for the onboard hydrogen storage tank of the UAV docked at the base station. The ground communication device is responsible for bidirectional data interaction with the airborne communication device of each UAV, receiving the expected energy replenishment demand packet uploaded by the UAV, and sending the electrical energy guidance excitation coefficient and hydrogen energy guidance excitation coefficient signals generated by the base station scheduling agent to the UAV. The airborne hydrogen-electric hybrid power subsystem acts as a follower, with each drone being an independent follower. It includes an airborne communication device, an airborne lithium battery pack, an airborne hydrogen storage tank, an airborne fuel cell stack, a DC / DC converter, an electronic speed controller, a motor drive module, and propellers; wherein: The airborne communication device is responsible for bidirectional data interaction with the ground communication device, receiving the electrical energy guiding excitation coefficient and hydrogen energy guiding excitation coefficient signals sent by the base station, and uploading the expected energy replenishment demand package of the device to the base station, which includes the final state of battery charge in flight.

[0052] Specifically, the construction of the base station scheduling agent in step S2 includes: a neural network structure, a scalarization module, and a network update mechanism; its training uses an experience replay pool for sample management; wherein: The neural network structure includes an Actor main network, an Actor target network, a dual Critic main network, and a dual Critic target network; The outputs of both the dual-Critic main network and the dual-Critic target network are N-dimensional Q-value vectors, where N is the total number of optimization objectives. Each dimension corresponds to the cumulative expected reward of an optimization objective (economic operating cost, power supply reliability, UAV mission success rate). The mechanism of minimizing the Q-values ​​in the dual-Critic network effectively suppresses the overestimation of Q-values ​​and improves training stability.

[0053] The experience replay pool is a priority experience replay pool, used by the base station scheduling agent to store and sample interaction samples. Unlike traditional uniform random sampling experience replay, the priority experience replay mechanism allocates different sampling probabilities based on the learning value of the samples, allowing historical experiences that contribute more significantly to the improvement of the current strategy to be replayed more frequently, thereby significantly improving data utilization efficiency and training convergence speed. Its operation mechanism consists of three stages: (1) Storage: Interactive samples are stored in a fixed-capacity circular buffer. Let the sample generated at the current time be... Where s is the current state, The actual actions executed after correction by the safety constraint layer, where r is the multi-objective component reward vector. Discount mask ( (This is the discount factor, and "done" is the round end marker). This is the next state. Newly stored samples are given the highest priority among all samples in the current buffer, ensuring that new samples can be sampled and learned first, accelerating the utilization of valuable experience; at the same time, since new samples represent the latest policy interaction results and have not yet been learned, giving them the highest priority also helps the agent adapt to changes in the environment in a timely manner.

[0054] (2) Sampling: When the number of samples stored in the buffer is sufficient, sampling is performed based on the priority of each sample. The sampling probability distribution is generated by exponentiation, and random sampling is performed according to this distribution. Let the priority of the k-th sample be denoted as . Then the probability of it being sampled is:

[0055] in This represents the number of valid samples that can be sampled at present. For the preset priority index ( This controls the strength of the impact of priority on sampling probability. When hour, It degenerates into uniform random sampling; when At that time, the higher the priority of the sample, the greater the probability of it being sampled.

[0056] It should be noted that the number of valid samples that can be sampled The value is the total number of samples stored in the current buffer minus 1. This is because each sampling requires simultaneously obtaining the current state 's' and the next state. In the circular buffer, samples are stored sequentially in chronological order, and the "next state" of the k-th sample is exactly the "current state" of the (k+1)-th sample. Therefore, the next state of the last valid sample has not yet been stored in the buffer and cannot be used as a complete sample; it must be excluded from the sampling range.

[0057] Furthermore, before calculating the sampling probability, the priority of all samples is first pruned to a value no lower than a preset lower limit. This operation aims to prevent a zero priority from resulting in a zero sampling probability, ensuring that every empirical sample retains the possibility of being resampled. This is related to the small positive constant introduced in the priority update. This constitutes a unified robust protection design.

[0058] Because priority sampling alters the original distribution of samples, directly using these samples for gradient updates introduces distribution bias, causing a systematic shift in the value estimation of the Critic network. To correct this bias, importance sampling weights need to be calculated for each sample:

[0059] in, The preset importance sampling index ( The strength of the bias correction is controlled. The calculated importance sampling weights for each sample are further normalized to the [0,1] interval:

[0060] This importance sampling weight will be used as a multiplier factor to determine the contribution of each sample in the subsequent Critic network loss function, thereby ensuring that the value estimation of the Critic network remains unbiased under the condition of priority sampling.

[0061] (3) Priority Update: After each network update, the priority of the sampled samples needs to be updated to reflect their current learning value in a timely manner. This invention uses the absolute value of the TD error obtained after the sampled samples are evaluated by the network as the new priority and writes it back to the corresponding sample position. Specifically, the absolute value of the deviation between the mean of the dual Critic scalar Q values ​​and the scalar TD target value is used as the TD error. :

[0062] Add a preset positive small constant to the absolute value of the TD error. This will then be used as the new priority for that sample:

[0063] in The purpose is to ensure that even if the TD error is zero, the sample still retains a lowest non-zero priority, preventing it from never being sampled. This design requirement is complementary to the priority lower bound pruning operation in the sampling phase: The priority is guaranteed to be positive from the source of the update, while the lower limit pruning provides additional protection during the sampling phase. Together, they ensure that each sample has a non-zero sampling probability at any time.

[0064] The intuitive meaning of this update mechanism is that the larger the absolute value of the TD error, the greater the prediction bias of the state-action pair corresponding to the sample under the current Critic network evaluation. In other words, this experience has a higher "surprise" or learning value for the current policy. After giving it higher priority, the sample will be sampled more frequently in subsequent training, allowing the agent to focus on the experience areas that have not yet been fully learned under the current policy, thereby improving learning efficiency.

[0065] Finally, the updated priorities are written back to the corresponding sample's position in the priority experience replay pool, achieving dynamic priority maintenance. It should be noted that only the priorities of the samples sampled and participating in this update are updated; the priorities of the remaining samples in the buffer remain unchanged. This "lazy update" strategy effectively controls computational overhead while ensuring algorithm effectiveness.

[0066] The scalarization module is configured as follows: By introducing a nonlinear Chebyshev scalarization function, the multi-objective Q-value vector is converted into a scalarized Q-value. :

[0067] Where N is the number of optimization objectives. Let the current preference weight be the i-th objective. For the dynamically updated utopian reference point of the i-th objective, , These are the outputs of the dual Critic master network, respectively.

[0068] The core idea of ​​this scalarization function is as follows: First, take the smaller Q-value of each objective in the dual-Critic network (to suppress overestimation); then calculate the weighted Chebyshev distance between the Q-value of each objective and its utopian reference point; finally, take the maximum value among all objectives. Through the maximum operator, this function preserves the distance information between each optimization objective and the ideal point, especially the constraint information of the worst-performing objective, fundamentally avoiding the problem that traditional linear weighted scalarization easily leads to optimization results biased towards extreme solutions, and can cover the complete non-convex Pareto optimal front.

[0069] Based on the scalarization module, the network update process is as follows: Sample the next state from the experience pool. Multi-objective reward vector And a discount mask, using the target network to calculate the next action. And truncated noise is added to smooth the strategy and enhance exploration capabilities; Obtain the multi-objective Q-value vector of the dual-Critic target network by taking the minimum value in each dimension. ; Constructing multi-objective target value vectors ,in , d is the discount factor and d is the termination flag; if d is true, then mask=0, and y=r. The Chebyshev scalarization function is used to scalarize the multi-objective target value vector to obtain the scalarized TD target value. :

[0070] The loss function of the Critic network is the mean squared error weighted by importance sampling weights:

[0071] in, For importance sampling weights, To scalarize the Q value, To quantify the TD target value.

[0072] Specifically, the base station scheduling agent's state space S, action space A, and multi-objective component reward vector... The definition is as follows: (1) The state space S is an observation vector constructed based on hourly scheduling step size, including the base station's own state and the equivalent state of the UAV swarm: The base station's own status includes: the charge status of the energy storage device s1; the remaining water storage ratio of the water storage device s2; the remaining hydrogen storage ratio of the hydrogen storage device s3; and the real-time output power of the photovoltaic power generation device s4. The equivalent state of the drone swarm is a macroscopic time-series variable generated by the base station scheduling agent after performing aggregation and continuity evaluation on the expected power replenishment demand packets fed back by each follower; Wherein, the scheduling time period t = 1, 2, ..., T, where T is the total number of time periods divided by the day-ahead scheduling cycle; for each time period t, the equivalent state of the UAV swarm includes: The equivalent number of drones requiring additional power replenishment in each time period is s5; the total power replenishment requirement in each time period is s6; the total hydrogen replenishment requirement in each time period is s7. The above aggregate variables are specifically defined as follows:

[0073]

[0074]

[0075] In the formula, the summation iterates through all drones i=1,2,…,N, where: The energy allocation coefficient is defined as the proportion of the intersection of the resupply time window and the scheduling period t of the i-th UAV to its total resupply time, and satisfies the following condition: The refueling time window is determined jointly by the landing period in the expected energy demand package and the refueling duration based on the expected hydrogen replenishment amount; when the refueling time window does not cover the period t, ; As a status indicator, if the final state of battery charge during flight... Below the ideal continuation threshold ,but ,otherwise ; The additional power required, determined by the base station scheduling agent after conducting a continuity assessment of the i-th drone, is specifically calculated as follows: when hour, ;when hour, ,in The rated capacity of the airborne power lithium battery pack; The expected amount of hydrogen replenishment in the expected energy replenishment demand package reported by the i-th UAV; (2) The action space A is a continuous action vector, which, within a scheduling step, includes the operating power instructions and excitation strategy instructions for each controllable energy conversion and storage device of the base station, specifically including: The charging and discharging power coefficient of the energy storage device is a1, where a1∈[-1,1], a1>0 indicates discharging, and a1<0 indicates charging. Its absolute value is multiplied by the rated charging and discharging power of the energy storage device to obtain the actual charging and discharging power. The water discharge flow coefficient of the water storage device is a2, a2∈[0,1], which is multiplied by the rated water discharge flow of the water storage device to obtain the actual water discharge flow. The hydrogen release flow rate coefficient of the hydrogen storage device is a3, where a3∈[0,1]. Multiplying it by the rated hydrogen release flow rate of the hydrogen storage device yields the actual hydrogen release flow rate. The operating power coefficient a4 of the water electrolysis hydrogen production unit, a4∈[0,1], is multiplied by the rated power of the water electrolysis hydrogen production unit to obtain the actual operating power; The operating power coefficient of the air-water intake device is a5, where a5∈[0,1]. Multiplying it by the rated power of the air-water intake device yields the actual operating power. The electrical energy guiding excitation coefficient is a6, where a6∈[0,1]; The hydrogen energy guiding excitation coefficient is a7, where a7∈[0,1]; Wherein, the electrical energy guiding excitation coefficient a6 and the actual electrical energy guiding excitation value The mapping relationship between them is as follows:

[0076] The hydrogen energy guiding excitation coefficient a7 and the actual hydrogen energy guiding excitation value The mapping relationship between them is as follows:

[0077] In the formula, and These are the preset lower and upper limits of the electrical energy guiding excitation value, respectively. and These are the preset lower and upper limits of the hydrogen energy guiding excitation value, respectively; The actual electrical energy guided excitation value and actual hydrogen energy guiding incentive value It is used to directly participate in the objective function calculation of the follower energy management agent.

[0078] (3) The multi-objective sub-item reward vector There are three optimization objectives, all of which are transformed into maximization directions, and are defined as follows: (31) r1: The objective of minimizing economic operating costs is transformed into the reward component in the maximization direction, defined as the negative value of the total economic operating cost within the scheduling period:

[0079] Electricity purchase cost :

[0080] In the formula: The duration of a single scheduling period; The external grid time-of-use electricity price for time period t; The power exchanged between the base station and the upstream power grid is determined by the power balance equation:

[0081] in: The total power load inside the base station during time period t. ; , These are the rated powers of the water electrolysis hydrogen production unit and the air-to-water extraction unit, respectively. Other auxiliary load constants; , These are the charging power and discharging power of the energy storage device, respectively. , , The rated charge and discharge power of the energy storage device; Equipment operation and maintenance costs :

[0082] In the formula: 、 、 、 、 These are the unit power or unit flow rate operation and maintenance cost coefficients for energy storage devices, water electrolysis hydrogen production devices, air water intake devices, water storage devices, and hydrogen storage devices, respectively. The rated discharge flow rate of the water storage device; The rated hydrogen release flow rate for the hydrogen storage device; Water network replenishment cost :

[0083] in, The water supply rate of the water transmission network during time period t is determined by the water balance relationship:

[0084] In the formula: The water consumption rate per unit power of the water electrolysis hydrogen production unit; The water production rate per unit power of the air-to-water intake device; Water price for supplementing the water supply network; Safety boundary penalty items :

[0085] In the formula: s1(t), s2(t), and s3(t) represent the state of charge of the energy storage device, the remaining water storage ratio of the water storage device, and the remaining hydrogen storage ratio of the hydrogen storage device at the end of the current time period, respectively. 、 These are the safety lower limits for water storage devices and hydrogen storage devices, respectively. 、 、 Preset penalty weight coefficient; (32) r2: Target for maximizing power supply reliability:

[0086] Where: Total load requirement of base stations Local power supply capability The local power supply capacity is the smaller of the sum of photovoltaic and energy storage discharge and the total load demand; the power supply capacity exceeding the load demand is not included. To prevent the denominator from being zero, a positive decimal is preset. This target adopts the complementary value of the standard load failure probability LOLP. The numerator is the cumulative power shortage that local resources cannot meet the total load demand within the scheduling period, and the denominator is the cumulative power demand of the total load. The value of r2 is in the range of [0,1]. The larger the value, the higher the power supply reliability of the base station when it relies only on local photovoltaic and energy storage devices. (33) r3: Objective to maximize the success rate of UAV missions

[0087] In the formula: N is the total number of UAVs that perform flight missions within the scheduling period; , These are the required power and hydrogen refueling amounts for the i-th drone, respectively, defined in the same state space. , These represent the total power replenishment demand and total hydrogen replenishment demand for drone swarms, respectively. , s6(t), s7(t), and s3(t) represent the total charging capacity and total hydrogen refueling capacity that the base station can actually provide during the scheduling cycle, respectively; s6(t), s7(t), and s3(t) represent the total energy demand, total hydrogen demand, and remaining hydrogen storage ratio of the hydrogen storage device for time period t, respectively, as defined in the state space; a3 is the hydrogen release flow coefficient of the hydrogen storage device, as defined in the action space. The rated hydrogen release flow rate for the hydrogen storage device; The rated hydrogen storage capacity of the hydrogen storage device, in units of... Consistent; The total charging power that the base station can actually provide during time period t is constrained by photovoltaic, energy storage, grid power purchase and charging facility capacity, and is given by the simulation environment after the action is executed. The duration of a single scheduling period; To prevent the denominator from being zero, a positive decimal is preset; r3∈[0,1], the larger the value, the more sufficient the base station's power supply guarantee for the drone swarm. When the total demand is zero, the corresponding ratio is 1.

[0088] All three optimization objectives are uniformly converted into scalar Q values ​​through the nonlinear Chebyshev scalarization module mentioned above, achieving mathematical unification from multiple objectives to a single objective.

[0089] Specifically, the offline master-slave game deduction in step S3 includes the following sub-steps: Step S3-1, Generate the initial guidance stimulus signal: The base station scheduling agent constructs an initial state vector for the state space S, inputs it into the trained Actor main network, and generates and publishes the initial electric energy-guided excitation coefficient curve and the hydrogen energy-guided excitation coefficient curve through a complete forward inference. The equivalent state components of the UAV swarm (s5, s6, s7) in the initial state vector are given in the form of estimated values ​​based on UAV mission prediction information, serving as the starting point for game iteration.

[0090] Step S3-2, follower response: After receiving the guidance stimulus coefficient curve, each of the follower energy management agents uses the guidance stimulus coefficient curve and the predetermined flight mission power profile as inputs to perform full-cycle power allocation through its policy network, generating its own data including the landing period and the desired hydrogen refueling amount. and the final state of battery charge during flight The expected power replenishment demand package is fed back to the base station scheduling agent; This step embodies the rational response mechanism of followers in a master-slave game: given the leader's strategy (guiding incentive coefficient), each follower outputs a power allocation action through a policy network that has undergone reinforcement learning training. This policy network has embedded the guiding incentive coefficient into its reward function, so its output action naturally aims to minimize the energy cost per flight weighted by the guiding incentive coefficient. Since followers do not interact directly with each other, each follower only needs to make independent decisions based on its own task requirements and the guiding incentive signals it receives. This distributed decision-making method effectively reduces the communication and computational complexity of the system.

[0091] Step S3-3, Leader further optimization: After collecting all expected power replenishment request packets, the base station scheduling agent performs the following sub-steps: S3-3-1, Demand Aggregation: Using the energy distribution coefficient The expected hydrogen replenishment amount of each follower Aggregate the total hydrogen demand for energy replenishment in each time period The energy supply distribution coefficient The duration of the intersection between the resupply time window and the scheduling period t of the i-th UAV is determined by the proportion of its total resupply time. S3-3-2, Follow-up Assessment: The final state of battery charge of each follower during flight Conduct a follow-up assessment. If Below the preset ideal connection threshold If the drone needs additional power before its next takeoff, the connection status indicator will be activated. Additional power required Set the connection status flags for each drone. and additional power replenishment needs According to the energy supply distribution coefficient Aggregate the data to obtain the total replenishment power demand for each time period. Equivalent number of drones requiring additional charging at different times .

[0092] S3-3-3, Strategy Update: The base station scheduling agent will include the aggregated... , , The complete state vector of the state space S, including the network, is input into its Actor main network. The network outputs the action vector, which is then decoded to obtain the updated base station day-ahead production scheduling scheme and the guidance incentive coefficient curve. This updated guidance incentive coefficient curve will be republished to each follower in the next iteration.

[0093] Steps S3-4, Iteration and Convergence: Repeat steps S3-2 (follower response) and S3-3 (leader re-optimization) until the change in the guiding incentive coefficient curve between two adjacent iterations is less than a preset threshold ε (range of values). to The system reaches Stackelberg equilibrium. In equilibrium, given the optimal response from followers, the leader cannot increase its utility by unilaterally changing the incentive coefficient, and followers cannot further reduce their energy costs by changing their response strategies. The system achieves global coordination optimality. At equilibrium, the base station scheduling agent decodes and outputs a day-ahead production scheduling scheme from its final action vector, including the next day's electricity purchase plan, electrolysis hydrogen production plan, and energy storage charging and discharging plan. It determines the settlement incentive coefficient for each time period and records the reference energy management strategy generated by each follower energy management agent under the equilibrium incentive coefficient.

[0094] The ideal continuation threshold The value range is 0.5 to 0.7; the preset threshold To guide the convergence criterion of the excitation coefficient curve, the range of values ​​is: to .

[0095] The training of the base station scheduling agent adopts a policy delayed update mechanism: after every two updates to the Critic main network, the Actor main network is updated once; during training, the target network parameters gradually approach the main network parameters through a soft update method. , ,in , These are the parameters for the Actor's main network and target network, respectively. , These are the parameters of the k-th Critic main network and the target network, respectively (k=1,2). This is the soft update coefficient.

[0096] During the training process of the base station scheduling agent, a global parameter update mechanism is triggered after each round: (1) Update the Utopia reference point : After each training round, calculate the cumulative reward for each objective in that round. For each objective i, update the Utopia reference point only when a better cumulative reward is obtained: ,in The preset positive decimal is used to avoid numerical instability caused by the utopian reference point coinciding with the current optimal value; this update mechanism ensures that the utopian reference point is always slightly higher than the currently known optimal solution, continuously driving the algorithm to explore a better Pareto front.

[0097] (2) Adaptive update of preference weights : After each preset K training rounds, the convergence of each objective is calculated. :

[0098] in The cumulative reward for the i-th objective in the most recent round. The cumulative reward reference value for this objective recorded when the Utopia reference point is first updated during the initial training phase; Calculate the original preference weights based on the convergence and introduce a preset lower bound for the weights. Perform cropping and normalization:

[0099] This causes the weight allocation to be tilted toward targets that have not yet been fully optimized, and prevents the weight of any target from degenerating to zero.

[0100] The utopian reference point and preference weights are updated at different frequencies, forming a dual-timescale optimization mechanism. The frequently updated utopian reference point ensures the algorithm always targets a challenging objective; the less frequently updated preference weights, based on long-term convergence information, guide the algorithm to focus more on objectives that are difficult to optimize. This design balances the aggressiveness of exploration with the stability of optimization.

[0101] The base station scheduling agent performs the following single-step interaction and update process in each training round: (1) Obtain an initial state s, wherein the state s contains each component in the state space S; (2) Action selection and safety constraints: Inputting state s into the Actor main network yields deterministic actions. After correction by the safety constraint layer, the output safety action is performed. The safety constraint layer includes: limiting the state of charge (s1) of the energy storage device to the range of 0.2 to 0.8, the remaining water storage ratio (s2) of the water storage device to the range of 0.1 to 0.9, the remaining hydrogen storage ratio (s3) of the hydrogen storage device to no more than 1.0, and the power grid interaction power to no more than the rated capacity of the transformer. (3) Environmental interaction: implement The environment returns to the next state. Multi-objective component reward vector and the round-ending flag "done"; (4) Sample storage: Generate discount mask , As a discount factor; the sample Store the sample in the priority experience replay pool and assign it the highest priority in the current replay pool. (5) Experience playback and network updates: When the number of samples in the replay pool exceeds the preset batch size, the samples are prioritized. The sampling probability distribution is generated by exponentiation, and the importance sampling weights are calculated. The following sub-steps are performed on the sampled samples: 5.1 Constructing the scalarized TD target value : Calculate the next action using the target network Add truncated noise; obtain the multi-objective Q-value vector of the dual-Critic target network, and obtain the minimum value in each dimension. Construct a multi-objective target value vector The y-value is scalarized using the Chebyshev scalarization function to obtain... If done is true, then mask = 0, and y = r. ; 5.2 Update the dual-Critic master network: Calculate the scalar Q-value of the current state-action pair. , The parameters of the Critic main network are updated using the mean square error weighted by importance sampling weights as the loss function. 5.3 Update sample priority: The absolute value of the deviation between the mean of the dual Critic scalar Q values ​​and the scalar TD target value As the TD error, the absolute value of the TD error is written back to the corresponding sample position as the new priority. 5.4 Delayed updates to the Actor main network: After every two updates to the Critic main network, minimize the scalar Q value output by the first Critic main network. Update the Actor's main network parameters with this as the goal. ; 5.5 Soft update target network: After each update of the Actor main network, the soft update factor is applied. Update the target network parameters; (6) State transition: make If done is true, the current round ends; otherwise, return to step (2).

[0102] Specifically, the output and application of the game equilibrium result in step S4 are as follows: According to the aforementioned daily production scheduling plan, the base station performs electricity purchase, photovoltaic power generation, electrolytic hydrogen production, and energy storage charging and discharging operations in various time periods of the following day; when the UAV performs actual flight missions, its onboard energy management intelligent agent uses the equilibrium guidance excitation coefficient curve as input to perform real-time power allocation between fuel cells and lithium batteries online; when the UAV lands to replenish energy, the energy replenishment settlement is carried out according to the settlement guidance excitation coefficient of the landing time period.

[0103] Specifically, the follower energy management agent in step S2 is constructed by including: Neural network architecture: includes follower Actor main network, follower Actor target network, follower Critic main network and follower Critic target network, wherein the Critic network adopts a double Q network structure; Training mechanism: A strategy of delayed update is adopted, in which the Actor main network is updated once after every two updates of the Critic main network, so that the update frequency of the Actor main network is lower than that of the Critic main network. The aforementioned architecture and training mechanism together form the foundation for the implementation of the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm.

[0104] The state space of the follower agent Including: the drone's current battery state of charge The drone's current remaining hydrogen content Current flight power requirements , determined by the power profile of the predetermined flight mission; Current received actual electrical energy guiding excitation value and actual hydrogen energy guiding incentive value (Originated by linear mapping of the incentive coefficient issued by the leader); The action space of the follower agent For a continuous scalar: Fuel cell output power coefficient Multiply by the rated power of the airborne fuel cell stack to obtain the actual fuel cell output power. ; Lithium battery output power Determined by instantaneous power balance constraints: ,in This is for DC / DC converter efficiency; this power balance relationship ensures that at any given time, the sum of the output power of the fuel cell and the lithium battery can precisely meet the flight power requirements of the drone.

[0105] The single-step reward function of the follower energy management agent for:

[0106] The first item only charges for the discharge power of the lithium battery, while the charging process is indirectly powered by the fuel cell through hydrogen consumption. For airborne fuel cell stack efficiency, Hydrogen has a low calorific value; The penalty weight is applied to deviations in battery state of charge. This reward function uses the leader's guiding incentive signals as energy cost weights, so that followers naturally develop a compatible response to the leader's incentive signals while minimizing their own energy costs.

[0107] The training process for the follower agent is as follows: Under different flight mission power profiles and different combinations of guidance and stimulus signals, with the goal of maximizing cumulative reward, the follower agent and criterion networks are trained by collecting experience samples through interaction with the training environment. At each control step, the follower agent obtains the state space from the environment. The current state is determined by the fuel cell power coefficient output by the Actor main network. The safe action is obtained after correction by the safety constraint layer. After the action is performed, the environment returns to the next state and the instant reward is given. Interaction samples are stored in an experience replay pool. When the sample size is sufficient, batches of samples are randomly sampled from the pool. The target Q-value is calculated using a dual-Critic target network. The Critic main network is updated using mean squared error as the loss function, and the Actor main network is updated according to the policy-delayed update frequency. Finally, the target network parameters are updated using a soft update method. Through the above training, the follower agent learns a power allocation strategy that minimizes energy costs under different guidance excitation signals and flight missions.

[0108] The trained follower energy management agent plays the following two key roles in the entire master-slave game and its actual operation: (1) Demand feedback role in master-slave game: In the offline master-slave game simulation described in step S3, the follower agent receives the guidance incentive coefficient curve released by the leader, and uses the guidance incentive coefficient curve and the predetermined flight mission power profile as input to perform full-cycle power allocation through its policy network, generating a power allocation including landing time and expected hydrogen replenishment amount. and the final state of battery charge during flight The expected power replenishment demand package is fed back to the base station scheduling agent, providing a basis for decision-making for the leader's demand aggregation, continuity assessment and strategy update.

[0109] (2) Online real-time power allocation role: In the next day's actual execution phase described in step S4, the UAV will input the equalization guidance excitation coefficient and real-time flight status into its trained policy network, output power allocation action online, and execute it after being corrected by the safety constraint layer to complete real-time adaptive energy management in flight.

[0110] It should be specifically noted that the above-described follower agent based on the TD3 algorithm is a preferred embodiment of the present invention. The deep reinforcement learning framework for the follower can be implemented using other policy gradient algorithms (such as DDPG, SAC, etc.). Its core lies in the follower's incentive-compatible behavior pattern of generating a power allocation scheme and feeding back the expected energy replenishment package through a trained neural network model based on the guiding incentive signals issued by the leader.

[0111] This invention also provides a base station-UAV energy scheduling system for solving master-slave game using DRL. The base station-UAV energy scheduling system for solving master-slave game using DRL can be implemented by executing the process steps of the base station-UAV energy scheduling method for solving master-slave game using DRL. That is, those skilled in the art can understand the base station-UAV energy scheduling method for solving master-slave game using DRL as a preferred embodiment of the base station-UAV energy scheduling system for solving master-slave game using DRL.

[0112] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0113] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A base station-UAV energy scheduling method for solving master-slave game using DRL, characterized in that, include: Step S1: Establish a two-layer architecture of physical and information layers for master-slave game theory; wherein, the physical layer is used to clarify the equipment composition and energy flow relationship between the ground UAV base station subsystem as the energy supply leader and the airborne hydrogen-electric hybrid power subsystem as the energy consumer follower; the information layer is used to clarify the information interaction protocol between the leader and the follower; the leader issues guidance incentive signals to the follower, and the follower feeds back to the leader a desired energy replenishment demand package including landing time, expected hydrogen replenishment amount, and final battery charge state of flight; Step S2: Construct follower energy management agent and base station scheduling agent based on deep reinforcement learning for the leader and follower in the master-slave game, respectively, and train the follower energy management agent and base station scheduling agent separately to obtain the trained follower energy management agent and the trained base station scheduling agent. Step S3: Based on the trained follower energy management agent and the trained base station scheduling agent, trigger the offline master-slave game deduction before the execution date, seek the Stackelberg equilibrium, and obtain the base station production scheduling scheme before the execution date, the settlement guidance incentive coefficient for each time period, and the output of the reference energy management strategy for each UAV. Step S4: Use the base station daytime production scheduling scheme, settlement guidance incentive coefficients for each time period, and reference energy management strategies for each UAV obtained from the game equilibrium for actual execution the next day.

2. The base station-UAV energy scheduling method for solving master-slave game using DRL as described in claim 1, characterized in that, The guiding excitation signal includes: electrical energy guiding excitation coefficient and hydrogen energy guiding excitation coefficient; The electrical energy incentive coefficient and the hydrogen energy incentive coefficient are dimensionless strategy variables formulated by the leader to guide the followers' electricity consumption behavior and hydrogen consumption behavior, respectively. The higher the value, the higher the unit cost of the corresponding energy, thereby incentivizing the followers to reduce the consumption of that energy or adjust their energy consumption time.

3. The base station-UAV energy scheduling method for solving master-slave game using DRL as described in claim 1, characterized in that, The base station scheduling agent includes a state space S, an action space A, and a multi-objective component reward vector. ; Wherein, the state space S is an observation vector constructed based on hourly scheduling compensation, including: the base station's own state and the equivalent state of the UAV swarm; The base station's own status includes: the charging status of the energy storage device s1, the remaining water storage ratio of the water storage device s2, the remaining hydrogen storage ratio of the hydrogen storage device s3, and the real-time output power of the photovoltaic power generation device s4. The equivalent state of the drone swarm is a macroscopic time-series variable generated by the base station scheduling agent after performing aggregation and continuity evaluation on the expected power replenishment demand packets fed back by each follower; Wherein, the scheduling period t=1,2,…,T, where T is the total number of time periods divided by the daily scheduling cycle; for each time period t, the equivalent state of the UAV swarm includes: the equivalent number of UAVs that need additional power replenishment in each time period s5, the total power replenishment requirement in each time period s6, and the total hydrogen replenishment requirement in each time period s7. The summation iterates through all drones i=1,2,…,N; The energy allocation coefficient is defined as the proportion of the intersection of the resupply time window and the scheduling period t of the i-th UAV to its total resupply time, and satisfies the following condition: The refueling time window is determined jointly by the landing period in the expected energy demand package and the refueling duration based on the expected hydrogen replenishment amount; when the refueling time window does not cover the period t, ; As a status indicator, if the final state of battery charge during flight... Below the ideal continuation threshold ,but ,otherwise ; The additional power required is determined by the base station scheduling agent after performing a continuity assessment on the i-th drone; when hour, ;when hour, ,in, The rated capacity of the airborne power lithium battery pack; The expected amount of hydrogen replenishment in the expected energy replenishment demand package reported by the i-th UAV; The action space A is a continuous action vector that includes the operating power instructions and incentive strategy instructions of each controllable energy conversion and storage device of the base station within a scheduling step. The action space A includes: the charging and discharging power coefficient of the energy storage device a1, a1∈[-1,1], a1>0 indicates discharging, a1<0 indicates charging, and its absolute value is multiplied by the rated charging and discharging power of the energy storage device to obtain the actual charging and discharging power. The water discharge flow coefficient of the water storage device is a2, a2∈[0,1], which is multiplied by the rated water discharge flow of the water storage device to obtain the actual water discharge flow. The hydrogen release flow rate coefficient of the hydrogen storage device is a3, where a3∈[0,1]. Multiplying it by the rated hydrogen release flow rate of the hydrogen storage device yields the actual hydrogen release flow rate. The operating power coefficient a4 of the water electrolysis hydrogen production unit, a4∈[0,1], is multiplied by the rated power of the water electrolysis hydrogen production unit to obtain the actual operating power; The operating power coefficient of the air-water intake device is a5, where a5∈[0,1]. Multiplying it by the rated power of the air-water intake device yields the actual operating power. The electrical energy guiding excitation coefficient is a6, where a6∈[0,1]; The hydrogen energy guiding excitation coefficient is a7, where a7∈[0,1]; Wherein, the electrical energy guiding excitation coefficient a6 and the actual electrical energy guiding excitation value The mapping relationship between them is as follows: The hydrogen energy guiding excitation coefficient a7 and the actual hydrogen energy guiding excitation value The mapping relationship between them is as follows: In the formula, and These are the preset lower and upper limits of the electrical energy guiding excitation value, respectively. and These are the preset lower and upper limits of the hydrogen energy guiding excitation value, respectively; The multi-objective sub-reward vector There are three optimization objectives, all of which are transformed into maximization directions, including: r1: The objective of minimizing economic operating costs is transformed into a reward component in the maximization direction, defined as the negative value of the total economic operating cost within the scheduling period: Among them, the cost of purchasing electricity from the power grid : In the formula: The duration of a single scheduling period; The external grid time-of-use electricity price for time period t; The power exchanged between the base station and the upstream power grid is determined by the power balance equation: in: The total power load inside the base station during time period t. ; , These are the rated powers of the water electrolysis hydrogen production unit and the air-to-water extraction unit, respectively. Other auxiliary load constants; , These are the charging power and discharging power of the energy storage device, respectively. , , The rated charge and discharge power of the energy storage device; Equipment operation and maintenance costs : In the formula: 、 、 、 、 These are the unit power or unit flow rate operation and maintenance cost coefficients for energy storage devices, water electrolysis hydrogen production devices, air water intake devices, water storage devices, and hydrogen storage devices, respectively. The rated discharge flow rate of the water storage device; The rated hydrogen release flow rate for the hydrogen storage device; Water network replenishment cost : in, The water supply rate of the water transmission network during time period t is determined by the water balance relationship: In the formula: The water consumption rate per unit power of the water electrolysis hydrogen production unit; The water production rate per unit power of the air-to-water intake device; Water price for supplementing the water supply network; Safety boundary penalty items : In the formula: s1(t), s2(t), and s3(t) represent the state of charge of the energy storage device, the remaining water storage ratio of the water storage device, and the remaining hydrogen storage ratio of the hydrogen storage device at the end of the current time period, respectively. 、 These are the safety lower limits for water storage devices and hydrogen storage devices, respectively. 、 、 Preset penalty weight coefficient; r2: Power supply reliability maximization objective: Where: Total load requirement of base stations Local power supply capability The local power supply capacity is the smaller of the sum of photovoltaic and energy storage discharge and the total load demand; the power supply capacity exceeding the load demand is not included. To prevent the denominator from being zero, a positive decimal is preset. This target adopts the complementary value of the standard load failure probability LOLP. The numerator is the cumulative power shortage that local resources cannot meet the total load demand within the scheduling period, and the denominator is the cumulative power demand of the total load. The value of r2 is in the range of [0,1]. The larger the value, the higher the power supply reliability of the base station when it relies only on local photovoltaic and energy storage devices. r3: Maximize the success rate of drone missions In the formula: N is the total number of UAVs that perform flight missions within the scheduling period; , These are the required power and hydrogen refueling amounts for the i-th drone, respectively, defined in the same state space. , These represent the total power replenishment demand and total hydrogen replenishment demand for drone swarms, respectively. , s6(t), s7(t), and s3(t) represent the total charging capacity and total hydrogen refueling capacity that the base station can actually provide during the scheduling cycle, respectively; s6(t), s7(t), and s3(t) represent the total energy demand, total hydrogen demand, and remaining hydrogen storage ratio of the hydrogen storage device for time period t, respectively, as defined in the state space; a3 is the hydrogen release flow coefficient of the hydrogen storage device, as defined in the action space. The rated hydrogen release flow rate for the hydrogen storage device; The rated hydrogen storage capacity of the hydrogen storage device, in units of... Consistent; The total charging power that the base station can actually provide during time period t is constrained by photovoltaic, energy storage, grid power purchase and charging facility capacity, and is given by the simulation environment after the action is executed. The duration of a single scheduling period; To prevent the denominator from being zero, a positive decimal is preset; r3∈[0,1], the larger the value, the more sufficient the base station's power supply guarantee for the drone swarm. When the total demand is zero, the corresponding ratio is 1.

4. The base station-UAV energy scheduling method for solving master-slave game using DRL as described in claim 3, characterized in that, The base station scheduling agent adopts a multi-objective Actor-Critic architecture with multi-objective sub-item reward vectors and nonlinear scalarization modules. It is used to generate updated guidance incentive signals and base station day-ahead production scheduling schemes through its policy network and value network after integrating the expected energy replenishment demand package fed back by followers. The base station scheduling agent performs the following single-step interaction and update in each training round, including: Obtain state s and input state s into the Actor main network to obtain deterministic actions. After being corrected by a preset safety constraint layer, a safe action is output. ; Perform safety actions The environment interaction returns to the next state. Multi-objective component reward vector and the round-end flag "done"; generate a discount mask. , Discount factor; Sample Store the sample in the priority experience replay pool and assign it the highest priority in the current replay pool. When the number of samples in the replay pool exceeds the preset batch size, the samples are prioritized. The sampling probability distribution is generated by exponentiation, and the importance sampling weights are calculated. The following sub-steps are performed on the sampled samples: Calculate the next action using the target network And add truncated noise; Obtain the multi-objective Q-value vector of the dual-Critic target network by taking the minimum value in each dimension. ; Constructing multi-objective target value vectors The multi-objective target value vector y is scalarized using a nonlinear Chebyshev scalarization function to obtain... If done is true, then mask = 0, and y = r. ; Calculate the scalar Q-value of the current state-action pair. , The parameters of the Critic main network are updated using the mean square error weighted by importance sampling weights as the loss function. in, For importance sampling weights, To scalarize the Q value, To quantify the TD target value; The absolute value of the deviation between the mean of the dual Critic scalar Q values ​​and the scalar TD target value As the TD error, the absolute value of the TD error is written back to the corresponding sample position as the new priority. After every two updates to the Critic main network, minimize the scalar Q value output by the first Critic main network. Update the Actor's main network parameters with this as the goal. ; After each update of the Actor main network, the soft update factor is applied. Update the target network parameters; make If done is true, the current round ends; otherwise, the execution is repeated until the current round ends.

5. The base station-UAV energy scheduling method for solving master-slave game using DRL as described in claim 4, characterized in that, The experience replay pool is a priority experience replay pool, including: Interaction samples are stored in a fixed-capacity circular buffer. Each sample contains a state, a multi-objective sub-reward vector, a discount mask, and an action. Newly stored samples are assigned the highest priority in the current buffer to ensure that new samples can be sampled and learned preferentially. Based on the priority of each sample The sampling probability distribution is generated by exponentiation, and random sampling is performed according to this distribution. Importance sampling weights are then calculated to correct for distribution bias introduced by priority sampling. To preset hyperparameters, It degenerates into uniform sampling; The absolute value of the TD error obtained after the sampled sample is evaluated by the network is used as the new priority, and it is written back to the corresponding sample position to realize the dynamic update of the priority.

6. The base station-UAV energy scheduling method for solving master-slave game using DRL as described in claim 1, characterized in that, The follower energy management agent adopts an Actor-Critic architecture of policy network and value network to generate a flight power allocation scheme based on the guidance and incentive signals received from the leader, and thereby obtain a desired energy replenishment demand package including landing time, expected hydrogen replenishment amount and final battery state of charge.

7. The base station-UAV energy scheduling method for solving master-slave game using DRL as described in claim 1, characterized in that, Step S3 includes: Step S3.1: Generate initial guidance stimulus signal: The base station scheduling agent constructs an initial state vector in the state space S, inputs it into its Actor main network, and generates and publishes the initial electric energy guiding excitation coefficient curve and hydrogen energy guiding excitation coefficient curve. Step S3.2: Follower Response: After receiving the guidance stimulus coefficient curve, each of the follower energy management agents uses the guidance stimulus coefficient curve and the predetermined flight mission power profile as inputs to perform full-cycle power allocation through its policy network, generating its own data including the landing period and the desired hydrogen refueling amount. and the final state of battery charge during flight The expected power replenishment demand package is fed back to the base station scheduling agent; Step S3.3; Leader Re-optimization: After collecting all expected power replenishment request packets, the base station scheduling agent performs the following sub-steps: Step S3.3.1: Requirement Aggregation: Using the energy distribution coefficient The expected hydrogen replenishment amount of each follower Aggregate the total hydrogen demand for energy replenishment in each time period ; S3.3.2: Succession Assessment: The final state of battery charge of each follower during flight Conduct a connection assessment and determine the connection status indicators. And the additional power required This is aggregated into the total replenishment power demand for each time period. And the equivalent number of drones that need additional charging at different times. ; S3.3.3: Strategy Update: The base station scheduling agent will include the aggregated... , , The complete state vector of the state space S, including the Actor main network, is input into its Actor main network. The network outputs the action vector, and after decoding, the updated base station day-ahead production scheduling scheme and the guiding incentive coefficient curve are obtained. Repeatedly trigger steps S3.2 to S3.3 until the change amplitude of the guiding incentive coefficient curve in two adjacent iterations is less than the preset threshold ε, reaching Stackelberg equilibrium; at equilibrium, the base station scheduling agent decodes and outputs a day-ahead production scheduling scheme containing the next day's power purchase plan, electrolysis hydrogen production plan, and energy storage charging and discharging plan from its final action vector, determines the settlement guiding incentive coefficient for each time period, and records the reference energy management strategy generated by each follower energy management agent under the equilibrium guiding incentive coefficient.

8. The base station-UAV energy scheduling method for solving master-slave game using DRL as described in claim 1, characterized in that, The training of the base station scheduling agent adopts a policy delayed update mechanism, including: updating the Actor main network once after updating the Critic main network twice; during training, the target network parameters gradually approach the main network parameters through a soft update method. , ,in , These are the parameters for the Actor's main network and target network, respectively. , These are the parameters of the k-th Critic main network and the target network, respectively, where k = 1, 2, This is the soft update coefficient.

9. The base station-UAV energy scheduling method for solving master-slave game using DRL as described in claim 1, characterized in that, During the training process of the base station scheduling agent, a global parameter update mechanism is triggered after each round, including: Update Utopia Reference Points ,include: After each training round, calculate the cumulative reward for each objective in that round. For each objective i, update the Utopia reference point only when a better cumulative reward is obtained: ,in, It is a preset positive decimal to avoid numerical instability caused by the utopian reference point coinciding with the current optimal value; Adaptive update of preference weights : After each preset K training rounds, the convergence of each objective is calculated. : in, The cumulative reward for the i-th objective in the most recent round. The cumulative reward reference value for this objective recorded when the Utopia reference point is first updated during the initial training phase; Calculate the original preference weights based on the convergence and introduce a preset lower bound for the weights. Perform cropping and normalization: This causes the weight allocation to be tilted toward targets that have not yet been fully optimized, and prevents the weight of any target from degenerating to zero.

10. A base station-UAV energy scheduling system for solving master-slave game using DRL, characterized in that, include: Module M1: Establishes a two-layer architecture of physical and information layers for master-slave game theory; wherein, the physical layer is used to clarify the equipment composition and energy flow relationship between the ground UAV base station subsystem as the energy supply leader and the airborne hydrogen-electric hybrid power subsystem as the energy consumer follower; the information layer is used to clarify the information interaction protocol between the leader and the follower; the leader issues guidance and incentive signals to the follower, and the follower feeds back to the leader a desired energy replenishment demand package including landing time, expected hydrogen replenishment amount, and final battery charge state of flight; Module M2: Construct follower energy management agents and base station scheduling agents based on deep reinforcement learning for the leader and follower in the master-slave game, respectively, and train the follower energy management agents and base station scheduling agents separately to obtain the trained follower energy management agents and trained base station scheduling agents. Module M3: Based on the trained follower energy management agent and the trained base station scheduling agent, the module performs offline master-slave game deduction before the execution date to seek Stackelberg equilibrium and obtain the base station day-ahead production scheduling scheme, the settlement guidance incentive coefficient for each time period, and the output of the reference energy management strategy for each UAV. Module M4: The base station daytime production scheduling scheme obtained from the game equilibrium, the settlement guidance incentive coefficients for each time period, and the reference energy management strategies for each UAV are used for actual execution the next day.

Citation Information

Patent Citations

  • Hydrogen fuel cell hybrid power unmanned aerial vehicle energy self-adaptive optimization system and method

    CN119167791B

  • Energy management method for hydrogen fuel hybrid power unmanned aerial vehicle of OPSA-ECMS

    CN119623313A