A method and system for energy management of a composite power supply of a UAV based on deep reinforcement learning

Abstract

The application discloses a kind of unmanned plane compound power energy management method and system based on deep reinforcement learning, it is related to unmanned plane compound power energy management technical field.Due to having vertical take-off and landing ability and efficient cruising performance, the application demand of hybrid vertical take-off and landing unmanned plane in logistics, monitoring and patrol and other fields is increasing rapidly.Lack of customized strategy specially for hybrid vertical take-off and landing unmanned plane.This study proposes a kind of multi-objective deep reinforcement learning framework, aims at maximizing the overall efficiency of fuel cell hybrid power system.A comprehensive modeling method is constructed, including hybrid power assembly, unmanned aerial vehicle power demand curve and efficiency-oriented learning algorithm.Comparison experiment with equivalent consumption minimization strategy verifies that the proposed method significantly improves the overall system efficiency under various flight conditions, while effectively reducing the operating stress of fuel cell.

Description

Technical Field

[0001] This invention relates to the field of composite power management technology for unmanned aerial vehicles (UAVs), and more specifically, to a method and system for composite power management of UAVs based on deep reinforcement learning. Background Technology

[0002] Hybrid vertical takeoff and landing (H-VTOL) unmanned aerial vehicles (UAVs) combine the structural features of fixed-wing and rotary-wing aircraft. This dual-configuration design allows the system to possess the high speed and long endurance advantages of fixed-wing aircraft, as well as the inherent vertical maneuverability of rotary-wing platforms. Currently, these UAVs primarily employ propulsion systems such as pure electric, internal combustion engines, fuel cells, or hydrogen energy. Although they are inferior to traditional fixed-wing UAVs in terms of energy consumption and speed, they are not limited by takeoff and landing sites, possessing superior environmental adaptability and broader application prospects. H-VTOL UAVs can achieve vertical takeoff by generating vertical thrust exceeding their own weight through their propulsion system. After climbing to a safe altitude, they switch to horizontal flight mode, relying on the aerodynamic lift of their wings for cruising. These UAVs feature low takeoff and landing requirements, hovering capability, long endurance, and large operating radius, making them a key research focus in UAV technology development and demonstrating increasingly broad application potential.

[0003] To compensate for the slow transient response of fuel cells, a hybrid architecture combining fuel cells and auxiliary power sources is often used in practical flight applications. Managing the complex energy flow in such multi-source systems requires the deployment of coordinated energy management strategies. Typically, the core objectives of energy management strategies focus on three aspects: first, limiting fuel cells to operate within their optimal efficiency or steady-state range; second, maximizing the synergistic potential of each power unit under fluctuating operating conditions; and third, improving system stability and energy economy through global optimization, while extending component lifespan. From a methodological perspective, mainstream energy management strategies can be broadly categorized into three types: rule-based strategies, optimization-based strategies, and intelligence-based strategies.

[0004] Rule-based energy management strategies (such as fuzzy logic control and state machines) are widely used due to their simple structure, reliable operation, and high computational efficiency. Among them, the state machine method classifies the system operating conditions by monitoring parameters such as load power and battery state of charge (SOC), and then executes corresponding control commands.

[0005] Currently, a power switching technology using solid-state relays has been designed for fuel cell / solar-battery hybrid UAVs to cope with flight load fluctuations and flexibly adjust the power circuit. Similarly, an energy management strategy based on dynamic balance rules is proposed, defining three operating modes according to the difference between the state of charge (SOC) and hydrogen reserves, effectively preventing premature energy depletion and improving system reliability. A state machine is used for power management, decoupling system operation into startup, charging, and high-power phases based on battery terminal voltage and bus current. To evaluate the performance of the fuel cell / lithium polymer battery hybrid architecture, a multi-dimensional conditional strategy integrating voltage, current, hydrogen state, and SOC is adopted. Although such methods are easy to implement in engineering, their strategy design heavily relies on expert experience, making it fundamentally difficult to guarantee the system's globally optimal performance.

[0006] Therefore, it is necessary to use optimized energy management strategies guided by specific design objectives. This involves constructing a cost function that integrates system state and physical constraints, and then applying optimization algorithms to calculate the theoretically optimal control sequence for power allocation. Optimized energy management strategies are often limited by high computational costs and heavily rely on accurate system modeling, making them difficult to apply in real-time online scenarios. Furthermore, these methods typically require prior knowledge of the global information of the mission load curve, necessitating a trade-off between computational speed and optimization accuracy. To address these limitations, a data-driven intelligent technology, represented by neural networks and reinforcement learning, is needed. This technology aims to meet flight load requirements while simultaneously optimizing total system losses, fuel cell stress, battery state-of-charge balance, and DC bus voltage stability, all while meeting flight load demands. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for managing the composite power supply of unmanned aerial vehicles (UAVs) based on deep reinforcement learning, so as to solve the above-mentioned problems existing in the prior art.

[0008] In a first aspect, embodiments of the present invention provide a method for managing the composite power supply of a UAV based on deep reinforcement learning, comprising:

[0009] Acquire battery circuit simulation data and force data; the battery circuit simulation data includes proton exchange membrane fuel cell simulation data and lithium battery pack simulation data; the force data represents the force data of the UAV under different vertical and horizontal flight conditions;

[0010] Based on the force data, the required power for flight is obtained; the required power for flight is either the required power for vertical flight or the required power for horizontal flight.

[0011] Construct a proton exchange membrane fuel power loss model and a lithium battery pack power loss model;

[0012] By employing a deep deterministic strategy gradient algorithm, a proton exchange membrane fuel power loss model, and a lithium battery pack power loss model, multi-objective energy management is performed based on the battery circuit simulation data and flight power requirements to obtain the fuel cell output power and the lithium battery pack output power.

[0013] Optionally, the step of using a deep deterministic strategy gradient algorithm, a proton exchange membrane fuel power loss model, and a lithium battery pack power loss model, based on the battery circuit simulation data and flight power requirements, to perform multi-objective energy management and obtain the fuel cell output power and lithium battery pack output power includes:

[0014] The simulated data for the lithium battery pack includes the battery state of charge.

[0015] Obtain the bus voltage deviation value and the fuel cell output power at the initial moment; the fuel cell output power at the initial moment represents the output power of the proton exchange membrane fuel cell in the equivalent circuit at the initial moment; the bus voltage deviation value represents the voltage deviation value of the bus connected to the equivalent circuit;

[0016] A state space is constructed based on the flight power requirement, battery state of charge, bus voltage deviation, and fuel cell output power at the initial moment.

[0017] The action space is defined as the fuel cell output power at time t; the fuel cell output power represents the output power of the proton exchange membrane fuel cell in the equivalent circuit.

[0018] A reward function is constructed based on the proton exchange membrane fuel power loss model and the lithium battery pack power loss model, using the simulation data of the proton exchange membrane fuel cell and the simulation data of the lithium battery pack.

[0019] Based on the state space, action space, and reward function, the fuel cell output power at time t is obtained using a deep deterministic policy gradient algorithm.

[0020] The output power of the lithium battery pack at time t is obtained based on the fuel cell output power and the flight power requirement at time t.

[0021] Optionally, the step of constructing a reward function based on the proton exchange membrane fuel power loss model and the lithium battery pack power loss model, using the proton exchange membrane fuel cell simulation data and the lithium battery pack simulation data, includes:

[0022] The proton exchange membrane fuel power loss model and the lithium battery pack power loss model were used to obtain the proton exchange membrane fuel power loss value and the lithium battery pack power loss value, respectively.

[0023] Based on the fuel cell output power, battery state of charge, bus voltage deviation, proton exchange membrane fuel power loss, and lithium battery pack power loss at the initial moment, the power change value, state of charge deviation value, bus voltage fluctuation value, and total power loss value are obtained.

[0024] A reward function is constructed based on the power change value, the state of charge deviation value, the bus voltage fluctuation value, and the total power loss value.

[0025] The reward function is as follows:

[0026]

[0027] in, This is the current reward value. The weighting coefficients, This represents the total power loss value. Indicates the power change value. This represents the deviation from the state of charge. This represents the bus voltage fluctuation value. This is a physical constraint penalty term.

[0028] Optionally, the step of obtaining the power change value, state of charge deviation value, bus voltage fluctuation value, and total power loss value based on the fuel cell output power, battery state of charge, bus voltage deviation value, proton exchange membrane fuel power loss value, and lithium battery pack power loss value at the initial time includes:

[0029] The total power loss value is obtained by adding the proton exchange membrane fuel power loss value to the lithium battery pack power loss value.

[0030] The power change value is obtained based on the fuel cell output power at the initial moment;

[0031] The formula for obtaining the power change value is as follows:

[0032]

[0033] in, This represents the power change value. This indicates the rated power of a proton exchange membrane fuel cell. The fuel output power at the current time point. This represents the fuel output power at the previous time point;

[0034] The squared difference between the battery's state of charge and a reference state of charge is used as the state of charge deviation value; the reference state of charge refers to the reference state of charge of the lithium battery.

[0035] The square of the bus voltage deviation is taken as the bus voltage fluctuation value.

[0036] Optionally, the state space is as follows:

[0037]

[0038] in, Let be the state vector at time t. Let be the power required for flight at time t; SOC(t) is the state of charge of the battery at time t. Let be the fuel cell output power at time t-1. Let t be the bus voltage deviation value at time t.

[0039] Optionally, obtaining the flight power requirement based on the force data includes:

[0040] Based on the force data, the drag state of vertical flight is detected to obtain the vertical flight power requirement; the vertical flight power requirement represents the power required by the UAV during vertical takeoff, landing and hovering phases.

[0041] Based on the force data, the drag state of horizontal flight is detected to obtain the power required for horizontal flight; the power required for horizontal flight represents the power required by the UAV during the horizontal cruise phase.

[0042] Optionally, the construction of the proton exchange membrane fuel power loss model and the lithium battery pack power loss model includes:

[0043] Multiple data pairs are acquired; the data pairs are the fuel cell output power and the corresponding fuel cell loss power obtained by discrete sampling at historical time points;

[0044] Based on the data pairs, a power loss model for proton exchange membrane fuel cells is constructed through quadratic polynomial fitting.

[0045] Based on the proton exchange membrane fuel cell power loss model, a lithium battery pack power loss model is constructed.

[0046] Optionally, the step of constructing a lithium battery pack power loss model based on the proton exchange membrane fuel cell power loss model includes:

[0047] Construct a first lithium battery pack internal resistance model; the first lithium battery pack internal resistance model represents the values ​​of ohmic internal resistance and polarization internal resistance during charging and discharging;

[0048] The first lithium battery pack internal resistance model includes a charging ohmic internal resistance model, a charging polarization internal resistance model, a discharging ohmic internal resistance model, and a discharging polarization internal resistance model.

[0049] Based on the internal resistance model of the first lithium battery pack, discrete sampling is performed to obtain lithium battery pack data pairs; the lithium battery pack data pairs represent data pairs sampled during discharge and charging of the battery charge state and ohmic internal resistance, and the battery charge state and polarization internal resistance, respectively.

[0050] Based on the lithium battery pack data pairs, a lithium battery pack internal resistance model is constructed through cubic polynomial fitting.

[0051] Based on the internal resistance model of the lithium battery pack, the power loss model of the lithium battery pack is obtained by calculating charging and discharging respectively.

[0052] The power loss model of the lithium battery pack is shown below:

[0053]

[0054] in, This indicates the power loss value of the lithium battery pack. Represents current. A resistor representing ohmic internal resistance. This represents the value of the polarization resistance.

[0055] Optionally, the obtained internal resistance model of the lithium battery pack is as follows:

[0056]

[0057] in, and (x∈{c,d}, i∈{0,1,2,3}) are the polynomial fitting coefficients for charging and discharging modes, respectively. Indicates the battery's state of charge. This indicates the value of the ohm resistance during charging. This indicates the value of the polarization resistance during charging. This indicates the value of the ohm resistance during discharge. This indicates the value of the polarization resistance during discharge.

[0058] Secondly, embodiments of the present invention provide a drone composite power management system based on deep reinforcement learning, including an equivalent circuit simulating a proton exchange membrane fuel cell, a lithium battery pack, and a supercapacitor:

[0059] The equivalent circuit includes a proton exchange membrane fuel cell equivalent circuit, a lithium battery pack equivalent circuit, and a supercapacitor equivalent circuit; the proton exchange membrane fuel cell equivalent circuit represents a simulation circuit containing a proton exchange membrane fuel cell; the lithium battery pack equivalent circuit represents a simulation circuit containing a lithium battery pack; and the supercapacitor equivalent circuit represents a simulation circuit containing a supercapacitor.

[0060] The equivalent circuit of the proton exchange membrane fuel cell is connected to the DC bus via a boost converter; the equivalent circuit of the lithium battery pack is connected to the DC bus via a bidirectional boost converter; the equivalent circuit of the supercapacitor in the equivalent circuit is directly connected in parallel to the DC bus.

[0061] The equivalent circuit of the proton exchange membrane fuel cell includes a circuit model constructed from activation resistor, concentration loss resistor, and ohmic loss resistor, which is used to obtain simulation data of the proton exchange membrane fuel cell.

[0062] The equivalent circuit of the lithium battery pack is a circuit model constructed using the Thevenin equivalent circuit, which is used to obtain simulated data of the lithium battery pack.

[0063] The supercapacitor equivalent circuit represents a circuit model constructed from capacitors and resistors.

[0064] Compared with the prior art, the embodiments of the present invention achieve the following beneficial effects:

[0065] This invention also provides a method and system for managing the composite power supply of unmanned aerial vehicles (UAVs) based on deep reinforcement learning.

[0066] This invention constructs a system model integrating PEMFC, battery, and supercapacitor, and combines it with a fast loss calculation method based on polynomial fitting to support the improvement of large-scale training efficiency. The task is modeled using a Markov decision process, and the system is collaboratively optimized using a composite reward function, simultaneously optimizing global system loss, fuel cell stress, state of charge regulation, and DC bus voltage stability. When the UAV's load power changes significantly, the energy management strategy based on deep reinforcement learning enables the battery to exhibit rapid charge and discharge characteristics, with power changes primarily compensated by battery power, thereby reducing the power change rate of the PEMFC and relatively reducing the operational stress of the PEMFC. Simultaneously, the energy management strategy based on deep reinforcement learning effectively improves the overall system efficiency. Attached Figure Description

[0067] Figure 1 This is a flowchart of a method for managing the composite power supply of a drone based on deep reinforcement learning, provided by an embodiment of the present invention.

[0068] Figure 2 This is a schematic diagram of the equivalent circuit in a deep reinforcement learning-based UAV composite power management system provided in an embodiment of the present invention.

[0069] Figure 3 This is a schematic diagram illustrating the charging and discharging efficiency of a lithium battery pack in a deep reinforcement learning-based composite power management method for unmanned aerial vehicles (UAVs) according to an embodiment of the present invention.

[0070] Figure 4This is a schematic diagram illustrating the performance of the vertical takeoff phase in a deep reinforcement learning-based UAV composite power management method provided in an embodiment of the present invention.

[0071] Figure 5 This is a schematic diagram illustrating the performance of the flight mode switching stage in a deep reinforcement learning-based UAV composite power management method provided in an embodiment of the present invention.

[0072] Figure 6 This is a schematic diagram illustrating the performance of the fixed-wing cruise phase in a deep reinforcement learning-based UAV composite power management method provided in an embodiment of the present invention.

[0073] Figure 7 This is a schematic diagram illustrating the performance of the rotor hovering phase in a deep reinforcement learning-based UAV composite power management method provided in an embodiment of the present invention.

[0074] Figure 8 This is a schematic diagram illustrating the performance of a deep reinforcement learning-based UAV composite power management method during the frequent maneuvering phase, as provided in an embodiment of the present invention.

[0075] Figure 9 This is a schematic diagram illustrating the performance of the vertical landing phase in a deep reinforcement learning-based UAV composite power management method provided in an embodiment of the present invention. Detailed Implementation

[0076] The present invention will now be described in detail with reference to the accompanying drawings.

[0077] Example 1:

[0078] like Figure 1 As shown, this embodiment of the invention provides a method for managing the power of a UAV composite power source based on deep reinforcement learning. The method includes:

[0079] S101: Acquire battery circuit simulation data and force data; the battery circuit simulation data includes proton exchange membrane fuel cell simulation data and lithium battery pack simulation data; the force data represents the force data of the UAV under different vertical and horizontal flight conditions.

[0080] The lithium battery pack simulation data includes the battery state of charge; the proton exchange membrane fuel cell simulation data includes the fuel cell output power at the initial moment.

[0081] The force data includes the total mass of the UAV (m), gravitational acceleration (g), air density (p), and vertical climb rate (m / g). ), vertical projection reference area of ​​the fuselage ( Vertical drag coefficient ( ), cruising speed ( ), Wing Reference Area ( ), zero-lift drag coefficient ( ), lift coefficient ( Oswald efficiency factor (e), wing aspect ratio (AR), overall lift-to-drag ratio (E), and total efficiency of the horizontal propulsion system ( ).

[0082] S102: Based on the force data, obtain the required power for flight; the required power for flight is either the required power for vertical flight or the required power for horizontal flight.

[0083] S103: Construct a proton exchange membrane fuel power loss model and a lithium battery pack power loss model;

[0084] S104: Using a deep deterministic strategy gradient algorithm, a proton exchange membrane fuel power loss model, and a lithium battery pack power loss model, multi-objective energy management is performed based on the battery circuit simulation data and flight power requirements to obtain the fuel cell output power and the lithium battery pack output power.

[0085] Optionally, the step of using a deep deterministic strategy gradient algorithm, a proton exchange membrane fuel power loss model, and a lithium battery pack power loss model, based on the battery circuit simulation data and flight power requirements, to perform multi-objective energy management and obtain the fuel cell output power and lithium battery pack output power includes:

[0086] The simulated data for the lithium battery pack includes the battery state of charge.

[0087] Obtain the bus voltage deviation value and the fuel cell output power at the initial moment; the fuel cell output power at the initial moment represents the output power of the proton exchange membrane fuel cell in the equivalent circuit at the initial moment; the bus voltage deviation value represents the voltage deviation value of the bus connected to the equivalent circuit.

[0088] The formula for calculating the bus voltage deviation is as follows: The The voltage value of the bus connected to the equivalent circuit at time t is represented by the following: This is the DC bus reference voltage of the equivalent circuit.

[0089] The state space is constructed based on the flight power requirement, battery state of charge, bus voltage deviation, and fuel cell output power at the initial moment.

[0090] The state space is as follows:

[0091]

[0092] in, Let be the state vector at time t. Let be the power required for flight at time t; SOC(t) is the state of charge of the battery at time t. Let be the fuel cell output power at time t-1. Let t be the bus voltage deviation value at time t.

[0093] The action space is defined as the fuel cell output power at time t; the fuel cell output power represents the output power of the proton exchange membrane fuel cell in the equivalent circuit.

[0094] The action space is as follows:

[0095]

[0096] in, Represents the action space at time t. Let represent the fuel cell output power at time t. and This indicates the rated range of fuel cell output power.

[0097] A reward function is constructed based on the proton exchange membrane fuel power loss model and the lithium battery pack power loss model, using the simulation data of the proton exchange membrane fuel cell and the simulation data of the lithium battery pack.

[0098] Based on the state space, action space, and reward function, the fuel cell output power at time t is obtained using a deep deterministic policy gradient algorithm.

[0099] The output power of the lithium battery pack at time t is obtained based on the fuel cell output power and the flight power requirement at time t.

[0100] The method for obtaining the output power of the lithium battery pack is as follows:

[0101]

[0102] in, Let be the output power of the fuel cell at time t. Let be the output power of the lithium battery pack at time t. Let t be the power required for flight.

[0103] Optionally, the step of constructing a reward function based on the proton exchange membrane fuel power loss model and the lithium battery pack power loss model, using the proton exchange membrane fuel cell simulation data and the lithium battery pack simulation data, includes:

[0104] The proton exchange membrane fuel power loss model and the lithium battery pack power loss model were used to obtain the proton exchange membrane fuel power loss value and the lithium battery pack power loss value, respectively.

[0105] Specifically, the fuel cell output power at the initial moment is input into the proton exchange membrane fuel power loss model to obtain the proton exchange membrane fuel power loss value. The battery state of charge is input into the lithium battery pack power loss model to obtain the lithium battery pack power loss value.

[0106] The power loss value of the lithium battery pack includes the loss value of ohmic internal resistance and polarization internal resistance; the loss value of the ohmic resistance represents the Joule heat loss value generated by the current passing through the ohmic internal resistance; and the polarization loss value represents the irreversible heat loss value generated during the polarization process.

[0107] Based on the fuel output power, battery state of charge, bus voltage deviation, proton exchange membrane fuel power loss, and lithium battery pack power loss, the power change value, state of charge deviation value, bus voltage fluctuation value, and total power loss value are obtained.

[0108] A reward function is constructed based on the power change value, the state of charge deviation value, the bus voltage fluctuation value, and the total power loss value.

[0109] The reward function is as follows:

[0110]

[0111] in, This is the current reward value. The weighting coefficients, This represents the total power loss of the system. Indicates the power change value. This represents the deviation from the state of charge. This represents the bus voltage fluctuation value. This is a physical constraint penalty term.

[0112] Optionally, the step of obtaining the power change value, state of charge deviation value, bus voltage fluctuation value, and total power loss based on the battery state of charge, bus voltage deviation value, proton exchange membrane fuel power loss value, and lithium battery pack power loss value includes:

[0113] The total power loss value is obtained by adding the proton exchange membrane fuel power loss value to the lithium battery pack power loss value.

[0114] The method for obtaining the total power loss value is as follows:

[0115]

[0116] in, This represents the total power loss value. This indicates the fuel power loss value of the exchange membrane. This indicates the power loss value of the lithium battery pack.

[0117] The power change value is obtained based on the fuel cell output power at the initial moment.

[0118] The formula for obtaining the power change value is as follows:

[0119]

[0120] in, This represents the power change value. This indicates the rated power of a proton exchange membrane fuel cell. Let be the output power of the fuel cell at time t. Let t be the output power of the fuel cell at time t-1.

[0121] The squared difference between the battery's state of charge and a reference state of charge is used as the state of charge deviation value; the reference state of charge refers to the reference state of charge of the lithium battery.

[0122] The formula for obtaining the deviation value of the state of charge is as follows:

[0123]

[0124] in, This represents the deviation from the state of charge. For reference state of charge, This indicates the battery's state of charge.

[0125] The square of the bus voltage deviation is taken as the bus voltage fluctuation value.

[0126] The formula for the bus voltage fluctuation value is as follows:

[0127]

[0128] in, This represents the bus voltage fluctuation value. This is the bus voltage deviation value. This is the DC bus reference voltage;

[0129] Optionally, obtaining the flight power requirement based on the force data includes:

[0130] Based on the force data, the drag state of vertical flight is detected to obtain the vertical flight power requirement; the vertical flight power requirement represents the power required by the UAV during vertical takeoff, landing and hovering.

[0131] The method for calculating the power required for vertical flight includes:

[0132]

[0133] in, It is the aerodynamic drag in the vertical direction. air density, Vertical climb rate The reference area for the vertical projection of the fuselage. This is the vertical drag coefficient. For vertical flight power, The overall efficiency coefficient of the vertical dynamic system. is the thrust margin coefficient, m represents the total mass of the UAV, and g represents the gravitational acceleration.

[0134] Based on the force data, the drag state of horizontal flight is detected to obtain the power required for horizontal flight; the power required for horizontal flight represents the power required by the UAV during the horizontal cruise phase.

[0135] The method for calculating the power required for horizontal flight includes:

[0136]

[0137] in, It is the horizontal aerodynamic drag. For cruising speed, For wing reference area, Zero-lift drag coefficient, Here, e is the lift coefficient, e is the Oswald efficiency factor, and AR is the wing aspect ratio. The power required for horizontal flight is given by E, where E is the lift-to-drag ratio of the entire aircraft. Let m represent the total efficiency of the horizontal propulsion system, g represent the total mass of the UAV, and m represent the gravitational acceleration.

[0138] Optionally, the construction of the proton exchange membrane fuel power loss model and the lithium battery pack power loss model based on the equivalent circuit includes:

[0139] Multiple data pairs are acquired; the data pairs are the fuel cell output power and the corresponding fuel cell loss power obtained by discrete sampling at historical time points.

[0140] Based on the data pairs, a power loss model for proton exchange membrane fuel cells is constructed through quadratic polynomial fitting.

[0141] The net output power model of the proton exchange membrane fuel cell is shown below:

[0142]

[0143] in, α represents the output power of the proton exchange membrane fuel cell in the equivalent circuit, and α, β, and γ are fitting coefficients.

[0144] Using the above method, the losses can be directly obtained from the output power of the proton exchange membrane fuel cell in the effective circuit, without having to calculate the losses by considering the ohmic loss resistor, activation resistor, and concentration loss resistor.

[0145] Based on the proton exchange membrane fuel cell power loss model, a lithium battery pack power loss model is constructed.

[0146] Optionally, the step of constructing a lithium battery pack power loss model based on the proton exchange membrane fuel cell power loss model includes:

[0147] Construct a first lithium battery pack internal resistance model; the first lithium battery pack internal resistance model represents the values ​​of ohmic internal resistance and polarization internal resistance during charging and discharging;

[0148] The internal resistance model of the first lithium battery pack is shown below:

[0149]

[0150] in, This is an empirical coefficient. This indicates the value of the ohm resistance during charging. This indicates the value of the polarization resistance during charging. This indicates the value of the ohm resistance during discharge. This indicates the value of the polarization resistance during discharge. This indicates the battery's state of charge.

[0151] Based on the internal resistance model of the first lithium battery pack, discrete sampling is performed to obtain lithium battery pack data pairs; the lithium battery pack data pairs represent data pairs sampled during discharge and charging of the battery charge state and ohmic internal resistance, and the battery charge state and polarization internal resistance, respectively.

[0152] Based on the lithium battery pack data pairs, a lithium battery pack internal resistance model is constructed through cubic polynomial fitting.

[0153] Based on the internal resistance model of the lithium battery pack, the power loss model of the lithium battery pack is obtained by calculating charging and discharging respectively.

[0154] The power loss model of the lithium battery pack is shown below:

[0155]

[0156] in, This indicates the power loss value of the lithium battery pack. This represents the current of the lithium battery pack in the equivalent circuit. This represents the ohmic internal resistance of the lithium battery pack in the equivalent circuit. This represents the value of the polarization internal resistance of the lithium battery pack in the equivalent circuit.

[0157] Optionally, the obtained internal resistance model of the lithium battery pack is as follows:

[0158]

[0159] in, and (x∈{c,d}, i∈{0,1,2,3}) are the polynomial fitting coefficients for charging (c) and discharging (d) modes, respectively. Indicates the battery's state of charge. This indicates the value of the ohm resistance during charging. This indicates the value of the polarization resistance during charging. This indicates the value of the ohm resistance during discharge. This indicates the value of the polarization resistance during discharge.

[0160] Example 2:

[0161] Based on the above-described deep reinforcement learning-based UAV hybrid power management method, this invention also provides a deep reinforcement learning-based UAV hybrid power management system, which includes an equivalent circuit simulating a proton exchange membrane fuel cell, a lithium battery pack, and a supercapacitor.

[0162] The equivalent circuit includes a proton exchange membrane fuel cell equivalent circuit (PEMFC model), a lithium battery pack equivalent circuit (Battery model), and a supercapacitor equivalent circuit (SC model); the proton exchange membrane fuel cell equivalent circuit represents a simulation circuit containing a proton exchange membrane fuel cell; the lithium battery pack equivalent circuit represents a simulation circuit containing a lithium battery pack; and the supercapacitor equivalent circuit represents a simulation circuit containing a supercapacitor.

[0163] The equivalent circuit of the proton exchange membrane fuel cell is connected to the DC bus via a boost converter; the equivalent circuit of the lithium battery pack is connected to the DC bus via a buck-boost converter; and the equivalent circuit of the supercapacitor in the equivalent circuit is directly connected in parallel to the DC bus.

[0164] The equivalent circuit of the proton exchange membrane fuel cell includes a circuit model constructed from activation resistors, concentration loss resistors, and ohmic loss resistors.

[0165] The equivalent circuit of the proton exchange membrane fuel cell includes a circuit model constructed from activation resistors, concentration loss resistors, and ohmic loss resistors, which is used to obtain simulation data of the proton exchange membrane fuel cell.

[0166] The equivalent circuit of the lithium battery pack is a circuit model constructed using the Thevenin equivalent circuit, which is used to obtain simulation data of the lithium battery pack.

[0167] Among them, such as Figure 2 As shown. The equivalent circuit is also connected to the drone (Load).

[0168] Wherein, the equivalent circuit is as follows Figure 2 As shown.

[0169] Example 3:

[0170] A method for comparing the efficiency of the equivalent circuit of the present invention (DRL) and the equivalent energy minimization strategy (ECMS) under various flight conditions:

[0171] To accurately quantify energy conversion performance, this study establishes efficiency calculation models for both the charging and discharging stages.

[0172] When the battery is in a discharging state, the discharge efficiency Defined as the ratio of terminal output power to the total electrochemical power generated, its expression is:

[0173]

[0174] in, This indicates the battery's terminal output power.

[0175] Conversely, when the battery is charging, the charging efficiency is... Defined as the ratio of stored effective chemical power to total input power:

[0176]

[0177] in, This represents the amplitude of the input charging power. The characteristic curve of the battery charging and discharging efficiency is shown below. Figure 3 As shown.

[0178] To verify the superiority of the proposed deep reinforcement learning framework, a hardware-in-the-loop comparison experiment was conducted under the same operating scenario, comparing the deep reinforcement learning strategy with the equivalent energy minimization strategy. Key performance indicators such as power allocation, overall system efficiency, and fuel cell operating stress were systematically evaluated. Technically, Haar wavelet decomposition was used to decouple the PEMFC output power into different high-frequency and low-frequency components. The standard deviation of the resulting high-frequency component was used to quantify the operating load, which was defined as the PEMFC operating stress. Notably, an increase in the standard deviation of this component indicates accelerated PEMFC aging. The experimental verification covered the following operating conditions: 1) vertical takeoff; 2) flight mode transition; 3) fixed-wing cruise; 4) rotor hovering; 5) frequent maneuvers; and 6) vertical landing.

[0179] (1) Vertical takeoff

[0180] Figure 4 (a) illustrates the system power allocation under the equivalent consumption minimization strategy, while Figure 4 (b) presents the results under the deep reinforcement learning method. Figure 4 (c) shows the instantaneous values ​​of high-frequency power fluctuations in the PEMFC under each strategy, and calculates its operating stress accordingly. It can be seen that deep reinforcement learning significantly reduces the operating stress of the PEMFC from 35.10 W to 1.05 W. Meanwhile, Figure 4 (d) shows that the system achieves higher overall efficiency under the DDPG-based strategy compared to the equivalent consumption minimization strategy, with the average efficiency increasing from 41.3% to 53.3%.

[0181] (a) Power allocation based on the equivalent consumption minimization strategy; (b) Power allocation based on the gradient of the deep deterministic strategy; (c) Comparison of battery state of charge between the equivalent consumption minimization strategy and the deep reinforcement learning strategy; (d) Comparison of the overall system efficiency between the equivalent consumption minimization strategy and the deep reinforcement learning strategy.

[0182] (2) Flight mode switching

[0183] Figure 5 (a) illustrates the system power allocation under the equivalent consumption minimization strategy, while Figure 5 (b) presents the results under the deep reinforcement learning method. Figure 5 (c) shows the instantaneous values ​​of high-frequency power fluctuations in the PEMFC under each strategy, and calculates its operating stress accordingly. It can be seen that deep reinforcement learning significantly reduces the operating stress of the PEMFC from 26.30 W to 16.04 W. Meanwhile, Figure 5(d) shows that the system achieves higher overall efficiency under the deep deterministic policy gradient method compared to the equivalent consumption minimization strategy, with the average efficiency increasing from 39.8% to 61.0%.

[0184] Figure 5 Performance during flight mode transition. (a) Power allocation based on the equivalent energy minimization strategy; (b) Power allocation based on the gradient of the deep deterministic strategy; (c) Comparison of battery SOC between the equivalent energy minimization strategy and deep reinforcement learning; (d) Comparison of overall system efficiency between the equivalent energy minimization strategy and deep reinforcement learning.

[0185] (3) Fixed-wing cruise

[0186] Figure 6 (a) illustrates the system power allocation under the equivalent consumption minimization strategy, while Figure 6 (b) presents the results under the deep reinforcement learning method. Figure 6 (c) shows the instantaneous values ​​of high-frequency power fluctuations in the PEMFC under each strategy, and calculates its operating stress accordingly. It can be seen that deep reinforcement learning significantly reduces the operating stress of the PEMFC from 9.22 W to 0.01 W. Meanwhile, Figure 6 (d) shows that the system achieves higher overall efficiency under the deep deterministic policy gradient method compared to the equivalent consumption minimization strategy, with the average efficiency increasing from 38.1% to 59.7%.

[0187] Figure 6 Performance of fixed-wing aircraft during the cruise phase. (a) Power allocation based on the equivalent energy minimization strategy; (b) Power allocation based on the gradient of the deep deterministic strategy; (c) Comparison of battery SOC between the equivalent energy minimization strategy and deep reinforcement learning; (d) Comparison of overall system efficiency between the equivalent energy minimization strategy and deep reinforcement learning.

[0188] (4) Rotor hovering

[0189] Figure 7 (a) illustrates the system power allocation under the equivalent consumption minimization strategy, while Figure 7 (b) presents the results under the deep reinforcement learning method. Figure 7 (c) shows the instantaneous values ​​of high-frequency power fluctuations in the PEMFC under each strategy, and calculates its operating stress accordingly. It can be seen that deep reinforcement learning significantly reduces the operating stress of the PEMFC from 99.95 W to 36.94 W. Meanwhile, Figure 7 (d) shows that the system achieves higher overall efficiency under the deep deterministic policy gradient method compared to the equivalent consumption minimization strategy, with the average efficiency increasing from 46.4% to 64.2%.

[0190] Figure 7 Performance during rotor hovering. (a) Power allocation based on the equivalent energy minimization strategy; (b) Power allocation based on the gradient of the deep deterministic strategy; (c) Comparison of battery SOC between the equivalent energy minimization strategy and deep reinforcement learning; (d) Comparison of overall system efficiency between the equivalent energy minimization strategy and deep reinforcement learning.

[0191] (5) Frequent maneuvers

[0192] Figure 8 (a) illustrates the system power allocation under the equivalent consumption minimization strategy, while Figure 8 (b) presents the results under the deep reinforcement learning method. Figure 8 (c) shows the instantaneous values ​​of high-frequency power fluctuations in the PEMFC under each strategy, and calculates its operating stress accordingly. It can be seen that deep reinforcement learning significantly reduces the operating stress of the PEMFC from 47.86 W to 1.96 W. Meanwhile, Figure 8 (d) shows that the system achieves higher overall efficiency under the deep deterministic policy gradient method compared to the equivalent consumption minimization strategy, with the average efficiency increasing from 45.4% to 62.5%.

[0193] Figure 8 Performance during frequent maneuvers. (a) Power allocation based on the equivalent cost minimization strategy; (b) Power allocation based on the gradient of the deep deterministic strategy; (c) Comparison of battery SOC between the equivalent cost minimization strategy and deep reinforcement learning; (d) Comparison of overall system efficiency between the equivalent cost minimization strategy and deep reinforcement learning.

[0194] (6) Vertical landing

[0195] Figure 9 (a) illustrates the system power allocation under the equivalent consumption minimization strategy, while Figure 9 (b) presents the results under the deep reinforcement learning method. Figure 9 (c) shows the instantaneous values ​​of high-frequency power fluctuations in the PEMFC under each strategy, and calculates its operating stress accordingly. It can be seen that deep reinforcement learning increases the operating stress of the PEMFC from 17.59 W to 20.74 W, a relative increase of 15.19%. Meanwhile, Figure 9 (d) shows that the system achieves higher overall efficiency under the deep deterministic policy gradient method compared to the equivalent consumption minimization strategy, with the average efficiency increasing from 42.8% to 70.4%.

[0196] Figure 9Performance during the vertical landing phase. (a) Power allocation based on the equivalent consumption minimization strategy; (b) Power allocation based on the gradient of the deep deterministic strategy; (c) Comparison of battery SOC between the equivalent consumption minimization strategy and deep reinforcement learning; (d) Comparison of overall system efficiency between the equivalent consumption minimization strategy and deep reinforcement learning.

[0197] In summary, under six specified operating conditions, when the load power changes significantly, the energy management strategy based on deep reinforcement learning enables the battery to exhibit rapid charge and discharge characteristics. The power change is primarily compensated by the battery power, thereby reducing the power change rate of the PEMFC and relatively decreasing its operational stress. Simultaneously, the energy management strategy based on deep reinforcement learning effectively improves the overall system efficiency.

[0198] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0199] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0200] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the apparatus according to embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

Claims

1. A method for managing the power of a UAV composite power source based on deep reinforcement learning, characterized in that, include: Acquire battery circuit simulation data and force data; the battery circuit simulation data includes proton exchange membrane fuel cell simulation data and lithium battery pack simulation data; the force data represents the force data of the UAV under different vertical and horizontal flight conditions; Based on the force data, the required power for flight is obtained; the required power for flight is either the required power for vertical flight or the required power for horizontal flight. Construct a proton exchange membrane fuel power loss model and a lithium battery pack power loss model; By employing a deep deterministic strategy gradient algorithm, a proton exchange membrane fuel power loss model, and a lithium battery pack power loss model, multi-objective energy management is performed based on the battery circuit simulation data and flight power requirements to obtain the fuel cell output power and the lithium battery pack output power.

2. The method for managing the composite power supply of a UAV based on deep reinforcement learning according to claim 1, characterized in that, The process involves using a deep deterministic strategy gradient algorithm, a proton exchange membrane fuel power loss model, and a lithium battery pack power loss model. Based on the battery circuit simulation data and flight power requirements, multi-objective energy management is performed to obtain the fuel cell output power and the lithium battery pack output power, including: The lithium battery pack simulation data includes the battery state of charge; the proton exchange membrane fuel cell simulation data includes the fuel cell output power at the initial moment. Obtain the bus voltage deviation value and the fuel cell output power at the initial moment; the fuel cell output power at the initial moment represents the output power of the proton exchange membrane fuel cell in the equivalent circuit at the initial moment; the bus voltage deviation value represents the voltage deviation value of the bus connected to the equivalent circuit; A state space is constructed based on the flight power requirement, battery state of charge, bus voltage deviation, and fuel cell output power at the initial moment. The action space is defined as the fuel cell output power at time t; the fuel cell output power represents the output power of the proton exchange membrane fuel cell in the equivalent circuit. A reward function is constructed based on the proton exchange membrane fuel power loss model and the lithium battery pack power loss model, using the simulation data of the proton exchange membrane fuel cell and the simulation data of the lithium battery pack. Based on the state space, action space, and reward function, the fuel cell output power at time t is obtained using a deep deterministic policy gradient algorithm. The output power of the lithium battery pack at time t is obtained based on the fuel cell output power and the flight power requirement at time t.

3. The method for managing the composite power supply of a UAV based on deep reinforcement learning according to claim 2, characterized in that, The reward function is constructed based on the proton exchange membrane fuel power loss model and the lithium battery pack power loss model, using the simulation data of the proton exchange membrane fuel cell and the simulation data of the lithium battery pack. The function includes: The proton exchange membrane fuel power loss model and the lithium battery pack power loss model were used to obtain the proton exchange membrane fuel power loss value and the lithium battery pack power loss value, respectively. Based on the fuel cell output power, battery state of charge, bus voltage deviation, proton exchange membrane fuel power loss, and lithium battery pack power loss at the initial moment, the power change value, state of charge deviation value, bus voltage fluctuation value, and total power loss value are obtained. A reward function is constructed based on the power change value, the state of charge deviation value, the bus voltage fluctuation value, and the total power loss value. The reward function is as follows: in, This is the current reward value. The weighting coefficients, This represents the total power loss value. Indicates the power change value. This represents the deviation from the state of charge. This represents the bus voltage fluctuation value. This is a physical constraint penalty term.

4. The method for managing the composite power supply of a UAV based on deep reinforcement learning according to claim 3, characterized in that, The process of obtaining power change, state of charge deviation, bus voltage fluctuation, and total power loss based on the fuel cell output power, battery state of charge, bus voltage deviation, proton exchange membrane fuel power loss, and lithium battery pack power loss at the initial time includes: The total power loss value is obtained by adding the proton exchange membrane fuel power loss value to the lithium battery pack power loss value. The power change value is obtained based on the fuel cell output power at the initial moment; The formula for obtaining the power change value is as follows: in, This represents the power change value. This indicates the rated power of a proton exchange membrane fuel cell. The fuel output power at the current time point. This represents the fuel output power at the previous time point; The squared difference between the battery's state of charge and a reference state of charge is used as the state of charge deviation value; the reference state of charge refers to the reference state of charge of the lithium battery. The square of the bus voltage deviation is taken as the bus voltage fluctuation value.

5. The method for managing the composite power supply of a UAV based on deep reinforcement learning according to claim 2, characterized in that, The state space is shown below: in, Let be the state vector at time t. Let be the power required for flight at time t; SOC(t) is the state of charge of the battery at time t. Let be the fuel cell output power at time t-1. Let t be the bus voltage deviation value at time t.

6. The method for managing the composite power supply of a UAV based on deep reinforcement learning according to claim 1, characterized in that, The process of obtaining the required flight power based on the force data includes: Based on the force data, the drag state of vertical flight is detected to obtain the vertical flight power requirement; the vertical flight power requirement represents the power required by the UAV during vertical takeoff, landing and hovering phases. Based on the force data, the drag state of horizontal flight is detected to obtain the power required for horizontal flight; the power required for horizontal flight represents the power required by the UAV during the horizontal cruise phase.

7. The method for managing the composite power supply of a UAV based on deep reinforcement learning according to claim 1, characterized in that, The construction of the proton exchange membrane fuel power loss model and the lithium battery pack power loss model includes: Multiple data pairs are acquired; the data pairs are the fuel cell output power and the corresponding fuel cell loss power obtained through discrete sampling at historical time points; Based on the data pairs, a power loss model for proton exchange membrane fuel cells is constructed through quadratic polynomial fitting. Based on the proton exchange membrane fuel cell power loss model, a lithium battery pack power loss model is constructed.

8. The method for managing the composite power supply of a UAV based on deep reinforcement learning according to claim 5, characterized in that, The construction of a lithium battery pack power loss model based on the proton exchange membrane fuel cell power loss model includes: Construct a first lithium battery pack internal resistance model; the first lithium battery pack internal resistance model represents the values ​​of ohmic internal resistance and polarization internal resistance during charging and discharging; The first lithium battery pack internal resistance model includes a charging ohmic internal resistance model, a charging polarization internal resistance model, a discharging ohmic internal resistance model, and a discharging polarization internal resistance model. Based on the internal resistance model of the first lithium battery pack, discrete sampling is performed to obtain lithium battery pack data pairs; the lithium battery pack data pairs represent data pairs sampled during discharge and charging of the battery charge state and ohmic internal resistance, and the battery charge state and polarization internal resistance, respectively. Based on the lithium battery pack data pairs, a lithium battery pack internal resistance model is constructed through cubic polynomial fitting. Based on the internal resistance model of the lithium battery pack, the power loss model of the lithium battery pack is obtained by calculating the charging and discharging respectively. The power loss model of the lithium battery pack is shown below: in, This indicates the power loss value of the lithium battery pack. Represents current. A resistor representing ohmic internal resistance. This represents the value of the polarization resistance.

9. The method for managing the composite power supply of a UAV based on deep reinforcement learning according to claim 8, characterized in that, The obtained internal resistance model of the lithium battery pack is shown below: in, and (x∈{c,d}, i∈{0,1,2,3}) are the polynomial fitting coefficients for charging and discharging modes, respectively. Indicates the battery's state of charge. This indicates the value of the ohm resistance during charging. This indicates the value of the polarization resistance during charging. This indicates the value of the ohm resistance during discharge. This indicates the value of the polarization resistance during discharge.

10. A composite power management system for unmanned aerial vehicles (UAVs) based on deep reinforcement learning, characterized in that, This includes equivalent circuits simulating proton exchange membrane fuel cells, lithium battery packs, and supercapacitors: The equivalent circuit includes a proton exchange membrane fuel cell equivalent circuit, a lithium battery pack equivalent circuit, and a supercapacitor equivalent circuit; the proton exchange membrane fuel cell equivalent circuit represents a simulation circuit containing a proton exchange membrane fuel cell; the lithium battery pack equivalent circuit represents a simulation circuit containing a lithium battery pack; and the supercapacitor equivalent circuit represents a simulation circuit containing a supercapacitor. The equivalent circuit of the proton exchange membrane fuel cell is connected to the DC bus via a boost converter; the equivalent circuit of the lithium battery pack is connected to the DC bus via a bidirectional boost converter; the equivalent circuit of the supercapacitor in the equivalent circuit is directly connected in parallel to the DC bus. The equivalent circuit of the proton exchange membrane fuel cell includes a circuit model constructed from activation resistor, concentration loss resistor, and ohmic loss resistor, which is used to obtain simulation data of the proton exchange membrane fuel cell. The equivalent circuit of the lithium battery pack is a circuit model constructed using the Thevenin equivalent circuit, which is used to obtain simulated data of the lithium battery pack. The supercapacitor equivalent circuit represents a circuit model constructed from capacitors and resistors.

Images