A Hybrid Power Management Method for Unmanned Aerial Vehicles Based on ECMS and PPO Algorithms

Abstract

This invention discloses a hybrid power energy management method for UAVs based on ECMS and PPO algorithms, belonging to the field of quadcopter UAV power technology. The implementation method is as follows: A quadcopter dynamics model, an engine simulation model considering altitude, and a UAV hybrid power system model are constructed. UAV flight parameters are collected. An equivalent fuel consumption rate considering battery energy consumption is constructed using an ECMS strategy. A multi-objective optimization reward function is constructed, consisting of the equivalent fuel consumption rate, SOC stability, and battery temperature. The PPO algorithm iteratively solves the objective optimization problem, continuously updating parameters according to the principle of finding actions with increasing reward values ​​and parameter update formulas. The output reward value converges to the action value with the maximum, thus satisfying the objective function and achieving the optimal result for UAV hybrid power energy management. This enables continuous real-time control of engine output power and optimal energy allocation, improving the economy, range, and reliability of the hybrid system.

Description

Technical Field

[0001] This invention relates to a hybrid power energy management method for a quadrotor drone based on ECMS and PPO algorithms, belonging to the field of quadrotor drone power technology. Background Technology

[0002] The applications of drones are constantly expanding. In the military field, they are widely used for reconnaissance, attack, and logistics missions. In the civilian field, they are widely used for pesticide spraying, environmental monitoring, disaster search and rescue, and logistics transportation. This expansion of application scenarios has led to increasingly complex and challenging missions for drones, placing higher demands on their power systems. For pure electric drones, the limited energy density of the battery restricts their flight time; research shows that most current pure electric drones have a flight time of 30 minutes to 2 hours. While pure gasoline-powered drones offer significantly longer flight times compared to pure electric systems, their engines often struggle to respond in real-time to sudden changes in mission conditions, resulting in poor power response under harsh operating conditions.

[0003] Hybrid power systems combine electric and gasoline power. Compared to gasoline systems, hybrid systems effectively reduce fuel consumption; compared to electric systems, they extend flight time. Currently, hybrid systems have seen some application and development in the drone industry. The energy management strategy of hybrid systems is a key factor affecting the drone's power performance, flight range, and economic performance.

[0004] Traditional energy management strategies for hybrid drone systems aim to reduce the drone's total fuel consumption. In the Equivalent Energy Minimization (ECMS) strategy, equivalent energy consumption is the sum of the actual fuel consumption of the internal combustion engine and the equivalent electrical energy consumption. The ECMS strategy can transform a global optimization problem into a real-time optimization problem, enabling online energy management. However, the ECMS strategy can only perform single-objective optimization and cannot maintain a relatively stable state of charge (SOC) of the battery during operation, nor does it consider other issues such as battery temperature rise. Summary of the Invention

[0005] The main objective of this invention is to provide a hybrid power energy management method for unmanned aerial vehicles (UAVs) based on ECMS and PPO algorithms. This method involves constructing a quadcopter dynamics model, an engine simulation model considering altitude, and a UAV hybrid power system model. By collecting UAV flight parameters, an equivalent fuel consumption rate considering battery energy consumption is constructed using the ECMS strategy. A multi-objective optimization reward function is then built, consisting of the equivalent fuel consumption rate, SOC stability, and battery temperature. The optimal engine control curve is then solved using the near-end strategy optimization (PPO) algorithm, achieving continuous control of the hybrid system and reducing total system fuel consumption while maintaining the stability and reliability of the engine and battery.

[0006] The objective of this invention is achieved through the following technical solutions.

[0007] The present invention discloses a method for managing hybrid power energy of unmanned aerial vehicles based on ECMS and PPO algorithms, comprising the following steps:

[0008] Step 1: Establish a nonlinear dynamic model for each component of the quadrotor, and collect flight parameters of the quadrotor UAV, including rotor speed, flight altitude h, and power battery voltage U. Superimpose the fuselage model, fuselage force and torque model, rotor model, and rotor force and torque model to obtain the overall nonlinear dynamic model of the multirotor. Obtain the overall power requirement based on the design conditions, and combine the power requirement data with the engine hybrid power system model constructed in subsequent steps 2, 3, 4, and 5 for simulation calculation.

[0009] Field experiments were conducted to collect flight status parameters of a pure electric multi-rotor drone. These parameters included rotor speed, battery SOC, and flight altitude. The overall power requirement of the experimental rotorcraft was mapped as follows:

[0010]

[0011] In the formula P req Where Cp is the required power, w is the rotor power coefficient, and i is the rotor speed.

[0012] Step 2: Establish a hybrid power system model, which includes a rotary engine, an alternator, a rectifier and voltage regulator, a power battery pack, and a battery management system.

[0013] A rotary engine uses a generator to provide electrical energy to the motor or to charge the power battery.

[0014] An alternator converts the kinetic energy of a rotary engine into electrical energy;

[0015] The rectifier and voltage regulator converts the AC power from the alternator into DC power and stabilizes the voltage near the power battery voltage to charge it.

[0016] The power battery pack provides electrical energy to the motor and stores the kinetic energy of the engine.

[0017] The engine controller primarily controls the electronic throttle and adjusts the rotary engine speed.

[0018] The battery management system monitors the SOC status of the power battery in real time.

[0019] The engine drives the generator, whose output is rectified and regulated to provide a stable voltage range. This stable voltage range is then connected in parallel with the power battery pack for output. The rectifier and voltage regulator achieve complete decoupling between the engine, generator set, and power battery, ensuring that the engine's operating conditions are unaffected by the aircraft's flight conditions and battery terminal voltage, allowing it to always operate in a high-efficiency range without the need for frequent control of engine speed changes.

[0020] The formula for calculating the total power requirement of a multi-rotor unmanned aerial vehicle (UAV) system is as follows:

[0021] P req =P ICE +P Bat

[0022] Where P req P is the total power demand of the system. ICE P is the engine output power. Bat This refers to the battery's output power.

[0023] Step 3: Establish a rotary engine simulation model considering altitude. Based on the rotary engine simulation models established at different altitudes, modify the engine model's intake environment parameters, including temperature, pressure, and air density, according to international standard atmospheric standards. Establish engine models at different altitudes, using engine speed and torque as inputs and engine fuel consumption as outputs. Adjust the simulation parameters at different altitudes (h1, h2…h) according to the rotorcraft's flight altitude. n Fuel consumption b1, b2…b n By performing Lagrange linear interpolation, the engine performance parameters at various altitudes within a certain altitude range can be obtained, resulting in a lookup table model for the engine at varying altitudes, thus improving the accuracy of the model in predicting engine performance at different altitudes.

[0024] The input-output relationship of the rotary engine is as follows:

[0025] f c =f(n,T) cmd ,h)

[0026] Where f c For rotary engine fuel consumption, T cmd Let n be the torque, n be the rotational speed, and h be the altitude.

[0027] The variable altitude interpolation model for the rotary engine is as follows:

[0028]

[0029] h represents the current altitude, L b To interpolate the output fuel consumption rate, b i This indicates the fuel consumption rate at different altitudes.

[0030] The simplified sub-model of the rotary engine leakage is as follows:

[0031]

[0032] Where m is the leakage rate of the mixed working fluid through the orifice per unit time, A is the average effective cross-sectional area of ​​the leaking gap, P is the gas pressure inside the cylinder, ρ is the gas density inside the cylinder, P0 and ρ0 are the ambient air pressure and density of the leaking gas, respectively, and γ is the adiabatic index.

[0033] Step four involves using the changes in the battery's SOC and capacity, and deriving and calculating the battery temperature based on the heat generation formula and heat balance equation to establish a battery heat generation model. The main reasons for the temperature rise of the power battery include heat generated by ohmic internal resistance and heat generated by internal chemical reactions.

[0034] The heat generation formula and the heat balance equation are expressed as follows:

[0035]

[0036]

[0037] In the formula, U oc The open-circuit voltage of the circuit is represented by m. b Indicates battery quality, c b A represents the average specific heat capacity of the battery. b Q represents the heat exchange coefficient of the battery. h T represents the rate of heat generation of a battery. en T represents ambient temperature. Bat Indicates battery temperature, I Bat U represents the charging and discharging current. t denoted by terminal voltage, and h represents the heat exchange coefficient.

[0038] Step 5: Equivalently convert lithium battery energy into fuel consumption, and construct an equivalent fuel consumption function model for the UAV that considers engine fuel consumption and equivalent fuel consumption of electric energy using the Hamiltonian optimization function. The energy consumption of the power battery is represented by the solution of the power system cost function, thereby improving the accuracy of the system's fuel economy assessment.

[0039] The state of charge (SOC) change rate of the power battery is:

[0040]

[0041] Where I is the battery bus output current and Q is the battery capacity.

[0042] Ignoring battery internal resistance, the battery power is calculated as: P b =IV b I is the battery output current, V b Let be the open-circuit voltage of the battery. The rate of change of the battery's state of charge (SOC) at this point is written as:

[0043]

[0044] The Hamiltonian function is used to describe the equivalent fuel consumption rate of the entire system.

[0045]

[0046] Where λ is the conversion coefficient for converting electrical energy into fuel consumption, H f This represents the total equivalent fuel consumption rate.

[0047] Step six involves constructing a multi-objective optimization reward function, incorporating equivalent fuel consumption rate, SOC stability, and battery temperature into the optimization objectives, and approximating the optimal solution of the evaluation function using the ideal point method. This process determines the weight coefficients of each component of the multi-objective optimization reward function, replacing the traditional empirical method and achieving the best trade-off between system economy and reliability.

[0048] The reward function is expressed as:

[0049]

[0050] Where α, β, and γ are the weights of each item.

[0051] Battery SOC stability has two meanings: first, the battery SOC should not be too high to prevent overcharging from damaging the battery itself; second, the remaining charge should not be too low, which would make it impossible to guarantee the safety and reliability of flight.

[0052]

[0053] Among them, SOC high SOC low These represent the high and low thresholds of battery SOC, respectively.

[0054] The function of this term is to ensure that the battery's operating temperature does not become too high; its expression is:

[0055]

[0056] The ideal point method is used to determine the weight coefficients of each item. The specific steps are as follows:

[0057] S61 first solves for the optimal solution of each single objective optimization, and denotes it as the ideal point:

[0058] F*=(f1 * f2 * f3 * )

[0059] S62 constructs the evaluation function:

[0060]

[0061] S63 solves for the optimal solution under the above evaluation functions. Recorded as:

[0062]

[0063] The expressions for the weight coefficients of each item in S64 are as follows:

[0064]

[0065] This allows us to determine the weighting coefficients, and based on these coefficients, achieve the optimal balance between system economy and reliability.

[0066] Step 7: Iteratively solve the target optimization problem constructed in Step 6 using the PPO algorithm. Calculate the reward value corresponding to each action value based on the PPO algorithm. Continuously update the parameters of the neural network according to the principle of finding actions with increasing reward values ​​and the parameter update formula. Output the action value where the reward value converges to the maximum, which is the optimal result of UAV hybrid power energy management that satisfies the objective function. The optimal result of UAV hybrid power energy management refers to the engine power optimal control curve that maximizes the reward function of the multi-objective optimization in Step 6.

[0067] The PPO algorithm network structure includes an actor network and a critic network. The input to the actor network is the state s, including the battery SOC value and the battery temperature T. bat The input to the current state is the required power; the output is the parameters of the action probability distribution function, i.e., the parameter set m of the probability distribution function of the engine output power. The input to the critic network is the same as that of the actor, and the output is the value of that state.

[0068] s = [P cmd SOC,T Bat ]

[0069] Given state s0, the probability distribution function parameters of the output power are obtained through the actor network. Then, an engine output power a0 is obtained by sampling according to the probability distribution function. The action a0 is input into the environment to obtain a new state s1 and a reward r1. The state value v(s0) is obtained through the output of the critic network. In this way, the experience (s0, a0, r1, v(s0)) can be obtained and put into the experience pool. The significance of the experience pool is to facilitate the calculation of the cumulative discounted reward v(s) of the state on a trajectory. t ) and action advantage A(s t ,a t ).

[0070] The loss function for training the Actor network is defined as follows, which aims to maximize the action advantage:

[0071]

[0072] in, The advantage function is defined as:

[0073]

[0074] δ t =rt+γv(s) t+1 )-v(s t )

[0075] The state values ​​v(s0) and v(s1) are obtained through the output of the critic network, and γλ is the discount factor.

[0076] Where r t The definition of (θ) is:

[0077]

[0078] The update process of an Actor network involves retrieving all data from the experience pool, calculating the loss function, and updating the gradient over several steps using gradient ascent. After the update is complete, the experience pool is emptied, ready for the next new Actor network to interact with the environment and collect data.

[0079] After the Actor network is updated, data is taken from the experience pool to update the Critic function.

[0080] Discount Returns G t The calculation formula is as follows:

[0081] G t =r t+1 +γr t+2 +…+γ T-t r T+1 +γ T+1-t v(sT+1 )

[0082] Where v(S) T+1 ) represents the network's predicted value.

[0083] The update process of the Critic function involves updating the calculated discount reward G. t The Critic network predicts the current state value v(s) t We calculate the difference and use MSEloss as the loss function to train the network.

[0084] Based on the PPO algorithm above, after several iterations of training, the optimal engine power control curve that maximizes the reward function of the multi-objective optimization in step six is ​​obtained.

[0085] Step 8: The engine output power of the quadcopter drone is followed by the optimal engine output power curve obtained in Step 7. The battery output power is obtained through the relationship between the system demand power, engine output power, and power battery output power in Step 2. Based on the hybrid power energy management results optimized by the ECMS algorithm and PPO algorithm, the hybrid power energy management of the drone is carried out to improve the fuel economy, range performance, and power battery working stability of the hybrid power system.

[0086] Beneficial effects:

[0087] (1) The UAV hybrid power energy management method based on ECMS and PPO algorithm disclosed in this invention iteratively solves the target optimization problem through PPO algorithm, calculates the reward value corresponding to each action value based on PPO algorithm, and continuously updates the parameters of the neural network according to the principle of finding actions with increasing reward value and parameter update formula, outputting the action value that converges to the maximum reward value, that is, the optimal result of UAV hybrid power energy management that satisfies the objective function, and obtains the optimal control curve of engine output power, realizing continuous real-time control of engine output power and optimal energy allocation, improving the economy, range and reliability of hybrid system.

[0088] (2) The UAV hybrid power energy management method based on ECMS and PPO algorithm disclosed in this invention constructs a multi-objective optimization reward function, incorporating equivalent fuel consumption rate, SOC stability and battery temperature into the optimization objectives, so that the energy management strategy takes into account both the economy and reliability of the hybrid power system.

[0089] (2) The UAV hybrid power energy management method based on ECMS and PPO algorithm disclosed in this invention approximates the optimal solution of the evaluation function by the ideal point method, determines the weight coefficients of each item of the multi-objective optimization reward function, replaces the traditional empirical method, and achieves the best balance between economy and reliability.

[0090] (3) The UAV hybrid power energy management method based on ECMS and PPO algorithm disclosed in this invention uses Hamilton optimization function to construct UAV equivalent fuel consumption function model that considers engine fuel consumption and electric energy equivalent fuel consumption, solves the problem that the traditional single fuel consumption rate index cannot characterize the energy consumption of the power battery, and improves the accuracy of the system fuel economy assessment. Attached Figure Description

[0091] Figure 1 A schematic diagram of the process for the UAV hybrid power energy management method based on ECMS and PPO algorithms of the present invention.

[0092] Figure 2 This is a structural diagram of a hybrid power system.

[0093] Figure 3 This is a diagram illustrating the fuel consumption model of a rotary engine at varying altitudes. Detailed Implementation

[0094] The implementation method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0095] like Figure 1 As shown in this embodiment, the UAV hybrid power energy management method based on ECMS and PPO algorithm includes the following steps:

[0096] Step 1: Establish a nonlinear dynamic model for each component of the quadrotor, and collect flight parameters of the quadrotor UAV, including rotor speed, flight altitude h, and power battery voltage U. Superimpose the fuselage model, fuselage force and torque model, rotor model, and rotor force and torque model to obtain the overall nonlinear dynamic model of the multirotor. Obtain the power requirements of the entire aircraft based on the design conditions, and perform simulation calculations by combining the power requirements data with the engine hybrid power system model described later.

[0097] The overall nonlinear dynamic model of the quadcopter is constructed based on the component models of the quadcopter, specifically including: fuselage model, fuselage force and moment model; rotor model, rotor force and moment model;

[0098] The forces and moments of the above component models are superimposed to obtain the overall nonlinear dynamic model of the multirotor. The power requirements of the whole machine are obtained according to the design conditions. The power requirements data are combined with the engine hybrid power system model described later for simulation calculation.

[0099] Field experiments were conducted to collect flight status parameters of a pure electric multi-rotor drone. These parameters included rotor speed, battery SOC, and flight altitude. The overall power requirement of the experimental rotorcraft was mapped as follows:

[0100]

[0101] In the formula P req Where Cp is the required power, w is the rotor power coefficient, and i is the rotor speed.

[0102] Step 2: Establish a hybrid power system model, which includes a rotary engine, an alternator, a rectifier and voltage regulator, a power battery pack, and a battery management system.

[0103] The structure of the hybrid power system is shown in the attached figure. Figure 2 As shown, Figure 2 The arrows without annotations indicate the direction of energy transfer. The output of the engine-generator set is rectified and regulated to provide a stable voltage range, which is then connected in parallel with the power battery pack for output. The rectifier and voltage regulator achieves complete decoupling between the engine-generator set and the power battery, ensuring that the engine's operating conditions are not affected by the aircraft's flight conditions or the battery terminal voltage, allowing it to always operate in a high-efficiency range without the need for frequent control of engine speed changes.

[0104] The formula for calculating the total power requirement of a multi-rotor unmanned aerial vehicle (UAV) system is as follows:

[0105] P req =P ICE +P Bat (2)

[0106] Where P req P is the total power demand of the system. ICE P is the engine output power. Bat This refers to the battery's output power.

[0107] Step 3: Establish a rotary engine simulation model considering altitude. Based on the rotary engine simulation models established at different altitudes, modify the engine model's intake environment parameters, including temperature, pressure, and air density, according to international standard atmospheric standards. Establish engine models at different altitudes, using engine speed and torque as inputs and engine fuel consumption as outputs. Adjust the simulation model according to the rotorcraft's flight altitude at different altitudes h1, h2…h. n Fuel consumption b1, b2…b n By performing Lagrange linear interpolation, the engine performance parameters at various altitudes within a certain altitude range can be obtained, resulting in a lookup table model for the engine at varying altitudes, thus improving the accuracy of the model in predicting engine performance at different altitudes.

[0108] The simplified sub-model of the rotary engine leakage is as follows:

[0109]

[0110] Where m is the leakage rate of the mixed working fluid through the orifice per unit time, A is the average effective cross-sectional area of ​​the leaking gap, P is the gas pressure inside the cylinder, ρ is the gas density inside the cylinder, P0 and ρ0 are the ambient air pressure and density of the leaking gas, respectively, and γ is the adiabatic index.

[0111] The rotary engine simulation model was established based on different altitudes. The intake environment parameters of the engine model were modified according to the international standard atmosphere, including temperature, pressure and air density.

[0112] Engine models were established at different altitudes, with engine speed and torque as inputs and engine fuel consumption as outputs. Models were then applied to different altitudes h1, h2…h according to the rotorcraft's flight altitude. n Fuel consumption b1, b2…b n By performing Lagrange linear interpolation, the engine performance parameters at various altitudes within a certain altitude range can be obtained, thus obtaining a lookup table model for the engine at varying altitudes.

[0113] The input-output relationship of the rotary engine is as follows:

[0114] f c =f(n,T) cmd ,h) (4)

[0115] Where f c For rotary engine fuel consumption, T cmd Let n be the torque, n be the rotational speed, and h be the altitude.

[0116] The variable altitude interpolation model for the rotary engine is as follows:

[0117]

[0118] h represents the current altitude, L b To interpolate the output fuel consumption rate, b i This indicates the fuel consumption rate at different altitudes.

[0119] Step 4: By analyzing the changes in the State of Charge (SOC) and battery capacity of the power battery, and based on the heat generation formula and heat balance equation, derive and calculate the battery temperature to establish a battery heat generation model. The main reasons for the temperature rise of the power battery include heat generation from ohmic internal resistance and heat generation from internal chemical reactions. The heat generation formula and heat balance equation are expressed as follows:

[0120]

[0121]

[0122] In the formula, U oc The open-circuit voltage of the circuit is represented by m. b Indicates battery quality, cb A represents the average specific heat capacity of the battery. b Q represents the heat exchange coefficient of the battery. h T represents the rate of heat generation of a battery. en T represents ambient temperature. Bat Indicates battery temperature, I Bat U represents the charging and discharging current. t denoted by terminal voltage, and h represents the heat exchange coefficient.

[0123] Step 5: Equivalently convert lithium battery energy into fuel consumption, construct an equivalent fuel consumption function model for the UAV that considers engine fuel consumption and equivalent fuel consumption of electric energy using the Hamiltonian optimization function, characterize the energy consumption of the power battery through the solution of the power system cost function, and improve the accuracy of the fuel economy assessment of the system.

[0124] The state of charge (SOC) change rate of the power battery is:

[0125]

[0126] Where I is the battery bus output current and Q is the battery capacity.

[0127] Ignoring battery internal resistance, the battery power is calculated as: P b =IV b I is the battery output current, V b Let V be the open-circuit voltage of the battery. At this point, the rate of change of the battery's state of charge (SOC) can be written as:

[0128]

[0129] The Hamiltonian function is used to describe the equivalent fuel consumption rate of the entire system.

[0130]

[0131] Where λ is the conversion coefficient for converting electrical energy into fuel consumption, H f This represents the total equivalent fuel consumption rate.

[0132] Step 6: By constructing a multi-objective optimization reward function, the equivalent fuel consumption rate, SOC stability, and battery temperature are incorporated into the optimization objectives. The optimal solution of the evaluation function is approximated by the ideal point method, and the weight coefficients of each item in the multi-objective optimization reward function are determined. This replaces the traditional empirical method and achieves the best trade-off between system economy and reliability.

[0133] The reward function can be expressed as:

[0134]

[0135] Where α, β, and γ are the weights of each item.

[0136] Battery SOC stability has two meanings: first, the battery SOC should not be too high to prevent overcharging from damaging the battery itself; second, the remaining charge should not be too low, which would make it impossible to guarantee the safety and reliability of flight.

[0137]

[0138] Among them, SOC high SOC low These represent the high and low thresholds of battery SOC, which are set to 0.8 and 0.3 respectively in this invention.

[0139] The function of this term is to ensure that the battery's operating temperature does not become too high; its expression is:

[0140]

[0141] Among them, T high Set to 45℃.

[0142] To determine the weight coefficients for each item, the ideal point method is used, and the specific steps are as follows:

[0143] S61 first solves for the optimal solution of each single objective optimization, and denotes it as the ideal point:

[0144] F*=(f1 * f2 * f3 * (14)

[0145] S62 constructs the evaluation function:

[0146]

[0147] S63 solves for the optimal solution under the above evaluation functions. Recorded as:

[0148]

[0149] The expressions for the weight coefficients of each item in S64 are as follows:

[0150]

[0151] Therefore, the weighting coefficients are determined as α = 1, β = 0.86, and γ = 0.45.

[0152] Step 7: Iteratively solve the target optimization problem constructed in Step 6 using the PPO algorithm. Calculate the reward value corresponding to each action value based on the PPO algorithm. Continuously update the parameters of the neural network according to the principle of finding actions with increasing reward values ​​and the parameter update formula. Output the action value where the reward value converges to the maximum, which satisfies the objective function of the optimal result for UAV hybrid power energy management.

[0153] S71 network architecture:

[0154] The PPO algorithm network structure includes an actor network and a critic network. The actor network takes a state s as input, including the battery SOC value, battery temperature, and the power demand at that moment; its output is the parameters of the action probability distribution function, i.e., the parameter set m of the probability distribution function of the engine output power. The critic network takes the same input as the actor network and outputs the value of that state.

[0155] s = [P cmd SOC,T Bat (18)

[0156] The process of S72 generating experience:

[0157] Given state s0, the probability distribution function parameters of the output power are obtained through the actor network. Then, an engine output power a0 is obtained by sampling according to the probability distribution function. The action a0 is input into the environment to obtain a new state s1 and a reward r1. The state value v(s0) is obtained through the output of the critic network. In this way, the experience (s0, a0, r1, v(s0)) can be obtained and put into the experience pool. The significance of the experience pool is to facilitate the calculation of the cumulative discounted reward v(s) of the state on a trajectory. t ) and action advantage A(s t ,a t ).

[0158] S73 Actor Network:

[0159] The loss function for training the Actor network is defined as follows, which aims to maximize the action advantage:

[0160]

[0161] in, The advantage function is defined as:

[0162]

[0163] δ t =rt+γv(s) t+1 )-v(s t (20)

[0164] The state values ​​v(s0) and v(s1) are obtained through the output of the critic network, and γλ is the discount factor.

[0165] Where r t The definition of (θ) is:

[0166]

[0167] Update method: First, retrieve all data from the experience pool, calculate the loss function, use gradient ascent, and update the gradient several times. After the update is complete, empty the experience pool and wait for the next new actor network to interact with the environment and collect data.

[0168] S74 Critic Network

[0169] After the Actor network is updated, data is taken from the experience pool to update the Critic function.

[0170] Discount Returns G t The calculation formula is as follows:

[0171] G t =r t+1 +γr t+2 +…+γ T-t r T+1 +γ T+1-t v(s T+1 ) (twenty two)

[0172] Where v(S) T+1 ) represents the network's predicted value.

[0173] Update method: Calculated discount return G t The Critic network predicts the current state value v(s) t The difference is calculated, and MSEloss is used as the loss function to train the network.

[0174] Based on the above PPO algorithm process, after several iterations of training, the optimal energy allocation scheme is finally obtained.

[0175] Step 8: The engine output power of the quadcopter drone is followed by the optimal engine output power curve obtained in Step 7. The battery output power is obtained through the relationship between the system demand power, engine output power, and power battery output power in Step 1. The hybrid energy management strategy optimized by the ECMS algorithm and PPO algorithm significantly improves the fuel economy, range performance, and power battery working stability of the hybrid system.

[0176] A simulation model and energy management strategy were built using Python. To verify the method proposed in this example, hybrid systems with different energy management strategies were tested under the same operating conditions for two hours. The simulation results are shown in Table 1.

[0177] Table 1 Performance Comparison of Energy Management Methods

[0178] Final SOC Total fuel consumption (kg) Battery stable temperature (°C) State machine control strategy 0.264 1.236 51.2 Fuzzy control strategy 0.225 1.205 48.8 PPO 0.269 1.159 45.5

[0179] It should be understood that the above-described specific details are a further detailed explanation of the purpose, technical solution and beneficial effects of the invention. The above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hybrid power energy management method for unmanned aerial vehicles based on ECMS and PPO algorithms, characterized in that: Includes the following steps: Step 1: Establish a nonlinear dynamic model of the quadrotor for each component, and collect flight parameters of the quadrotor UAV, including rotor speed, flight altitude h, and power battery voltage U; superimpose the fuselage model, fuselage force and torque model, rotor model, and rotor force and torque model to obtain the overall nonlinear dynamic model of the multirotor; obtain the power requirement of the whole machine according to the design conditions; and combine the power requirement data with the engine hybrid power system model constructed in subsequent steps 2, 3, 4, and 5 for simulation calculation. Step 2: Establish a hybrid power system model, which includes a rotary engine, an alternator, a rectifier and voltage regulator, a power battery pack, and a battery management system. A rotary engine uses a generator to provide electrical energy to the motor or to charge the power battery. An alternator converts the kinetic energy of a rotary engine into electrical energy; The rectifier and voltage regulator converts the AC power from the alternator into DC power and stabilizes the voltage near the power battery voltage to charge it. The power battery pack provides electrical energy to the motor and stores the kinetic energy of the engine. The engine controller primarily controls the electronic throttle and adjusts the rotary engine speed. The battery management system monitors the SOC status of the power battery in real time. The engine drives the generator to work. The generator output is rectified and regulated to output a stable voltage range. The stable voltage range is connected in parallel with the power battery pack for output. The rectifier and voltage regulator realizes the complete decoupling of the engine-generator set and the power battery, ensuring that the engine operating condition is not affected by the flight condition of the aircraft and the battery terminal voltage, so that it always works in the high efficiency range and does not need to frequently control the engine speed changes. Step 3: Establish a rotary engine simulation model considering altitude; based on the rotary engine simulation models established at different altitudes, modify the engine model's intake environment parameters, including temperature, pressure, and air density, according to international standard atmospheric standards; establish engine models at different altitudes, using engine speed and torque as inputs and engine fuel consumption as outputs, and adjust the models at different altitudes h1, h2…h according to the rotorcraft's flight altitude. n Fuel consumption b1, b2…b n By performing Lagrange linear interpolation, we can obtain the engine performance parameters at various altitudes within a certain altitude range, obtain a lookup table model for the engine at varying altitudes, and improve the accuracy of the model in predicting engine performance at different altitudes. Step 4: By analyzing the changes in the SOC and battery capacity of the power battery, and based on the heat generation formula and heat balance equation, calculate the battery temperature and establish a battery heat generation model; the main reasons for the temperature rise of the power battery include heat generation from ohmic internal resistance and heat generation from internal chemical reactions. Step 5: Equivalently convert lithium battery energy into fuel consumption, construct an equivalent fuel consumption function model for the UAV that considers engine fuel consumption and equivalent fuel consumption of electric energy using the Hamiltonian optimization function, characterize the energy consumption of the power battery through the solution of the power system cost function, and improve the accuracy of the system fuel economy assessment. Step 6: By constructing a multi-objective optimization reward function, the equivalent fuel consumption rate, SOC stability, and battery temperature are incorporated into the optimization objectives. The optimal solution of the evaluation function is approximated by the ideal point method, and the weight coefficients of each item in the multi-objective optimization reward function are determined. This replaces the traditional empirical method and achieves the best trade-off between system economy and reliability. Step 7: Iteratively solve the target optimization problem constructed in Step 6 using the PPO algorithm. Calculate the reward value corresponding to each action value based on the PPO algorithm. Continuously update the parameters of the neural network according to the principle of finding actions with increasing reward values ​​and the parameter update formula. Output the action value where the reward value converges to the maximum, which is the optimal result of UAV hybrid power energy management that satisfies the objective function. The optimal result of UAV hybrid power energy management refers to the engine power optimal control curve that maximizes the reward function of the multi-objective optimization in Step 6. Step 8: The engine output power of the quadcopter drone is followed by the optimal engine output power curve obtained in Step 7. The battery output power is obtained through the relationship between the system demand power, engine output power, and power battery output power in Step 2. Based on the hybrid power energy management results optimized by the ECMS algorithm and PPO algorithm, the hybrid power energy management of the drone is carried out to improve the fuel economy, range performance, and power battery working stability of the hybrid power system.

2. The UAV hybrid power energy management method based on ECMS and PPO algorithm as described in claim 1, characterized in that: The implementation method for step one is as follows: Field experiments were conducted to collect flight status parameters of a pure electric multi-rotor drone. These parameters included rotor speed, battery SOC, and flight altitude. The overall power requirement of the experimental rotorcraft was mapped as follows: In the formula P req Where Cp is the required power, w is the rotor power coefficient, and i is the rotor speed.

3. The UAV hybrid power energy management method based on ECMS and PPO algorithm as described in claim 2, characterized in that: In step two, The formula for calculating the total power requirement of a multi-rotor unmanned aerial vehicle (UAV) system is as follows: P req =P ICE +P Bat Where P req P is the total power demand of the system. ICE P is the engine output power. Bat This refers to the battery's output power.

4. The UAV hybrid power energy management method based on ECMS and PPO algorithm as described in claim 3, characterized in that: In step three, The input-output relationship of the rotary engine is as follows: f c =f(n,T cmd ,h) Where f c For rotary engine fuel consumption, T cmd Let n be the torque, n be the rotational speed, and h be the altitude. The variable altitude interpolation model for the rotary engine is as follows: h represents the current altitude, L b To interpolate the output fuel consumption rate, b i This indicates the fuel consumption rate at different altitudes. The simplified sub-model of the rotary engine leakage is as follows: Where m is the leakage rate of the mixed working fluid through the orifice per unit time, A is the average effective cross-sectional area of ​​the leaking gap, P is the gas pressure inside the cylinder, ρ is the gas density inside the cylinder, P0 and ρ0 are the ambient air pressure and density of the leaking gas, respectively, and γ is the adiabatic index.

5. The UAV hybrid power energy management method based on ECMS and PPO algorithm as described in claim 4, characterized in that: In step four, The heat generation formula and the heat balance equation are expressed as follows: In the formula, U oc The open-circuit voltage of the circuit is represented by m. b Indicates battery quality, c b A represents the average specific heat capacity of the battery. b Q represents the heat exchange coefficient of the battery. h T represents the rate of heat generation of a battery. en T represents ambient temperature. Bat Indicates battery temperature, I Bat U represents the charging and discharging current. t denoted by terminal voltage, and h represents the heat exchange coefficient.

6. The UAV hybrid power energy management method based on ECMS and PPO algorithm as described in claim 5, characterized in that: In step five, The state of charge (SOC) change rate of the power battery is: Where I is the battery bus output current and Q is the battery capacity. Ignoring battery internal resistance, the battery power is calculated as: P b =IV b I is the battery output current, V b Let be the open-circuit voltage of the battery. The rate of change of the battery's state of charge (SOC) at this point is written as: The Hamiltonian function is used to describe the equivalent fuel consumption rate of the entire system. Where λ is the conversion coefficient for converting electrical energy into fuel consumption, H f This represents the total equivalent fuel consumption rate.

7. The UAV hybrid power energy management method based on ECMS and PPO algorithm as described in claim 6, characterized in that: In step six, The reward function is expressed as: r=-(αH f +βf SOC +γf TBat ) Where α, β, and γ are the weights of each item; Battery SOC stability has two meanings: first, the battery SOC should not be too high to prevent overcharging from damaging the battery itself; second, the remaining charge should not be too low, which would make it impossible to guarantee the safety and reliability of flight. Among them, SOC high SOC low These represent the high and low thresholds of battery SOC, respectively. The function of this term is to ensure that the battery's operating temperature does not become too high; its expression is: 。 8. The UAV hybrid power energy management method based on ECMS and PPO algorithm as described in claim 7, characterized in that: Step six uses the ideal point method to determine the weight coefficients for each item. The specific steps are as follows: S61 first solves for the optimal solution of each single objective optimization, and denotes it as the ideal point: F*=(f1 * f2 * f3 * ) S62 constructs the evaluation function: S63 solves for the optimal solution under the above evaluation functions. Recorded as: The expressions for the weight coefficients of each item in S64 are as follows: This allows us to determine the weighting coefficients, and based on these coefficients, achieve the optimal balance between system economy and reliability.

9. The UAV hybrid power energy management method based on ECMS and PPO algorithm as described in claim 1 or 2, characterized in that: In step seven, The PPO algorithm network structure includes an actor network and a critic network. The input to the actor network is the state s, including the battery SOC value and the battery temperature T. bat The required power at this moment; the output is the parameters of the action probability distribution function, that is, the parameter set m of the probability distribution function of the engine output power; the input of the critic network is the same as that of the actor, and the output is the value of this state; s=[P cmd ,SOC,T Bat ] Given the state s0, the probability distribution function parameters of the output power are obtained through the actor network. Then, an engine output power a0 is obtained by sampling according to the probability distribution function. The action a0 is input into the environment to obtain a new state s1 and a reward r1. The state value v(s0) is obtained through the output of the critic network. In this way, the experience (s0, a0, r1, v(s0)) can be obtained and put into the experience pool. The significance of the experience pool is to facilitate the calculation of the cumulative discounted reward v(s) of the state on a trajectory. t ) and action advantage A(s t ,a t ); The loss function for training the Actor network is defined as follows, which aims to maximize the action advantage: in, The advantage function is defined as: d t =rt+γv(s t+1 )-v(s t ) The state values ​​v(s0) and v(s1) are obtained through the output of the critic network, and γλ is the discount factor. Where r t The definition of (θ) is: The update process of the Actor network involves retrieving all data from the experience pool, calculating the loss function, and updating the gradient using the gradient ascent method over several steps. After the update is complete, the experience pool is emptied, and the network waits for the next new Actor network to interact with the environment and collect data. After the Actor network is updated, data is taken from the experience pool to update the Critic function; Discount Returns G t The calculation formula is as follows: G t =r t+1 +γr t+2 +…+γ T-t r T+1 +γ T+1-t v(s T+1 ) Where v(S) T+1 () represents the network's predicted value; The Critic function update process involves updating the calculated discount reward G. t The Critic network predicts the current state value v(s) t The difference is calculated, and MSEloss is used as the loss function to train the network. Based on the PPO algorithm above, after several iterations of training, the optimal engine power control curve that maximizes the reward function of the multi-objective optimization in step six is ​​obtained.

Images