Energy management method for fuel cell hybrid vehicle

Abstract

This invention discloses a fuel cell hybrid electric vehicle (FCHEV) energy management method. The main steps include establishing a simulation environment, training conditions, and verification conditions; building actor and critic networks and their target networks; training the energy management strategy to obtain inheritable network parameters; and loading the network parameters into the vehicle controller for online application. This method utilizes a Beta strategy to improve the standard SAC algorithm to enhance optimization performance. Through multiple simulation experiments, appropriate weight coefficients were determined, and health constraints were emphasized to reduce driving costs and extend the service life of the FCHEV. The method achieves performance very close to a dynamic programming (DP) benchmark, and simulation results across different driving cycles demonstrate good adaptability.

Description

Technical Field

[0001] The present invention relates to an energy management method for a fuel cell hybrid vehicle, and in particular to the development of an energy management strategy based on deep reinforcement learning and taking into account the health status of the energy system. Background Art

[0002] The traditional transportation industry accounts for approximately 20% of global greenhouse gas emissions and air pollution, placing a heavy burden on environmental protection and energy security. Automotive companies and research institutions are continuously striving to develop new vehicles to replace traditional internal combustion engine vehicles. Currently, there are three mainstream technologies: hybrid electric vehicles (HEVs), fuel cell electric vehicles, and pure electric vehicles.

[0003] In recent years, fuel cells have attracted increasing attention due to their advantages such as high efficiency, pollution-free, fast refueling and low noise. However, fuel cells have the disadvantages of slow dynamic response and poor stability under conditions of rapid power demand. To ensure the sustainability of output power, high-energy-density power batteries are usually equipped to serve as auxiliary energy together with the fuel cell. The power battery pack provides peak power to smooth the fluctuations in the fuel cell output power. However, hybrid energy storage makes the vehicle's power and energy flow more complex, so it is of great significance to develop efficient and reasonable energy management and optimization strategies to give full play to the performance and advantages of fuel cell hybrid electric vehicles (FCHEVs).

[0004] With the development of artificial intelligence (AI) technology, energy management strategies (EMS) based on reinforcement learning (RL) and deep reinforcement learning (DRL) algorithms have been widely studied. As an advanced DRL algorithm, the Soft Actor Critic (SAC) algorithm demonstrates better convergence and lower hyperparameter sensitivity than other algorithms. SAC is based on the maximum entropy DRL framework, in which actors maximize entropy while maximizing expected returns to enhance exploration. The Gaussian strategy used in standard SAC algorithms in existing technologies inevitably introduces estimation bias, which can slow down the training process and even lead to poor convergence. Therefore, eliminating the impact of this bias on algorithm performance has important practical applications. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: on the basis of the standard SAC algorithm, a fuel cell hybrid vehicle energy management method considering health is proposed based on an improved SAC algorithm, and the Beta strategy is used to replace the Gaussian strategy in the standard SAC algorithm to obtain better optimization performance.

[0006] The present invention adopts the following technical solutions:

[0007] A fuel cell hybrid vehicle energy management method includes the following steps:

[0008] S1. Build a simulation environment, pre-load the efficiency diagram of the quasi-steady-state electric motor model and the fuel cell output characteristic curve as prior knowledge, and construct a fuel cell hybrid electric vehicle (FCHEV) model. This model includes the FCHEV powertrain structure, the fuel cell hydrogen consumption model and life model, and the power battery electric-thermal-life coupling model. Input the constructed training operating conditions as FCHEV driving data.

[0009] S2. Create an Actor network and a Critic network based on the SAC algorithm and neural network, build a training network for the FCHEV model and the fuel cell hybrid vehicle health constraint energy management strategy, and set the state space, action space, and reward function;

[0010] The S3 and SAC agents interact with the simulation environment. Based on the established actor network, critic network, and reward function, they introduce a Beta strategy to propose an improved SAC algorithm for offline training of the fuel cell hybrid vehicle health constraint energy management strategy, thereby obtaining an inheritable parameterized neural network strategy.

[0011] S4. Load the parameterized neural network strategy obtained through offline training into the vehicle controller of the hybrid electric vehicle to realize real-time online application; the target domain FCHEV executes the trained energy management strategy.

[0012] As a further preferred solution, step S1 includes the following sub-steps:

[0013] S101. Use Python to build a simulation environment for the FCHEV model and energy management strategy, and obtain the speed and acceleration of the vehicle in the simulation scenario through an interactive interface.

[0014] S102. Inputting an efficiency diagram of a quasi-steady-state motor model and a fuel cell output characteristic curve, wherein the efficiency diagram of the quasi-steady-state motor model is used to construct a relationship between motor efficiency and wheel speed and torque, and the corresponding motor efficiency is obtained by interpolation, thereby obtaining the required power of the vehicle at any time; the fuel cell output characteristic curve is used to construct a relationship between fuel cell power and hydrogen consumption rate and fuel cell stack efficiency, thereby solving for the hydrogen consumption rate at any time;

[0015] S103 , inputting an FCHEV driving data set, which consists of highway conditions and urban road conditions, and constructing a mixed cycle including low-speed to high-speed conditions for various roads.

[0016] As a further preferred solution, step S2 includes the following sub-steps:

[0017] S201. Construct the power system structure of the FCHEV model;

[0018] S202. Constructing a fuel cell hydrogen consumption model and lifespan model for the FCHEV model:

[0019] S203, constructing a power battery electric-thermal-life coupling model of the FCHEV model;

[0020] S204, defining state space, action space and reward function;

[0021] S205: Construct a target network of the Actor network and the Critic network to train the energy management strategy of the fuel cell hybrid electric vehicle FCHEV.

[0022] Further, in step S201, based on the fuel cell hybrid electric bus,

[0023] At time step t, the longitudinal traction of the vehicle is calculated as follows:

[0024]

[0025] Here m is the total mass of the vehicle; f is the rolling resistance coefficient; θ is the road slope, A is the area in front of the vehicle, and C D is the air resistance coefficient, δ is the rotational mass coefficient, and g is the acceleration due to gravity;

[0026] Wheel speed W w and drive shaft torque T w As shown below:

[0027]

[0028] Here r w is the wheel radius;

[0029] Motor speed W m and torque T m Calculated as follows:

[0030]

[0031] Here R fd is the final drive gear ratio, η fd is the efficiency of the drive shaft;

[0032] The power required by the vehicle is calculated by interpolating the efficiency diagram of the quasi-steady-state motor as follows:

[0033]

[0034] Here η m is the motor efficiency;

[0035] P req As shown below:

[0036] P req =P DC / DC +P bat (5)

[0037] Here P DC / DC is the output power of the DC / DC converter, P bat It is the power of lithium-ion battery pack, including charging and discharging process.

[0038] Furthermore, in the step S202, the fuel cell hydrogen consumption model and life model are used to construct the fuel cell group, and the hydrogen consumption rate of the fuel cell group is Calculated as follows:

[0039]

[0040] Here L v Indicates the lower calorific value of hydrogen, equal to 120kJ / g, η fcs Indicates the efficiency of the fuel cell stack, power P fcs and hydrogen combustion rate and efficiency η fcs The relationship between is represented by the fuel cell stack output characteristic curve;

[0041] The overall performance degradation of the fuel cell system is expressed as a discrete expression for four different types of adverse driving condition load change cycles:

[0042]

[0043] Here n is the number of time steps, d ss (t), d low (t), d high (t), d cha (t) are the performance degradation caused by the start-stop condition, low power condition, high power load and load change condition at time t.

[0044] Furthermore, in step S203, an electric-thermal-lifetime coupled model is used to construct a power battery system. The model includes three sub-models: a second-order RC electric model, a two-state thermal model, and an energy throughput aging model. Specifically,

[0045] (1) In the second-order RC electrical model, two RC branches are used to simulate the polarization effect, and the governing equation is as follows:

[0046]

[0047] V t (t) = V oc (SoC)+V p1(t)+V p2 (t)+R S I(t) (11)

[0048] Where I(t) and V t (t) is the load current and terminal voltage at time step t, V p1 and V p2 is the polarization voltage across the RC branch, which is determined by the capacitor C p1 and C p2 and resistor R p1 、R P2 parameterization;

[0049] (2) In the two-state thermal model, according to the principle of conservation of thermal energy, the following equation is given:

[0050]

[0051]

[0052] Where, T s (t), T c (t)T a (t), T f (t) are the battery surface temperature, core temperature, internal average temperature and ambient temperature, all in °C; R c and R u It is the thermal resistance caused by heat conduction inside the battery and convection on the battery surface; C c and C s is the equivalent thermal capacitance of the battery cell and battery surface; the heat generation rate affected by ohmic heat, polarization heat, and irreversible entropy heat is represented by H(t), which is calculated by the following equation:

[0053] H(t)=I(t)[V p1 (t)+V p2 (t)+R s (t)I(t)]+I(t)[T a (t)+273]E n (SoC,t) (15)

[0054] Among them E n Represents the entropy change during the electrochemical reaction;

[0055] (3) The energy throughput model is used to evaluate battery degradation. Based on the fact that the battery can withstand a certain amount of cumulative charge flow before being scrapped, the battery health SOH is dynamically calculated as follows:

[0056]

[0057] Where Δt is the current duration, N(c,T a) is the equivalent number of cycles until the battery system reaches the end of its life; the capacity loss empirical model based on the Arrhenius equation, considering the effects of discharge rate C-rate (c) and internal temperature, the equation is as follows:

[0058]

[0059] where ΔC n is the percentage of capacity loss, B(c) represents the pre-exponential factor, R is the ideal gas constant equal to 8.314 J / (mol·K), z is the power law factor equal to 0.55, Ah represents the ampere-hour throughput, and E a represents the activation energy in J / mol:

[0060] E a (c)=31700-370.3·c (18)

[0061] When C n When the battery drops by 20%, it will reach the end of its life. The derivation of Ah and N is as follows:

[0062]

[0063] N(c,T a )=3600·Ah(c,T a ) / C n (20)

[0064] Finally, the SoH change is calculated based on the given current, temperature and battery dynamics using Equation (16) to understand the aging of the battery pack.

[0065] Furthermore, in step S204, the speed, acceleration and battery SoC information in the FCHEV model and the energy management strategy are integrated to define the state space as follows:

[0066] s=[SOC,SOH bat ,SOH fcs ,P bat ,P fcs ,v,a] (21)

[0067] Among them, SOC is the state of charge of the battery, SOH bat Is the health status of the power battery, SOH fcs is the health status of the fuel cell stack, P bat Power battery power, P fcs is the power of the fuel cell stack, v is the vehicle speed, and a is the vehicle acceleration;

[0068] Define the action space as the output power of the fuel cell system:

[0069] a=P fcs ∈[0,60]kW (22)

[0070] Based on reducing the hydrogen consumption of the fuel cell system, reducing the health degradation of the power battery and fuel cell system, and keeping the battery SOC within a reasonable margin, the reward function is defined as follows:

[0071]

[0072] Where ρ1, ρ2, ρ3 are the price of hydrogen, the price of fuel cell system replacement and the price of power battery pack replacement respectively. The weight coefficient ω is used to determine the relative importance of capital cost to battery SOC value. ref Represents the reference value of SOC.

[0073] As a further preferred solution, in step S3, the SAC agent interacts with the simulation environment, obtains the current environment state information, selects and executes actions according to the strategy, enters a new environment state, and obtains rewards from the environment feedback. At the same time, the state, action, and reward information are stored, and this cycle repeats.

[0074] As a further preferred solution, in step S3, an improved SAC algorithm is applied to the interaction model between the agent and the simulation environment. Specifically, a Beta strategy is introduced and Beta distribution is used to reduce the impact of deviation on algorithm performance. According to the definition of Beta distribution, the strategy is expressed as follows:

[0075]

[0076] Where α and β are the shape parameters of the Beta distribution, φ is a neural network parameter, Γ(n) = (n-1)! is the gamma function that extends the factorial to real numbers, where only α and β > 1 are considered, corresponding to the case where the Beta distribution is concave and unimodal.

[0077] As a further preferred solution, the steps for offline training the fuel cell hybrid vehicle health constraint energy management strategy based on the improved SAC algorithm in step S3 are as follows:

[0078] S301. Initialize the Actor network and Critic network of the energy management strategy EMS and its target network; define a storage space M as the experience replay pool and initialize it;

[0079] S302, from the current strategy π φ Sample and execute action a in (x|s), obtain the reward r at the current moment and the state s′ at the next moment, save the experience (s, a, r, s′) to the experience replay pool M, and update the state s←s′;

[0080] S303. Randomly sample from the experience replay pool M to obtain a small batch of samples N(s,a,r,s′);

[0081] S304. Train the Critic network by minimizing the soft Bellman residual equation:

[0082]

[0083] Among them, M is the experience replay pool, (s t ,a t ,r t ,s t+1 ) is a small batch of samples randomly drawn from it;

[0084] S305. Train the Actor network by minimizing the expected Kullback-Leibler divergence:

[0085]

[0086] At each time step, the action is determined by the current policy, which is determined by the output of the policy network. The constructed Beta distribution as shown in formula (24) is obtained, and random sampling is performed on the Beta distribution to obtain the current action a t ;

[0087] S306, automatically adjust the temperature coefficient, and the calculation target of its gradient is as follows:

[0088]

[0089] Where target entropy is the negative of the action dimension;

[0090] S307, soft update the target critic network with parameter θ′, the soft update is controlled by the step size factor τ:

[0091] θ′←(1-τ)θ′+τθ (28)

[0092] S308. Repeat steps S302 to S307 until the preset maximum number of iterations is reached, the training is completed, and then the final parameterized neural network π is output, saved and downloaded as the learning strategy.

[0093] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0094] 1. Considering the health degradation of the fuel cell system and the power battery pack, the present invention proposes a fuel cell hybrid vehicle energy management method based on an improved SAC algorithm that considers the health status of the energy system.

[0095] 2. Considering the estimation bias caused by the Gaussian distribution of the standard SAC method, the present invention adopts Beta distribution to improve the optimization performance.

[0096] 3. Determine appropriate weight coefficients through extensive simulation experiments and emphasize health constraints to reduce driving costs and extend the service life of FCHEV.

[0097] 4. The proposed strategy achieves performance very close to the dynamic programming (DP) benchmark, and simulation results in different driving cycles show good adaptability, outperforming other DRL methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 It is a fuel cell hybrid electric vehicle energy management framework based on the improved SAC method that considers the health status of the energy system;

[0099] Figure 2 This is a power battery model diagram;

[0100] Figure 3 is the motor efficiency diagram;

[0101] Figure 4 is the fuel cell system output characteristic curve;

[0102] Figure 5(a) is a diagram for training the mixed cycle (Mix-train);

[0103] Figure 5(b) is a mixed-valid graph used for verification;

[0104] Figure 6 This is a diagram of the power system structure of a fuel cell hybrid vehicle;

[0105] Figure 7(a) is a probability density function diagram of the Gaussian distribution;

[0106] Figure 7(b) is the probability density function of the Beta distribution;

[0107] Figure 8 This is a schematic diagram of the relationship between the energy management modules of a fuel cell hybrid vehicle. DETAILED DESCRIPTION

[0108] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the application are further described in detail below with reference to the accompanying drawings. The described embodiments are only part of the embodiments involved in the present invention. All non-innovative embodiments based on this embodiment by other researchers in the field are within the scope of protection of the present invention.

[0109] The present invention proposes a fuel cell hybrid vehicle energy management method, such as Figure 1 The specific steps are as follows:

[0110] Step S1: Constructing a simulation environment, pre-loading the efficiency diagram of the quasi-steady-state motor model and the fuel cell output characteristic curve as prior knowledge, and building a fuel cell hybrid electric vehicle (FCHEV) model. The model includes the FCHEV power system structure, the fuel cell hydrogen consumption model and life model, and the power battery electric-thermal-life coupling model; inputting the constructed training operating conditions as driving data for the FCHEV;

[0111] Step S2: creating an Actor network and a Critic network based on the SAC algorithm and the neural network, constructing a training network for the FCHEV model and the fuel cell hybrid vehicle health constraint energy management strategy, and setting the state space, action space, and reward function;

[0112] In step S3, the SAC agent interacts with the simulation environment and, based on the constructed actor network, critic network, and reward function, proposes an improved SAC algorithm by introducing the Beta strategy to perform offline training on the health constraint energy management strategy of the fuel cell hybrid vehicle, thereby obtaining an inheritable parameterized neural network strategy.

[0113] Step S4: Load the parameterized neural network strategy obtained through offline training into the vehicle controller of the hybrid electric vehicle to realize real-time online application; the target domain FCHEV executes the trained energy management strategy.

[0114] In a preferred embodiment of the present invention, step S1 specifically includes the following steps:

[0115] Step S101: Use Python to build a simulation environment for the FCHEV model and energy management strategy. Obtain the speed and acceleration of the vehicle in the simulation scenario through an interactive interface. Simulate the lithium-ion battery pack through an electric-thermal-aging model consisting of a second-order RC electric model, a two-state thermal model, and an energy throughput aging model. This allows the SoH value of the battery to be calculated at any time. The power battery model is as follows: Figure 2 shown.

[0116] Step S102: Inputting an efficiency diagram of a quasi-steady-state motor model and a fuel cell output characteristic curve, wherein the efficiency diagram of the quasi-steady-state motor model is used to construct a relationship between motor efficiency and wheel speed and torque, and the corresponding motor efficiency is obtained by interpolation, thereby obtaining the required power of the vehicle at any time; the fuel cell output characteristic curve is used to construct a relationship between fuel cell power and hydrogen consumption rate and fuel cell stack efficiency, thereby solving for the hydrogen consumption rate at any time;

[0117] Specifically, the efficiency diagram of the quasi-steady-state motor model and the prior knowledge of the fuel cell output characteristic curve are input, and the display function relationship is obtained through the interpolation fitting method, including two sets of function relationships: (1) the functional relationship between the motor speed, torque and efficiency; (2) the relationship between the fuel cell power, hydrogen consumption rate and fuel cell stack efficiency. And draw a graph, such as Figure 3 and Figure 4 As shown, the above functional relationship is used to solve the required power and hydrogen consumption rate of the vehicle at any time.

[0118] Step S103: Input the FCHEV driving dataset, which consists of both highway and urban road conditions. A mixed cycle (Mix-train) is constructed, as shown in Figure 5(a), encompassing both low-speed and high-speed conditions. This allows the training results of the present invention to be applied to a variety of roads. Experiments show that this cycle includes the China Light Vehicle Test Cycle - Passenger Car (CLTC-P) and the West Virginia University Interstate (WVU-INTER) cycles. The driving distance in this data set is 39.438 kilometers.

[0119] In addition, a mixed cycle (Mix-valid) including West Virginia University City (WVU-city) and Highway Fuel Economy Test (HWFET) is constructed as shown in Figure 5(b) to test the robustness of the obtained strategy. The driving distance in this set of data is 21.822 kilometers.

[0120] In a preferred embodiment of the present invention, step S2 includes the following sub-steps:

[0121] Step S201: constructing a power system structure of an FCHEV model;

[0122] Step S202: Constructing a fuel cell hydrogen consumption model and lifespan model of the FCHEV model:

[0123] Step S203: constructing a power battery electric-thermal-life coupling model of the FCHEV model;

[0124] Step S204: define the state space, action space and reward function;

[0125] Step S205 : constructing a target network of the Actor network and the Critic network to train the energy management strategy of the fuel cell hybrid electric vehicle FCHEV.

[0126] Specifically, first, in step S201, the research object of the present invention is a fuel cell hybrid electric bus, whose power system structure is as follows: Figure 6 At time step t, the longitudinal traction of the vehicle is calculated as follows:

[0127]

[0128] Here m is the total mass of the vehicle; f is the rolling resistance coefficient; θ is the road slope, A is the area in front of the vehicle, and C D is the air resistance coefficient, δ is the rotational mass coefficient, and g is the acceleration due to gravity.

[0129] Then the wheel speed W w and drive shaft torque T w It can be expressed as follows:

[0130]

[0131] Here r w is the wheel radius.

[0132] Motor speed W m and torque T m It can then be calculated as follows:

[0133]

[0134] Here R fd is the final drive gear ratio, η fd is the efficiency of the drive shaft.

[0135] The power required by the vehicle can be calculated as follows:

[0136]

[0137] Here η m is the motor efficiency, which is obtained by interpolation from the efficiency diagram of the quasi-steady-state motor.

[0138] Correspondingly, P req can be represented as follows:

[0139] P req =P DC / DC +P bat (33)

[0140] Here P DC / DC is the output power of the DC / DC converter, P bat It is the power of lithium-ion battery pack, including charging and discharging process.

[0141] Furthermore, in the step S202 of constructing the fuel cell hydrogen consumption model and life model of the FCHEV model,

[0142] The fuel cell, the primary power source for the FCHEV, converts the chemical energy of hydrogen and oxygen into electrical energy. A fuel cell hydrogen consumption model and lifespan model are used to construct the fuel cell stack. The hydrogen consumption rate m of the fuel cell stack can be calculated as follows:

[0143]

[0144] Here L v Indicates the lower calorific value of hydrogen, equal to 120kJ / g, η fcs Indicates the efficiency of the fuel cell stack. Power P fcs and hydrogen combustion rate and efficiency η fcs The relationship between is represented by the fuel cell stack output characteristic curve.

[0145] The overall performance degradation of the fuel cell system can be expressed using discrete expressions for four different types of adverse driving condition load variation cycles:

[0146]

[0147] Here n is the number of time steps, d ss (t), d low (t), d high (t), d cha (t) are the performance degradation caused by the start-stop condition, low power condition, high power load and load change condition at time t.

[0148] Next, in the step S203 of constructing the power battery electric-thermal-life coupling model of the FCHEV model,

[0149] As the second energy storage device of FCHEV, the power battery pack can provide peak power for the vehicle and smooth the output of the fuel cell system.

[0150] The electric-thermal-lifetime coupled model is used to construct the power battery system, which includes three sub-models: a second-order RC electric model, a two-state thermal model, and an energy throughput aging model.

[0151] (1) In the second-order RC electrical model, two RC branches are used to simulate the polarization effect, and the governing equation is as follows:

[0152]

[0153]

[0154] V t (t) = V oc (SoC)+V p1 (t)+V p2 (t)+RS I(t) (39)

[0155] Where I(t) and V t (t) is the load current and terminal voltage at time step t, V p1 and V p2 is the polarization voltage across the RC branch, which is determined by the capacitor C p1 and C p2 and resistor R p1 、R P2 Parameterized.

[0156] (2) In the two-state thermal model, according to the principle of conservation of thermal energy, the following equation is given:

[0157]

[0158] Where, T s (t), T c (t)T a (t), T f (t) are the battery surface temperature, core temperature, internal average temperature and ambient temperature, respectively, all in °C. c and R u It is the thermal resistance caused by heat conduction inside the battery and convection on the battery surface. c and C s is the equivalent thermal capacitance of the battery cell and battery surface. The heat generation rate, which is the combined effect of ohmic heat, polarization heat, and irreversible entropy heat, is represented by H(t) and can be calculated using the following equation:

[0159] H(t)=I(t)[V p1 (t)+V p2 (t)+R s (t)I(t)]+I(t)[T a (t)+273]E n (SoC,t) (43)

[0160] Among them E n It represents the entropy change during the electrochemical reaction.

[0161] (3) The energy throughput model is used to evaluate battery degradation, based on the fact that the battery can withstand a certain amount of cumulative charge flow before it is scrapped. The dynamic calculation of the battery health (SOH) is as follows:

[0162]

[0163] Where Δt is the current duration, N(c,T a) is the equivalent number of cycles until the battery system reaches the end of its life. The capacity loss empirical model based on the Arrhenius equation takes into account the effects of discharge rate C-rate (c) and internal temperature. The equation is as follows:

[0164]

[0165] where ΔC n is the percentage of capacity loss, B(c) represents the pre-exponential factor, R is the ideal gas constant equal to 8.314 J / (mol·K), z is the power law factor equal to 0.55, Ah represents the ampere-hour throughput, and E a represents the activation energy in J / mol:

[0166] E a (c)=31700-370.3·c (46)

[0167] When C n When the battery drops by 20%, it will reach the end of its life. The derivation of Ah and N is as follows:

[0168]

[0169] Finally, the SoH change can be calculated based on the given current, temperature and battery dynamics using Equation (16) to understand the aging of the battery pack.

[0170] Furthermore, in step S204, the state space is defined as follows by integrating the speed, acceleration, and battery SoC information in the FCHEV model and the energy management strategy:

[0171] s=[SOC,SOH bat ,SOH fcs ,P bat ,P fcs ,v,a] (49)

[0172] Among them, SOC is the state of charge of the battery, SOH bat Is the health status of the power battery, SOH fcs is the health status of the fuel cell stack, P bat Power battery power, P fcs is the power of the fuel cell stack, v is the vehicle speed, and a is the vehicle acceleration. Define the action space as the output power of the fuel cell system:

[0173] a=P fcs ∈[0,60]kW (50)

[0174] The energy management method for a fuel cell hybrid vehicle based on the improved SAC algorithm described in this invention takes into account health. The energy management strategy has three optimization objectives: 1) reducing hydrogen consumption of the fuel cell system; 2) reducing the degradation of the power battery and fuel cell system; and 3) maintaining the battery SOC within a reasonable margin. Therefore, the reward function is defined as follows:

[0175]

[0176] Where ρ1, ρ2, and ρ3 are the hydrogen price, fuel cell system replacement price, and power battery pack replacement price, respectively. This means that the first two objectives can be normalized by capital cost. The weight coefficient ω determines the relative importance of capital cost relative to the battery SOC value and should be fully explored to obtain better optimization performance. SOC ref is the reference value of SOC, which is 0.5.

[0177] Finally, in step S205, an Actor network is constructed, which is recorded as

[0178] where θ π are network parameters, the input of the Actor network is the current state s, and the output is the probability distribution of action a.

[0179] Construct a Critic network, recorded as Q(s,a|θ Q ),θ Q are network parameters. The input of the Critic network is the current state s and the action a that reparameterizes the probability distribution of the Actor network output. The output is the value function.

[0180] Establish the target network Q′(s,a|θ of the Critic network i Q′ ), the network structure and parameters of the target network are the same as those of the corresponding network, denoted by θ i Q′ are the parameters of the Critic target network.

[0181] The constructed Actor network and Critic network target network are used to train the energy management strategy of fuel cell hybrid electric vehicle (FCHEV).

[0182] In a preferred embodiment of the present invention, in step S3, the intelligent agent in the SAC framework interacts with the simulation environment, obtains the current environment state information, selects and executes actions according to the strategy, enters a new environment state, and obtains rewards from the environment feedback. At the same time, it stores information such as state, action, and reward, and repeats this cycle.

[0183] In order to make the model converge faster and achieve better training results, the SAC algorithm is improved in step S3. The improved SAC algorithm is adopted, and the Gaussian strategy in the standard SAC algorithm is replaced by the Beta strategy. The specific instructions are as follows:

[0184] The Gaussian strategy of the standard SAC algorithm is defined as follows:

[0185]

[0186] In the formula and are the mean and standard deviation of the normal distribution, which are the strategy π φ The output of (x|s). However, the action space of EMS is finite, while the Gaussian policy corresponds to an infinite support probability distribution, which introduces bias. To fully explore the policy space early in training, a larger value of σ is required, but this will lead to greater bias. Furthermore, the actions output by the Gaussian policy can only be executed by the DRL agent after truncation. The truncated actions are also used to calculate the state value function and the log probability gradient. Not only does it suffer from the same bias problem, but it also introduces another bias by subtracting the baseline function.

[0187] Considering the estimation bias caused by the Gaussian distribution of the standard SAC method, in order to eliminate the impact of the bias on the algorithm performance, a strategy with limited support probability distribution is needed. Therefore, the present invention adopts Beta distribution to improve the optimization performance.

[0188] We introduce the Beta strategy. According to the definition of Beta distribution, the strategy is expressed as follows:

[0189]

[0190] Where α and β are the shape parameters of the Beta distribution, which are the outputs of the policy neural network with parameter φ And Γ(n)=(n-1)! is the gamma function that extends the factorial to real numbers.

[0191] The most significant difference between the Beta strategy and the Gaussian strategy is that the Beta distribution has bounded intervals that describe the probability of success, where α-1 and β-1 can be thought of as the counts of successes and failures. The Beta strategy is unbiased because no probability density falls outside the bounds. We only consider cases where α, β > 1, corresponding to the case where the Beta distribution is concave and unimodal. The probability density function of the Gaussian distribution is shown in Figure 7(a), and the probability density function of the Beta distribution is shown in Figure 7(b).

[0192] In a preferred embodiment of the present invention, Figure 8As shown, the fuel cell hybrid vehicle energy management system includes a deep reinforcement learning agent and an interactive environment. In step S3, the agent interacts with the environment and, based on the constructed SAC network and reward function, performs offline training on the health constraint energy management strategy using an improved SAC algorithm to obtain an inheritable parameterized neural network strategy. Specifically, the steps include:

[0193] Step S301: Initialize the Actor network and Critic network of the energy management strategy EMS and their target network; define a storage space M as an experience replay pool and initialize it.

[0194] Step S302: From the current strategy π φ Sample and execute action a in (x|s), obtain the reward r at the current moment and the state s′ at the next moment, save the experience (s, a, r, s′) to the experience replay pool M, and update the state s←s′.

[0195] Step S303: Randomly sample from the experience replay pool M to obtain a small batch of samples N(s,a,r,s′)

[0196] Step S304: Train the Critic network by minimizing the soft Bellman residual equation:

[0197]

[0198] Among them, M is the experience replay pool, (s t ,a t ,r t ,s t+1 ) is a small batch of samples randomly drawn from it.

[0199] Step S305: Train the Actor network by minimizing the expected Kullback-Leibler divergence:

[0200]

[0201] At each time step, the action is determined by the current policy, which is determined by the output of the policy network. The constructed Beta distribution as shown in formula (24) is obtained, and random sampling is performed on the Beta distribution to obtain the current action a t .

[0202] Step S306: Automatically adjust the temperature coefficient, and the calculation target of its gradient is as follows:

[0203]

[0204] where target entropy is the negative of the action dimension.

[0205] Step S307: Soft update the target critic network with parameter θ′. The soft update is controlled by the step size factor τ:

[0206] θ′←(1-τ)θ′+τθ (57)

[0207] Step S308: Repeat steps 2 to 7 until the training is completed, and then output, save and download the final parameterized neural network π as the learning strategy.

[0208] Comparative experiments show that the deep reinforcement learning energy management strategy for the fuel cell hybrid electric vehicle (FCHEV) proposed in this invention has a performance gap of 5.12% compared with the energy management strategy based on dynamic programming in terms of driving cost, but is 4.72% better in terms of equivalent hydrogen consumption.

[0209] In addition, the invention has similar performance in the verification cycle, which shows that the energy management strategy proposed in the present invention has good adaptability.

[0210] The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A fuel cell hybrid vehicle energy management method, characterized in that: Here are the steps: S1. Build a simulation environment, pre-load the efficiency diagram of the quasi-steady-state electric motor model and the fuel cell output characteristic curve as prior knowledge, and construct a fuel cell hybrid electric vehicle (FCHEV) model. This model includes the FCHEV powertrain structure, the fuel cell hydrogen consumption model and life model, and the power battery electric-thermal-life coupling model. Input the constructed training operating conditions as FCHEV driving data. S2. Create an Actor network and a Critic network based on the SAC algorithm and neural network, build a training network for the FCHEV model and the fuel cell hybrid vehicle health constraint energy management strategy, and set the state space, action space, and reward function. This includes the following sub-steps: S201. Construct the power system structure of the FCHEV model; S202. Constructing a fuel cell hydrogen consumption model and lifespan model for the FCHEV model: S203, constructing a power battery electric-thermal-life coupling model of the FCHEV model; S204, defining state space, action space and reward function; S205, constructing a target network of the actor network and the critic network to train the energy management strategy of the fuel cell hybrid electric vehicle FCHEV; In step S201, based on a fuel cell hybrid electric bus, at time step t, the longitudinal traction of the vehicle is calculated as follows: Here m is the total mass of the vehicle, f is the rolling resistance coefficient, θ is the road slope, A is the area in front of the vehicle, and C D is the air resistance coefficient, δ is the rotational mass coefficient, and g is the acceleration due to gravity; Wheel speed W w and drive shaft torque T w As shown below: Here r w is the wheel radius; Motor speed W m and torque T m Calculated as follows: Here R fd is the final drive gear ratio, η fd is the efficiency of the drive shaft; The power required by the vehicle is calculated by interpolating the efficiency diagram of the quasi-steady-state motor as follows: Here η m is the motor efficiency; P req As shown below: P req =P DC / DC +P bat (5) Here P DC / DC is the output power of the DC / DC converter, P bat It is the power of lithium-ion battery pack, including charging and discharging process; The S3 and SAC agents interact with the simulation environment. Based on the established actor and critic networks and reward functions, they introduce a Beta strategy and propose a SAC algorithm to perform offline training on the health constraint energy management strategy of the fuel cell hybrid vehicle, obtaining an inheritable parameterized neural network strategy. S4. Load the parameterized neural network strategy obtained through offline training into the vehicle controller of the hybrid electric vehicle to realize real-time online application; the target domain FCHEV executes the trained energy management strategy.

2. A fuel cell hybrid vehicle energy management method according to claim 1, characterized in that: The step S1 includes the following sub-steps: S101. Use Python to build a simulation environment for the FCHEV model and energy management strategy, and obtain the speed and acceleration of the vehicle in the simulation scenario through an interactive interface. S102. Inputting an efficiency diagram of a quasi-steady-state motor model and a fuel cell output characteristic curve, wherein the efficiency diagram of the quasi-steady-state motor model is used to construct a relationship between motor efficiency and wheel speed and torque, and the corresponding motor efficiency is obtained by interpolation, thereby obtaining the required power of the vehicle at any time; the fuel cell output characteristic curve is used to construct a relationship between fuel cell power and hydrogen consumption rate and fuel cell stack efficiency, thereby solving for the hydrogen consumption rate at any time; S103 , inputting an FCHEV driving data set, which consists of highway conditions and urban road conditions, and constructing a mixed cycle including low-speed to high-speed conditions for various roads.

3. The fuel cell hybrid vehicle energy management method according to claim 1, characterized in that: In step S202, a fuel cell group is constructed using a fuel cell hydrogen consumption model and a lifespan model. The hydrogen consumption rate of the fuel cell group is Calculated as follows: Here L v Indicates the lower calorific value of hydrogen, equal to 120kJ / g, η fcs Indicates the efficiency of the fuel cell stack, power P fcs and hydrogen combustion rate and efficiency η fcs The relationship between is represented by the fuel cell stack output characteristic curve; The overall performance degradation of the fuel cell system is expressed as a discrete expression for four different types of adverse driving condition load change cycles: Here n is the number of time steps, d ss (t), d low (t), d high (t), d cha (t) are the performance degradation caused by the start-stop condition, low power condition, high power load and load change condition at time t.

4. The fuel cell hybrid vehicle energy management method according to claim 3, characterized in that: In step S203, the power battery system is constructed using an electric-thermal-lifetime coupled model. The model includes three sub-models: a second-order RC electric model, a two-state thermal model, and an energy throughput aging model. Specifically, (1) In the second-order RC electrical model, two RC branches are used to simulate the polarization effect, and the governing equation is as follows: V t (t)=V oc (SoC)+V p1 (t)+V p2 (t)+R S I(t) (11) Where I(t) and V t (t) is the load current and terminal voltage at time step t, V p1 and V p2 is the polarization voltage across the RC branch, which is determined by the capacitor C p1 and C p2 and resistor R p1 、R P2 parameterization; (2) In the two-state thermal model, according to the principle of conservation of thermal energy, the following equation is given: Where, T s (t), T c (t)T a (t), T f (t) are the battery surface temperature, core temperature, internal average temperature and ambient temperature, all in °C; R c and R u It is the thermal resistance caused by heat conduction inside the battery and convection on the battery surface; C c and C s is the equivalent thermal capacitance of the battery cell and battery surface; the heat generation rate affected by ohmic heat, polarization heat, and irreversible entropy heat is represented by H(t), which is calculated by the following equation: H(t)=I(t)[V p1 (t)+V p2 (t)+R s (t)I(t)]+I(t)[T a (t)+273]E n (SoC,t) (15) Among them E n Represents the entropy change during the electrochemical reaction; (3) The energy throughput model is used to evaluate battery degradation. Based on the fact that the battery can withstand a certain amount of cumulative charge flow before being scrapped, the battery health SOH is dynamically calculated as follows: Where Δt is the current duration, N(c,T a ) is the equivalent number of cycles until the battery system reaches the end of its life; the capacity loss empirical model based on the Arrhenius equation, considering the effects of discharge rate C-rate (c) and internal temperature, the equation is as follows: where ΔC n is the percentage of capacity loss, B(c) represents the pre-exponential factor, R is the ideal gas constant equal to 8.314 J / (mol·K), z is the power law factor equal to 0.55, Ah represents the ampere-hour throughput, and E a represents the activation energy in J / mol: E a (c)=31700-370.3·c (18) When C n When the battery drops by 20%, it will reach the end of its life. The derivation of Ah and N is as follows: N(c,T a )=3600·Ah(c,T a ) / C n (20) The SoH change is calculated according to the given current, temperature and battery dynamics using equation (16) to understand the aging of the battery pack.

5. The fuel cell hybrid vehicle energy management method according to claim 3, characterized in that: In step S204, the speed, acceleration and battery SoC information in the FCHEV model and energy management strategy are integrated to define the state space as follows: s=[SOC,SOH bat ,SOH fcs ,P bat ,P fcs ,v,a] (21) Among them, SOC is the state of charge of the battery, SOH bat Is the health status of the power battery, SOH fcs is the health status of the fuel cell stack, P bat Power battery power, P fcs is the power of the fuel cell stack, v is the vehicle speed, and a is the vehicle acceleration; Define the action space as the output power of the fuel cell system: a=P fcs ∈[0,60]kW (22) Based on reducing the hydrogen consumption of the fuel cell system, reducing the health degradation of the power battery and fuel cell system, and keeping the battery SOC within a reasonable margin, the reward function is defined as follows: Where ρ1, ρ2, ρ3 are the price of hydrogen, the price of fuel cell system replacement and the price of power battery pack replacement respectively. The weight coefficient ω is used to determine the relative importance of capital cost to battery SOC value. ref Represents the reference value of SOC.

6. The fuel cell hybrid vehicle energy management method according to claim 1, characterized in that: In step S3, the SAC agent interacts with the simulation environment, obtains the current environment state information, selects and executes actions according to the strategy, enters a new environment state, and obtains rewards from the environment feedback. At the same time, the state, action, and reward information are stored, and the cycle repeats.

7. The fuel cell hybrid vehicle energy management method according to claim 6, characterized in that: In step S3, the Beta strategy is introduced and the SAC algorithm is applied to the interaction model between the agent and the simulation environment. According to the definition of Beta distribution, the strategy is expressed as follows: Where α and β are the shape parameters of the Beta distribution, φ is a neural network parameter, Γ(n) = (n-1)! is the gamma function that extends the factorial to real numbers, where only α and β > 1 are considered, corresponding to the case where the Beta distribution is concave and unimodal.

8. The fuel cell hybrid vehicle energy management method according to claim 1, characterized in that: The steps for offline training of the fuel cell hybrid vehicle health constraint energy management strategy in step S3 are as follows: S301. Initialize the Actor network and Critic network of the energy management strategy EMS and its target network; define a storage space M as the experience replay pool and initialize it; S302, from the current strategy π φ Sample and execute action a in (x|s), obtain the reward r at the current moment and the state s′ at the next moment, save the experience (s, a, r, s′) to the experience replay pool M, and update the state s←s′; S303. Randomly sample from the experience replay pool M to obtain a small batch of samples N(s,a,r,s′); S304. Train the Critic network by minimizing the soft Bellman residual equation: Among them, M is the experience replay pool, (s t ,a t ,r t ,s t+1 ) is a small batch of samples randomly drawn from it; S305. Train the Actor network by minimizing the expected Kullback-Leibler divergence: At each time step, the action is determined by the current policy, which is determined by the output of the policy network. The constructed Beta distribution as shown in formula (24) is obtained, and random sampling is performed on the Beta distribution to obtain the current action a t ; S306, automatically adjust the temperature coefficient, and the calculation target of its gradient is as follows: Where target entropy is the negative of the action dimension; S307. Soft update is performed on the target critic network with parameter θ′. The soft update is controlled by the step size factor τ: θ′←(1-τ)θ′+τθ (28) S308. Repeat steps S302 to S307 until the preset maximum number of iterations is reached, the training is completed, and then the final parameterized neural network π is output, saved and downloaded as the learning strategy.

Images