A Reinforcement Learning-Based Control Method for Operating Point Switching in Fuel Cell Engine Systems
By using a reinforcement learning-based control model for the operating point switching of a fuel cell engine system, and optimizing the control strategy using a critical neural network and recursive least squares method, the problem of slow response during high-efficiency power point switching of fuel cells is solved, achieving fast and smooth power point switching and improving the power response capability of hybrid electric vehicles.
Patent Information
- Application Number
- CN202310901554.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-21
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-07-21
AI Technical Summary
There is a lack of research on efficient power point switching control for fuel cells in the current technology, which results in hybrid vehicles not responding quickly and smoothly when power demand changes.
By employing a reinforcement learning-based approach, a control model for switching operating points of a fuel cell engine system is constructed. The control strategy is then optimized using a critical neural network and recursive least squares method to achieve rapid and smooth switching of the fuel cell between its high-efficiency power points.
It achieves excellent and optimal transient control and rapid response of fuel cells during high-efficiency power point switching, and the intake manifold pressure and cathode pressure can quickly reach the reference trajectory, thus improving the power response capability of hybrid vehicles.
Smart Images

Figure CN117174959B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of fuel cell control technology, and in particular to a method for switching the operating point of a fuel cell engine system based on reinforcement learning. Background Technology
[0002] A fuel cell is an electrochemical device that converts the chemical energy of gaseous fuel into electrical energy, and is widely considered a future alternative to stationary and mobile power sources. Because fuel cells directly convert fuel energy into electrical energy, their maximum efficiency is not limited by the Carnot cycle.
[0003] In fuel cell, battery, and supercapacitor (FC+B+UC) hybrid electric vehicles, the fuel cell serves as the primary power source, providing continuous power. When peak power is required for vehicle start-up, acceleration, hill climbing, etc., the battery and supercapacitor can provide this power independently or in combination, keeping the fuel cell power at its highest efficiency point.
[0004] When a vehicle requires high power output for an extended period, switching between different high-efficiency power points of the fuel cell and then using batteries and supercapacitors to provide or recover power assists in meeting the vehicle's power demands, thus achieving a rational distribution of energy. However, current research on fuel cells mainly focuses on the optimal control of the gas supply system, with very little research on the optimal transient control of high-efficiency power point switching. Summary of the Invention
[0005] To address the aforementioned issues, this invention provides a reinforcement learning-based control method for switching operating points in a fuel cell engine system. This invention controls the gas flow rate through the air compressor, enabling the fuel cell to exhibit excellent optimal transient control and rapid response when switching between two high-efficiency power points. The reinforcement learning method solves the problem of optimal transient control for switching between two power points in a hybrid electric vehicle fuel cell. According to simulation results, the intake manifold pressure and cathode pressure in the two observed states respond more rapidly during the switching between the two power points, both quickly and smoothly reaching the reference trajectory.
[0006] Specifically, this invention provides a method for controlling the operating point switching of a fuel cell engine system based on reinforcement learning, the specific steps of which are as follows:
[0007] S1. Establish an air intake system model for the fuel cell engine;
[0008] First, establish models for the air compressor system, intake manifold, and cathode of the fuel cell intake system. Then, based on these three models, obtain the fuel cell intake system model, as shown below:
[0009]
[0010] in, Φ is the control input for the intake system; ρ is the normalized compressor flow rate; a air density; d c U is the rotating diameter of the compressor motor; c This refers to the blade tip rotation speed of the compressor motor; The control input for the intake manifold pressure; R a T is the gas constant; cp V represents the temperature of the air leaving the compressor. sm The volume of the gas supply manifold; This refers to the airflow rate at the intake manifold outlet. T is the control input for cathode pressure; fc V represents the temperature of the fuel cell stack. ca R is the volume of the cathode; R is the universal gas constant; M a The molar mass of air; This is the inlet mass flow rate of the cathode; This is the outlet mass flow rate of the cathode; Here is the molar mass of oxygen. The mass flow rate of oxygen consumed by the cathode reaction;
[0011] S2. Constructing a transient control scheme for operating point switching of a fuel cell engine system based on reinforcement learning;
[0012] S21. Construct the HJB equations for the transient control model of the operating point switching of the fuel cell engine system, and calculate the optimal control strategy parameter σ. c ;
[0013] S22. Using approximate value functions, construct the Bellman equation for the critic neural network to achieve the optimal control strategy parameter σ. c Reinforcement learning;
[0014] S23. In the reinforcement learning process, the weights μ of the critic neural network are updated using the Bellman equation error and the recursive least squares method.
[0015] S3. Construct a transient control system for the operating point switching of a fuel cell engine system, and describe the optimal regulation problem of the control system as follows:
[0016]
[0017] Where Minimize is the target to be minimized; e(0) is the initial tracking error; u e Here are the parameters to be optimized in the control system; J() is the objective function of the control system; Subject to is the objective function; e(t) is the tracking error. Its derivative; xd (t) represents the given reference value; G represents the dynamic input of the control system; u e (t) is the given input; To control the dynamics within the system;
[0018] S4. Combining the fuel cell engine intake system model from step S1, build a simulation model in the MATLAB / Simulink environment. Based on steps S2 and S3, construct the transient control scheme and control system for the fuel cell engine system's operating point switching, and build the controller. Conduct a simulation experiment on the transient control of the fuel cell engine system's operating point switching to obtain the reference output quantity P in the experiment. smd ,P cad and reference load current I st The corresponding steady-state feedforward control quantities u at steady state d By repeating the experiment under balanced conditions and with reference output and reference load current, the critical neural network weights ω and Bellman error δ are obtained to determine the steady-state feedback control quantity μ of the fuel cell engine. This enables the switching control between feedforward and feedback at the operating point of the fuel cell engine system.
[0019] Preferably, the establishment of the fuel cell intake system air compressor system model in step S1 specifically includes:
[0020] S111. Using a lumped rotational parameter model with inertia, the dynamic characteristics of the compressor's rotational angular velocity are obtained, as shown below:
[0021]
[0022] Where, ω cp J is the angular velocity of the compressor. cp τ is the combined inertia of the compressor and motor. cm τ is the motor torque of the compressor. cp The torque required to drive the compressor; dt is the differential symbol;
[0023] S112. Obtain the compressor motor torque using the static motor equations, as shown below:
[0024]
[0025] Where, k t R cm and k v These are the first, second, and third constants of the motor, respectively; V cm η is the compressor motor voltage; cm The mechanical efficiency of the motor;
[0026] S113. Using thermodynamic equations, the torque required to drive the compressor is obtained as follows:
[0027]
[0028] Among them, C p T is the specific heat capacity of air. atm For air temperature; η cp P represents the compressor's efficiency. sm P is the intake manifold pressure. atm Standard atmospheric pressure; γ is the specific heat coefficient of air; W cp The air flow rate output by the air compressor;
[0029] S114. The dimensionless parameter Ψ obtained using the Jensen & Kristensen method is shown below:
[0030]
[0031] Where Ψ is the dimensionless parameter obtained by the Jensen & Kristensen method; T cp,in The temperature before entering the air compressor; P cp,out The pressure to leave the air compressor; P cp,in The pressure entering the air compressor;
[0032] The blade tip rotation speed of the compressor motor, U c As shown below:
[0033]
[0034] Where, N cr This refers to the compressor motor speed;
[0035] S115. Obtain the normalized compressor flow rate Φ, as shown below:
[0036]
[0037] Where Φ is the normalized compressor flow rate; Φ max , β and Ψ max These are the first, second, and third polynomials for Mach number M, respectively; M is the Mach number.
[0038] The first, second, and third polynomials of the Mach number M are as follows:
[0039]
[0040] Among them, a i b is the i-th regression coefficient of the first polynomial of Mach number M; ic is the i-th regression coefficient of the second polynomial of Mach number M; i Let be the i-th regression coefficient of the third polynomial of Mach number M;
[0041] The Mach number M is as follows:
[0042]
[0043] The mass flow rate W of the air cp As shown below:
[0044]
[0045] S116. Describe the temperature T of air after compression by an air compressor using thermodynamics. cp,out As shown below:
[0046]
[0047] Among them, T cp,out The temperature of the air after compression by the air compressor;
[0048] S117. Establish a model of the air compressor system for the fuel cell intake system.
[0049] Preferably, the fuel cell intake system intake manifold model in step S1 is specifically:
[0050] S121. Using the law of conservation of mass, obtain the mass of the gas inside the intake manifold, as shown below:
[0051]
[0052] Where m is the mass of gas accumulated in the intake manifold; W in W represents the intake airflow rate of the intake manifold. out This refers to the outlet flow rate of the intake manifold;
[0053] S122. The kinetic equation for intake manifold pressure obtained through the ideal gas equation is as follows:
[0054]
[0055] The outlet flow rate W of the intake manifold out As shown below:
[0056] W out =K sm,out (P sm -Pc a) ;
[0057] Among them, K sm,out P is the flow constant; caCathode pressure;
[0058] S123. Construct a model of the intake manifold of the fuel cell intake system.
[0059] Preferably, the cathode model of the fuel cell intake system in step S1 is specifically:
[0060] S131. Using the ideal gas law and the law of conservation of mass, the dynamic pressure equation of the cathode is obtained as follows:
[0061]
[0062] Among them, W ca,in The mass flow rate of air entering the cathode; W ca,out The mass flow rate of the air exiting the cathode; T represents the mass flow rate of air consumed at the cathode. fc V represents the internal temperature of the fuel cell. ca The volume of air entering the cathode;
[0063] S132, Air mass flow rate W entering the cathode ca,in As shown below:
[0064]
[0065] Where, φ atm The relative humidity of the air entering the cathode; P sat M is the saturated vapor pressure inside the cathode. v ω is the molar mass of steam; atm The angular velocity of the air compressor;
[0066] The saturated vapor pressure P inside the cathode sat The following relationship must be satisfied:
[0067] log 10 (P sat )=-1.69×10 -10 T 4 +3.85×10 -7 T 3 -3.39×10 -4 T 2 +0.143T-20.92;
[0068] Where T represents the operating period of the control system;
[0069] S133. Obtain the cathode outlet air mass flow rate W by measuring the difference between the cathode pressure and atmospheric pressure. ca,out As shown below:
[0070]
[0071] Where, k ca,out The cathode outlet orifice plate constant;
[0072] S134. Calculate oxygen consumption using the principle of electrochemical reaction, as shown below:
[0073]
[0074] Where N is the number of fuel cell layers; F a I is Faraday's constant; st This refers to the current in the fuel cell stack.
[0075] S135. Construct a cathode model for the fuel cell intake system.
[0076] Preferably, the state space of the fuel cell intake system model in step S1 is as follows:
[0077]
[0078] Where x = [p sm ,p ca ] T This refers to the state of the fuel cell intake system. Its derivative; F(x) represents the internal dynamics of the fuel cell intake system; For the control input of the intake system;
[0079] The feedforward control variable of the system is:
[0080]
[0081] in, This is the feedforward control quantity of the system; The derivative for a given reference value.
[0082] Preferably, in step S21, the HJB equations of the transient control model for the operating point switching of the fuel cell engine system are constructed, and the optimal control strategy parameter σ is calculated. c Specifically, it includes:
[0083] S211. The mathematical model of the fuel cell is represented by a continuous-time nonlinear affine system, as shown below:
[0084]
[0085] in, Let be the state of the nonlinear affine system at time t. Its derivative; For control input of nonlinear affine systems; The dynamics within a nonlinear affine system; The input-state dynamics of the nonlinear affine system; x0 is the initial value of the objective function;
[0086] S212. Define the performance index of a continuous-time nonlinear affine system as follows:
[0087]
[0088] Where J(x0,u) is the objective function of the performance index; Q(x) is a positive definite continuous function; R is a positive definite matrix; and r(x,u) is the value of the objective function at the current time.
[0089] S213. Define the control strategy as a state feedback control law;
[0090] S2131. The value function V(x0) associated with the given feedback control strategy u=μ(x) is:
[0091]
[0092] Where V(x0) is the control strategy value function;
[0093] S2132, Optimal control strategy parameter σ c The HJB equation needs to be satisfied, and the calculation relationship is shown below:
[0094]
[0095] Among them, V * The function is the minimum value function; g represents the dynamic input of the critic neural network.
[0096] S2133, Optimal Control Strategy μ * The method for obtaining it is as follows:
[0097]
[0098] Where, μ * The optimal control strategy;
[0099] S2134, V at the same time * It is the only optimal value function, and the method for determining it is as follows;
[0100]
[0101] Where min is the local minimum function;
[0102] S214. Calculate the HJB equation; use a critical neural network structure to implement policy evaluation and policy improvement, and use two neural networks to solve the HJB equation.
[0103] S2141, The control system strategy evaluation method is as follows:
[0104]
[0105] Among them, V i Let μ be the value function of the i-th iteration; t is the first moment of the control system; μ i This is the control strategy for the i-th iteration;
[0106] S2142, The method for improving the control system strategy is as follows:
[0107]
[0108] Where, μ i+1 This is the control strategy for the (i+1)th iteration; Gradient symbol; The value function of the i-th iteration when the input is x.
[0109] S2143, the minimum value function V * The method for determining (x0) is as follows:
[0110]
[0111] S215. Obtain the optimal solution of the HJB equation using a critic neural network. Given a control policy μ(t) to approximate the true value function V(x), as shown below:
[0112]
[0113] Where, σ c σ represents the optimal control strategy parameters. c (x) is the activation function of the critic neural network; ω c ω is the weight vector of the critic neural network; j L represents the weights of the critic neural network; L is the total number of weights in the critic neural network.
[0114] The critical neural network activation function σ c (χ), as shown below:
[0115]
[0116] in, Let c be an element in the activation function of the critic neural network, where c ∈ 1, 2, d ∈ 1, 2, ..., 8;
[0117] S216. Obtain the optimal control strategy parameter σ c .
[0118] Preferably, in step 22, an approximate value function is used to construct the Bellman equation of the critic neural network to achieve the optimal control strategy parameter σ. c The reinforcement learning is as follows:
[0119]
[0120] in, The state of the nonlinear affine system at time τ; To establish the control law for the fuel cell engine; Let τ be the state of the system at time t+T; τ be the second time step of the control system; and r be the value of the objective function at the current time.
[0121] Next, the Bellman error of the critic neural network is calculated as follows:
[0122]
[0123] Wherein, δ(x(t),T) is the Bellman error equation; is the approximate weight vector of the critic neural network.
[0124] Preferably, the step 23 of updating the weights of the critic neural network using the Bellman equation error and recursive least squares method specifically involves:
[0125] In the reinforcement learning process, the weights of the critic neural network are iteratively updated using the Bellman equation error, thereby iteratively updating the control policy, as shown below:
[0126]
[0127] in, This is the partial derivative of the activation function; Let be the approximate weight vector of the i-th iteration of the critic neural network;
[0128] The weights of the critic neural network are updated using recursive least squares, as shown below:
[0129]
[0130] in, Let be the derivative of the approximate weight vector of the critic neural network; P is the matrix gain of the critic neural network. Its derivative; δ is the Bellman error.
[0131] Preferably, the construction of the transient control system for switching the operating point of the fuel cell engine system in step S3 specifically includes:
[0132] S31. Construct a transient control error system for the operating point switching of a fuel cell engine system, transforming the optimal tracking problem into an optimal adjustment problem, and defining the tracking error as:
[0133] e(t) = x(t) - x d (t);
[0134] Where e(t) is the tracking error of the fuel cell engine system; x d (t) represents the given state reference value of the fuel cell engine system;
[0135] S32. Obtain the transient control error system for operating point switching of the fuel cell engine system by using the tracking error e(t) of the fuel cell engine system and the reference trajectory:
[0136]
[0137] in, F is the derivative of the tracking error of the fuel cell engine system. e For the first internal dynamics of the fuel cell engine system; f e For the second internal dynamics of the fuel cell engine system;
[0138] S33. The performance indicators of the transient control system for switching the operating point of the fuel cell engine system are defined as follows:
[0139]
[0140] Where e(0) is the initial tracking error;
[0141] S34. Obtain the transient control system for switching the operating point of the fuel cell engine system.
[0142] Preferably, the transient control simulation experiment for switching the operating point of the fuel cell engine system in step S4 specifically includes:
[0143] S41. Set the intake manifold pressure P of the fuel cell intake system. sm Cathode pressure P ca The critical neural network ω and the optimal control strategy parameters σ c The initial value;
[0144] S42, The optimal control strategy parameter σ c The optimal control strategy μ for the (i+1)th iteration is calculated by substituting the critic neural network ω into the Bellman error equation. i+1 ;
[0145] S43. The optimal control strategy μ in the (i+1)th iteration. i+1 Input scalar of the transient control system for switching operating points of the fuel cell engine system Combined, as new input to the system;
[0146] S44. Obtain the intake manifold pressure P of the fuel cell intake system through Simulink model simulation. sm and cathode pressure P ca ;
[0147] S45. Obtain the normalized compressor flow rate Φ;
[0148] S451. Obtain the optimal control strategy parameter σ c ;
[0149] e(t) = x(t) - x d (t);
[0150] S452. Calculate the normalized compressor flow rate Φ;
[0151] Φ=σ c (x(t+T))-σ c (x(t));
[0152] S46. Using the Bellman error calculation method of the critic neural network, obtain the Bellman error δ;
[0153] S47. Use the recursive least squares method to obtain the weights of the (i+1)th iteration of the critic neural network.
[0154] S48. Return to step S42 until the critic neural network weights are calculated. Once a stable state is reached, the cycle stops.
[0155] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0156] This invention provides a reinforcement learning-based control method for switching operating points in a fuel cell engine system. Compared to traditional methods, this invention controls the gas flow through the air compressor, thereby enabling the fuel cell to have good optimal transient control and fast response capability when switching between two high-efficiency power points. The reinforcement learning method solves the optimal transient control of the two power point switching of a hybrid electric vehicle fuel cell. According to simulation results, the intake manifold pressure and cathode pressure in the two observed states respond more quickly when switching between the two power points, and can quickly and smoothly reach the reference trajectory. Attached Figure Description
[0157] Figure 1 This is a flowchart of the operating point switching control method for a fuel cell engine system based on reinforcement learning according to the present invention;
[0158] Figure 2 This is a structural diagram of the gas supply system according to the present invention;
[0159] Figure 3 This is a block diagram of an air compressor according to the present invention;
[0160] Figure 4 This is an intake manifold block diagram according to the present invention;
[0161] Figure 5 This is a cathode block diagram according to the present invention;
[0162] Figure 6 This is a structural diagram of the control method according to the present invention;
[0163] Figure 7 This is a schematic diagram of the convergence parameters of the two power point switching neural network according to the present invention;
[0164] Figure 8 This is a schematic diagram of the convergence parameters of the four power point switching neural network according to the present invention;
[0165] Figures 9a-9d This is a schematic diagram of the load current switching between two power points according to the present invention;
[0166] Figures 10a-10d This is a schematic diagram of the load current switching at four power points according to the present invention;
[0167] Figure 11 According to the present invention, the intake manifold pressure P is switched between two power points. sm Trajectory diagram;
[0168] Figure 12 According to the present invention, the cathode pressure P is switched at two power points. ca Trajectory diagram;
[0169] Figure 13 This is a schematic diagram of the trajectory of the two power point switching control law u* according to the present invention;
[0170] Figure 14 This is a schematic diagram of the trajectory of the four power point switching control law u* according to the present invention;
[0171] Figure 15 According to the present invention, the intake manifold pressure P is switched at four power points. sm Trajectory diagram;
[0172] Figure 16 According to the present invention, the cathode pressure P is switched at four power points. ca Trajectory diagram. Detailed Implementation
[0173] Exemplary embodiments, features, and aspects of the present invention will now be described in detail with reference to the accompanying drawings. Although various parameter values and aspects of the embodiments are shown, the simulation process need not be performed with exactly the same parameters and aspects unless specifically indicated.
[0174] The present invention provides a reinforcement learning-based operating point switching control method for fuel cell engine systems, such as... Figure 1 The flowchart shown includes:
[0175] S1. Establish an air intake system model for the fuel cell engine;
[0176] First, a model of the fuel cell intake system air compressor system is established. The air compressor system model is divided into two parts, such as... Figure 3 This is a block diagram of an air compressor according to the present invention. The first part is a static compressor diagram, which determines the airflow through the compressor and calculates the outlet air temperature using thermodynamic equations. The second part is the compressor's rotational angular velocity ω. cp The combined inertia J of the compressor and motor cp Specifically:
[0177] S111. Using a lumped rotational parameter model with inertia, the dynamic characteristics of the compressor's rotational angular velocity are obtained, as shown below:
[0178]
[0179] Where, ω cp J is the angular velocity of the compressor. cp τ is the combined inertia of the compressor and motor. cm τ is the motor torque of the compressor. cp The torque required to drive the compressor; dt is the differential symbol.
[0180] S112. Obtain the compressor motor torque using the static motor equations, as shown below:
[0181]
[0182] Where, k t R cm and k v These are the first, second, and third constants of the motor, respectively; V cm η is the compressor motor voltage; cm This refers to the mechanical efficiency of the motor.
[0183] S113. Using thermodynamic equations, the torque required to drive the compressor is obtained as follows:
[0184]
[0185] Among them, Cp T is the specific heat capacity of air. atm For air temperature; η cp P represents the compressor's efficiency. sm P is the intake manifold pressure. atm Standard atmospheric pressure; γ is the specific heat coefficient of air; W cp This refers to the air flow rate output by the air compressor.
[0186] S114. The dimensionless parameter Ψ obtained using the Jensen & Kristensen method is shown below:
[0187]
[0188] Where Ψ is the dimensionless parameter obtained by the Jensen & Kristensen method; T cp,in The temperature before entering the air compressor; P cp,out The pressure to leave the air compressor; P cp,in The pressure entering the air compressor.
[0189] The blade tip rotation speed of the compressor motor, U c As shown below:
[0190]
[0191] Where, N cr This refers to the compressor motor speed.
[0192] S115. Obtain the normalized compressor flow rate Φ, as shown below:
[0193]
[0194] Where Φ is the normalized compressor flow rate; Φ max , β and Ψ max These are the first, second, and third polynomials of the Mach number M, respectively; M is the Mach number.
[0195] The first, second, and third polynomials for Mach number M are shown below:
[0196]
[0197] Among them, a i b is the i-th regression coefficient of the first polynomial of Mach number M; i c is the i-th regression coefficient of the second polynomial of Mach number M; i Let be the i-th regression coefficient of the third polynomial of Mach number M.
[0198] The Mach number M is shown below:
[0199]
[0200] air mass flow rate W cp As shown below:
[0201]
[0202] S116. Describe the temperature T of air after compression by an air compressor using thermodynamics. cp,out As shown below:
[0203]
[0204] Among them, T cp,out This refers to the temperature of the air after it has been compressed by the air compressor.
[0205] Then, the fuel cell intake system intake manifold model, the intake manifold only includes the manifold pressure P. sm A state variable, such as Figure 4 The diagram shown is an intake manifold block diagram. In practice, the humidifier and cooler are relatively small; for ease of control, they are integrated into the intake manifold. Specifically:
[0206] S121. Using the law of conservation of mass, obtain the mass of the gas inside the intake manifold, as shown below:
[0207]
[0208] Where m is the mass of gas accumulated in the intake manifold; W in W represents the intake airflow rate of the intake manifold. out This refers to the outlet flow rate of the intake manifold;
[0209] S122. The kinetic equation for intake manifold pressure obtained through the ideal gas equation is as follows:
[0210]
[0211] Intake manifold outlet flow rate W out As shown below:
[0212] W out =K sm,out (P sm -P ca );
[0213] Among them, K sm,out P is the flow constant; ca This is the cathode pressure.
[0214] Secondly, the cathode model of the fuel cell intake system includes only one state variable, cathode pressure. The cathode model incorporates not only electrochemical reactions but also thermodynamics and fluid dynamics, such as... Figure 5 The diagram shown is of the cathode. Specifically:
[0215] S131. Using the ideal gas law and the law of conservation of mass, the dynamic pressure equation of the cathode is obtained as follows:
[0216]
[0217] Among them, W ca,in The mass flow rate of air entering the cathode; W ca,out The mass flow rate of the air exiting the cathode; T represents the mass flow rate of air consumed at the cathode. fc V represents the internal temperature of the fuel cell. ca The volume of air entering the cathode;
[0218] S132, Air mass flow rate W entering the cathode ca,in As shown below:
[0219]
[0220] Where, φ atm The relative humidity of the air entering the cathode; P sat M is the saturated vapor pressure inside the cathode. v ω is the molar mass of steam; atm The angular velocity of the air compressor;
[0221] Cathode internal saturated vapor pressure P sat The following relationship must be satisfied:
[0222] log 10 (P sat )=-1.69×10 -10 T 4 +3.85×10 -7 T 3 -3.39×10 -4 T 2 +0.143T-20.92;
[0223] Where T represents the operating period of the control system;
[0224] S133. Obtain the cathode outlet air mass flow rate W by measuring the difference between the cathode pressure and atmospheric pressure. ca,out As shown below:
[0225]
[0226] Where, kca,out The cathode outlet orifice plate constant;
[0227] S134. Calculate oxygen consumption using the principle of electrochemical reaction, as shown below:
[0228]
[0229] Where N is the number of fuel cell layers; F a I is Faraday's constant; st This refers to the current in the fuel cell stack.
[0230] Finally, based on the above three models, the fuel cell intake system model is obtained as follows:
[0231]
[0232] in, Φ is the control input for the intake system; ρ is the normalized compressor flow rate; a air density; d c U is the rotating diameter of the compressor motor; c This refers to the blade tip rotation speed of the compressor motor; The control input for the intake manifold pressure; R a T is the gas constant; cp V represents the temperature of the air leaving the compressor. sm The volume of the gas supply manifold; This refers to the airflow rate at the intake manifold outlet. T is the control input for cathode pressure; fc V represents the temperature of the fuel cell stack. ca R is the volume of the cathode; R is the universal gas constant; M a The molar mass of air; This is the inlet mass flow rate of the cathode; This is the outlet mass flow rate of the cathode; Here is the molar mass of oxygen. The mass flow rate of oxygen consumed by the cathode reaction.
[0233] like Figure 2 According to the gas supply system structure diagram of the present invention, the fuel cell air intake system model includes an air compressor system model, an intake manifold model, and a cathode model. The specific physical parameter values and constants in the model are shown in the following tables: Table 1 is the system simulation parameters, Table 2 is the air compressor characteristic regression coefficient, and Table 3 is the air supply system simulation parameters.
[0234] Table 1 System Simulation Parameters
[0235]
[0236] Table 2 Regression coefficients of air compressor characteristics
[0237]
[0238] Table 3 Simulation parameters of the air supply system
[0239]
[0240] The state space of the fuel cell intake system model is shown below:
[0241]
[0242] Where x = [p sm ,p ca ] T This refers to the state of the fuel cell intake system. Its derivative; F(x) represents the internal dynamics of the fuel cell intake system; This is the control input for the intake system.
[0243] The feedforward control variable of the system is:
[0244]
[0245] in, This is the feedforward control quantity of the system; The derivative for a given reference value.
[0246] S2. Constructing a transient control scheme for operating point switching of a fuel cell engine system based on reinforcement learning;
[0247] S21. Construct the HJB equations for the transient control model of the operating point switching of the fuel cell engine system, and calculate the optimal control strategy parameter σ. c Specifically, it includes:
[0248] S211. The mathematical model of the fuel cell is represented by a continuous-time nonlinear affine system, as shown below:
[0249]
[0250] in, Let be the state of the nonlinear affine system at time t. Its derivative; For control input of nonlinear affine systems; The dynamics within a nonlinear affine system; The input-state dynamics of the nonlinear affine system; x0 is the initial value of the objective function;
[0251] S212. Define the performance index of a continuous-time nonlinear affine system as follows:
[0252]
[0253] Where J(x0,u) is the objective function of the performance index; Q(x) is a positive definite continuous function; R is a positive definite matrix; and r(x,u) is the value of the objective function at the current time.
[0254] S213. Define the control strategy as a state feedback control law;
[0255] S2131. The value function V(x0) associated with the given feedback control strategy u=μ(x) is:
[0256]
[0257] Where V(x0) is the control strategy value function;
[0258] S2132, Optimal control strategy parameter σ c The HJB equation needs to be satisfied, and the calculation relationship is shown below:
[0259]
[0260] Among them, V * The function is the minimum value function; g represents the dynamic input of the critic neural network.
[0261] S2133, Optimal Control Strategy μ * The method for obtaining it is as follows:
[0262]
[0263] Where, μ * The optimal control strategy;
[0264] S2134, V at the same time * It is the only optimal value function, and the method for determining it is as follows;
[0265]
[0266] Where min is the local minimum function;
[0267] S214. Calculate the HJB equation; use a critical neural network structure to implement policy evaluation and policy improvement, and use two neural networks to solve the HJB equation.
[0268] S2141, The control system strategy evaluation method is as follows:
[0269]
[0270] Among them, Vi Let μ be the value function of the i-th iteration; t is the first moment of the control system; μ i This is the control strategy for the i-th iteration;
[0271] S2142, The method for improving the control system strategy is as follows:
[0272]
[0273] Where, μ i+1 This is the control strategy for the (i+1)th iteration; Gradient symbol; The value function of the i-th iteration when the input is x.
[0274] S2143, Minimum Value Function V * The method for determining (x0) is as follows:
[0275]
[0276] S215. Obtain the optimal solution of the HJB equation using a critic neural network. Given a control policy μ(t) to approximate the true value function V(x), as shown below:
[0277]
[0278] Where, σ c σ represents the optimal control strategy parameters. c (x) is the activation function of the critic neural network; ω c ωj is the weight vector of the critic neural network; L is the total number of weights in the critic neural network.
[0279] critic neural network activation function σ c (χ), as shown below:
[0280]
[0281] in, Let c be an element in the activation function of the critic neural network, where c ∈ 1, 2, d ∈ 1, 2, ..., 8;
[0282] S216. Obtain the optimal control strategy parameter σ c .
[0283] S22. Using approximate value functions, construct the Bellman equation for the critic neural network to achieve the optimal control strategy parameter σ. c The reinforcement learning is as follows:
[0284]
[0285] in, The state of the nonlinear affine system at time τ; To establish the control law for the fuel cell engine; Let τ be the state of the system at time t+T; τ be the second time step of the control system; and r be the value of the objective function at the current time.
[0286] Next, the Bellman error of the critic neural network is calculated as follows:
[0287]
[0288] Wherein, δ(x(t),T) is the Bellman error equation; is the approximate weight vector of the critic neural network.
[0289] S23. In the reinforcement learning process, the weights μ of the critic neural network are updated using the Bellman equation error and the recursive least squares method, specifically as follows:
[0290] In the reinforcement learning process, the weights of the critic neural network are iteratively updated using the Bellman equation error, thereby iteratively updating the control policy, as shown below:
[0291]
[0292] in, This is the partial derivative of the activation function; Let be the approximate weight vector of the i-th iteration of the critic neural network;
[0293] The weights of the critic neural network are updated using recursive least squares, as shown below:
[0294]
[0295] in, Let be the derivative of the approximate weight vector of the critic neural network; P is the matrix gain of the critic neural network. Its derivative; δ is the Bellman error.
[0296] S3. Construct a transient control system for the operating point switching of the fuel cell engine system, such as... Figure 6 The diagram shown is a structural diagram of the control method according to the present invention; specifically:
[0297] S31. Construct a transient control error system for the operating point switching of a fuel cell engine system, transforming the optimal tracking problem into an optimal adjustment problem, and defining the tracking error as:
[0298] e(t) = x(t) - x d(t);
[0299] Where e(t) is the tracking error of the fuel cell engine system; x d (t) represents the given state reference value of the fuel cell engine system;
[0300] S32. Obtain the transient control error system for operating point switching of the fuel cell engine system by using the tracking error e(t) of the fuel cell engine system and the reference trajectory:
[0301]
[0302] in, F is the derivative of the tracking error of the fuel cell engine system. e For the first internal dynamics of the fuel cell engine system; f e For the second internal dynamics of the fuel cell engine system;
[0303] S33. The performance indicators of the transient control system for switching the operating point of the fuel cell engine system are defined as follows:
[0304]
[0305] Where e(0) is the initial tracking error;
[0306] S34. Obtain the transient control system for switching the operating point of the fuel cell engine system.
[0307] S35. The optimal regulation problem of the control system is described as follows:
[0308]
[0309] Where Minimize is the target to be minimized; e(0) is the initial tracking error; u e Here are the parameters to be optimized in the control system; J() is the objective function of the control system; Subject to is the objective function; e(t) is the tracking error. Its derivative; x d (t) represents the given reference value; G represents the dynamic input of the control system; u e (t) is the given input; To control the dynamics within the system.
[0310] S4. Combining the fuel cell engine intake system model from step S1, build a simulation model in the MATLAB / Simulink environment. Based on the transient control scheme for operating point switching of the fuel cell engine system constructed in step S2 and the transient control system for operating point switching of the fuel cell engine system constructed in step S3, build a controller and conduct a simulation experiment on the transient control of operating point switching of the fuel cell engine system. The matrix gain P of the recursive least squares algorithm is initialized to P0 = 10. 6 I 24×24 The initial weights of the Critic-Actor neural network are 0. 24×1 The initial value of the system state is x(0) = {2.8759e5, 2.7091e5}. T .
[0311] This invention performs Simulink simulations by switching between two power points and four power points in pairs.
[0312] When switching between two power points, which are 41.58 kW and 46.26 kW respectively, the weight of the performance index is... With R=10 and a period of 10s, the load current changes from 200A to 260A. Figure 7 The diagram shows the current of two power switching loads.
[0313] When switching between the four power points sequentially, these four power points are 41.58KW, 44.89KW, 46.26KW, and 45.96KW, respectively. The weights of the performance indicators are... R = 10.
[0314] With a period of 10 seconds, the load current changes from 200A to 230A in the first 500 seconds, from 230A to 260A in the 500-700 seconds, and from 260A to 280A in the 700-900 seconds. Figure 8 The diagram shows the load current switching at four power points.
[0315] Specifically:
[0316] S41. Set the intake manifold pressure P of the fuel cell intake system. sm Cathode pressure P ca The critical neural network ω and the optimal control strategy parameters σ c The initial value.
[0317] S42, The optimal control strategy parameter σ c The optimal control strategy μ for the (i+1)th iteration is calculated by substituting the critic neural network ω into the Bellman error equation. i+1 .
[0318] S43. The optimal control strategy μ in the (i+1)th iteration. i+1 Input scalar of the transient control system for switching operating points of the fuel cell engine system Combined, it serves as a new input to the system.
[0319] S44. Obtain the intake manifold pressure P of the fuel cell intake system through Simulink model simulation. sm and cathode pressure P ca .
[0320] S45, Obtain the normalized compressor flow rate Φ.
[0321] S451. Obtain the optimal control strategy parameter σ c ;
[0322] e(t) = x(t) - x d (t);
[0323] S452. Calculate the normalized compressor flow rate Φ;
[0324] Φ=σ c (x(t+T))-σ c (x(t));
[0325] S46. Using the Bellman error calculation method of the critic neural network, obtain the Bellman error δ.
[0326] S47. Use the recursive least squares method to obtain the weights of the (i+1)th iteration of the critic neural network.
[0327] S48. Return to step S42 until the critic neural network weights are calculated. Once a stable state is reached, the cycle stops.
[0328] Obtain the reference output quantity P in the experiment smd ,P cad and reference load current I st By repeating the experiment with the corresponding steady-state control variables and equilibrium states at steady state, and under the conditions of reference output and reference load current, the weights ω of the critic neural network and the Bellman error δ are obtained respectively, thus obtaining the control law μ of the fuel cell engine under steady-state conditions.
[0329] Figure 9 shows a schematic diagram of two power point switching load current according to the present invention. Figure 9(a) is a schematic diagram of the first group of two power point switching load currents, Figure 9(b) is a schematic diagram of the second group of two power point switching load currents, Figure 9(c) is a schematic diagram of the third group of two power point switching load currents, and Figure 9(d) is a schematic diagram of the fourth group of two power point switching load currents. Figure 10 shows a schematic diagram of four power point switching load currents according to the present invention. Figure 10(a) is a schematic diagram of the first group of four power point switching load currents, Figure 10(b) is a schematic diagram of the second group of four power point switching load currents, Figure 10(c) is a schematic diagram of the third group of four power point switching load currents, and Figure 10(d) is a schematic diagram of the fourth group of four power point switching load currents. The simulation results show that the weights of the critic neural network converge at the end of the learning process and are very close in each iteration. Figure 13 The diagram shown is a schematic representation of the trajectory of the two power point switching control laws u^* according to the present invention; as follows: Figure 14 The diagram shown is a schematic representation of the trajectory of the four power point switching control law u^* according to the present invention; as follows: Figure 11 The diagram shown is a schematic of the intake manifold pressure P_sm trajectory during two power point switching according to the present invention; as follows: Figure 12 The diagram shown is a schematic representation of the cathode pressure P_ca trajectory at two power points according to the present invention; as follows: Figure 15 The diagram shown is a schematic representation of the intake manifold pressure P_sm trajectory during four power point switching according to the present invention; as follows: Figure 16 The diagram shows the trajectory of the cathode pressure P_ca at four power point switching according to the present invention. From the four diagrams above, it can be seen that under the near-optimal control strategy u... * Under the cathode pressure P sm and intake manifold pressure P ca The response speed is faster than that under the initial control strategy u. Therefore, the control method proposed in this invention can effectively track the intake manifold pressure and cathode pressure when the fuel cell switches power, achieving optimal transient control for power switching. This meets the power response requirements of the fuel cell and is a relatively good solution for practical engineering problems.
[0330] The beneficial effects of this invention are as follows: This invention provides a reinforcement learning-based control method for switching operating points in a fuel cell engine system. Compared with traditional methods, this invention controls the gas flow rate through the air compressor, thereby enabling the fuel cell to have good optimal transient control and rapid response capabilities when switching between two high-efficiency power points. The reinforcement learning method solves the problem of optimal transient control for switching between two power points in a hybrid electric vehicle fuel cell. According to simulation results, the intake manifold pressure and cathode pressure in the two observed states respond more rapidly during the switching between the two power points, and both can quickly and smoothly reach the reference trajectory. Analysis of the embodiments proves that this scheme can effectively track the intake manifold pressure and cathode pressure when the fuel cell switches power, achieving optimal transient control for power switching, meeting the power response requirements of the fuel cell, and providing a good solution for practical engineering problems.
[0331] Finally, it should be noted that the above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions or parameters described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for controlling the operating point switching of a fuel cell engine system based on reinforcement learning, characterized in that: It includes the following steps: S1. Establish an air intake system model for the fuel cell engine; First, establish models for the air compressor system, intake manifold, and cathode of the fuel cell intake system. Then, based on these three models, obtain the fuel cell intake system model, as shown below: in, Φ is the control input for the intake system; ρ is the normalized compressor flow rate; a air density; d c U is the rotating diameter of the compressor motor; c This refers to the blade tip rotation speed of the compressor motor; The control input for the intake manifold pressure; R a T is the gas constant; cp V represents the temperature of the air leaving the compressor. sm The volume of the gas supply manifold; This refers to the airflow rate at the intake manifold outlet. T is the control input for cathode pressure; fc V represents the temperature of the fuel cell stack. ca R is the volume of the cathode; R is the universal gas constant; M a The molar mass of air; This is the inlet mass flow rate of the cathode; This is the outlet mass flow rate of the cathode; Here is the molar mass of oxygen. The mass flow rate of oxygen consumed by the cathode reaction; S2. Constructing a transient control scheme for operating point switching of a fuel cell engine system based on reinforcement learning; S21. Construct the HJB equations for the transient control model of the operating point switching of the fuel cell engine system, and calculate the optimal control strategy parameter σ. c ; S22. Using approximate value functions, construct the Bellman equation for the critic neural network to achieve the optimal control strategy parameter σ. c Reinforcement learning; S23. In the reinforcement learning process, the weights μ of the critic neural network are updated using the Bellman equation error and the recursive least squares method. S3. Construct a transient control system for the operating point switching of a fuel cell engine system, and describe the optimal regulation problem of the control system as follows: Where Minimize is the target to be minimized; e(0) is the initial tracking error; u e Here are the parameters to be optimized in the control system; J() is the objective function of the control system; Subjectto is the objective function; e(t) is the tracking error. Its derivative; x d (t) represents the given reference value; G represents the dynamic input of the control system; u e (t) represents the given input; f represents the dynamics within the control system, belonging to... S4. Combining the fuel cell engine intake system model from step S1, build a simulation model in the MATLAB / Simulink environment. Based on steps S2 and S3, construct the transient control scheme and control system for the fuel cell engine system's operating point switching, and build the controller. Conduct a simulation experiment on the transient control of the fuel cell engine system's operating point switching to obtain the reference output quantity P in the experiment. smd ,P cad and reference load current I st The corresponding steady-state feedforward control quantities u at steady state d In the equilibrium state, the experiment was repeated. Under the conditions of reference output and reference load current, the weights ω of the critic neural network and the Bellman error δ were obtained respectively, so as to obtain the feedback control quantity μ of the fuel cell engine under steady-state conditions, and realize the feedforward and feedback switching control of the fuel cell engine system operating point.
2. The method for controlling the operating point switching of a fuel cell engine system based on reinforcement learning according to claim 1, characterized in that: The step S1 involves establishing a fuel cell intake system air compressor system model, specifically as follows: S111. Using a lumped rotational parameter model with inertia, the dynamic characteristics of the compressor's rotational angular velocity are obtained, as shown below: Where, ω cp J is the angular velocity of the compressor. cp τ is the combined inertia of the compressor and motor. cm τ is the motor torque of the compressor. cp The torque required to drive the compressor; dt is the differential symbol; S112. Obtain the compressor motor torque using the static motor equations, as shown below: Where, k t R cm and k v These are the first, second, and third constants of the motor, respectively; V cm η is the compressor motor voltage; cm The mechanical efficiency of the motor; S113. Using thermodynamic equations, the torque required to drive the compressor is obtained as follows: Among them, C p T is the specific heat capacity of air. atm For air temperature; η cp P represents the compressor's efficiency. sm P is the intake manifold pressure. atm Standard atmospheric pressure; γ is the specific heat coefficient of air; W cp This refers to the air flow rate output by the air compressor. S114. The dimensionless parameter Ψ obtained using the Jensen & Kristensen method is shown below: Where Ψ is the dimensionless parameter obtained by the Jensen & Kristensen method; T cp,in The temperature before entering the air compressor; P cp,out The pressure to leave the air compressor; P cp,in The pressure entering the air compressor; The blade tip rotation speed of the compressor motor, U c As shown below: Where, N cr This refers to the compressor motor speed; S115. Obtain the normalized compressor flow rate Φ, as shown below: Where Φ is the normalized compressor flow rate; Φ max , β and Ψ max These are the first, second, and third polynomials of Mach number M, respectively; M is the Mach number. The first, second, and third polynomials of the Mach number M are as follows: Among them, a i b is the i-th regression coefficient of the first polynomial of Mach number M; i c is the i-th regression coefficient of the second polynomial of Mach number M; i Let be the i-th regression coefficient of the third polynomial of Mach number M; The Mach number M is as follows: The mass flow rate W of the air cp As shown below: S116. Describe the temperature T of air after compression by an air compressor using thermodynamics. cp,out As shown below: Among them, T cp,out The temperature of the air after compression by the air compressor; S117. Establish a model of the air compressor system for the fuel cell intake system.
3. The method for controlling the operating point switching of a fuel cell engine system based on reinforcement learning according to claim 1, characterized in that: The step S1 involves establishing an intake manifold model for the fuel cell intake system, specifically as follows: S121. Using the law of conservation of mass, obtain the mass of the gas inside the intake manifold, as shown below: Where m is the mass of gas accumulated in the intake manifold; W in W represents the intake airflow rate of the intake manifold. out This refers to the outlet flow rate of the intake manifold; S122. The kinetic equation for intake manifold pressure obtained through the ideal gas equation is as follows: The outlet flow rate W of the intake manifold out As shown below: W out =K sm,out (P sm -P ca ); Among them, K sm,out P is the flow constant; ca Cathode pressure; S123. Construct a model of the intake manifold of the fuel cell intake system.
4. The method for controlling the operating point switching of a fuel cell engine system based on reinforcement learning according to claim 1, characterized in that: The establishment of the cathode model of the fuel cell intake system in step S1 is specifically as follows: S131. Using the ideal gas law and the law of conservation of mass, the dynamic pressure equation of the cathode is obtained as follows: Among them, W ca,in The mass flow rate of air entering the cathode; W ca,out The mass flow rate of the air exiting the cathode; T represents the mass flow rate of air consumed at the cathode. fc V represents the internal temperature of the fuel cell. ca The volume of air entering the cathode; S132, Air mass flow rate W entering the cathode ca,in As shown below: Where, φ atm The relative humidity of the air entering the cathode; P sat M is the saturated vapor pressure inside the cathode. v ω is the molar mass of steam; atm The angular velocity of the air compressor; The saturated vapor pressure P inside the cathode sat The following relationship must be satisfied: log 10 (P sat )=-1.69×10 -10 T 4 +3.85×10 -7 T 3 -3.39×10 -4 T 2 +0.143T-20.92; Where T represents the operating period of the control system; S133. Obtain the cathode outlet air mass flow rate W by measuring the difference between the cathode pressure and atmospheric pressure. ca,out As shown below: Where, k ca,out The cathode outlet orifice plate constant; S134. Calculate oxygen consumption using the principle of electrochemical reaction, as shown below: Where N is the number of fuel cell layers; F a I is Faraday's constant; st This refers to the current in the fuel cell stack. S135. Construct a cathode model for the fuel cell intake system.
5. The method for controlling the operating point switching of a fuel cell engine system based on reinforcement learning according to claim 1, characterized in that: The state space of the fuel cell intake system model in step S1 is shown below: Where x = [p sm p ca ] T This refers to the state of the fuel cell intake system. Its derivative; F(x) represents the internal dynamics of the fuel cell intake system; For the control input of the intake system; The feedforward control variable of the system is: in, This is the feedforward control quantity of the system; The derivative for a given reference value.
6. The method for controlling the operating point switching of a fuel cell engine system based on reinforcement learning according to claim 1, characterized in that: In step S21, the HJB equations of the transient control model for the operating point switching of the fuel cell engine system are constructed, and the optimal control strategy parameter σ is calculated. c Specifically, it includes: S211. The mathematical model of the fuel cell is represented by a continuous-time nonlinear affine system, as shown below: in, Let be the state of the nonlinear affine system at time t. Its derivative; For control input of nonlinear affine systems; The dynamics within a nonlinear affine system; The input-state dynamics of the nonlinear affine system; x0 is the initial value of the objective function; S212. Define the performance index of a continuous-time nonlinear affine system as follows: Where J(x0, u) is the objective function of the performance index; Q(x) is a positive definite continuous function; R is a positive definite matrix; and r(x, u) is the value of the objective function at the current time. S213. Define the control strategy as a state feedback control law, which includes the following sub-steps: S2131. The value function V(x0) associated with the given control strategy u=μ(x) is: Where V(x0) is the control strategy value function; S2132, Optimal control strategy parameter σ c The HJB equation needs to be satisfied, and the calculation relationship is shown below: Among them, V * The function is the minimum value function; g represents the dynamic input of the critic neural network. S2133, Optimal Control Strategy μ * The method for obtaining it is as follows: Where, μ * The optimal control strategy; S2134, V at the same time * It is the only optimal value function, and the method for determining it is as follows; Where min is the local minimum function; S214. Calculate the HJB equation; use a critical neural network structure to implement policy evaluation and policy improvement, and use two neural networks to solve the HJB equation. S2141, The control system strategy evaluation method is as follows: Among them, V i Let μ be the value function of the i-th iteration; t is the first moment of the control system; μ i This is the control strategy for the i-th iteration; S2142, The method for improving the control system strategy is as follows: Where, μ i+1 This is the control strategy for the (i+1)th iteration; Gradient symbol; The value function of the i-th iteration when the input is x. S2143, the minimum value function V * The method for determining (x0) is as follows: S215. Obtain the optimal solution of the HJB equation using a critic neural network. Given a control policy μ(t) to approximate the true value function V(x), as shown below: Where, σ c σ represents the optimal control strategy parameters. c (x) is the activation function of the critic neural network; ω c ω is the weight vector of the critic neural network; j L represents the weights of the critic neural network; L is the total number of weights in the critic neural network. The critical neural network activation function σ c (x), as shown below: in, Let c be an element in the activation function of the critic neural network, where c ∈ 1, 2, d ∈ 1, 2, ..., 8; S216. Obtain the optimal control strategy parameter σ c .
7. The method for controlling the operating point switching of a fuel cell engine system based on reinforcement learning according to claim 1, characterized in that: In step 22, the Bellman equation of the critic neural network is constructed using an approximate value function to achieve the optimal control strategy parameter σ. c The reinforcement learning is as follows: in, The state of the nonlinear affine system at time τ; To establish the control law for the fuel cell engine; Let τ be the state of the system at time t+T; τ be the second time step of the control system; and r be the value of the objective function at the current time. Next, the Bellman error of the critic neural network is calculated as follows: Wherein, δ(x(t), T) is the Bellman error equation; is the approximate weight vector of the critic neural network.
8. The method for controlling the operating point switching of a fuel cell engine system based on reinforcement learning according to claim 1, characterized in that: The step 23, which updates the weights of the critic neural network using the Bellman equation error and recursive least squares, specifically involves: In the reinforcement learning process, the weights of the critic neural network are iteratively updated using the Bellman equation error, thereby iteratively updating the control policy, as shown below: in, This is the partial derivative of the activation function; Let be the approximate weight vector of the i-th iteration of the critic neural network; The weights of the critic neural network are updated using recursive least squares, as shown below: in, Let be the derivative of the approximate weight vector of the critic neural network; P is the matrix gain of the critic neural network. Its derivative; δ is the Bellman error.
9. The method for controlling the operating point switching of a fuel cell engine system based on reinforcement learning according to claim 1, characterized in that: The construction of the transient control system for switching the operating point of the fuel cell engine system in step S3 is specifically as follows: S31. Construct a transient control error system for the operating point switching of a fuel cell engine system, transforming the optimal tracking problem into an optimal adjustment problem, and defining the tracking error as: e(t)=x(t)-x d (t); Where e(t) is the tracking error of the fuel cell engine system; x d (t) represents the given state reference value of the fuel cell engine system; S32. Obtain the transient control error system for operating point switching of the fuel cell engine system by using the tracking error e(t) of the fuel cell engine system and the reference trajectory: in, F is the derivative of the tracking error of the fuel cell engine system. e For the first internal dynamics of the fuel cell engine system; f e For the second internal dynamics of the fuel cell engine system; S33. The performance indicators of the transient control system for switching the operating point of the fuel cell engine system are defined as follows: Where e(0) is the initial tracking error; S34. Obtain the transient control system for switching the operating point of the fuel cell engine system.
10. The method for controlling the operating point switching of a fuel cell engine system based on reinforcement learning according to claim 1, characterized in that: The transient control simulation experiment for switching the operating point of the fuel cell engine system in step S4 is as follows: S41. Set the intake manifold pressure P of the fuel cell intake system. sm Cathode pressure P ca The critical neural network ω and the optimal control strategy parameters σ c The initial value; S42, The optimal control strategy parameter σ c The optimal control strategy μ for the (i+1)th iteration is calculated by substituting the critic neural network ω into the Bellman error equation. i+1 ; S43. The optimal control strategy μ in the (i+1)th iteration. i+1 Input scalar of the transient control system for switching operating points of the fuel cell engine system Combined, as a new input to the system; S44. Obtain the intake manifold pressure P of the fuel cell intake system through Simulink model simulation. sm and cathode pressure P ca ; S45. Obtain the normalized compressor flow rate Φ; S451. Obtain the optimal control strategy parameter σ c ; e(t)=x(t)-x d (t); S452. Calculate the normalized compressor flow rate Φ; F = s c (x(t+T))-σ c (x(t)); S46. Using the Bellman error calculation method of the critic neural network, obtain the Bellman error δ; S47. Use the recursive least squares method to obtain the weights of the (i+1)th iteration of the critic neural network. S48. Return to step S42 until the critic neural network weights are calculated. Once a stable state is reached, the cycle stops.
Citation Information
Patent Citations
Fuel cell control method and system based on model predictive control
CN112072142A
Fuel cell air supply system control method based on fusion type model prediction
CN116154238A
