Real-time control method for fuel cell gas supply system based on MODDPG algorithm

Abstract

The invention discloses a real-time control method for a fuel cell gas supply system based on an MODDPG algorithm. The real-time control method comprises the following steps: step 1, establishing a model of the fuel cell gas supply system; step 2, acquiring actual state information of the fuel cell gas supply system; 3, performing real-time control on the fuel cell gas supply system based on an MODDPG algorithm; according to the method, the MODDPG algorithm fusing the Pareto theory and the DDPG algorithm is provided, and the problem that global optimization is difficult to achieve in complex multi-objective optimization of a traditional discrete action control strategy is solved. Through a multi-target tradeoff mechanism of the algorithm, the optimization requirement of the fuel cell gas supply system in a complex dynamic environment can be better adapted, and the control strategy realizes optimal balance among different targets, so that the applicability and robustness of the control method are enhanced, and the overall efficiency and stability of system operation are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of fuel cell gas supply system control, and discloses a fuel cell gas supply system real-time control method based on a MODDPG algorithm. Background Art

[0002] As an important part of the fuel cell system, the fuel cell gas supply system directly affects the electrochemical reaction process, system efficiency and overall stability of the fuel cell. Fuel cells convert chemical energy into electrical energy through electrochemical reactions. Many factors such as the supply speed, pressure stability, and humidity control of the reaction gas directly affect its performance. Therefore, real-time control of the gas supply system has become an important research field that affects the overall efficiency, power output, and life of the fuel cell system.

[0003] At present, the technology of fuel cell gas supply system is developing rapidly, and the research mainly focuses on improving the system response speed, reducing power loss and achieving stable and reliable gas supply control. The gas supply system is usually composed of air compressors, valves, pipelines and humidity control devices, and needs to provide appropriate air flow and pressure for the fuel cell stack under different load conditions. At present, the main gas supply control methods used include traditional linear control methods, model predictive control (MPC) and rule-based feedback control. However, these methods have many limitations in practical applications and are difficult to cope with the complex nonlinear dynamic characteristics of fuel cell systems and high-dimensional, multi-objective optimization requirements.

[0004] The main technical bottleneck of the fuel cell gas supply system lies in the lack of real-time and multi-objective balancing capabilities of the control system. First, under high load change conditions, the reaction requirements of the fuel cell will change rapidly, and the response speed of traditional control methods is often slow, making it difficult to meet the needs of rapid dynamic changes in the system. Secondly, the gas supply system needs to comprehensively consider multiple mutually constrained objectives such as gas flow, system pressure, energy consumption, and response speed. The complexity of this multi-objective optimization problem has increased significantly, and existing control algorithms are difficult to achieve a global optimal solution. In addition, the system nonlinearity and uncertainty problems caused by environmental disturbances and operating condition fluctuations also place higher requirements on the robustness and stability of the gas supply control method.

[0005] In recent years, with the development of intelligent control technology, the application of reinforcement learning methods in complex system control has received widespread attention. As a type of efficient policy optimization algorithm in reinforcement learning, the deep deterministic policy gradient algorithm (DDPG) significantly improves the convergence speed and stability by introducing dual-strategy networks and improvements to the target network. However, the traditional DDPG algorithm still has limitations in multi-objective trade-offs and adaptability to complex environments. Direct application in the real-time control of fuel cell gas supply systems faces problems such as poor adaptability and insufficient target optimization effect. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention proposes a real-time control method for a fuel cell gas supply system based on the MODDPG algorithm, which aims to improve the adaptability of the control algorithm in a high-dimensional complex environment by introducing a multi-objective trade-off strategy, and achieve rapid optimization and balance of multiple objectives. The improved control method can comprehensively consider the reaction requirements of the fuel cell, the energy consumption and dynamic performance of the gas supply system, and has higher control accuracy, robustness and real-time response capabilities, thereby significantly improving the overall performance and energy efficiency of the fuel cell system.

[0007] The present invention discloses a fuel cell gas supply system real-time control method based on MODDPG algorithm, and the fuel cell gas supply system real-time control method based on MODDPG algorithm comprises the following steps:

[0008] Step 1, establishing a model of a fuel cell gas supply system;

[0009] Step 2, obtaining actual status information of the fuel cell gas supply system;

[0010] Step 3, real-time control of the fuel cell gas supply system based on the MODDPG algorithm;

[0011] In step 1, the following steps are also included:

[0012] Step 11, establishing a cathode flow channel model;

[0013] Step 12, establishing an anode flow channel model;

[0014] Step 3 also includes the following steps:

[0015] Step 31, taking the oxygen supply effect and hydrogen supply effect of the fuel cell gas supply system as two objectives, multi-objective optimization is performed based on the MODDPG algorithm;

[0016] Step 32, adjust the reward function of the MODDPG algorithm.

[0017] Furthermore, in step 11, using the law of conservation of mass, the dynamic equations of the three gas components can be constructed:

[0018]

[0019] in, and are the masses of oxygen and nitrogen in the cathode of the stack, respectively; and and are the mass flow rates of oxygen and nitrogen flowing into and out of the cathode of the fuel cell stack, respectively;

[0020] The partial pressures of oxygen, nitrogen and other gases in the cathode gas mixture are obtained using the ideal gas state equation:

[0021]

[0022] Among them, V ca Indicates the volume of the cathode of the battery stack; P sat is the saturation pressure of water vapor, represents the cathode oxygen partial pressure, represents the cathode nitrogen partial pressure, P ca represents the total cathode gas pressure, R represents the gas constant, T st Indicates the stack temperature, represents the molar mass of oxygen, represents the molar mass of nitrogen;

[0023] The oxygen excess ratio is used to measure the degree of cathode oxygen flow supply:

[0024]

[0025] It is the ratio of oxygen supply to reaction volume.

[0026] Furthermore, in step 12, the kinetic properties of hydrogen can be derived based on the law of conservation of mass:

[0027]

[0028] in, is the mass of hydrogen in the anode of the stack, and are the mass flow rates of hydrogen flowing into and out of the anode of the stack, The mass flow rate of hydrogen consumed by the anode electrochemical reaction;

[0029] The hydrogen partial pressure in the anode gas mixture is obtained using the ideal gas state equation:

[0030]

[0031] V an is the stack anode volume, is the mass of hydrogen in the anode of the stack, is the hydrogen partial pressure at the anode of the stack, P an is the total anode gas pressure;

[0032] Use hydrogen excess ratio Measuring the extent of hydrogen supply:

[0033]

[0034] It is the ratio of the inflow of hydrogen to the reaction amount.

[0035] Furthermore, in step 31, the following steps are also included:

[0036] Step 311: The current state information s of the agent t Input into the Actor evaluation network, the output is the corresponding action or a randomly generated action a t , and the action a t Input into the environment, and get reward r(s) through interaction with the environment t ,a t ) and the next state information s t+1 ; The status information s t+1 Input into the Actor evaluation network, and loop through step 311 until a certain number of samples e are stored. t =(s t , a t ,r(s t , a t ), s t+1 );

[0037] Step 312, randomly extract a portion of samples e from the experience pool, first convert the state information s in the sample e t Input into the Actor evaluation network to get the corresponding action a t , and then the status information s t and the corresponding action a t Input into the Critic evaluation network together to obtain the state information s t and action a t The cumulative reward Q(s) under t , a t |θ), θ represents the parameters of the Critic evaluation network; by finding Q(s t , a t |θ) to obtain the loss function and back propagate it, and use the gradient ascent method to update the parameters in the Actor evaluation network, that is, through back propagation, the loss function in state s is obtained. t Next Q(s t , a t |θ) The action corresponding to the maximum expected value;

[0038] Step 313: The state information s in the sample e sampled in step 312 is t Input into the Actor evaluation network to get the corresponding action a after the Actor evaluation network is updated t , and then the status information s t and the corresponding action a tThe two are input into the Critic evaluation network together, and the Actor evaluation network is updated in the state information s t and action a t The cumulative reward Q(s) under t , a t |θ);

[0039] Step 314: the next state information s in the sample e sampled in step 312 is t+1 Input into the Actor target network to get the corresponding action a t+1 , and then the status information s t+1 and the corresponding action a t+1 Input into the Critic target network together to obtain the state information s t+1 and action a t+1 The cumulative reward Q(s) under t+1 , a t+1 |θ′), where θ′ is the parameter of the Critic target network;

[0040] Step 315, back propagation of the loss function Loss, using the gradient descent method to update the parameters in the Critic evaluation network;

[0041] Step 316, looping steps 311-315, after each C step, updating the parameters of the Actor evaluation network and the Critic evaluation network to the Actor target network and the Critic target network.

[0042] Furthermore, in step 312, in order to improve the single-objective DDPG algorithm into a multi-objective DDPG algorithm, the reward function is improved from a single objective to a multi-objective by combining the Pareto theory, that is, from a numerical value to an array, and each numerical value in the array is an objective of the reward function.

[0043] Furthermore, in step 32, the reward function for measuring the effect of oxygen and hydrogen supply can be specifically expressed as:

[0044]

[0045] The reward function for the oxygen supply effect, is the reward function for the hydrogen supply effect, Because the amount of The reward generated by the control error, Because the amount of The reward generated by the change in the absolute value of the control error, The bonus is due to the change of air compressor voltage. Because the amount of The reward generated by the control error, Because the amount of The reward generated by the change in the absolute value of the control error, The reward is due to the change in the speed of the hydrogen circulation pump;

[0046] In order to ensure the controlled and The error gradually decreases and converges to 0. It is necessary to consider the error and

[0047]

[0048]

[0049] The controlled amount The control error, The controlled amount Control error of

[0050] A decrease in the absolute value of the error indicates that the control effect has been improved, and a positive reward value is given. On the contrary, an increase in the absolute value of the error gives a negative reward value:

[0051]

[0052]

[0053] Among them, δ represents a very small positive value;

[0054] Give positive rewards when the action volume changes slightly, and give negative rewards when the action volume changes significantly:

[0055]

[0056]

[0057] Wherein, ΔV′ represents the change of the normalized value of the air compressor voltage V, and S′ represents the change of the normalized value of the speed S of the hydrogen circulation pump.

[0058] Furthermore, the fuel cell gas supply system real-time control method based on the MODDPG algorithm further includes the following steps:

[0059] Step 4, hardware-in-the-loop testing, specifically includes the following steps:

[0060] Step 41, model building and simulation initialization;

[0061] Accurately model the physical model of the fuel cell gas supply system, including the air supply system, hydrogen supply system, fuel cell stack dynamics model, etc. The model needs to be calibrated with experimental data to ensure that its dynamic response and physical characteristics are consistent with the real system;

[0062] Step 42, controller connection and communication test;

[0063] Physically connect the controller hardware based on the MODDPG algorithm to the real-time simulation platform to test and verify whether the communication interface between the controller and the simulation platform is stable and whether the data transmission is accurate;

[0064] Step 43, multi-operating condition testing and verification;

[0065] By setting up a variety of operating scenarios, the response performance of the controller under complex conditions can be tested;

[0066] Step 44, disturbance test and robustness verification;

[0067] Introduce simulated external disturbances and nonlinear interferences to verify the robustness and adaptability of the control method;

[0068] Step 45, data analysis and performance evaluation;

[0069] The monitoring and data acquisition system records the controller's output response, simulation system status and performance indicators in real time, and conducts a comprehensive evaluation of the control method's real-time performance, stability, response speed, and multi-objective optimization effect.

[0070] The beneficial effects achieved by the present invention are:

[0071] The present invention conducts in-depth analysis and modeling of the fuel cell gas supply system. The present invention constructs a simplified but effective mathematical model for the cathode and anode flow channels, which can reflect the output power, gas flow and pressure change characteristics of the system in real time during operation. This model provides an accurate and fast dynamic response basis for the real-time optimization control of the gas supply system, significantly improves the control accuracy and the overall performance stability of the system, and can effectively meet the response requirements of the fuel cell system under different working conditions.

[0072] The present invention proposes a MODDPG algorithm that integrates Pareto theory and DDPG algorithm, which overcomes the problem that traditional discrete action control strategies are difficult to achieve global optimality in complex multi-objective optimization. Through the multi-objective trade-off mechanism of the algorithm, it can better adapt to the optimization needs of the fuel cell gas supply system under complex dynamic environments, so that the control strategy can achieve the optimal balance between different objectives, thereby enhancing the applicability and robustness of the control method and improving the overall efficiency and stability of the system operation.

[0073] The present invention aims at the problem of real-time supply of reaction gas in the fuel cell gas supply system. The present invention takes air compressors and hydrogen circulation pumps as control objects, and constructs a multi-objective reward function that comprehensively considers the absolute error, absolute change and control action change of the oxygen excess ratio and hydrogen excess ratio. On the basis of achieving real-time control, the accuracy and flexibility of the control strategy are optimized to ensure the stable supply of reaction gas, thereby improving the operating efficiency and life of the fuel cell. Through the multi-objective optimization control method, the present invention realizes more efficient gas supply system control to meet the dynamic needs of the fuel cell under different working conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a flow chart of a real-time control method of a fuel cell gas supply system based on a MODDPG algorithm proposed by the present invention;

[0075] Figure 2 It is a flow chart of the MODDPG algorithm in the real-time control method of the fuel cell gas supply system based on the MODDPG algorithm proposed by the present invention. DETAILED DESCRIPTION

[0076] The present invention will be further described below in conjunction with specific embodiments, and the advantages and features of the present invention will become clearer as the description proceeds. However, these embodiments are exemplary only and do not constitute any limitation to the scope of the present invention. It should be understood by those skilled in the art that the details and forms of the technical solution of the present invention may be modified or replaced without departing from the spirit and scope of the present invention, but these modifications and replacements all fall within the scope of protection of the present invention.

[0077] like Figure 1 As shown, the present invention proposes a real-time control method for a fuel cell gas supply system based on a MODDPG algorithm, the method comprising the following steps:

[0078] Step 1: Establish a model of the fuel cell gas supply system.

[0079] The gas supply system of the proton exchange membrane fuel cell is mainly composed of two parts: the hydrogen path and the air path. During the operation of the fuel cell, hydrogen is supplied through a high-pressure hydrogen storage tank, while air is introduced by an air compressor and enters the fuel cell stack separately, and is distributed to each fuel cell monomer along multiple gas flow channels. The two gases undergo electrochemical reactions under the action of the electrode catalyst and the proton exchange membrane, and the generated reaction products are discharged from the stack in the form of a water-gas mixture through multiple components such as the exhaust valve, drain valve, and back pressure valve. After proper treatment, these products are either recycled into the stack to continue the reaction, or discharged into the atmosphere.

[0080] Specifically, the following steps are also included:

[0081] Step 11: Establish cathode flow channel model

[0082] After the oxygen in the air enters the cathode of the fuel cell stack, it combines with the H+ conducted in the electrolyte and the electrons in the external circuit to generate water. Therefore, the gas flow channel on the cathode side of the stack mainly contains three components: oxygen, water and nitrogen. Using the law of conservation of mass, the dynamic equations of these three gas components can be constructed:

[0083]

[0084] in, and are the masses of oxygen and nitrogen in the cathode of the stack, respectively; and and are the mass flow rates of oxygen and nitrogen flowing into and out of the cathode of the fuel cell stack, respectively; The mass flow rate of oxygen consumed by the stack reaction is specifically expressed as:

[0085]

[0086] in, is the molar mass of oxygen.

[0087] The partial pressures of oxygen, nitrogen and other gases in the cathode gas mixture are obtained using the ideal gas state equation:

[0088]

[0089] Among them, V ca Indicates the volume of the cathode of the battery stack, P sat is the saturation pressure of water vapor, represents the cathode oxygen partial pressure, represents the cathode nitrogen partial pressure, P ca represents the total cathode gas pressure, R represents the gas constant, T st Indicates the stack temperature, represents the molar mass of oxygen, represents the molar mass of nitrogen.

[0090] The oxygen excess ratio is used to measure the degree of cathode oxygen flow supply, which is specifically defined as the ratio of oxygen supply to reaction amount:

[0091]

[0092] Step 12: Establish anode flow channel model

[0093] The gas in the anode flow channel of the fuel cell stack is mainly composed of hydrogen and water vapor, and the dynamic behavior of the hydrogen flow can be modeled in a similar way to the oxygen flow in the cathode of the fuel cell stack. During the operation of the fuel cell, the kinetic characteristics of hydrogen can be derived based on the law of conservation of mass:

[0094]

[0095] in, is the mass of hydrogen in the anode of the stack; and are the mass flow rates of hydrogen flowing into and out of the stack anode, respectively; is the mass flow rate of hydrogen consumed by the internal reaction of the fuel cell stack, which is specifically expressed as:

[0096]

[0097] in, is the molar mass of hydrogen.

[0098] The hydrogen partial pressure in the anode gas mixture is obtained using the ideal gas state equation:

[0099]

[0100] Among them, V an is the stack anode volume, is the mass of hydrogen in the anode of the stack, is the hydrogen partial pressure at the anode of the stack, P an is the total anode gas pressure.

[0101] Use hydrogen excess ratio A measure of the degree of hydrogen supply, defined as the ratio of the inflow of hydrogen to the amount of reaction:

[0102]

[0103] Step 2: Obtain actual status information of the fuel cell gas supply system. ΔV′ and ΔS′ are the acquired state quantities of the fuel cell gas supply system.

[0104] Step 3, real-time control of the fuel cell gas supply system based on the MODDPG algorithm;

[0105] Specifically, the following steps are also included:

[0106] Step 31, MODDPG algorithm

[0107] Assume that the continuous state set of the agent is S and the continuous action sequence is A. When the current state of the agent is s t ∈S, take action at ∈A, the state of the agent is transferred to a new state s under the influence of the environment. t+1 ∈S, the immediate reward generated is r(s t , a t ). Deep reinforcement learning selects the agent’s actions by maximizing the cumulative reward, that is, taking into account both immediate rewards and future rewards, and continuously improving the strategy π to maximize the cumulative reward. The strategy corresponding to the maximum cumulative reward is the optimal strategy π * (a|s). The strategy π is a series of actions taken by the agent from the beginning to the end.

[0108] The agent state is s t , take action a t The optimal action value function Q * (s t , a t )for:

[0109]

[0110] Among them, R t is the cumulative reward with discount, γ is the discount factor, γ∈[0,1], T is the termination time. The optimal action value function Q * (s t , a t ) follows the Bellman equation:

[0111] Q * (s t , a t )=E[r(s t , a t )+γQ * (s t+1 , a t+1 )|s t , a t ] (10)

[0112] like Figure 1 As shown, the main process of the MODDPG algorithm is as follows:

[0113] Step 311, such as Figure 1 As shown in the dashed box ①, the current state information s of the agent t Input into the Actor Evaluation Network, the output is the corresponding action (i.e., utilization), or a randomly generated action (i.e., exploration), denoted as a t , and the action a t Input into the environment, and get reward r(s) through interaction with the environment t , a t ) and the next state information st+1 . Send the current status information t , selected action a t , the reward r(s t , a t ) and the next state s t+1 Store it in the experience pool. Then transfer the status information s t+1 Input into the Actor evaluation network, and loop through step 311 until a certain number of e are stored. t =(s t , a t ,r(s t , a t ), s t+1 ), in this process, the parameters in the Actor evaluation network are not updated. Among them, the ε-greedy algorithm is used to achieve a balance between exploration and utilization when selecting actions, ensuring sufficient exploration and reasonable utilization:

[0114]

[0115] Among them, ε∈[0,1] is the exploration rate. The algorithm chooses exploration with a probability of ε and selects the action of the Actor evaluation network output with a probability of 1-ε. Therefore, in order to ensure the performance of the deep reinforcement learning algorithm and prevent it from falling into the local optimum, ε is generally set to a larger initial value to ensure sufficient exploration ability. As the iteration proceeds, the value of ε gradually decreases to ensure full utilization and accelerate the convergence of the algorithm.

[0116] Step 312, such as Figure 1 As shown in the dashed box ②, a part of samples e is randomly selected from the experience pool. First, the state information s in e is t Input into the Actor evaluation network to get the corresponding action a t , and then the status information s t and the corresponding action a t Input into the Critic evaluation network together to obtain the state information s t and action a t The cumulative reward Q(s) under t ,a t |θ), θ represents the parameters of the Critic evaluation network. By finding Q(s t ,a t |θ) to obtain the loss function and back-propagate, and use the gradient ascent method to update the parameters in the Actor evaluation network, that is, through back-propagation, the loss function in state s is obtained. t Next Q(s t ,a t|θ) is the action corresponding to the maximum expected value. Note that in this step, only the parameters in the Actor evaluation network are updated, and the parameters in the Critic evaluation network remain unchanged.

[0117] Since the reward function r in the single-objective DDPG algorithm is a numerical value, its cumulative reward Q is also a numerical value. Similarly, the expected value of the cumulative reward and the loss function are both numerical values. When improving the single-objective DDPG algorithm to a multi-objective DDPG algorithm, the reward function is improved from a single objective to a multi-objective by combining the Pareto theory, that is, from a numerical value to an array, and each numerical value in the array is a target of the reward function. After the improvement, there are two problems that need to be solved, one is how to select the maximum cumulative reward Q, and the other is how to calculate the loss function and perform back propagation.

[0118] First, regarding how to select the maximum cumulative reward Q, that is, how to compare the size of the cumulative reward Q, the Pareto optimal frontier is introduced to select the maximum cumulative reward Q.

[0119] Pareto optimal frontier: Assume that there are i objective functions in the multi-objective problem, and A and B are its two feasible solutions. If all the objective function values ​​of solution A are better than solution B, then solution A is said to be better than solution B, that is, solution A dominates solution B; if only some objective functions of solution A are better than solution B, then solution A and solution B are said to be indistinguishable, that is, solution A does not dominate solution B. If the objective function value of solution A is better than any other solution in the feasible space, then A is called the optimal solution; if no other solution is better than solution A in the feasible space, then solution A is called the Pareto optimal solution. For multi-objective optimization problems, there is generally no optimal solution, but there are multiple Pareto optimal solutions, and all Pareto optimal solutions constitute the Pareto optimal frontier.

[0120] Therefore, when selecting the maximum cumulative reward Q, we first solve the Pareto optimal frontier about Q, and then formulate a rule on the Pareto optimal frontier to select the maximum cumulative reward Q, such as randomly selecting a Q as the maximum cumulative reward, or selecting the Q with the smallest target as the maximum cumulative reward, etc.

[0121] Secondly, how to find the loss function and perform back propagation. Since multiple targets generate multiple loss functions, the gradient is calculated based on multiple loss functions and back propagated.

[0122] Step 313, as Figure 1 As shown in the dashed box ③, the state information s in e sampled in (2) is t Input into the Actor evaluation network to get the corresponding action a after the Actor evaluation network is updated t , and then the status information s t and the corresponding action at The two are input into the Critic evaluation network together, and the Actor evaluation network is updated in the state information s t and action a t The cumulative reward Q(s) under t , a t |θ).

[0123] Step 314, such as Figure 1 As shown in the dashed box ④, the next state information s in e sampled in (2) is t+1 Input into the Actor target network to get the corresponding action a t+1 , and then the status information s t+1 and the corresponding action a t+1 Input into the Critic target network together to obtain the state information s t+1 and action a t+1 The cumulative reward Q(s) under t+1 , a t+1 |θ′), where θ′ is the parameter of the Critic target network.

[0124] Step 315, as Figure 1 As shown in the dashed box ⑤, the loss function Loss is back-propagated, and the gradient descent method is used to update the parameters in the Critic evaluation network. The loss function is the mean square error (MSE), and the loss calculation formula is:

[0125]

[0126] Among them, E is the number of e sampled from the experience pool.

[0127] Step 316, looping steps 311-315, after each C step, updating the parameters of the Actor evaluation network and the Critic evaluation network to the Actor target network and the Critic target network.

[0128] Step 32, reward function

[0129] In order to ensure the controlled and The error gradually decreases and converges to 0. It is necessary to consider the error and When the error is small, the reward is a large positive value, and as the error decreases, the reward value should increase gradually. When the error is large, the reward is a small negative value:

[0130]

[0131] The controlled amount The control error, The controlled amount Control error of

[0132] At the same time, in order to better keep the control amount moving in the direction of reducing the error at any time, the change in the absolute value of the error and Incorporate into the reward function. A decrease in the absolute value of the error indicates that the control effect has improved, and a positive reward value is given. On the contrary, an increase in the absolute value of the error gives a negative reward value:

[0133]

[0134] Here, δ represents a very small positive value.

[0135] In addition, in order to avoid frequent and significant changes in the working state of the actuator in the actual process, thereby reducing equipment loss, the effect of the change in action volume is introduced into the reward function. When the change in action volume is small, a positive reward is given, and when the change in action volume is large, a negative reward is given:

[0136]

[0137] Among them, V′ and S′ represent the normalized values ​​of the air compressor voltage V and the hydrogen circulation pump speed S, respectively. V and S are the action quantities of the deep reinforcement learning algorithm. ΔV′ and ΔS′ are the state variables of the deep reinforcement learning algorithm. The reward function for measuring the oxygen and hydrogen supply effects can be specifically expressed as the sum of the above three items, which are also the two optimization objectives of the MODDPG algorithm:

[0138]

[0139] Step 4: Hardware-in-the-loop testing

[0140] Hardware-in-the-loop testing is an important means to verify and optimize the real-time control method of the fuel cell gas supply system based on the MODDPG algorithm. It can realistically simulate the actual system operation in a simulation environment to ensure the stability, response speed and performance optimization capabilities of the control algorithm under complex working conditions. Through HIL testing, comprehensive testing and verification of multiple scenarios and working conditions can be carried out without affecting the safety and stability of the actual fuel cell system, laying a solid foundation for the engineering application of the algorithm.

[0141] Step 41, model building and simulation initialization. First, the physical model of the fuel cell gas supply system is accurately modeled, including the air supply system, hydrogen supply system, fuel cell stack dynamics model, etc. The model needs to be calibrated with experimental data to ensure that its dynamic response and physical characteristics are consistent with the real system. Then, the model is loaded and initialized on the real-time simulation platform to simulate different working conditions and load changes.

[0142] Step 42, controller connection and communication test. Physically connect the controller hardware based on the MODDPG algorithm to the real-time simulation platform, test and verify whether the communication interface between the controller and the simulation platform is stable and whether the data transmission is accurate. Ensure the real-time transmission of control instructions and the accurate reception of feedback data.

[0143] Step 43, multi-condition testing and verification. By setting up a variety of operating scenarios, including different load changes, environmental disturbances, changes in gas flow and pressure requirements, the response performance of the controller under complex conditions is tested. The focus is on verifying the real-time performance, response speed and multi-objective balance ability of the improved multi-objective TD3 algorithm under rapidly changing conditions, including the coordinated control effect of multiple objectives such as gas flow, system pressure, energy consumption and stability.

[0144] Step 44, disturbance test and robustness verification. Introduce simulated external disturbances and nonlinear disturbances, such as random load changes, gas supply disturbances, etc., to verify the robustness and adaptability of the control method. Observe and record the stability and adjustment ability of the controller in response to complex nonlinear disturbance conditions to ensure the robust operation of the system in a complex environment.

[0145] Step 45, data analysis and performance evaluation. The output response of the controller, the simulation system status and performance indicators are recorded in real time by the monitoring and data acquisition system, and the real-time performance, stability, response speed, multi-objective optimization effect, etc. of the control method are comprehensively evaluated. The controller performance data under different working conditions are compared, and the control parameters are optimized to further improve the practical application effect of the algorithm.

[0146] The above are only specific steps of the present invention and do not constitute any limitation to the protection scope of the present invention; any technical solutions formed by equivalent transformation or equivalent replacement shall fall within the protection scope of the present invention; some parts not elaborated in detail in the present invention belong to the known technologies of those skilled in the art.

Claims

1. A real-time control method for a fuel cell gas supply system based on the MODDPG algorithm, characterized in that: The fuel cell gas supply system real-time control method based on the MODDPG algorithm comprises the following steps: Step 1, establishing a model of a fuel cell gas supply system; Step 2, obtaining actual status information of the fuel cell gas supply system; Step 3, real-time control of the fuel cell gas supply system based on the MODDPG algorithm; In step 1, the following steps are also included: Step 11, establishing a cathode flow channel model; Step 12, establishing an anode flow channel model; Step 3 also includes the following steps: Step 31, taking the oxygen supply effect and hydrogen supply effect of the fuel cell gas supply system as two objectives, multi-objective optimization is performed based on the MODDPG algorithm; Step 32, adjust the reward function of the MODDPG algorithm.

2. The real-time control method of the fuel cell gas supply system based on the MODDPG algorithm according to claim 1 is characterized in that: In step 11, using the law of conservation of mass, the dynamic equations of the three gas components can be constructed: in, and are the masses of oxygen and nitrogen in the cathode of the stack, respectively; and and are the mass flow rates of oxygen and nitrogen flowing into and out of the cathode of the fuel cell stack, respectively; The partial pressures of oxygen, nitrogen and other gases in the cathode gas mixture are obtained using the ideal gas state equation: Among them, V ca Indicates the volume of the cathode of the battery stack; P sat is the saturation pressure of water vapor, represents the cathode oxygen partial pressure, represents the cathode nitrogen partial pressure, P ca represents the total cathode gas pressure, R represents the gas constant, T st Indicates the stack temperature, represents the molar mass of oxygen, represents the molar mass of nitrogen; The oxygen excess ratio is used to measure the degree of cathode oxygen flow supply: It is the ratio of oxygen supply to reaction volume.

3. The real-time control method of the fuel cell gas supply system based on the MODDPG algorithm according to claim 1 is characterized in that: In step 12, the kinetic properties of hydrogen can be derived based on the law of conservation of mass: in, is the mass of hydrogen in the anode of the stack, and are the mass flow rates of hydrogen flowing into and out of the anode of the stack, The mass flow rate of hydrogen consumed by the anode electrochemical reaction; The hydrogen partial pressure in the anode gas mixture is obtained using the ideal gas state equation: V an is the stack anode volume, is the mass of hydrogen in the anode of the stack, is the hydrogen partial pressure at the anode of the stack, P an is the total anode gas pressure; Use hydrogen excess ratio Measuring the extent of hydrogen supply: It is the ratio of the inflow of hydrogen to the reaction amount.

4. The real-time control method for a fuel cell gas supply system based on the MODDPG algorithm according to claim 1 is characterized in that: In step 31, the following steps are also included: Step 311: The current state information s of the agent t Input into the Actor evaluation network, the output is the corresponding action or a randomly generated action a t , and the action a t Input into the environment, and get reward r(s) through interaction with the environment t ,a t ) and the next state information s t+1 ; The status information s t+1 Input into the Actor evaluation network, and loop through step 311 until a certain number of samples e are stored. t =(s t ,a t ,r(s t ,a t ),s t+1 ); Step 312, randomly extract a portion of samples e from the experience pool, first convert the state information s in the sample e t Input into the Actor evaluation network to get the corresponding action a t , and then the status information s t and the corresponding action a t Input into the Critic evaluation network together to obtain the state information s t and action a t The cumulative reward Q(s) under t ,a t |θ), θ represents the parameters of the Critic evaluation network; by finding Q(s t ,a t |θ) to obtain the loss function and back-propagate, and use the gradient ascent method to update the parameters in the Actor evaluation network, that is, through back-propagation, the loss function in state s is obtained. t Next Q(s t ,a t |θ) The action corresponding to the maximum expected value; Step 313: The state information s in the sample e sampled in step 312 is t Input into the Actor evaluation network to get the corresponding action a after the Actor evaluation network is updated t , and then the status information s t and the corresponding action a t The two are input into the Critic evaluation network together, and the Actor evaluation network is updated in the state information s t and action a t The cumulative reward Q(s) under t ,a t |θ); Step 314: the next state information s in the sample e sampled in step 312 is t+1 Input into the Actor target network to get the corresponding action a t+1 , and then the status information s t+1 and the corresponding action a t+1 Input into the Critic target network together to obtain the state information s t+1 and action a t+1 The cumulative reward Q(s) under t+1 ,a t+1 |θ ′ ), where θ ′ are the parameters of the Critic target network; Step 315, back propagation of the loss function Loss, using the gradient descent method to update the parameters in the Critic evaluation network; Step 316, looping steps 311-315, after each C step, updating the parameters of the Actor evaluation network and the Critic evaluation network to the Actor target network and the Critic target network.

5. The real-time control method of the fuel cell gas supply system based on the MODDPG algorithm according to claim 4 is characterized in that: In step 312, in order to improve the single-objective DDPG algorithm into a multi-objective DDPG algorithm, the reward function is improved from a single objective to a multi-objective by combining the Pareto theory, that is, from a numerical value to an array, and each numerical value in the array is an objective of the reward function.

6. The real-time control method for a fuel cell gas supply system based on the MODDPG algorithm according to claim 4 is characterized in that: In step 32, the reward function for measuring the oxygen and hydrogen supply effects can be specifically expressed as: The reward function for the oxygen supply effect, is the reward function for the hydrogen supply effect, Because the amount of The reward generated by the control error, Because the amount of The reward generated by the change in the absolute value of the control error, The bonus is due to the change of air compressor voltage. Because the amount of The reward generated by the control error, Because the amount of The reward generated by the change in the absolute value of the control error, The reward is due to the change in the speed of the hydrogen circulation pump; In order to ensure the controlled and The error gradually decreases and converges to 0. It is necessary to consider the error and The controlled amount The control error, The controlled amount Control error of A decrease in the absolute value of the error indicates that the control effect has been improved, and a positive reward value is given. On the contrary, an increase in the absolute value of the error gives a negative reward value: Among them, δ represents a very small positive value; Give positive rewards when the action volume changes slightly, and give negative rewards when the action volume changes significantly: Wherein, ΔV′ represents the change of the normalized value of the air compressor voltage V, and S′ represents the change of the normalized value of the speed S of the hydrogen circulation pump.

7. The real-time control method for a fuel cell gas supply system based on the MODDPG algorithm according to claim 1 is characterized in that: The fuel cell gas supply system real-time control method based on the MODDPG algorithm also includes the following steps: Step 4, hardware-in-the-loop testing, specifically includes the following steps: Step 41, model building and simulation initialization; Accurately model the physical model of the fuel cell gas supply system, including the air supply system, hydrogen supply system, fuel cell stack dynamics model, etc. The model needs to be calibrated with experimental data to ensure that its dynamic response and physical characteristics are consistent with the real system; Step 42, controller connection and communication test; Physically connect the controller hardware based on the MODDPG algorithm to the real-time simulation platform to test and verify whether the communication interface between the controller and the simulation platform is stable and whether the data transmission is accurate; Step 43, multi-operating condition testing and verification; By setting up a variety of operating scenarios, the response performance of the controller under complex conditions can be tested; Step 44, disturbance test and robustness verification; Introduce simulated external disturbances and nonlinear interferences to verify the robustness and adaptability of the control method; Step 45, data analysis and performance evaluation; The monitoring and data acquisition system records the controller's output response, simulation system status and performance indicators in real time, and conducts a comprehensive evaluation of the control method's real-time performance, stability, response speed, and multi-objective optimization effect.

Images