A temperature control method for water-cooled fuel cells based on incentive learning

Abstract

The present invention relates to a temperature control method for a water-cooled fuel cell based on incentive learning. The present invention utilizes an incentive learning algorithm to achieve autonomous learning and optimization of temperature control through an intelligent agent sensing the environment, selecting actions and optimizing methods. In this method, first, the inlet and outlet water temperature data of the water-cooled fuel cell are collected in real time by sensors. These data are used to construct a state space, and the intelligent agent selects appropriate actions according to the current state to adjust the temperature control method. In order to achieve autonomous learning and optimization, a suitable incentive learning algorithm is selected, and a reward function is designed to evaluate the behavior and control method of the intelligent agent. The intelligent agent continuously updates the method and parameters through interactive feedback with the environment to improve the accuracy, stability and efficiency of temperature control. The present invention can adapt to different working conditions and requirements, has accurate and stable temperature control capabilities, and improves the working efficiency and safety of the fuel cell system.

Description

Technical Field

[0001] The present invention relates to the technical field of fuel cell thermal control, and in particular to a water-cooled fuel cell temperature control method based on incentive learning. Background Art

[0002] Fuel cells are an environmentally friendly energy technology that converts chemical energy directly into electrical energy. They offer advantages such as high efficiency, low pollution, and sustainability. However, the performance and lifespan of fuel cells are affected by many factors, temperature being a key parameter. Proper temperature control is crucial to ensuring safe and stable fuel cell operation, improving energy conversion efficiency, and extending the lifespan.

[0003] In fuel cells, temperature control is crucial for maintaining catalyst activity, promoting reaction rates, and preventing component damage. Furthermore, fuel cell operating temperatures are also affected by external environmental conditions, such as climate change and changes in load demand. Therefore, designing an efficient and reliable temperature control method is crucial for the performance and stability of fuel cell systems.

[0004] Currently, commonly used fuel cell temperature control methods include traditional PID control, model predictive control, and fuzzy control. However, these control methods have limitations in fuel cell temperature control, such as difficulty in parameter adjustment, sensitivity to model errors, complex rule design, and high computational overhead.

[0005] To overcome the limitations of traditional control methods, control methods based on incentive learning have gained increasing attention in recent years. Incentive learning methods, through a trial-and-error learning approach, are capable of handling nonlinear systems and model uncertainty. They can autonomously learn the dynamic characteristics of a system without requiring an accurate system model. They are also robust enough to adapt to long time delays and changing environments. By interacting with the environment, incentive learning methods enable online optimization, avoiding the need for accurate models and complex computations in traditional model predictive control. Overall, fuel cell temperature control using incentive learning methods exhibits good adaptability and robustness, enabling adaptation to diverse operating conditions and requirements while improving temperature control stability. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a water-cooled fuel cell temperature control method based on incentive learning. Through the interaction between the intelligent agent and the environment, the control method is adjusted according to the feedback signal, so that the fuel cell system can autonomously learn and optimize temperature control to adapt to different working conditions and needs.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a water-cooled fuel cell temperature control method based on incentive learning, comprising the following steps:

[0008] Step 1: Obtain the fuel cell inlet water temperature T in And outlet water temperature T out data;

[0009] Step 2: Establish a state-action-reward model. Use the acquired fuel cell inlet and outlet water temperature data to construct a state set, then define an action set. Finally, define a reward function R to evaluate the quality of each action in a specific state.

[0010] Step 3: Based on the state-action-reward model, the controller is trained to incentivize learning algorithms, enabling it to generate decision-making strategies and gradually improve its performance. During training, the controller learns the optimal actions to take under different states through interaction with the environment, evaluates the quality of each action through reward signals, and adjusts the state-action value function based on the reward signals. With continuous training and optimization, the controller gradually learns to make the best decisions in a given environment to maximize the cumulative reward.

[0011] Step 4: At each time step, the trained controller is used to select the optimal action for fuel cell temperature control based on the current fuel cell state. The controller selects the optimal action corresponding to the current state based on the state-action value function to obtain the maximum reward value.

[0012] Step 5: Based on the selected optimal action, execute the corresponding control method to adjust the temperature of the fuel cell;

[0013] Step 6: After executing the control method, update the state-action-reward model based on the actual control effect, compare the difference between the actual reward and the expected reward stored in the controller, provide feedback and adjust the model to continuously optimize the performance of the control method, and use the incentive learning algorithm to update the parameters of the model to make it better adapt to the actual situation.

[0014] In a preferred embodiment, the state-action-reward model in step 2 is a Markov decision model MDP, which is expressed as:

[0015] MDP = {S, Act, R, P, γ}.

[0016] In a preferred embodiment, in the Markov decision model expression, S is the state set, Act is the action set, R is the reward function, P is the state transition probability, and γ is the discount factor. In the model-free algorithm, P is difficult to know in advance, γ is determined by the environment, and the intelligent agent needs to learn through interaction with the environment.

[0017] In a preferred embodiment, the reward function R in step 2 is a function used to evaluate the behavior of the agent in reinforcement learning. The reward function R gives a numerical value as feedback based on the action taken by the agent in the environment and the state of the environment to guide the learning process of the agent. It is usually expressed as R(s,a), where s represents the state of the environment and a represents the action taken by the agent.

[0018] In a preferred embodiment, the intelligent agent refers to an individual that participates in the learning and decision-making process, obtains incentive signals through interaction with the environment, and makes decisions and takes actions based on these signals.

[0019] In a preferred embodiment, the interaction between the controller and the environment in step 3 refers to the reward value obtained by the agent during the interaction with the environment. In incentive learning, the agent learns the optimal behavior strategy through interaction with the environment. After the agent performs an action, the environment will give a reward or punishment to indicate the quality of the action. The goal of the agent is to maximize the cumulative reward value and gradually learn the optimal behavior strategy by trying different actions and observing the feedback from the environment.

[0020] In a preferred embodiment, the state-action value function in step 3 is also called the Q-function, which is a function used in reinforcement learning to evaluate the value of performing a specific action in a given state. The Q-function represents the cumulative reward that can be obtained by taking a certain action in a certain state, and is expressed as:

[0021] Q(S,Act)←Q(S,Act)+α[R+γmax Act Q(S′,Act′)-Q(S,Act)].

[0022] In a preferred embodiment, in the Q function expression, Q(S, Act) represents the Q value corresponding to the current state and the actual action taken; t Q(S′, Act′), S′, Act′ represent the Q value, S value, and Act value corresponding to the state and action taken at the next moment, respectively; R represents the reward value; α represents the learning rate; γ is the discount factor; It can be seen from the formula that the better the action, the higher the Q value.

[0023] In a preferred embodiment, the fuel cell temperature is adjusted in step 5 by adjusting the fuel cell inlet water temperature to 338.15K, and the difference between the outlet water temperature and the inlet water temperature is 283.15K.

[0024] In a preferred embodiment, the incentive learning algorithm in step 6 is intended to adjust the parameters of the model according to the reward signal obtained through interaction with the environment so that the intelligent agent can learn the optimal strategy.

[0025] Compared with existing technologies, the present invention has the following advantages: It proposes a water-cooled fuel cell temperature control method based on incentive learning. This method utilizes interactive learning between an intelligent agent and its environment to maximize cumulative rewards, automatically adjusting the fuel cell system's circulating water pump flow rate and cooling fan speed to achieve precise temperature control. Through training and optimization of the incentive learning algorithm, this method can adapt to different operating environments and conditions and achieve precise temperature control. Furthermore, the automated control process implemented through programming is simple and efficient, making it easy to integrate with specific engineering applications and solving the temperature control problem of water-cooled fuel cells in engineering applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A fuel cell temperature control Markov decision model constructed for an embodiment of the present invention;

[0027] Figure 2 Flowchart of the Q-Learning algorithm used in this embodiment of the present invention;

[0028] Figure 3 A controller for the Q-Learning algorithm in an embodiment of the present invention controls the training process;

[0029] Figure 4 This is a schematic structural diagram of a water-cooled fuel cell temperature control system based on an embodiment of the present invention; DETAILED DESCRIPTION

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0031] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0032] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0033] Based on Figure 1-4 The schematic structural diagram of the water-cooled fuel cell temperature control system is shown. This embodiment proposes a water-cooled fuel cell temperature control method based on incentive learning.

[0034] The water-cooled fuel cell temperature control system includes a temperature sensor, a fuel cell stack, a water tank, a circulating water pump, a radiator, a cooling fan, and a Q-learning controller.

[0035] 1) Obtain the fuel cell inlet and outlet water temperature data. The fuel cell inlet water temperature T in And outlet water temperature T out The data is measured by sensors in real time.

[0036] 2) Using an incentive learning algorithm, a state-action-reward model is established. The acquired fuel cell inlet and outlet water temperature data is used to construct a state set. The action set is then defined as adjusting the cooling water flow rate and the cooling fan speed. Finally, a reward function is defined to evaluate the performance of each action in a specific state.

[0037] Figure 1 In a specific case of this embodiment, the selected incentive learning algorithm is the Q-learning algorithm, which is constructed based on the Markov decision model. Figure 1 In the state-action-reward model, the Q-learning controller uses the ε-greedy strategy to specify the action to be performed based on the temperature T provided by the fuel cell model, and finally receives a reward R based on the quality of the action performed.

[0038] The Q-learning algorithm is a value-based agent algorithm in reinforcement learning. It is also a tabular algorithm used to solve MDP problems with a clear reward mechanism. Figure 2 This is a flowchart of the Q-learning algorithm. The behavior in the Q-learning algorithm adopts the ε-greedy strategy, also known as the ε-greedy strategy. This strategy is based on the gradual improvement of the Q-value table and has two parts: exploration and exploitation. The ε-greedy process of this embodiment is as follows:

[0039] ① Initialize the value of ε to 0.1;

[0040] ② At each decision point, generate a random number r between 0 and 1;

[0041] ③If r<ε (i.e. r<0.1), then explore and choose a random action;

[0042] ④ If r ≥ ε (i.e., r ≥ 0.1), then take advantage and choose the action that is currently considered to be the best. This means that in most cases, choose the action or strategy that is currently known to yield the greatest reward.

[0043] The Q value table mentioned above is shown in the following table:

[0044] Q value table

[0045]

[0046]

[0047] The rows represent all states observed from the environment model, and the columns represent different actions. The corresponding data in the table is the Q value. The Q values ​​corresponding to different state actions represent different strategies. As the number of iterations increases, the table will gradually improve. The Q value is updated as follows:

[0048] Q(S,Act)←Q(S,Act)+α[R+γmax Act Q(S′,Act′)-Q(S,Act)]

[0049] In the formula, Q(S, Act) represents the Q value corresponding to the current state and the actual action taken; Q(S′, Act′), S′, and Act′ represent the Q value, S value, and Act value corresponding to the state and action taken at the next moment, respectively; R represents the reward value; α represents the learning rate; and γ is the discount factor. As can be seen from the formula, the better the action, the higher the Q value.

[0050] 3) At each time step, the trained model is used to select the optimal action for fuel cell temperature control based on the current fuel cell state. The model predicts the expected reward for each action based on the current state and selects the action with the highest expected reward as the current control method.

[0051] Figure 3 The controller of the Q-learning algorithm controls the training process, wherein the training parameter setting part includes initializing the state space, action space, reward function, and parameters required by the Q-learning algorithm. When establishing the Q table, the initial row only has the initial value of the state. The new state value is obtained after each subsequent step and then added to the Q table in real time. The maximum number of rounds is set according to the complexity of the task. In this embodiment, the maximum number of rounds is set to 200. A larger number of rounds can improve the convergence, stability, and learning effect of the Q-learning algorithm, but it will increase the time overhead and computing resource requirements. The entire control task is decomposed into two subtasks: the action sets of task 1 and task 2 are:

[0052] Task 1:

[0053] Act = {Fan off, fan position 1, fan position 2, fan position 3, fan position 4, fan position 5}

[0054] Task 2:

[0055] Act={Water pump gear 1, water pump gear 2, water pump gear 3, water pump gear 4, water pump gear 5}

[0056] Where gears are different duty cycles, with the duty cycles increasing proportionally from gear 1 to gear 5 to 100%. Task 1 is to judge the current inlet water temperature, and the agent autonomously judges and selects the optimal action from the action set of Task 1 to execute; Task 2 is to judge the current inlet and outlet water temperature difference, and the agent autonomously judges and selects the optimal action from the action set of Task 2 to execute. Both subtasks belong to the reinforcement learning control process, and the selection of the required parameters is controlled by the Q-learning algorithm. The advantage of this is that each subtask can be optimized as an independent individual, and the reward for each subtask is more clear. The reward function of Task 1 and the reward function of Task 2 are respectively:

[0057] R1=100-50|e1| 2 -0.05D1

[0058] R2=100-50|e2| 2 -0.05D2

[0059] In the formula, the first term 100 is the basic reward value, the second term e is the difference between the current temperature value after control or the temperature difference and the set value. In this term, the smaller the difference between the current temperature value after control and the temperature set value, the greater the reward. The third term D is related to the selected gear, and its value is the number of gears selected. This term is set to minimize the control cost.

[0060] If all states and behaviors satisfy the formula:

[0061] |Q(S,Act)-Q(S′,Act′)∣<ξ

[0062] The round ends (where ξ is a tiny amount, indicating minimal change in state and behavior between the previous and next moments). Through continuous cyclic training, the agent can learn the optimal strategy from the environment and make the best decision when faced with new states to maximize the cumulative reward.

[0063] 4) At each time step, the trained controller (agent) selects the optimal action for fuel cell temperature control based on the current fuel cell state. The controller selects the optimal action for the current state based on the state-action value function to maximize the reward.

[0064] Figure 4 This is a simplified system control structure diagram in the embodiment. The temperature sensor collects the fuel cell inlet water temperature T in and the fuel cell outlet water temperature T outAfter receiving the temperature parameters, the parameters are transmitted to the incentive module and the trained Q-learning controller. The incentive module outputs the reward R to the Q-learning controller according to the reward function. The Q-learning controller performs autonomous control based on the received temperature parameters and performs optimal control on Task 1 and Task 2 according to the principle of maximizing cumulative rewards, so that the difference between the inlet water temperature and the inlet and outlet water temperature of the system meets the required temperature control effect.

[0065] 5) Based on the selected optimal action, the corresponding control method is executed to adjust the temperature of the fuel cell. For example, if the model chooses to increase the cooling efficiency of the water cooling system to reduce the temperature, the action of increasing the gear of the circulating water pump or the gear of the cooling fan will be executed to achieve this;

[0066] 6) After executing the control method, the state-action-reward model is updated according to the actual control effect. The difference between the actual reward and the expected reward stored in the controller is compared. The model can be fed back and adjusted to continuously optimize the performance of the control method. The incentive learning algorithm is used to update the parameters of the model to make it better adapt to the actual situation.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A water-cooled fuel cell temperature control method based on incentive learning, characterized in that: The following steps are involved: Step 1: Obtain the fuel cell inlet water temperature T in And outlet water temperature T out data; Step 2: Establish a state-action-reward model. Use the acquired fuel cell inlet and outlet water temperature data to construct a state set, then define an action set. Finally, define a reward function R to evaluate the quality of each action in a specific state. Step 3: Based on the state-action-reward model, the controller is trained to incentivize learning algorithms, enabling it to generate decision-making strategies and gradually improve its performance. During training, the controller learns the optimal actions to take under different states through interaction with the environment, evaluates the quality of each action through reward signals, and adjusts the state-action value function based on the reward signals. With continuous training and optimization, the controller gradually learns to make the best decisions in a given environment to maximize the cumulative reward. Step 4: At each time step, the trained controller is used to select the optimal action for fuel cell temperature control based on the current fuel cell state. The controller selects the optimal action corresponding to the current state based on the state-action value function to obtain the maximum reward value. Step 5: Based on the selected optimal action, execute the corresponding control method to adjust the temperature of the fuel cell; Step 6: After executing the control method, update the state-action-reward model based on the actual control effect, compare the difference between the actual reward and the expected reward stored in the controller, provide feedback and adjust the model to continuously optimize the performance of the control method, and use the incentive learning algorithm to update the model parameters to make it better adapt to the actual situation; The state-action-reward model in step 2 is a Markov decision model MDP, which is expressed as: In the Markov decision model expression, S is the state set, Act is the action set, R is the reward function, and P is the state transition probability. is the discount factor. In the model-free algorithm, P is difficult to know in advance. It is determined by the environment, and the agent needs to learn through interaction with the environment; The reward function R in step 2 is a function used to evaluate the behavior of the agent in reinforcement learning. The reward function R gives a numerical value as feedback based on the actions taken by the agent in the environment and the state of the environment to guide the learning process of the agent; The intelligent agent is an individual that participates in the learning and decision-making process, obtains incentive signals through interaction with the environment, and makes decisions and takes actions based on these signals; In step 2, the action set is defined as adjusting the flow of cooling water and adjusting the speed of the cooling fan; The entire control task is decomposed into two subtasks: the action sets of Task 1 and Task 2 are: Task 1: Act = {Fan off, fan gear 1, fan gear 2, fan gear 3, fan gear 4, fan gear 5} Task 2: Act ={water pump gear 1, water pump gear 2, water pump gear 3, water pump gear 4, water pump gear 5}; The reward functions for task 1 and task 2 are: , , where the first term 100 is the basic reward value, the second term e is the difference between the current temperature value after control and the set value. In this term, the smaller the difference between the current temperature value after control and the temperature set value, the greater the reward, and the third term D is related to the selected gear, and its value is the number of selected gears.

2. The water-cooled fuel cell temperature control method based on incentive learning according to claim 1, characterized in that: The interaction between the controller and the environment in step 3 refers to the reward value obtained by the agent during the interaction with the environment. In incentive learning, the agent learns the optimal behavior strategy through interaction with the environment. After the agent performs an action, the environment will give a reward or punishment to indicate the quality of the action. The goal of the agent is to maximize the cumulative reward value and gradually learn the optimal behavior strategy by trying different actions and observing the feedback from the environment.

3. The water-cooled fuel cell temperature control method based on incentive learning according to claim 1, characterized in that: The state-action value function in step 3 is also called the Q function. It is a function used in reinforcement learning to evaluate the value of performing a specific action in a given state. The Q function represents the cumulative reward that can be obtained by taking a certain action in a certain state. The expression is: 。 4. The water-cooled fuel cell temperature control method based on incentive learning according to claim 3, characterized in that: In the Q function expression, Indicates the Q value corresponding to the current state and the actual action taken; 、 、 Represents the Q value, S value and value; R represents the reward value; represents the learning rate; is the discount factor; it can be seen from the formula that the better the action, the higher the Q value.

5. The water-cooled fuel cell temperature control method based on incentive learning according to claim 1, characterized in that: The fuel cell temperature is adjusted in step 5 to adjust the fuel cell inlet water temperature to 338.15K, and the difference between the outlet water temperature and the inlet water temperature is 283.15K.

6. The water-cooled fuel cell temperature control method based on incentive learning according to claim 1, characterized in that: The incentive learning algorithm in step 6 is designed to adjust the parameters of the model according to the reward signal obtained through interaction with the environment so that the agent can learn the optimal strategy.

Images