Water-cooled fuel cell temperature control method based on incentive learning

Through the water-cooled fuel cell temperature control method based on excitation learning, the interaction between the agent and the environment is used to establish a state-action-reward model and train the excitation learning algorithm controller, the difficulty of parameter adjustment of the temperature control method in the prior art is solved and The model error sensitivity problem achieves accurate and stable control of fuel cell temperature.

CN118919776AActive Publication Date: 2024-11-08MINJIANG UNIVERSITY +1

Patent Information

Application Number
CN202410948934.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

The existing fuel cell temperature control methods have problems such as difficulty in adjusting parameters, sensitive model errors, complex rule design, and large calculation overhead, making it difficult to adapt to different working conditions and environmental changes.

Method used

The water-cooled fuel cell temperature control method based on excitation learning is adopted, through the interaction between the agent and the environment, a state-action-reward model is established, the excitation learning algorithm controller is trained, and the temperature control is optimized independently to adapt to different working conditions and needs.

Benefits of technology

It realizes precise control of fuel cell temperature, adapts to different operating environments and working conditions, simplifies the automated control process, and improves the stability and adaptability of temperature control.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118919776A_ABST
    Figure CN118919776A_ABST
Patent Text Reader

Abstract

The invention relates to a water-cooled fuel cell temperature control method based on incentive learning, which realizes autonomous learning and optimization of temperature control by utilizing an incentive learning algorithm and sensing an environment, selecting actions and optimizing a method through an intelligent agent. The method comprises the following steps: firstly, collecting inlet water temperature and outlet water temperature data of the water-cooled fuel cell in real time through a sensor; the data are used for constructing a state space, and the intelligent agent selects proper actions to adjust the temperature control method according to the current state. In order to realize autonomous learning and optimization, a proper incentive learning algorithm is selected, and a reward function is designed to evaluate behaviors and a control method of the intelligent agent. The intelligent agent continuously updates the method and the parameters through interactive feedback with the environment so as to improve the accuracy, the stability and the efficiency of temperature control. The system can adapt to different working conditions and requirements, has accurate and stable temperature control capability, and improves the working efficiency and safety of the fuel cell system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of fuel cell thermal control, and in particular to a water-cooled fuel cell temperature control method based on incentive learning. Background Art

[0002] Fuel cells are an environmentally friendly energy technology that converts chemical energy directly into electrical energy. They have the advantages of high efficiency, low pollution and sustainability. However, the performance and life of fuel cells are affected by many factors, among which temperature is an important parameter. Reasonable temperature control is essential to ensure the safe and stable operation of fuel cells, improve energy conversion efficiency and extend service life.

[0003] In fuel cells, temperature control is critical to maintaining catalyst activity, promoting reaction rates, and preventing component damage. At the same time, the operating temperature of fuel cells is also affected by external environmental conditions, such as climate change and changes in load demand. Therefore, designing an efficient and reliable temperature control method is crucial to the performance and stability of the fuel cell system.

[0004] At present, commonly used fuel cell temperature control methods include traditional PID control, model predictive control and fuzzy control, etc. However, these control methods have some limitations in fuel cell temperature control, such as difficult parameter adjustment, sensitive model errors, complex rule design and high computational overhead.

[0005] In order to overcome the limitations of traditional control methods, control methods based on incentive learning have gradually attracted attention in recent years. The incentive learning method can handle nonlinear systems and model uncertainties through a trial-and-error learning method. It can autonomously learn the dynamic characteristics of the system without an accurate system model. The incentive learning method can also adapt to long time lags and changing environments and has a certain degree of robustness. Through interaction with the environment, the incentive learning method can achieve online optimization, avoiding the problem of accurate models and complex calculations in traditional model predictive control. In general, fuel cell temperature control using incentive learning method has good adaptability and robustness, can adapt to different working conditions and requirements, and improve the stability of temperature control. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a water-cooled fuel cell temperature control method based on incentive learning. Through the interaction between the intelligent agent and the environment, the control method is adjusted according to the feedback signal, so that the fuel cell system can autonomously learn and optimize temperature control to adapt to different working conditions and needs.

[0007] To achieve the above object, the present invention adopts the following technical solution: a water-cooled fuel cell temperature control method based on incentive learning, comprising the following steps:

[0008] Step 1: Obtain the fuel cell inlet water temperature T in And outlet water temperature T out data;

[0009] Step 2: Establish a state-action-reward model, use the obtained fuel cell inlet and outlet water temperature data to construct the state set, then define the action set, and finally define the reward function R to evaluate the pros and cons of each action in a specific state;

[0010] Step 3: Based on the state-action-reward model, the controller is trained to stimulate the learning algorithm so that it can generate decision-making strategies and gradually improve its performance. During the training process, the controller learns the best actions to take in different states through interaction with the environment, evaluates the quality of each action through reward signals, and adjusts the state-action value function according to the reward signals. With continuous training and optimization, the controller gradually learns to make the best decision in a given environment to obtain the maximum cumulative reward.

[0011] Step 4: At each time step, the best action for fuel cell temperature control is selected using the trained controller according to the current state of the fuel cell. The controller selects the best action corresponding to the current state according to the state-action value function to obtain the maximum reward value.

[0012] Step 5: According to the selected optimal action, a corresponding control method is executed to adjust the temperature of the fuel cell;

[0013] Step 6: After executing the control method, update the state-action-reward model according to the actual control effect, compare the difference between the actual reward and the expected reward stored in the controller, feedback and adjust the model to continuously optimize the performance of the control method, and use the incentive learning algorithm to update the parameters of the model to better adapt it to the actual situation.

[0014] In a preferred embodiment, the state-action-reward model in step 2 is a Markov decision model MDP, which is expressed as:

[0015] MDP={S,Act,R,P,γ}.

[0016] In a preferred embodiment, in the Markov decision model expression, S is the state set, Act is the action set, R is the reward function, P is the state transition probability, and γ is the discount factor. In the model-free algorithm, P is difficult to know in advance, γ is determined by the environment, and the agent needs to learn through interaction with the environment.

[0017] In a preferred embodiment, the reward function R in step 2 is a function used to evaluate the behavior of the agent in reinforcement learning. The reward function R gives a numerical value as feedback based on the action taken by the agent in the environment and the state of the environment to guide the learning process of the agent. It is usually expressed as R(s,a), where s represents the state of the environment and a represents the action taken by the agent.

[0018] In a preferred embodiment, the intelligent agent refers to an individual that participates in the learning and decision-making process, obtains incentive signals through interaction with the environment, and makes decisions and takes actions based on these signals.

[0019] In a preferred embodiment, the interaction between the controller and the environment in step 3 refers to the reward value obtained by the agent during the interaction with the environment. In incentive learning, the agent learns the optimal behavior strategy by interacting with the environment. After the agent performs an action, the environment will give a reward or punishment to indicate the quality of the action. The goal of the agent is to maximize the cumulative reward value and gradually learn the optimal behavior strategy by trying different actions and observing the feedback from the environment.

[0020] In a preferred embodiment, the state-action value function in step 3 is also called the Q-function, which is a function used in reinforcement learning to evaluate the value of performing a specific action in a given state. The Q-function represents the cumulative reward that can be obtained by taking a certain action in a certain state, and the expression is:

[0021] Q(S,Act)←Q(S,Act)+α[R+γmax Act Q(S′,Act′)-Q(S,Act)].

[0022] In a preferred embodiment, in the Q function expression, Q(S, Act) represents the Q value corresponding to the current state and the actual action taken; t Q(S′,Act′), S′, Act′ represent the Q value, S value and Act value corresponding to the state and action taken at the next moment respectively; R represents the reward value; α represents the learning rate; γ is the discount factor; It can be seen from the formula that the better the action, the higher the Q value.

[0023] In a preferred embodiment, the fuel cell temperature is adjusted in step 5 by adjusting the fuel cell inlet water temperature to 338.15K, and the difference between the outlet water temperature and the inlet water temperature is 283.15K.

[0024] In a preferred embodiment, the incentive learning algorithm in step 6 is intended to adjust the parameters of the model according to the reward signal obtained through interaction with the environment so that the agent can learn the optimal strategy.

[0025] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a water-cooled fuel cell temperature control method based on incentive learning. The method utilizes the interactive learning between the intelligent agent and the environment to automatically adjust the circulating water pump flow and the cooling fan speed of the fuel cell system in a way that maximizes the cumulative reward, so as to achieve precise temperature control. Through the training and optimization of the incentive learning algorithm, the method can adapt to different operating environments and working conditions and obtain precise temperature control effects. At the same time, the automated control process implemented by programming is simple and efficient, which is conducive to combining with specific engineering applications to solve the temperature control problem of water-cooled fuel cells in engineering applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A fuel cell temperature control Markov decision model constructed for an embodiment of the present invention;

[0027] Figure 2 A flow chart of the Q-Learning algorithm used in the embodiment of the present invention;

[0028] Figure 3 A controller for the Q-Learning algorithm in an embodiment of the present invention controls the training process;

[0029] Figure 4 It is a schematic structural diagram of a water-cooled fuel cell temperature control system based on an embodiment of the present invention; DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0031] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0032] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0033] Based on Figure 1-4 The structural schematic diagram of the water-cooled fuel cell temperature control system is shown. This embodiment proposes a water-cooled fuel cell temperature control method based on incentive learning.

[0034] The water-cooled fuel cell temperature control system includes a temperature sensor, a fuel cell stack, a water tank, a circulating water pump, a radiator, a cooling fan, and a Q-learning controller.

[0035] 1) Obtain the fuel cell inlet and outlet water temperature data. The fuel cell inlet water temperature T in And outlet water temperature T out The data is measured by sensors in real time.

[0036] 2) Use the incentive learning algorithm to establish a state-action-reward model, and use the acquired fuel cell inlet and outlet water temperature data to construct the state set. Then define the action set as adjusting the flow of cooling water and adjusting the speed of the cooling fan. Finally, define the reward function to evaluate the pros and cons of each action in a specific state.

[0037] Figure 1 In a specific case of this embodiment, the selected incentive learning algorithm is the Q-learning algorithm, which is constructed based on the Markov decision model. Figure 1 State-Action-Reward Model, the Q-learning controller uses the ε-greedy strategy to specify the action to be performed through the temperature T provided by the fuel cell model, and finally receives a reward R based on the quality of the action performed.

[0038] The Q-learning algorithm is a value-based agent algorithm in reinforcement learning. It is also one of the tabular algorithms and is used to solve MDP problems with a clear reward mechanism. Figure 2 This is a flowchart of the Q-learning algorithm. In the Q-learning algorithm, the behavior adopts the ε-greedy strategy, also known as the ε-greedy strategy. This strategy is gradually improved based on the Q value table and has two parts, namely exploration and utilization. The ε-greedy process of this embodiment is as follows:

[0039] ① Initialize the value of ε to 0.1;

[0040] ② At each decision point, generate a random number r between 0 and 1;

[0041] ③If r<ε (i.e. r<0.1), then explore and choose a random action;

[0042] ④ If r ≥ ε (i.e. r ≥ 0.1), then take advantage and choose the currently considered best action. This means that in most cases, choose the currently known operation or strategy that can obtain the maximum return.

[0043] The Q value table mentioned above is shown in the following table:

[0044] Q value table

[0045]

[0046]

[0047] The rows represent all states observed from the environment model, the columns represent different actions, and the corresponding data in the table are Q values. The Q values ​​corresponding to different state actions represent different strategies. As the number of iterations increases, the table will gradually improve. The Q value is updated as follows:

[0048] Q(S,Act)←Q(S,Act)+α[R+γmax Act Q(S′,Act′)-Q(S,Act)]

[0049] In the formula, Q(S, Act) represents the Q value corresponding to the current state and the actual action taken; Q(S′, Act′), S′, Act′ represent the Q value, S value, and Act value corresponding to the state and the action taken at the next moment, respectively; R represents the reward value; α represents the learning rate; and γ is the discount factor. As can be seen from the formula, the better the action, the higher the Q value.

[0050] 3) At each time step, the trained model is used to select the best action for fuel cell temperature control based on the current state of the fuel cell. The model predicts the expected reward for each action based on the current state and selects the action with the highest expected reward as the current control method.

[0051] Figure 3 The controller of the Q-learning algorithm controls the training process, wherein the training parameter setting part includes initializing the state space, action space, reward function and parameters required by the Q-learning algorithm; when establishing the Q table, the initial row only has the initial value of the state, and the new state value is obtained after each subsequent step and then added to the Q table in real time; the maximum number of rounds is set according to the complexity of the task. In this embodiment, the maximum number of rounds is set to 200. A larger number of rounds can improve the convergence, stability and learning effect of the Q-learning algorithm, but it will increase the time overhead and computing resource requirements; the entire control task is decomposed into two subtasks: the action sets of task 1 and task 2 are respectively:

[0052] Task 1:

[0053] Act = {Fan off, fan position 1, fan position 2, fan position 3, fan position 4, fan position 5}

[0054] Task 2:

[0055] Act={Water pump gear 1, water pump gear 2, water pump gear 3, water pump gear 4, water pump gear 5}

[0056] In the formula, the gear position is a different duty cycle, and the duty cycle increases proportionally from 1st gear to 5th gear to 100%. Task 1 is to judge the current inlet water temperature, and the intelligent agent autonomously judges to select an optimal action from the action set of Task 1 to execute; Task 2 is to judge the current inlet and outlet water temperature difference, and the intelligent agent autonomously judges to select an optimal action from the action set of Task 2 to execute. Both subtasks belong to the reinforcement learning control process, and the selection of required parameters is controlled by the Q-learning algorithm. The advantage of this is that each subtask can be optimized as an independent individual, and the reward of each subtask is clearer. The reward function of Task 1 and the reward function of Task 2 are:

[0057] R1=100-50|e1| 2 -0.05D1

[0058] R2=100-50|e2| 2 -0.05D2

[0059] The first item 100 in the formula is the basic reward value, the second item e is the difference between the current temperature value after control or the temperature difference and the set value. In this item, the smaller the difference between the current temperature value after control and the temperature set value, the greater the reward. The third item D is related to the selected gear, and its value is the number of gears selected. This item is set to minimize the control cost.

[0060] If all states and behaviors satisfy the formula:

[0061] |Q(S,Act)-Q(S′,Act′)∣<ξ

[0062] Then this round ends (where ξ is a tiny amount, which means that the state and behavior change very little before and after). After continuous cyclic training, the agent can learn the optimal strategy from the environment and make the best decision when facing a new state to maximize the cumulative reward.

[0063] 4) At each time step, according to the current state of the fuel cell, the trained controller (agent) is used to select the best action for fuel cell temperature control. The controller selects the best action corresponding to the current state according to the state-action value function to obtain the maximum reward value;

[0064] Figure 4 This is a simplified system control structure diagram in the embodiment. The temperature sensor collects the fuel cell inlet water temperature T in The fuel cell outlet water temperature T outAfter receiving the temperature parameters, the parameters are transmitted to the incentive module and the trained Q-learning controller. The incentive module outputs the reward R to the Q-learning control according to the reward function. The Q-learning controller performs autonomous control based on the received temperature parameters and performs optimal control on Task 1 and Task 2 according to the principle of maximizing accumulated rewards, so that the inlet water temperature and the inlet and outlet water temperature difference of the system meet the required temperature control effect.

[0065] 5) According to the selected optimal action, the corresponding control method is executed to adjust the temperature of the fuel cell. For example, if the model chooses to increase the cooling efficiency of the water cooling system to reduce the temperature, the action of increasing the gear of the circulating water pump or increasing the gear of the cooling fan will be executed to achieve this;

[0066] 6) After executing the control method, the state-action-reward model is updated according to the actual control effect, and the difference between the actual reward and the expected reward stored in the controller is compared. The model can be fed back and adjusted to continuously optimize the performance of the control method, and the incentive learning algorithm is used to update the parameters of the model to better adapt it to the actual situation.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the embodiments, a person skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A water-cooled fuel cell temperature control method based on incentive learning, characterized in that: The following steps are involved: Step 1: Obtain the fuel cell inlet water temperature T in And outlet water temperature T out data; Step 2: Establish a state-action-reward model, use the obtained fuel cell inlet and outlet water temperature data to construct the state set, then define the action set, and finally define the reward function R to evaluate the pros and cons of each action in a specific state; Step 3: Based on the state-action-reward model, the controller is trained to stimulate the learning algorithm so that it can generate decision-making strategies and gradually improve its performance. During the training process, the controller learns the best actions to take in different states through interaction with the environment, evaluates the quality of each action through reward signals, and adjusts the state-action value function according to the reward signals. With continuous training and optimization, the controller gradually learns to make the best decision in a given environment to obtain the maximum cumulative reward. Step 4: At each time step, the best action for fuel cell temperature control is selected using the trained controller according to the current state of the fuel cell. The controller selects the best action corresponding to the current state according to the state-action value function to obtain the maximum reward value. Step 5: According to the selected optimal action, executing a corresponding control method to adjust the temperature of the fuel cell; Step 6: After executing the control method, update the state-action-reward model according to the actual control effect, compare the difference between the actual reward and the expected reward stored in the controller, feedback and adjust the model to continuously optimize the performance of the control method, and use the incentive learning algorithm to update the parameters of the model to better adapt it to the actual situation.

2. A water-cooled fuel cell temperature control method based on incentive learning according to claim 1, characterized in that: The state-action-reward model in step 2 is a Markov decision model MDP, which is expressed as: MDP={S,Act,R,P,γ}.

3. The water-cooled fuel cell temperature control method based on incentive learning according to claim 2 is characterized in that: In the Markov decision model expression, S is the state set, Act is the action set, R is the reward function, P is the state transition probability, and γ is the discount factor. In the model-free algorithm, P is difficult to know in advance, and γ is determined by the environment. The agent needs to learn through interaction with the environment.

4. The water-cooled fuel cell temperature control method based on incentive learning according to claim 1, characterized in that: The reward function R in step 2 is a function used to evaluate the behavior of the agent in reinforcement learning. The reward function R gives a numerical value as feedback based on the action taken by the agent in the environment and the state of the environment to guide the learning process of the agent. It is usually expressed as R(s,a), where s represents the state of the environment and a represents the action taken by the agent.

5. The water-cooled fuel cell temperature control method based on incentive learning according to claim 3 is characterized in that: The intelligent agent is an individual that participates in the learning and decision-making process, obtains incentive signals through interaction with the environment, and makes decisions and takes actions based on these signals.

6. The water-cooled fuel cell temperature control method based on incentive learning according to claim 1, characterized in that: The interaction between the controller and the environment in step 3 refers to the reward value obtained by the agent during the interaction with the environment. In incentive learning, the agent learns the optimal behavior strategy through interaction with the environment. After the agent performs an action, the environment will give a reward or punishment to indicate the quality of the action. The goal of the agent is to maximize the cumulative reward value and gradually learn the optimal behavior strategy by trying different actions and observing the feedback from the environment.

7. The water-cooled fuel cell temperature control method based on incentive learning according to claim 1, characterized in that: The state-action value function in step 3 is also called the Q-function, which is a function used in reinforcement learning to evaluate the value of performing a specific action in a given state. The Q-function represents the cumulative reward that can be obtained by taking a certain action in a certain state, and the expression is: Q(S,Act)←Q(S,Act)+α[R+γmax Act Q(S′,Act′)-Q(S,Act)]。 8. The water-cooled fuel cell temperature control method based on incentive learning according to claim 7, characterized in that: In the Q function expression, Q(S, Act) represents the Q value corresponding to the current state and the actual action taken; Q(S′, Act′), S′, Act′ respectively represent the Q value, S value and Act value corresponding to the state and the action taken at the next moment; R represents the reward value; α represents the learning rate; γ is the discount factor; it can be seen from the formula that the better the action, the higher the Q value.

9. The water-cooled fuel cell temperature control method based on incentive learning according to claim 1, characterized in that: The fuel cell temperature is adjusted in step 5 to adjust the fuel cell inlet water temperature to 338.15K, and the difference between the outlet water temperature and the inlet water temperature is 283.15K.

10. The water-cooled fuel cell temperature control method based on incentive learning according to claim 1, characterized in that: The incentive learning algorithm in step 6 aims to adjust the parameters of the model according to the reward signal obtained through interaction with the environment so that the agent can learn the optimal strategy.

Citation Information

Patent Citations

  • Battery control system, control method thereof and battery control equipment

    CN112259764A

  • Metro air conditioning system energy-saving control method based on ActorCritic algorithm

    CN113901713A

  • Micro-fluidic chip heat dissipation method and system based on reinforcement learning algorithm

    CN116595879A

  • Energy-saving optimization control method of dehumidification unit system based on reinforcement learning

    CN117190405A

  • Refrigeration station edge control method based on reinforcement learning

    CN117663553A

Cited By

  • Intelligent flow battery control model construction method, control method and device

    CN119812398A

  • Automatic temperature control heating method and system

    CN119960523A

  • Fuel cell adaptive optimization calculation method and system based on deep learning

    CN122338103A