Water electrolysis efficiency dynamic scheduling method and system based on reinforcement learning

Abstract

The invention relates to the field of information automation management, and particularly discloses a water electrolysis efficiency dynamic scheduling method and system based on reinforcement learning. According to the method, a state space is constructed through multi-parameter dynamic perception, an improved depth deterministic strategy gradient (DDPG) algorithm is used for learning, a strategy is optimized by means of a self-adaptive exploration strategy and a multi-time scale learning mechanism, model migration and rapid starting are achieved, real-time feedback and closed-loop control are constructed, the water electrolysis efficiency is improved, and energy consumption is reduced. The method comprises a reward function and an exploration rate adjustment formula, so that the strategy optimization direction is accurate, and the exploration process is reasonable. The system comprises a data acquisition module, a processing module, a reinforcement learning module, a control execution module and a feedback monitoring module, and all the modules work cooperatively to improve the system performance.

Description

Technical Field

[0001] The present invention relates to the technical field of hydrogen production by electrolysis of water, and in particular to a method and system for dynamic scheduling of water electrolysis efficiency based on reinforcement learning. Background Art

[0002] With the growing global demand for clean energy, water electrolysis has attracted widespread attention as a sustainable hydrogen production method. However, current water electrolysis technology still faces numerous challenges in terms of efficiency and energy consumption, limiting its large-scale commercial application. Difficulty in optimizing operating parameters: The efficiency of the water electrolysis process is affected by multiple operating parameters, such as electrolysis voltage, current density, electrolyte temperature, and electrolyte concentration. These parameters are interdependent and mutually constrained, making it difficult to accurately optimize them using traditional empirical or fixed control strategies. In actual operation, it is often impossible to find the optimal parameter combination based on real-time operating conditions, resulting in low water electrolysis efficiency and high energy consumption. Inadequate dynamic response: Actual water electrolysis hydrogen production systems are subject to interference from various external factors, such as fluctuations in power supply and changes in feedstock availability. Existing systems lack a rapid and effective response mechanism to these dynamic changes. For example, when the power supply fluctuates, the operating parameters of the water electrolysis process cannot be adjusted in a timely manner to adapt to the power fluctuations and maintain high hydrogen production efficiency, which can lead to interruptions in the hydrogen production process or a significant decrease in efficiency. Summary of the Invention

[0003] The present invention provides a method for dynamic scheduling of water electrolysis efficiency based on reinforcement learning, comprising the following steps:

[0004] By deploying multiple sensors in the water electrolysis hydrogen production system, real-time data on electrolysis voltage, current density, electrolyte temperature, electrolyte concentration, hydrogen production, oxygen production, and power supply fluctuations are collected;

[0005] The collected data are fused and processed to construct the state space of the water electrolysis hydrogen production system. Data preprocessing methods such as normalization are used to make data of different dimensions comparable.

[0006] Dynamically optimize the operating parameters of water electrolysis using an improved deep deterministic policy gradient (DDPG) algorithm, wherein the improved DDPG algorithm includes an optimized actor network and critic network structure and an improved reward function;

[0007] Adopting an adaptive exploration strategy to balance exploration and exploitation in the reinforcement learning process;

[0008] Design a multi-timescale learning mechanism to adapt to the temporal characteristics of different parameter changes in the water electrolysis hydrogen production system;

[0009] Use model transfer technology to apply the trained model parameters to the new water electrolysis hydrogen production system and perform fine-tuning;

[0010] A real-time feedback and closed-loop control system is established, the actions output by the reinforcement learning algorithm are applied to the water electrolysis hydrogen production system, and the system feedback data is re-input into the reinforcement learning algorithm.

[0011] Furthermore, in the state space construction, the collected data are normalized using the following formula:

[0012] Among them, x is the original data, xmin is the minimum value of the data type, xmax is the maximum value of the data type, and xnorm is the normalized data.

[0013] Furthermore, in the improved DDPG algorithm, the Actor network is used to generate action strategies, the Critic network is used to evaluate action values, and the improved reward function R is calculated as follows:

[0014] R=w1×ΔE+w2×(-ΔP)+w3×Swhere Δ E is the improvement ratio of water electrolysis efficiency, ΔP is the energy consumption change ratio, S is the system stability index, w1, w2, w3 are weight coefficients, and w1+w2+w3=1.

[0015] Furthermore, in the adaptive exploration strategy, the exploration rate epsilon is dynamically adjusted according to the following rules: at the beginning of training, a higher initial exploration rate epsilon0 is set; as the number of training steps t increases, the exploration rate epsilon gradually decreases. When the average reward AR obtained by the agent in n consecutive steps is greater than the preset threshold TR, the exploration rate epsilon is decayed according to the formula epsilon = epsilontimes*alpha (0<alpha<1); when the average reward AR of the agent in m consecutive steps is less than the preset threshold TRlow, the exploration rate epsilon is increased according to the formula epsilon = epsilontimes*beta (beta>1).

[0016] Furthermore, in the multi-time-scale learning mechanism, for short-term changing parameters, the learning rate etas is large; for medium-term changing parameters, the learning rate etam is moderate; for long-term changing parameters, the learning rate etal is small, and etas>etam>etal is satisfied.

[0017] Furthermore, in the model migration technology, the model parameters trained on multiple water electrolysis hydrogen production systems of different scales and operating conditions are used as the initial values of the new model. A small amount of operating data is collected on the new system, and by fine-tuning some of the model parameters, the model can quickly adapt to the new system environment.

[0018] Furthermore, in the real-time feedback and closed-loop control system, sensors collect operating status data of the water electrolysis hydrogen production system in real time, input the data into a reinforcement learning algorithm, and the algorithm outputs optimized operating parameters based on the current status. The operating parameters are converted into control instructions through the controller and sent to the actuator. The actuator adjusts the operation of the water electrolysis hydrogen production system, forming a continuous feedback optimization cycle.

[0019] Furthermore, the multiple sensors include voltage sensors and current sensors installed at the electrodes of the electrolytic cell, temperature sensors and concentration sensors on the electrolyte storage tank and circulation pipeline, gas flow sensors on the hydrogen and oxygen output pipelines, and connected power monitoring equipment.

[0020] Furthermore, the actuator includes an electric regulating valve, a controller of a heating or cooling device, and a power controller, which are respectively used to adjust the electrolyte flow, electrolyte temperature, and electrolysis voltage and current density.

[0021] Furthermore, this method can be applied to water electrolysis hydrogen production systems of different scales and operating conditions to improve water electrolysis efficiency, reduce energy consumption, and enhance the stability and adaptability of the system in dynamic environments.

[0022] Another aspect of the present invention provides a dynamic scheduling system for water electrolysis efficiency based on reinforcement learning, comprising:

[0023] The data acquisition module consists of multiple sensors deployed in the water electrolysis hydrogen production system, which is used to collect real-time data on electrolysis voltage, current density, electrolyte temperature, electrolyte concentration, hydrogen production, oxygen production, and power supply fluctuations;

[0024] The data processing and state space construction module is used to fuse the collected data, construct the state space of the water electrolysis hydrogen production system, and use data preprocessing methods such as normalization to make data of different dimensions comparable;

[0025] A reinforcement learning optimization module uses an improved deep deterministic policy gradient (DDPG) algorithm to dynamically optimize the operating parameters of water electrolysis. The improved DDPG algorithm includes an optimized actor network and critic network structure and an improved reward function;

[0026] Exploration strategy module, which implements adaptive exploration strategy to balance exploration and exploitation in the reinforcement learning process;

[0027] Multi-time scale learning module, which designs a multi-time scale learning mechanism to adapt to the temporal characteristics of different parameter changes in the water electrolysis hydrogen production system;

[0028] The model migration module uses model migration technology to apply the trained model parameters to the new water electrolysis hydrogen production system and perform fine-tuning;

[0029] The real-time feedback and closed-loop control module applies the actions output by the reinforcement learning algorithm to the water electrolysis hydrogen production system, and re-inputs the system feedback data into the reinforcement learning algorithm to form a closed-loop control;

[0030] A controller, which converts the optimized operating parameters output by the reinforcement learning algorithm into control instructions;

[0031] The actuator, including the electric regulating valve, the controller of the heating or cooling device and the power controller, adjusts the operating parameters of the water electrolysis hydrogen production system according to the control instructions.

[0032] Beneficial effects

[0033] Breaking through the traditional monitoring model of a single or a few parameters, this system comprehensively collects multi-parameter data from water electrolysis hydrogen production systems, constructing a state space containing a variety of key information. Through data fusion and preprocessing, it provides richer and more accurate input for reinforcement learning algorithms, enabling them to more comprehensively understand the system state and make more optimized decisions.

[0034] Innovative improvements have been made to the Deep Deterministic Policy Gradient (DDPG) algorithm, optimizing the Actor and Critic network structures to enhance the network's learning capabilities. Furthermore, a carefully designed reward function, closely centered around core metrics such as water electrolysis efficiency, energy consumption, and system stability, guides the agent to learn an optimal strategy that better meets actual needs, effectively improving water electrolysis efficiency and reducing energy consumption.

[0035] The proposed adaptive exploration strategy dynamically adjusts the exploration rate based on the reinforcement learning training progress and environmental changes. This strategy prevents the agent from being overly blind or overly conservative during exploration, improving the algorithm's convergence speed and learning efficiency, enabling the agent to more quickly find the optimal strategy and adapt to different water electrolysis hydrogen production system operating conditions.

[0036] The multi-timescale learning mechanism fully considers the temporal characteristics of parameter changes in the water electrolysis hydrogen production system. Different learning rates are used for short-term, medium-term, and long-term changes, enabling the reinforcement learning algorithm to more accurately capture system dynamics, adjust strategies in a timely manner, and improve the system's adaptability and stability in complex dynamic environments.

[0037] Using model transfer technology, we can apply the experience and strategies gained from training on multiple systems to new systems. By initializing the new model parameters and performing minor fine-tuning, we can quickly start the model on the new system, significantly reducing the training time and cost, and improving the algorithm's versatility and practicality.

[0038] By building a real-time feedback and closed-loop control system, we achieve a close integration of the reinforcement learning algorithm with the water electrolysis hydrogen production system. Through real-time monitoring and feedback, we continuously optimize operating parameters, forming a continuous improvement cycle to ensure that the system always operates in an optimal state, further improving water electrolysis efficiency and system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 System operation principle flow chart;

[0040] Figure 2 The overall architecture of the water electrolysis hydrogen production system;

[0041] Figure 3 Improved DDPG algorithm structure diagram;

[0042] Figure 4 Closed-loop control system workflow diagram. DETAILED DESCRIPTION

[0043] Example 1

[0044] The main purpose of this invention is to provide a dynamic scheduling method for water electrolysis efficiency based on reinforcement learning. Through an innovative algorithm model, it can achieve dynamic optimization of water electrolysis operating parameters. It can sense the operating status of the system and changes in the external environment in real time, and automatically adjust parameters such as electrolysis voltage, current density, electrolyte temperature and concentration to improve water electrolysis efficiency, reduce energy consumption, and enhance the stability and adaptability of the system in dynamic environments.

[0045] A method for dynamic scheduling of water electrolysis efficiency based on reinforcement learning, comprising the following steps:

[0046] By deploying multiple sensors in the water electrolysis hydrogen production system, real-time data on electrolysis voltage, current density, electrolyte temperature, electrolyte concentration, hydrogen production, oxygen production, and power supply fluctuations are collected;

[0047] The collected data are fused and processed to construct the state space of the water electrolysis hydrogen production system. Data preprocessing methods such as normalization are used to make data of different dimensions comparable.

[0048] Dynamically optimize the operating parameters of water electrolysis using an improved deep deterministic policy gradient (DDPG) algorithm, wherein the improved DDPG algorithm includes an optimized actor network and critic network structure and an improved reward function;

[0049] Adopting an adaptive exploration strategy to balance exploration and exploitation in the reinforcement learning process;

[0050] Design a multi-timescale learning mechanism to adapt to the temporal characteristics of different parameter changes in the water electrolysis hydrogen production system;

[0051] Use model transfer technology to apply the trained model parameters to the new water electrolysis hydrogen production system and perform fine-tuning;

[0052] A real-time feedback and closed-loop control system is established, the actions output by the reinforcement learning algorithm are applied to the water electrolysis hydrogen production system, and the system feedback data is re-input into the reinforcement learning algorithm.

[0053] In the state space construction, the collected data are normalized by the following formula:

[0054] Among them, x is the original data, xmin is the minimum value of the data type, xmax is the maximum value of the data type, and xnorm is the normalized data.

[0055] In the improved DDPG algorithm, the Actor network is used to generate action strategies, the Critic network is used to evaluate action values, and the improved reward function R is calculated as follows:

[0056] R=w1×ΔE+w2×(-ΔP)+w3×Swhere Δ E is the ratio of water electrolysis efficiency improvement, Δ P is the energy consumption change ratio, S is the system stability index, w1, w2, w3 are weight coefficients, and w1+w2+w3=1.

[0057] In the adaptive exploration strategy, the exploration rate epsilon is dynamically adjusted according to the following rules: at the beginning of training, a higher initial exploration rate epsilon0 is set; as the number of training steps t increases, the exploration rate epsilon gradually decreases. When the average reward AR obtained by the agent in n consecutive steps is greater than the preset threshold TR, the exploration rate epsilon is decayed according to the formula epsilon = epsilon times * alpha (0 < alpha < 1); when the average reward AR of the agent in m consecutive steps is less than the preset threshold TRlow, the exploration rate epsilon is increased according to the formula epsilon = epsilon times * beta (beta > 1).

[0058] In the multi-time scale learning mechanism, for short-term changing parameters, the learning rate etas is large; for medium-term changing parameters, the learning rate etam is moderate; for long-term changing parameters, the learning rate etal is small and full.

[0059] etas > etam > etal.

[0060] In the model transfer technology, the model parameters trained on electrolytic water hydrogen production systems of multiple different scales and operating conditions are used as the initial values of the new model. A small amount of operating data is collected on the new system, and by fine-tuning some parameters of the model, the model can quickly adapt to the new system environment.

[0061] In the real-time feedback and closed-loop control system, sensors collect the operating state data of the electrolytic water hydrogen production system in real time, input the data into the reinforcement learning algorithm, and the algorithm outputs optimized operating parameters according to the current state. Through the controller, the operating parameters are converted into control instructions and sent to the actuator, and the actuator adjusts the operation of the electrolytic water hydrogen production system to form a continuous feedback optimization cycle.

[0062] The multiple sensors include voltage sensors and current sensors installed at the electrodes of the electrolyzer, temperature sensors and concentration sensors on the electrolyte storage tank and circulation pipeline, gas flow sensors on the hydrogen and oxygen output pipelines, and connected power monitoring equipment.

[0063] The actuator includes an electric control valve, a controller for heating or cooling devices, and a power supply controller, which are respectively used to adjust the electrolyte flow rate, electrolyte temperature, and electrolysis voltage and current density.

[0064] This method can be applied to electrolytic water hydrogen production systems of different scales and operating conditions to improve the electrolytic water efficiency, reduce energy consumption, and enhance the stability and adaptability of the system in a dynamic environment.

[0065] Embodiment 2

[0066] On the other hand, the present invention also provides a dynamic scheduling system for electrolytic water efficiency based on reinforcement learning, including:

[0067] A data acquisition module, composed of multiple sensors deployed in the electrolytic water hydrogen production system, is used to collect electrolysis voltage, current density, electrolyte temperature, electrolyte concentration, hydrogen production, oxygen production, and power supply fluctuation data in real time;

[0068] A data processing and state space construction module is used to fuse and process the collected data, construct the state space of the electrolytic water hydrogen production system, and make the data of different dimensions comparable by using data preprocessing methods such as normalization;

[0069] A reinforcement learning optimization module uses an improved deep deterministic policy gradient (DDPG) algorithm to dynamically optimize the electrolytic water operating parameters. The improved DDPG algorithm includes an optimized Actor network and Critic network structure and an improved reward function;

[0070] Exploration strategy module, which implements adaptive exploration strategy to balance exploration and exploitation in the reinforcement learning process;

[0071] Multi-time scale learning module, which designs a multi-time scale learning mechanism to adapt to the temporal characteristics of different parameter changes in the water electrolysis hydrogen production system;

[0072] The model migration module uses model migration technology to apply the trained model parameters to the new water electrolysis hydrogen production system and perform fine-tuning;

[0073] The real-time feedback and closed-loop control module applies the actions output by the reinforcement learning algorithm to the water electrolysis hydrogen production system, and re-inputs the system feedback data into the reinforcement learning algorithm to form a closed-loop control;

[0074] A controller, which converts the optimized operating parameters output by the reinforcement learning algorithm into control instructions;

[0075] The actuator, including the electric regulating valve, the controller of the heating or cooling device and the power controller, adjusts the operating parameters of the water electrolysis hydrogen production system according to the control instructions.

[0076] Multi-parameter dynamic sensing and state space construction: Multiple sensors are deployed in the water electrolysis hydrogen production system to collect real-time data such as electrolysis voltage, current density, electrolyte temperature, electrolyte concentration, hydrogen production, oxygen production, and power supply fluctuations. This data is integrated and processed to construct the state space of the water electrolysis hydrogen production system. Data preprocessing methods such as normalization are used to make data of different dimensions comparable, facilitating subsequent processing by the reinforcement learning algorithm.

[0077] Improved reinforcement learning algorithm model based on Deep Deterministic Policy Gradient (DDPG): This paper introduces an improved Deep Deterministic Policy Gradient (DDPG) algorithm. Based on the traditional DDPG algorithm, the structures of the Actor network and Critic network are optimized. The number of layers and neurons in the network is increased to improve the network's fitting ability, enabling it to better learn the complex state-action mapping relationship of the water electrolysis system. At the same time, the algorithm's reward function design is improved, with improved water electrolysis efficiency, reduced energy consumption, and system stability as the primary metrics for rewards. For example, positive rewards are given when water electrolysis efficiency increases by a certain percentage; corresponding rewards are also given when energy consumption decreases or the system maintains stable operation in the face of external interference, while penalties are given when these decreases.

[0078] Adaptive Exploration Strategy: This paper proposes an adaptive exploration strategy to balance exploration and exploitation during reinforcement learning. Initially, a high exploration rate is used to enable the agent to extensively explore diverse action spaces and quickly discover potentially effective strategies. As training progresses, the exploration rate is dynamically adjusted based on the agent's learning progress and environmental changes. When the agent achieves good results with its current strategy, the exploration rate is gradually reduced to further exploit the learned strategy. When the agent encounters performance bottlenecks or experiences significant environmental changes, the exploration rate is appropriately increased to search for a more optimal strategy.

[0079] Multi-timescale learning mechanism: A multi-timescale learning mechanism is designed to adapt to the temporal characteristics of different parameter changes in the water electrolysis hydrogen production system. Parameter changes during the water electrolysis process are divided into short-term, medium-term, and long-term changes. For short-term changes, such as momentary fluctuations in power supply, a fast-response learning strategy is adopted to adjust actions in a timely manner. For medium-term changes, such as slow changes in electrolyte concentration over time, a moderate learning rate is adopted. For long-term changes, such as performance changes caused by equipment aging, a slower learning rate is adopted. This multi-timescale learning mechanism enables the reinforcement learning algorithm to more effectively learn optimal strategies at different time scales.

[0080] Model migration and rapid startup: Reinforcement learning models are trained on multiple water electrolysis hydrogen production systems of varying scales and operating conditions, accumulating extensive experience and strategies. When applied to a new water electrolysis hydrogen production system, model migration technology is employed, using the trained model parameters as the initial values for the new model. A small amount of training and fine-tuning is then performed on the new system, enabling rapid startup and adaptation to the new system environment, reducing training time and costs.

[0081] Real-time feedback and closed-loop control: A real-time feedback and closed-loop control system is established, applying the actions output by the reinforcement learning algorithm (i.e., optimized operating parameters) to the water electrolysis hydrogen production system. Sensors monitor the system's operating status and performance indicators in real time, and this feedback data is fed back into the reinforcement learning algorithm to form a closed-loop control system. This continuously optimizes operating parameters and improves water electrolysis efficiency, ensuring the system maintains optimal operating conditions in dynamic environments.

[0082] The reinforcement learning-based dynamic scheduling method for water electrolysis efficiency works as follows: First, a multi-parameter dynamic perception module collects real-time operating data from the water electrolysis hydrogen production system through sensors, constructs a state space, performs preprocessing, and then inputs it into the improved DDPG algorithm model. Based on the current state, the agent selects actions (i.e., adjusts the operating parameters of the water electrolysis system) under the guidance of an adaptive exploration strategy. These actions act on the water electrolysis hydrogen production system, changing the system's operating state and generating new states and rewards. The rewards are calculated based on a modified reward function that reflects changes in water electrolysis efficiency, energy consumption, and system stability.

[0083] The multi-timescale learning mechanism adjusts the agent's learning rate based on the temporal characteristics of different parameter changes, enabling the agent to more effectively learn optimal strategies at different timescales. The agent uses new states and rewards to update the parameters of the actor and critic networks, continuously optimizing its strategy. When applied to a new water electrolysis hydrogen production system, the model migration module migrates pre-trained model parameters, performing minor fine-tuning for rapid startup. The real-time feedback and closed-loop control module continuously feeds the system's real-time operating status back to the reinforcement learning algorithm, forming a closed-loop control system that continuously optimizes the operating parameters and improves water electrolysis efficiency.

[0084] Example 3

[0085] (1) Hardware deployment

[0086] Sensor Installation: High-precision voltage and current sensors are installed at the electrodes of the electrolyzer to accurately measure the electrolysis voltage and current density. Temperature and concentration sensors are installed on the electrolyte storage tank and circulation pipelines to monitor the electrolyte temperature and concentration in real time. Gas flow sensors are installed on the hydrogen and oxygen output pipelines to measure hydrogen and oxygen production. Furthermore, power monitoring equipment is connected to obtain data on power supply fluctuations. Ensure that the sensors are installed in appropriate locations to accurately collect data and do not affect the normal operation of the water electrolysis hydrogen production system.

[0087] Connecting the controller to the actuator: Select a reliable controller, such as a programmable logic controller (PLC) or industrial computer. Connect the controller to the sensor to obtain real-time sensor data. Also, connect the controller to the actuator, such as an electric control valve to adjust electrolyte flow and concentration, a heating or cooling device controller to adjust electrolyte temperature, and a power supply controller to adjust electrolysis voltage and current density. Ensure that the controller can accurately receive sensor data and precisely control the actuator's movements based on the control instructions output by the algorithm.

[0088] (2) Software Programming

[0089] Data acquisition and preprocessing program development: Use Python or C++ to write a data acquisition program to implement real-time sensor data acquisition, calibration, and filtering. Use data normalization algorithms, such as min-max normalization or Z-score normalization, to map data from different ranges to a unified interval for easier processing by reinforcement learning algorithms. Write a data storage program to store collected data in a database for subsequent analysis and model training.

[0090] Reinforcement Learning Algorithm Implementation: Implement a modified DDPG algorithm based on a deep learning framework such as TensorFlow or PyTorch. Write the code for the Actor and Critic networks, initializing parameters based on the designed network structure. Implement the code logic for the adaptive exploration strategy, dynamically adjusting the exploration rate based on training progress and environmental changes. Write the code for a multi-timescale learning mechanism, adjusting the learning rate based on the temporal characteristics of different parameter changes.

[0091] Model training and optimization: We collect extensive operational data from water electrolysis hydrogen production systems, including parameter changes under different operating conditions and corresponding hydrogen production efficiency and energy consumption data. We use this data to train a reinforcement learning model, optimizing the parameters of the actor and critic networks through continuous iteration. During training, we adjust algorithm hyperparameters, such as the learning rate and discount factor, based on model performance metrics, such as improvements in water electrolysis efficiency and reductions in energy consumption, to improve model performance.

[0092] Real-time control program development: This involves writing a real-time control program to convert the output of the reinforcement learning algorithm (i.e., optimized operating parameters) into control commands that the controller can recognize. This program implements real-time feedback and closed-loop control logic, re-inputting real-time data collected by sensors into the reinforcement learning algorithm to continuously update the model and optimize the control strategy. A program is also written to communicate with the controller to ensure that control commands are accurately transmitted to the controller.

[0093] Model Migration and Application Development: Develop a model migration program to migrate trained model parameters to the new water electrolysis hydrogen production system. Write a program to fine-tune the model on the new system, optimizing the migrated model based on the new system's characteristics and a small amount of operational data. Develop a user interface program to facilitate operators monitoring the water electrolysis hydrogen production system's operating status, viewing model training and optimization results, and configuring necessary parameters.

[0094] (3) System testing and optimization

[0095] Simulated Environment Testing: A test platform simulating a water electrolysis hydrogen production system was built and tested under various operating conditions, such as power supply fluctuations, electrolyte concentration variations, and temperature changes. This validated the multi-parameter dynamic perception module's ability to accurately collect and process data, the reinforcement learning algorithm's ability to correctly learn the optimal strategy, and the effectiveness of the adaptive exploration strategy, multi-timescale learning mechanism, and model transfer technology. Through simulation testing, data was collected and analyzed for model performance metrics, such as electrolysis efficiency improvement, energy consumption reduction, and convergence speed, to optimize the algorithm's parameters and structure.

[0096] Actual system testing: The reinforcement learning-based dynamic water electrolysis efficiency scheduling system was deployed and tested on an actual water electrolysis hydrogen production device. During operation, the system's operating status and performance indicators were monitored and compared with traditional control methods. The system's responsiveness and stability in the face of external disturbances such as power fluctuations and changes in raw material supply were observed. Based on the actual test results, the system's hardware configuration and software algorithms were further optimized to ensure stable and efficient operation, achieving the goals of improving water electrolysis efficiency and reducing energy consumption.

[0097] Beneficial effects

[0098] Improve water electrolysis efficiency: By optimizing the operating parameters of water electrolysis, the water electrolysis hydrogen production system can maintain a high water electrolysis efficiency under different operating conditions. Compared with traditional control methods, it can significantly increase hydrogen production and meet the growing demand for clean energy.

[0099] Reduced energy consumption: Precise parameter control effectively reduces energy consumption in the water electrolysis process, reduces energy waste, reduces hydrogen production costs, improves the economic feasibility of water electrolysis hydrogen production technology, and helps promote its large-scale commercial application.

[0100] Enhanced system adaptability: Adaptive exploration strategies, multi-time-scale learning mechanisms, and real-time feedback and closed-loop control systems enable the water electrolysis hydrogen production system to quickly adapt to changes in the external environment, such as power supply fluctuations and raw material supply changes, thereby improving the stability and reliability of the system and reducing hydrogen production interruptions or efficiency reductions caused by external interference.

[0101] Shorten the model application cycle: Model migration and rapid startup technology greatly shortens the training time and cost of reinforcement learning models on new systems, enabling the technology to be more quickly applied to water electrolysis hydrogen production systems of different scales and working conditions, accelerating the promotion and application of the technology.

[0102] Example 4

[0103] Parameter coding and population initialization: Binary coding is used to process the key operating parameters in the water electrolysis process, including electrolysis voltage, current density, electrolyte temperature and concentration. Taking the electrolysis voltage as an example, if its actual operating range is 2V-4V, this range is divided into several discrete values. Through 8-bit binary coding, 256 different coding combinations can be generated, corresponding to different voltage values, thereby realizing the discrete coding of continuous parameters and meeting the input requirements of the genetic algorithm. Similarly, current density, electrolyte temperature and concentration are similarly coded. After the coding design is completed, an initial population containing a certain number of individuals is randomly generated. Each individual is composed of the above parameter coding combination, representing a complete set of water electrolysis operating parameter solutions.

[0104] Fitness function design: A fitness function is constructed to evaluate the quality of each individual. This function comprehensively considers key factors such as water electrolysis efficiency, energy consumption, and system stability. In actual scenarios, if the current main goal is to improve hydrogen production efficiency, the weight of water electrolysis efficiency in the fitness function should be appropriately increased; if energy conservation is more important, the weight of energy consumption factors should be increased. Taking the pursuit of high hydrogen production efficiency as an example, the weight of water electrolysis efficiency is set to a higher value, such as 0.6, the weight of energy consumption is set to 0.3, and the weight of system stability is set to 0.1. Based on these weights and the actual values of the corresponding parameters, the fitness value of each individual is calculated. The higher the fitness value, the better the parameter combination corresponding to the individual.

[0105] Genetic Operations:

[0106] Selection: Roulette wheel selection is used to calculate the probability of selection based on each individual's fitness. Individuals with higher fitness occupy a larger area on the wheel and have a greater probability of being selected for the next generation. For example, if individual A's fitness accounts for 10% of the total fitness of the entire population, it has a 10% probability of being selected in each selection operation. Through repeated selection, superior individuals have more opportunities to pass on their genes to the next generation.

[0107] Crossover: Set the crossover probability, for example, to 0.8, which means there is an 80% probability of performing a crossover on the selected individuals. Randomly select two individuals as parents and randomly determine a crossover point in their coding sequences. Assume that individual A's coding sequence is [10101010] and individual B's coding sequence is [01010101]. The crossover point is chosen at position 4. After the crossover, the resulting new individuals C and D are [10100101] and [01011010]. Crossover promotes gene exchange within the population, increasing population diversity and helping to search for optimal parameter combinations.

[0108] Mutation: Mutate the genes of an individual according to a pre-set mutation probability, such as 0.01. Mutation randomly changes the value of one or more bits in the individual's code. For example, if the code for individual E is [11001100], and the third bit mutates, the mutated code for individual E′ becomes [11101100]. Mutation prevents the algorithm from prematurely falling into a local optimum and introduces new possibilities into the search process.

[0109] Iterative Optimization and Parameter Determination: The genetic operations described above are repeated over multiple generations. In each generation, the fitness of each individual is calculated, and the population is updated accordingly. Based on extensive testing and practical application experience, the number of iterations is set to 100. After 100 generations, the individual with the highest fitness value is selected from the final population. The corresponding water electrolysis operating parameter combination is the result of genetic algorithm optimization, which is then applied to the actual water electrolysis hydrogen production system.

[0110] Example 5

[0111] Data Collection and Preprocessing: Extensive historical operating data from the water electrolysis hydrogen production system under various operating conditions is collected, covering relevant parameters such as electrolysis voltage, current density, electrolyte temperature and concentration, hydrogen production, oxygen production, and ambient temperature and humidity. The collected data is carefully cleaned to remove outliers caused by sensor failures, data recording errors, and other factors. For example, if the electrolysis voltage at a given moment significantly exceeds the normal range and does not match other relevant parameters, the data point is considered an outlier and deleted. The data is then normalized to map data of varying ranges to the [0, 1] interval. For example, assuming the electrolysis voltage range is 2V-4V, normalization is performed using the formula (x_{norm} = frac{x - x_{min}}{x_{max} - x_{min}}) (where x is the original value, (x_{min}) is the minimum value of 2V, and (x_{max}) is the maximum value of 4V). This allows the neural network to more easily learn the characteristics and patterns in the data.

[0112] Neural Network Model Construction: A multi-layer feedforward neural network is constructed, consisting of an input layer, hidden layers, and an output layer. The number of input layer nodes is determined by the number of input data features. If six features (electrolysis voltage, current density, electrolyte temperature, concentration, hydrogen production, and ambient temperature) are selected as input, the number of input layer nodes is 6. The number of hidden layer nodes is set to 2-3, and the number is determined based on experience and multiple experiments. Generally, the principle of gradual decrease is followed, such as setting 128 nodes in the first layer and 64 nodes in the second layer. The number of hidden layer nodes should ensure sufficient learning capacity of the network while avoiding overfitting. The number of output layer nodes is determined by the number of parameters to be predicted. If the goal is to predict water electrolysis efficiency and energy consumption, the number of output layer nodes is 2.

[0113] Model Training and Optimization: The neural network is trained using the backpropagation algorithm, with mean squared error (MSE) used as the loss function to measure the difference between the model's predicted values and the actual values. During training, the Adam optimizer is used to automatically adjust the neural network's weights and biases to minimize the loss function. The collected dataset is divided into training, validation, and test sets with a 70% / 15% / 15% split. During training, the model is trained on the training set. After a certain number of training iterations (e.g., 100), the model's performance is evaluated on the validation set, observing the change in the loss function value on the validation set. If the loss function value on the validation set stops decreasing or shows an upward trend, the model may be overfitting. In this case, adjust model parameters (such as reducing the number of hidden layer nodes, adding regularization terms, etc.) or use other optimization methods until the model performs well on the validation set. Finally, evaluate the trained model on the test set to ensure good generalization.

[0114] Real-time control application: The trained neural network model is deployed in an actual water electrolysis hydrogen production system. During system operation, sensors collect real-time operating data from the water electrolysis hydrogen production system. This data is input into the neural network model, which predicts the water electrolysis efficiency and energy consumption based on the learned rules. The predicted results are compared with the pre-set target values, and based on the difference, the controller adjusts the operating parameters of the water electrolysis process. For example, if the predicted water electrolysis efficiency is lower than the target value, the controller can appropriately increase the electrolysis voltage or adjust the electrolyte concentration to achieve real-time optimization control of the water electrolysis process, ensuring that the system always maintains a near-optimal operating state.

[0115] Example 6

[0116] Initialization of the Particle Swarm Optimization algorithm: The operating parameters of water electrolysis—namely, electrolysis voltage, current density, electrolyte temperature, and concentration—are considered to be the positions of particles in a multidimensional space. Each particle represents a specific combination of these operating parameters. At the beginning of the algorithm, the particle swarm is randomly initialized, and each particle is randomly assigned an initial position within the parameter range. For example, if the electrolysis voltage range is 2V-4V, each particle is randomly assigned an initial voltage value within this range as one dimension of its initial position. The initial velocity of the particle is also set to 0. The particle swarm size is set, for example, to 30 particles. The choice of size affects the algorithm's search efficiency and accuracy. After multiple experiments, this size has been determined to be appropriate for the current scenario. Learning factors (c_1) and (c_2) are determined. These values are generally between 1 and 2. Here, (c_1 = c_2 = 1.5) are set. These control the step size of the particle's movement toward its historical optimal position and the global optimal position. Set the inertia weight w, with an initial value of 0.9, and gradually decrease with the increase of the number of iterations. For example, w decreases by 0.01 with each iteration. The inertia weight is used to balance the global search and local search capabilities of the particle.

[0117] Fuzzy logic controller design: Determine the fuzzy input and output variables. The fuzzy input variables are the deviation e between the current water electrolysis efficiency and the target efficiency, as well as the rate of change of the deviation ec. The output variables are adjustments to the water electrolysis operating parameters, such as the electrolysis voltage adjustment (Delta V) and the current density adjustment (Delta J). Fuzzification is performed on the input and output variables, dividing them into multiple fuzzy linguistic variables. For example, the deviation e is divided into five fuzzy sets: "negatively large," "negatively small," "zero," "positively small," and "positively large." A corresponding membership function is defined for each fuzzy set, using a triangular membership function in this case. A fuzzy rule base is constructed based on the experience of domain experts and extensive experimental data. For example, when e is "negatively large" and ec is "negatively small," the fuzzy rule stipulates increasing the electrolysis voltage and current density. After obtaining the fuzzy inference results, defuzzification methods such as the centroid method are used to convert the fuzzy outputs into precise parameter adjustments for application in a practical water electrolysis hydrogen production system.

[0118] Particle swarm optimization and fuzzy logic collaboration: During the iterative process of the particle swarm optimization algorithm, each particle updates its speed and position based on its historical optimal position pbest and global optimal position gbest. In each iteration, the system performance, including indicators such as electrolysis efficiency and energy consumption, is calculated for the electrolysis parameter combination corresponding to the particle's current position. During this performance calculation, the parameters are adjusted using a fuzzy logic controller, and system performance is calculated based on the adjusted parameters. If a particle's performance exceeds its historical optimal performance, its pbest is updated; if its performance exceeds the global optimal performance, its global optimal position gbest is updated. After multiple experiments and practical application verification, the number of iterations was set to 50. At the end of the iteration, the particle at the global optimal position is selected from the particle swarm. The electrolysis operating parameter combination represented by this particle is the optimal parameter combination obtained through the combination of particle swarm optimization and fuzzy logic. This combination is then applied to a practical electrolysis hydrogen production system to achieve optimized control of the electrolysis process.

Claims

1. A dynamic scheduling method for water electrolysis efficiency based on reinforcement learning, characterized in that: It includes the following steps: By deploying a variety of sensors in the electrolytic water hydrogen production system, real-time collection of electrolysis voltage, current density, electrolyte temperature, electrolyte concentration, hydrogen production, oxygen production, and power supply fluctuation data is carried out; The collected data is fused and processed to construct the state space of the electrolytic water hydrogen production system, and data preprocessing methods such as normalization are used to make data of different dimensions comparable; The improved Deep Deterministic Policy Gradient (DDPG) algorithm is used to dynamically optimize the electrolysis operation parameters. The improved DDPG algorithm includes an optimized Actor network and Critic network structure and an improved reward function; An adaptive exploration strategy is adopted to balance exploration and exploitation in the reinforcement learning process; A multi-time scale learning mechanism is designed to adapt to the time characteristics of different parameter changes in the electrolytic water hydrogen production system; The model transfer technology is used to apply the trained model parameters to a new electrolytic water hydrogen production system and perform fine-tuning; A real-time feedback and closed-loop control system is established. The actions output by the reinforcement learning algorithm are applied to the electrolytic water hydrogen production system, and the system feedback data is input into the reinforcement learning algorithm again.

2. The method for dynamic scheduling of water electrolysis efficiency based on reinforcement learning according to claim 1, characterized in that: In the construction of the state space, the collected data is normalized through the following formula: Where, x is the original data, xmin is the minimum value of this data type, xmax is the maximum value of this data type, and xnorm is the normalized data.

3. The method for dynamic scheduling of water electrolysis efficiency based on reinforcement learning according to claim 1, characterized in that: In the improved DDPG algorithm, the Actor network is used to generate action policies, the Critic network is used to evaluate action values, and the calculation formula of the improved reward function R is: R = w1×ΔE + w2×(-ΔP) + w3×S Among them, Δ E is the ratio of water electrolysis efficiency improvement, Δ P is the energy consumption change ratio, S is the system stability index, w1, w2, w3 are weight coefficients, and w1+w2+w3=1.

4. The method for dynamic scheduling of water electrolysis efficiency based on reinforcement learning according to claim 1, characterized in that: In the adaptive exploration strategy, the exploration rate epsilon is dynamically adjusted according to the following rules: At the beginning of training, a higher initial exploration rate epsilon0 is set; as the number of training steps t increases, the exploration rate epsilon gradually decreases. When the average reward AR obtained by the agent in consecutive n steps is greater than the preset threshold TR, the exploration rate epsilon decays according to the formula epsilon = epsilontimes*alpha (0 < alpha < 1); when the average reward AR of the agent in consecutive m steps is less than the preset threshold TRlow, the exploration rate epsilon increases according to the formula epsilon = epsilontimes*beta (beta > 1).

5. The method for dynamic scheduling of water electrolysis efficiency based on reinforcement learning according to claim 1, characterized in that: In the multi-time scale learning mechanism, for short-term changing parameters, the learning rate etas is larger; for medium-term changing parameters, the learning rate etam is moderate; for long-term changing parameters, the learning rate etal is smaller, and it satisfies etas > etam > etal.

6. The method for dynamic scheduling of water electrolysis efficiency based on reinforcement learning according to claim 1, characterized in that: In the model transfer technology, the model parameters trained on multiple electrolytic water hydrogen production systems with different scales and working conditions are used as the initial values of the new model. A small amount of operation data is collected on the new system, and by fine-tuning some parameters of the model, the model can quickly adapt to the new system environment.

7. The method for dynamic scheduling of water electrolysis efficiency based on reinforcement learning according to claim 1, characterized in that: In the real-time feedback and closed-loop control system, sensors collect operating status data of the water electrolysis hydrogen production system in real time, input the data into a reinforcement learning algorithm, and the algorithm outputs optimized operating parameters based on the current status. The operating parameters are converted into control instructions by the controller and sent to the actuator. The actuator adjusts the operation of the water electrolysis hydrogen production system, forming a continuous feedback optimization cycle.

8. The method for dynamic scheduling of water electrolysis efficiency based on reinforcement learning according to claim 1, characterized in that: The various sensors include voltage sensors and current sensors installed at the electrodes of the electrolytic cell, temperature sensors and concentration sensors on the electrolyte storage tank and circulation pipeline, gas flow sensors on the hydrogen and oxygen output pipelines, and connected power monitoring equipment.

9. The method for dynamic scheduling of water electrolysis efficiency based on reinforcement learning according to claim 1, characterized in that: The actuator includes an electric regulating valve, a controller for a heating or cooling device, and a power controller, which are respectively used to adjust the electrolyte flow, electrolyte temperature, and electrolysis voltage and current density.

10. A dynamic scheduling system for water electrolysis efficiency based on reinforcement learning, characterized in that: include: The data acquisition module consists of multiple sensors deployed in the water electrolysis hydrogen production system, which is used to collect real-time data on electrolysis voltage, current density, electrolyte temperature, electrolyte concentration, hydrogen production, oxygen production, and power supply fluctuations; The data processing and state space construction module is used to fuse the collected data, construct the state space of the water electrolysis hydrogen production system, and use data preprocessing methods such as normalization to make data of different dimensions comparable; A reinforcement learning optimization module uses an improved deep deterministic policy gradient (DDPG) algorithm to dynamically optimize the operating parameters of water electrolysis. The improved DDPG algorithm includes an optimized actor network and critic network structure and an improved reward function; Exploration strategy module, which implements adaptive exploration strategy to balance exploration and exploitation in the reinforcement learning process; Multi-time scale learning module, which designs a multi-time scale learning mechanism to adapt to the temporal characteristics of different parameter changes in the water electrolysis hydrogen production system; The model migration module uses model migration technology to apply the trained model parameters to the new water electrolysis hydrogen production system and perform fine-tuning; The real-time feedback and closed-loop control module applies the actions output by the reinforcement learning algorithm to the water electrolysis hydrogen production system, and re-inputs the system feedback data into the reinforcement learning algorithm to form a closed-loop control; A controller, which converts the optimized operating parameters output by the reinforcement learning algorithm into control instructions; The actuator, including the electric regulating valve, the controller of the heating or cooling device and the power controller, adjusts the operating parameters of the water electrolysis hydrogen production system according to the control instructions.

Images