Wind-solar-hydrogen system operation optimization method based on hierarchical verification deep reinforcement learning

Abstract

This invention discloses a method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning, belonging to the field of new energy hydrogen production system operation optimization technology. The method includes: collecting environmental and equipment information to construct a system state vector; establishing a multi-agent framework based on the soft actor-critic algorithm, defining the energy storage battery and electrolyzer as independent agents, and outputting control commands through continuous-discrete hybrid actions. Before execution, a Level I safety verification is performed, and the state is updated after passing; a fusion reward is calculated based on the new state using a global-local fusion reward mechanism. State samples are stored in an experience buffer for network parameter updates. During training, a Level II model performance verification is periodically performed, and qualified models are included in the candidate set; after training, a Level III deployment verification is performed, and the optimal model is selected for use in the actual system. This invention improves the operational efficiency, equipment lifespan, and system deployment reliability of wind-solar hydrogen production systems under fluctuating environments.

Description

Technical Field

[0001] This application relates to the field of operation optimization technology for new energy hydrogen production systems, and in particular to an operation optimization method for wind and solar hydrogen production systems based on hierarchical verification deep reinforcement learning. Background Technology

[0002] Renewable energy hydrogen production systems centered on wind and solar power are facing increasingly stringent requirements regarding their economic efficiency, efficiency, and reliability due to their crucial role in enhancing grid flexibility and achieving large-scale "green hydrogen" supply. In wind-solar hydrogen production systems, the coordinated optimization and control of heterogeneous equipment such as energy storage batteries and electrolyzers is key to improving overall system performance. Traditional operation optimization methods primarily rely on mathematical programming problems constructed from precise physical models for solution. However, these methods depend on accurate system modeling and suffer from limitations such as modeling difficulties, computational complexity, and poor real-time performance when facing the strong fluctuations in wind and solar resources and complex interactions between equipment. Therefore, researching an intelligent optimization method capable of autonomously adapting to complex environments and achieving multi-objective coordination is of great significance for improving the operational economy, reliability, and engineering practicality of wind-solar hydrogen production systems.

[0003] Currently, research has applied deep reinforcement learning techniques to wind and solar hydrogen production systems. However, their action space design often fails to accurately reflect the actual control characteristics of equipment such as energy storage batteries and electrolyzers, which involve both continuous adjustment and discrete switching. This results in generated commands frequently violating the physical constraints of the equipment. Furthermore, their reward mechanisms often focus on overall system-level indicators, lacking fine-grained guidance for equipment-level behavior, making it difficult to achieve an effective balance between improving system efficiency and ensuring long-term reliable operation of equipment. In addition, existing methods generally lack systematic safety and reliability verification mechanisms during model training and deployment. On the one hand, it is difficult to guarantee that all actions output by the agent meet actual safety constraints during training; on the other hand, the trained models lack robustness assessments under various extreme or unknown conditions. This leads to high deployment risks and poor adaptability in practical engineering, limiting their industrial application. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing an operation optimization method for wind and solar hydrogen production systems based on hierarchical verification deep reinforcement learning.

[0005] According to a first aspect of the embodiments of this application, a method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning is provided, comprising: S1: Collect environmental data and equipment information, and obtain the current state vector after feature extraction; S2: A heterogeneous multi-agent framework is constructed based on the soft actor-critic algorithm of hierarchical verification deep reinforcement learning. The energy storage battery and the electrolyzer are defined as independent agents. Each agent contains a policy network, a value network and a target value network, and a central value network and its target value network are added. S3: Receive the current state vector through each policy network, and output continuous and discrete instructions through continuous-discrete hybrid mapping; S4: Perform Level I security verification on the instruction. If the verification is successful, the valid instruction is executed. Repeat S1 to obtain a new state vector. S5: Based on the new state vector, the global reward and the fusion reward are calculated by the global-local fusion reward mechanism; S6: The current state vector, new state vector, valid instructions, global reward, and fusion reward are combined to form a sample and stored in the experience buffer. S7: Randomly sample from the experience buffer and input the central value network and its target value network, the value networks of each agent and their target value networks, and synchronously update all network parameters. S8: Repeat S1 to S5 for multiple rounds of training and select several rounds to perform Level II model validation on the policy network. Those that pass are stored in the candidate set. S9: After training, the candidate set is subjected to Level III deployment verification, and the optimal strategy network parameter combination is selected as the control model for final deployment.

[0006] Optionally, in S3, the strategy network of the energy storage battery agent simultaneously outputs continuous control commands for charging and discharging power, discrete control commands for charging / discharging direction, and discrete control commands for start / stop conditions. The strategy network of the electrolyzer agent simultaneously outputs continuous control commands for operating power and discrete control commands for operating / hot standby / cold standby / stop conditions.

[0007] Optionally, in S4, the Level I safety verification criteria include system power balance constraints, electrolytic cell continuous-discrete command conflict constraints, electrolytic cell maximum operating power constraints, electrolytic cell operating / hot standby / cold standby power constraints, energy storage battery continuous-discrete command conflict constraints, energy storage battery maximum power constraints, and energy storage battery state of charge constraints.

[0008] Optionally, in S5, the global reward represents three types of overall system performance: wind and solar energy absorption rate, hydrogen production efficiency, and energy surplus rate; the fusion reward is obtained by weighting the global reward and local reward through a reward coordination factor driven by multi-index feedback, and the reward coordination factor driven by multi-index feedback is generated by nonlinear mapping based on the multi-dimensional performance indicators of the system operating status; the local reward of the electrolyzer represents two types of equipment behavior: electrolyzer power fluctuation and electrolyzer operating condition switching frequency; the local reward of the energy storage battery represents three types of equipment behavior: energy storage battery power fluctuation, energy storage battery charge and discharge cycle count, and energy storage battery overcharge and over-discharge.

[0009] Optionally, in S7, the central value network and the value networks of each agent update their respective network parameters by minimizing the temporal difference error. The target value network of the central value network and the target value networks of each agent update their respective network parameters by soft updating. The policy network of each agent updates its respective network parameters by maximizing the policy gradient of the expected return regularized by entropy regularization.

[0010] Optionally, in S8, the verification criteria for the Level II model include whether the cumulative global reward has converged, whether the cumulative local reward has converged, and whether the cumulative pass rate of the Level I security verification instruction is lower than a preset threshold.

[0011] Optionally, in S9, the Level III deployment verification is tested on independent datasets for five scenarios: historical typical day wind and solar power output, extreme typical day wind and solar power output, partial shutdown of energy storage batteries, partial shutdown of electrolyzers, and simultaneous partial shutdown of energy storage batteries and electrolyzers.

[0012] Optionally, the optimal strategy network parameter combination is the optimal parameter combination after comprehensively considering five indicators: wind and solar power absorption rate, hydrogen production efficiency, energy surplus rate, number of electrolyzer operating condition switching times, and number of energy storage battery charge and discharge cycles.

[0013] Optionally, the reward component characterizing the power fluctuation behavior of the electrolytic cell in the local reward of the electrolytic cell is calculated using a piecewise fluctuation barrier function; the reward component characterizing the power fluctuation behavior of the energy storage battery in the local reward of the energy storage battery is calculated using a piecewise fluctuation barrier function; and the reward component characterizing the overcharge and over-discharge behavior of the energy storage battery in the local reward of the energy storage battery is calculated using a state-of-charge safety barrier function.

[0014] According to a second aspect of the embodiments of this application, an electronic device is provided, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0015] The technical solutions provided by the embodiments of this application may include the following beneficial effects: Based on environmental data and equipment status information of the wind-solar hydrogen production system, this invention constructs a soft actor-commentator deep reinforcement learning network framework based on hierarchical verification deep reinforcement learning. Each energy storage battery and electrolyzer is defined as an independent intelligent agent, and a central value network is added for coordination. The system can fully learn and utilize the differences and cooperative relationships between heterogeneous devices, thereby achieving significant optimization of the cooperative scheduling relationship of multiple devices in the wind-solar hydrogen production system and improving the overall cooperative control capability of the system. This invention designs a dedicated continuous-discrete hybrid mapping mechanism for the physical control characteristics of energy storage batteries and electrolyzers, enabling the action commands generated by the intelligent agent to accurately match the executable interface of the actual equipment. This effectively overcomes the problem of mismatch between the action space and actual physical constraints in the traditional DRL method, thereby ensuring the feasibility and safety of the optimization strategy in engineering applications. In the process of performing operation optimization calculations for wind and solar hydrogen production systems, this invention constructs a reward mechanism that integrates global and local rewards. This mechanism guides the agent to actively smooth equipment operation behavior and reduce harmful losses while pursuing the optimal overall system performance, thereby effectively extending the service life of key equipment and improving the long-term production economy of wind and solar hydrogen production systems. This invention fully considers the volatility of wind and solar resources and the diversity of actual operating conditions, and introduces a hierarchical verification mechanism that runs through the training and deployment stages. It can safely monitor the training process and select strategy algorithms that are convergent, stable and have strong generalization ability as the actual output, thereby ensuring that the final deployed optimization strategy has excellent adaptability, stability and reliability.

[0016] The above general description and the following detailed description are exemplary and illustrative only, and are not intended to limit this application. Attached Figure Description

[0017] Figure 1 A flowchart of the wind-solar hydrogen production system operation optimization method based on hierarchical verification deep reinforcement learning provided in the embodiments of the present invention.

[0018] Figure 2 This is a schematic diagram of the wind-solar hydrogen production system provided in an embodiment of the present invention.

[0019] Figure 3 The solar power output curves for typical days in spring, summer, autumn and winter provided for embodiments of the present invention.

[0020] Figure 4 A typical electrolytic cell power curve for spring provided for embodiments of the present invention.

[0021] Figure 5 The power curve of a typical electrolytic cell in summer is provided for an embodiment of the present invention.

[0022] Figure 6 The power curve of a typical electrolytic cell in autumn is provided for an embodiment of the present invention.

[0023] Figure 7 A typical electrolytic cell power curve for winter provided for embodiments of the present invention.

[0024] Figure 8 The hydrogen production curves for typical days in spring, summer, autumn and winter are provided for embodiments of the present invention.

[0025] Figure 9 A comparison chart of CPU utilization during the training phase of the DVRL-SAC and SPEA2 algorithms provided in this embodiment of the invention.

[0026] Figure 10 The diagram shows the convergence curves of the overall performance under different reward configurations provided in the embodiments of the present invention.

[0027] Figure 11 The bar chart shows the comprehensive scores of the DVRL-SAC and SPEA2 algorithms provided in this embodiment of the invention in five test scenarios. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0031] Figure 1 This is a flowchart illustrating an operational optimization method for a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning, according to an exemplary embodiment. Figure 1 As shown, the method may include the following steps: S1: Collect environmental data and equipment information, and obtain the current state vector after feature extraction; Specifically, constructing such Figure 2 The system shown consists of a wind power generation system, a photovoltaic power generation system, an energy storage battery, and four alkaline electrolyzers, forming a wind-solar hydrogen production system. The main parameters of the system are shown in Table 1.

[0032] Table 1: Parameters of wind-solar hydrogen production system;

[0033] The effectiveness and performance of the proposed method were compared and verified using the above settings. Figure 3 The solar power output curves for typical days in each season are given. Figure 3 As shown, wind resources are abundant and stable in spring, with long hours of sunshine; sunshine starts about 1 hour earlier in summer than in spring, and wind power fluctuates wildly; solar power output is moderate in autumn, and wind power output is stable; winter has the shortest sunshine hours, the lowest photovoltaic output, and low but stable wind power output.

[0034] Because the power output fluctuates randomly in different seasons and the dimensions of equipment state vary, directly inputting these parameters into a deep neural network can lead to unstable gradient updates and slow convergence. Therefore, this system normalizes each continuous variable and performs one-hot encoding on the discrete state variables. Finally, all normalized environmental features, equipment state features, and pre-stored parameters are concatenated to obtain the current system state vector. The expression is as follows:

[0035] in Let this be the system state vector at the current moment. For the current moment, The current wind speed, For ambient temperature, Light intensity, This is a battery status indicator. This is the normalization coefficient for the number of start-stop cycles. This represents the current output power of the wind turbine. This refers to the rated power of the fan. This represents the current output power of the photovoltaic system. Rated power of photovoltaic, This represents the current total output power of renewable energy. For the ideal total power of the system, The total power output of renewable energy is set. This represents the current theoretical maximum hydrogen production capacity. This is the current battery state of charge value. This is the maximum permissible state of charge value for the battery. This is the weighting coefficient for energy dispatch.

[0036] S2: A heterogeneous multi-agent framework is constructed based on the soft actor-critic algorithm of hierarchical verification deep reinforcement learning. The energy storage battery and the electrolyzer are defined as independent agents. Each agent contains a policy network, a value network and a target value network, and a central value network and its target value network are added. Specifically, each device is defined as an independent intelligent agent. Each energy storage battery intelligent agent is responsible for controlling the charging and discharging power, direction, and start / stop. Each electrolyzer intelligent agent independently controls its operating power and operating conditions (operation / hot standby / cold standby / shutdown). A central value network is added to coordinate global information and aggregate global states to coordinate the action tendencies of the heterogeneous intelligent agents. To verify the advantages of the above heterogeneous multi-agent framework and central value network in the collaborative and decoupled control under complex wind and solar conditions, this embodiment compares the Divided Level Verification Reinforcement Learning-based Soft Actor-Critic (DVRL-SAC) algorithm and the traditional Strength Pareto Evolutionary Algorithm 2 (SPEA2) using an NVIDIA RTX 4090 GPU computing platform in a Python 3.10 environment. The key parameter configurations of the two algorithms in the experiment are shown in Table 2.

[0037] Table 2: Algorithm parameters;

[0038] Figure 3The wind and solar power output curves for typical days in spring, summer, autumn, and winter provided in this embodiment of the invention show that wind power fluctuates dramatically in summer and there is a significant energy shortage in winter. Under these complex conditions, the DVRL-SAC algorithm, through the collaboration of independent agents and a central value network, demonstrates significantly better performance than the traditional SPEA2 algorithm. (1) Decoupling of control logic for heterogeneous equipment: In view of the different physical requirements of stable operation of electrolytic cells and high-frequency response of energy storage batteries, independent strategy network realizes effective decoupling of control logic. Figure 5 The power curves of a typical electrolytic cell in summer are given. Figure 5 It can be seen that the power curves of each electrolyzer are smooth under the DVRLSAC algorithm, and the equipment coordination difference is only 198.9kW, which is far lower than the 382.3kW of the SPEA2 algorithm, and the coordination control capability is improved by 47.9%. Through the aggregation and coordination of global information by the central value network, the power allocation of multiple electrolyzers in the face of severe fluctuations is ensured, enabling the energy storage intelligent agent to take priority in handling high-frequency fluctuations and significantly improving the system's operational stability.

[0039] (2) The central value network achieves global collaborative optimization: By aggregating global information, the central value network achieves better scheduling logic under extreme working conditions such as winter. Figure 7 A typical electrolytic cell power curve for winter is given. Figure 8 The graphs of 24-hour hydrogen production on typical days in each season are given. Figure 7 and Figure 8 It can be seen that under extreme winter operating conditions, the equipment coordination difference of DVRLSAC is only 235.4kW, an improvement of 73.9% compared to 903.0kW of SPEA2, and the average correlation coefficient of the electrolyzer is 0.866 (0.563 for SPEA2). This mechanism uses a global reward signal to guide the energy storage battery to discharge reasonably when wind and solar power are insufficient and to charge when there is excess, and guides the electrolyzer to accurately perform hot / cold standby switching, avoiding unnecessary start-ups and shutdowns. This results in an increase in hydrogen production of 3.02% and 1.63% in winter and summer, respectively, directly verifying that the global coordination mechanism effectively improves the energy conversion efficiency of the system while ensuring the long-term operation of the equipment.

[0040] S3: Receive the current state vector through each policy network, and output continuous and discrete instructions through continuous-discrete hybrid mapping; Specifically, Figure 2This is a schematic diagram of the wind-solar hydrogen production system provided in this embodiment of the invention. This example uses one energy storage battery and four electrolyzers. The main parameters of the system are shown in Table 1. The strategy network of the energy storage battery's intelligent agent simultaneously outputs continuous control commands for charging and discharging power, discrete control commands for charging / discharging direction, and discrete control commands for start / stop operation conditions. The strategy network of the electrolyzer's intelligent agent simultaneously outputs continuous control commands for operating power and discrete control commands for operating / hot standby / cold standby / stop operation conditions. The strategy network adopts a 3-layer fully connected network with 256 neurons in each layer and a learning rate of [missing information]. To address the training challenge caused by the non-differentiability of discrete actions, this invention designs a continuous-discrete hybrid mapping mechanism. Each agent's policy network simultaneously generates continuous control parameters and discrete choices, outputting differentiable hybrid actions through Gaussian sampling and GumbelSoftmax sampling, respectively. This allows the policy network to simultaneously optimize continuous and discrete control quantities end-to-end. This mechanism approximates the non-differentiable discrete sampling process as continuous, ensuring that gradients can propagate back from the loss function to the policy network, thus achieving end-to-end joint optimization of the DVRLSAC algorithm. The above mechanism exhibits the following effects in practical simulations: Instruction consistency logic verification: The traditional SPEA2 algorithm often leads to logical conflicts such as "power command available but system shutdown" due to post-processing mapping. This invention achieves near-zero continuous discrete conflict rate by pairing instruction outputs. Figure 4 The power curve of a typical electrolytic cell in spring shows that the power fluctuation of the electrolytic cell in spring dropped significantly from 394.2kW ​​(SPEA2) to 71.3kW (DVRLSAC), effectively suppressing the power spikes caused by frequent adjustments due to command conflicts.

[0041] Power trajectory smoothing: Hybrid mapping enables devices to intelligently adjust power and reduce unnecessary operating condition switching. Figure 5 The power curves of a typical electrolytic cell in summer are given. Figure 5 The DVRLSAC showed a total power variation coefficient of 41.21%, which is better than SPEA2's 48.12%. Especially during the sunrise startup phase (6-8 am), the DVRLSAC showed a 95.1% improvement in startup smoothness compared to SPEA2. Figure 6 A typical electrolytic cell power curve for autumn is presented. Figure 6 The total power variation coefficient of the DVRLSAC in the middle is 22.49%, while that of SPEA2 is 31.21%, resulting in a system stability improvement of 28.0%. Figure 7 The diagram shows the power curve of a typical electrolytic cell in winter. During the low load period (0-7h) at night in winter, the DVRLSAC achieves a completely balanced output (7000kW, total standard deviation of 0), while the SPEA2 has an uneven distribution, highlighting the precise control of the hybrid action mechanism under extreme conditions.

[0042] High efficiency of end-to-end training: The introduction of Gumbel-Softmax sampling makes discrete actions differentiable and gradients can be backpropagated to the policy network, thus achieving end-to-end training. Figure 9 DVRLSAC and A comparison chart of CPU utilization during the training phase of the algorithm. Figure 9 The results show that, thanks to the full utilization of gradient information by the differentiable architecture, the DVRL-SAC algorithm, although having a slightly higher instantaneous CPU utilization during training, has a 35.3% lower total computation time and only requires 30 seconds to deploy. Its overall efficiency is significantly better than the traditional SPEA2 algorithm, which cannot utilize gradient backpropagation.

[0043] The continuous-discrete hybrid mapping mechanisms for the energy storage battery intelligent agent and the electrolyzer intelligent agent are as follows: 1. The agent's policy network simultaneously outputs continuous control commands for charging and discharging power, discrete control commands for charging / discharging direction, and discrete control commands for start / stop operation. Its continuous-discrete hybrid mapping mechanism includes: The policy network of the energy storage battery agent uses a number of neurons. A multi-layer fully connected network structure is responsible for generating a set of Gaussian-distributed real number pairs. and 2 groups of classification distributions vector , ; After sampling using a Gaussian distribution, the output charging and discharging power continuous control command is expressed as follows:

[0044] In the formula, for Continuous control command for charging and discharging power at all times; It is the hyperbolic tangent function; Let be a random variable, and let it follow a standard normal distribution, i.e. ; This represents the maximum charge and discharge power of the energy storage battery.

[0045] After sampling by Gumbel-Softmax, the discrete control command for charging / discharging direction is output, and the expression is as follows:

[0046] In the formula, for Discrete control commands for charging / discharging direction at any given time; arg max is the maximum value index operation. ; This is the Gumbel noise vector; These are the temperature parameters for the charging / discharging direction.

[0047] After sampling by Gumbel-Softmax, the discrete control commands for start / stop conditions are output, and the expressions are as follows:

[0048] In the formula, for Discrete control commands for start / stop operation at all times. ; This is the Gumbel noise vector; Temperature parameters for start / stop operation.

[0049] 2. The strategy network of the electrolytic cell agent simultaneously outputs continuous control commands for operating power and discrete control commands for operating / hot standby / cold standby / shutdown conditions. Its continuous-discrete hybrid mapping mechanism includes: The policy network of the electrolytic cell agent uses a number of neurons. A multi-layer fully connected network structure is responsible for generating a set of Gaussian-distributed real number pairs. and a set of logits vectors for categorical distributions. ; The operating power continuous control command is generated by sampling using a Gaussian distribution, and its expression is as follows:

[0050] In the formula, for Continuous power control commands are executed at all times; It is the hyperbolic tangent function; Let be a random variable, and let it follow a standard normal distribution, i.e. ; This represents the maximum power output of the electrolytic cell under operating conditions.

[0051] After sampling by Gumbel-Softmax, the discrete control commands for running / hot standby / cold standby / shutdown conditions are output, and the expressions are as follows:

[0052] In the formula, for Discrete control commands for continuous operation / hot standby / cold standby / shutdown conditions. For maximum value index operation, ; This is the Gumbel noise vector; These are the operating temperature parameters for the electrolytic cell.

[0053] S4: Perform Level I security verification on the instruction. If the verification is successful, the valid instruction is executed. Repeat S1 to obtain a new state vector.

[0054] Specifically, Figure 2 The diagram below shows the structure of the wind and solar hydrogen production system provided in this embodiment of the invention. This embodiment takes one energy storage battery and four electrolyzers as an example. The main parameters of the system are shown in Table 1. Figure 3 The image shows the solar power output curves for typical days in spring, summer, autumn, and winter, as provided in this embodiment of the invention. Figure 3 The data shows that summer sunlight starts about 1 hour earlier than spring, and wind power fluctuations are severe. Figure 5 This is a typical summer electrolytic cell power curve provided in an embodiment of the present invention. Figure 5 The DVRLSAC total power variation coefficient is 41.21%, which is better than SPEA2's 48.12%, indicating that during periods of severe wind power fluctuations in summer, Level I safety verification effectively avoids dangerous power commands, allowing the agent to learn within safe boundaries. Therefore, a hard constraint verification layer must be added before action execution. The verification basis includes seven constraints: system power balance constraint, electrolyzer continuous-discrete command conflict constraint, electrolyzer maximum operating power constraint, electrolyzer operating / hot standby / cold standby power constraint, energy storage battery continuous-discrete command conflict constraint, energy storage battery maximum power constraint, and energy storage battery state of charge constraint. Details are as follows: (1) The maximum operating power constraint expression of the electrolytic cell is as follows:

[0055] (2) The power constraint expressions for electrolytic cell operation / hot standby / cold standby are as follows:

[0056] In the formula, for time; The rated operating power of the electrolytic cell under hot standby conditions; Rated operating power of the electrolytic cell under cold standby conditions; This represents the lower boundary of the operating power threshold for the hot standby mode of the electrolytic cell and the upper boundary of the operating power threshold for the cold standby mode of the electrolytic cell. This represents the lower boundary of the operating power threshold for the electrolytic cell in cold standby mode. This represents the upper boundary of the operating power threshold for the electrolytic cell.

[0057] (3) The expression for the conflict constraint between continuous and discrete instructions in the electrolytic cell is as follows:

[0058] In the formula, This is a continuous-discrete command conflict flag for the electrolytic cell. This indicates a conflict between continuous and discrete commands for the electrolytic cell. This indicates that the continuous-discrete instructions for the electrolytic cell satisfy the conflict constraint; This is the operating condition indicator bit for the continuous control command of the electrolytic cell's operating power, and its expression is as follows:

[0059] (4) The maximum power constraint expression for the energy storage battery is as follows:

[0060] In the formula, This represents the maximum charge and discharge power of the energy storage battery.

[0061] (5) The state of charge constraint expression of the energy storage battery is as follows:

[0062] In the formula, and They are respectively and The charge level of the energy storage battery at all times; The time interval between adjacent moments; and These represent the lower and upper safe limits for the charge capacity of the energy storage battery, respectively.

[0063] (6) The expression for the continuous-discrete instruction conflict constraint of the energy storage battery is as follows:

[0064] In the formula, This is a continuous-discrete command conflict flag for energy storage batteries. This indicates a conflict between continuous and discrete instructions for energy storage batteries. This indicates that the continuous-discrete instructions for the energy storage battery satisfy the conflict constraint. This is the operating condition indicator bit for the continuous control command of the energy storage battery's operating power, and its expression is as follows:

[0065] (7) The expression for the system power balance constraint is as follows:

[0066] In the formula, for The power generation capacity of wind turbine units at all times for The power generation capacity of photovoltaic systems at any given time; This refers to the number of electrolyzers in a wind-solar hydrogen production system. The number of energy storage battery devices in a wind-solar hydrogen production system; for Time of the first Continuous control commands for the operating power of each electrolytic cell for Time of the first Continuous control commands for the operating power of each energy storage battery for Time of the first Discrete control commands for the charging / discharging direction of an energy storage battery.

[0067] The valid instructions verified by the above seven safety constraints are issued to the energy storage battery and electrolyzer for execution, and the system enters the next moment. Subsequently, the environmental data and equipment information collection process described in S1 is repeated, and after feature extraction, a new state vector for the next moment is obtained. This new state vector will serve as the input basis for the calculation of the global-local fusion reward mechanism in S5 and will participate in the sample construction of the experience buffer in S6.

[0068] S5: Based on the new state vector, the global reward and the fusion reward are calculated by the global-local fusion reward mechanism; Optimizing the operation of a wind-solar-hydrogen production system is a multi-objective problem. It aims to maximize the wind and solar energy utilization rate and hydrogen production efficiency (overall system performance) while suppressing power fluctuations, reducing operating condition switching frequency, and avoiding overcharging and over-discharging (long-term equipment lifespan). A single reward signal is insufficient to balance these conflicting objectives. Therefore, this invention designs a global-local fusion reward mechanism: the global reward represents three types of overall system performance—wind and solar energy utilization rate, hydrogen production efficiency, and energy surplus rate—guiding improvements in both. The local rewards include local rewards for the electrolyzer and the energy storage battery, effectively suppressing harmful behaviors such as power fluctuations, operating condition switching, and overcharging and over-discharging through piecewise fluctuation barrier functions and state-of-charge safety barrier functions. The fusion reward is obtained by weighting the global and local rewards through a reward coordination factor driven by multi-index feedback, achieving a leap from static trade-offs with fixed weights to dynamic coordination based on state awareness.

[0069] Specifically, Figure 2 This is a schematic diagram of the wind-solar hydrogen production system provided in this embodiment of the invention. This example uses one energy storage battery and four electrolyzers. The main parameters of the system are shown in Table 1. The DVRL-SAC algorithm and the SPEA2 algorithm are compared under this system, and the verification results are as follows: (1) The strong guiding effect of global rewards on the overall performance of the system: Figure 4The power curves of typical electrolytic cells in spring are presented. Thanks to the incentives of hydrogen production efficiency and wind and solar power integration rate in the global reward, the DVRL-SAC algorithm achieved an average load factor of 91.6% in spring, higher than the 90.6% of the SPEA2 algorithm. Under the extreme energy shortage conditions in winter, the average load factor of the DVRL-SAC algorithm reached 75.7%, 2% higher than the SPEA2 algorithm. Especially during the afternoon recovery period (14-18h), the average total power reached 9061.2kW, with a power recovery advantage of 1157.0kW, demonstrating the dynamic gain capability of the global reward under fluctuating resources.

[0070] (2) The protective effect of local rewards on equipment life: Local rewards significantly suppress equipment loss through the power fluctuation barrier function. Figures 4 to 7 The following figures illustrate the electrolyzer power curves under typical daytime conditions in spring, summer, autumn, and winter using the DVRLSAC and SPEA2 algorithms: Figures 4 to 7 As shown, the power variation coefficient of DVRL-SAC was significantly improved compared to SPEA2 in spring, summer, and autumn (14.4% improvement in summer stability and 28.0% improvement in autumn stability). Especially during high-fluctuation periods such as sunrise (6-8 am) in summer and nighttime (10-11 pm) in autumn, power smoothness was improved by 95.1% and 69.7%, respectively. During low-load periods in winter, DVRL-SAC achieved perfectly balanced output with a total power standard deviation of 0, verifying the effectiveness of local rewards in suppressing harmful power fluctuations and extending equipment lifespan.

[0071] (3) Dynamic balance of multiple objectives driven by multi-indicator feedback: In this embodiment, the weight of the coordination factor is dynamically adjusted according to hydrogen production efficiency, power fluctuation and SOC deviation. Figures 4 to 7 The following figures illustrate the electrolyzer power curves under typical daytime conditions in spring, summer, autumn, and winter using the DVRLSAC and SPEA2 algorithms: Figures 4 to 7 As shown, when the system is in a steady state, the coordination factor approaches 1, emphasizing global rewards to pursue maximum efficiency; when the system is in a state of violent fluctuation, the coordination factor is sensitively reduced, strengthening local rewards to extend equipment life. Figure 10 Convergence curves for overall performance under different reward configurations. Figure 10 The scheme using the coordination factor of this invention (Group D) achieved convergence in about 2200 rounds, which is about 21.4% faster than the fixed weight strategy (2800 rounds). The final overall performance score reached 0.96, which is 13%~31% higher than the pure efficiency-oriented strategy, the pure protection-oriented strategy and the fixed weight strategy.

[0072] In summary, the global-local fusion reward mechanism effectively solves the dynamic balance problem between production efficiency and equipment health. While ensuring stable operation around the clock (smoothness improved by up to 95.1%), it achieves efficient energy conversion and consumption, verifying its excellence in multi-objective optimization under complex stochastic operating conditions.

[0073] The specific formulas for calculating global rewards, local rewards, and fusion rewards are as follows: (1) Based on the global reward characterizing the three types of overall system performance information—wind and solar energy absorption rate, hydrogen production efficiency, and energy surplus rate—the global reward is calculated as follows:

[0074] In the formula, for Real-time global rewards for Incentives for high absorption rates of scenic spots for Incentives for improving hydrogen production efficiency; for The energy surplus rate penalty term is used to calculate the wind and solar power absorption rate reward term, the hydrogen production efficiency reward term, and the energy surplus rate penalty term. Their expressions are as follows:

[0075]

[0076]

[0077] In the formula, for The actual amount of renewable energy consumed at any given time. For total renewable energy; for Actual hydrogen production at any given time This represents the theoretical maximum hydrogen production capacity. for The electrical energy that has been absorbed by the system but not converted into hydrogen; , As the reward coefficient, This is the penalty coefficient.

[0078] (2) The local reward characterizes the independent behavioral characteristics of each device and consists of two parts: the local reward of the energy storage battery and the local reward of the electrolyzer. The local reward is calculated as follows:

[0079] In the formula, for Local rewards at any given moment; For the number of energy storage batteries, This refers to the number of electrolytic cells; and In respectively Time of the first The local reward value of the energy storage battery in Taiwan and in Time of the first The local reward value of the electrolyzer is given by the following formula: The local reward of the energy storage battery represents three types of equipment behavior: power fluctuation, number of charge-discharge cycles, and overcharge / over-discharge; the local reward of the electrolyzer represents two types of equipment behavior: power fluctuation and number of electrolyzer operating condition switching cycles. The local rewards of the energy storage battery and the electrolyzer are calculated using the following expressions:

[0080]

[0081] In the formula, For piecewise fluctuation barrier functions, For the state-of-charge safety barrier function; In order to be in Time of the first The number of start-stop cycles for Taiwan's energy storage batteries. In order to be in Time of the first Number of start-ups and shutdowns of the TECH solvent cell; exist Time of the first The power standard deviation of the TSMC electrolytic cell, among which For the first Taiwan Electrolytic Cell Operating power at any given time For the first Taiwan Electrolytic Cell The average power within the sliding window at any given time; In order to be in Time of the first The absolute value of the power fluctuation of Taiwan's energy storage batteries, among which No. Taiwan's energy storage batteries The charging and discharging power at any given time No. The charging and discharging power of the energy storage battery at the previous moment; In order to be in Time of the first The state of charge of Taiwan's energy storage batteries; The value of the sliding window is determined based on the system sampling period and the device thermal inertia time constant, so as to achieve a balance between response sensitivity and statistical stability. , and This is the penalty coefficient for energy storage batteries. and Let be the penalty coefficient of the electrolytic cell agent, where is the penalty coefficient. , , , and It consists of fixed parameters set in advance based on system characteristics and control objectives, used to constrain equipment behavior and balance system performance with equipment lifespan in a multi-objective optimization.

[0082] In (2), power fluctuations in the energy storage battery and electrolyzer are unavoidable. Severe fluctuations will accelerate equipment aging. The reward components characterizing the equipment behavior during power fluctuations in the local rewards of the energy storage battery and the local rewards of the electrolyzer are calculated using a piecewise fluctuation barrier function. The aim is to construct a nonlinear response range of fluctuation sensitivity to distinguish between "normal adjustment fluctuations" and "harmful severe fluctuations". The expression is as follows:

[0083] In the formula, For power fluctuations, the energy storage battery intelligent agent is... For the intelligent agent of the electrolytic cell ; The fluctuation threshold; and For shape parameters, , >0 and This causes the penalty function to abruptly change at the threshold, making drastic fluctuations more costly and ensuring optimal device operation.

[0084] In (2), the energy storage battery has an optimal operating range. Deviating from the optimal point will accelerate aging, and exceeding the limit will cause immediate damage. The state-of-charge safety barrier function is used to define the safe potential energy field for equipment operation and to calculate the reward component in the local reward of the energy storage battery that represents the overcharge and over-discharge behavior of the energy storage battery. This is used to guide the energy storage battery to work within the safe range and as close to the optimal point as possible. The expression is as follows:

[0085] In the formula, This refers to the state of charge of the energy storage battery. , For the safety boundary of the state of charge, This is the midpoint of the safe range; This is a shape parameter, and its value depends on the sensitivity of the selected energy storage battery to deviation. This is the amplitude parameter, used to control the maximum value of the penalty.

[0086] (3) The fusion reward is obtained by weighting the global reward and the local reward. It is used to simultaneously reflect the overall system performance goal and the individual device behavior constraint in a single reward signal, so as to achieve multi-objective collaborative optimization. The expression is as follows:

[0087] In the formula, This is a reward coordination factor driven by multi-indicator feedback. The coordination factor is driven by a set of multi-dimensional performance characteristics reflecting the real-time operating state of the system. These multi-dimensional performance characteristics include, but are not limited to, indicators reflecting the system's energy conversion efficiency, operational stability, and the health status of key equipment. By constructing a comprehensive performance representation function and combining it with a nonlinear mapping mechanism to generate the coordination factor, it can dynamically allocate a proportional distribution between system-level global rewards and equipment-level local rewards based on the real-time deviation of the system's operating state, thereby achieving synergistic optimization of global performance goals and local behavioral constraints. In this embodiment, the system monitors three indicators—hydrogen production efficiency, power fluctuation, and energy storage battery state of charge—to reflect the system's recent operating performance. The calculation... The steps are as follows: 1) Define key performance indicators. In this example, hydrogen production efficiency, power fluctuation, and energy storage battery state of charge are selected to reflect the system's operating status in terms of efficiency, stability, and safety, respectively. Calculate the average hydrogen production efficiency, average power fluctuation, and average energy storage battery state of charge deviation, as shown in the following expressions:

[0088]

[0089]

[0090] In the formula, This represents the average hydrogen production efficiency. For average power fluctuation, This represents the average deviation of the state of charge of the energy storage battery. The preset sliding window length for system performance evaluation is used to calculate the historical time steps of recent average performance indicators.

[0091] 2) Calculate the standardized deviations of hydrogen production efficiency, power fluctuation, and energy storage battery state of charge relative to the desired targets, using the following expressions:

[0092]

[0093]

[0094] In the formula, Target hydrogen production efficiency; The upper limit of target power fluctuation; This represents the upper limit of the target energy storage battery's state of charge deviation.

[0095] 3) Construct a comprehensive regulation signal by weighting the standardized deviations of hydrogen production efficiency, power fluctuation, and energy storage battery state of charge to form a comprehensive regulation signal, as shown in the following expression:

[0096] In the formula, , , A positive gain coefficient is used to adjust the relative importance of each deviation term in the synthesized signal.

[0097] 4) Weight mapping, using The activation function ensures that the larger the overall signal, the closer the weight is to 1, and the smaller the overall signal, the closer the weight is to 0, thus achieving a balance between global and local rewards. Its expression is as follows:

[0098] In the formula, For activation functions; For comprehensive adjustment signals.

[0099] S6: The current state vector, new state vector, valid instructions, global reward, and fusion reward are combined to form a sample and stored in the experience buffer. Specifically, Table 2 shows the algorithm parameters. In this step, the transition samples at each time step (consisting of the current state, valid instructions verified by Level I security, the new state, the global-fusion reward, and the termination flag) are stored in the experience buffer. This buffer uses a first-in, first-out (FIFO) strategy, and its capacity is set to... (This embodiment) When storage exceeds capacity, the oldest sample is automatically discarded. To enhance the agent's generalization ability, the following will be implemented: Figure 3 The data of typical days in the four seasons are mixed and stored in the same buffer, which enables the agent to learn the diverse weather conditions throughout the year, effectively avoiding model overfitting to a single season, and providing an independent and identically distributed sample basis for random sampling of S7. Figure 9 A comparison chart of CPU utilization during the training phase for the DVRL-SAC and SPEA2 algorithms is presented. Figure 9 The results show that, thanks to the improved sample utilization rate of the experience replay mechanism, although the CPU utilization rate of DVRL-SAC during training (16.41%) is higher than that of SPEA2 (7.96%), the overall computation time is reduced by 35.3%, and the deployment phase only takes 30 seconds. This demonstrates that the experience buffer, combined with large-batch sampling (batch size...), effectively improves the efficiency of sample utilization. While ensuring training diversity, it significantly improves computational efficiency.

[0100] S7: Randomly sample from the experience buffer and input the central value network and its target value network, the value networks of each agent and their target value networks, and synchronously update all network parameters. Specifically, this step synchronously updates all networks using a soft actor-critic algorithm based on hierarchical validation deep reinforcement learning: the central value network and the value networks of each agent update their respective network parameters by minimizing temporal difference errors; the central target value network and the target value networks of each agent update their respective network parameters using a soft update method; the policy network of each agent is optimized based on maximizing the policy gradient of the expected reward with entropy regularization; the training parameters are shown in Table 2, and the discount factor is... Soft update coefficient Experience replay pool capacity Batch size The training process consisted of 3000 rounds. A comparison between the DVRL-SAC algorithm and the SPEA2 algorithm verified the significant advantages of this update mechanism in global coordination and performance optimization. Centralized training supports global coordination: Figures 4 to 7 The electrolyzer power curves for DVRLSAC and SPEA2 under typical daytime conditions in spring, summer, autumn, and winter are shown respectively. Figures 4 to 7 As shown, under typical operating conditions in all four seasons, the electrolytic cell power curve produced by DVRLSAC is smooth and oscillating. Especially during the low-load period at night in winter (0-7h), this algorithm achieved a perfectly balanced output (7000kW) with a total standard deviation of 0, while SPEA2 distribution was unbalanced. This proves that the central value network successfully captured the global coordination signal during the centralized training phase, achieving accurate distribution among heterogeneous devices.

[0101] Improvement of overall performance by policy gradient optimization: Figure 8 The data shows that hydrogen production has increased across all four seasons (3.02% increase in winter and 1.63% increase in summer), indicating that the network update mechanism in S7 has balanced consumption efficiency and hydrogen production efficiency, achieving simultaneous growth of multiple objectives.

[0102] Computational efficiency and deployment performance: Figure 9 A comparison chart of CPU utilization during the training phase for the DVRL-SAC and SPEA2 algorithms is presented. Figure 9 The results show that, thanks to the gradient optimization mechanism, although DVRL-SAC's instantaneous CPU utilization during training (16.41%) is slightly higher than SPEA2's (7.96%), its total computation time is reduced by 35.3%. During the deployment phase, decisions can be made in just 30 seconds, and the CPU utilization drops to 9.51%, meeting the real-time control requirements of complex landscape scenes.

[0103] In conclusion, Figures 4 to 9 Together, it was demonstrated that the update mechanism of the soft actor-critic algorithm based on hierarchical verification deep reinforcement learning designed in S7 (double Q network, delayed soft update, entropy regularization, and experience replay) significantly outperforms the traditional SPEA2 algorithm in terms of convergence stability, multi-agent cooperation, overall hydrogen production performance, and computational efficiency. The specific implementation steps of S7 are as follows: (1) Construct a buffer zone. At each time step, the system constructs a storage transfer tuple consisting of the current state, the valid instructions verified by Level I security, the new state, the global-fusion reward, and the termination flag. And set the experience buffer capacity. and batch size .

[0104] (2) Randomly sample a batch of samples from the experience buffer, perform temporal difference learning, and calculate the target with entropy regularization. The value, expressed as follows:

[0105] In the formula, For the first The target of entropy regularization for each sample value; For the first Global-fusion reward for each sample; For the first The termination flag for each sample, where 1 indicates termination and 0 indicates no termination; Discount factor; For the first The action of sampling under the new state in each sample; For the first The new state of each sample; For the first Take action in the new state of each sample. The probability, is the entropy regularization coefficient. Wherein, the action at the next time step... The target value is obtained by reparameterized sampling, which is consistent with the continuous-discrete hybrid mapping mechanism in S3, ensuring the consistency between target value calculation and strategy generation.

[0106] (3) Update the value network parameters by minimizing the mean squared error loss, as shown in the following expression:

[0107] In the formula, For the first A value network loss function; For the first The estimated value of a state-action pair for each value network; For the first The target of each sample value.

[0108] (4) By maximizing the expected value The policy network parameters are updated by subtracting the policy entropy from the value, as shown in the following expression:

[0109] In the formula, Let the objective function of the policy network be denoted as . For policy network parameters; is the entropy regularization coefficient, used to balance the weight between the expected cumulative return of the policy and the policy entropy; For the first Take an action based on the current state of each sample. The probability; the output of the policy network is as described in S3, that is, the mean of a Gaussian distribution. and logarithmic standard deviation The specific action is obtained through reparameterized sampling.

[0110] (5) Automatically adjust the entropy regularization coefficient to maintain the target entropy level, as shown in the following expression:

[0111] In the formula, The loss function is the entropy regularization coefficient, used for automatic adjustment. To maintain the target entropy level; The target entropy value is typically set to a negative action dimension.

[0112] (6) Double Q The network uses a soft update method to synchronize its target value network parameters, as shown in the following expression:

[0113] In the formula, For the first One target value network parameter; For the first One current value network parameter; This is the soft update coefficient, which should be set to a positive number less than 1 to balance the stability of network parameter updates with the learning speed.

[0114] S8: Repeat S1 to S5 for multiple rounds of training and select several rounds to perform Level II model validation on the policy network. Those that pass are stored in the candidate set. Specifically, this step performs Level II model validation on the policy network of each agent every preset number of rounds (m rounds) during training. The Level II model validation criteria include whether the cumulative global reward converges, whether the cumulative local reward converges, and whether the cumulative instruction pass rate of the Level I safety validation is below a preset threshold (90%). After Level II model validation, multiple candidate models are selected. The effectiveness of Level II model validation is demonstrated by comparing the DVRL-SAC algorithm and the SPEA2 algorithm. Stability verification: Figures 4 to 7 The electrolyzer power curves for typical days in all four seasons show that the curves become smoother and less oscillating in the later stages of training (fluctuations of only 71.3kW in spring and 0 standard deviation at night in winter), proving that the reward mechanism has guided the model to converge to a stable range and meets the convergence condition.

[0115] Generalization capability verification: Figure 3 The solar power output curves for typical days in each season are given. Figure 8 The graphs show the 24-hour hydrogen production curves for typical days in spring, summer, autumn, and winter, as follows: Figure 3 and Figure 8 As shown, the selected candidate models perform well in all four seasons. Under extreme energy shortage conditions in winter, the efficient hydrogen production time of the algorithm in this invention accounts for 12.5% ​​(compared to 0% for the comparative algorithm), and the hydrogen production volume is significantly improved in all four seasons (3.02% improvement in winter), proving that the model does not suffer from seasonal overfitting and has good generalization ability across operating conditions.

[0116] Safety robustness: Figure 5 The period of severe fluctuations in wind power in mid-summer and Figure 7 During the extreme energy shortage period in midwinter, none of the candidate models triggered safety violations or system crashes, ensuring that the models met the prerequisites for entering the candidate set.

[0117] S9: After training, the candidate set is subjected to Level III deployment verification, and the optimal combination of policy network parameters is selected as the control model for final deployment. Specifically, the Level III deployment verification employs five independent test scenarios (historical typical day wind and solar power output, extreme typical day wind and solar power output, partial shutdown of energy storage batteries, partial shutdown of electrolyzers, and simultaneous partial shutdown of both energy storage batteries and electrolyzers). The optimal strategy network parameter combination is selected from the candidate set as the final deployment control model. In this embodiment, each scenario runs for 100 evaluation rounds (24 hours per round), and a normalized comprehensive score S is calculated based on wind and solar power absorption rate, hydrogen production efficiency, energy surplus rate, number of equipment switching times, and number of cycles. Figure 11 The bar charts show the combined scores of the DVRL-SAC and SPEA2 algorithms in five test scenarios. Figure 11The results show that DVRL-SAC outperforms SPEA2 in all scenarios. In typical historical daytime scenarios, DVRL-SAC achieved an overall score of 0.94, while SPEA2 scored 0.87, representing an 8.0% improvement. Under extreme and fault conditions, the advantages of this invention increase with complexity: performance improved by 17.1% in extreme weather scenarios and by 21.4% in scenarios with dual equipment outages. This demonstrates that the model still possesses strong adaptive adjustment capabilities and robustness even with partial system failure.

[0118] To further verify deployment efficiency Figure 9 The comparison of CPU utilization between the DVRL-SAC and SPEA2 algorithms during the training phase is shown. Figure 9 The results show that, thanks to the use of gradient information by the differentiable framework, although DVRL-SAC's instantaneous CPU utilization during training (16.41%) is slightly higher than that of traditional algorithms, the total computation time is reduced by 35.3%. Moreover, during the deployment phase, the model only needs 30 seconds to complete the optimization response, and the deployment CPU utilization is only 9.51%. This fully demonstrates the engineering advantage of DVRL-SAC: "train once, deploy multiple times".

[0119] In summary, the optimal parameter combination was selected as the optimal strategy network parameter combination after verification in Level III deployment. This model combines high production efficiency with robustness across operating conditions and was ultimately determined as the control model for the actual wind and solar hydrogen production system.

[0120] The formula for calculating the overall score is as follows:

[0121] In the formula, For average hydrogen production, This represents the average hydrogen production efficiency. The average consumption rate, This represents the average number of start-stop cycles. This represents the maximum number of start-stop cycles allowed. The standard deviation of hydrogen production. The standard deviation of efficiency; , , , , , These are the weighting coefficients for indicators such as wind and solar energy absorption rate, hydrogen production efficiency, energy surplus rate, number of times the electrolyzer operates under different conditions, and number of charge and discharge cycles of the energy storage battery.

[0122] As demonstrated by the above embodiments, the wind-solar hydrogen production system operation optimization method based on hierarchical verification deep reinforcement learning proposed in this application addresses key challenges such as the high volatility of multi-source renewable energy, multi-equipment collaborative scheduling, and adaptability to complex operating conditions. It innovatively constructs a multi-agent deep reinforcement learning framework based on hierarchical verification deep reinforcement learning, combining a continuous-discrete hybrid action mechanism and a global-local reward collaborative mechanism to achieve fine-grained optimization and control of core equipment such as energy storage batteries and electrolyzers. Simulation results under typical seasonal operating conditions verify that this method significantly outperforms traditional optimization algorithms in multiple indicators, including wind-solar absorption rate, hydrogen production efficiency, energy surplus rate, number of electrolyzer operating condition switching times, and number of energy storage battery charge-discharge cycles. It exhibits stronger capacity improvement and lower power fluctuations, especially under extreme winter conditions and highly volatile summer environments, fully demonstrating its superior adaptive optimization capabilities and engineering practical value. This method not only significantly improves the overall operational efficiency and intelligence level of wind-solar hydrogen production systems but also provides strong theoretical support and broad application prospects for multi-objective collaborative optimization and intelligent operation management of complex systems in the field of renewable energy hydrogen production.

[0123] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method for optimizing the operation of a wind and solar hydrogen production system based on hierarchical verification deep reinforcement learning.

[0124] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the above-described method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning.

[0125] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0126] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical validation deep reinforcement learning, characterized in that, include: S1: Collect environmental data and equipment information, and obtain the current state vector after feature extraction; S2: A heterogeneous multi-agent framework is constructed based on the soft actor-critic algorithm of hierarchical verification deep reinforcement learning. The energy storage battery and the electrolyzer are defined as independent agents. Each agent contains a policy network, a value network and a target value network, and a central value network and its target value network are added. S3: Receive the current state vector through each policy network, and output continuous and discrete instructions through continuous-discrete hybrid mapping; S4: Perform Level I security verification on the instruction. If the verification is successful, the valid instruction is executed. Repeat S1 to obtain a new state vector. S5: Based on the new state vector, the global reward and the fusion reward are calculated by the global-local fusion reward mechanism; S6: The current state vector, new state vector, valid instructions, global reward, and fusion reward are combined to form a sample and stored in the experience buffer. S7: Randomly sample from the experience buffer and input the central value network and its target value network, the value networks of each agent and their target value networks, and synchronously update all network parameters. S8: Repeat S1 to S5 for multiple rounds of training and select several rounds to perform Level II model validation on the policy network. Those that pass are stored in the candidate set. S9: After training, the candidate set is subjected to Level III deployment verification, and the optimal strategy network parameter combination is selected as the control model for final deployment.

2. The method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning according to claim 1, characterized in that, In S3, the strategy network of the energy storage battery agent simultaneously outputs continuous control commands for charging and discharging power, discrete control commands for charging / discharging direction, and discrete control commands for start / stop conditions. The strategy network of the electrolyzer agent simultaneously outputs continuous control commands for operating power and discrete control commands for operating / hot standby / cold standby / stop conditions.

3. The method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning according to claim 1, characterized in that, In S4, the Level I safety verification criteria include system power balance constraints, electrolytic cell continuous-discrete command conflict constraints, electrolytic cell maximum operating power constraints, electrolytic cell operating / hot standby / cold standby power constraints, energy storage battery continuous-discrete command conflict constraints, energy storage battery maximum power constraints, and energy storage battery state of charge constraints.

4. The method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning according to claim 1, characterized in that, In S5, the global reward represents three types of overall system performance: wind and solar power absorption rate, hydrogen production efficiency, and energy surplus rate. The fusion reward is obtained by weighting the global reward and local reward through a reward coordination factor driven by multi-index feedback. The reward coordination factor driven by multi-index feedback is generated by nonlinear mapping based on the multi-dimensional performance indicators of the system's operating status. The local reward of the electrolyzer represents two types of equipment behavior: electrolyzer power fluctuation and electrolyzer operating condition switching frequency. The local reward of the energy storage battery represents three types of equipment behavior: energy storage battery power fluctuation, energy storage battery charge and discharge cycle count, and energy storage battery overcharge and over-discharge.

5. The method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning according to claim 1, characterized in that, In S7, the central value network and the value networks of each agent update their respective network parameters by minimizing the temporal difference error. The target value network of the central value network and the target value networks of each agent update their respective network parameters by soft updating. The policy network of each agent updates its respective network parameters by maximizing the policy gradient of the expected return regularized by entropy regularization.

6. The method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning according to claim 1, characterized in that, In S8, the verification criteria for the Level II model include whether the cumulative global reward has converged, whether the cumulative local reward has converged, and whether the cumulative pass rate of the Level I security verification instruction is lower than a preset threshold.

7. The method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning according to claim 1, characterized in that, In S9, the Level III deployment verification is tested on independent datasets for five scenarios: historical typical day wind and solar power output, extreme typical day wind and solar power output, partial shutdown of energy storage batteries, partial shutdown of electrolyzers, and simultaneous partial shutdown of energy storage batteries and electrolyzers.

8. The method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning according to claim 1, characterized in that, The optimal strategy network parameter combination is the optimal combination of parameters after comprehensively considering five indicators: wind and solar power absorption rate, hydrogen production efficiency, energy surplus rate, number of electrolyzer operating condition switching times, and number of energy storage battery charge and discharge cycles.

9. The method for optimizing the operation of a wind-solar hydrogen production system based on hierarchical verification deep reinforcement learning according to claim 4, characterized in that, The reward component characterizing the power fluctuation behavior of the electrolytic cell in the local reward is calculated using a piecewise fluctuation barrier function. The reward component characterizing the power fluctuation behavior of the energy storage battery in the local reward is calculated using a piecewise fluctuation barrier function. The reward component characterizing the overcharge and over-discharge behavior of the energy storage battery in the local reward is calculated using a state-of-charge safety barrier function.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

Images