Hydropower station primary frequency modulation governor parameter optimization method based on reinforcement learning
By constructing a hydroelectric coupling simulation model and optimizing governor parameters through reinforcement learning, the problems of regulation lag and frequency instability in traditional methods under uneven operating conditions were solved, and efficient regulation of hydropower units under multiple operating conditions and grid frequency stability were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NORTHWEST A & F UNIV
- Filing Date
- 2026-04-22
- Publication Date
- 2026-06-05
AI Technical Summary
Existing methods for optimizing the parameters of hydropower unit governors are ineffective in dealing with high-frequency and extreme conditions when the operating conditions are unevenly distributed, resulting in regulation lag and frequency instability, which makes it difficult to meet the requirements of grid frequency stability and economic assessment.
A water-mechanical-electric coupling simulation model is constructed. The governor parameters are optimized through reinforcement learning. The head and initial power are used as state inputs, PID parameters are used as action outputs, and the first frequency regulation pass rate is used as the reward signal. The reinforcement learning model is trained to achieve the optimal strategy and adaptively adjust the governor parameters.
It enables adaptive dynamic adjustment of governor parameters, improves the primary frequency regulation qualification rate and power regulation stability of hydropower units across the entire operating range, and enhances the safety and reliability of the power grid frequency.
Smart Images
Figure CN122159395A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of hydropower unit speed control technology, specifically relating to a method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning. Background Technology
[0002] Hydropower units play a crucial role in new power systems, undertaking key tasks such as frequency regulation, peak shaving, and emergency support. Their operation frequently involves small-amplitude power adjustments and switching between multiple operating conditions. However, existing governor parameters are often optimized based on the assumption of a uniform operating condition distribution, which deviates significantly from reality. This results in insufficient influence of high-frequency and extreme operating conditions on parameter selection, potentially leading to regulation lag or unstable frequency recovery in certain operating ranges. Furthermore, traditional PID parameter determination relies on manual experience or single-objective search algorithms, failing to achieve global optimization.
[0003] In summary, traditional governor parameter optimization methods have limitations in terms of multi-dimensional performance balance and operating condition adaptability. These limitations not only affect grid frequency stability but also fail to meet the stringent assessment requirements of the "two detailed rules," thereby harming the revenue from power plant frequency regulation ancillary services. Therefore, it is urgent to overcome the limitations of traditional methods and achieve a balance between grid regulation needs and the safe and stable operation of generating units. Summary of the Invention
[0004] The purpose of this invention is to propose a method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning, in order to solve the problems in the prior art.
[0005] Therefore, this invention provides a method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning, comprising: Construct a water-mechanical-electric coupling simulation model; Frequency perturbations were applied to the hydro-mechanical coupling simulation model under different operating conditions, and the primary frequency regulation performance index under each operating condition was calculated. Using the aforementioned water-mechanical-electric coupling simulation model as the environment, water head and initial power as state inputs, speed governor control parameters as action outputs, and the aforementioned primary frequency regulation performance index as reward signals, a reinforcement learning model is constructed. The reinforcement learning model is trained using a preset algorithm to obtain the optimal policy, which can output the optimal control parameters based on the current state. The optimal strategy was then applied to the actual hydropower station speed regulation system.
[0006] In some embodiments, the construction of the hydro-mechanical coupling simulation model includes: A water diversion system model was established based on the transfer function method. A turbine model is established based on the full characteristic curves of the turbine and through interpolation. A synchronous generator model was built using Simulink. The water diversion system model, turbine model, and generator model are integrated, and the initial parameter of the governor is set to the proportional coefficient K. p =1.0, integral coefficient K i =1.0, differential coefficient K d =0.0.
[0007] In some implementations, frequency perturbations are applied to the hydro-mechanical coupling simulation model under different operating conditions, and the primary frequency regulation performance index under each operating condition is calculated, including: Multiple operating conditions are determined, which are jointly defined by the head and the initial power. A uniform frequency perturbation is applied to the hydro-turbine coupling simulation model under the aforementioned operating conditions to obtain the active power response of the unit under the corresponding operating conditions. The primary frequency modulation pass rate is calculated based on the target frequency deviation and power recovery characteristics.
[0008] In some implementations, the primary frequency modulation pass rate is calculated based on the target frequency deviation and power recovery characteristics using the following formula:
[0009]
[0010]
[0011] Among them, W the and W prac P represents the theoretical integrated energy and the actual integrated energy, respectively; r P0 and P0 represent rated power and initial power, respectively; Δf represents frequency deviation; f r Indicates the rated frequency; e p represents the rotational speed unequal rate; D represents the first frequency regulation pass rate.
[0012] In some implementations, performance zones are formed based on the primary frequency modulation pass rate, with 60% as the primary frequency modulation pass threshold; The operating condition where the primary frequency regulation pass rate is higher than 60% is the qualified zone; otherwise, it is the assessment zone.
[0013] In some implementations, the transition input of the reinforcement learning model includes head and initial power, and the transition vector is:
[0014] Where h is the operating head; P0 is the initial power; The action output of the reinforcement learning model is a PID controller parameter, defined as:
[0015] Among them, K p K is the proportionality coefficient. i K is the integral coefficient. d These are the differential coefficients; The reward function is defined as:
[0016] Wherein, PFR is the first frequency modulation pass rate, Tr is the settling time, and Ts is the settling time.
[0017] In some implementations, the objective function of the preset algorithm is:
[0018] in, The probability ratio of the strategy; This is the advantage function estimate, used to characterize the degree of improvement of the current action relative to the average performance.
[0019] In some implementations, the preset algorithm combines the value function and the entropy regularization term to form an overall optimization objective function, which is:
[0020] Among them, V θ (s) t Let S[π] be the state value function. θ ] represents the policy entropy term, used to balance exploration and convergence.
[0021] Beneficial effects: This invention establishes a hydroelectric coupling simulation model and quantitatively evaluates and partitions the frequency regulation performance under different head and initial power conditions. It introduces a reinforcement learning agent and environment interaction mechanism into the governor parameter optimization process, realizing the adaptive dynamic adjustment of governor parameters according to operating conditions. This effectively solves the systemic problem that traditional fixed parameter governors are difficult to balance regulation safety, dynamic response and regulation accuracy under multiple operating conditions and multiple indicators. It significantly improves the primary frequency regulation qualification rate of hydropower units in the whole operating range, enhances the stability of power regulation process, and supports the safe and reliable operation of the power grid frequency. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 A flowchart of a primary frequency regulator parameter optimization method provided in an embodiment of the present invention; Figure 2 This is a zoning diagram of the primary frequency modulation performance before optimization, provided in an embodiment of the present invention. Figure 3 This is an optimized primary frequency modulation performance partitioning diagram provided in an embodiment of the present invention. Detailed Implementation
[0024] The invention will be more readily understood by referring to the following detailed description of preferred embodiments and included examples. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. In case of conflict, the definitions in this specification shall prevail.
[0025] This invention provides a method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning, and describes the technical concept.
[0026] See Figure 1 This invention provides a method for optimizing the primary frequency regulation performance of a hydropower station based on reinforcement learning, comprising the following steps: A hydro-turbine-electric coupling simulation model was established. This model is coupled with the water diversion system, turbine, and generator. Specifically: A hydroelectric coupling model is constructed using a water intake system modeled with the transfer function method, interpolated turbine full characteristic curves, and a synchronous generator model built with Simulink. This model accurately simulates the water hammer effect and pressure fluctuation dynamics of the water intake system using the transfer function method, comprehensively covers the unit's full operating characteristics using interpolated turbine full characteristic curves, and integrates a synchronous generator model built with Simulink to accurately reflect the transient processes of electromechanical coupling and grid interaction. The constructed hydroelectric coupling simulation model can faithfully reproduce the dynamic response of the entire energy conversion chain from hydraulic to mechanical to electrical in a hydropower station. This provides an accurate, reliable, and comprehensive digital simulation environment covering the entire operating range for subsequent performance evaluation and reinforcement learning training, effectively supporting the credibility and practicality of governor parameter optimization research.
[0027] Set unified governor parameters for the hydro-mechanical coupling model under different operating conditions. The governor parameters include: K P The value is 1.0, K I The value is 1.0, K D The value is set to 0.0. By pre-setting a unified set of governor parameters for the hydro-mechanical coupling model under all different head and initial power conditions, the model adopts the same control strategy under different operating conditions. This allows for the unbiased revelation of the inherent performance differences and limitations of the traditional fixed-parameter method under different operating conditions. This facilitates the subsequent application of a unified frequency perturbation to obtain simulation results and performance partitioning, intuitively revealing the specific areas and extent to which parameter fixation leads to low primary frequency regulation pass rates under certain operating conditions.
[0028] A uniform frequency perturbation was applied to the hydraulic-electric coupling simulation model under different operating conditions. The primary frequency regulation qualification rate was calculated for each operating condition, and performance partitioning was formed based on the primary frequency regulation qualification rate. Figure 2 This section details the process of applying standardized frequency perturbations to the hydro-mechanical coupling simulation model under all operating conditions. Based on the comparison between theoretical and actual integrated power outputs under each perturbation, the primary frequency regulation pass rate for each operating condition is accurately calculated. This achieves a standardized and quantitative evaluation of the fixed-parameter governor's performance across the entire operating range.
[0029] Specifically, based on the operating characteristics of the hydropower unit, multiple operating conditions are determined. Each operating condition is jointly defined by the head and initial power. A uniform frequency disturbance is applied to the hydro-turbine coupling simulation model under each operating condition to obtain the active power response of the unit under different operating conditions. The primary frequency regulation qualification rate is calculated based on the target frequency deviation and power recovery characteristics. The calculation formula is as follows:
[0030]
[0031]
[0032] In the formula, and These represent the theoretical integrated energy and the actual integrated energy, respectively. and These represent rated power and initial power, respectively. Indicates frequency deviation; Indicates the rated frequency; Indicates the rotational speed inequality; This indicates the first frequency modulation pass rate.
[0033] Simultaneously, performance zoning is established based on the primary frequency regulation pass rate, with 60% as the pass threshold. Operating conditions with a primary frequency regulation pass rate above 60% are classified as the pass zone, otherwise as the assessment zone. Specifically, performance zoning transforms abstract numerical calculation results into intuitive and visual performance distribution maps. This not only clearly reveals the strength and weakness distribution and spatial patterns of the hydropower unit's primary frequency regulation performance across the entire operating range under fixed parameters, intuitively locating the weak links and specific operating conditions where performance fails to meet existing control strategies, but also provides clear and focused target areas for subsequent optimization work. This guides the reinforcement learning agent to prioritize exploration and learning in the assessment zone where performance is weak, enabling efficient allocation of optimization resources and ultimately ensuring that the optimized parameter strategy can systematically and purposefully improve the overall performance level. The hydroelectric coupling simulation model is encapsulated as a reinforcement learning agent, i.e., a reinforcement learning model. The water head and initial power are used as observations, i.e., state inputs, PID parameters are used as action outputs, the first frequency modulation pass rate is used as a reward signal, and the adjustment time and settling time are used as constraints. By encapsulating the hydro-mechanical coupling simulation model into a reinforcement learning environment, and using an interactive framework with head and initial power as state inputs, PID parameters as action outputs, first-time frequency regulation pass rate as the core reward signal, and adjustment time and settling time as dynamic performance constraints, a learning and feedback training platform capable of simulating real control processes has been successfully constructed.
[0034] This enables the intelligent agent to explore and learn under the constraints of meeting dynamic process requirements, based on the current operating conditions, using parameter adjustment as the execution means, and the frequency regulation qualification rate as the direct optimization goal. Thus, the complex multi-condition, multi-objective parameter optimization problem is transformed into a sequential decision-making problem that can be solved autonomously through data-driven learning. This allows the intelligent agent to autonomously discover and approximate the optimal PID parameter combination under different operating conditions without relying on human experience or preset rules, ultimately achieving adaptive speed controller parameters.
[0035] Specifically, the agent takes the head and initial power as state inputs, and uses the PID parameter K... p K i K d For the action output, based on the first frequency modulation pass rate, As a reward signal, and constrained by settling time and settling time, the agent's state input includes head and initial power, and the state vector is defined as follows:
[0036] Where h is the operating head (in meters) and P0 is the initial power (in megawatts). The action output is a PID controller parameter, defined as:
[0037] Among them, K p K is the proportionality coefficient. i K is the integral coefficient. d These are the differential coefficients; The reward function is defined as:
[0038] Where PFR is the first frequency modulation pass rate, T r To adjust the time, T s For stable time.
[0039] Then, the near-end strategy optimization PPO algorithm is used to train the agent in a reinforcement learning environment to obtain the optimal governor parameters under different operating conditions. The trained strategy is then embedded into the hydropower station speed regulation system to adjust the governor parameters and obtain the optimized primary frequency regulation qualification rate.
[0040] By leveraging its policy probability ratio pruning mechanism and advantage function estimation, the algorithm effectively stabilizes the training process and encourages continuous policy updates towards maximizing cumulative rewards. Simultaneously, the value function and policy entropy regularization introduced by the algorithm balance long-term rewards with policy exploration during optimization, enabling the agent to efficiently and robustly learn a nonlinear mapping policy from head and initial power states to optimal PID parameters. The knowledge gained from preliminary simulations, evaluations, and partitioning is transformed into a practically deployable parameter decision function with generalization capabilities.
[0041] After embedding the optimal strategy obtained from training into the hydropower station's speed regulation system, the system can automatically and quickly call upon the optimal governor parameters that match the real-time monitored operating conditions (head, power), realizing an improvement in governor parameters from fixed settings to online adaptive adjustment. The system applying the new strategy was then subjected to full-condition simulation testing, and the optimized primary frequency regulation pass rate was calculated. The significant reduction in the test area and the overall improvement in the full-condition pass rate directly validated this approach, demonstrating that it systematically solves the inherent defect of traditional methods in failing to consider dynamic performance under multiple operating conditions, and substantially improves the reliability and overall performance level of hydropower units participating in grid primary frequency regulation.
[0042] Specifically, the objective function of the algorithm is defined as:
[0043] The probability ratio of the strategy; This is the advantage function estimate, used to characterize the degree of improvement of the current action relative to the average performance.
[0044] The algorithm comprehensively considers the value function and the entropy regularization term, and its overall optimization objective function is:
[0045] Among them, V θ (s) t Let S[π] be the state value function. θ The term ] represents the policy entropy, used to balance exploration and convergence. The optimized partitioning result is as follows: Figure 3 As shown.
[0046] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning, characterized in that, include: Construct a water-mechanical-electric coupling simulation model; Frequency perturbations were applied to the hydro-mechanical coupling simulation model under different operating conditions, and the primary frequency regulation performance index under each operating condition was calculated. Using the aforementioned water-mechanical-electric coupling simulation model as the environment, water head and initial power as state inputs, speed governor control parameters as action outputs, and the aforementioned primary frequency regulation performance index as reward signals, a reinforcement learning model is constructed. The reinforcement learning model is trained using a preset algorithm to obtain the optimal policy, which can output the optimal control parameters based on the current state. The optimal strategy was then applied to the actual hydropower station speed regulation system.
2. The method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning according to claim 1, characterized in that, The construction of the hydroelectric coupling simulation model includes: A water diversion system model was established based on the transfer function method. A turbine model is established based on the full characteristic curves of the turbine and through interpolation. A synchronous generator model was built using Simulink. The water diversion system model, turbine model, and generator model are integrated, and the initial parameter of the governor is set to the proportional coefficient K. p =1.0, integral coefficient K i =1.0, differential coefficient K d =0.
0.
3. The method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning according to claim 1, characterized in that, Frequency perturbations were applied to the hydro-mechanical coupling simulation model under different operating conditions, and the primary frequency regulation performance index under each operating condition was calculated, including: Multiple operating conditions are determined, which are jointly defined by the head and the initial power. A uniform frequency perturbation is applied to the hydro-turbine coupling simulation model under the aforementioned operating conditions to obtain the active power response of the unit under the corresponding operating conditions. The primary frequency modulation pass rate is calculated based on the target frequency deviation and power recovery characteristics.
4. The method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning according to claim 3, characterized in that, The primary frequency modulation pass rate, based on the target frequency deviation and power recovery characteristics, is calculated using the following formula: Among them, W the and W prac P represents the theoretical integrated energy and the actual integrated energy, respectively; r P0 and P0 represent rated power and initial power, respectively; Δf represents frequency deviation; f r Indicates the rated frequency; e p represents the rotational speed unequal rate; D represents the first frequency regulation pass rate.
5. The method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning according to claim 4, characterized in that, Performance zones are formed based on the primary frequency modulation pass rate, with 60% as the primary frequency modulation pass threshold. The operating condition where the primary frequency regulation pass rate is higher than 60% is the qualified zone; otherwise, it is the assessment zone.
6. The method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning according to claim 1, characterized in that, The transition inputs of the reinforcement learning model include the head and the initial power, and the transition vector is: Where h is the operating head; P0 is the initial power; The action output of the reinforcement learning model is the PID controller parameter, defined as: Among them, K p K is the proportionality coefficient. i K is the integral coefficient. d These are the differential coefficients; The reward function is defined as: Where PFR is the first frequency modulation pass rate, T r To adjust the time, T s For stable time.
7. The method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning according to claim 1, characterized in that, The objective function of the preset algorithm is: in, The probability ratio of the strategy; This is the advantage function estimate, used to characterize the degree of improvement of the current action relative to the average performance.
8. The method for optimizing the parameters of a primary frequency regulator in a hydropower station based on reinforcement learning according to claim 7, characterized in that, The preset algorithm combines the value function and the entropy regularization term to form an overall optimization objective function, which is: Among them, V θ (s) t Let S[π] be the state value function. θ ] represents the policy entropy term, used to balance exploration and convergence.