Hydropower station peak regulation operation strategy optimization method, system and equipment based on reinforcement learning and storage medium
By using a digital twin environment and agent training based on reinforcement learning, the problems of real-time decision-making and optimal strategy in the peak-shaving operation of hydropower stations were solved, and the efficient and stable operation of hydropower stations in complex environments was achieved.
Patent Information
- Application Number
- CN202511779845.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-03-20
AI Technical Summary
Existing technologies are unable to quickly and accurately formulate peak-shaving operation strategies for hydropower stations that meet real-time needs while also taking into account long-term benefits in complex and ever-changing environments. In particular, they are unable to achieve real-time decision-making and optimal strategies when facing fluctuations in renewable energy output, changes in market prices, and sudden grid events.
A reinforcement learning-based approach is adopted to construct a digital twin environment and design a reinforcement learning algorithm suitable for peak-shaving operation of hydropower stations. The agent is trained in the digital twin environment to obtain data in real time and generate the optimal scheduling strategy. By combining offline training and online fine-tuning, the strategy can achieve self-optimization and adaptability.
It enables rapid generation of scheduling decisions in complex and ever-changing environments, balances short-term and long-term interests, maximizes power generation revenue, improves the operational stability and economic benefits of hydropower stations, and eliminates the dependence on fixed models and prior knowledge.
Smart Images
Figure CN121707038A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of power system optimization scheduling and artificial intelligence applications, and in particular to a method, system, equipment and storage medium for optimizing peak-shaving operation strategies of hydropower stations based on reinforcement learning. Background Technology
[0002] With the large-scale integration of new energy sources into the power grid, the dramatic fluctuations in power output and the instantaneous changes in market prices place higher demands on the peak-shaving operation strategies of hydropower stations. How to quickly and accurately formulate peak-shaving operation strategies that meet both real-time needs and long-term benefits in a complex and ever-changing environment has become a key issue that needs to be addressed.
[0003] Current peak-shaving operation strategies for hydropower stations mainly include static rule-based scheduling methods, which pre-determine scheduling diagrams or curves based on historical data and the experience of dispatchers. However, this method is rigid and lagging, and cannot respond quickly to dynamic conditions such as fluctuations in renewable energy output, changes in market prices, and sudden grid events. It also includes model-based methods based on traditional optimization, such as dynamic programming and linear programming. Although these methods attempt to establish accurate mathematical models to solve for optimal solutions, the computation time increases exponentially when facing complex problems such as hydropower dispatch, which are high-dimensional, nonlinear, and highly random, making it difficult to meet the real-time decision-making requirements of the electricity market. Summary of the Invention
[0004] In view of the above-mentioned problems, the present invention provides a method, system, equipment and storage medium for optimizing the peak-shaving operation strategy of hydropower stations based on reinforcement learning.
[0005] Therefore, the technical problem solved by this invention is: how to overcome the dilemma that existing technologies cannot simultaneously achieve real-time decision-making, optimal strategy, and long-term goals, and to provide a method for optimizing the peak-shaving operation strategy of hydropower stations that can perceive dynamic changes in real time, automatically adjust scheduling strategies, balance short-term and long-term interests, and maximize economic benefits.
[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for optimizing the peak-shaving operation strategy of a hydropower station based on reinforcement learning, comprising: Collect data related to hydropower station operation, power grid and market data, and combine them with operational constraints to build a digital twin environment; Based on the constructed digital twin environment, a reinforcement learning algorithm suitable for peak-shaving operation of hydropower stations is designed. Initialize the agent in the digital twin environment, train the agent in the constructed digital twin environment using reinforcement learning algorithms, and optimize it based on the training results; Real-time data acquisition of hydropower station operation, power grid load, and electricity price data is used to input the trained and optimized agent to generate the optimal scheduling strategy for the current moment.
[0007] As a preferred scheme for optimizing the peak-shaving operation strategy of hydropower stations based on reinforcement learning, the following is proposed: The collection of hydropower station operation-related data, power grid and market-related data, and the construction of a digital twin environment in conjunction with operational constraints include: Based on collected data related to hydropower station operation, power grid and market data, and combined with operational constraints, a physical model of the hydropower station is established. The physical model of the hydropower station reflects the water balance of the reservoir, the output characteristics of the generator units, the flood discharge capacity, and simulates the dynamic changes of the external environment.
[0008] As a preferred scheme for optimizing the peak-shaving operation strategy of hydropower stations based on reinforcement learning, the following is proposed: The collection of hydropower station operation-related data, power grid and market-related data, and the construction of a digital twin environment in conjunction with operational constraints also include: Define a state space that encompasses all the information needed to describe key environmental information, including the operating status of the hydropower station and external environmental information. Define an action space, which is a decision vector that the agent can execute. For the peak shaving problem of hydropower stations, the actions are continuous. Design a reward function to guide the agent's learning, balance short-term economic benefits with long-term operational goals, provide positive incentives for power generation revenue, and punish water abandonment, deviation from ecological flow, and violation of operational constraints.
[0009] As a preferred scheme for optimizing the peak-shaving operation strategy of hydropower stations based on reinforcement learning, the following is proposed: Based on the constructed digital twin environment, the reinforcement learning algorithm designed for the peak-shaving operation of the hydropower station includes: Based on the constructed digital twin environment, a reinforcement learning algorithm suitable for peak-shaving operation of hydropower stations is designed. Taking the current state as input, the algorithm outputs an action or the probability distribution of the action, which is a refined operation strategy for one or more future scheduling cycles, including the unit output vector and the total discharge flow.
[0010] As a preferred scheme for optimizing the peak-shaving operation strategy of hydropower stations based on reinforcement learning, the following is proposed: The reinforcement learning algorithm designed for peak-shaving operation of hydropower stations based on the constructed digital twin environment also includes: Reinforcement learning algorithms employ stochastic policies, where the agent selects actions according to a certain probability distribution in each state. For discrete and continuous action spaces, the policy function returns the probability distribution of actions or the action value and the probability distribution parameters of the actions, respectively.
[0011] As a preferred scheme for optimizing the peak-shaving operation strategy of hydropower stations based on reinforcement learning, the following is proposed: The reinforcement learning algorithm designed for peak-shaving operation of hydropower stations based on the constructed digital twin environment also includes: The quality of a state or state-action pair is evaluated using value functions, including a state value function that measures the expected future cumulative reward of following a policy in a given state, and an action value function that measures the expected future cumulative reward of following a policy after choosing an action in a given state.
[0012] The beneficial effects of this preferred technical solution are as follows: the introduction of the value function provides a quantitative indicator for the agent to evaluate the quality of states and state-action pairs. The state value function and action value function can help the agent better understand the long-term effects of different states and actions, thereby guiding the agent to choose better actions, improving the quality of the strategy and the learning efficiency of the agent.
[0013] As a preferred scheme for optimizing the peak-shaving operation strategy of hydropower stations based on reinforcement learning, the following is proposed: The process of initializing the agent in a digital twin environment, training the agent using a reinforcement learning algorithm within the constructed digital twin environment, and optimizing it based on the training results includes: Using historical operational data and a pre-built environment simulator, the agent is trained offline. During the training process, the agent continuously iterates and updates the neural network parameters through simulated scheduling operations until the policy reaches a convergent state. The latest real-time data is then used to fine-tune the agent that has completed offline training online.
[0014] The beneficial effects of this preferred technical solution are as follows: Offline training utilizes historical data and environmental simulators, enabling the agent to learn and optimize strategies in a large number of simulated scenarios, thereby improving the stability and reliability of the strategies. Online fine-tuning, combined with the latest real-time data, allows the agent to quickly adapt to environmental changes, maintaining the effectiveness and timeliness of the strategies, and ensuring that the hydropower station can achieve optimal peak-shaving operation under different environments.
[0015] Secondly, the present invention provides a reinforcement learning-based hydropower station peak-shaving operation strategy optimization system, comprising: The digital twin environment construction module is used to collect data related to hydropower station operation, power grid and market data, and combine them with operational constraints to construct a digital twin environment. The reinforcement learning algorithm design module is used to design reinforcement learning algorithms suitable for peak-shaving operation of hydropower stations based on the constructed digital twin environment. The agent training and optimization module is used to initialize the agent in the digital twin environment, use reinforcement learning algorithms to train the agent in the constructed digital twin environment, and optimize it based on the training results. The real-time scheduling strategy generation module is used to acquire hydropower station operation data, power grid load data, and electricity price data in real time, and input them into the trained and optimized agent to generate the optimal scheduling strategy for the current moment.
[0016] Thirdly, the present invention provides a computer device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method for optimizing the peak-shaving operation strategy of hydropower stations based on reinforcement learning are implemented.
[0017] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of a reinforcement learning-based method for optimizing the peak-shaving operation strategy of a hydropower station.
[0018] The beneficial effects of this invention are as follows: This invention does not rely on fixed rules or prior models. After offline training, the model can quickly generate scheduling decisions during runtime, with a computation speed far exceeding that of traditional algorithms. It can promptly respond to dynamic operating conditions such as drastic fluctuations in new energy output, instantaneous changes in market prices, and sudden grid events. By balancing short-term and long-term rewards through a time discount factor, it finds the optimal balance between short-term peak-shaving demand and long-term water resource utilization, avoiding the problems of sacrificing strategy optimality or real-time decision-making in traditional methods. The model can sense changes in electricity prices in real time and dynamically adjust peak-shaving strategies to maximize power generation revenue. At the same time, by utilizing the online exploration and learning capabilities of reinforcement learning, the scheduling strategy can self-optimize and continuously evolve based on actual feedback data, freeing it from dependence on fixed models and prior knowledge, improving the robustness and adaptability of the strategy, thereby ensuring the stable and efficient operation of hydropower stations in complex and ever-changing market environments. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1This is an overall flowchart of a reinforcement learning-based optimization method for peak-shaving operation strategy of hydropower stations provided by the present invention. Detailed Implementation
[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.
[0022] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a method for optimizing the peak-shaving operation strategy of a hydropower station based on reinforcement learning, including: S1: Collect data related to hydropower station operation, power grid and market data, and combine them with operational constraints to build a digital twin environment; S2: Based on the constructed digital twin environment, design a reinforcement learning algorithm suitable for peak-shaving operation of hydropower stations; S3: Initialize the agent in the digital twin environment, use reinforcement learning algorithms to train the agent in the constructed digital twin environment, and optimize it based on the training results; S4: Real-time acquisition of hydropower station operation data, power grid load data, and electricity price data, inputting them into the trained and optimized agent to generate the optimal scheduling strategy for the current moment.
[0023] It should be noted that, through S1-S4, from data collection and environment construction to algorithm design, agent training and optimization, and finally to the generation of real-time scheduling strategies in practical applications, a complete and closed-loop intelligent scheduling system for hydropower station peak-shaving operation is formed. Utilizing a digital twin environment and reinforcement learning algorithms, the agent can continuously learn and adapt to the complex and ever-changing operating environment and market demands of the hydropower station, effectively improving the decision-making efficiency and accuracy of peak-shaving operation. This provides strong support for achieving efficient, stable, and economical operation of hydropower stations, helps enhance their competitiveness in the electricity market, and better meets the peak-shaving needs of the power grid and the stability requirements of social electricity consumption.
[0024] Example 2, refer to Figure 1 As an embodiment of the present invention, based on the previous embodiment, a method for optimizing the peak-shaving operation strategy of a hydropower station based on reinforcement learning is provided, including: In this embodiment, the step S1 above, which involves collecting hydropower station operation-related data, power grid and market-related data, and combining them with operational constraints to construct a digital twin environment, includes: Hydropower station operation-related data includes: Hydrological conditions: inflow, rainfall, evaporation, etc.
[0025] Reservoir status: real-time water level, reservoir capacity curve, etc.
[0026] Generator status: number of units, maximum and minimum generating power, etc.
[0027] Data related to power grids and markets includes: Power grid load demand: Real-time load demand curve.
[0028] Electricity market pricing: real-time electricity price, peak-valley pricing rules, ancillary service market prices, etc.
[0029] Fluctuations in renewable energy output: Optional input, real-time forecast data of renewable energy power generation such as wind power and photovoltaic power.
[0030] Operational constraints include: Dispatch constraints: upper and lower limits of reservoir water level, flood discharge constraints, ecological flow requirements, etc.
[0031] Safety constraints: water level fluctuation limits, flood control requirements, equipment operation safety restrictions, etc.
[0032] Based on the collected information, a physical model of the hydropower station was established, including the water balance equation of the reservoir, the output curve of the generator unit, and the flood discharge capacity curve.
[0033] Simulate the external environment, including dynamic changes such as power grid load demand curves, electricity price fluctuations, and hydrological conditions.
[0034] In another possible implementation, a physical model of the hydropower station can be established based on the principles of system dynamics. The various components of the hydropower station, such as the reservoir, generator units, and flood discharge facilities, are considered as interconnected subsystems. The operation of the hydropower station is described by establishing causal relationships and feedback mechanisms between these subsystems. For example, the impact of changes in reservoir water level on generator output, and the feedback effect of changes in generator output on the power grid load, can be analyzed to establish corresponding mathematical models.
[0035] In another possible implementation, when building the physical model of a hydropower station, deep learning algorithms from machine learning, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), can be used to learn and model the operational data of the hydropower station. By training on a large amount of historical data, the model can automatically learn the physical characteristics and operational patterns of the hydropower station, thereby establishing a physical model of the hydropower station. For example, CNNs can be used to analyze image data of the hydropower station to identify reservoir water levels, unit operating status, etc.; RNNs can be used to process time series data to predict the future operation of the hydropower station.
[0036] Furthermore, define the state space: Let the state be... S t The key information used to describe the environment, including all information needed for decision-making such as the operating status of the hydropower station and external environmental information, is defined as: S t ={ W t , Q in,t , L t , P t , T t} in, W t This is the current water level. Q in,t This represents the current inbound flow. L t To meet the power grid load demand, P t For market electricity prices, T t This is the information for the current moment.
[0037] Define the action space: Let the action... A t It is the decision vector that the agent can execute. For the peak-shaving problem of a hydropower station, the action space is usually continuous and is defined as: A t ={ P gen,t , Q spill,t} in, P gen,t It is the total power generation capacity of the hydropower station. Q spill,t It is the discharge flow rate.
[0038] Design the reward function: Reward function R t It is the core of guiding the learning direction of intelligent agents, and it requires balancing short-term economic benefits with long-term operational goals, expressed as: R t =α ⋅ R evenuet −β ⋅ P enaltyspill −γ ⋅ P enaltyeco −d ⋅P enaltyconstraint The dimensions and orders of magnitude of each component in the reward function need to be consistent. Specifically, R evenuet It is the revenue from electricity generation, calculated based on the recent average market electricity price. P mean and power station rated capacity P rated Normalize, R evenuet =( P gen , t×P t ) / ( P rated × P mean The value range is [0,1]. P enaltyspill It is a punishment for the act of wasting water, and the amount of water wasted will be... Q spill Utilizing the reservoir's maximum water discharge capacity Q spillmax Normalize, P enaltyspill =( Q spill / Q spillmax ) 2 ; P enaltyeco It is a penalty for deviating from the minimum ecological flow. P enaltyconstraint It is a penalty for violating operational constraints such as water level and power output; α , β , c , d Let be an adjustable weighting coefficient. i =[ α , β , c , d The initial value of the weight coefficient can be set to 1. The weight is adjusted through a trigger mechanism. Specifically, the trigger condition is that the system continuously monitors operating indicators. The weight adjuster is triggered when the following conditions occur: Market pattern shift: such as from "flat period" to "peak period" or "valley period", the characteristics of electricity prices change significantly.
[0039] Seasonal variations: abundant water during the flood season or scarce water during the dry season.
[0040] Target preference adjustment: The superior dispatching department issued new dispatching instructions, emphasizing different objectives such as ensuring power supply, protecting the ecology, or maintaining water levels.
[0041] Performance degradation: The recent average cumulative reward has consistently fallen below the preset threshold.
[0042] The weight coefficients can be adjusted using a meta-learning method based on policy gradients: ① Offline training generates performance corresponding to different weights, forming a prior knowledge base; ② When adjustment is triggered, the system retrieves high-performance weights from the knowledge base for similar scenarios as the initial point based on the current environmental state (such as high electricity price, low water level); ③ Subsequently, on real-world data within a short time window, perform a small number of online gradient updates. The update direction is to increase the weights of behaviors that have recently made positive contributions to cumulative rewards and decrease the weights of behaviors that result in penalties; ④ Update weights: i new = i + or *∇θ J ( i ),in i new = [ α ', β ', c ', d '] represents the updated weight. J ( i () represents recent accumulated rewards. or It has a very small learning rate to ensure smooth adjustment.
[0043] In another possible implementation, building a digital twin environment can also leverage big data analytics to deeply mine and analyze collected data related to hydropower station operation, power grid, and market conditions. This allows for the extraction of potential patterns and characteristics from the data, which, combined with operational constraints, can then be used with specialized digital twin modeling software to construct a highly realistic and interactive digital twin environment. For example, by analyzing years of hydrological data, predictions can be made about reservoir water level changes under different seasonal and weather conditions. These predictions can then be integrated into the digital twin environment to more accurately reflect the actual situation.
[0044] In another possible implementation, building a digital twin environment can also employ multi-sensor fusion technology. This involves deploying various types of sensors at key locations within the hydropower station to acquire real-time operational data, such as water level, flow rate, and unit temperature. Simultaneously, real-time data from the power grid and the market is collected, fused, and then, combined with operational constraints, used cloud computing and edge computing technologies to construct a distributed digital twin environment. This approach improves data acquisition efficiency and processing speed, enabling the digital twin environment to respond more promptly to changes in actual conditions.
[0045] In this embodiment, the reinforcement learning algorithm designed based on the constructed digital twin environment in step S2 above, suitable for the peak-shaving operation of the hydropower station, includes: Based on the constructed digital twin environment, a reinforcement learning algorithm suitable for peak-shaving operation of a hydropower station is designed, with the current state as the input. S t (Such as reservoir water level, power grid load, electricity price, etc.), the output is an action. A t (e.g., power generation, water wastage) or the probability distribution of actions, i.e., refined operation strategies for one or more future scheduling cycles, including: Unit output vector: The optimal power generation allocated to each operating unit at each decision time step (e.g., 15 minutes or 1 hour), in the form of [ W 1, W 2,..., W n ] .
[0046] Total discharge flow: includes the flow rate through the generator set and the flow rate of water discharged without passing through the generator set.
[0047] The policy function of a reinforcement learning algorithm defines how an agent chooses an action in a given state, and is expressed as: in, Indicates the state Select action strategy, This means that in a given state Under the conditions, take action The probability; that is, using a random strategy, the agent selects actions according to a certain probability distribution in each state.
[0048] For a discrete action space, the policy function returns the probability distribution for each action.
[0049] For a continuous action space, the policy function returns an action value or a probability distribution parameter of the action (such as mean and variance).
[0050] The value function evaluates the merits of a state or state-action pair, including: State value function: measures the state value at a given time. S t The expected cumulative future rewards that can be obtained by following strategy π are as follows: in, Indicates the state Next, follow the strategy Expected future cumulative rewards Indicates policy-based Expectation operation, It is a discount factor used to discount future rewards, reflecting the agent's emphasis on future rewards. Indicates at time The rewards received.
[0051] Action value function: measures the value of an action in a given state. S t Select action A t Then, follow the expected future cumulative rewards that can be obtained by strategy π: in, Indicates the state Select action Then, follow the strategy. The expected value of future cumulative rewards; coefficient c k The closer a value is to 1, the more emphasis is placed on long-term returns, determining the agent's level of focus on future rewards. The criteria for this determination are as follows: ① Hydropower scheduling is a typical long-term planning problem; therefore, γ must be set to a relatively high value, usually selected between [0.97, 0.999]. ② Similar to the weighting coefficients, the optimal value of γ is also determined through hyperparameter optimization in the offline phase. Different values of γ, such as 0.99, 0.98, and 0.97, are tested in the validation environment to select the value that best performs the model in terms of long-term cumulative rewards (such as the total revenue over a scheduling year).
[0052] In another possible implementation, the refined operation strategy can be output using a multi-objective optimization method, comprehensively considering multiple objectives such as the power generation benefits of the hydropower station, the ecological and environmental impact, and the grid stability. In the reinforcement learning algorithm, multiple reward functions are set, each corresponding to a different objective. These reward functions are combined into a comprehensive reward function through weighted summation. Then, during training, the agent makes decisions based on the comprehensive reward function, outputting a refined operation strategy that simultaneously satisfies multiple objectives. For example, in the reward function, a certain weight is assigned to power generation revenue, and another weight is assigned to the deviation of ecological flow; by adjusting the weights, the relationship between different objectives is balanced.
[0053] In another possible implementation, the refined operation strategy can be combined with real-time monitoring and forecasting technologies to dynamically adjust the strategy based on real-time operating data of the hydropower station and future forecast information. In a digital twin environment, data such as water level, flow rate, and unit status of the hydropower station are acquired in real time. Simultaneously, information such as weather forecasts and grid load predictions is used to predict the hydropower station's operation over a future period. Then, based on real-time data and forecast information, the refined operation strategy output by the reinforcement learning algorithm is adjusted in real time to ensure the strategy's effectiveness and adaptability. For example, when heavy rain is predicted in the next few days, the unit output and flood discharge flow can be adjusted in advance to cope with potential flood disasters.
[0054] Considering the continuous action space and high stability requirements of hydropower station scheduling, this embodiment selects the following advanced deep reinforcement learning algorithm: DDPG (Deep Deterministic Policy Gradient): Applicable to continuous action space problems, it directly outputs a definite optimal action.
[0055] PPO (Proximal Policy Optimization): Improves the stability of the training process by limiting the magnitude of policy updates. It is a widely used and robust algorithm.
[0056] SAC (Soft Actor-Critic): By introducing a maximum entropy objective, it encourages agents to explore more thoroughly, and is particularly suitable for handling environments with high randomness and uncertainty.
[0057] In another possible implementation, when designing a reinforcement learning algorithm suitable for peak-shaving operation of a hydropower station, the advantages of genetic algorithms and reinforcement learning algorithms can be combined to design a hybrid algorithm. First, the initial parameters of the reinforcement learning algorithm are optimized using a genetic algorithm to find a better set of initial parameters. Then, the reinforcement learning algorithm is trained and learned in a digital twin environment, continuously adjusting the parameters to improve the algorithm's performance. For example, in a genetic algorithm, parameters such as the learning rate and discount factor of the reinforcement learning algorithm are used as genes on a chromosome, and the optimal parameter combination is found through operations such as selection, crossover, and mutation.
[0058] In another possible implementation, the idea of transfer learning can be introduced when designing reinforcement learning algorithms suitable for peak-shaving operation of hydropower stations. This involves using peak-shaving operation data from other similar hydropower stations and pre-trained reinforcement learning models to initialize and pre-train the current hydropower station's reinforcement learning algorithm. Then, fine-tuning can be performed in the digital twin environment of the current hydropower station, enabling the algorithm to adapt to new environments and tasks more quickly. For example, a reinforcement learning model from a geographically close and similarly sized hydropower station can be used as the base model, and some of its parameters can be transferred to the current hydropower station's algorithm, followed by training and optimization in the current environment.
[0059] In this embodiment, step S3 above, which initializes the agent in the digital twin environment, trains the agent using a reinforcement learning algorithm in the constructed digital twin environment, and optimizes it based on the training results, includes: Using historical operational data (hydrology, load, electricity price, etc.) and a pre-built environmental simulator, the agent is thoroughly trained offline. Through millions of simulations, the agent iteratively updates its neural network parameters until the policy converges.
[0060] The trained agent is fine-tuned online using the latest real-time data to enable it to adapt to slow changes in the environment.
[0061] The model's performance was evaluated using a set of test datasets that were not used in the training, to test its generalization ability, robustness, and applicability in various scenarios such as different hydrological conditions, power grid demands, and market environments.
[0062] In this embodiment, the real-time acquisition of hydropower station operation data, grid load data, and electricity price data in step S4, and the input of the trained and optimized agent to generate the optimal scheduling strategy for the current moment, includes: Real-time hydrological data, grid load, electricity price, etc. are input into the reinforcement learning model to generate the current optimal scheduling strategy.
[0063] The agent updates the scheduling strategy periodically using a rolling window to ensure real-time performance and dynamic adaptability.
[0064] The actual operating results (such as the water level after execution and the actual power generation) are compared with the predictions of the agent. The information from the comparison is used as new data for the continuous optimization and adjustment of the agent, forming a complete closed-loop adaptive system.
[0065] Example 3: The above is an illustrative scheme of a reinforcement learning-based hydropower station peak-shaving operation strategy optimization method according to this embodiment. It should be noted that the technical solution of a reinforcement learning-based hydropower station peak-shaving operation strategy optimization system belongs to the same concept as the above-described reinforcement learning-based hydropower station peak-shaving operation strategy optimization method. Details not described in detail in the reinforcement learning-based hydropower station peak-shaving operation strategy optimization system in this embodiment can be found in the description of the reinforcement learning-based hydropower station peak-shaving operation strategy optimization method described above.
[0066] This embodiment also provides a reinforcement learning-based hydropower station peak-shaving operation strategy optimization system, including: The digital twin environment construction module is used to collect data related to hydropower station operation, power grid and market data, and combine them with operational constraints to construct a digital twin environment. The reinforcement learning algorithm design module is used to design reinforcement learning algorithms suitable for peak-shaving operation of hydropower stations based on the constructed digital twin environment. The agent training and optimization module is used to initialize the agent in the digital twin environment, use reinforcement learning algorithms to train the agent in the constructed digital twin environment, and optimize it based on the training results. The real-time scheduling strategy generation module is used to acquire hydropower station operation data, power grid load data, and electricity price data in real time, and input them into the trained and optimized agent to generate the optimal scheduling strategy for the current moment.
[0067] This embodiment also provides an electronic device applicable to a reinforcement learning-based method for optimizing peak-shaving operation strategies in hydropower stations, including: The system includes a memory and a processor. The memory stores computer-executable instructions, and the processor executes these instructions to implement a reinforcement learning-based optimization method for peak-shaving operation of hydropower stations, as proposed in the above embodiments.
[0068] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements a method for optimizing the peak-shaving operation strategy of a hydropower station based on reinforcement learning as proposed in the above embodiment.
[0069] The storage medium proposed in this embodiment belongs to the same inventive concept as the reinforcement learning-based hydropower station peak-shaving operation strategy optimization method proposed in the above embodiment. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.
[0070] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for optimizing peak-shaving operation strategy of hydropower stations based on reinforcement learning, characterized in that, include: Collect data related to hydropower station operation, power grid and market data, and combine them with operational constraints to build a digital twin environment; Based on the constructed digital twin environment, a reinforcement learning algorithm suitable for peak-shaving operation of hydropower stations is designed. Initialize the agent in the digital twin environment, train the agent in the constructed digital twin environment using reinforcement learning algorithms, and optimize it based on the training results; Real-time data acquisition of hydropower station operation, power grid load, and electricity price data is used to input the trained and optimized agent to generate the optimal scheduling strategy for the current moment.
2. The method for optimizing peak-shaving operation strategy of hydropower stations based on reinforcement learning as described in claim 1, characterized in that, The collection of hydropower station operation-related data, power grid and market-related data, and the construction of a digital twin environment in conjunction with operational constraints include: Based on collected data related to hydropower station operation, power grid and market data, and combined with operational constraints, a physical model of the hydropower station is established. The physical model of the hydropower station reflects the water balance of the reservoir, the output characteristics of the generator units, the flood discharge capacity, and simulates the dynamic changes of the external environment.
3. The method for optimizing peak-shaving operation strategy of hydropower stations based on reinforcement learning as described in claim 2, characterized in that, The collection of hydropower station operation-related data, power grid and market-related data, and the construction of a digital twin environment in conjunction with operational constraints also include: Define a state space that encompasses all the information needed to describe key environmental information, including the operating status of the hydropower station and external environmental information. Define an action space, which is a decision vector that the agent can execute. For the peak shaving problem of hydropower stations, the actions are continuous. Design a reward function to guide the agent's learning, balance short-term economic benefits with long-term operational goals, provide positive incentives for power generation revenue, and punish water abandonment, deviation from ecological flow, and violation of operational constraints.
4. The method for optimizing peak-shaving operation strategy of hydropower stations based on reinforcement learning as described in claim 3, characterized in that, Based on the constructed digital twin environment, the reinforcement learning algorithm designed for the peak-shaving operation of the hydropower station includes: Based on the constructed digital twin environment, a reinforcement learning algorithm suitable for peak-shaving operation of hydropower stations is designed. Taking the current state as input, the algorithm outputs an action or the probability distribution of the action, which is a refined operation strategy for one or more future scheduling cycles, including the unit output vector and the total discharge flow.
5. The method for optimizing peak-shaving operation strategy of hydropower stations based on reinforcement learning as described in claim 4, characterized in that, The reinforcement learning algorithm designed for peak-shaving operation of hydropower stations based on the constructed digital twin environment also includes: Reinforcement learning algorithms employ stochastic policies, where the agent selects actions according to a certain probability distribution in each state. For discrete and continuous action spaces, the policy function returns the probability distribution of actions or the action value and the probability distribution parameters of the actions, respectively.
6. The method for optimizing peak-shaving operation strategy of hydropower stations based on reinforcement learning as described in claim 5, characterized in that, The reinforcement learning algorithm designed for peak-shaving operation of hydropower stations based on the constructed digital twin environment also includes: The quality of a state or state-action pair is evaluated using value functions, including a state value function that measures the expected future cumulative reward of following a policy in a given state, and an action value function that measures the expected future cumulative reward of following a policy after choosing an action in a given state.
7. The method for optimizing peak-shaving operation strategy of hydropower stations based on reinforcement learning as described in claim 6, characterized in that, The process of initializing the agent in a digital twin environment, training the agent using a reinforcement learning algorithm within the constructed digital twin environment, and optimizing it based on the training results includes: Using historical operational data and a pre-built environment simulator, the agent is trained offline. During the training process, the agent continuously iterates and updates the neural network parameters through simulated scheduling operations until the policy reaches a convergent state. The latest real-time data is then used to fine-tune the agent that has completed offline training online.
8. A hydropower station peak-shaving operation strategy optimization system based on reinforcement learning, using the method described in any one of claims 1 to 7, characterized in that, include: The digital twin environment construction module is used to collect data related to hydropower station operation, power grid and market data, and combine them with operational constraints to construct a digital twin environment. The reinforcement learning algorithm design module is used to design reinforcement learning algorithms suitable for peak-shaving operation of hydropower stations based on the constructed digital twin environment. The agent training and optimization module is used to initialize the agent in the digital twin environment, use reinforcement learning algorithms to train the agent in the constructed digital twin environment, and optimize it based on the training results. The real-time scheduling strategy generation module is used to acquire hydropower station operation data, power grid load data, and electricity price data in real time, and input them into the trained and optimized agent to generate the optimal scheduling strategy for the current moment.
9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.