A water, wind and light scheduling method considering constraint correction and average reward reinforcement learning

Abstract

This invention belongs to the field of energy dispatching and intelligent control technology, and discloses a water-wind-solar complementary operation method considering constraint correction and average reward reinforcement learning. The method includes: S1 constructing a mathematical model for water-wind-solar complementary operation and applying constraint processing; S2 constructing a water-wind-solar complementary operation environment suitable for reinforcement learning; S3 constructing a reinforcement learning SAC agent suitable for water-wind-solar complementary operation; S4 training the parameters of the SAC agent using a large amount of historical data to learn the optimal strategy that maximizes environmental rewards; S5 performing actual optimization scheduling of water-wind-solar complementary operation, deploying the trained strategy network to the actual scheduling system, implementing optimized scheduling, and generating a complete scheduling scheme. This invention constructs a reinforcement learning environment suitable for water-wind-solar complementary operation, enabling the reinforcement learning agent to directly enforce hard constraints within the environment, avoiding the instability of traditional punitive soft constraints, and significantly improving the feasibility and stability of the scheduling strategy.

Description

Technical Field

[0001] This invention belongs to the technical field of energy dispatch and intelligent control, and more specifically, relates to a water, wind and solar dispatch method that considers constraint correction and average reward reinforcement learning. Background Technology

[0002] With the large-scale integration of renewable energy, the complementary operation of hydropower, wind power, and photovoltaic power generation has become an important way to improve the absorption capacity of renewable energy. Hydropower has the characteristics of strong regulation flexibility and rapid start-up, which can be used to smooth the fluctuations in wind power and photovoltaic output; wind power and photovoltaic power can provide additional power support during the dry season or under low water level conditions. However, because the multi-energy joint operation process is affected by a variety of factors such as hydrological inflow, wind and solar resource conditions, and power plant operation constraints, its optimization scheduling problem has typical characteristics of multiple constraints, high dimension, and strong coupling. Traditional solution methods are difficult to quickly obtain high-quality scheduling solutions under complex operating conditions.

[0003] Existing optimal scheduling methods for hydropower-wind-solar hybrid systems mainly include mathematical programming and evolutionary algorithms. First, mathematical programming methods, including dynamic programming, linear programming, and nonlinear programming, typically require simplified modeling of hydrological processes and power station characteristics. When solving continuous action spaces, high-dimensional state spaces, and strongly nonlinear hydraulic characteristics, they are prone to problems such as computational dimensionality explosion or convergence difficulties, making it difficult to simultaneously achieve refined constraint expression and efficient solution. Second, evolutionary algorithms, as typical intelligent optimization methods, usually require a large number of iterations to converge. Their solution efficiency is greatly affected by parameter settings and the initial population, making it difficult to quickly generate high-quality scheduling schemes in large-scale cascade systems.

[0004] In summary, existing mathematical programming and evolutionary algorithms for multi-energy joint scheduling problems suffer from high model complexity, insufficient solution efficiency, and limited adaptability to high-dimensional states and strong uncertainties. Furthermore, traditional methods often represent constraints using penalty mechanisms, lacking the ability to directly model hard constraints, which may lead to unstable constraint satisfaction. Therefore, there is an urgent need for a water, wind, and solar scheduling method that considers constraint correction and average reward reinforcement learning, capable of simultaneously handling the characteristics of multiple energy sources, reservoir dynamics, and complex operational constraints, and rapidly obtaining high-quality scheduling schemes in high-dimensional state spaces to achieve multi-energy joint operation. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a water-wind-solar scheduling method that considers constraint correction and average reward reinforcement learning. Its purpose is to construct a unified scheduling environment for water-wind-solar complementary operation, establish a reinforcement learning solution framework that can truly reflect the scheduling physical process and the recursive relationship of rewards, and realize fast and efficient optimized scheduling of water-wind-solar complementary systems.

[0006] To achieve the above objectives, according to one aspect of the present invention, a water, wind, and solar scheduling method considering constraint correction and average reward reinforcement learning is provided, comprising the following steps:

[0007] S1 constructs a mathematical model for the operation of water-wind-solar hybrid system and performs constraint processing;

[0008] S2 constructs a water-wind-solar hybrid operating environment suitable for reinforcement learning. The mathematical model obtained in step S1 and the water level corridor mapping results are encapsulated into a reinforcement learning environment. This environment consists of a state space, an action space, a state transition function, and a reward function. The environment is in a state... Next action The next state will be obtained. With instant rewards ;

[0009] S3 constructs a reinforcement learning SAC agent suitable for hydro-wind-solar hybrid operation; on the environment constructed in step S2, a SAC-based agent is constructed; the SAC agent consists of a policy network (Actor), two independent value networks (Critic), and two corresponding target value networks (Target Critic); the SAC agent learns the optimal scheduling strategy through interaction with the environment.

[0010] S4 trains the parameters of the SAC agent constructed in step S3 using a large amount of historical data, and learns the optimal strategy that maximizes environmental rewards.

[0011] After the S5 hydro-wind-solar hybrid actual optimization scheduling is completed in step S4, the trained strategy network is deployed to the actual scheduling system and optimized scheduling is implemented to generate a complete scheduling scheme.

[0012] Preferably, S1 specifically includes the following steps:

[0013] S1.1 Construct a mathematical model for the integrated operation of hydropower, wind power, and solar power. With the objective of maximizing the total power generation of the combined operation of hydropower, wind power, and solar power, the mathematical model is established as follows:

[0014]

[0015] In the formula: This represents the total power generation of the complementary system. , and Let represent the power output of the wind power station, photovoltaic power station, and hydropower station respectively during time period t. T This represents the total number of time periods during the scheduling period. It is the length of the scheduling period, in hours;

[0016] System constraints include reservoir characteristic constraints, water balance constraints, water level constraints, flow constraints, power output constraints, and channel capacity constraints;

[0017] S1.2 Utilizes the concept of water level corridors to uniformly map water level, flow rate, and output constraints to the feasible water level range for a given time period, thereby achieving direct modeling of hard constraints;

[0018] Based on the operational constraints of the hydropower station, the maximum allowable water level for time period t is calculated in reverse. With minimum allowable water level And form a reverse water level range The maximum allowable water level for time period t is calculated by recursively estimating the water level for time period t in the forward direction based on the water level for time period t-1. With minimum allowable water level and constitute a positive water level range. .

[0019] Preferably, S2 specifically includes the following steps:

[0020] The state space of the S2.1 hydro-wind-solar hybrid operating environment includes time period, temperature, radiation intensity, wind speed, water level, upper and lower limits of water level for the next time period, and inflow.

[0021] S2.2 defines the action space of the hydro-wind-solar hybrid operating environment as the reservoir outflow, i.e., the action in time period t. Outbound flow rate during time period t ;

[0022] S2.3 Based on the state space defined in step S2.1 and the action space defined in step S2.2, given the current state and the action to be executed, the environment completes the state transition according to the reservoir water balance relationship and system operation constraints. Specifically, the state transition function of the hydro-wind-solar hybrid operating environment includes two parts: state transition and action correction. Given the action... The water level at time t+1 is obtained from the state transition equation. If the water level exceeds the feasible range, hard constraints are applied based on the corridor to obtain the corrected state and action.

[0023] S2.4 After the state transition is completed in step S2.3, the environment calculates the immediate reward based on the current state and the executed action; the reward function of the hydro-wind-solar hybrid operating environment is the state. Next action Rewards This includes the total amount of hydropower, wind power, and solar power generation, as well as penalties for violating water level constraints.

[0024] Preferably, in step S2.1, the state during time period t is: :

[0025]

[0026] In the formula, and Let t be the ambient temperature and solar radiation intensity of the photovoltaic power station during time period t. For the wind speed of the power plant, These are the upper and lower limits of the water level for the time period t+1. The reservoir water level during time period t. This refers to the inbound flow during the scheduling period;

[0027] In step S2.3, given the action The water level at time t+1 is obtained from the state transition equation. :

[0028]

[0029] In the formula, , These represent the reservoir water level and inflow rate at time t. This represents the water level-capacity curve of a reservoir.

[0030] If the water level exceeds the feasible range, then a hard constraint correction will be made based on the corridor:

[0031]

[0032] In the formula, This refers to the corrected outflow and reservoir water level.

[0033] Preferably, the state in step S2.4 Next Action Rewards :

[0034]

[0035] In the formula, Let t be the total power generation during period t. Penalties for violating water level constraints, This is the penalty coefficient;

[0036] in, and The calculation formula is as follows:

[0037]

[0038] In the formula, This indicates the maximum capacity of the outgoing channel.

[0039] Preferably, S3 specifically includes the following steps:

[0040] S3.1 Constructs a policy network, which consists of a multilayer perceptron with hidden nodes of size `hidden_size` and hidden layers of `n`, and stores the current state. The mapping is to continuous action distribution parameters, and the policy network input is the current state. The output action distribution has a mean and variance. By sampling this distribution and mapping it to the decision action range, the actual actions to be executed can be obtained. ;

[0041] S3.2 Based on the policy network generating scheduling actions in step S3.1, a value network is further constructed to estimate the value function corresponding to the policy network state-action. To reduce overestimation bias, two independent Critic networks are used. Each value network is a multilayer perceptron structure with the same number of hidden layers and nodes as the policy network, and the parameters are independent of each other, thereby achieving robust evaluation of the value of scheduling actions.

[0042] S3.3 Based on the two sets of value networks in step S3.2, construct target value networks with the same structure to estimate the value function of the state-action in the next time period. The target value network has the same structure as the value network. During training, the value network parameters are synchronized in a soft update manner, thereby providing a relatively smooth and stable learning target when updating the value function and improving the stability of the SAC training process.

[0043] Preferably, S4 specifically includes the following steps:

[0044] S4.1 Initialize Policy Network Parameters Value network parameters Set target value network parameters Construct an experience replay buffer with a capacity of buffer_size; set the learning rate. Soft update parameters Sampling batch size and total number of training steps;

[0045] The S4.2 agent interacts with the environment according to the current policy and continuously collects data. The data is then stored in the playback pool;

[0046] S4.3 Randomly sample small batches of data from the replay pool to update the policy and value network parameters; to adapt to the operation of hydro-wind-solar hybrid systems with fixed scheduling cycles, the average value of all future rewards is used. As the target value of the value network;

[0047] The average of all future rewards The expression is as follows:

[0048]

[0049] In the formula, and Let N represent the rewards for time period t and time period t+k, respectively, and let N be the number of remaining scheduling steps from the current time period t. This represents the average of all rewards during period t and all future rewards. The average of the t+1 period and all future rewards estimated by the target value network.

[0050] Preferably, the loss function of the value network in step S4.3 The value network parameters are updated by minimizing the loss function using mean squared error:

[0051]

[0052] In the formula, The estimated results are for the value network i-to-t time period and the average of all future rewards;

[0053] The policy network is updated via policy gradients, with the objective of maximizing the value network estimate. :

[0054]

[0055] In the formula, Let the loss function be the policy network. For the policy network in state The estimated value of the lower value network;

[0056] Based on soft update parameters The parameters of the target value network are updated using a soft update method to maintain learning stability.

[0057]

[0058] In the formula, , These are the value network parameters and the target value network parameters, respectively.

[0059] Preferably, the optimized scheduling implemented in S5 specifically includes the following steps:

[0060] S5.1 Receive real-time or short-term forecast data, including but not limited to temperature, radiation intensity, wind speed, water level, and inflow during the scheduling period; input the above data into the hydro-wind-solar hybrid operation environment constructed in step S2, and construct the state vector for the current time period based on the environmental state definition. ;

[0061] S5.2 policy network outputs scheduling actions based on the current environment state. The trained policy network is invoked to generate outbound traffic decisions for the current time period. The actions are mapped through the internal environment and transformed into actual hydropower station discharge flow control commands.

[0062] S5.3 will take action Input the operating environment, and obtain the environment state for the next time period according to the environment state transition mechanism in step S2. and the actual power generation during that period The environment automatically corrects actions that violate the constraints based on the hard constraints of the water level corridor.

[0063] S5.4 Repeat steps S5.2 and S5.3, iterating through each time period until the end of the scheduling period. After the scheduling is completed, a complete scheduling scheme is generated, which includes the reservoir water level process for each time period; the discharge flow process; the power output process of hydropower, wind power, and photovoltaic power for each time period; and the statistical results of the combined hydropower, wind power, and photovoltaic power generation. This scheme serves as the basis for actual operation control or scheduling decisions.

[0064] According to another aspect of the present invention, a computer device is provided, including a processor and a memory, wherein the memory stores a computer program that, when loaded and executed by the processor, implements the water, wind and solar power joint scheduling method as described in any of the above technical solutions.

[0065] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores a program that, when loaded by a processor, implements the water, wind and solar joint scheduling method as described in any of the above technical solutions.

[0066] In summary, compared with the prior art, the water, wind, and solar scheduling method considering constraint correction and average reward reinforcement learning provided by this invention has the following beneficial effects:

[0067] 1. This invention addresses the characteristic of joint scheduling of water, wind, and solar power with a clearly defined scheduling cycle termination by proposing a value estimation method based on the average reward of future time periods. This method replaces the traditional discounted cumulative return form, enabling the value network to more accurately reflect the comprehensive benefits of scheduling decisions throughout the remaining scheduling cycle. This improves the stability and accuracy of value estimation and enhances the efficiency of strategy learning.

[0068] 2. This invention constructs a reinforcement learning environment suitable for hydro-wind-solar hybrid operation by introducing a water level corridor constraint mechanism. It unifies the multi-dimensional coupled constraints such as flow constraints and output constraints in reservoir operation into feasible water level intervals for different time periods, and directly executes them as hard constraints during environmental state transitions, thereby improving the feasibility and stability of the scheduling process.

[0069] 3. This invention deeply integrates the constraint correction mechanism with the reinforcement learning environment, performs real-time correction of actions that do not meet the constraints during the state transition stage, and introduces a constraint violation penalty term into the reward function, effectively guiding the agent to form a scheduling strategy that satisfies the operating constraints and reducing the probability of the scheduling strategy violating the constraints.

[0070] 4. This invention achieves optimized scheduling and operation of a hydro-wind-solar hybrid system based on a reinforcement learning framework. It can fully utilize hydropower, wind power, and solar energy resources under complex conditions, and achieve an overall improvement in power generation efficiency under multi-energy complementary operation. It has strong engineering applicability and promotion value. Attached Figure Description

[0071] Figure 1 This is an overall flowchart of a water, wind, and solar scheduling method that considers constraint correction and average reward reinforcement learning according to the present invention.

[0072] Figure 2 This is a schematic diagram of the interaction structure between the water-wind-solar hybrid operating environment and the SAC intelligent agent of the present invention.

[0073] Figure 3 This is a flowchart of the SAC agent training process of the present invention.

[0074] Figure 4 This is a schematic diagram of the change in the value function during the training process of the agent in this embodiment of the invention.

[0075] Figure 5 This is a schematic diagram of the total reward change during the training process of the agent in an embodiment of the present invention.

[0076] Figure 6 This is a daily total power generation curve of the optimized scheduling results in an embodiment of the present invention.

[0077] Figure 7 This is a water level and flow rate process diagram of a typical daily optimized scheduling result in an embodiment of the present invention.

[0078] Figure 8 This is a diagram showing the output process of a typical day's optimized scheduling result in an embodiment of the present invention. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0080] This invention provides a water, wind, and solar scheduling method that considers constraint correction and average reward reinforcement learning. Please refer to [link to relevant documentation]. Figure 1 It includes the following steps:

[0081] S1 constructs a mathematical model for the operation of a water-wind-solar hybrid system and applies constraints:

[0082] S1.1 Construct a mathematical model for the integrated operation of hydropower, wind power, and solar power. With the objective of maximizing the total power generation of the combined operation of hydropower, wind power, and solar power, the following optimization model is established:

[0083] (1)

[0084] In the formula: This represents the total power generation of the complementary system. , and Let represent the power output of the wind power station, photovoltaic power station, and hydropower station respectively during time period t. T This represents the total number of time periods during the scheduling period. It is the length of the scheduling period, in hours;

[0085] System constraints include reservoir characteristic constraints, water balance constraints, water level constraints, flow constraints, power output constraints, and channel capacity constraints;

[0086] S1.2 utilizes the concept of water level corridors to map water level, flow rate, and output constraints to a feasible water level interval for a given time period, enabling direct modeling of hard constraints:

[0087] (1) Reverse calculation of water level intervals

[0088] Based on the operational constraints of the hydropower station, the maximum allowable water level for time period t is calculated in reverse. With minimum allowable water level And form a reverse water level range Taking the reverse minimum water level constraint as an example, the calculation method is as follows:

[0089] (2)

[0090] (3)

[0091] (4)

[0092] In the formula, These represent the initial minimum operable water level at time t, obtained by inversely calculating the minimum flow constraint, minimum output constraint, and water level constraint of the power station during time period t, respectively. This represents a function for calculating water level at the end of a known time period. Inbound flow and time of day In this case, the initial water level of the time period is determined by the "electricity-based water level determination" method. ; This represents the water level-capacity curve of a reservoir. , and These represent the lower limit of the reservoir's discharge flow, the lower limit of its hydropower output, and the lower limit of its water level during time period t, respectively. This indicates the allowable range of water level changes within the reservoir's operational time intervals. These represent the final water levels during the scheduling period; This represents the lowest operational water level at the beginning of time period t, obtained through reverse recursion.

[0093] (2) Forward recursion of water level intervals

[0094] Similarly, the maximum allowable water level for time period t is derived by recursively calculating the water level for time period t in the forward direction based on the water level for time period t-1. With minimum allowable water level and constitute a positive water level range. Taking the positive minimum water level constraint as an example, the calculation method is as follows:

[0095] (5)

[0096] (6)

[0097] In the formula, These represent the lowest operable water level at the end of time period t, obtained by forward calculation of the power station's maximum flow constraint, maximum output constraint, and water level constraint during time period t. , , ,and These represent the upper limit of the discharge flow, the upper limit of hydropower output, the inflow flow, and the water level during the t-1 time period, respectively.

[0098] S2 constructs a water-wind-solar hybrid operating environment suitable for reinforcement learning. The mathematical model obtained in step S1 and the water level corridor mapping results are encapsulated into a reinforcement learning environment. This environment consists of a state space, an action space, a state transition function, and a reward function. The environment is in a state... Next action The next state will be obtained. With instant rewards ;

[0099] S2 specifically includes the following steps:

[0100] The state space of the S2.1 hydro-wind-solar hybrid operating environment includes time period, temperature, radiation intensity, wind speed, water level, upper and lower limits of water level for the next time period, and inflow. The state at time period t is... :

[0101] (7)

[0102] In the formula, and Let t be the ambient temperature and solar radiation intensity of the photovoltaic power station during time period t. For the wind speed of the power plant, These are the upper and lower limits of the water level for the time period t+1. The reservoir water level during time period t. This refers to the inbound flow during the scheduling period;

[0103] S2.2 defines the action space of the hydro-wind-solar hybrid operating environment as the reservoir outflow, i.e., the action in time period t. Outbound flow rate during time period t ;

[0104] S2.3 Based on the state space defined in step S2.1 and the action space defined in step S2.2, given the current state and the action to be executed, the environment completes the state transition according to the reservoir water balance relationship and system operation constraints. Specifically, the state transition function of the hydro-wind-solar hybrid operating environment includes two parts: state transition and action correction. Given the action... The water level at time t+1 is obtained from the state transition equation. :

[0105] (8)

[0106] In the formula, and Let t represent the reservoir water level and inflow rate during time period t. This represents the water level-capacity curve of a reservoir.

[0107] If the water level exceeds the feasible range, then a hard constraint correction will be made based on the corridor:

[0108] (9)

[0109] In the formula, This refers to the corrected outflow and reservoir water level.

[0110] S2.4 After the state transition is completed in step S2.3, the environment calculates the immediate reward based on the current state and the executed action; the reward function of the hydro-wind-solar hybrid operating environment is the state. Next action Rewards :

[0111] (10)

[0112] In the formula, Let t be the total power generation during period t. Penalties for violating water level constraints, This is the penalty coefficient;

[0113] in, and The calculation formula is as follows:

[0114] (11)

[0115] In the formula, This indicates the maximum capacity of the outgoing channel.

[0116] S3 constructs a reinforcement learning SAC agent suitable for hydro-wind-solar hybrid operation; on the environment constructed in step S2, constructs an SAC-based agent; please refer to... Figure 2 The SAC agent consists of a policy network (Actor), two independent value networks (Critic), and two corresponding target value networks (Target Critic). The agent learns the optimal scheduling strategy through interaction with the environment;

[0117] S3 specifically includes the following steps:

[0118] S3.1 Constructs a policy network, which consists of a multilayer perceptron with hidden nodes of size `hidden_size` and hidden layers of `n`, and stores the current state. The mapping is to continuous action distribution parameters, and the policy network input is the current state. The output action distribution has a mean and variance. By sampling this distribution and mapping it to the decision action range, the actual actions to be executed can be obtained. ;

[0119] S3.2 Based on the policy network generating scheduling actions in step S3.1, a value network is further constructed to estimate the value function corresponding to the policy network state-action. To reduce overestimation bias, two independent Critic networks are used. Each value network is a multilayer perceptron structure with the same number of hidden layers and nodes as the policy network, and the parameters are independent of each other, thereby achieving robust evaluation of the value of scheduling actions.

[0120] S3.3 Based on the two sets of value networks in step S3.2, construct target value networks with the same structure to estimate the value function of the state-action in the next time period. The target value network has the same structure as the value network. During training, the value network parameters are synchronized in a soft update manner, thereby providing a relatively smooth and stable learning target when updating the value function and improving the stability of the SAC training process.

[0121] S4 trains the parameters of the SAC agent using a large amount of historical data, learning the optimal policy that maximizes environmental rewards. (See also...) Figure 3 S4 specifically includes the following steps:

[0122] S4.1 Initialize Policy Network Parameters Value network parameters Set target value network parameters Construct an experience replay buffer with a capacity of buffer_size; set the learning rate. Soft update parameters The sampling batch size (batch_size) and the total number of training steps (total_timesteps);

[0123] The S4.2 agent interacts with the environment according to the current policy and continuously collects data. The data is then stored in the playback pool;

[0124] S4.3 Randomly sample small batches of data from the replay pool to update the policy and value network parameters; to adapt to the operation of hydro-wind-solar hybrid systems with fixed scheduling cycles, the average value of all future rewards is used. As the target value of the value network, the average future reward is defined as:

[0125] (12)

[0126] In the formula, and Let N represent the rewards for time period t and time period t+k, respectively, and let N be the number of remaining scheduling steps from the current time period t. This represents the average of all rewards during period t and all future rewards. The average of the t+1 period estimated by the target value network and all future rewards;

[0127] Loss function of value network The value network parameters are updated by minimizing the loss function using mean squared error:

[0128] (13)

[0129] The policy network is updated via policy gradients, with the objective of maximizing the value network estimate. :

[0130] (14)

[0131] Based on soft update parameters The parameters of the target value network are updated using a soft update method to maintain learning stability.

[0132] (15)

[0133] In the formula, , These are the value network parameters and the target value network parameters, respectively.

[0134] In the actual optimized scheduling of S5 hydro-wind-solar hybrid systems, after training in step S4, the trained policy network is deployed to the actual scheduling system, and optimized scheduling is implemented according to the following steps to generate a complete scheduling scheme:

[0135] S5.1 Receive real-time or short-term forecast data, including but not limited to temperature, radiation intensity, wind speed, water level, and inflow during the scheduling period; input the above data into the hydro-wind-solar hybrid operation environment constructed in step S2, and construct the state vector for the current time period based on the environmental state definition. ;

[0136] S5.2 policy network outputs scheduling actions; based on the current environment state. The trained policy network is invoked to generate outbound traffic decisions for the current time period. The actions are mapped through the internal environment and transformed into actual hydropower station discharge flow control commands.

[0137] S5.3 will take action Input the operating environment, and obtain the environment state for the next time period according to the environment state transition mechanism in step S2. and the actual power generation during that period The environment automatically corrects actions that violate the constraints based on the hard constraints of the water level corridor.

[0138] S5.4 Repeat steps S5.2 and S5.3, iterating through each time period until the end of the scheduling period. After the scheduling is completed, a complete scheduling scheme is generated, which includes the reservoir water level process for each time period; the discharge flow process; the power output process of hydropower, wind power, and photovoltaic power for each time period; and the statistical results of the combined hydropower, wind power, and photovoltaic power generation. This scheme can be used as the basis for actual operation control or scheduling decisions.

[0139] The following are specific examples:

[0140] This embodiment uses a hydropower station and its connected wind and solar power stations as an example to form a hydro-wind-solar hybrid operating system. The installed capacities of the hydropower station, wind power station, and solar power station are 16 million kW, 1.41 million kW, and 3.08 million kW, respectively. The number of hidden nodes in the intelligent agent network is set to hidden_size to 256, the number of hidden layers n is 4, and the learning rate is... 0.0003, soft update parameter The parameters are: 0.0005, experience replay buffer size (buffer_size) of 100,000, batch size (batch_size) of 1024, and total training steps (total_timesteps) of 200,000. The agent is trained based on two years of historical data to obtain a scheduling strategy suitable for hydro-wind-solar hybrid operation. To verify the technical effectiveness of this invention (HWS_SAC), conventional reinforcement learning models (SAC and TD3) are used as comparison schemes.

[0141] Table 1 Daily scheduling results for different models

[0142]

[0143] Please refer to the process of change of the two value functions during training. Figure 4 As can be seen, after approximately 50,000 training steps, both value estimates gradually stabilize, indicating that the corresponding value networks have essentially converged. However, the value network trained using the improved time-averaged reward method exhibits significantly lower variance and smaller fluctuations throughout the training process, reflecting a more stable training process. In contrast, the value estimate based on the cumulative reward method shows a significant increase again after approximately 165,000 training steps, indicating some instability in the later training stages. For a detailed explanation of the changes in total reward during training, please refer to [link to relevant documentation]. Figure 5 For ease of display and viewing, a moving average of the total reward is plotted as follows: Figure 5 As shown in b in the figure, HWS_SAC and SAC have similar training effects. TD3's moving average is better than HWS_SAC, but TD3 training is unstable, frequently violating constraints and resulting in negative rewards. The optimization effect of the trained agent was verified using one year of real-world data; the results can be found in [link to relevant documentation]. Figure 6 See Table 1. The results show that HWS_SAC achieves the best optimization effect, maintaining the lowest curtailment and fewest constraint violations while achieving the highest total power generation. HWS_SAC achieved a total power generation of 163.77 GWh, which is 1.26% and 4.25% higher than SAC and TD3, respectively; at the same time, its curtailment rate decreased by 63.8% and 89.9%, respectively. The results demonstrate that the proposed method can significantly improve energy utilization efficiency without sacrificing operational feasibility, exhibiting good robustness and adaptability.

[0144] To further analyze the scheduling characteristics of different methods, one typical day each during the flood season and the non-flood season was selected. The water level and flow processes of the scheduling results can be found in [reference needed]. Figure 7 Please refer to the output process. Figure 8The scheduling results of the HWS_SAC of this invention exhibit clear characteristics that conform to the physical laws of hydropower operation: the learned strategy tends to maintain the reservoir water level at a relatively high level and release water more concentratedly under high head conditions, thereby effectively improving power generation efficiency. Simultaneously, this strategy can achieve a stable distribution of water throughout the scheduling period, avoiding both excessive water release in the early stages leading to premature water depletion and large-scale power curtailment caused by concentrated water release in the later stages. In contrast, traditional SAC, due to its use of cumulative reward for value estimation, learns a strategy that excessively focuses on immediate rewards at the beginning of the scheduling period, exhibiting a more aggressive water release behavior, thus increasing early power generation. However, as water resources decrease later, this strategy can only operate at near-minimum outflow. This scheduling pattern of excessive water consumption in the early stages persists in both flood and non-flood seasons, leading to an increased frequency of constraint violations. TD3 typically retains a certain amount of water in the early stages of the scheduling period to raise the reservoir water level and enhance hydropower efficiency in subsequent periods, and then significantly increases water release in the later stages of the scheduling to increase power generation and meet the final water level constraint. While this water resource allocation model is reasonable to some extent, its adaptability to changes in inflow and wind and solar power output is relatively limited. In actual operation, this strategy often keeps reservoir water levels close to the daily upper limit for extended periods, followed by concentrated water releases, which can easily lead to significant water and power wastage during the flood season.

[0145] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A water, wind, and solar scheduling method considering constraint correction and average reward reinforcement learning, characterized in that: Includes the following steps: S1 constructs a mathematical model for the operation of a water-wind-solar hybrid system and applies constraints, including the following steps: S1.1 Construct a mathematical model for the integrated operation of hydropower, wind power and photovoltaic power, with the goal of maximizing the total power generation of the combined operation of hydropower, wind power and photovoltaic power. The system constraints include reservoir characteristic constraints, water balance constraints, water level constraints, flow constraints, output constraints and channel capacity constraints. S1.2 Utilizes the concept of water level corridors to uniformly map water level, flow rate, and output constraints to the feasible water level range for a given time period, thereby achieving direct modeling of hard constraints; Based on the operational constraints of the hydropower station, the maximum allowable water level for time period t is calculated in reverse. With minimum allowable water level And form a reverse water level range The maximum allowable water level for time period t is calculated by recursively estimating the water level for time period t in the forward direction based on the water level for time period t-1. With minimum allowable water level and constitute a positive water level range. ; S2 constructs a water-wind-solar hybrid operating environment suitable for reinforcement learning. The mathematical model obtained in step S1 and the mapping result of the water level corridor are encapsulated into a reinforcement learning environment, which consists of a state space, action space, state transition function, and reward function. The environment is in the state... Next action The next state will be obtained. With instant rewards ; S2 specifically includes the following steps: The state space of the S2.1 hydro-wind-solar hybrid operating environment includes time period, temperature, radiation intensity, wind speed, water level, upper and lower limits of water level for the next time period, and inflow. S2.2 defines the action space of the hydro-wind-solar hybrid operating environment as the reservoir outflow, i.e., the action in time period t. Outbound flow rate during time period t ; S2.3 Based on the state space defined in step S2.1 and the action space defined in step S2.2, given the current state and the action to be executed, the environment completes the state transition according to the reservoir water balance relationship and system operation constraints. Specifically, the state transition function of the hydro-wind-solar hybrid operating environment includes two parts: state transition and action correction. Given the action... The water level at time t+1 is obtained from the state transition equation. If the water level exceeds the feasible range, hard constraints are applied based on the corridor to obtain the corrected state and action. S2.4 After the state transition is completed in step S2.3, the environment calculates the immediate reward based on the current state and the executed action; the reward function of the hydro-wind-solar hybrid operating environment is the state. Next action Rewards This includes the total amount of hydropower, wind power, and solar power generation, as well as penalties for violating water level constraints; S3 constructs a reinforcement learning SAC agent suitable for water-wind-solar complementary operation; on the environment constructed in step S2, a SAC-based agent is constructed; the SAC agent consists of a policy network, two independent value networks and two corresponding target value networks, and the SAC agent learns the optimal scheduling strategy through interaction with the environment. S4 trains the parameters of the SAC agent constructed in step S3 using a large amount of historical data, and learns the optimal strategy that maximizes environmental rewards. After the S5 hydro-wind-solar hybrid actual optimization scheduling is completed in step S4, the trained strategy network is deployed to the actual scheduling system and optimized scheduling is implemented to generate a complete scheduling scheme.

2. The water, wind, and solar scheduling method considering constraint correction and average reward reinforcement learning as described in claim 1, characterized in that: In step S1.1, the mathematical model is as follows: In the formula: This represents the total power generation of the complementary system. , and Let represent the power output of the wind power station, photovoltaic power station, and hydropower station respectively during time period t. T This represents the total number of time periods during the scheduling period. It is the length of the scheduling period, in hours.

3. The water, wind, and solar scheduling method considering constraint correction and average reward reinforcement learning as described in claim 2, characterized in that: In step S2.1, the state during time period t is: : In the formula, and Let t be the ambient temperature and solar radiation intensity of the photovoltaic power station during time period t. For the wind speed of the power plant, These are the upper and lower limits of the water level for the time period t+1. The reservoir water level during time period t. This refers to the inbound flow during the scheduling period; In step S2.3, given the action The water level at time t+1 is obtained from the state transition equation. : In the formula, , These represent the reservoir water level and inflow rate at time t. This represents the water level-capacity curve of a reservoir. If the water level exceeds the feasible range, then a hard constraint correction will be made based on the corridor: In the formula, This refers to the corrected outflow and reservoir water level.

4. A water, wind, and solar scheduling method considering constraint correction and average reward reinforcement learning as described in claim 3, characterized in that: State in step S2.4 Next Action Rewards : In the formula, Let t be the total power generation during period t. Penalties for violating water level constraints, This is the penalty coefficient; in, and The calculation formula is as follows: In the formula, This indicates the maximum capacity of the outgoing channel.

5. A water, wind, and solar scheduling method considering constraint correction and average reward reinforcement learning as described in claim 1, characterized in that: S3 specifically includes the following steps: S3.1 Constructs a policy network, which consists of a multilayer perceptron with hidden nodes of size `hidden_size` and hidden layers of `n`, and stores the current state. The mapping is to continuous action distribution parameters, and the policy network input is the current state. The output action distribution has a mean and variance. By sampling this distribution and mapping it to the decision action range, the actual actions to be executed can be obtained. ; S3.2 Based on the policy network generating scheduling actions in step S3.1, a value network is further constructed to estimate the value function corresponding to the policy network state-action. To reduce overestimation bias, two independent Critic networks are used. Each value network is a multilayer perceptron structure with the same number of hidden layers and nodes as the policy network, and the parameters are independent of each other, thereby achieving robust evaluation of the value of scheduling actions. S3.3 Based on the two sets of value networks in step S3.2, construct target value networks with the same structure to estimate the value function of the state-action in the next time period. The target value network has the same structure as the value network. During training, the value network parameters are synchronized in a soft update manner, thereby providing a relatively smooth and stable learning target when updating the value function and improving the stability of the SAC training process.

6. A water, wind, and solar scheduling method considering constraint correction and average reward reinforcement learning as described in claim 1, characterized in that: S4 specifically includes the following steps: S4.1 Initialize Policy Network Parameters Value network parameters Set target value network parameters Construct an experience replay buffer with a capacity of buffer_size; set the learning rate. Soft update parameters Sampling batch size and total number of training steps; The S4.2 agent interacts with the environment according to the current policy and continuously collects data. The data is then stored in the playback pool; S4.3 Randomly sample small batches of data from the replay pool to update the policy and value network parameters; to adapt to the operation of hydro-wind-solar hybrid systems with fixed scheduling cycles, the average value of all future rewards is used. As the target value of the value network; The average of all future rewards The expression is as follows: In the formula, and Let N represent the rewards for time period t and time period t+k, respectively, and let N be the number of remaining scheduling steps from the current time period t. This represents the average of all rewards during period t and all future rewards. The average of the t+1 period and all future rewards estimated by the target value network.

7. A water, wind, and solar scheduling method considering constraint correction and average reward reinforcement learning as described in claim 6, characterized in that: The loss function of the value network in step S4.3 The value network parameters are updated by minimizing the loss function using mean squared error: In the formula, The estimated results are for the value network i-to-t time period and the average of all future rewards; The policy network is updated via policy gradients, with the objective of maximizing the value network estimate. : In the formula, Let the loss function be the policy network. For the policy network in state The estimated value of the lower value network; Based on soft update parameters The parameters of the target value network are updated using a soft update method to maintain learning stability. In the formula, , These are the value network parameters and the target value network parameters, respectively.

8. A water, wind, and solar scheduling method considering constraint correction and average reward reinforcement learning as described in claim 1, characterized in that: Implementing optimized scheduling in S5 specifically includes the following steps: S5.1 Receive real-time or short-term forecast data, including temperature, radiation intensity, wind speed, water level, and inflow during the scheduling period; input the above data into the hydro-wind-solar hybrid operation environment constructed in step S2, and construct the state vector for the current time period based on the environmental state definition. ; S5.2 policy network outputs scheduling actions based on the current environment state. The trained policy network is invoked to generate outbound traffic decisions for the current time period. The actions are mapped through the internal environment and transformed into actual hydropower station discharge flow control commands. S5.3 will take action Input the operating environment, and obtain the environment state for the next time period according to the environment state transition mechanism in step S2. and the actual power generation during that period The environment automatically corrects actions that violate the constraints based on the hard constraints of the water level corridor. S5.4 Repeat steps S5.2 and S5.3, iterating through each time period until the end of the scheduling period. After the scheduling is completed, a complete scheduling scheme is generated, which includes the reservoir water level process for each time period; the discharge flow process; the power output process of hydropower, wind power, and photovoltaic power for each time period; and the statistical results of the combined hydropower, wind power, and photovoltaic power generation. This scheme serves as the basis for actual operation control or scheduling decisions.

9. A computer device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when loaded and executed by the processor, implements the water, wind and solar scheduling method as described in any one of claims 1 to 8.

Images