Intra-day scheduling method and system for water-wind-light power system based on constraint reinforcement learning

Abstract

The invention discloses a constraint reinforcement learning-based intra-day scheduling method and system for a water-wind-light electric power system. The method comprises the following steps: constructing an intra-day optimization scheduling model oriented to water, fire and storage and direct current delivery collaboration, wherein the model comprises an objective function and constraint conditions; converting the intra-day optimization scheduling model into a Markov decision process, and defining a state, an action, an award and cost; constructing a neural network, wherein the neural network comprises a strategy network and a value network; training the neural network through imitation learning and self-supervised learning algorithms; a Lagrangian multiplier method is used to convert a constraint optimization problem into a minimum-maximum optimization problem, and strategy network parameters and Lagrangian multipliers are alternately updated; and obtaining historical operation data and an environment state of the water-wind-light power system, and inputting the historical operation data and the environment state into the trained neural network to obtain a scheduling decision scheme. The efficiency, safety and reliability of generating the scheduling scheme can be improved.

Description

Technical Field

[0001] This invention relates to the field of power system technology, and in particular to a method, system, storage medium, and electronic device for intraday scheduling of hydropower, wind power, and solar power systems based on constrained reinforcement learning. Background Technology

[0002] To address the volatility and randomness of wind and solar power generation, clean energy bases typically incorporate energy storage as a flexible regulation resource to enhance the grid's peak-shaving capacity. Furthermore, due to the distinct regional characteristics of my country's energy source-load distribution, clean energy is mainly concentrated in the "Three Norths" (Northeast, North, and Northwest China) and Southwest regions, where local consumption capacity is insufficient. Therefore, cross-regional transmission via high-voltage direct current (HVDC) has become a crucial means of absorbing clean energy. Multi-energy complementarity of wind, solar (hydro), thermal, and energy storage, along with HVDC transmission, has become a prominent feature of high-proportion renewable energy transmission systems.

[0003] However, the intraday dispatching of high-proportion renewable energy power transmission systems currently faces numerous challenges. First, with the increasing proportion of renewable energy generation, the total power deviation under the same forecast accuracy increases significantly, requiring frequent adjustments to power generation plans during the intraday phase. This weakens the reference value of day-ahead plans, leading to substantial deviations between real-time power flow and day-ahead plans. Second, as renewable energy replaces conventional power sources, the system's conventional regulation capacity is compressed, and the flexibility of resource coordination during the intraday phase is not fully utilized. Large-scale renewable energy and load control measures are becoming increasingly frequent in extreme scenarios. Third, renewable energy generation exhibits anti-peak-shaving characteristics, significantly increasing the difficulty of system regulation. Intraday dispatching models relying on conventional units for deviation proportional allocation have limited global computational capabilities and lack forward-looking capabilities, making it difficult to perceive future operational risks and reserve regulation capacity. Therefore, how to improve the utilization rate of existing channels and promote renewable energy consumption through multi-resource collaboration and multi-time-period joint optimization during the intraday phase, while achieving efficient solutions within the specified time, has become a critical issue that urgently needs to be addressed.

[0004] Data-driven deep reinforcement learning methods offer the advantage of end-to-end decision-making, eliminating the reliance on precise physical models and effectively handling complex problems such as high-dimensionality, nonlinearity, and uncertainty, providing an intelligent solution. However, the operation of hydropower, wind power, and solar power systems is significantly affected by seasonality and experiences frequent intraday power flow changes, resulting in complex and diverse operating scenarios. Existing reinforcement learning methods suffer from limited feature extraction capabilities, insufficient decision-making abilities, and weak generalization. Furthermore, the reduced safety margin of new power systems places higher demands on decision-making security. Summary of the Invention

[0005] This invention provides a method, system, storage medium, and electronic device for intraday scheduling of hydropower, wind power, and solar power systems based on constrained reinforcement learning. It can overcome the limitations of traditional reinforcement learning algorithms in complex constrained optimization problems, such as difficulty in convergence and poor policy feasibility, improve the efficiency and reliability of generating scheduling schemes, and enhance the capacity for clean energy consumption.

[0006] This invention provides a method for intraday scheduling of hydropower, wind power, and solar power systems based on constrained reinforcement learning, comprising: Construct an intraday optimal scheduling model for the coordinated operation of hydropower, thermal energy storage, and DC power transmission, including the objective function and constraints; The intraday optimization scheduling model is transformed into a Markov decision process, defining state, action, reward, and cost; Construct a neural network, which includes a policy network and a value network; The neural network is trained using a deep reinforcement learning algorithm; The constrained optimization problem is transformed into a mini-max optimization problem using the Lagrange multiplier method, and the policy network parameters and Lagrange multipliers are updated alternately. Historical operating data and environmental conditions of the hydropower, wind power, solar power, and electric power systems are acquired, and the historical operating data and environmental conditions are input into a trained neural network to obtain a scheduling decision scheme.

[0007] Furthermore, according to the above-mentioned intraday scheduling method for hydro-wind-solar power systems based on constraint reinforcement learning, the objective function is: ; in, To optimize the number of time periods ， As a set of adjustable resources, water, thermal power, energy storage, and direct current (DC) were selected as the control targets. and To regulate resources i exist t Adjustments to the current day's plan, both positive and negative, should be made continuously. To regulate resources i The unit adjustment cost; The constraints include node injection power constraints, AC power flow constraints, thermal power operation constraints, cascade hydropower operation constraints, energy storage operation constraints, and DC operation constraints.

[0008] Furthermore, according to the above-mentioned intraday scheduling method for hydropower, wind power, and solar power systems based on constraint reinforcement learning, the policy network includes a gated recurrent unit. The gated recurrent unit represents the dynamic correlation between historical operating states and future scheduling actions through hidden states. The processing procedure of the gated recurrent unit is expressed by the following formula: ; ; ; ; in, and These represent updating the door and resetting the door, respectively. , express t Candidate hidden states and hidden states at time step. , , , , , Let represent the weight matrix and bias vector of the update gate, reset gate, and candidate hidden state, respectively. This is the activation function.

[0009] Furthermore, according to the above-mentioned intraday scheduling method for hydro-wind-solar-power systems based on constraint reinforcement learning, the value network includes a graph convolutional network, which is used to transform environmental states into graph data and a high-order representation of the graph data; No. l The calculation formula for layer graph convolutional networks is as follows: ; in, For the first l The node feature matrix of the layer It is an adjacency matrix with self-connections. It is the identity matrix. yes The degree matrix, , It is a non-linear activation function. For the first l The weight matrix that can be trained in a layer GCN.

[0010] Furthermore, according to the above-mentioned intraday scheduling method for hydropower systems based on constrained reinforcement learning, the training process of the neural network includes: Phase 1: Introducing an imitation learning mechanism to train the policy network: Constructing a supervised learning loss based on historical state sequences and corresponding historical scheduling decisions, and training the policy network based on the supervised learning loss; The second stage involves predicting rewards and costs, calculating the mean squared error between the predicted and actual values ​​of rewards and costs, and training the value network based on the mean squared error.

[0011] Furthermore, according to the above-mentioned intraday scheduling method for hydro-wind-solar power systems based on constraint reinforcement learning, the method further includes: An unsafe action adjustment module based on expert experience is constructed to correct the actions output by the policy network during training.

[0012] Furthermore, according to the above-mentioned intraday scheduling method for hydro-wind-solar power systems based on constraint reinforcement learning, the correction of the actions output by the policy network includes: Obtain the adjusted power generation plan and the allowable operating range of each controllable resource from the intelligent agent; Determine if the balancing machine exceeds its limits; if so, perform a secondary distribution of the unbalanced power. Determine if there is a line exceeding the limit. If so, adjust the line exceeding the limit based on sensitivity.

[0013] This invention also provides a daytime dispatching system for hydropower, wind power, solar power, and electric power systems based on constrained reinforcement learning, comprising: The scheduling model construction module is used to build an intraday optimized scheduling model for the coordination of hydropower, thermal energy storage and DC power transmission, including the objective function and constraints. The transformation module is used to transform the intraday optimized scheduling model into a Markov decision process, defining the state, action, reward, and cost. A neural network building module is used to build neural networks, which include a policy network and a value network. An auxiliary training module is used to train the neural network using a deep reinforcement learning algorithm; The update module is used to transform the constrained optimization problem into a mini-maximum optimization problem using the Lagrange multiplier method, and alternately updates the policy network parameters and Lagrange multipliers; The decision-making module is used to acquire historical operating data and environmental conditions of the hydropower system, wind power system, and solar power system, and input the historical operating data and environmental conditions into a trained neural network to obtain a scheduling decision scheme.

[0014] The present invention also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described constrained reinforcement learning-based intraday scheduling methods for hydro-wind-solar power systems.

[0015] The present invention also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for the steps in the intraday scheduling method for hydro-wind-solar power systems based on constraint reinforcement learning described in any of the preceding claims.

[0016] This invention provides a method, system, storage medium, and electronic equipment for intraday scheduling of hydropower, wind power, and solar power systems based on constrained reinforcement learning. First, an intraday optimal scheduling model considering the coordinated operation of hydropower, thermal power, energy storage, and DC transmission is constructed and transformed into a constrained Markov decision process. Second, a neural network is designed and trained. Then, a minimum-maximum optimization problem is solved based on the Lagrange multiplier method, optimizing by alternately updating the original and dual variables and adaptively adjusting the penalty weights of safety constraints. Finally, the updated neural network is trained to predict scheduling decision schemes. This application overcomes the limitations of traditional reinforcement learning algorithms in complex constrained optimization problems, such as difficulty in convergence and poor policy feasibility, accelerating the efficiency and reliability of generating scheduling schemes and improving the clean energy absorption capacity. Attached Figure Description

[0017] The technical solution and other beneficial effects of the present invention will become apparent from the following detailed description of specific embodiments of the invention, in conjunction with the accompanying drawings.

[0018] Figure 1 A flowchart of a daytime scheduling method for hydro-wind-solar power systems based on constraint reinforcement learning provided in an embodiment of the present invention.

[0019] Figure 2 A flowchart illustrating the training process of a neural network provided in an embodiment of the present invention.

[0020] Figure 3 This is a flowchart illustrating the correction of actions output by a policy network, provided as an embodiment of the present invention.

[0021] Figure 4 A schematic diagram of the intraday scheduling system for hydropower, wind power, solar power, and electric power systems based on constraint reinforcement learning, provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] This invention provides a method, system, storage medium, and electronic device for intraday scheduling of hydropower systems based on constrained reinforcement learning. The intraday scheduling system for hydropower systems based on constrained reinforcement learning provided by this invention can be integrated into an electronic device, such as a terminal or server. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0024] Please see Figure 1 , Figure 1 The flowchart illustrates a constrained reinforcement learning-based intraday scheduling method for hydro-wind-solar-power systems, which is applied in electronic devices and includes the following steps: S1. Construct an intraday optimal scheduling model for the coordinated operation of hydropower, thermal energy storage, and DC power transmission, including the objective function and constraints.

[0025] In one embodiment, intraday adjustment cost is used as the optimization objective, i.e., the objective function: ; in, To optimize the number of time periods (in one specific embodiment, the scheduling time interval is selected as 15 minutes), ) ， As a set of adjustable resources, water, thermal power, energy storage, and direct current (DC) were selected as the control targets. and To regulate resources i exist t Adjustments to the current day's plan, both positive and negative, should be made continuously. To regulate resources i The unit adjustment cost.

[0026] The intraday optimized scheduling model includes the following constraints: (1) Node injection power constraints ; ; in, , Let be the active and reactive power flowing from node i to node j at time t; , , , , , , These represent the intraday planned values ​​or ultra-short-term forecast values ​​at time t for the thermal power, hydropower, energy storage, DC, photovoltaic, wind power, and load connected to node i; the intraday plans for hydropower, thermal power, energy storage, and DC transmission are... , This is the base value for the previous day; Let i be the set of nodes connected to node i.

[0027] (2) Constraints of communication trends ; ; ; ; in, , Let be the conductance and susceptance of branch ij; Let be the voltage amplitude at node i at time t; Let be the phase angle difference between nodes i and j at time t; The maximum allowable transmission power for branch ij; , This represents the upper and lower limits of the allowed voltage amplitude at node i.

[0028] (3) Constraints on thermal power plant operation The operating constraints of thermal power units take into account the upper and lower limits of output and the ramping constraints.

[0029] ; ; in, The start-up and shutdown status of thermal power plant i at time t, as determined by the previous plan; , and These are the upper and lower limits of active power output and the ramp-up limit for thermal power plant i.

[0030] (4) Operational constraints of cascade hydropower In addition to the upper and lower limits of power output and the ramping constraint, cascade hydropower stations also include constraints on hydropower conversion, water balance, power generation flow, upper and lower limits of reservoir capacity, and first and last reservoir capacity.

[0031] ; ; ; ; ; ; in, , , Let i be the hydropower conversion efficiency, initial head, and head coefficient of hydropower station i. , Let i be the hydropower head and power generation flow rate of hydropower station i at time t; Let be the natural inflow of hydropower station i at time t; assuming hydropower station j is located upstream of hydropower station i, its water flow time delay is . ; , Let i be the minimum and maximum allowable power generation flow rates for hydropower station i; , Let i be the minimum and maximum allowable reservoir capacity of hydropower station i.

[0032] (5) Constraints on Energy Storage Operation Energy storage constraints include charge and discharge power constraints, state of charge and discharge constraints, state of charge (SOC) transition constraints, upper and lower limits of SOC constraints, and first and last SOC constraints.

[0033] ; ; ; ; ; ; in, , Let i be a binary variable representing the charging and discharging state of energy storage at time t; , Let i be the charging and discharging power of energy storage at time t; Let i be the state of charge of energy storage at time t; , , The maximum allowable charge / discharge power and upper and lower limits of state of charge for energy storage i; , The charging and discharging efficiency of energy storage i; Let i be the capacity of energy storage. For scheduling time intervals, this application takes .

[0034] (6) DC operation constraints Considering that high-voltage direct current converter equipment should not be frequently adjusted and that the power transmission curve exhibits a stepped shape, DC operation constraints include transmission capacity limits, adjustment range limits, ramping constraints, ramping state constraints, and minimum adjustment time interval limits.

[0035] ; ; ; ; ; ; in, The operating status of DC i at time t as determined by the previous plan; , The minimum and maximum allowable transmission power for DC i; This is to adjust the current plan within a certain limit; , Let i be the binary variable representing the upward and downward adjustment states of DC at time t; For DC i, the ramp limit; This is the minimum allowed adjustment time interval.

[0036] S2 transforms the intraday optimization scheduling model into a Markov decision process, defining states, actions, rewards, and costs.

[0037] For the intraday optimal scheduling problem involving multiple resources, with thermal power, hydropower, energy storage, and DC power as the control targets, the main elements of its constrained Markov decision process model are defined as follows: (1) Status: This includes the current operating status and allowable operating range of controllable resources, the load rate of each line and the voltage value of each node, as well as the load and ultra-short-term forecast of new energy sources for the next period and the day-ahead power generation plan of controllable resources.

[0038] ; in, and It is a controllable resource i exist t The lower and upper limits of the allowed operating range at any given time; For the line i exist t Load rate at any given time.

[0039] Considering operational constraints, the allowable operating ranges for thermal power, hydropower, energy storage, and DC power are calculated as follows. ; ; ; ; ; ; ; ; in, This is the time interval since the last DC power adjustment.

[0040] hydroelectric units and The calculation method is as follows: Substituting the water balance equation into the constraints of hydropower conversion and reservoir capacity limits, we can obtain... ; ; Substituting the above formula into the hydroelectric conversion constraint, we can obtain... ; For single-step decision-making, the above formula only has Let be the decision variables. Therefore, the upper and lower limits of the hydropower unit's output are transformed into the following formula, which can be solved analytically using the method of finding the extrema of a univariate function.

[0041] ; .

[0042] 2) Actions: These include power generation commands for thermal and hydropower, power charging and discharging commands for energy storage, and DC scheduling adjustment commands. The initial action range output by the intelligent agent's neural network is [-1, 1], which is then converted into power scheduling commands based on the allowable operating range of the controllable equipment. ; 3) Reward: Transform the objective function into the agent's immediate reward. Since the training objective of reinforcement learning is to maximize the cumulative reward, it is the negative value of the objective function.

[0043] ; After the agent completes the correction of the controllable resource power generation plan, any remaining power deviation in the system will be handled by the balancing machine. If the remaining power deviation exceeds the upper and lower limits of the balancing machine's adjustable capacity, the adjustment task for this round is considered a failure, and the agent receives a negative reward. .

[0044] 6) Cost: The cost function is used to evaluate whether the agent's actions violate constraints. This application's embodiments categorize the constraints of the optimization scheduling problem into three types: physical constraints, device-level safety constraints, and system-level safety constraints. Physical constraints primarily use equations to describe the laws of electrical physics and their underlying mechanisms, such as the AC power flow equations of the system, the hydroelectric conversion relationship and water balance equations of hydropower units, and the state-of-charge transition equations of energy storage. Within the reinforcement learning framework, these physical constraints are embedded in the simulation environment, enabling real-time updates of device and system states through environmental simulation, effectively avoiding the limitations of model-driven methods in approximating nonlinear constraints. For device-level safety constraints, except for the first and last reservoir capacity constraints and the first and last SOC constraints which involve long-term coupling constraints, the action space designed in this application ensures that safety constraints are strictly satisfied, shielding a large number of invalid actions that violate safety constraints. Therefore, the agent only needs to focus on system-level safety constraints and device-level long-term coupling safety constraints, reducing the difficulty of the agent's optimization exploration. , , and The scenarios in which the agent's actions violated line transmission capacity constraints, node voltage constraints, and the first and last state constraints of hydropower and energy storage were evaluated.

[0045] ; ; ; ; Considering that the reservoir capacity and energy storage state of charge need to be restored to a level close to the initial state at the end of the dispatch cycle, while a large deviation in state is allowed in other stages, this application introduces a time-varying weighting coefficient, making In the final stage of scheduling, a larger weighting coefficient is applied to guide the agent to prioritize the execution of the reservoir capacity and charge state recovery control strategy. In summary, the agent's cost is... ; in, The penalty cost for early termination of a round; , , and These are the weighting coefficients for each item.

[0046] S3, construct a neural network, which includes a policy network and a value network.

[0047] In one embodiment, the neural network integrates temporal memory and graph feature embedding to enhance its ability to represent the temporal operating state and the relationship between the power grid topology. Step S3 includes: S31, design a policy network architecture that integrates temporal memory.

[0048] Because hydropower and energy storage are subject to power constraints, their optimization decisions are strongly coupled with time. That is, current scheduling actions not only affect immediate rewards and costs, but also influence the feasible decision space for multiple future time periods through variables such as reservoir capacity, hydropower head, and energy storage state of charge.

[0049] Traditional policy networks based on fully connected neural networks, due to structural limitations, cannot explicitly capture the dependencies of time series, resulting in short-sighted scheduling decisions and insufficient dynamic adaptability in complex scenarios. Therefore, this application introduces a Gated Recurrent Unit (GRU) into the policy network. Through hidden states, the GRU represents the dynamic relationship between historical running states and future scheduling actions, achieving a memory function and assisting the agent in making optimal decisions with limited local observations, maximizing efficiency within the scheduling cycle. Compared to Long Short-Term Memory (LSTM) networks, GRU has fewer training parameters and higher computational efficiency. Its shorter gradient propagation path effectively alleviates the gradient vanishing problem. Its calculation formula is shown below: ; ; ; ; in, and These represent updating the door and resetting the door, respectively. , express t Candidate hidden states and hidden states at time step. , , , , , Let represent the weight matrix and bias vector of the update gate, reset gate, and candidate hidden state, respectively. This is the activation function.

[0050] S32, Value Network Architecture with Feature Embedded Design Graphs In the design of the value network architecture, a Graph Convolutional Network (GCN) is introduced to extract power grid graph features. Power system topology has graph data format. The busbar configuration diagram includes node V, the branch configuration diagram data of transmission lines or transformers, and the signals on the configuration diagram of electrical physical quantities such as line current, node voltage, and node injected power.

[0051] Traditional models based on fully connected neural networks neglect the topological coupling of electrical physical quantities in the spatial dimension, and the fully connected mechanism forces all neurons to calculate the electrical physical characteristics of all nodes, failing to fully reflect the inherent physical mechanisms of the power system. Therefore, the embodiments of this application will consider the environmental state. The data is transformed into graph data, and a high-order representation of the graph structure data is obtained by aggregating the feature information of itself and its neighboring nodes based on GCN.

[0052] No. l The calculation formula for layer graph convolutional networks is as follows: ; in, For the first l The node feature matrix of the layer It is an adjacency matrix with self-connections. It is the identity matrix. yes The degree matrix, , It is a non-linear activation function. For the first l The weight matrix that can be trained in a layer GCN.

[0053] Assumption Unconnected nodes correspond to zero elements in the matrix, while connected nodes are non-zero. This is achieved through... conduct The left-multiplication transformation enables node feature updates to rely solely on sparse computation using the feature information of its first-order neighboring nodes, achieving a feature propagation mechanism that conforms to the physical laws of power systems. The first-layer GCN establishes local correlations through feature aggregation of first-order neighboring nodes. The second-layer GCN further expands the receptive domain based on the first layer, achieving aggregation of second-order neighboring node information through the hidden features output from the previous layer. This hierarchical, progressive structure based on GCN establishes a multi-level feature extraction system from local to non-local perspectives, enhancing the state representation capability of the value network.

[0054] S4 uses imitation learning and self-supervised learning algorithms to assist in training the neural network.

[0055] Intraday scheduling with multi-resource collaboration is a dynamic optimization problem involving high-dimensional decision variables and coupled security constraints. Traditional reinforcement learning methods rely solely on reward / cost signals to guide the agent, leading to low training efficiency and potential convergence to infeasible strategies that frequently violate security constraints, thus failing to meet the reliability requirements of practical scheduling operations. Considering that historical scheduling decision databases contain rich feasible scheduling experience, this application proposes a two-stage training framework guided by historical knowledge. Figure 2 The training flowchart of the neural network provided in this embodiment of the invention includes the following steps: S41, First stage: Introducing an imitation learning mechanism to train the policy network: Constructing a supervised learning loss based on historical state sequences and corresponding historical scheduling decisions, and training the policy network based on the supervised learning loss.

[0056] The first stage employs an imitation learning method, utilizing historical scheduling decisions to construct supervised learning signals and guide the policy network to quickly converge to the neighborhood of feasible policies. This is followed by a free exploration phase where the agent performs a refined search within the feasible region, optimizing economic objectives while ensuring policy safety. The training loss function for the first stage is defined as follows: ; In the formula, To use historical decision database The extracted state-action pairs.

[0057] S42, Second Stage: Predict rewards and costs, calculate the mean squared error between the predicted and actual values ​​of rewards and costs, and train the value network based on the mean squared error.

[0058] This application introduces a self-supervised learning method, using reward / cost as labels to establish a prediction auxiliary task, enhancing the agent's information extraction capability in high-dimensional state space. The value network is decomposed into a feature extraction module and a value function decision module, with an additional reward / cost decision module. Most existing reinforcement learning methods adopt an end-to-end approach to learn the mapping from state to value function, implicitly including the state representation process. However, these methods have limited ability to extract state information features and ignore the rich potential information of the environment during reinforcement learning training, reducing the model's approximation accuracy of the value function and sample utilization. Therefore, this application constructs an auxiliary training task to provide supervision signals for model training, assisting the upstream feature extraction module in updating network parameters and improving the value network's ability to perceive events such as power imbalance, line over-limit, and voltage over-limit.

[0059] Assuming the feature extraction module outputs: , and The prediction network, representing the reward and cost respectively, outputs the following predictions: ; ; The network parameters of the feature extraction module and the reward / cost prediction module are optimized by minimizing the mean square error between the predicted output and the true value.

[0060] S5 uses the Lagrange multiplier method to transform the constrained optimization problem into a mini-maximum optimization problem, and alternately updates the policy network parameters and Lagrange multipliers.

[0061] SAC (Soft Actor-Critic) is an off-policy algorithm suitable for continuous action spaces. The policy network generates a Gaussian distribution of actions instead of directly outputting deterministic actions, thus avoiding the monotony of actions. Furthermore, by introducing entropy regularization, SAC balances cumulative reward and policy randomness during policy optimization, effectively enhancing the agent's exploratory ability and preventing premature convergence to local optima. Therefore, this embodiment extends the SAC algorithm to the field of constrained reinforcement learning and, based on the aforementioned agent (neural network) architecture, proposes the CSAC algorithm, which imposes entropy constraints on the reinforcement learning training objective. ,in Let be the target entropy of the agent.

[0062] For constrained optimization problems, this embodiment introduces Lagrange multipliers to integrate the constraints into the objective function, transforming it into an unconstrained optimization problem, as shown below. ; In the formula, and These are the Lagrange multipliers corresponding to the agent's policy security constraints and policy stochasticity constraints, respectively.

[0063] The agent updates its inner and outer layer parameters alternately during each training iteration. The inner layer parameters are updated using policy gradient ascent, and the outer layer parameters are updated using dual gradient descent. The training framework is as follows: Figure 2 As shown.

[0064] Policy Network Update: Taking the inner layer maximization problem as the optimization objective, the loss function of the neural network is as follows. ; In the formula, This is an experience replay pool.

[0065] The network parameter update method is as follows: ; The learning rate is the parameter of the policy network.

[0066] Lagrange multiplier update: The outer minimization problem is used as the optimization objective, and its objective function is as follows. ; ; The update method for Lagrange multipliers is as follows: ; ; , The learning rate; It is the Lagrange multiplier Controlled Projection function within the range; Similarly.

[0067] Value Network Update: With the goal of minimizing the mean square Bellman error, the loss function of the neural network is as follows. ; ; ; ; in, and These are the online network of the agent and the target network, respectively. Both have the same network architecture, but the target network updates its parameters slowly to stabilize training, while this embodiment uses a soft update method. , This is the soft update coefficient.

[0068] In addition, the aforementioned self-supervised learning task updates the network parameters of the feature extraction module and the reward / cost prediction module of the value network.

[0069] Furthermore, this method also includes: An unsafe action adjustment module based on expert experience is constructed to correct the actions output by the policy network during training.

[0070] Specifically, constrained Markov decision processes aim to ensure that the cumulative degree of constraint violation does not exceed a specified threshold, but they cannot guarantee zero violation of single-step safety constraints, especially in scenarios unseen during the training phase, where data-driven methods perform poorly. Therefore, this application introduces an unsafe action adjustment module to post-process the actions generated by the agent, ensuring strict satisfaction of single-step constraints. This module focuses on the output constraints of the balancing machine and the transmission capacity constraints of the line, designing a secondary unbalanced power allocation mechanism and a sensitivity-based line over-limit adjustment mechanism. Figure 3 A flowchart for correcting the actions output by the policy network, as provided in an embodiment of the present invention, is shown below. Figure 3 As shown, the specific steps include: A1, obtain the adjusted power generation plan of the intelligent agent and the allowable operating range of each controllable resource.

[0071] A2, determine if there is a limit exceeding the balancing machine. If so, perform a secondary distribution of unbalanced power.

[0072] Since the remaining power deviation after the agent adjusts the power generation plan is borne by the balancing machine, a secondary power distribution is performed for unbalanced power exceeding the balancing machine's adjustable range. The balancing machine's over-limit power is calculated using the following formula. ; Based on the aforementioned allowable operating ranges of each controllable resource, and after adjustments by the intelligent agent within the day, the remaining upward adjustment capacity... and lowering ability The calculation method is as follows: ; ; After obtaining the imbalance and the remaining adjustment capacity of each resource, they are sequentially activated according to their adjustment costs from low to high until power balance is achieved. For example... A positive signal indicates a power shortage in the system, requiring increased output of thermal and hydropower, reduced energy storage charging power, or increased discharging power, and reduced DC power transmission; conversely, a positive signal indicates a power shortage.

[0073] A3. Determine if there is a line exceeding the limit. If so, adjust the line exceeding the limit based on the sensitivity.

[0074] The sensitivity-based line over-limit adjustment mechanism is as follows: if line i in the system exceeds the limit, the units are sorted according to their sensitivity to line i, the units with higher sensitivity are adjusted to eliminate line power, and then the units with lower sensitivity are adjusted in reverse to achieve power balance of the system.

[0075] In power systems with a low proportion of renewable energy, intraday adjustment methods based on expert experience are usually effective. However, with the continuous increase in renewable energy penetration, traditional methods still have shortcomings in multi-resource coordination, multi-time period coordination, and multi-objective balancing, making it difficult to support objectives such as ensuring safety, ensuring consumption, and ensuring supply. In contrast, reinforcement learning methods, as a near-global optimization solution, significantly reduce system operation risks and adjustment difficulty even when single-step safety constraints cannot be met. Therefore, this application selects intraday adjustment methods based on expert experience as a supplement for secondary correction in emergency situations.

[0076] S6 acquires historical operating data and environmental status of the hydropower system, wind power, solar power system, and inputs the historical operating data and environmental status into the trained neural network to obtain the scheduling decision scheme.

[0077] The dispatching decision scheme specifically refers to the daily power generation plan.

[0078] Based on the method described in the above embodiments, this embodiment will further describe the intraday scheduling system of hydro-wind-solar-power system based on constraint reinforcement learning. The intraday scheduling system of hydro-wind-solar-power system based on constraint reinforcement learning can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other device. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0079] Please see Figure 4 , Figure 4 This invention specifically describes a constrained reinforcement learning-based intraday scheduling system for hydro-wind-solar-power systems, applied in electronic devices. This constrained reinforcement learning-based intraday scheduling system for hydro-wind-solar-power systems may include: The scheduling model construction module is used to build an intraday optimization scheduling model for hydropower, thermal energy storage and DC power transmission, including the objective function and constraints. The transformation module is used to transform the intraday optimized scheduling model into a Markov decision process, defining the state, action, reward, and cost. A neural network building module is used to build neural networks, which include a policy network and a value network. An auxiliary training module is used to train the neural network using a deep reinforcement learning algorithm; The update module is used to transform the constrained optimization problem into a mini-maximum optimization problem using the Lagrange multiplier method, and alternately updates the policy network parameters and Lagrange multipliers; The decision-making module is used to acquire historical operating data and environmental conditions of the hydropower system, wind power system, and solar power system, and input the historical operating data and environmental conditions into a trained neural network to obtain a scheduling decision scheme.

[0080] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0081] In addition, this embodiment of the invention also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps in any embodiment of the constrained reinforcement learning-based intraday scheduling method for hydro-wind-solar-power systems provided in this embodiment of the invention. Therefore, it can achieve the beneficial effects that any constrained reinforcement learning-based intraday scheduling method for hydro-wind-solar-power systems provided in this embodiment of the invention can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0082] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the constrained reinforcement learning-based intraday scheduling method for hydro-wind-solar-power systems provided by the present invention.

[0083] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0084] Since the instructions stored in the storage medium can execute the steps in any embodiment of the intraday scheduling method for hydro-wind-solar-power systems based on constraint reinforcement learning provided in the embodiments of the present invention, the beneficial effects that any intraday scheduling method for hydro-wind-solar-power systems based on constraint reinforcement learning provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0085] The foregoing has provided a detailed description of a method, system, storage medium, and electronic device for intraday scheduling of hydropower, wind power, and solar power systems based on constrained reinforcement learning, as provided in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for day-ahead scheduling of a hydro-wind-solar power system based on constrained reinforcement learning, characterized in that, The method comprises: constructing an intra-day optimal scheduling model for water, fire, storage and DC transmission coordination, including an objective function and constraint conditions; transforming the intra-day optimal scheduling model into a Markov decision process, defining states, actions, rewards and costs; constructing a neural network, the neural network comprising a policy network and a value network; training the neural network through a deep reinforcement learning algorithm; using the Lagrange multiplier method to transform the constraint optimization problem into a minimum-maximum optimization problem, and alternately updating the policy network parameters and the Lagrange multiplier; obtaining historical operation data and environmental states of a water, wind, light and power system, and inputting the historical operation data and the environmental states into the trained neural network to obtain a scheduling decision scheme.

2. The method of claim 1, wherein the method is a method of intraday dispatching of a hydro-hydrokinetic power system based on constraint reinforcement learning. The objective function is: ； Wherein, Optimizing the number of time periods ， The adjustable resource set, water, fire, storage and direct current are selected as the control objects, And The control resource i In t The positive and negative adjustment amount of the day-ahead plan at the moment, The control resource i The unit adjustment cost of The constraint conditions include node injection power constraints, alternating current flow constraints, thermal power operation constraints, cascade hydropower operation constraints, energy storage operation constraints and DC operation constraints.

3. The method of claim 1, wherein the method is a method of intraday dispatching of a hydro-hydrokinetic power system based on constraint reinforcement learning. The policy network comprises a gated recurrent unit, which represents the dynamic correlation between historical operation states and future scheduling actions through hidden states, and the processing process of the gated recurrent unit is represented by the following formula: ； ； ； ； wherein, and denote the update gate and the reset gate, respectively, , denote t the candidate hidden state and the hidden state at time t, , , , , , denote the weight matrices and the bias vectors of the update gate, the reset gate, and the candidate hidden state, respectively, is an activation function.

4. The method of claim 1, wherein, The value network comprises a graph convolution network, which is used to transform environmental states into graph data and high-order representations of graph data; l The computational formula of the layer graph convolution network is shown below​ ； wherein, is the degree matrix of the graph, l is the degree matrix of the graph, is the adjacency matrix with self-loops, is the identity matrix, is the degree matrix of the graph, is the degree matrix of the graph, , is a non-linear activation function, is the degree matrix of the graph, l is the degree matrix of the graph.

5. The method of claim 1, wherein, The training process of the neural network comprises: a first stage: introducing an imitation learning mechanism to train the policy network: constructing a supervised learning loss based on historical state sequences and corresponding historical scheduling decisions, and training the policy network based on the supervised learning loss; a second stage: predicting rewards and costs, calculating the mean square error between the predicted values and the true values of the rewards and costs, and training the value network based on the mean square error.

6. The method of claim 1, wherein, The method further comprises: constructing an unsafe action adjustment module based on expert experience to correct the actions output by the policy network during the training process.

7. The method of claim 6, wherein the method is a method of intraday dispatching of a hydro-hydrokinetic power system based on constrained reinforcement learning. The correction of the actions output by the policy network comprises: obtaining the adjusted generation plan of the agent and the allowed operation interval of each controllable resource; determining whether there is a balance machine out-of-limit, and if there is, performing a secondary distribution of unbalanced power; determining whether there is a line out-of-limit, and if there is, adjusting the line out-of-limit based on the sensitivity.

8. A water, wind and solar power system intra-day dispatching system based on constraint reinforcement learning, characterized in that, It comprises: a scheduling model construction module for constructing an intra-day optimal scheduling model for water, fire, storage and DC transmission coordination, including an objective function and constraint conditions; a transformation module for transforming the intra-day optimal scheduling model into a Markov decision process, defining states, actions, rewards and costs; a neural network construction module for constructing a neural network, the neural network comprising a policy network and a value network; an auxiliary training module for training the neural network through a deep reinforcement learning algorithm; an update module for using the Lagrange multiplier method to transform the constraint optimization problem into a minimum-maximum optimization problem, and alternately updating the policy network parameters and the Lagrange multiplier; a decision module for obtaining historical operation data and environmental states of a water, wind, light and power system, and inputting the historical operation data and the environmental states into the trained neural network to obtain a scheduling decision scheme.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions adapted to be loaded by a processor to execute the method for daily scheduling of a water-wind-solar power system based on constraint reinforcement learning according to any one of claims 1-7.

10. An electronic device, comprising: A processor and a memory are included, the processor is electrically connected with the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in the method for daily scheduling of a water-wind-solar power system based on constraint reinforcement learning according to any one of claims 1-7.

Images