A low-carbon dispatching control method and device for a power system, a terminal device, and a storage medium

Abstract

The application discloses a low-carbon scheduling control method and device for a power system, a terminal equipment and a storage medium, and relates to the technical field of power system scheduling control. The method comprises the following steps: constructing a low-carbon economic scheduling model of the power system based on a Markov decision process according to information that new energy and a carbon trading mechanism jointly participate in the optimized scheduling of the power system. According to the model, a low-carbon economic scheduling strategy of the power system based on an inequality reduction gradient and cross-round information propagation is proposed. The trained constraint processing actor network is integrated into a power system scheduling device based on the proposed reinforcement learning method, and the scheduling scheme of the generator unit and the energy storage is made according to the real-time state information of the system. The application is based on the demand of the low-carbon economic scheduling of the power system, overcomes the defects of the general constraint reinforcement learning, such as the reduced learning ability and poor compatibility, and greatly improves the decision economy of the low-carbon scheduling control of the power system.

Description

Technical Field

[0001] The present invention relates to the field of power grid dispatching technology, and in particular to a low-carbon dispatching control method, apparatus, terminal equipment and storage medium for a power system. Background Art

[0002] Energy conservation and emission reduction in the power system are gradually transitioning from relying solely on technical means such as renewable energy generation to a combined "electricity-carbon" approach that comprehensively considers carbon trading market factors. Wind power generation technology is mature and large-scale. Rationally guiding wind power to participate in the real-time dispatch of the power system is an important means to achieve the low-carbon goals of the power system. Therefore, in the face of the random volatility of renewable energy, establishing and solving a low-carbon economic dispatch model for power systems containing wind power is of great significance for the construction of new power systems and the realization of "dual carbon" goals. However, there is currently no real-time and rapid control strategy for low-carbon dispatch of power systems under random fluctuations, resulting in the poor economic efficiency of existing low-carbon dispatch control of power systems.

[0003] Therefore, there is an urgent need for an efficient and safe low-carbon dispatching control strategy for power systems, so as to solve the problems of long solution time and poor decision-making economy in the existing low-carbon dispatching control of power systems under random fluctuations. Summary of the Invention

[0004] Embodiments of the present invention provide a low-carbon dispatch control method, apparatus, terminal device, and storage medium for a power system to solve the problem of low decision-making economy in the existing low-carbon dispatch control of power systems containing random fluctuations of renewable energy.

[0005] To solve the above problems, an embodiment of the present invention provides a low-carbon dispatch control method for a power system, comprising:

[0006] Based on the Markov decision process, a Markov decision process model of the low-carbon economic dispatch model of the power system is constructed;

[0007] According to the preset reinforcement learning method based on inequality-reduced gradient and cross-round information propagation and Markov decision process model, the actor network is trained offline, and the constraint-processing actor network is obtained after the offline training is completed;

[0008] Integrating the constraint processing actor network into a power system dispatching device so that the power system dispatching device outputs a dispatching plan for each power generation device and energy storage in the power system based on the constraint processing actor network and real-time status information of the power system;

[0009] According to the scheduling plan, the power generation equipment and energy storage equipment in the power system are scheduled.

[0010] As an improvement of the above solution, the low-carbon economic dispatch model includes: an objective function and constraints;

[0011] The objective function satisfies the following conditions:

[0012]

[0013] Where, F W Including the cost of wind turbine power generation and wind curtailment penalty costs C1 is the power generation cost coefficient of wind turbine (yuan / MW·h); P W,i,t is the power generation of the i-th wind turbine per unit time (MW); C2 is the wind abandonment penalty coefficient (yuan / MW·h); P S,i,t is the predicted power of the i-th wind turbine in unit time (MW); b j 、c j is the power generation cost coefficient of the jth thermal power unit; C3 is the energy storage operation cost coefficient (yuan / MW·h), P es,k,t is the kth energy storage charge and discharge power per unit time, and positive / negative values ​​represent charge / discharge power respectively;

[0014] The constraints satisfy the following conditions:

[0015]

[0016] U i,min ≤U i,t ≤U i,max

[0017] Soc k,min ≤Soc k,t ≤Soc k,max

[0018]

[0019] In the formula, the first set of constraints above are the unit ramping, energy storage charging and discharging power, wind power output and unit output constraints, r Gup,j and r Gdown,j are the up and down ramp rates of the thermal power unit, ΔT is the time interval, P charge,max and P discharge,max are the maximum charge / discharge power of energy storage, P G,i,max and P G,i,min are the upper and lower limits of the active power output of the unit, P W,i,max and P W,i,min are the upper and lower limits of the active power output of the wind turbine respectively; the second set of constraints is the system power balance constraint, P G,i,t 、P W,i,t 、P es,i,tare the active power injected into node i at time t by thermal power, wind power and energy storage respectively, Q G,i,t is the reactive power injected into node i by the thermal power unit at time t, P L,i,t and Q L,i,t are the active and reactive load demands of node i at time t, θ ij,t is the voltage difference between nodes i and j; B ij and G ij are the susceptance and conductance between nodes i and j, respectively, U i,t is the voltage of node i at time t; the other five constraints are the constraints related to voltage and energy storage Soc, U i,max and U i,min are the upper and lower limits of node i respectively; Soc k,max and Soc k,min are the upper and lower limits of the state of charge of energy storage k, Soc k,t is the state of charge of energy storage k at time t, E es,k,t and E es,k,max The actual power and upper limit capacity of energy storage k at time t, Soc k,t-1 is the state of charge at the previous moment, β es,k,t is the charge / discharge state, represented by a 0-1 variable, η charge,k,t and η discharge,k,t Represent the charging / discharging efficiency of energy storage, P charge,k,t and P discharge,k,t Respectively represent the charging / discharging power of energy storage.

[0020] As an improvement to the above solution, the Markov decision process model includes: an action space, a state space, and a reward; the Markov decision process model for constructing a low-carbon economic dispatch model for a power system based on the Markov decision process includes:

[0021] The output power and energy storage power of the low-carbon economic dispatch model are used as the action space; wherein the action space satisfies the following conditions:

[0022] A=[P G ,P es ]

[0023] Where, P G ,P es Respectively represent the output power and energy storage power of conventional thermal power units;

[0024] The load demand, node voltage, energy storage charge state and time of the low-carbon economic dispatch model are used as the state space, which is in the following form:

[0025] S=[P L ,V,Soc,t]

[0026] Where, P L ,V,Soc,t are load demand, node voltage, energy storage state of charge, and time t respectively.

[0027] Based on the objective function and constraints of the low-carbon economic dispatch model, a reward for the power system is obtained; the reward meets the following conditions:

[0028]

[0029] s t =[P L (t),V t ,Soc(t),t]

[0030] a t =[P G (t),P es (t)]

[0031] Where r t The agent is in state s at time t t Next, perform action a t The reward obtained after W 、F G 、F ESS 、 Respectively represent wind power cost, thermal power generation cost, energy storage operation cost, and carbon trading cost;

[0032] Based on the action space, the state space and the reward, a Markov decision process model of the power system is constructed.

[0033] As an improvement to the above solution, the actor network is trained offline according to the preset reinforcement learning method based on inequality-reduced gradient and cross-round information propagation and Markov decision process model, including:

[0034] In each episode of offline training, constraint processing operations are executed at a preset number of time steps based on the Markov decision process model and the current actor network. Constraint processing operations are executed once per time step, and the corrective actions, power system state information, and rewards generated by each constraint processing operation are stored in the experience replay pool.

[0035] In each episode of offline training, when the number of executed time steps reaches the number of update steps set in the episode, a small batch of trajectories with the minimum number of batches is randomly extracted from the experience replay pool; the state and action pairs in the small batch of trajectories are scored by the critic network to obtain a score value; based on the score value, a loss value calculation formula for the current actor network is constructed, and according to the loss value calculation formula, the actor network parameters of the current actor network are updated using gradient descent, and the next constraint processing operation is performed through the actor network containing the updated actor network parameters; the critic network parameters are updated using the cross-round information propagation update algorithm to obtain the updated critic network parameters, and the state and action pairs in the next small batch of trajectories are scored using the updated critic network parameters;

[0036] When the number of episodes of the offline training reaches a preset episode number threshold, the offline training of the actor network is completed, and a constraint-processed actor network is obtained.

[0037] As an improvement to the above solution, the constraint processing operation includes:

[0038] Obtain the current power system status information;

[0039] Inputting the power system state information into a neural network in an actor network so that the neural network outputs a basic action; wherein the basic action is output power;

[0040] Based on the basic action, the non-basic action is solved by the commercial solver and all the equality constraints in the low-carbon economic dispatch model, and the basic action and the non-basic action are combined into the initial action of the actor network; wherein the non-basic action is the energy storage power; a=(a B ,a N ) is the initial action of the constraint processing actor network, a B For basic actions, a N It is a non-basic action;

[0041] Based on the equality constraints and inequality constraints in the low-carbon economic dispatch model, the initial action is corrected by the inequality reduction gradient formula to obtain the corrective action; wherein the inequality reduction gradient satisfies the following conditions:

[0042]

[0043] In the formula, the simplified gradient formula of the inequality is the action of multiple iterations to correct the unsatisfied constraints. Therefore, and Represents the corrective action at the kth iteration update η α is the learning rate, Δa B Represents the difference corrected at each iteration, so that the action gradually satisfies the equality and inequality constraints corresponding to the constraints in the low-carbon economic dispatch model;

[0044] The corrective action and the power system status information are input into the Markov decision process model to obtain a reward, and the inequality reduced gradient algorithm is re-executed; wherein, after obtaining the reward, the corrective action, the power system status information and the reward are input into the experience replay pool.

[0045] As an improvement to the above solution, the loss value calculation formula of the current actor network is constructed based on the score value. According to the loss value calculation formula, the actor network parameters are updated using gradient descent on the current actor network, including:

[0046] Construct the loss calculation formula of the current actor network based on the scoring value;

[0047] According to the loss calculation formula and the mini-batch trajectory, the loss value of the actor network parameters is calculated using the gradient descent formula to maximize the negative value of the reward to complete the update of the actor network parameters; the gradient descent formula satisfies the following conditions:

[0048]

[0049] Where, κ j is the Lagrange multiplier, is the multiplier κ j The learning rate, is κ j The value of the kth iteration and Where k is the number of updates for neural network training, and each update Gradually increase.

[0050] As an improvement to the above solution, the cross-round information propagation update algorithm includes:

[0051] Based on the mini-batch trajectory, calculate the initial loss value of the critic network parameters according to the loss calculation method of the critic network;

[0052] The difference feature is extracted from the data "state-action pair" in the small batch trajectory, and then this difference is added to the initial loss value to form the final loss value of the critic network parameter. The critic network parameters are updated based on the final loss value of the critic network parameter; wherein the final loss value of the critic network parameter satisfies the following conditions:

[0053] d(s,a,s′,a′)=MLP(υ),υ=[υ1,υ2,...,V k ],

[0054]

[0055] Where d(s,a,s′,a′) is the difference between the state-action pairs (s,a) and (s′,a′), MLP(υ) is, υ is, MSE() is, Scoring allows the agent to more quickly identify the differences between states, speeding up the training of the algorithm.

[0056] Accordingly, an embodiment of the present invention further provides a low-carbon dispatch control device for a power system, comprising: a model building module, a network training module, a network integration module, and a dispatch module;

[0057] The model building module is used to build a Markov decision process model of a low-carbon economic dispatch model of the power system based on the Markov decision process;

[0058] The network training module is used to perform offline training on the actor network according to a preset reinforcement learning method based on inequality-reduced gradient and cross-round information propagation and a Markov decision process model, and obtain a constraint-processing actor network after the offline training is completed;

[0059] The network integration module is used to integrate the constraint processing actor network into the power system dispatching device, so that the power system dispatching device outputs the dispatching plan of each power generation equipment and energy storage in the power system based on the constraint processing actor network and the real-time status information of the power system;

[0060] The scheduling module is used to schedule the power generation equipment and energy storage equipment in the power system according to the scheduling plan.

[0061] Correspondingly, an embodiment of the present invention also provides a computer terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements a low-carbon dispatching control method for a power system as described in the present invention.

[0062] Correspondingly, an embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a low-carbon dispatch control method for a power system as described in the present invention.

[0063] As can be seen from the above, the present invention has the following beneficial effects:

[0064] The present invention provides a low-carbon dispatch control method for power systems, comprising: constructing a low-carbon economic dispatch model for power systems based on a Markov decision process based on information on the joint participation of new energy and carbon trading mechanisms in the optimal dispatch of power systems. According to the model, a low-carbon economic dispatch strategy for power systems based on reinforcement learning of inequality-reduced gradients and cross-round information propagation is proposed; based on the proposed reinforcement learning method, a trained constraint processing actor network is integrated into a power system dispatch device and put into use, and a dispatch plan for generator sets and energy storage is made based on the real-time status information of the system. Based on the needs of low-carbon economic dispatch of power systems, the present invention processes the general form of equality and inequality constraints of the dispatch problem through a constraint processing actor network and implements the policy network training of the method; and utilizes cross-round information propagation to update the critic network parameters to improve the learning ability of the intelligent agent. Based on the proposed method, the defects of reduced learning ability and poor compatibility of general constraint reinforcement learning are overcome, and the economy of low-carbon dispatch control decisions of power systems is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 1 is a flow chart of a low-carbon dispatch control method for a power system provided by an embodiment of the present invention;

[0066] Figure 2 1 is a schematic structural diagram of a low-carbon dispatch control device for a power system provided by an embodiment of the present invention;

[0067] Figure 3 This is a schematic diagram of the structure of a terminal device provided by one embodiment of the present invention;

[0068] Figure 4 This is a schematic diagram of the specific structure of the constraint processing actor network provided by one embodiment of the present invention;

[0069] Figure 5 is a flowchart of a method for updating critic network parameters provided by one embodiment of the present invention;

[0070] Figure 6 1 is a flow chart of a reinforcement learning algorithm provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0071] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0072] Example 1

[0073] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a low-carbon dispatch control method for a power system provided by an embodiment of the present invention, such as Figure 1 As shown, this embodiment includes steps 101 to 102, and each step is specifically as follows:

[0074] Step 101: Based on the Markov decision process, a Markov decision process model of a low-carbon economic dispatch model of the power system is constructed.

[0075] In this embodiment, the operating status of the power system at each moment is obtained.

[0076] In this embodiment, the low-carbon economic dispatch model includes: an objective function and constraints;

[0077] The objective function satisfies the following conditions:

[0078]

[0079] Where, F W Including the cost of wind turbine power generation and wind curtailment penalty costs C1 is the power generation cost coefficient of wind turbine (yuan / MW·h); P W,i,t is the power generation of the i-th wind turbine per unit time (MW); C2 is the wind abandonment penalty coefficient (yuan / MW·h); P S,i,t is the predicted power of the i-th wind turbine in unit time (MW); b j 、c j is the power generation cost coefficient of the jth thermal power unit; C3 is the energy storage operation cost coefficient (yuan / MW·h), P es,k,t is the kth energy storage charge and discharge power per unit time, and positive / negative values ​​represent charge / discharge power respectively;

[0080] The constraints satisfy the following conditions:

[0081]

[0082] U i,min ≤U i,t≤U i,max

[0083] Soc k,min ≤Soc k,t ≤Soc k,max

[0084]

[0085] In the formula, the first set of constraints above are the unit ramping, energy storage charging and discharging power, wind power output and unit output constraints, r Gup,j and r Gdown,j are the up and down ramp rates of the thermal power unit, ΔT is the time interval, P charge,max and P discharge,max are the maximum charge / discharge power of energy storage, P G,i,max and P G,i,min are the upper and lower limits of the active power output of the unit, P W,i,max and P W,i,min are the upper and lower limits of the active power output of the wind turbine respectively; the second set of constraints is the system power balance constraint, P G,i,t 、P W,i,t 、P es,i,t are the active power injected into node i at time t by thermal power, wind power and energy storage respectively, Q G,i,t is the reactive power injected into node i by the thermal power unit at time t, P L,i,t and Q L,i,t are the active and reactive load demands of node i at time t, θ ij,t is the voltage difference between nodes i and j; B ij and G ij are the susceptance and conductance between nodes i and j, respectively, U i,t is the voltage of node i at time t; the other five constraints are the constraints related to voltage and energy storage Soc, U i,max and U i,min are the upper and lower limits of node i respectively; Soc k,max and Soc k,min are the upper and lower limits of the state of charge of energy storage k, Soc k,t is the state of charge of energy storage k at time t, E es,k,t and E es,k,max The actual power and upper limit capacity of energy storage k at time t, Soc k,t-1 is the state of charge at the previous moment, β es,k,t is the charge / discharge state, represented by a 0-1 variable, η charge,k,t and η discharge,k,t Represent the charging / discharging efficiency of energy storage, P charge,k,t and P discharge,k,t Respectively represent the charging / discharging power of energy storage.

[0086] In a specific embodiment, the low-carbon economic dispatch model includes:

[0087] ①Objective function

[0088] 1) Carbon trading model

[0089] The carbon quota is calculated based on the power generation of the system units, and the carbon emissions are calculated based on the power generation of conventional thermal power units. The system carbon transaction cost is calculated by combining the carbon quota and carbon emissions. The system carbon quota is proportional to the power generation of the units. The carbon quota E d Expressed as:

[0090]

[0091] Where P D,t is the sum of the output of conventional units and wind power in unit time t, specifically expressed as M and N are the number of wind power and conventional units respectively; η is the carbon emission quota per unit of electricity, t / (MW·h); P W,i,t and P G,j,t Represents the output power of different wind power and conventional thermal power units at time t. Since the carbon emissions generated by wind power operation are much smaller than those of conventional thermal power units (negligible), carbon emissions are only counted for thermal power units. Carbon emissions of thermal power units E p Expressed as:

[0092]

[0093] Where T is the scheduling period, P b =P G,j,t / P G,j,max is the unit coal consumption rate, μ j =(y j P b +z j ), the coal consumption coefficient of the jth thermal power unit is y j and z j ;μ j ρP G,j,t is the carbon emissions of the jth unit in unit time t; ρ is the carbon conversion coefficient, kg / (MW·h). Finally, the carbon trading cost is expressed as:

[0094]

[0095] In the formula is the carbon trading cost, is the carbon trading price (yuan / t).

[0096] 2) Low-carbon economic dispatch model for power systems including wind power

[0097] The objective function of the low-carbon economic dispatch model of the power system containing wind power can be expressed as:

[0098]

[0099] F in the formula W 、F G 、F ESS 、 Respectively represent wind power cost, thermal power generation cost, energy storage operation cost, and carbon trading cost. W Including the cost of wind turbine power generation and wind curtailment penalty costs The specific expression is as follows:

[0100]

[0101] C1 in formula (6) is the power generation cost coefficient of wind turbine (yuan / MW·h); P W,i,t is the power generation of the i-th wind turbine per unit time (MW); C2 is the wind abandonment penalty coefficient (yuan / MW·h); P S,i,t is the predicted power of the i-th wind turbine in unit time (MW). The power generation cost of conventional thermal power units F G As shown below:

[0102]

[0103] Among them, b j 、c j is the power generation cost coefficient of the jth thermal power unit. In order to reduce the impact of wind power fluctuations on the power grid and to ensure the power transmission balance of the power grid to the greatest extent, a certain amount of energy storage is configured in the system. The operation and maintenance cost of the energy storage is F ESS The expression is as follows:

[0104]

[0105] Where C3 is the energy storage operation cost coefficient (yuan / MW·h), P es,k,t is the kth energy storage charge and discharge power per unit time, and positive / negative values ​​represent the charge / discharge power, respectively.

[0106] In this embodiment, the Markov decision process model includes: an action space, a state space, and a reward; the Markov decision process model for constructing a low-carbon economic dispatch model for a power system based on the Markov decision process includes:

[0107] The output power and energy storage power of the low-carbon economic dispatch model are used as the action space; wherein the action space satisfies the following conditions:

[0108] A=[P G,P es ]

[0109] Where, P G ,P es Respectively represent the output power and energy storage power of conventional thermal power units;

[0110] The load demand, node voltage, energy storage charge state and time of the low-carbon economic dispatch model are used as the state space, which is in the following form:

[0111] S=[P L ,V,Soc,t]

[0112] Where, P L ,V,Soc,t are load demand, node voltage, energy storage state of charge, and time t respectively.

[0113] Based on the objective function and constraints of the low-carbon economic dispatch model, a reward for the power system is obtained; the reward meets the following conditions:

[0114]

[0115] s t =[P L (t),V t ,Soc(t),t]

[0116] a t =[P G (t),P es (t)]

[0117] Where r t The agent is in state s at time t t Next, perform action a t The reward obtained after W 、F G 、F ESS 、 Respectively represent wind power cost, thermal power generation cost, energy storage operation cost, and carbon trading cost;

[0118] Based on the action space, the state space and the reward, a Markov decision process model of the power system is constructed.

[0119] In a specific embodiment, the low-carbon economic dispatch Markov decision process model includes:

[0120] Based on the mathematical representation of the low-carbon economic dispatch model for wind power systems, a Markov decision process is used here to describe a series of state transitions, primarily consisting of a tuple (S, A, R, P, γ). S represents the state space of the agent, A represents the action space taken by the agent, R represents the reward, P is the state transition function, and γ is the reward discount factor. After converting the low-carbon economic dispatch model for wind power systems into a Markov decision process and determining the state space, action space, and reward function, the reinforcement learning method proposed later can be used to interactively train the agent with the model environment of the low-carbon economic dispatch problem for wind power systems to solve the problem.

[0121] Step 102: Perform offline training on the actor network according to the preset reinforcement learning method based on inequality-reduced gradient and cross-round information propagation and the Markov decision process model, and obtain the constraint-processing actor network after the offline training is completed.

[0122] In this embodiment, the actor network is trained offline according to the preset reinforcement learning method based on inequality-reduced gradient and cross-round information propagation and Markov decision process model, including:

[0123] In each episode of offline training, constraint processing operations are executed at a preset number of time steps based on the Markov decision process model and the current actor network. Constraint processing operations are executed once per time step, and the corrective actions, power system state information, and rewards generated by each constraint processing operation are stored in the experience replay pool.

[0124] In each episode of offline training, when the number of executed time steps reaches the number of update steps set in the episode, a small batch of trajectories with the minimum number of batches is randomly extracted from the experience replay pool; the state and action pairs in the small batch of trajectories are scored by the critic network to obtain a score value; based on the score value, a loss value calculation formula for the current actor network is constructed, and according to the loss value calculation formula, the actor network parameters of the current actor network are updated using gradient descent, and the next constraint processing operation is performed through the actor network containing the updated actor network parameters; the critic network parameters are updated using the cross-round information propagation update algorithm to obtain the updated critic network parameters, and the state and action pairs in the next small batch of trajectories are scored using the updated critic network parameters;

[0125] When the number of episodes of the offline training reaches a preset episode number threshold, the offline training of the actor network is completed, and a constraint-processed actor network is obtained.

[0126] In this embodiment, the constraint processing operation includes:

[0127] Obtain the current power system status information;

[0128] Inputting the power system state information into a neural network in an actor network so that the neural network outputs a basic action; wherein the basic action is output power;

[0129] Based on the basic action, the non-basic action is solved by the commercial solver and all the equality constraints in the low-carbon economic dispatch model, and the basic action and the non-basic action are combined into the initial action of the actor network; wherein the non-basic action is the energy storage power; a=(a B ,a N ) is the initial action of the constraint processing actor network, a B For basic actions, a N It is a non-basic action;

[0130] Based on the equality constraints and inequality constraints in the low-carbon economic dispatch model, the initial action is corrected by the inequality reduction gradient formula to obtain the corrective action; wherein the inequality reduction gradient satisfies the following conditions:

[0131]

[0132] In the formula, the simplified gradient formula of the inequality is the action of multiple iterations to correct the unsatisfied constraints. Therefore, and Represents the corrective action at the kth iteration update η α is the learning rate, Δa B Represents the difference corrected at each iteration, so that the action gradually satisfies the equality and inequality constraints corresponding to the constraints in the low-carbon economic dispatch model;

[0133] The corrective action and the power system status information are input into the Markov decision process model to obtain a reward, and the inequality reduced gradient algorithm is re-executed; wherein, after obtaining the reward, the corrective action, the power system status information and the reward are input into the experience replay pool.

[0134] In this embodiment, the loss value calculation formula of the current actor network is constructed based on the score value, and the actor network parameters are updated using gradient descent on the current actor network according to the loss value calculation formula, including:

[0135] Construct the loss calculation formula of the current actor network based on the scoring value;

[0136] According to the loss calculation formula and the mini-batch trajectory, the loss value of the actor network parameters is calculated using the gradient descent formula to maximize the negative value of the reward to complete the update of the actor network parameters; the gradient descent formula satisfies the following conditions:

[0137]

[0138] Where, κ j is the Lagrange multiplier, is the multiplier κ j The learning rate, is κ j The value of the kth iteration and Where k is the number of updates for neural network training, and each update Gradually increase.

[0139] In this embodiment, the cross-round information propagation update algorithm includes:

[0140] Based on the mini-batch trajectory, calculate the initial loss value of the critic network parameters according to the loss calculation method of the critic network;

[0141] The difference feature is extracted from the data "state-action pair" in the small batch trajectory, and then this difference is added to the initial loss value to form the final loss value of the critic network parameter. The critic network parameters are updated based on the final loss value of the critic network parameter; wherein the final loss value of the critic network parameter satisfies the following conditions:

[0142] d(s,a,s′,a′)=MLP(υ),υ=[υ1,v2,...,v k ],

[0143]

[0144] Where d(s,a,s′,a′) is the difference between the state-action pairs (s,a) and (s′,a′), MLP(v) is, v is, MSE() is, Scoring allows the agent to more quickly identify the differences between states, speeding up the training of the algorithm.

[0145] In a specific embodiment, the constraints of the low-carbon economic dispatch model include:

[0146]

[0147] Among them, t represents the time period, g j and fi Respectively represent only the current state s t and action a t There are m equality constraints in total. For the proposed algorithm, π θ represents the policy network (equivalent to the actor network in the Actor-Critic learning framework), the actor network in period t is based on the state s t The output action is a t =π θ (s t ), J R (π θ ) is the strategy π θ The objective function value under the condition of J is maximized. R (π θ ). In this case, equations (20)-(22) are supplemented with equation (21) relative to the general problem form processed by the reduced gradient algorithm. Therefore, based on the reduced gradient method, the present invention proposes an actor network (herein referred to as a constraint processing actor network) that combines inequality reduced gradient with a neural network.

[0148] The specific structure of the constraint processing actor network is as follows Figure 4 First, refer to the basic simplified gradient algorithm and change the action Divided into basic movements and non-basic actions Then, a neural network module outputs the initial basic action a B , and obtain the initial value a of the non-basic action based on the solver N The initial action without correction is However, based only on the initial action, it is impossible to guarantee that the equality and inequality constraints are fully satisfied at the same time and the decision-making economy is not possible. Finally, the proposed inequality reduction gradient algorithm is used to update the action within the region defined by the inequality constraint expressed in equation (21) while ensuring the satisfaction of equation (22). After k iterations of update and correction, the final feasible action a is obtained. k .

[0149] Detailed process of inequality simplified gradient update action a: Here, all constraints in equation (22) are defined as F(a;s) = 0, and a N =v N (a N ), Referring to the ordinary simplified gradient method, we can get N to a B The gradient of

[0150]

[0151] The basic simplified gradient algorithm does not consider the constraints described in formula (21). In order to ensure that the output of the constraint processing actor satisfies all constraints, it is proposed to limit the update of the equality constraint (22) to the area defined by the inequality constraint (21). The specific update is shown in formulas (24)-(28). Here, the sum of the maximum violations of inequality (21) is G(a B ,a N ), Similar to the basic reduced gradient algorithm, for a B and a N Update so that the action finally satisfies all constraints, specifically a k Updates as follows:

[0152]

[0153] η in formulas (27) and (28) α Equivalent to the learning rate when the basic minimalist gradient algorithm is updated, it is used here to control the update rate. and Indicates the action at the kth update According to equations (24)-(28), the sum and number of inequality violations during the iteration process will continue to decrease, and the final output will fully satisfy the equality and inequality constraints. The entire process from input state s to output a is defined as the complete constraint processing actor network (that is, the final policy network).

[0154] In a specific embodiment, when using TD update, state transition information can only be propagated to the previous state in the same round, which limits the data efficiency of the reinforcement learning algorithm. Figure 5 As shown, the present invention proposes a method for propagating fragment information across rounds, which mutually propagates state-action pairs in different rounds to improve the learning ability of the intelligent agent.

[0155] Figure 5 (a) is a schematic diagram of the cross-round propagation of state-action pairs. represents the trajectory of the kth round of interaction between the agent and the environment, and are the state-action pairs and their corresponding Q-values ​​for the k-th round trajectory at the nth period. When the algorithm feeds each batch of data into training, the cross-round information propagation algorithm passes the data in the batch through the similarity calculation network and the difference calculation network to obtain the difference d between the corresponding state-action pairs (s′, a′) and (s, a) in the batch, and uses this difference d to update the critic network. Figure 5 (b) is the specific process of cross-round information propagation. The specific steps of cross-round information propagation are as follows:

[0156] Similarity calculation: First, based on any given two state-action pairs (s′, a′) and (s, a), the multi-head encoder extracts features q and q′, and then uses the similarity function to calculate the similarity between features q and q′ (the similarity function here uses cosine similarity). The specific expression is as follows:

[0157]

[0158] In formula (29), i∈1,2,...,k, where i is the head number of the multi-head encoder.

[0159] Difference calculation: similarity v is obtained after feature extraction i After obtaining the value, a multi-layer perceptron (MLP) is used to infer the difference between (s′, a′) and (s, a), and finally the difference d and Q(s, a) are used to infer the value of Q(s′, a′), as shown below:

[0160] d(s,a,s′,a′)=MLP(υ),υ=[υ1,υ2,...,υ k ](30)

[0161] Q(s′,a′)←Q(s,a)+d(s,a,s′,a′)(31)

[0162] Finally, by combining the Q-value update method of the critic network with the constraint processing actor network through the cross-round information propagation method, this paper proposes a new constrained reinforcement learning method based on the soft actor-critic (SAC) framework [6]. In order to realize the training of the algorithm under the SAC framework, an additional Lagrange multiplier and a penalty term are introduced to transform the problem represented by Equations (20)-(22) into an unconstrained problem. The transformed expression of the problem is as follows:

[0163]

[0164] where κ j is the Lagrange multiplier, which is expressed as follows:

[0165]

[0166] is the multiplier κ j The learning rate, is κ j The value of the kth iteration and because It is monotonically increasing during training, and there must be Therefore, equation (32) is equivalent to the problem expressed by equations (20)-(22).

[0167] To better illustrate the reinforcement learning method based on inequality-reduced gradients and cross-episode information propagation, see the following steps:

[0168] Step 1: First, according to the reinforcement learning algorithm based on inequality simplified gradient and cross-round information propagation proposed in the present invention, the action A in the Markov decision process of the low-carbon economic dispatch model of the wind power system is converted into [P G ,P es ] is equivalent to the basic action a of the proposed algorithm B and non-basic actions a N Combination a (set P G =a B , P es =a N ,[P G ,P es ]=a).

[0169] Step 2: Based on Figure 6 The schematic diagram of the actor shown in the figure shows the s at each moment. t Input the actor network and let the actor network output an initial algorithm a B At this time, with a B is a known quantity. Based on the equality constraints in the low-carbon economic dispatch model of the power system containing wind power, the unknown action a is obtained by the solver. N , so we get the state s t The complete initial action a at this time cannot satisfy the inequality constraint. Therefore, we use the inequality reduced gradient method shown in formulas (23)-(28) to construct G(a B ,a N ), for a B and a N Perform iterative correction to take the inequality constraints into account. After k iterations, we get Make a k =[P G ,P es ] satisfies all the constraints of the optimization problem.

[0170] Step 3: Based on a k Interact with the environment of the low-carbon economic dispatch model of the wind power system to obtain the reward (that is, the cost value of the objective function).

[0171] Step 4: The algorithm is cyclical based on steps 1 to 3. Through the continuous interaction between the intelligent agent and the low-carbon economic dispatch model environment of the wind power system, the actor and critic networks are updated with the goal of minimizing formula (32) to achieve the optimal strategy network π θ training.

[0172] Step 5: When the policy network π is completed θ After training, when making decisions, only the state s at each moment needs to be input into the actor network t , the actor network can output the optimal action a, so that we can only know the state s at each moment in a random environment t , we can quickly get the optimal solution to the problem [P G ,P es ].

[0173] To better illustrate the reinforcement learning algorithm, based on the basic SAC framework, the actor and critic network parameter update process of the new constrained reinforcement learning algorithm proposed in this embodiment under the SAC framework is shown in Table 1 below:

[0174]

[0175]

[0176] It should be noted that ③ to ⑦ in Table 1 are specific steps for updating the input parameters according to the present invention.

[0177] Step 103: Integrate the constraint processing actor network into the power system dispatching device, so that the power system dispatching device outputs a dispatching plan for each power generation equipment and energy storage in the power system based on the constraint processing actor network and the real-time status information of the power system.

[0178] Step 104: Scheduling the power generation equipment and energy storage equipment in the power system according to the scheduling plan.

[0179] See also Figure 2 , Figure 2 2 is a schematic structural diagram of a low-carbon dispatch control device for a power system provided by an embodiment of the present invention, comprising: a model building module 201, a network training module 202, a network integration module 203 and a dispatch module 204;

[0180] The model building module is used to build a Markov decision process model of a low-carbon economic dispatch model of the power system based on the Markov decision process;

[0181] The network training module is used to perform offline training on the actor network according to a preset reinforcement learning method based on inequality-reduced gradient and cross-round information propagation and a Markov decision process model, and obtain a constraint-processing actor network after the offline training is completed;

[0182] The network integration module is used to integrate the constraint processing actor network into the power system dispatching device, so that the power system dispatching device outputs the dispatching plan of each power generation equipment and energy storage in the power system based on the constraint processing actor network and the real-time status information of the power system;

[0183] The scheduling module is used to schedule the power generation equipment and energy storage equipment in the power system according to the scheduling plan.

[0184] It can be understood that the above-mentioned system embodiment corresponds to the method embodiment of the present invention, which can implement the low-carbon dispatching control method for the power system provided by any of the above-mentioned method embodiments of the present invention.

[0185] This embodiment, through an actor network based on an inequality-reduced gradient algorithm for action correction, overcomes the current problem of constrained reinforcement learning, which lacks the ability to handle equality constraints and the versatility of its constraint processing methods. This significantly improves the versatility of the method while satisfying various power system constraints. For the first time, this approach improves the parameter update process from the perspective of cross-round information propagation and applies it to the proposed actor network based on an inequality-reduced gradient algorithm for action correction, forming a new constrained reinforcement learning algorithm that significantly enhances the algorithm's learning capabilities.

[0186] Example 2

[0187] See also Figure 3 , Figure 3 It is a schematic diagram of the terminal device structure provided by one embodiment of the present invention.

[0188] A terminal device of this embodiment includes: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the steps of the above-mentioned low-carbon dispatch control method for a power system in the embodiment are implemented, for example: Figure 1 Alternatively, when the processor executes the computer program, the functions of the modules in the above-mentioned device embodiments are realized, for example: Figure 2 All modules of the low-carbon dispatch control device for the power system are shown.

[0189] In addition, an embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the low-carbon dispatch control method for the power system as described in any of the above embodiments.

[0190] Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0191] The processor 301 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor 301 is the control center of the terminal device, connecting various parts of the entire terminal device using various interfaces and lines.

[0192] The memory 302 can be used to store the computer programs and / or modules. The processor 301 implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0193] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0194] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A low-carbon dispatch control method for a power system, characterized in that: include: Based on the Markov decision process, a Markov decision process model of the low-carbon economic dispatch model of the power system is constructed; According to the preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and Markov decision process model, the actor network is trained offline, and a constraint processing actor network is obtained after the offline training is completed; wherein, according to the preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and Markov decision process model, the actor network is trained offline, including: in each episode of offline training, based on the Markov decision process model and the current actor network, the constraint processing operation is executed according to the preset number of time steps; wherein, the constraint processing operation is executed once per time step, and the corrective action, power system status information and reward generated by each execution of the constraint processing operation are stored in the experience replay pool; in each episode of offline training, when the number of executed time steps reaches the update step number set for the episode, the constraint processing operation is executed in the experience replay pool. Randomly extract a small batch of trajectories with the minimum number of batches; score the state and action pairs in the small batch trajectories through the critic network to obtain a score value; construct a loss value calculation formula for the current actor network based on the score value, update the actor network parameters of the current actor network using gradient descent according to the loss value calculation formula, and perform the next constraint processing operation through the actor network containing the updated actor network parameters; update the critic network parameters using a cross-round information propagation update algorithm to obtain updated critic network parameters, and score the state and action pairs in the next small batch trajectories through the actor network containing the updated critic network parameters; when the number of episodes of the offline training reaches a preset episode number threshold, complete the offline training of the actor network and obtain the constraint processing actor network; Integrating the constraint processing actor network into a power system dispatching device so that the power system dispatching device outputs a dispatching plan for each power generation device and energy storage in the power system based on the constraint processing actor network and real-time status information of the power system; According to the scheduling plan, the power generation equipment and energy storage equipment in the power system are scheduled.

2. The low-carbon dispatch control method for a power system according to claim 1, characterized in that: The low-carbon economic dispatch model includes: an objective function and constraints; The objective function satisfies the following conditions: Where, F W Including the cost of wind turbine power generation and wind curtailment penalty costs C1 is the power generation cost coefficient of wind turbine (yuan / MW·h); P W,i,t is the power generation of the i-th wind turbine per unit time (MW); C2 is the wind abandonment penalty coefficient (yuan / MW·h); P S,i,t is the predicted power of the i-th wind turbine in unit time (MW); b j 、c j is the power generation cost coefficient of the jth thermal power unit; C3 is the energy storage operation cost coefficient (yuan / MW·h), P es,k,t is the kth energy storage charge and discharge power per unit time, and positive / negative values ​​represent charge / discharge power respectively; F G F is the power generation cost of thermal power units; ESS is the energy storage operation cost; M and N are the number of wind turbines and conventional turbines respectively; T is the scheduling period; The constraints satisfy the following conditions: IN i,min ≤U i,t ≤U i,max Soc k,min ≤Soc k,t ≤Soc k,max In the formula, the first set of constraints above are the unit ramping, energy storage charging and discharging power, wind power output and unit output constraints, r Gup,j and r Gdown,j are the up and down ramp rates of the thermal power unit, ΔT is the time interval, P charge,max and P discharge,max are the maximum charge / discharge power of energy storage, P G,i,max and P G,i,min are the upper and lower limits of active power output of thermal power units, P W,i,max and P W,i,min are the upper and lower limits of the active power output of the wind turbine respectively; the second set of constraints is the system power balance constraint, P G,i,t 、P W,i,t 、P es,i,t are the active power injected into node i at time t by thermal power, wind power and energy storage respectively, Q G,i,t is the reactive power injected into node i by the thermal power unit at time t, P L,i,t and Q L,i,t are the active and reactive load demands of node i at time t, θ ij,t is the voltage difference between nodes i and j; B ij and G ij are the susceptance and conductance between nodes i and j, respectively, U i,t is the voltage of node i at time t; the other five constraints are the constraints related to voltage and energy storage Soc, U i,max and U i,min are the upper and lower limits of node i respectively; Soc k,max and Soc k,min are the upper and lower limits of the state of charge of energy storage k, Soc k,t is the state of charge of energy storage k at time t, E es,k,t and E es,k,max The actual power and upper limit capacity of energy storage k at time t, Soc k,t-1 is the state of charge at the previous moment, β es,k,t is the charge / discharge state, represented by a 0-1 variable, η charge,k,t and η discharge,k,t Represent the charging / discharging efficiency of energy storage, P charge,k,t and P discharge,k,t Respectively represent the charging / discharging power of energy storage; P G,j,t is the output power of conventional thermal power unit at time t; P G,j,t-1 is the output power of the conventional thermal power unit at time t-1.

3. The low-carbon dispatch control method for a power system according to claim 2, characterized in that: The Markov decision process model includes: an action space, a state space, and a reward; the Markov decision process model for constructing a low-carbon economic dispatch model for a power system based on the Markov decision process includes: The output power and energy storage power of the low-carbon economic dispatch model are used as the action space; wherein the action space satisfies the following conditions: A=[P G ,P es ] Where, P G ,P es Respectively represent the output power and energy storage power of conventional thermal power units; The load demand, node voltage, energy storage charge state and time of the low-carbon economic dispatch model are used as the state space, which is in the following form: S=[P L ,V,Soc,t] Where, P L ,V,Soc,t are load demand, node voltage, energy storage charge state, and time respectively; Based on the objective function and constraints of the low-carbon economic dispatch model, a reward for the power system is obtained; the reward meets the following conditions: s t =[P L (t),V t ,Soc(t),t] a t =[P G (t),P es (t)] Where r t The agent is in state s at time t t Next, perform action a t The reward obtained after W 、F G 、F ESS 、 Respectively represent wind power cost, thermal power generation cost, energy storage operation cost, and carbon trading cost; Based on the action space, the state space and the reward, a Markov decision process model of the power system is constructed.

4. The low-carbon dispatch control method for power system according to claim 3, characterized in that: The constraint processing operation includes: Obtain the current power system status information; Inputting the power system state information into a neural network in an actor network so that the neural network outputs a basic action; wherein the basic action is output power; Based on the basic action, the non-basic action is solved by the commercial solver and all the equality constraints in the low-carbon economic dispatch model, and the basic action and the non-basic action are combined into the initial action of the actor network; wherein the non-basic action is the energy storage power; a=(a B ,a N ) is the initial action of the constraint processing actor network, a B For basic actions, a N It is a non-basic action; Based on the equality constraints and inequality constraints in the low-carbon economic dispatch model, the initial action is corrected by the inequality reduction gradient formula to obtain the corrective action; wherein the inequality reduction gradient satisfies the following conditions: In the formula, the simplified gradient formula of the inequality is the action of multiple iterations to correct the unsatisfied constraints. Therefore, and Represents the corrective action at the kth iteration update η α is the learning rate, Δa B Represents the difference corrected at each iteration, so that the action gradually satisfies the equality and inequality constraints corresponding to the constraints in the low-carbon economic dispatch model; The corrective action and the power system status information are input into the Markov decision process model to obtain a reward, and the inequality reduced gradient algorithm is re-executed; wherein, after obtaining the reward, the corrective action, the power system status information and the reward are input into the experience replay pool.

5. The low-carbon dispatch control method for power system according to claim 3, characterized in that: The loss value calculation formula of the current actor network is constructed based on the score value, and the actor network parameters are updated using gradient descent on the current actor network according to the loss value calculation formula, including: Construct the loss calculation formula of the current actor network based on the scoring value; According to the loss calculation formula and the mini-batch trajectory, the loss value of the actor network parameters is calculated using the gradient descent formula to maximize the negative value of the reward to complete the update of the actor network parameters; the gradient descent formula satisfies the following conditions: Where, κ j is the Lagrange multiplier, is the multiplier κ j The learning rate, is κ j The value of the kth iteration and Where k is the number of updates for neural network training, and each update Gradually increase.

6. The low-carbon dispatch control method for power system according to claim 3, characterized in that: The cross-round information propagation update algorithm includes: Based on the mini-batch trajectory, calculate the initial loss value of the critic network parameters according to the loss calculation method of the critic network; The difference feature is extracted from the data "state-action pair" in the small batch trajectory, and then the difference is added to the initial loss value to form the final loss value of the critic network parameter. The critic network parameters are updated based on the final loss value of the critic network parameter; wherein the final loss value of the critic network parameter satisfies the following conditions: d(s,a,s′,a′)=MLP(v),υ=[υ1,υ2,...,υ k ], Where d(s,a,s′,a′) is the difference between the state-action pairs (s,a) and (s′,a′), Scoring allows the agent to more quickly identify the differences between states, speeding up the training of the algorithm.

7. A low-carbon dispatch control device for a power system, characterized in that: include: Model building module, network training module, network integration module and scheduling module; The model building module is used to build a Markov decision process model of a low-carbon economic dispatch model of the power system based on the Markov decision process; The network training module is used to perform offline training on the actor network according to the preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and Markov decision process model, and obtain the constraint processing actor network after the offline training is completed; wherein, the offline training of the actor network according to the preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and Markov decision process model includes: in each episode of offline training, based on the Markov decision process model and the current actor network, executing the constraint processing operation according to the preset time steps; wherein, the constraint processing operation is executed once per time step, and the corrective action, power system status information and reward generated by each execution of the constraint processing operation are stored in the experience replay pool; in each episode of offline training, when the number of executed time steps reaches the update step number set by the episode, Randomly extract a small batch of trajectories with the minimum batch size from the experience replay pool; score the state and action pairs in the small batch trajectories through the critic network to obtain a score value; construct a loss value calculation formula for the current actor network based on the score value, and update the actor network parameters of the current actor network using gradient descent according to the loss value calculation formula, and perform the next constraint processing operation through the actor network containing the updated actor network parameters; use the cross-round information propagation update algorithm to update the critic network parameters to obtain the updated critic network parameters, and score the state and action pairs in the next small batch trajectories using the updated critic network parameters; when the number of offline training episodes reaches a preset episode number threshold, complete the offline training of the actor network and obtain the constraint processing actor network; The network integration module is used to integrate the constraint processing actor network into the power system dispatching device, so that the power system dispatching device outputs the dispatching plan of each power generation equipment and energy storage in the power system based on the constraint processing actor network and the real-time status information of the power system; The scheduling module is used to schedule the power generation equipment and energy storage equipment in the power system according to the scheduling plan.

8. A computer terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, a low-carbon dispatch control method for a power system as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a low-carbon dispatch control method for a power system according to any one of claims 1 to 6.

Images