Low-carbon dispatching control method and device for power system, terminal equipment and storage medium

Abstract

The invention discloses a low-carbon dispatching control method and device for an electric power system, terminal equipment and a storage medium, and the method comprises the steps: constructing an electric power system low-carbon economic dispatching model based on a Markov decision process according to the information of the joint participation of new energy and a carbon transaction mechanism in the optimal dispatching of the electric power system. According to the model, providing a reinforcement learning power system low-carbon economic dispatching strategy based on inequality simple gradient and cross-turn information propagation; based on the provided reinforcement learning method, the trained constraint processing actor network is integrated into a power system dispatching device to be put into application, and a generator set and energy storage dispatching scheme is made according to system real-time state information. Based on the demand of low-carbon economic dispatching of the power system, the method overcomes the defects that the learning capability of general constraint reinforcement learning is reduced and the compatibility is poor, and also greatly improves the decision economy of low-carbon dispatching control of the power system.

Description

Technical Field

[0001] The present invention relates to the field of power grid dispatching technology, and in particular to a low-carbon dispatching control method, device, terminal equipment and storage medium for a power system. Background Art

[0002] Energy conservation and emission reduction in the power system is gradually transitioning from relying solely on new energy power generation and other technical means in the past to a "electricity-carbon" joint means that comprehensively considers carbon trading market factors. Wind power generation technology is mature and large-scale. Reasonable guidance of wind power to participate in the real-time dispatch of the power system is an important means to achieve the low-carbon goal of the power system. Therefore, in the face of the random volatility of renewable energy, establishing and solving a low-carbon economic dispatch model for a power system containing wind power is of great significance for the construction of a new power system and the realization of the "dual carbon" goal. However, there is currently no real-time and rapid control strategy for low-carbon dispatch of power systems under random fluctuations, resulting in the problem of poor economic efficiency of the existing low-carbon dispatch control of the power system.

[0003] Therefore, there is an urgent need for an efficient and safe low-carbon dispatching control strategy for power systems, so as to solve the problems of long solution time and poor decision-making economy in the existing low-carbon dispatching control of power systems under random fluctuations. Summary of the invention

[0004] The embodiments of the present invention provide a low-carbon dispatching control method, apparatus, terminal equipment and storage medium for a power system to solve the problem of low decision-making economy in the existing low-carbon dispatching control of a power system containing random fluctuations of renewable energy.

[0005] In order to solve the above problems, an embodiment of the present invention provides a low-carbon dispatch control method for a power system, comprising:

[0006] Based on the Markov decision process, a Markov decision process model of the low-carbon economic dispatch model of the power system is constructed;

[0007] According to the preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and Markov decision process model, the actor network is trained offline, and the constraint processing actor network is obtained after the offline training is completed;

[0008] Integrate the constraint processing actor network into the power system dispatching device, so that the power system dispatching device outputs the dispatching plan of each power generation equipment and energy storage in the power system based on the constraint processing actor network according to the real-time status information of the power system;

[0009] According to the scheduling plan, the power generation equipment and energy storage equipment in the power system are scheduled.

[0010] As an improvement of the above scheme, the low-carbon economic dispatch model includes: an objective function and constraint conditions;

[0011] The objective function satisfies the following conditions:

[0012]

[0013] In the formula, F W Including the cost of wind turbine power generation and wind curtailment penalty costs C1 is the power generation cost coefficient of wind turbine (yuan / MW·h); P W,i,t is the power generation of the i-th wind turbine per unit time (MW); C2 is the wind abandonment penalty coefficient (yuan / MW·h); P S,i,t is the predicted power of the i-th wind turbine in unit time (MW); b j 、c j is the power generation cost coefficient of the jth thermal power unit; C3 is the energy storage operation cost coefficient (yuan / MW·h), P es,k,t is the kth energy storage charging and discharging power per unit time, and the positive / negative values ​​represent the charging / discharging power respectively;

[0014] The constraints satisfy the following conditions:

[0015]

[0016] U i,min ≤U i,t ≤U i,max

[0017] Soc k,min ≤Soc k,t ≤Soc k,max

[0018]

[0019] In the formula, the first set of constraints above are the unit climbing, energy storage charging and discharging power, wind power output and unit output constraints, r Gup,j and r Gdown,j are the up and down ramp rates of the thermal power unit, ΔT is the time interval, P charge,max and P discharge,max are the maximum charging / discharging power of energy storage, P G,i,max and P G,i,min are the upper and lower limits of the active power output of the unit, P W,i,max and P W,i,min are the upper and lower limits of the active power output of the wind turbines respectively; the second group of constraints is the system power balance constraint, P G,i,t , P W,i,t , P es,i,tare the active power injected into node i by thermal power, wind power and energy storage at time t, Q G,i,t is the reactive power injected into node i by the thermal power unit at time t, P L,i,t and Q L,i,t are the active and reactive load demands of node i at time t, θ ij,t is the voltage difference between nodes i and j; B ij and G ij are the susceptance and conductance between nodes i and j, respectively, U i,t is the voltage of node i at time t; the other five constraints are the constraints related to voltage and energy storage Soc, U i,max and U i,min are the upper and lower limits of node i respectively; Soc k,max and Soc k,min are the upper and lower limits of the state of charge of energy storage k, Soc k,t is the state of charge of energy storage k at time t, E es,k,t and E es,k,max The actual power and upper limit capacity of energy storage k at time t, Soc k,t-1 is the state of charge at the previous moment, β es,k,t is the charge / discharge state, represented by a 0-1 variable, η charge,k,t and η discharge,k,t Respectively represent the charging / discharging efficiency of energy storage, P charge,k,t and P discharge,k,t Respectively represent the charging / discharging power of energy storage.

[0020] As an improvement of the above scheme, the Markov decision process model includes: action space, state space and reward; the Markov decision process model for constructing a low-carbon economic dispatch model of the power system based on the Markov decision process includes:

[0021] The output power and energy storage power of the low-carbon economic dispatch model are used as the action space; wherein the action space satisfies the following conditions:

[0022] A=[P G ,P es ]

[0023] Where P G ,P es Respectively represent the output power and energy storage power of conventional thermal power units;

[0024] The load demand, node voltage, energy storage charge state and time of the low-carbon economic dispatch model are taken as the state space, which is in the following form:

[0025] S=[P L ,V,Soc,t]

[0026] Where P L ,V,Soc,t are load demand, node voltage, energy storage state of charge, and time t respectively.

[0027] Based on the objective function and constraints of the low-carbon economic dispatch model, a reward for the power system is obtained; the reward meets the following conditions:

[0028]

[0029] s t =[P L (t),V t ,Soc(t),t]

[0030] a t =[P G (t),P es (t)]

[0031] In the formula, r t is the agent in state s at time t t Next, perform action a t The reward obtained after W 、F G 、F ESS , They represent wind power cost, thermal power generation cost, energy storage operation cost, and carbon trading cost respectively;

[0032] Based on the action space, the state space and the reward, the Markov decision process model of the power system is constructed.

[0033] As an improvement of the above scheme, the actor network is trained offline according to the preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and Markov decision process model, including:

[0034] In each episode of offline training, based on the Markov decision process model and the current actor network, the constraint processing operation is executed according to the preset number of time steps; wherein, the constraint processing operation is executed once in each time step, and the corrective action, power system state information and reward generated by each execution of the constraint processing operation are stored in the experience replay pool;

[0035] In each episode of offline training, when the number of execution time steps reaches the number of update steps set in the episode, a small batch of trajectories with the minimum batch size is randomly selected from the experience replay pool; the state and action pairs in the small batch of trajectories are scored by the critic network to obtain a score value; the loss value calculation formula of the current actor network is constructed based on the score value, and according to the loss value calculation formula, the actor network parameters are updated using gradient descent for the current actor network, and the next constraint processing operation is performed through the actor network containing the updated actor network parameters; the critic network parameters are updated using the cross-round information propagation update algorithm to obtain the updated critic network parameters, and the state and action pairs in the next small batch of trajectories are scored by including the updated critic network parameters;

[0036] When the number of episodes of the offline training reaches a preset episode number threshold, the offline training of the actor network is completed to obtain a constraint-processed actor network.

[0037] As an improvement of the above solution, the constraint processing operation includes:

[0038] Obtain the current power system status information;

[0039] Inputting the power system state information into a neural network in an actor network so that the neural network outputs a basic action; wherein the basic action is output power;

[0040] Based on the basic action, non-basic actions are solved by using a commercial solver and all equation constraints in the low-carbon economic dispatch model, and the basic action and the non-basic action are combined into the initial action of the actor network; wherein the non-basic action is the energy storage power; a=(a B ,a N ) is the initial action of the constraint processing actor network, a B For basic actions, a N It is a non-basic action;

[0041] Based on the equality constraints and inequality constraints in the low-carbon economic dispatch model, the initial action is corrected by the inequality simplified gradient formula to obtain the corrective action; wherein the inequality simplified gradient satisfies the following conditions:

[0042]

[0043] In the formula, the reduced gradient formula of the inequality is the action of multiple iterations to correct the unsatisfied constraints. Therefore, and Represents the corrective action at the kth iteration update η α is the learning rate, Δa B It represents the difference corrected at each iteration, so that the action gradually satisfies the equality constraints and inequality constraints corresponding to the constraints in the low-carbon economic dispatch model;

[0044] The corrective action and the power system status information are input into the Markov decision process model, a reward is obtained, and the inequality reduced gradient algorithm is re-executed; wherein, after obtaining the reward, the corrective action, the power system status information and the reward are input into the experience replay pool.

[0045] As an improvement of the above solution, the loss value calculation formula of the current actor network is constructed based on the scoring value, and according to the loss value calculation formula, the actor network parameters are updated using gradient descent for the current actor network, including:

[0046] Construct the loss value calculation formula of the current actor network based on the scoring value;

[0047] According to the loss value calculation formula and the small batch trajectory, the loss value of the actor network parameters is calculated by the gradient descent formula to maximize the negative value of the reward to complete the update of the actor network parameters; wherein the gradient descent formula satisfies the following conditions:

[0048]

[0049] In the formula, κ j is the Lagrange multiplier, is the multiplier κ j The learning rate, is kappa j The value of the kth iteration and Where k is the number of updates for neural network training, and each update Gradually increasing.

[0050] As an improvement of the above solution, the cross-round information propagation update algorithm includes:

[0051] Based on the mini-batch trajectory, calculate the initial loss value of the critic network parameter according to the loss calculation method of the critic network;

[0052] The difference feature is extracted for the data "state-action pair" in the small batch trajectory, and then the difference is added to the initial loss value to form the final loss value of the critic network parameter, and the critic network parameter is updated based on the final loss value of the critic network parameter; wherein the final loss value of the critic network parameter satisfies the following conditions:

[0053] d(s,a,s′,a′)=MLP(υ),υ=[υ1,υ2,...,V k ],

[0054]

[0055] Where d(s,a,s′,a′) is the difference between the state-action pairs (s,a) and (s′,a′), MLP(υ) is, υ is, MSE() is, Scoring enables the agent to recognize the difference between states more quickly, speeding up the training of the algorithm.

[0056] Accordingly, an embodiment of the present invention further provides a low-carbon dispatch control device for a power system, comprising: a model building module, a network training module, a network integration module and a dispatch module;

[0057] The model building module is used to build a Markov decision process model of a low-carbon economic dispatch model of the power system based on the Markov decision process;

[0058] The network training module is used to perform offline training on the actor network according to a preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and a Markov decision process model, and obtain a constraint-processed actor network after the offline training is completed;

[0059] The network integration module is used to integrate the constraint processing actor network into the power system dispatching device, so that the power system dispatching device outputs the dispatching plan of each power generation equipment and energy storage in the power system based on the constraint processing actor network according to the real-time status information of the power system;

[0060] The scheduling module is used to schedule the power generation equipment and energy storage equipment in the power system according to the scheduling plan.

[0061] Correspondingly, an embodiment of the present invention further provides a computer terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a low-carbon dispatching control method for a power system as described in the present invention.

[0062] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a low-carbon dispatching control method for a power system as described in the present invention.

[0063] As can be seen from the above, the present invention has the following beneficial effects:

[0064] The present invention provides a low-carbon dispatch control method for an electric power system, comprising: constructing a low-carbon economic dispatch model for an electric power system based on a Markov decision process according to information on the joint participation of new energy and carbon trading mechanisms in the optimal dispatch of the electric power system. According to the model, a low-carbon economic dispatch strategy for an electric power system based on reinforcement learning of inequality simplified gradients and cross-round information propagation is proposed; based on the proposed reinforcement learning method, the trained constraint processing actor network is integrated into the electric power system dispatch device and put into use, and a dispatch plan for generator sets and energy storage is made according to the real-time status information of the system. Based on the demand for low-carbon economic dispatch of the electric power system, the present invention processes the general form of equality and inequality constraints of the dispatch problem through the constraint processing actor network and realizes the strategy network training of the method; and uses the method of updating the critic network parameters through cross-round information propagation to improve the learning ability of the intelligent agent. Based on the proposed method, the defects of reduced learning ability and poor compatibility of general constraint reinforcement learning are overcome, and the economy of low-carbon dispatch control decision-making of the electric power system is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a flow chart of a low-carbon dispatch control method for a power system provided by an embodiment of the present invention;

[0066] Figure 2 is a structural schematic diagram of a low-carbon dispatching control device for a power system provided by an embodiment of the present invention;

[0067] Figure 3 This is a schematic diagram of the structure of a terminal device provided by an embodiment of the present invention;

[0068] Figure 4 It is a specific structural diagram of a constraint processing actor network provided by an embodiment of the present invention;

[0069] Figure 5 is a flow chart of a method for updating critic network parameters provided by an embodiment of the present invention;

[0070] Figure 6 It is a flowchart of a reinforcement learning algorithm provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] Embodiment 1

[0073] See also Figure 1 , Figure 1 is a flow chart of a low-carbon dispatch control method for a power system provided by an embodiment of the present invention, such as Figure 1 As shown, this embodiment includes step 101 to step 102, and each step is specifically as follows:

[0074] Step 101: Based on the Markov decision process, a Markov decision process model of a low-carbon economic dispatch model of a power system is constructed.

[0075] In this embodiment, the operating status of the power system at each moment is obtained.

[0076] In this embodiment, the low-carbon economic dispatch model includes: an objective function and constraints;

[0077] The objective function satisfies the following conditions:

[0078]

[0079] In the formula, F W Including the cost of wind turbine power generation and wind curtailment penalty costs C1 is the power generation cost coefficient of wind turbine (yuan / MW·h); P W,i,t is the power generation of the i-th wind turbine per unit time (MW); C2 is the wind abandonment penalty coefficient (yuan / MW·h); P S,i,t is the predicted power of the i-th wind turbine in unit time (MW); b j 、c j is the power generation cost coefficient of the jth thermal power unit; C3 is the energy storage operation cost coefficient (yuan / MW·h), P es,k,t is the kth energy storage charging and discharging power per unit time, and the positive / negative values ​​represent the charging / discharging power respectively;

[0080] The constraints satisfy the following conditions:

[0081]

[0082] U i,min ≤U i,t≤U i,max

[0083] Soc k,min ≤Soc k,t ≤Soc k,max

[0084]

[0085] In the formula, the first set of constraints above are the unit climbing, energy storage charging and discharging power, wind power output and unit output constraints, r Gup,j and r Gdown,j are the up and down ramp rates of the thermal power unit, ΔT is the time interval, P charge,max and P discharge,max are the maximum charging / discharging power of energy storage, P G,i,max and P G,i,min are the upper and lower limits of the active power output of the unit, P W,i,max and P W,i,min are the upper and lower limits of the active power output of the wind turbines respectively; the second group of constraints is the system power balance constraint, P G,i,t , P W,i,t , P es,i,t are the active power injected into node i by thermal power, wind power and energy storage at time t, Q G,i,t is the reactive power injected into node i by the thermal power unit at time t, P L,i,t and Q L,i,t are the active and reactive load demands of node i at time t, θ ij,t is the voltage difference between nodes i and j; B ij and G ij are the susceptance and conductance between nodes i and j, respectively, U i,t is the voltage of node i at time t; the other five constraints are the constraints related to voltage and energy storage Soc, U i,max and U i,min are the upper and lower limits of node i respectively; Soc k,max and Soc k,min are the upper and lower limits of the state of charge of energy storage k, Soc k,t is the state of charge of energy storage k at time t, E es,k,t and E es,k,max The actual power and upper limit capacity of energy storage k at time t, Soc k,t-1 is the state of charge at the previous moment, β es,k,t is the charge / discharge state, represented by a 0-1 variable, η charge,k,t and η discharge,k,t Respectively represent the charging / discharging efficiency of energy storage, P charge,k,t and P discharge,k,t Respectively represent the charging / discharging power of energy storage.

[0086] In a specific embodiment, the low-carbon economic dispatch model includes:

[0087] ①Objective function

[0088] 1) Carbon trading model

[0089] The carbon quota is calculated based on the power generation of the system units, the carbon emissions are calculated based on the power generation of conventional thermal power units, and the system carbon trading cost is calculated by combining the carbon quota and carbon emissions. The system carbon quota is proportional to the power generation of the unit. d It is expressed as:

[0090]

[0091] Where P D,t is the sum of the output of conventional units and wind power in unit time t, specifically expressed as M and N are the number of wind power and conventional units respectively; η is the carbon emission allocation per unit of electricity, t / (MW·h); P W,i,t and P G,j,t Represent the output power of different wind power and conventional thermal power units at time t. Since the carbon emissions generated by wind power operation are much smaller than those of conventional thermal power units (negligible), carbon emissions are only taken into account for thermal power units. Carbon emissions of thermal power units E p It is expressed as:

[0092]

[0093] Where T is the scheduling period, P b =P G,j,t / P G,j,max is the unit coal consumption rate, μ j =(y j P b +z j ), the coal consumption coefficient of the jth thermal power unit is y j and z j ;μ j ρP G,j,t is the carbon emission of the jth unit in unit time t; ρ is the carbon conversion coefficient, kg / (MW·h). Finally, the carbon trading cost is expressed as:

[0094]

[0095] In the formula is the carbon trading cost, is the carbon trading price (yuan / t).

[0096] 2) Low-carbon economic dispatch model of power system including wind power

[0097] The objective function of the low-carbon economic dispatch model of the power system containing wind power can be expressed as:

[0098]

[0099] F in the formula W 、F G 、F ESS , They represent wind power cost, thermal power generation cost, energy storage operation cost, and carbon trading cost respectively. W Including the cost of wind turbine power generation and wind curtailment penalty costs The specific expression is as follows:

[0100]

[0101] C1 in formula (6) is the power generation cost coefficient of wind turbine (yuan / MW·h); P W,i,t is the power generation of the i-th wind turbine per unit time (MW); C2 is the wind abandonment penalty coefficient (yuan / MW·h); P S,i,t is the predicted power of the i-th wind turbine in unit time (MW). The power generation cost of conventional thermal power units F G As shown below:

[0102]

[0103] Among them, b j 、c j is the power generation cost coefficient of the jth thermal power unit. In order to reduce the impact of wind power fluctuations on the power grid and to ensure the power transmission balance of the power grid to a greater extent, a certain scale of energy storage is configured in the system. The operation and maintenance cost of energy storage is F ESS The expression is as follows:

[0104]

[0106] Where C3 is the energy storage operation cost coefficient (yuan / MW·h), P es,k,t is the kth energy storage charging and discharging power per unit time, and the positive / negative values ​​represent the charging / discharging power respectively.

[0107] In this embodiment, the Markov decision process model includes: an action space, a state space, and a reward; the Markov decision process model for constructing a low-carbon economic dispatch model of a power system based on the Markov decision process includes:

[0108] The output power and energy storage power of the low-carbon economic dispatch model are used as the action space; wherein the action space satisfies the following conditions:

[0109] A=[PG ,P es ]

[0110] Where P G ,P es Respectively represent the output power and energy storage power of conventional thermal power units;

[0111] The load demand, node voltage, energy storage charge state and time of the low-carbon economic dispatch model are taken as the state space, which is in the following form:

[0112] S=[P L ,V,Soc,t]

[0113] Where P L ,V,Soc,t are load demand, node voltage, energy storage state of charge, and time t respectively.

[0114] Based on the objective function and constraints of the low-carbon economic dispatch model, a reward for the power system is obtained; the reward meets the following conditions:

[0115]

[0116] s t =[P L (t),V t ,Soc(t),t]

[0117] a t =[P G (t),P es (t)]

[0118] In the formula, r t is the agent in state s at time t t Next, perform action a t The reward obtained after W 、F G 、F ESS , They represent wind power cost, thermal power generation cost, energy storage operation cost, and carbon trading cost respectively;

[0119] Based on the action space, the state space and the reward, the Markov decision process model of the power system is constructed.

[0120] In a specific embodiment, the low-carbon economic dispatch Markov decision process model includes:

[0121] Based on the mathematical expression of the low-carbon economic dispatch model of the power system containing wind power, the Markov decision process is used here to describe a series of state transfer processes, which is mainly composed of a tuple (S, A, R, P, γ). S represents the state space of the agent, A represents the action space taken by the agent, R represents the reward return, P is the state transfer function, and γ is the reward discount factor. After the low-carbon economic dispatch model of the power system containing wind power is converted into a Markov decision process and the state space, action space and reward function are determined, the reinforcement learning method proposed later can be used to allow the agent to interact with the model environment of the low-carbon economic dispatch problem of the power system containing wind power to train and solve the problem.

[0122] Step 102: According to the preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and Markov decision process model, the actor network is trained offline, and the constraint processing actor network is obtained after the offline training is completed.

[0123] In this embodiment, the actor network is trained offline according to the preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and Markov decision process model, including:

[0124] In each episode of offline training, based on the Markov decision process model and the current actor network, the constraint processing operation is executed according to the preset number of time steps; wherein, the constraint processing operation is executed once in each time step, and the corrective action, power system state information and reward generated by each execution of the constraint processing operation are stored in the experience replay pool;

[0125] In each episode of offline training, when the number of execution time steps reaches the number of update steps set in the episode, a small batch of trajectories with the minimum batch size is randomly selected from the experience replay pool; the state and action pairs in the small batch of trajectories are scored by the critic network to obtain a score value; the loss value calculation formula of the current actor network is constructed based on the score value, and according to the loss value calculation formula, the actor network parameters are updated using gradient descent for the current actor network, and the next constraint processing operation is performed through the actor network containing the updated actor network parameters; the critic network parameters are updated using the cross-round information propagation update algorithm to obtain the updated critic network parameters, and the state and action pairs in the next small batch of trajectories are scored by including the updated critic network parameters;

[0126] When the number of episodes of the offline training reaches a preset episode number threshold, the offline training of the actor network is completed to obtain a constraint-processed actor network.

[0127] In this embodiment, the constraint processing operation includes:

[0128] Obtain the current power system status information;

[0129] Inputting the power system state information into a neural network in an actor network so that the neural network outputs a basic action; wherein the basic action is output power;

[0130] Based on the basic action, non-basic actions are solved by using a commercial solver and all equation constraints in the low-carbon economic dispatch model, and the basic action and the non-basic action are combined into the initial action of the actor network; wherein the non-basic action is the energy storage power; a=(a B ,a N ) is the initial action of the constraint processing actor network, a B For basic actions, a N It is a non-basic action;

[0131] Based on the equality constraints and inequality constraints in the low-carbon economic dispatch model, the initial action is corrected by the inequality simplified gradient formula to obtain the corrective action; wherein the inequality simplified gradient satisfies the following conditions:

[0132]

[0133] In the formula, the reduced gradient formula of the inequality is the action of multiple iterations to correct the unsatisfied constraints. Therefore, and Represents the corrective action at the kth iteration update η α is the learning rate, Δa B It represents the difference corrected at each iteration, so that the action gradually satisfies the equality constraints and inequality constraints corresponding to the constraints in the low-carbon economic dispatch model;

[0134] The corrective action and the power system status information are input into the Markov decision process model, a reward is obtained, and the inequality reduced gradient algorithm is re-executed; wherein, after obtaining the reward, the corrective action, the power system status information and the reward are input into the experience replay pool.

[0135] In this embodiment, the loss value calculation formula of the current actor network is constructed based on the scoring value, and according to the loss value calculation formula, the actor network parameters are updated using gradient descent on the current actor network, including:

[0136] Construct the loss value calculation formula of the current actor network based on the scoring value;

[0137] According to the loss value calculation formula and the small batch trajectory, the loss value of the actor network parameters is calculated by the gradient descent formula to maximize the negative value of the reward to complete the update of the actor network parameters; wherein the gradient descent formula satisfies the following conditions:

[0138]

[0139] In the formula, κ j is the Lagrange multiplier, is the multiplier κ j The learning rate, is kappa j The value of the kth iteration and Where k is the number of updates for neural network training, and each update Gradually increasing.

[0140] In this embodiment, the cross-round information propagation update algorithm includes:

[0141] Based on the mini-batch trajectory, calculate the initial loss value of the critic network parameter according to the loss calculation method of the critic network;

[0142] The difference feature is extracted for the data "state-action pair" in the small batch trajectory, and then the difference is added to the initial loss value to form the final loss value of the critic network parameter, and the critic network parameter is updated based on the final loss value of the critic network parameter; wherein the final loss value of the critic network parameter satisfies the following conditions:

[0143] d(s,a,s′,a′)=MLP(υ),υ=[υ1,v2,...,v k ],

[0144]

[0145] Where d(s,a,s′,a′) is the difference between the state-action pairs (s,a) and (s′,a′), MLP(v) is, v is, MSE() is, Scoring enables the agent to recognize the difference between states more quickly, speeding up the training of the algorithm.

[0146] In a specific embodiment, the constraints of the low-carbon economic dispatch model include:

[0147]

[0148] Among them, t represents the time period, g j and fi Respectively represent only the current state s t and action a t There are m equality constraints in total. For the proposed algorithm, π θ represents the policy network (equivalent to the actor network in the Actor-Critic learning framework), the actor network in period t is based on the state s t The output action is a t =π θ (s t ), J R (π θ ) is the strategy π θ The objective function value under the condition, the optimization goal of the algorithm is to maximize J R (π θ ). At this time, equations (20)-(22) add equation (21) relative to the general problem form processed by the simplified gradient algorithm. Therefore, based on the simplified gradient method, the present invention proposes an actor network (herein referred to as a constraint processing actor network) that combines inequality simplified gradient with a neural network.

[0149] The specific structure of the constraint processing actor network is as follows Figure 4 First, refer to the basic simplified gradient algorithm and convert the action Divided into basic actions and non-basic actions Then, a neural network module outputs the initial basic action a B , and obtain the initial value a of the non-basic action based on the solver N The initial action without correction is However, based only on the initial action, it is impossible to guarantee that the equality and inequality constraints are fully satisfied at the same time and the economy of the decision. Finally, the proposed inequality simplified gradient algorithm is used to update the action in the region defined by the inequality constraint expressed in equation (21) and ensure the satisfaction of equation (22). After k iterations of update and correction, the final feasible action a is obtained. k .

[0150] Detailed process of inequality simplified gradient update action a: Here, all constraints in equation (22) are defined as F(a;s) = 0, let a N =v N (a N ), Referring to the ordinary simplified gradient method, we can get N to a B The gradient of

[0151]

[0152] The basic simplified gradient algorithm does not consider the constraints described in equation (21). In order to ensure that the output of the constraint processing actor satisfies all constraints, it is proposed to limit the update of the equality constraint (22) to the area defined by the inequality constraint (21). The specific update is shown in equations (24)-(28). Here, the sum of the maximum violations of inequality (21) is G(a B ,a N ), Similar to the basic minimalistic gradient algorithm, for a B and a N Update so that the action finally satisfies all constraints. k Updated as follows:

[0153]

[0154] η in formulas (27) and (28) α This is equivalent to the learning rate when the basic minimalistic gradient algorithm is updated, and is used here to control the update rate. and Indicates the action at the kth update According to equations (24)-(28), the sum and number of inequality violations will continue to decrease during the iteration process, and the final output will fully satisfy the equality and inequality constraints. The entire process from input state s to output a is defined as the complete constraint processing actor network (that is, the final policy network).

[0155] In a specific embodiment, when using TD update, state transition information can only be propagated to the previous state in the same round, which limits the data efficiency of the reinforcement learning algorithm. Figure 5 As shown, the present invention proposes a method for propagating fragment information across rounds, which mutually propagates state-action pairs in different rounds to enhance the learning ability of the intelligent agent.

[0156] Figure 5 (a) is a schematic diagram of the cross-round propagation of state-action pairs. represents the trajectory of the kth round of interaction between the agent and the environment, and They are the state-action pairs and their corresponding Q values ​​of the trajectory of the kth round in the nth period. When the algorithm sends each batch of data to training, the cross-round information propagation algorithm passes the data in the batch through the similarity calculation network and the difference calculation network to obtain the difference d between the corresponding state-action pairs (s′, a′) and (s, a) of the batch, and uses the difference d to update the critic network. Figure 5 (b) is the specific process of cross-round information propagation. The specific steps of cross-round information propagation are as follows:

[0157] Similarity calculation: First, based on any given two state-action pairs (s′, a′) and (s, a), the multi-head encoder extracts features q and q′, and then uses the similarity function to calculate the similarity between features q and q′ (the similarity function here uses cosine similarity). The specific expression is as follows:

[0158]

[0159] In formula (29), i∈1,2,...,k, where i is the head number of the multi-head encoder.

[0160] Difference calculation: similarity v is obtained after feature extraction i After the value is obtained, a multi-layer perceptron (MLP) is used to infer the difference between (s′, a′) and (s, a), and finally the difference d and Q(s, a) are used to infer the value of Q(s′, a′), as shown below:

[0161] d(s,a,s′,a′)=MLP(υ),υ=[υ1,υ2,...,υ k ](30)

[0162] Q(s′,a′)←Q(s,a)+d(s,a,s′,a′)(31)

[0163] Finally, the Q-value update method of the critic network using the cross-round information propagation method is combined with the constraint processing actor network. This paper proposes a new constrained reinforcement learning method based on the soft actor-critic (SAC) framework [6]. In order to implement the algorithm training under the SAC framework, an additional Lagrange multiplier and a penalty term are introduced to transform the problem represented by equations (20)-(22) into an unconstrained problem. The transformed expression of the problem is as follows:

[0164]

[0165] where κ j is the Lagrange multiplier, which is expressed as follows:

[0166]

[0167] is the multiplier κ j The learning rate, is kappa j The value of the kth iteration and because It is monotonically increasing during training, so there must be Therefore, equation (32) is equivalent to the problem expressed by equations (20)-(22).

[0168] To better illustrate the reinforcement learning method based on inequality-based reduced gradients and cross-episode information propagation, refer to the following steps:

[0169] Step 1: First, according to the reinforcement learning algorithm based on inequality simplified gradient and cross-round information propagation proposed in the present invention, the action A in the Markov decision process of the low-carbon economic dispatch model of the wind power system is converted into G ,P es ] is equivalent to the basic action a of the proposed algorithm B and non-basic actions a N Combination a (set P G =a B , P es =a N ,[P G ,P es ]=a).

[0170] Step 2: Based on Figure 6 The schematic diagram of the actor shown in the figure shows the s at each moment. t Input into the actor network and let the actor network output an initial algorithm a B At this time, with a B is a known quantity. Based on the equality constraints in the low-carbon economic dispatch model of the power system containing wind power, the unknown action a is obtained by the solver. N , so we get the state s t The complete initial action a is a. At this time, a cannot satisfy the inequality constraint. Therefore, we use the inequality simplified gradient method shown in formulas (23)-(28) to construct G(a B ,a N ), for a B and a N Perform iterative correction to take the inequality constraints into account. After k iterations, we get Make a k =[P G ,P es ] satisfies all constraints of the optimization problem.

[0171] Step 3: Based on a k Interact with the environment of the low-carbon economic dispatch model of the wind power system to obtain the reward (that is, the cost value of the objective function).

[0172] Step 4: The algorithm is cycled based on steps 1 to 3. Through the continuous interaction between the intelligent agent and the low-carbon economic dispatch model environment of the wind power system, the actor and critic networks are updated with the goal of minimizing equation (32) to achieve the optimal strategy network π θ training.

[0173] Step 5: When the strategy network π is completed θ After training, when making decisions, only the state s at each moment needs to be input into the actor network t , the actor network can output the optimal action a, so that we can only know the state s at each moment in a random environment t , we can quickly get the optimal solution to the problem [P G ,P es ].

[0174] To better illustrate the reinforcement learning algorithm, based on the basic SAC framework, the actor and critic network parameter update process of the new constrained reinforcement learning algorithm proposed in this embodiment under the SAC framework is shown in Table 1 below:

[0175]

[0176]

[0177] It should be noted that ③ to ⑦ in Table 1 are specific steps for updating the input parameters described in the present invention.

[0178] Step 103: Integrate the constraint processing actor network into the power system dispatching device, so that the power system dispatching device outputs a dispatching plan for each power generation equipment and energy storage in the power system based on the constraint processing actor network according to the real-time status information of the power system.

[0179] Step 104: Scheduling power generation equipment and energy storage equipment in the power system according to the scheduling plan.

[0180] See also Figure 2 , Figure 2 It is a structural schematic diagram of a low-carbon dispatch control device for a power system provided by an embodiment of the present invention, comprising: a model building module 201, a network training module 202, a network integration module 203 and a dispatch module 204;

[0181] The model building module is used to build a Markov decision process model of a low-carbon economic dispatch model of the power system based on the Markov decision process;

[0182] The network training module is used to perform offline training on the actor network according to a preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and a Markov decision process model, and obtain a constraint-processed actor network after the offline training is completed;

[0183] The network integration module is used to integrate the constraint processing actor network into the power system dispatching device, so that the power system dispatching device outputs the dispatching plan of each power generation equipment and energy storage in the power system based on the constraint processing actor network according to the real-time status information of the power system;

[0184] The scheduling module is used to schedule the power generation equipment and energy storage equipment in the power system according to the scheduling plan.

[0185] It can be understood that the above-mentioned system item embodiments correspond to the method item embodiments of the present invention, which can implement the low-carbon dispatching control method for the power system provided by any one of the above-mentioned method item embodiments of the present invention.

[0186] This embodiment overcomes the problem that the current constrained reinforcement learning cannot take into account the universality of equality constraint processing and its constraint processing method through the actor network based on the inequality simplified gradient algorithm to correct the action, and can greatly improve the universality of the method while meeting various constraints of the power system. For the first time, the parameter update process is improved from the perspective of cross-round information propagation, and applied to the proposed actor network based on the inequality simplified gradient algorithm to correct the action, forming a new constrained reinforcement learning algorithm to significantly improve the learning ability of the algorithm.

[0187] Embodiment 2

[0188] See also Figure 3 , Figure 3 It is a schematic diagram of the structure of a terminal device provided in one embodiment of the present invention.

[0189] A terminal device of this embodiment includes: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the steps of the above-mentioned low-carbon dispatch control method for a power system in the embodiment are implemented, for example: Figure 1 Alternatively, when the processor executes the computer program, the functions of each module in the above-mentioned device embodiments are realized, for example: Figure 2 All modules of a low carbon dispatch control device for a power system are shown.

[0190] In addition, an embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the low-carbon dispatching control method for the power system as described in any of the above embodiments.

[0191] Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0192] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 301 is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0193] The memory 302 can be used to store the computer program and / or module. The processor 301 implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0194] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0195] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A low-carbon dispatch control method for a power system, characterized in that: include: Based on the Markov decision process, a Markov decision process model of the low-carbon economic dispatch model of the power system is constructed; According to the preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and Markov decision process model, the actor network is trained offline, and the constraint processing actor network is obtained after the offline training is completed; Integrate the constraint processing actor network into the power system dispatching device, so that the power system dispatching device outputs the dispatching plan of each power generation equipment and energy storage in the power system based on the constraint processing actor network according to the real-time status information of the power system; According to the scheduling plan, the power generation equipment and energy storage equipment in the power system are scheduled.

2. The low-carbon dispatch control method for a power system according to claim 1, characterized in that: The low-carbon economic dispatch model includes: an objective function and constraints; The objective function satisfies the following conditions: In the formula, F W Including the cost of wind turbine power generation and wind curtailment penalty costs C1 is the power generation cost coefficient of wind turbine (yuan / MW·h); P W,i,t is the power generation of the i-th wind turbine per unit time (MW); C2 is the wind abandonment penalty coefficient (yuan / MW·h); P S,i,t is the predicted power of the i-th wind turbine in unit time (MW); b j 、c j is the power generation cost coefficient of the jth thermal power unit; C3 is the energy storage operation cost coefficient (yuan / MW·h), P es,k,t is the kth energy storage charging and discharging power per unit time, and the positive / negative values ​​represent the charging / discharging power respectively; The constraints satisfy the following conditions: IN i,min ≤U i,t ≤U i,max Soc k,min ≤Soc k,t ≤Soc k,max In the formula, the first set of constraints above are the unit climbing, energy storage charging and discharging power, wind power output and unit output constraints, r Gup,j and r Gdown,j are the up and down ramp rates of the thermal power unit, ΔT is the time interval, P charge,max and P discharge,max are the maximum charging / discharging power of energy storage, P G,i,max and P G,i,min are the upper and lower limits of the active power output of the unit, P W,i,max and P W,i,min are the upper and lower limits of the active power output of the wind turbines respectively; the second group of constraints is the system power balance constraint, P G,i,t , P W,i,t , P es,i,t are the active power injected into node i by thermal power, wind power and energy storage at time t, Q G,i,t is the reactive power injected into node i by the thermal power unit at time t, P L,i,t and Q L,i,t are the active and reactive load demands of node i at time t, θ ij,t is the voltage difference between nodes i and j; B ij and G ij are the susceptance and conductance between nodes i and j, respectively, U i,t is the voltage of node i at time t; the other five constraints are the constraints related to voltage and energy storage Soc, U i,max and U i,min are the upper and lower limits of node i respectively; Soc k,max and Soc k,min are the upper and lower limits of the state of charge of energy storage k, Soc k,t is the state of charge of energy storage k at time t, E es,k,t and E es,k,max The actual power and upper limit capacity of energy storage k at time t, Soc k,t-1 is the state of charge at the previous moment, β es,k,t is the charge / discharge state, represented by a 0-1 variable, η charge,k,t and η discharge,k,t Respectively represent the charging / discharging efficiency of energy storage, P charge,k,t and P discharge,k,t Respectively represent the charging / discharging power of energy storage.

3. The low-carbon dispatch control method for a power system according to claim 2, characterized in that: The Markov decision process model includes: action space, state space and reward; the Markov decision process model for constructing a low-carbon economic dispatch model of the power system based on the Markov decision process includes: The output power and energy storage power of the low-carbon economic dispatch model are used as the action space; wherein the action space satisfies the following conditions: A=[P G ,P es ] Where P G ,P es Respectively represent the output power and energy storage power of conventional thermal power units; The load demand, node voltage, energy storage charge state and time of the low-carbon economic dispatch model are taken as the state space in the following form: S=[P L ,V,Soc,t] Where P L ,V,Soc,t are load demand, node voltage, energy storage state of charge, and time t respectively. Based on the objective function and constraints of the low-carbon economic dispatch model, a reward for the power system is obtained; the reward meets the following conditions: s t =[P L (t),V t ,Soc(t),t] a t =[P G (t),P es (t)] In the formula, r t is the agent in state s at time t t Next, perform action a t The reward obtained after W 、F G 、F ESS , They represent wind power cost, thermal power generation cost, energy storage operation cost, and carbon trading cost respectively; Based on the action space, the state space and the reward, the Markov decision process model of the power system is constructed.

4. The low-carbon dispatch control method for a power system according to claim 3, characterized in that: The actor network is trained offline according to the preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and Markov decision process model, including: In each episode of offline training, based on the Markov decision process model and the current actor network, the constraint processing operation is executed according to the preset number of time steps; wherein, the constraint processing operation is executed once in each time step, and the corrective action, power system state information and reward generated by each execution of the constraint processing operation are stored in the experience replay pool; In each episode of offline training, when the number of execution time steps reaches the number of update steps set in the episode, a small batch of trajectories with the minimum batch size is randomly selected from the experience replay pool; the state and action pairs in the small batch of trajectories are scored by the critic network to obtain a score value; the loss value calculation formula of the current actor network is constructed based on the score value, and according to the loss value calculation formula, the actor network parameters are updated using gradient descent for the current actor network, and the next constraint processing operation is performed through the actor network containing the updated actor network parameters; the critic network parameters are updated using the cross-round information propagation update algorithm to obtain the updated critic network parameters, and the state and action pairs in the next small batch of trajectories are scored by including the updated critic network parameters; When the number of episodes of the offline training reaches a preset episode number threshold, the offline training of the actor network is completed to obtain a constraint-processed actor network.

5. The low-carbon dispatch control method for power system according to claim 4, characterized in that: The constraint processing operation includes: Obtain the current power system status information; Inputting the power system state information into a neural network in an actor network so that the neural network outputs a basic action; wherein the basic action is output power; Based on the basic action, non-basic actions are solved by using a commercial solver and all equation constraints in the low-carbon economic dispatch model, and the basic action and the non-basic action are combined into the initial action of the actor network; wherein the non-basic action is the energy storage power; a=(a B ,a N ) is the initial action of the constraint processing actor network, a B For basic actions, a N It is a non-basic action; Based on the equality constraints and inequality constraints in the low-carbon economic dispatch model, the initial action is corrected by the inequality simplified gradient formula to obtain the corrective action; wherein the inequality simplified gradient satisfies the following conditions: In the formula, the reduced gradient formula of the inequality is the action of multiple iterations to correct the unsatisfied constraints. Therefore, and Represents the corrective action at the kth iteration update η α is the learning rate, Δa B It represents the difference corrected at each iteration, so that the action gradually satisfies the equality constraints and inequality constraints corresponding to the constraints in the low-carbon economic dispatch model; The corrective action and the power system status information are input into the Markov decision process model, a reward is obtained, and the inequality reduced gradient algorithm is re-executed; wherein, after obtaining the reward, the corrective action, the power system status information and the reward are input into the experience replay pool.

6. The low-carbon dispatch control method for power system according to claim 4, characterized in that: The loss value calculation formula of the current actor network is constructed based on the scoring value, and according to the loss value calculation formula, the actor network parameters are updated using gradient descent on the current actor network, including: Construct the loss value calculation formula of the current actor network based on the scoring value; According to the loss value calculation formula and the small batch trajectory, the loss value of the actor network parameters is calculated by the gradient descent formula to maximize the negative value of the reward to complete the update of the actor network parameters; wherein the gradient descent formula satisfies the following conditions: In the formula, κ j is the Lagrange multiplier, is the multiplier κ j The learning rate, is kappa j The value of the kth iteration and Where k is the number of updates for neural network training, and each update Gradually increasing.

7. The low-carbon dispatch control method for power system according to claim 4, characterized in that: The cross-round information propagation update algorithm includes: Based on the mini-batch trajectory, calculate the initial loss value of the critic network parameter according to the loss calculation method of the critic network; The difference feature is extracted for the data "state-action pair" in the small batch trajectory, and then the difference is added to the initial loss value to form the final loss value of the critic network parameter, and the critic network parameter is updated based on the final loss value of the critic network parameter; wherein the final loss value of the critic network parameter satisfies the following conditions: d(s,a,s′,a′)=MLP(υ),υ=[υ1,υ2,...,υ k ], Where d(s,a,s′,a′) is the difference between the state-action pairs (s,a) and (s′,a′), MLP(υ) is, v is, and MSE() is, Scoring enables the agent to recognize the difference between states more quickly, speeding up the training of the algorithm.

8. A low-carbon dispatch control device for a power system, characterized in that: include: Model building module, network training module, network integration module and scheduling module; The model building module is used to build a Markov decision process model of a low-carbon economic dispatch model of the power system based on the Markov decision process; The network training module is used to perform offline training on the actor network according to a preset reinforcement learning method based on inequality simplified gradient and cross-round information propagation and a Markov decision process model, and obtain a constraint-processed actor network after the offline training is completed; The network integration module is used to integrate the constraint processing actor network into the power system dispatching device, so that the power system dispatching device outputs the dispatching plan of each power generation equipment and energy storage in the power system based on the constraint processing actor network according to the real-time status information of the power system; The scheduling module is used to schedule the power generation equipment and energy storage equipment in the power system according to the scheduling plan.

9. A computer terminal device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements a low-carbon dispatch control method for a power system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a low-carbon dispatch control method for a power system as described in any one of claims 1 to 7.

Images