Power system real-time low-carbon economic dispatching method and device based on parallel distributed flexible action device-evaluator, electronic equipment and storage medium

Abstract

The invention discloses an electric power system real-time low-carbon economic dispatching method and device based on a parallel distributed flexible action device-evaluator, electronic equipment and a storage medium, and the method comprises the steps: jointly participating in the real-time optimal dispatching information of an electric power system according to large-scale new energy and a carbon transaction mechanism; and constructing a real-time low-carbon economic dispatching model of the power system based on the Markov decision process. According to the model, a parallel distributed flexible action device-evaluator algorithm is proposed to train an agent offline to learn a low-carbon economic dispatching strategy; and integrating the trained intelligent agent into a power system dispatching device for application, and deciding a dispatching scheme of each generator set according to the real-time state information of the system. According to the method, on the basis of the real-time low-carbon economic dispatching requirement of a power system, the continuous action space of the dispatching problem is processed through a flexible action device-evaluator; and a distributed flexible strategy iteration framework and a parallel technology are utilized to improve a flexible action device-evaluator, so that the solving speed and precision are improved.

Description

Technical Field

[0001] This invention relates to the field of power grid dispatching technology, specifically to a real-time low-carbon economic dispatching method, device, electronic equipment, and storage medium for power systems based on a parallel distributed flexible actor-evaluator. Background Technology

[0002] With the increasing severity of global climate change, carbon emission reduction has become a core strategic goal for development in various countries. In promoting energy transition, China is gradually shifting from a traditional policy of controlling both energy consumption and carbon emissions to one of controlling both. Energy conservation and carbon reduction in the power system have become a crucial link in achieving low-carbon development. The power industry is moving from solely relying on new energy power generation technologies to a comprehensive "electricity-carbon" synergistic mechanism that considers the carbon trading market and the entire energy production and consumption chain. Effectively reducing carbon emissions during power system operation and optimizing dispatch economics to achieve synergistic development between low-carbon and economic goals is an important current research direction.

[0003] However, existing technologies still face many challenges in real-time low-carbon economic dispatch. The increasing penetration rate of renewable energy brings uncertainty to power system operation and exacerbates the complexity of carbon emission and economic optimization. Traditional dispatch methods are often inefficient and inaccurate when dealing with the randomness, nonlinearity, and dynamic changes in the system. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for real-time low-carbon economic dispatching of power systems based on a parallel distributed flexible actuator-evaluator. Implementing this invention improves the efficiency and accuracy of real-time power system dispatching.

[0005] An embodiment of the present invention provides a real-time low-carbon economic dispatch method for power systems based on a parallel distributed flexible actor-evaluator, comprising:

[0006] A real-time low-carbon economic dispatch model for the power system is constructed based on the Markov decision process. Specifically, the state space of the Markov decision process is constructed based on the active power output and reactive power output of thermal power units at time t-1, the voltage amplitude of all nodes at time t-1, the phase angle of all nodes at time t-1, the predicted active power output and predicted reactive power output of renewable energy units at time t, the predicted active load and predicted reactive load of load nodes at time t, and the predicted reactive load of load nodes at time t. The action space is constructed based on the output increment of thermal power units at PV nodes at time t, the voltage amplitude of PV nodes at time t, and the amount of wind and solar curtailment of renewable energy units at time t. The action space is constructed based on the given current state s. t Actions taken (a) t Below, and transition to the new state s t+1Construct a state transition function; construct a reward function based on the power generation cost of thermal power units, the cost of wind and solar curtailment, the cost of carbon emission trading, and cost penalty terms;

[0007] The agent corresponding to the real-time low-carbon economic dispatch model of the power system is trained offline according to the parallel distributed flexible actor-evaluator algorithm.

[0008] The offline trained agent is integrated into the power system dispatching equipment so that the power system dispatching equipment can generate a dispatching plan for each generator set in the power system based on real-time status information, and dispatch the generator sets in the power system according to the dispatching plan.

[0009] Furthermore, the state space includes:

[0010]

[0011] Among them, s t This refers to the state information at time t; This represents the active power output of the thermal power unit at time t-1. For the reactive power output of the thermal power unit at time t-1, U t-1 For the voltage magnitude and δ at time t-1 of all nodes t-1 Let be the phase angle of all nodes at time t-1. For the prediction of active power output of new energy generating units at time t, For the predicted reactive power output of the new energy unit at time t, For the predicted active power load at load node t, This represents the predicted reactive load at load node t.

[0012] Furthermore, the action space includes:

[0013]

[0014] Among them, a t This refers to the action information at time t; The output increment of the thermal power unit at time t, The voltage amplitude at PV node t at time t This represents the amount of wind and solar power curtailed by the renewable energy generating units at time t.

[0015] Furthermore, the state transition function includes:

[0016]

[0017] in, The actual output of the new energy unit at time t; The output of the new energy generating unit at time t; This represents the amount of wind and solar power curtailment at time t for renewable energy units. Let t be the active power of the thermal power unit at the PV node at time t; This represents the power output increment of the thermal power unit at time t; The active power injected into the thermal power unit at node i at time t; The active power injected by the new energy unit at node i at time t; The active power injected into the load at node i at time t; U i，t and U j，t The voltage magnitudes at nodes i and j are respectively represented by the vector U. t G ij and B ij and are the real and imaginary parts of the element in the i-th row and j-th column of the nodal admittance matrix, respectively; δ ij，t The phase angle difference between nodes i and j at time t is denoted by the vector δ. t ; The reactive power injected into the thermal power unit at node i at time t; The active power injected by the new energy unit at node i at time t; The active power injected into the load at node i at time t.

[0018] Furthermore, the reward function includes:

[0019] r k (s t a t s t+1 )=-f c C t -f d D t

[0020] Where r(s) t a t s t+1 ) represents the state s in the current state. t Take action a t The system transitions to a new state s t+1 The return; f c f is the cost scaling factor. d C is the penalty scaling factor; t D represents the total cost of the power system. t This is a cost penalty item;

[0021] The total cost of the power system is calculated using the following formula:

[0022]

[0023] Among them, C G The cost of generating electricity from thermal power units; C represents the active power output of thermal power unit g during time period t; CUR Costs associated with curtailing wind and solar power; This refers to the amount of wind and solar power curtailed by the new energy generator unit r during the time period t. Costs associated with carbon emissions trading; The active power injected into the new energy unit r at time t;

[0024] The cost penalty item is calculated using the following formula:

[0025]

[0026] Where, d t For restricted system variables; d min and d max These correspond to the lower and upper limits, respectively; f c and f d These represent the cost scaling factor and the penalty scaling factor, respectively.

[0027] The power generation cost of the thermal power unit is calculated using the following formula:

[0028]

[0029] Where G is the set of thermal power units; a g b g and c g Δt represents the power generation cost coefficient of thermal power unit g; Δt is the decision time interval.

[0030] The cost of wind and solar curtailment is calculated using the following formula:

[0031]

[0032] Where R is the set of new energy generator sets; η r Cost per unit of wind and solar curtailment;

[0033] The cost of carbon emission trading is calculated using the following formula:

[0034]

[0035] in, For carbon trading prices; E p For carbon quotas; E d Carbon emissions;

[0036] The carbon allowance is calculated using the following formula:

[0037]

[0038] Where h represents the carbon emission allocation per unit of electricity;

[0039] The carbon emissions are calculated using the following formula:

[0040]

[0041] Where μg is the unit coal consumption of the thermal power unit; ρ is the coal emission factor;

[0042] The unit coal consumption of the thermal power unit is calculated using the following formula:

[0043]

[0044] in, x is the upper limit of the active power output of thermal power unit g; g y g and z g This is the unit coal consumption coefficient.

[0045] The parallel distributed flexible actor-evaluator algorithm corresponds to a parallel technical architecture, which includes: a main network consisting of a global actor network, N local actor networks and an evaluator network, a target network consisting of a target global actor network and a target evaluator network, and a global experience replay pool.

[0046] The offline training of the agent corresponding to the real-time low-carbon economic dispatch model of the power system based on the parallel distributed flexible actor-evaluator algorithm includes:

[0047] Step 1: Generate the same number of simulators for the low-carbon economic dispatch model of the power system as the local action network, and distribute the generated simulators to each local action network in a one-to-one ratio.

[0048] Step 2: Initialize the evaluator network with random network parameters w. and target evaluator network Network parameters; initialize the global actor network τ with random network parameters θ. θ Target Global Action Network And the network parameters of N local actor networks; initialize the global experience replay pool R; where the network parameters of the N local actor networks are θ1, θ2, ... θ N The network parameters of the target evaluator network are w. - The network parameters of the target global actor network are θ. - .

[0049] Step 3: Set the number of rounds e = 1;

[0050] Step 4: For the e-th round, obtain the current state information;

[0051] Step 5: Set the time step t = 1;

[0052] Step 6: The N local actor networks interact with the corresponding simulator to perform parallel sampling, obtain the sample set, and store the sample set in the global experience replay pool R.

[0053] Step 7: Draw N samples from R;

[0054] Step 8: Update the evaluator network parameters w according to the first loss function;

[0055] Step 9: Update the global actor network parameters θ according to the second loss function;

[0056] Step 10: Update the entropy regularization coefficient α according to the third loss function;

[0057] Step 11: Parameter w of the target evaluator network - and target global actor network parameters θ - Update;

[0058] Step 12: Synchronize the parameters θ of the global actor network to the parameters θ1, θ2, ... θ of all local actor networks. N In order to update the parameters of the local actor network;

[0059] Step 13: Determine if the current time t is less than the preset time step T. If yes, execute t = t + 1 and jump to step 6. If no, jump to step 14. Where T is the preset time step.

[0060] Step 14: Determine if e is less than the preset number of rounds E. If yes, execute e = e + 1 and jump to step 4. If no, use the current global actor network as the trained agent.

[0061] The first loss function is:

[0062]

[0063] Among them, D KL (·) represents the KL divergence function; For Bellman operators; The output of the target evaluator network is from (s) t a t The mapping from flexible state actions to the distribution of rewards; The output of the target evaluator network is from (s) t a t The mapping from the flexible state action to the distribution of the reward; Z π (s t at ) is in state s t Perform action a t The flexible state action to reward; c is a constant; R is the data collected by the policy in the past; θ - Network parameters for the target global actor network; The action policy output by the target global actor network;

[0064] The evaluator network parameters w are updated using the following formula:

[0065]

[0066] Where w represents the network parameters of the evaluator network; λ Z J is the learning rate of the evaluator network; J is calculated using the following formula. Z (w):

[0067] The second loss function is:

[0068]

[0069] Among them, Q w (s t a t ) represents the expected value of a state-action pair; π θ This is the action policy output by the global actor network; here, reparameterization techniques can be used to modify action a. t The conversion is performed as follows:

[0070] The parameters θ of the global actor network are updated using the following formula:

[0071]

[0072] Where μ and σ represent the mean and variance of the global actor network output, respectively;

[0073]

[0074] Where θ represents the network parameters of the global actor network; λ π J is the learning rate of the global actor network; J is calculated using the following formula. π (θ):

[0075] The third loss function is:

[0076]

[0077] Where H0 is the target value;

[0078] The entropy regularization coefficient α is updated using the following formula:

[0079]

[0080] Where, λ α The learning rate for the entropy regularization coefficients is calculated using the following formula: α (α):

[0081] The target evaluator network parameters are updated using the following formula:

[0082] w - =τ w w+(1-τ w )w -

[0083] Where, τ w Update parameters for the target evaluator network; w - For the network parameters of the target evaluator network;

[0084] The target global actor network parameters are updated using the following formula:

[0085] θ - =τ θ θ+(1-τ θ )θ -

[0086] Where, τ θ θ represents the update parameters of the target global actor; θ represents the network parameters of the global actor network.

[0087] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments.

[0088] One embodiment of the present invention provides a real-time low-carbon economic dispatching device for power systems based on a parallel distributed flexible agent-evaluator, comprising: a model building module, an agent offline training module, and an agent deployment online dispatching module;

[0089] The model construction module is used to acquire a real-time low-carbon economic dispatch model for the power system based on a Markov decision process. Specifically, it constructs the state space of the Markov decision process based on the active power output and reactive power output of thermal power units at time t-1, the voltage amplitude and phase angle of all nodes at time t-1, the predicted active power output and predicted reactive power output of renewable energy units at time t, the predicted active load and predicted reactive load of load nodes at time t, and the predicted reactive load of load nodes at time t. It constructs the action space based on the output increment of thermal power units at PV nodes at time t, the voltage amplitude of PV nodes at time t, and the amount of wind and solar power curtailment at time t. Finally, it constructs the action space based on the given current state s. t Actions taken (a) t Below, and transition to the new state st+1 Construct a state transition function; construct a reward function based on the power generation cost of thermal power units, the cost of wind and solar curtailment, the cost of carbon emission trading, and cost penalty terms;

[0090] The offline training module for the intelligent agent is used to perform offline training on the intelligent agent corresponding to the real-time low-carbon economic dispatch model of the power system according to the parallel distributed flexible actor-evaluator algorithm.

[0091] The intelligent agent is deployed to the online scheduling module, which integrates the offline trained intelligent agent into the power system scheduling equipment, so that the power system scheduling equipment can generate a scheduling plan for each generator set in the power system based on real-time status information, and schedule the generator sets in the power system according to the scheduling plan.

[0092] Based on the above method embodiments, the present invention provides corresponding electronic device embodiments.

[0093] An embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the real-time low-carbon economic dispatch method for power systems based on parallel distributed flexible actuator-evaluator as described in any of the above-described method embodiments.

[0094] Based on the above method embodiments, the present invention provides corresponding storage medium embodiments.

[0095] One embodiment of the present invention provides a storage medium storing a computer program thereon, which, when executed by a processor, can implement the real-time low-carbon economic dispatch method for power systems based on a parallel distributed flexible actor-evaluator as described in any of the above-described method embodiments.

[0096] Compared with the prior art, the present invention has the following beneficial effects:

[0097] This invention provides a method, apparatus, electronic device, and storage medium for real-time low-carbon economic dispatching of power systems. The method constructs a real-time low-carbon economic dispatching model for power systems based on Markov decision processes, utilizing data from large-scale renewable energy and carbon trading mechanisms jointly participating in real-time optimized dispatching of the power system. Based on this model, a parallel distributed flexible actor-evaluator algorithm is proposed to offline train agents to learn effective low-carbon economic dispatching strategies. The trained agents are then integrated into a power system dispatching device for online application, determining dispatching schemes for each generating unit based on real-time system status information.

[0098] Based on the actual needs of real-time low-carbon economic dispatching of power systems, this invention efficiently processes the continuous action space of the dispatching problem through a flexible actor-evaluator; and improves the flexible actor-evaluator by utilizing a distributed flexible strategy iterative framework and parallel technology, thereby improving the solution speed and accuracy. Attached Figure Description

[0099] Figure 1 This is a flowchart illustrating a real-time low-carbon economic dispatch method for power systems based on a parallel distributed flexible actor-evaluator, provided by an embodiment of the present invention.

[0100] Figure 2 This is a schematic diagram of the specific training iteration process of the parallel distributed flexible actor-evaluator algorithm provided in an embodiment of the present invention.

[0101] Figure 3 This is a schematic diagram of a real-time low-carbon economic dispatching structure for a power system based on a parallel distributed flexible actor-evaluator, provided by an embodiment of the present invention. Detailed Implementation

[0102] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0103] like Figure 1 As shown, an embodiment of the present invention provides a real-time low-carbon economic dispatch method for power systems based on a parallel distributed flexible actor-evaluator, which includes at least the following steps:

[0104] Step S1: Construct a real-time low-carbon economic dispatch model for the power system based on a Markov decision process. Specifically, the state space of the Markov decision process is constructed based on the active power output and reactive power output of thermal power units at time t-1, the voltage amplitude and phase angle of all nodes at time t-1, the predicted active power output and predicted reactive power output of renewable energy units at time t, the predicted active load and predicted reactive load of load nodes at time t, and the predicted reactive load of load nodes at time t. The action space is constructed based on the output increment of thermal power units at PV nodes at time t, the voltage amplitude of PV nodes at time t, and the amount of wind and solar power curtailment at time t. The action space is constructed based on the given current state s. t Actions taken (a) t Below, and transition to the new state s t+1Construct a state transition function; construct a reward function based on the power generation cost of thermal power units, the cost of wind and solar curtailment, the cost of carbon emission trading, and cost penalty terms;

[0105] In a preferred embodiment, the state space includes:

[0106]

[0107] Among them, s t This refers to the state information at time t; This represents the active power output of the thermal power unit at time t-1. For the reactive power output of the thermal power unit at time t-1, U t-1 For the voltage magnitude and δ at time t-1 of all nodes t-1 Let be the phase angle of all nodes at time t-1. For the prediction of active power output of new energy generating units at time t, For the predicted reactive power output of the new energy unit at time t, For the predicted active power load at load node t, This represents the predicted reactive load at load node t.

[0108] The action space includes:

[0109]

[0110] Among them, a t This refers to the action information at time t; The output increment of the thermal power unit at time t, The voltage amplitude at PV node t at time t This represents the amount of wind and solar power curtailed by the renewable energy generating units at time t.

[0111] The state transition function includes:

[0112]

[0113] in, The actual output of the new energy unit at time t; The output of the new energy generating unit at time t; This represents the amount of wind and solar power curtailment at time t for renewable energy units. Let t be the active power of the thermal power unit at the PV node at time t; This represents the power output increment of the thermal power unit at time t; The active power injected into the thermal power unit at node i at time t; The active power injected by the new energy unit at node i at time t; The active power injected into the load at node i at time t; U i，t and Uj，t The voltage magnitudes at nodes i and j are respectively represented by the vector U. t G ij and B ij and are the real and imaginary parts of the element in the i-th row and j-th column of the nodal admittance matrix, respectively; δ ij，t The phase angle difference between nodes i and j at time t is denoted by the vector δ. t ; The reactive power injected into the thermal power unit at node i at time t; The active power injected by the new energy unit at node i at time t; The active power injected into the load at node i at time t.

[0114] The reward function includes:

[0115] r t (s t a t s t+1 )=-f c C t -f d D t

[0116] Where r(s) t a t s t+1 ) represents the state s in the current state. t Take action a t The system transitions to a new state s t+1 The return; f c f is the cost scaling factor. d C is the penalty scaling factor; t D represents the total cost of the power system. t This is a cost penalty item;

[0117] The total cost of the power system is calculated using the following formula:

[0118]

[0119] Among them, C G The cost of generating electricity from thermal power units; C represents the active power output of thermal power unit g during time period t; CUR Costs associated with curtailing wind and solar power; This refers to the amount of wind and solar power curtailed by the new energy generator unit r during the time period t. Costs associated with carbon emissions trading; The active power injected into the new energy unit r at time t;

[0120] The cost penalty item is calculated using the following formula:

[0121]

[0122] Where, d t For restricted system variables; d min and d max These correspond to the lower and upper limits, respectively; f c and f d These represent the cost scaling factor and the penalty scaling factor, respectively.

[0123] The power generation cost of the thermal power unit is calculated using the following formula:

[0124]

[0125] Where G is the set of thermal power units; a g b g and c g Δt represents the power generation cost coefficient of thermal power unit g; Δt is the decision time interval.

[0126] The cost of wind and solar curtailment is calculated using the following formula:

[0127]

[0128] Where R is the set of new energy generator sets; η r Cost per unit of wind and solar curtailment;

[0129] The cost of carbon emission trading is calculated using the following formula:

[0130]

[0131] in, For carbon trading prices; E p For carbon quotas; E d Carbon emissions;

[0132] The carbon allowance is calculated using the following formula:

[0133]

[0134] Where h represents the carbon emission allocation per unit of electricity;

[0135] The carbon emissions are calculated using the following formula:

[0136]

[0137] Where, μ g ρ represents the unit coal consumption of a thermal power unit; ρ is the coal emission factor.

[0138] The unit coal consumption of the thermal power unit is calculated using the following formula:

[0139]

[0140] in, x is the upper limit of the active power output of thermal power unit g; g y g and z g This is the unit coal consumption coefficient.

[0141] Step S2: Train the agent corresponding to the real-time low-carbon economic dispatch model of the power system offline according to the parallel distributed flexible actor-evaluator algorithm.

[0142] In one embodiment, the parallel distributed flexible actor-evaluator algorithm corresponds to a parallel technical architecture, which includes: a main network consisting of a global actor network, N local actor networks and an evaluator network, a target network consisting of a target global actor network and a target evaluator network, and a global experience replay pool.

[0143] like Figure 2 As shown, an embodiment of the present invention provides specific training iteration steps for the parallel distributed flexible actor-evaluator algorithm. The step of offline training of the agent corresponding to the real-time low-carbon economic dispatch model of the power system according to the parallel distributed flexible actor-evaluator algorithm includes:

[0144] Step 1: Generate the same number of simulators for the low-carbon economic dispatch model of the power system as the local action network, and distribute the generated simulators to each local action network in a one-to-one ratio.

[0145] Step 2: Initialize the network parameters of the evaluator network and the target evaluator network with random network parameters w; initialize the network parameters of the global actor network, the target global actor network, and N local actor networks with random network parameters θ; initialize the global experience replay pool R;

[0146] Step 3: Set the number of rounds e = 1;

[0147] Step 4: For the e-th round, obtain the current state information;

[0148] Step 5: Set the time step t = 1;

[0149] Step 6: The N local actor networks interact with the corresponding simulator to perform parallel sampling, obtain the sample set, and store the sample set in the global experience replay pool R.

[0150] Step 7: Draw N samples from R;

[0151] Step 8: Update the evaluator network parameters w according to the first loss function;

[0152] Step 9: Update the global actor network parameters θ according to the second loss function;

[0153] Step 10: Update the entropy regularization coefficient α according to the third loss function;

[0154] Step 11: Update the target evaluator network parameters and the target global actor network parameters;

[0155] Step 12: Synchronize the parameters θ of the global actor network to the parameters of all local actor networks to update the parameters of the local actor networks.

[0156] Step 13: Determine if the current time t is less than the preset time step T. If yes, execute t = t + 1 and jump to step 6. If no, jump to step 14. Where T is the preset time step.

[0157] Step Fourteen: Determine if e is less than the preset number of rounds E. If yes, execute e = e + 1 and jump to Step Four. If no, use the current global actor network as the trained agent. It should be noted that the agents used for scheduling subsequently refer only to the finally trained global actor network. Other networks are mainly used to assist in updating the global actor network. Once trained, other networks do not participate in the actual scheduling task; only the global actor network needs to be retained for integration into the scheduling system.

[0158] It should be noted here that the first loss function is:

[0159]

[0160] Among them, D KL (·) represents the KL divergence function; For Bellman operators; The output of the target evaluator network is from (s) t a t The mapping from flexible state actions to the distribution of rewards; The output of the target evaluator network is from (s) t a t The mapping from the flexible state action to the distribution of the reward; Z π (s t a t ) is in state s t Perform action a t The flexible state action to reward; c is a constant; R is the data collected by the policy in the past; θ - Network parameters for the target global actor network; The action policy output by the target global actor network;

[0161] The evaluator network parameters w are updated using the following formula:

[0162]

[0163] Where w represents the network parameters of the evaluator network; λ Z J is the learning rate of the evaluator network; J is calculated using the following formula. Z (w):

[0164] The second loss function is:

[0165]

[0166] Among them, Q w (s t a t ) represents the expected value of a state-action pair; π θ This is the action policy output by the global actor network; here, reparameterization techniques can be used to modify action a. t The conversion is performed as follows:

[0167] The parameters θ of the global actor network are updated using the following formula:

[0168]

[0169] Where μ and σ represent the mean and variance of the global actor network output, respectively;

[0170]

[0171] Where θ represents the network parameters of the global actor network; λ π J is the learning rate of the global actor network; J is calculated using the following formula. π (θ):

[0172] The third loss function is:

[0173]

[0174] Where H0 is the target value;

[0175] The entropy regularization coefficient α is updated using the following formula:

[0176]

[0177] Where, λ α The learning rate for the entropy regularization coefficients is calculated using the following formula: α (α):

[0178] The target evaluator network parameters are updated using the following formula:

[0179] w - =τ w w+(1-τ w )w -

[0180] Where, τ w Update parameters for the target evaluator network; w - For the network parameters of the target evaluator network;

[0181] The target global actor network parameters are updated using the following formula:

[0182] θ - =τ θ θ+(1-τ θ )θ -

[0183] Where, τ θ θ represents the update parameters of the target global actor; θ represents the network parameters of the global actor network.

[0184] It should be noted that deep reinforcement learning generally suffers from slow training convergence speed. Therefore, this invention introduces parallel technology, sharing the experience collected by multiple agents exploring multiple environments, and fully utilizing the computing resources of multi-core CPUs and GPUs to further improve training speed. The agents are local actor networks, and the environment is a simulation model of low-carbon economic dispatching of a power system. All local actor networks have the same network parameters, and all simulation models have the same parameters. Because the local actor networks are set with different random seeds, their output action policies differ, thus achieving the goal of sharing the experience collected by multiple agents exploring multiple environments and fully utilizing the computing resources of multi-core CPUs and GPUs to further improve training speed.

[0185] Furthermore, in the flexible actor-evaluator algorithm, the optimal maximum entropy policy is learned by alternating between flexible policy evaluation and flexible policy improvement. The flexible policy evaluation process calculates the expected value of the current policy after choosing an action in the current state. The flexible policy improvement process refines the policy based on the state-action pair value evaluation results, making the action selection under the current policy more optimal. The expected value of the state-action pair is defined as follows:

[0186]

[0187] in, Indicates from s t The initial entropy increase cumulative return; γ is the discount factor.

[0188] Unlike the flexible actor-evaluator algorithm, which only considers the expected value of state-action pairs, the distributed flexible policy iterative framework chooses to directly learn the distribution of rewards for flexible state-action pairs, taking into account the randomness of cumulative rewards. This can alleviate the problem of overestimating the state-action value of the algorithm and improve training speed and decision accuracy.

[0189] The flexible state-action pair reward under policy π can be represented as follows:

[0190]

[0191] Due to the stochastic policy, reward function, and state transition function, flexible state-action pairs with reward Z. π (s t a t This is typically a random variable, and its relationship with the expected value of a state-action pair is as follows:

[0192] Q(s t a t )=E[Z π (s t a t )]

[0193] Z π (Z π (s t a t )s t a t ): S×A→P(Z π (s t a t )) is defined as from (s t a t The mapping from the action-reward distribution of the flexible state to the distribution of the flexible state action-reward distribution is called the flexible state action-reward distribution or distribution value function. Within the maximum entropy framework, the distribution variable of the Bellman operator can be derived as follows:

[0194]

[0195] In the formula It is the Bellman operator. This indicates that two random variables follow the same probability law.

[0196] Assumption in yes The distribution of the distribution. Therefore, in the process of flexible policy improvement, the policy can be updated by minimizing the distribution distance between the Bellman operator and the current distribution, as follows:

[0197]

[0198] In the formula, d represents the metric for the distance between two distributions; for ease of calculation, many practical deep reinforcement learning algorithms use the Kullback-Leibler (KL) divergence as the metric, denoted as D. KL .

[0199] Step S3: Integrate the offline trained agent into the power system dispatching equipment so that the power system dispatching equipment can generate a dispatching scheme for each generator set in the power system based on real-time status information, and dispatch the generator sets in the power system according to the dispatching scheme.

[0200] In practice, the trained agent is integrated into the power system dispatching device and put into online application. The agent reads the real-time status information of the system, including the active and reactive power output of thermal power units at the previous moment, the voltage amplitude of all nodes at the previous moment, the phase angle of all nodes at the previous moment, the ultra-short-term predicted active power output of new energy units at the current moment, the ultra-short-term predicted active load of all nodes at the current moment, and the ultra-short-term predicted reactive load of all nodes at the current moment.

[0201] Based on the real-time status information of the system, the intelligent agent makes decisions on the action information of each generator unit. The action information includes: the output increment of the thermal power unit at the PV node at the current moment, the voltage amplitude of the PV node at the current moment, and the amount of wind and solar curtailment of the new energy units at the current moment.

[0202] Based on the operation information of each generator set, a power system dispatching plan is formed.

[0203] Specifically, after the offline training phase, the trained global agent network will be integrated into the power system's dispatch system for real-time online solution of low-carbon economic dispatch strategies. First, state information is constructed based on ultra-short-term forecasts of renewable energy sources and loads, and then input into the dispatch system. Subsequently, the agents in the dispatch system make online decisions regarding the actual output of each generating unit based on the state information. Next, the dispatch scheme is applied to the actual power system environment, causing changes in various state variables, which are then fed back to the dispatch system at the next time step. This process is repeated until full-day dispatch is completed.

[0204] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments.

[0205] like Figure 3 As shown, an embodiment of the present invention provides a real-time low-carbon economic dispatching device for power systems based on a parallel distributed flexible agent-evaluator, comprising: a model building module, an offline training module for intelligent agents, and an online dispatching module for intelligent agents;

[0206] The model construction module is used to acquire a real-time low-carbon economic dispatch model for the power system based on a Markov decision process. Specifically, it constructs the state space of the Markov decision process based on the active power output and reactive power output of thermal power units at time t-1, the voltage amplitude and phase angle of all nodes at time t-1, the predicted active power output and predicted reactive power output of renewable energy units at time t, the predicted active load and predicted reactive load of load nodes at time t, and the predicted reactive load of load nodes at time t. It constructs the action space based on the output increment of thermal power units at PV nodes at time t, the voltage amplitude of PV nodes at time t, and the amount of wind and solar power curtailment at time t. Finally, it constructs the action space based on the given current state s. t Actions taken (a) t Below, and transition to the new state s t+1 Construct a state transition function; construct a reward function based on the power generation cost of thermal power units, the cost of wind and solar curtailment, the cost of carbon emission trading, and cost penalty terms;

[0207] The offline training module for the intelligent agent is used to perform offline training on the intelligent agent corresponding to the real-time low-carbon economic dispatch model of the power system according to the parallel distributed flexible actor-evaluator algorithm.

[0208] The intelligent agent is deployed to the online scheduling module, which integrates the offline trained intelligent agent into the power system scheduling equipment, so that the power system scheduling equipment can generate a scheduling plan for each generator set in the power system based on real-time status information, and schedule the generator sets in the power system according to the scheduling plan.

[0209] It should be noted that the embodiments of the device described above correspond to the embodiments of the present invention described above, and can realize the real-time low-carbon economic dispatch method for power systems based on parallel distributed flexible actuator-evaluator as described in any one of the above embodiments of the present invention. Furthermore, the embodiments of the device described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without creative effort.

[0210] Based on the above-described method embodiments of the present invention, a corresponding embodiment of an electronic device is provided.

[0211] An embodiment of the present invention provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the real-time low-carbon economic dispatch method for power systems based on parallel distributed flexible actuator-evaluator as described in any one of the present invention, or, when the processor executes the computer program, it implements the functions of each module in the above-described device embodiments.

[0212] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the terminal device.

[0213] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0214] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0215] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0216] Based on the above method embodiments, the present invention provides corresponding storage medium embodiments;

[0217] Another embodiment of the present invention provides a storage medium including a stored computer program, wherein, when the computer program is running, the device where the storage medium is located executes any of the above-described power system real-time low-carbon economic dispatch methods based on parallel distributed flexible actor-judge of the present invention.

[0218] The aforementioned storage medium is a computer-readable storage medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0219] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0220] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A real-time low-carbon economic dispatching method for power system based on parallel distributed flexible actor-evaluator, characterized in that: include: A real-time low-carbon economic dispatch model for the power system is constructed based on the Markov decision process; wherein, the state space of the Markov decision process is constructed according to the active output of the thermal power unit at time t-1, the reactive output of the thermal power unit at time t-1, the voltage amplitude of all nodes at time t-1, the phase angle of all nodes at time t-1, the predicted active output of the new energy unit at time t, the predicted reactive output of the new energy unit at time t, the predicted active load of the load node at time t, and the predicted reactive load of the load node at time t; the action space is constructed according to the output increment of the thermal power unit on the PV node at time t, the voltage amplitude of the PV node at time t, and the amount of wind and solar power abandoned by the new energy unit at time t; according to the given current state s t 、Action taken a t Next, and transfer to the new state s t+1 Construct a state transfer function; construct a reward function based on the power generation cost of thermal power units, the cost of wind and solar power abandonment, the carbon emission rights trading cost and the cost penalty item; Offline training of an intelligent agent corresponding to the real-time low-carbon economic dispatch model of the power system is performed according to a parallel distributed flexible actor-judge algorithm; The intelligent agent that has completed offline training is integrated into the power system dispatching equipment, so that the power system dispatching equipment generates a dispatching plan for each generator set in the power system according to real-time status information, and dispatches the generator sets in the power system according to the dispatching plan.

2. The real-time low-carbon economic dispatching method for power system based on parallel distributed flexible actor-evaluator as claimed in claim 1, characterized in that: The state space includes: Among them, s t is the status information at time t; is the active power output of the thermal power unit at time t-1; is the reactive power output of the thermal power unit at time t-1, U t-1 is the voltage amplitude of all nodes at time t-1, δ t-1 is the phase angle of all nodes at time t-1, is the predicted active power output of the new energy unit at time t, is the predicted reactive power output of the new energy unit at time t, is the predicted active load at the load node at time t, is the predicted reactive load at the load node at time t.

3. The real-time low-carbon economic dispatching method for power system based on parallel distributed flexible actor-evaluator as claimed in claim 2, characterized in that: The action space includes: Among them, a t is the action information at time t; is the output increment of the thermal power unit at time t, is the voltage amplitude of the PV node at time t, is the amount of wind and solar power abandoned by the new energy unit at time t.

4. The real-time low-carbon economic dispatching method for power system based on parallel distributed flexible actor-evaluator as claimed in claim 3, characterized in that: The state transfer function comprises: in, is the actual output of the new energy unit at time t; is the output of the new energy unit at time t; is the amount of wind and solar power abandoned by the new energy unit at time t; is the active power of the thermal power unit at the PV node at time t; is the output increment of the thermal power unit at time t; is the active power injected by the thermal power unit at node i at time t; is the active power injected by the new energy unit at node i at time t; is the active power injected by the load at node i at time t; U i，t and U j，t are the voltage amplitudes at nodes i and j respectively, forming a vector U t ; G ij and B ij and are the real and imaginary parts of the elements in the i-th row and j-th column of the node admittance matrix, respectively; δ ij，t is the phase angle difference between nodes i and j at time t, forming a vector δ t ; is the reactive power injected by the thermal power unit at node i at time t; is the active power injected by the new energy unit at node i at time t; is the active power injected by the load at node i at time t.

5. The real-time low-carbon economic dispatching method for power system based on parallel distributed flexible actor-evaluator as claimed in claim 4, characterized in that: The reward function includes: r t (s t ,a t ,s t+1 )=-f c C t -f d D t Among them, r(s t , a t ,s t+1 ) is in the current state s t Take action a t , the system transfers to the new state s t+1 The return of c is the cost scaling factor; f d is the penalty scaling factor; C t is the total cost of the power system; D t is the cost penalty item; The total cost of the power system is calculated by the following formula: Among them, C G The cost of electricity generation for thermal power units; is the active power output of thermal power unit g during period t; C CUR Cost of curtailing wind and solar power; is the amount of wind and solar power abandoned by the new energy generator r in period t; The cost of carbon emission rights trading; is the active power injected by the new energy unit r at time t; The cost penalty term is calculated by the following formula: Among them, d t is a restricted system variable; d min and d max Corresponding to the lower limit and upper limit respectively; f c and f d Represent the cost scaling factor and penalty scaling factor respectively; The power generation cost of the thermal power unit is calculated by the following formula: Among them, G is the collection of thermal power units; a g , b g and c g is the power generation cost coefficient of thermal power unit g; Δt is the decision time interval; The cost of curtailing wind and solar power is calculated by the following formula: Where R is the set of new energy generating units; η r is the unit cost of wind and solar power abandonment; The carbon emission rights transaction cost is calculated by the following formula: in, is the carbon trading price; E p is the carbon quota; E d for carbon emissions; The carbon quota is calculated by the following formula: Among them, h is the carbon emission allocation quota per unit of electricity; The carbon emissions are calculated using the following formula: Among them, μ g is the unit coal consumption of thermal power units; ρ is the coal emission factor; The unit coal consumption of the thermal power unit is calculated by the following formula: in, is the upper limit of active output of thermal power unit g; x g ,y g and z g is the unit coal consumption coefficient.

6. The real-time low-carbon economic dispatch method for power system based on parallel distributed flexible actor-evaluator according to claim 5, wherein the parallel distributed flexible actor-evaluator algorithm corresponds to a parallel technical architecture, and the parallel technical architecture includes: A main network consisting of a global actor network, N local actor networks and a critic network, a target network consisting of a target global actor network and a target critic network, and a global experience replay pool; The offline training of the intelligent agent corresponding to the real-time low-carbon economic dispatch model of the power system according to the parallel distributed flexible actor-judge algorithm includes: Step 1: Generate the same number of simulators of the low-carbon economic dispatch model of the power system as the number of local actor networks, and distribute the generated simulators to each local actor network in a one-to-one ratio; Step 2: Initialize the network parameters of the judge network and the target judge network with random network parameters w; initialize the network parameters of the global actor network, the target global actor network and N local actor networks with random network parameters θ; initialize the global experience replay pool R; Step 3: Set the number of rounds e=1; Step 4: For the e-th round, obtain the current state information; Step 5: Set time step t=1; Step 6: N local actor networks interact with the corresponding simulators to perform parallel sampling to obtain a sample set, and store the sample set in the global experience replay pool R; Step 7: Extract N samples from R; Step 8: Update the judge network parameter w according to the first loss function; Step 9: Update the global actor network parameter θ according to the second loss function; Step 10: Update the entropy regularization coefficient α according to the third loss function; Step 11: Update the target evaluator network parameters and the target global actor network parameters; Step 12: Synchronize the parameters θ of the global actor network to the parameters of all local actor networks to update the parameters of the local actor networks; Step 13: determine whether the current time t is less than the preset time step T. If so, execute t=t+1 and jump to step 6. If not, jump to step 14; wherein T is the preset time step; Step 14: Determine whether e is less than the preset number of rounds E at this time. If so, execute e=e+1 and jump to step 4. If not, use the current global actor network as the trained intelligent agent.

7. The real-time low-carbon economic dispatching method for power system based on parallel distributed flexible actor-evaluator as claimed in claim 6, characterized in that: The first loss function is: Among them, D KL (·) is the KL divergence function; is the Bellman operator; The output of the target judge network is (s t , a t ) to the distribution of flexible state-action pairs of rewards; The output of the target judge network is (s t , a t ) to the distribution of flexible state-action pairs of rewards; Z π (s t , a t ) is in state s t Execute action a t The flexible state-action pair return; c is a constant; R is the data collected by the strategy in the past; θ - are the network parameters of the target global actor network; The action strategy output by the target global actor network; The judge network parameter w is updated by the following formula: Where w is the network parameter of the judge network; λ z is the learning rate of the judge network; J is calculated by the following formula z (w): The second loss function is: Among them, Q w (s t , a t ) is the expected value of the state-action pair; π θ is the action strategy output by the global actor network; here we can use the reparameterization technique to t To convert, specifically: The parameters θ of the global actor network are updated by the following formula: Where μ and σ represent the mean and variance of the global actor network output respectively; Where θ is the network parameter of the global actor network; λ π is the learning rate of the global actor network; J is calculated by the following formula π (θ): The third loss function is: Among them, H0 is the target value; The entropy regularization coefficient α is updated by the following formula: Among them, λ α is the learning rate of the entropy regularization term coefficient, and L is calculated by the following formula α (α): The target judge network parameters are updated using the following formula: w - =t w w+(1-τ w )w - Among them, τ w is the update parameter of the target judge network; w - are the network parameters of the target judge network; The target global actor network parameters are updated using the following formula: i - =t θ θ+(1-τ θ )i - Among them, τ θ are the update parameters of the target global actor; θ are the network parameters of the global actor network.

8. A real-time low-carbon economic dispatching device for power system based on parallel distributed flexible actor-evaluator, characterized in that: include: Model building module, agent offline training module and agent online scheduling module; The model building module is used to obtain a real-time low-carbon economic dispatch model for the power system based on the Markov decision process; wherein, the state space of the Markov decision process is constructed according to the active output of the thermal power unit at time t-1, the reactive output of the thermal power unit at time t-1, the voltage amplitude of all nodes at time t-1, the phase angle of all nodes at time t-1, the predicted active output of the new energy unit at time t, the predicted reactive output of the new energy unit at time t, the predicted active load of the load node at time t, and the predicted reactive load of the load node at time t; the action space is constructed according to the output increment of the thermal power unit at the PV node at time t, the voltage amplitude of the PV node at time t, and the amount of wind and solar power abandoned by the new energy unit at time t; according to the given current state s t 、Action taken a t Next, and transfer to the new state s t+1 Construct a state transfer function; construct a reward function based on the power generation cost of thermal power units, the cost of wind and solar power abandonment, the carbon emission rights trading cost and the cost penalty item; The intelligent agent offline training module is used to perform offline training on the intelligent agent corresponding to the real-time low-carbon economic dispatch model of the power system according to the parallel distributed flexible actor-judge algorithm; The intelligent agent is put into an online scheduling module, which is used to integrate the intelligent agent that has completed offline training into the power system scheduling equipment, so that the power system scheduling equipment can generate a scheduling plan for each generator set in the power system according to real-time status information, and schedule the generator sets in the power system according to the scheduling plan.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it can implement the real-time low-carbon economic dispatch method of the power system based on the parallel distributed flexible actor-evaluator as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can implement the real-time low-carbon economic dispatch method of the power system based on the parallel distributed flexible actor-evaluator as described in any one of claims 1 to 7.

Images