Power distribution system demand side response potential evaluation method based on robust reinforcement learning

Abstract

The invention provides a power distribution system demand side response potential evaluation method based on robust reinforcement learning, and belongs to the technical field of power grid dispatching information systems. The method aims at coping with the difference between a simulation power distribution system simulation model and a real power grid environment, attempts to disturb and destroy intelligent scheduling by establishing an aggressive adversarial reinforcement learning agent for the power system, and thus applies external disturbance to the power system. Through alternate training of the intelligent agent and the power system dispatching intelligent agent, a zero-sum game based on a Maximin strategy is formed, and robustness of the power system intelligent agent to external interference and power grid dispatching safety under external attacks are gradually enhanced. According to the method, a bridge between the simulation model and a real power grid environment is established, the technical advantages of robust reinforcement learning are utilized, dynamic optimization of the power system is achieved, challenges such as new energy access and load fluctuation are dealt with, the robustness and reliability of the power distribution system are improved, and safe operation of the power system is ensured.

Description

Technical Field

[0001] The invention relates to a method for evaluating the demand-side response potential of a distribution system based on robust reinforcement learning, and belongs to the technical field of power grid dispatching information. Background Art

[0002] The power system is an important infrastructure of modern society. However, traditional distribution systems often face many challenges, including grid load uncertainty, new energy integration, and the popularization of electric vehicles. Traditional power system optimization methods often have difficulty in coping with these complex and dynamic challenges, and the system robustness and efficiency are insufficient.

[0003] As an emerging intelligent optimization technology, robust reinforcement learning provides new ideas for solving complex problems in power systems. Traditional reinforcement learning methods usually face the challenge of testing in real environments, and the strategies they learn in simulated environments are difficult to directly translate into real-world applications. Therefore, robust reinforcement learning technology has become a new way to improve the robustness and adaptability of distribution systems. Summary of the invention

[0004] The purpose of this invention is to solve the technical problems that the existing distribution system scheduling and optimization technology based on deep reinforcement learning is not applicable to the generality of actual power scenarios, and creatively proposes a distribution system demand side response potential evaluation method based on robust reinforcement learning.

[0005] The present invention aims to utilize the technical advantages of robust reinforcement learning to achieve dynamic optimization of the power system by establishing a bridge between the simulation model and the actual power grid environment, so as to cope with challenges such as new energy access and load fluctuations, thereby improving the robustness and reliability of the distribution system and ensuring the safe dispatching and operation of the power system.

[0006] The innovations of the present invention include: the proposed method focuses on coping with the difference between the simulation model of the simulated power distribution system and the real power grid environment, and attempts to interfere with and destroy the intelligent dispatching by establishing an attacking adversarial reinforcement learning agent against the power system, thereby imposing external disturbances on the power system. These agents are alternately trained with the power system dispatching agent, forming a zero-sum game based on the Maximin strategy, gradually enhancing the robustness of the power system agent to external interference and the security of the power grid dispatching under external attacks.

[0007] The present invention is implemented by adopting the following technical solutions.

[0008] A demand side response potential evaluation method based on robust reinforcement learning theory includes the following steps:

[0009] Step S1: Using robust reinforcement learning technology, a simulation model of the simulated power distribution system is constructed.

[0010] The purpose is to simulate an attacking intelligent agent in a real power grid environment. The main goal of the intelligent agent is to impose external interference on the power system in an attempt to interfere with and destroy the automated dispatching of the distribution system.

[0011] Among them, the specific method of establishing an attacking intelligent agent that simulates the gap between the distribution system simulation model and the actual power grid environment is as follows:

[0012] Step S1.1: Analyze the distribution system simulation model based on dual-agent zero-sum game and robust reinforcement learning theory;

[0013] Step S1.2: Based on the analysis of step S1.1, an attack adversarial agent is established to simulate the gap between the distribution system simulation model and the actual power grid environment. Its training goal is to impose external disturbances on the power system in an attempt to prevent and destroy the automated dispatching of the distribution system agent.

[0014] Step S2: Establish a Markov simulation environment model for the power distribution system.

[0015] It covers traditional power generation units, distributed power generation units, energy storage devices, industrial, residential power loads and electric vehicle loads. The model is designed to accurately reflect the operation of the real power system.

[0016] Specifically, the steps include:

[0017] Step S2.1: Modeling the uncertain distributed renewable energy generation units, storage and discharge behaviors, coordination between distributed devices, and electric vehicle loads in the distribution system;

[0018] Step S2.2: Taking into full consideration the unit models established in step S2.1, a Markov simulation environment model of the power distribution system is established.

[0019] Step S3: Based on the Markov simulation environment model in step S2, alternate training of the power system dispatching agent and the attacking agent is performed.

[0020] In this process, the two agents compete against each other, forming a two-person zero-sum game multi-agent reinforcement learning model with a Maximin strategy.

[0021] Specifically, the steps include:

[0022] Step S3.1: The active power injection of the generation unit, energy storage unit, load unit, reactive power injection of the load unit, the maximum active and reactive output of the distributed generation unit, and the state of charge of the energy storage unit are modeled as the power system dispatching main agent N in the dual-agent Markov zero-sum game model of the distribution system. 1The adversarial agent N that perturbs the state of the master agent 2 The Markov state parameter.

[0023] Step S3.2: Model the active and reactive reduction values ​​of the distributed generation units and the active and reactive injection values ​​(extraction values) of the energy units (referring to the energy storage units or energy storage devices) as the power system dispatching main agent N in the dual-agent Markov zero-sum game model of the distribution system. 1 The Markov action parameter.

[0024] Step S3.3: The master agent N 1 The action of perturbing the Markov observation state of the power distribution system is modeled as the adversarial agent that perturbs the state of the main agent against N in the dual-agent Markov zero-sum game model of the power distribution system. 2 The Markov action parameter.

[0025] Step S3.4: The energy loss of distributed generation units during output reduction, the energy loss of power lines, and the energy loss of energy storage units during energy storage and discharge are modeled as reference indicators of the reward function of the dual-agent Markov zero-sum game model of the distribution system.

[0026] Step S3.5: In the dual-agent Markov game model of the power distribution system, a reference index of the penalty function for violating the safety constraint is set.

[0027] The goal of this step is to ensure that violations of security regulations are punished during the model learning process to maintain the security of the system.

[0028] Step S4: The Actor-Critic algorithm is used to solve the multi-agent reinforcement learning model of the Maximin strategy proposed in step S3 to improve the robustness of the distribution system and evaluate its impact on the demand-side response potential.

[0029] Specifically, the steps include:

[0030] Step S4.1: Determine the Maximin strategy solution target of the state perturbation-based robust reinforcement learning method;

[0031] Step S4.2: Use the Actor-Critic algorithm to solve the solution objective determined in step S4.1, improve the robustness of the distribution system, and evaluate its impact on the demand-side response potential.

[0032] Beneficial Effects

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] 1. The present invention applies external disturbances to the power system and adopts an attack-type adversarial reinforcement learning agent to improve the robustness of the power system agent to external interference and the security of the system, thereby ensuring the stability and safety of the power grid dispatching operation.

[0035] 2. By combining the Actor-Critic algorithm and the multi-agent reinforcement learning model, the system continuously optimizes the flow distribution while also evaluating the demand-side response potential, providing a more comprehensive evaluation and improvement path for the system.

[0036] 3. This invention uses a two-person zero-sum game multi-agent reinforcement learning model with a Maximin strategy. Our method improves the system's ability to respond to external attacks during the training process, thereby enhancing the security of the power distribution system. This can ensure that the system can still maintain safe operation in the face of various abnormal situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of a flow chart of an embodiment of the present invention.

[0038] Figure 2 Schematic diagram of the overall framework of the Markov game model of an embodiment of the present invention.

[0039] Figure 3 Schematic diagram of Markov simulation environment modeling of a power distribution system according to an embodiment of the present invention.

[0040] Figure 4 1 is a flow chart of the Actor-Critic algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0042] Example

[0043] like Figure 1 As shown, a method for evaluating the demand side response potential of a distribution system based on robust reinforcement learning includes the following steps:

[0044] Step S1: Establish an attacking agent that simulates the gap between the distribution system simulation model and the actual power grid environment. Its training goal is to impose external disturbances on the power system in an attempt to prevent and destroy the automatic dispatching of the distribution system agent.

[0045] Specifically, the steps include:

[0046] Step S1.1: Analyze the distribution system simulation model based on dual-agent zero-sum game and robust reinforcement learning theory.

[0047] Specifically, the dual-agent γ-discounted zero-sum Markov game MDP model is represented as a seven-tuple: (S, A1, A2, P, R, γ, ρ 0 ). Where S represents the state set of the system; A1 and A2 represent a set of continuous actions taken by two agents in the game; P: S×A1×A2×S'→R, P represents the transition probability density function jointly determined by the two agents, S' represents the next state set; r: S×A1×A2→R, r represents the immediate reward value fed back by the environment to the two game agents; R represents a missing or incorrectly filled in, usually a set of reward functions or the reward function itself, which describes the rewards obtained by the agent in different states; ρ 0 Represents the discount factor.

[0048] Let the main agent N 1 The strategy μ is learned after training, against the agent N 2 The strategy ν is learned after training. For the dual-agent γ discounted zero-sum Markov game problem, the immediate reward function r μ,v for:

[0049]

[0050] Where E represents the expected value operator, which is used to calculate the average value under given conditions; a 1 、a 2 They represent the actions taken by the two agents / decision makers in state s according to their respective strategies μ and v; μ(s) and v(s) represent the action choices taken by strategies μ and v in state s respectively; s represents the current state of the system; r(s, a 1 ,a 2 ) means that in state s, when the agent / decision maker takes action a 1 and a 2 After that, the environment returns an immediate reward.

[0051] A two-agent γ-discounted zero-sum Markov game, regarded as the master agent N 1 Maximize the γ discounted reward. At the same time, the adversarial agent N 2 A zero-sum game process that minimizes the γ discounted reward.

[0052] In the robust reinforcement learning setting, there is a set U of uncertainties. Formally, U is expressed as:

[0053] U∈{S,A,R,P}

[0054] The uncertain variables contained in this set may be the state s∈S, action a∈A, reward function r∈R, and transition probability density function P in reinforcement learning.

[0055] The goal of the robust reinforcement learning algorithm is to learn an attack strategy ν through training to maximize the performance of the uncertain variable u∈U, that is, to try to disrupt the performance of the strategy π that maximizes the reward function. In form, it is expressed as the following Maximin robust optimization problem:

[0056]

[0057] Among them, R π (u) is the mathematical expected cumulative return under the strategy π with the uncertain variable u as the independent variable. By Markov modeling the natural uncertainty that occurs in real applications and the errors designed by humans, adversarial attacks on uncertain variables u of different forms are realized. The adversarial training framework driven by the above model is defined as a robust reinforcement learning method, thereby assuming that an attacking adversarial agent conducts adversarial attacks to simulate natural uncertainty and human-designed errors to improve the performance and robustness of the main agent strategy π.

[0058] Step S1.2: Based on the analysis of step S1.1, an attack adversarial agent is established to simulate the gap between the distribution system simulation model and the actual power grid environment. Its training goal is to impose external disturbances on the power system in an attempt to prevent and destroy the automatic dispatching of the distribution system agent.

[0059] like Figure 2 As shown in the figure, the power system dispatching main intelligent agent N is simulated based on the Markov game model of the distribution system demand side response. 1 With adversarial agent N 2 In this model, the establishment of adversarial agents involves simulating the large-scale integration of distributed generation units in a real power grid environment, and conducting adversarial training for the complexity, uncertainty and volatility brought about by these factors. The training goal is to disturb the Markov observation state of the power system in an attempt to interfere with, prevent and destroy the automated safe dispatching of the main agent of the power system.

[0060] Step S2: Establish a Markov simulation environment model of a power distribution system with traditional power generation units, distributed power generation units, storage / discharge devices, industrial and residential power loads, and electric vehicle loads.

[0061] Specifically, the steps include:

[0062] Step S2.1: Model the highly uncertain distributed renewable energy generation units, storage and discharge behaviors, coordination between large-scale distributed devices, and the growing electric vehicle load in the distribution system. Including:

[0063] Power generation unit U G , these units ideally only inject electrical energy into the distribution system:

[0064] U G = {U TG , U DG}

[0065] Among them, U TG It is a traditional power generation unit; U DG A distributed power generation unit.

[0066] To facilitate simulation analysis, the distributed generation units are:

[0067] U DG = {U SDG , U WDG}

[0068] Among them, U SDG and U WDG They represent wind power generation units and photovoltaic power generation units respectively.

[0069] Distributed power storage unit U DES The energy storage unit can both extract and store electrical energy from the power distribution system and inject the stored electrical energy into the power distribution system.

[0070] Load unit U L ,include:

[0071] U L = {U IL , U RL , U EVL}

[0072] Among them, U IL It is the industrial power load unit, U RL is the residential electricity load unit, U EVL It is a movable electric vehicle load unit. Different types of loads have different characteristics. The proportion of industrial load ranks first in the electricity consumption structure and is relatively constant; the residential electricity load has a strong temporal fluctuation; the biggest feature of electric vehicle load is that it is movable.

[0073] The above three types of loads can only draw electrical energy from the distribution system.

[0074] Step S2.2: Taking into full consideration the unit models established in step S2.1, a Markov simulation environment model of the power distribution system is established.

[0075] like Figure 3 The Markov simulation environment of the power distribution system fully considers the highly uncertain distributed renewable energy generation units and power storage behaviors of power users, the coordination between large-scale distributed devices, and the growing use of electric vehicle loads. TGDistributed generation unit U with intermittent generation characteristics DG (including wind power generation unit U SDG , Photovoltaic power generation unit U WDG ), a distributed energy storage unit U that can realize bidirectional energy flow with the distribution system DES , load units U with different load characteristics L (Including industrial power load unit U IL , Residential electricity load unit U RL , Mobile electric vehicle load unit U EVL ).

[0076] Step S3.1: The active power injection and reactive power injection of the generation unit, energy storage unit, and load unit, the maximum active output and reactive output of the distributed generation unit, and the charge state of the energy storage unit are modeled as the power system dispatching main agent N in the dual-agent Markov zero-sum game model of the distribution system. 1 The adversarial agent N that perturbs the state of the master agent 2 The Markov state parameter.

[0077] Among them, the state set S: the power system dispatching main intelligent agent N 1 The adversarial agent N that perturbs the state of the master agent 2 The Markov state s t ∈S, are defined by the following formula:

[0078]

[0079] Where t is the minimum time step in the training process;

[0080] They represent the value i at time t and the value u at time t. g From Collection U G , these values ​​represent the active and reactive reduction values ​​of distributed generation units at a certain point in time; similarly, and represents another set of values ​​for different indicators i at time t, where u des is from the set U DES An element selected from , these values ​​represent the active and reactive injection (extracted values) of the energy storage unit at a certain point in time; and represents another set of indicators for quantity i at time t, where u l is from the set U L An element selected in . and They represent the maximum active and reactive outputs of distributed generation units respectively; Indicates the state of charge (SOC) of the energy storage unit.

[0081] Step S3.2: Model the active and reactive reduction values ​​of distributed generation units and the active and reactive injection / extraction values ​​of energy units as the power system dispatching master agent N in the dual-agent Markov zero-sum game model of the distribution system. 1 The Markov action parameter.

[0082] Action Set A 1 :Power system dispatching main intelligent agent N 1 The Markov action set a 1,t ∈A 1 , defined by the following formula:

[0083]

[0084] in, Represent the active and reactive reduction values ​​of distributed generation units, u dg represents a distributed generation unit; Respectively represent the active and reactive injection (or extraction) values ​​of the energy storage unit, u des Represents an individual or specific type of energy storage unit.

[0085] Step S3.3: dispatch the main intelligent agent N to the power system 1 The action of applying disturbance to the Markov observation state is modeled as the adversarial agent N of the power system in the dual-agent Markov zero-sum game model of the distribution system. 2 The Markov action parameter.

[0086] Action Set A 2 :Adversarial agent N 2 The action is to the main agent N 1 The Markov observation state is perturbed, so its action set A 2 with s defined in step S3.1 t has the same form as the dimensions.

[0087] Step S3.4: The energy loss of distributed generation units during output reduction, the energy loss of power lines, and the energy loss of energy storage units during energy storage and discharge are modeled as reference indicators of the reward function of the dual-agent Markov zero-sum game model of the distribution system.

[0088] Reward function R: For the dual-agent Markov zero-sum game model based on the virtual power plant, the reward functions of the two agents should be opposite to each other. t |∈R is defined as follows:

[0089] |r t |=|ΔE cur +ΔE el +ΔE des |

[0090] The reward function is composed of the energy loss during the operation of the virtual power plant, ΔE cur Represents distributed generation unit U DG Energy loss during output reduction; ΔE des Represents the power line energy loss; ΔE des Represents energy storage unit U DES Energy loss during the process of storing and discharging energy.

[0091] Step S3.5: Model the penalty function reference index for violating the safety constraint of the dual-agent Markov zero-sum game model of the distribution system.

[0092] The power system is an important infrastructure of modern society. Any operation that violates relevant safety constraints during the dispatching process may cause shutdown or permanent damage to some units, resulting in huge economic losses. Therefore, it is necessary to ensure that the applied control decisions are safe to ensure that the operating state of the driving power system does not violate key physical constraints or cause instability or safety issues.

[0093] The reinforcement learning automated scheduling decision-making scheme based on virtual power plants has two most typical safety issues. The penalty set C is used to quantify the penalty feedback for violating the safety scheduling constraints:

[0094] First, the voltage value range of each node B is safely restricted. The node voltage safety constraint is particularly critical to ensure the stable operation of each unit of the entire virtual power plant. The penalty function term c is set using the following formula: 1 ∈C:

[0095]

[0096] Among them, V i is the voltage value of node i, and its upper and lower bounds are Voltage safety constraints.

[0097] Second, there are capacity constraints on the transmission lines in the virtual power plant, that is, the maximum power that can pass through the lines. These safety constraints are set to ensure that the lines do not overheat and cause failures or fire disasters. The penalty function term c is set using the following formula: 2 ∈C:

[0098]

[0099] Among them, S ij is the power value from node i to node j, and the power flow is calculated The line capacity security constraint.

[0100] Step S4: Use the Actor-Critic algorithm to solve the two-player zero-sum game multi-agent reinforcement learning model of the Maximin strategy proposed in step S3, and finally achieve power flow optimization with enhanced robustness of the distribution system.

[0101] Specifically, the steps include:

[0102] Step S4.1: Determine the Maximin strategy solution target of the state perturbation based robust reinforcement learning method.

[0103] This is a reinforcement learning method that combines policy optimization and value function estimation. In this framework, the agent continuously learns how to choose actions in a given state and evaluates the value of these actions. For robust reinforcement learning methods based on state perturbations, the focus is on system robustness and safety.

[0104]

[0105] Among them, Q μ (s,a) represents the behavior value function of selecting action a in state s, V μ (s) is in strategy The value function of the next state s. Represents the distribution of state s according to p μ Sampling is performed and the expected value of the expression is taken. This expected value represents the behavior value function Q under a given strategy μ. μ (s,a) and value function V μ (s) Perform the operation and obtain the expected value.

[0106] The Actor-Critic algorithm continuously optimizes this expected objective function, allowing the agent to learn better strategies. Among them, Actor (policy network) is responsible for learning how to choose actions, and Critic (value network) evaluates the value of choosing the action.

[0107] In the iterative process, the distribution system dispatches the main intelligent agent N 1 The Actor network is responsible for learning the strategy of generating actions and evaluating the value of these actions through the Critic network. 2 In game training, perturbations are also imposed on the observed state of the main agent in order to obtain a better confrontation strategy.

[0108] The Actor-Critic algorithm continuously optimizes the strategy during the iteration process, so that the distribution system dispatching agent can gradually improve its robustness to external disturbances and maintain more robust system security under the same interference conditions. The Actor-Critic algorithm is a reinforcement learning method, the core of which is to train two networks at the same time: Actor and Critic. In this scenario, the Actor is responsible for learning how to choose actions, while the Critic evaluates the value of these actions.

[0109] In the context of power distribution systems, N 1 (i.e. Actor) is responsible for formulating the dispatch strategy of the power system. It constantly tries different behaviors (i.e. different dispatch strategies) and adjusts its behavior strategy based on the reward signal obtained. The reward signal here can be the energy consumption of the system under different strategies or other evaluation indicators.

[0110] N 2 (Critic) acts as an adversarial agent and observes N by applying perturbations 1 state to counter N 1 Its task is to evaluate N 1 The value of the strategy adopted in a specific state is thus N 1 Provide guidance or feedback.

[0111] This Actor-Critic algorithm continuously optimizes the objective function, that is, maximizing N 1 In the power distribution system, this means N 1 We will continue to learn better scheduling strategies to adapt to different external environmental interferences and maintain a more robust system operation state under similar interferences.

[0112]

[0113] Among them, R μ represents the expected return of the strategy μ; T represents the time step, and represents the cumulative return in the time series 0 to T.

[0114] At the same time, the penalty target form expectation is minimized as follows:

[0115]

[0116] Attacking adversarial agent N 2 The goal is to learn the optimal network parameters θ′ of the perturbation strategy ν through the deep reinforcement learning algorithm, 1The observed state is disturbed, so that its output energy loss is large and it is easy to violate the scheduling decision of safety constraints. In contrast to the main agent, the reward target form of the above formula is minimized, while the penalty target form of the following formula is maximized.

[0117] In essence, the two agents constitute a dual-agent zero-sum game reinforcement learning model:

[0118]

[0119]

[0120] The Actor-Critic method combines the policy network and the value network. In terms of policy updates, it can use algorithms such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO). These algorithms limit the update range, that is, constrain the difference between the new policy and the old policy, to ensure that the policy adjustment is not too large. This can stabilize training and reduce instability caused by policy updates.

[0121] In the Critic part, the value network is used to evaluate the value of the state, which helps guide the update direction of the Actor strategy network. Actor-Critic will adjust the generated action strategy based on the state value information given by the Critic, making the selected action more likely to achieve a better state:

[0122]

[0123] in, V represents the loss function of the Critic network, which is the square loss of the difference between the actual reward and the estimated reward; ω represents the gradient operator with respect to the parameter ω; s t Indicates the current state; V ω (s t ) represents the Critic network's response to state s t estimated value.

[0124] is equivalent to:

[0125]

[0126] in, represents the gradient operator with respect to the parameter ω.

[0127] The Lagrange multiplier β is an iteratively updated parameter. Its function is to weigh the trade-off between reward and penalty in the decision strategy μ parameter θ. The specific process of updating the Actor-Critic algorithm is shown in Algorithm 1:

[0128]

[0129]

[0130] Figure 4 4 is a flow chart of the Actor-Critic algorithm of an embodiment of the present invention.

[0131] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A demand-side response potential evaluation method based on robust reinforcement learning theory, characterized in that: The following steps are involved: Step S1: using robust reinforcement learning technology to construct a simulation model of the simulated power distribution system; Among them, the specific method of establishing an attacking intelligent agent that simulates the gap between the distribution system simulation model and the actual power grid environment is as follows: Step S1.1: Analyze the distribution system simulation model based on dual-agent zero-sum game and robust reinforcement learning theory; Step S1.2: Based on the analysis of step S1.1, an attack adversarial agent is established to simulate the gap between the distribution system simulation model and the actual power grid environment. Its training goal is to impose external disturbances on the power system in an attempt to prevent and destroy the automatic dispatching of the distribution system agent; Step S2: Establishing a Markov simulation environment model of the power distribution system; Step S2.1: Modeling the uncertain distributed renewable energy generation units, storage and discharge behaviors, coordination between distributed devices, and electric vehicle loads in the distribution system; Step S2.2: Taking into full consideration the unit models established in step S2.1, a Markov simulation environment model of the power distribution system is established; Step S3: Based on the Markov simulation environment model of step S2, alternate training of the power system dispatching agent and the attacking agent is performed; In this process, two agents compete against each other to form a two-person zero-sum game multi-agent reinforcement learning model with a Maximin strategy. Step S3.1: The active power injection of the power generation unit, the energy storage unit, the load unit, the reactive power injection of the load unit, the maximum active output and reactive output of the distributed generation unit, and the charge state of the energy storage unit are modeled as the Markov state parameters of the power system dispatching main agent N1 and the adversarial agent N2 that performs disturbance attacks on the main agent state in the dual-agent Markov zero-sum game model of the distribution system; Step S3.2: Model the active and reactive reduction values ​​of the distributed generation units and the active and reactive injection values ​​of the energy units as Markov action parameters of the power system dispatching main agent N1 in the dual-agent Markov zero-sum game model of the distribution system; Step S3.3: The action of applying disturbance to the Markov observation state of the main agent N1 is modeled as the Markov action parameter of the adversarial agent N2 that performs disturbance attack on the state of the main agent in the dual-agent Markov zero-sum game model of the power distribution system; Step S3.4: Model the energy loss of distributed generation units in the process of reducing output, the energy loss of power lines, and the energy loss of energy storage units in the process of storing and discharging energy as the reference indicators of the reward function of the dual-agent Markov zero-sum game model of the distribution system; Step S3.5: In the dual-agent Markov game model of the power distribution system, a reference index of the penalty function for violating the safety constraint is set; Step S4: Using the Actor-Critic algorithm, the multi-agent reinforcement learning model of the Maximin strategy proposed in step S3 is solved to improve the robustness of the distribution system and evaluate its impact on the demand-side response potential; Step S4.1: Determine the Maximin strategy solution target of the state perturbation-based robust reinforcement learning method; Step S4.2: Use the Actor-Critic algorithm to solve the solution objective determined in step S4.1, improve the robustness of the distribution system, and evaluate its impact on the demand-side response potential.

2. The demand-side response potential evaluation method based on robust reinforcement learning theory according to claim 1, characterized in that: In step S1.1, the dual-agent γ-discounted zero-sum Markov game MDP model is represented as a seven-tuple: (S, A1, A2, P, R, γ, ρ0); Among them, S represents the state set of the system; A1 and A2 represent a set of continuous actions taken by two agents in the game; P: S×A1×A2×S'→R, P represents the transition probability density function jointly determined by the two agents, and S' represents the next state set; r: S×A1×A2→R, r represents the immediate reward value fed back by the environment to the two game agents; R represents a missing or incorrectly filled in, which is a set of reward functions or the reward function itself, describing the rewards obtained by the agent in different states; ρ0 represents the discount factor; Assume that the main agent N1 is trained to learn the strategy μ, and the adversarial agent N2 is trained to learn the strategy ν; for the dual-agent γ discounted zero-sum Markov game problem, the immediate reward function r μ,v for: Where E represents the expected value operator, which is used to calculate the average value under given conditions; a1 and a2 represent the actions taken by the two agents / decision makers in state s according to their respective strategies μ and v; μ(s) and v(s) represent the action choices taken by strategies μ and ν in state s respectively; s represents the current state of the system; r(s, a1, a2) represents the immediate reward returned by the environment after the agent / decision maker takes actions a1 and a2 in state s; A two-agent γ-discounted zero-sum Markov game, which is viewed as a zero-sum game process in which the main agent N1 maximizes the γ-discounted reward; at the same time, the adversary agent N2 minimizes the γ-discounted reward; In the robust reinforcement learning setting, there exists an uncertainty set U represented by: U∈{S,A,R,P} The uncertain variables contained in this set may be the state s∈S, action a∈A, reward function r∈R, and transition probability density function P in reinforcement learning; The goal of the robust reinforcement learning algorithm is to learn an attack strategy ν through training to maximize the performance of the uncertain variable u∈U, that is, to try to disrupt the performance of the maximum reward function strategy π, which is expressed as the following Maximin robust optimization problem: Among them, R π (u) is the mathematical expected cumulative return under the strategy π, with the uncertain variable u as the independent variable. By performing Markov modeling on the natural uncertainties that occur in real applications and the errors designed by humans, adversarial attacks on uncertain variables u in different forms are achieved.

3. The demand-side response potential evaluation method based on robust reinforcement learning theory according to claim 1, characterized in that: In step S2.1, the power generation unit U G , these units ideally only inject electrical energy into the distribution system: IN G ={U TG ,IN DG } Among them, U TG It is a traditional power generation unit; U DG For distributed generation units; Distributed generation units are: IN DG ={U SDG ,IN WDG } Among them, U SDG and U WDG They represent wind power generation unit and photovoltaic power generation unit respectively; Distributed power storage unit U DES , the energy storage unit can both extract and store electrical energy from the power distribution system and inject the stored electrical energy into the power distribution system; Load unit U L ,include: IN L ={U IL ,IN RL ,IN EVL } Among them, U IL It is the industrial power load unit, U RL is the residential electricity load unit, U EVL It is a mobile electric vehicle load unit; The above three types of loads can only draw electrical energy from the distribution system.

4. The demand-side response potential evaluation method based on robust reinforcement learning theory according to claim 1, characterized in that: In step S2.2, the distribution system Markov simulation environment includes a conventional power generation unit U TG Distributed generation unit U with intermittent generation characteristics DG , a distributed energy storage unit U that can achieve two-way energy flow with the distribution system DES , load units U with different load characteristics L .

5. The demand-side response potential evaluation method based on robust reinforcement learning theory according to claim 1, characterized in that: Step S3.1 includes: State set S: Markov state s of the power system dispatching main agent N1 and the adversarial agent N2 that perturbs the main agent state t ∈S, are defined by the following formula: Where t is the minimum time step in the training process; They represent the value i at time t and the value u at time t. g From the collection U G , these values ​​represent the active and reactive reduction values ​​of distributed generation units at a certain point in time; similarly, and represents another set of values ​​for different indicators i at time t, where u des is from the set U DES An element selected from , these values ​​represent the active and reactive power injection of the energy storage unit at a certain point in time; and represents another set of indicators for quantity i at time t, where u l is an element selected from the set Ux; and Respectively represent the maximum active output and reactive output of distributed generation units; Indicates the charge state of the energy storage unit; Step S3.2 includes: Action set A1: Markov action set a of the main intelligent agent N1 in power system dispatching 1,t ∈A1, defined by the following formula: in, Represent the active and reactive reduction values ​​of distributed generation units, u dg represents a distributed generation unit; Respectively represent the active and reactive injection values ​​of the energy storage unit, u des Represents an individual or specific type of energy storage unit; In step S3.3, the action set A2 is included: the action of the adversarial agent N2 is to perturb the Markov observation state of the main agent N1. Its action set A2 is consistent with the s defined in step S3.

1. t The form and dimensions are the same; In step S3.4, the reward function R is included: For the dual-agent Markov zero-sum game model based on the virtual power plant, the reward functions of the two agents should be opposite to each other; |r t |∈R is defined as follows: |r t |=|ΔE cur +ΔE el +ΔE des Δ The reward function is composed of the energy loss during the operation of the virtual power plant, ΔE cur Represents distributed generation unit U DG Energy loss during output reduction; ΔE des Represents the power line energy loss; ΔE des Represents energy storage unit U DES Energy loss during storage and discharge; In step S3.5, the penalty set C is used to quantify the penalty feedback for violating the safe scheduling constraint: Safety restrictions on the voltage value range of each node B; node voltage safety constraints are particularly critical for ensuring the stable operation of each unit of the entire virtual power plant; the penalty function term c1∈C is set using the following formula: Among them, V i is the voltage value of node i, and its upper and lower bounds V i max 、V i max Voltage safety constraints; The transmission lines in the virtual power plant have line capacity limitations, that is, the maximum power limit passing through the line; set safety constraints to ensure that the line will not overheat and cause failures or fire disasters; set the penalty function term c2∈C using the following formula: Among them, S ij is the power value from node i to node j, and the power flow is calculated The line capacity security constraint.

6. The demand-side response potential evaluation method based on robust reinforcement learning theory according to claim 1, characterized in that: In step S4.1, for the robust reinforcement learning method based on state perturbations, the focus is on system robustness and safety: Among them, Q μ (s,a) represents the behavior value function of selecting action a in state s, V μ (s) is in strategy The value function of the next state s; Represents the distribution of state s according to p μ Sampling is performed and the expected value of the expression is taken; this expected value represents the behavior value function Q under a given strategy μ. μ (s,a) and value function V μ (s) Perform the operation and obtain the expected value; The Actor-Critic algorithm enables the agent to learn better strategies by continuously optimizing the expected objective function. Among them, the policy network Actor is responsible for learning how to choose an action, and the value network Critic evaluates the value of choosing the action; In the iterative process, the Actor network of the main intelligent agent N1 of the power distribution system dispatch is responsible for learning the strategy of generating actions and evaluating the value of these actions through the Critic network; the attacking adversarial agent N2 is also in the game training, by applying perturbations to the observed state of the main intelligent agent in order to obtain a better adversarial strategy; The Actor-Critic algorithm continuously optimizes strategies during the iteration process, so that the distribution system dispatching agent can gradually improve its robustness to external disturbances and maintain more robust system security under the same interference conditions. The Actor-Critic algorithm is a reinforcement learning method, the core of which is to train two networks at the same time: the actor and the critic. In this scenario, the actor is responsible for learning how to choose actions, while the critic evaluates the value of these actions. In the power distribution system, this means that N1 will continuously learn better dispatching strategies to adapt to different external environmental interferences and maintain a more robust system operation status under similar interferences; Among them, R μ represents the expected return of strategy μ; T represents the time step, which represents the cumulative return in the time series 0 to T; At the same time, the penalty target form expectation is minimized as follows: The goal of attacking adversarial agent N2 is to learn the optimal network parameters θ′ of the perturbation strategy ν through the deep reinforcement learning algorithm, and to perturb the observed state of the main agent N1, so that it outputs scheduling decisions with large energy loss and easy to violate safety constraints; contrary to the main agent, the reward target form of the above formula is minimized, and the penalty target form of the following formula is maximized; In essence, the two agents constitute a dual-agent zero-sum game reinforcement learning model: In the Critic part, the value network is used to evaluate the value of the state, which helps guide the update direction of the Actor strategy network; Actor-Critic adjusts the generated action strategy based on the state value information given by the Critic, making the selected action more likely to achieve a better state: in, V represents the loss function of the Critic network, which is the square loss of the difference between the actual reward and the estimated reward; ω represents the gradient operator with respect to the parameter ω; s t Indicates the current state; V ω (s t ) represents the Critic network's response to state s t estimate of the value of; is equivalent to: in, represents the gradient operator with respect to the parameter ω.

Images