A demand response closed-loop control method, system, device and medium

Abstract

The application discloses a demand response closed-loop control method, system, device and medium, comprising: proposing a double-adversarial MDP model based on burst power failure awareness, constructing an agent for a microgrid operator to solve minimization and maximization problems respectively; considering the influence of power distribution network burst power failure, constructing an emergency regulation preference model, under the premise that the emergency regulation response amount is greater than the power protection plan amount, constructing an emergency response node set that can be connected to the power failure node and still normally operate, reducing the risk of large-area power failure. And propose a data-driven closed-loop precise robust power control method based on double-adversarial DQN, through the continuous interaction of two agents with the power distribution network environment, learning the optimal demand response closed-loop control strategy, and combining the output results of another agent to train the model, the two are iterated and solved alternately, and the robust optimal solution is obtained, realizing the precise control of data-driven closed-loop precise robust power.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power distribution network management, and particularly relates to a demand response closed-loop control method, system, device and medium. BACKGROUND

[0002] Demand response closed-loop control plays a crucial role in modern power distribution network management. Closed-loop control strategies can effectively alleviate network load fluctuations, optimize power generation resource allocation, improve energy use efficiency, and help balance supply and demand. Especially during peak demand, demand response closed-loop control can dynamically allocate resources and promote the development of low-carbon economy and smart energy systems by stimulating demand-side participation. However, with high proportions of distributed source, load and storage resources accessing the power distribution network, wind power and photovoltaic power generation have high randomness and volatility, and user-side power fluctuations and voltage instability pose great challenges to demand response closed-loop feedback and the safe and stable operation of the power distribution network.

[0003] Microgrid operators can aggregate distributed power sources, energy storage, controllable loads and other user-side resources through controllers to participate in demand response, reduce power distribution network losses, improve demand response revenue, and achieve mutual benefits between the power distribution network and users.

[0004] Existing demand response closed-loop control methods usually consider the uncertainty of renewable energy and the operating characteristics of microgrids at day-ahead and intra-day time scales, use robust optimization methods, adjust the implementation plan based on the response results, achieve closed-loop control, reduce scheduling costs, and improve calculation speed. Deep Q-network (DQN) algorithm is a machine learning method that enables agents to learn how to act in an environment to maximize some cumulative reward, has the advantages of high computational efficiency and good convergence characteristics, and can achieve demand response closed-loop control optimization and regulation.

[0005] However, the existing demand response closed-loop control technology mainly has the following problems: the uncertainty of the power grid-communication network double-network coupling and the demand response system closed-loop feedback means are not considered, the demand response decision cannot be accurately adjusted according to the response result, the robustness, real-time performance and accuracy of the user-side resource participation demand response model are reduced, and it is difficult to realize the accurate power closed-loop control and optimization decision of demand response. Moreover, the influence of sudden power failure on demand response closed-loop feedback is not considered, which leads to the reduction of high power protection priority load during sudden power failure, and it is difficult to guarantee the continuous power supply stability of high power protection priority load. Further, the deep reinforcement learning method of single agent decision feedback learning does not consider the antagonism between multi-agent optimization goals, is not suitable for solving the min-max double-layer optimization problem obtained by robust peer-to-peer transformation, has the problems of slow convergence speed, poor optimization ability and difficult to guarantee robustness, and is difficult to realize the data-driven accurate closed-loop control of user-side resource demand response. SUMMARY

[0006] The present application provides a demand response closed-loop control method, system, device and medium, which is used to realize demand response closed-loop control under the uncertainty of double-network operation, improve the convergence degree and optimization ability of the algorithm, and improve the stability of power distribution network operation.

[0007] Therefore, the first aspect of the present application provides a demand response closed-loop control method, which is applied to a technical architecture composed of a power distribution network dispatching center, a microgrid operator, and source-network-load-storage controllable resources; wherein the microgrid operator controls the source-network-load-storage controllable resources through a microgrid controller.

[0008] The method comprises:

[0009] S1, a two-stage robust optimization model of power grid-communication network double-network operation uncertainty is established with the minimum weighted difference between power distribution network loss and microgrid operator benefit as the optimization goal;

[0010] S2, an agent agent max and an agent agent min are constructed for the microgrid operator to solve the maximization problem and the minimization problem, and the agent agent max and the agent agent m i n corresponding to the state space based on the sudden power failure awareness to determine the emergency control preference set of the node controllable resources, and the action space and the reward function are designed, wherein the agent agent max and the agent agent min both include a corresponding evaluation network, a target network and an experience replay pool;

[0011] S3, initialize the iteration number, and initialize the iteration stop condition and the uncertain variable of the power grid-communication network dual-network operation uncertainty two-stage robust optimization model according to the reward function;

[0012] S4, the intelligent agent agent max and the intelligent agent agent min determines the current observable state, inputs it into the evaluation network, and selects the first action with the maximum state-action value by using an e-greedy strategy, while the intelligent agent agent max and the intelligent agent agent min executes the first action and calculates the reward;

[0013] S5, it is judged whether the iteration stop condition is met, if yes, the optimal solution of the power grid-communication network dual-network operation uncertainty model in this period is output, and the regulation and control decision is issued to the microgrid controller, otherwise the iteration number is updated and the step S4 is returned;

[0014] S6, when the source network load storage controllable resource responds to the regulation and control decision, the network loss and benefit in this period are calculated, and the action chain of the intelligent agent agent max and the intelligent agent agent min is generated and stored in the experience replay pool, and experience data is extracted from the experience replay pool to calculate the time sequence difference error of the intelligent agent agent max and the intelligent agent agent min ;

[0015] S7, the parameters of the evaluation network and the parameters of the target network are updated according to the time sequence difference error, and it is judged whether the current period t is greater than the total period T, if yes, the optimization is ended, otherwise the period is updated and the step S3 is returned.

[0016] Optionally, the power grid-communication network dual-network operation uncertainty two-stage robust optimization model is represented as:

[0017]

[0018] The constraint conditions of the power grid-communication network dual-network operation uncertainty two-stage robust optimization model include: distribution network branch flow constraint, node voltage and line current safety constraint, and controllable resource output constraint;

[0019] In the formula, is the optimization objective, which represents the minimum weighted difference between the distribution network loss and the microgrid operator benefit, x is the set of deterministic decision variables, which is composed of the set of source network load storage controllable resource outputs, y is the set of uncertain variables, which is composed of the uncertain factors of each node of the power grid α i,t , which takes any value in the interval [-1, 1].

[0020] Optionally, the state space representation based on the burst outage awareness is:

[0021]

[0022] wherein, and are the demand response incentive prices paid by the distribution network to the distributed photovoltaic, energy storage device and controllable load in the microgrid operator respectively in the t period, is the electricity price of the energy storage device purchasing electricity from the distribution network in the t period, and the specific price depends on the time-of-use electricity price of the distribution network; E(t-1) = {E 1,t-1 ,...,E i,t-1 ,...,E N,t-1} is the set of node energy storage capacity in the t-1 period; and are the reward outputs of the last iteration of another agent; σ(t) is a burst outage indication variable, and when the distribution network bursts, σ(t) = 1, otherwise σ(t) = 0; is the set of emergency regulation preferences of the node controllable resource, wherein Z = {DG, ES, CL} represents the specific power equipment type participating in emergency response at node i in the t period, is the emergency regulation preference at node i in the t period, which is specifically represented as:

[0023]

[0024] wherein, is the set of controllable loads that still have demand response capability under burst outage; is the controllable load power preservation priority issued by the distribution network in the t period, which is divided into four priorities: critical equipment, important facilities, general commercial and residential electricity, and non-essential electricity. The higher the controllable load power preservation priority, the smaller the value of ; is the price paid by the distribution network for emergency response in the t period, which depends on the time-of-use electricity price of the source network load storage controllable resource type; represents the adjustable active power of the source network load storage controllable resource type; is the historical average response performance index, which is represented as:

[0025]

[0026] wherein, respectively represent the average duration of response, response speed and response reliability of the power equipment demand response obtained according to historical experience information.

[0027] Optionally, the agent agent maxand the agent min The action space of the agent is respectively represented as:

[0028] Ψ max (t) = {Ψ 1,max (t),..., Ψ i,max (t),..., Ψ N,max (t)}.

[0029] Ψ min (t) = {Ψ 1,min (t),..., Ψ i,min (t),..., Ψ N,min (t)}.

[0030] The reward function in real time is represented as:

[0031]

[0032] In the formula, is an optimization objective of an optimization problem, that is, a weighted difference between minimization of power grid loss and micro-grid operator revenue; v is a balance weight; and W is a larger real number; is an event function, if a branch power flow constraint is met, then otherwise Similarly, if a controllable resource output constraint is met, then otherwise

[0033] Optionally, the reward function is used to initialize an iteration stop condition, and specifically,

[0034]

[0035] In the formula, e is a distance parameter.

[0036] Optionally, the time series difference error is represented as:

[0037]

[0038] In the formula, represents a number of action chains extracted from the experience pool; Ψ min (t) represents an action; represents a state-action value function; represents the evaluation network; h min (t) represents a state-action value updated by the target network after the distribution network master station obtains a reward The update formula of the target network is:

[0039]

[0040] wherein, β min is a discount factor, used to measure the influence of future time slot rewards on the current state-action value, the greater the value, the greater the influence of future rewards on the current action policy, and the algorithm pays more attention to long-term benefits; θ min (t) represents the reward value; S min (t+1) represents the next state; Ψ min (t+1) represents the next action. represents the target network.

[0041] Optionally, the update formula of the parameter of the evaluation network is:

[0042]

[0043] wherein, t mod T' represents the remainder of t divided by T'.

[0044] The second aspect of the present application provides a demand response closed-loop control system, applied to a technical architecture composed of a power distribution network dispatching center, a microgrid operator, and source-grid-load-storage controllable resources; wherein the microgrid operator regulates the source-grid-load-storage controllable resources through a microgrid controller;

[0045] The system comprises:

[0046] A first construction unit is configured to establish a power grid-communication network dual-network operation uncertainty two-stage robust optimization model with the optimization target of minimizing the weighted difference between power grid loss and microgrid operator revenue.

[0047] A second construction unit is configured to construct an agent agent max and an agent agent m i n to solve the maximization problem and the minimization problem, and design the agent agent max and the agent agent min corresponding state space based on power outage burst awareness to determine the emergency regulation preference set of node controllable resources, and design the action space and the reward function, wherein the agent agent max and the agent agent min each comprise a corresponding evaluation network, a target network, and an experience replay pool.

[0048] An initialization unit is configured to initialize the number of iterations, and initialize the iteration stop condition according to the reward function, and the uncertain variables of the power grid-communication network dual-network operation uncertainty two-stage robust optimization model.

[0049] A first calculation unit is configured to perform the following steps through the agent agent max and the agent agentmin The current observable state is determined and input to the evaluation network, and an e-greedy strategy is used to select a first action with the maximum current state-action value, while the agent max and the agent min The first action is executed and the reward is calculated.

[0050] A judgment unit is configured to judge whether the iteration stop condition is met, and if so, output the optimal solution of the power grid-communication network dual-network operation uncertainty model at this time period, and issue the control decision to the microgrid controller, otherwise, update the iteration number and trigger the initialization module.

[0051] A second calculation unit is configured to calculate the network loss and benefit at this time period when the source network load storage controllable resources respond to the control decision, and generate the action chain of the agent max and the agent m i n and store it in the experience replay pool, and extract experience data from the experience replay pool to calculate the time difference error of the agent max and the agent min .

[0052] An update unit is configured to update the parameters of the evaluation network and the parameters of the target network according to the time difference error, and judge whether the current time period t is greater than the total time period T, if so, end the optimization, otherwise, update the time period and trigger the first calculation unit.

[0053] The third aspect of the present application provides a demand response closed-loop control device, the device comprising a processor and a memory:

[0054] The memory is configured to store program code and transmit the program code to the processor.

[0055] The processor is configured to execute the steps of the demand response closed-loop control method according to the instructions in the program code.

[0056] The fourth aspect of the present application provides a computer readable storage medium for storing program code, the program code being used to execute the demand response closed-loop control method of the first aspect.

[0057] From the above technical solutions, the present application has the following advantages:

[0058] (1) The application proposes a double-adversarial MDP model based on burst power outage awareness, which constructs an agent for microgrid operators to solve the minimization problem and the maximization problem, considers the impact of power outage of the distribution network, and constructs an emergency control preference model based on the load power protection priority, economic incentive level, user power preference, demand response historical data and other information issued by the distribution network. Under the premise that the emergency control response amount is greater than the power protection plan amount, an emergency response node set that can connect to the outage node and still operate normally is constructed, which can reduce the risk of large-scale power outage and ensure the continuous power supply stability of high power protection priority loads.

[0059] (2) The application proposes a data-driven closed-loop precise robust power control method based on double-adversarial DQN, which continuously interacts with the distribution network environment through two agents, learns the optimal demand response closed-loop control strategy, and trains the model combined with the output results of another agent. Iterative solution is alternately iterated, which can improve the convergence speed and optimization ability of the algorithm, get the robust optimal solution, and realize the precise control of data-driven closed-loop precise robust power.

[0060] (3) Based on the double-adversarial MDP model and the data-driven closed-loop precise robust power control method based on double-adversarial DQN, the demand response closed-loop control under the uncertainty of double-network operation is realized, the convergence degree and optimization ability of the algorithm are improved, and the stability of the operation of the distribution network is improved. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 Flowchart of a demand response closed-loop control method provided in an embodiment of the application Figure 1 ;

[0062] Figure 2 Flowchart of a demand response closed-loop control method provided in an embodiment of the application Figure 2 ;

[0063] Figure 3 Demand response closed-loop control technology architecture considering double-network operation uncertainty provided in an embodiment of the application

[0064] Figure 4 Demand response closed-loop control technology principle considering double-network operation uncertainty provided in an embodiment of the application

[0065] Figure 5 Structure diagram of a demand response closed-loop control system provided in an embodiment of the application. DETAILED DESCRIPTION

[0066] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0067] Please refer to Figure 1 、 2 and 4, a demand response closed-loop control method provided in the embodiments of the present application is applied to a technical architecture composed of a power distribution network dispatching center, a microgrid operator, and source-grid-load-storage controllable resources. The microgrid operator regulates and controls the source-grid-load-storage controllable resources through a microgrid controller.

[0068] The technical architecture is shown in Figure 3 , mainly including three layers of the power distribution network dispatching center, the microgrid operator, and the source-grid-load-storage controllable resources.

[0069] Power distribution network dispatching center: The power distribution network dispatching center divides user-side demand response control into T time periods, and the set is The demand response demand is issued to the microgrid operator, the microgrid operator collects controllable resource data of each node of the microgrid, formulates a demand response plan to reduce network loss and improve operation income, and realizes closed-loop control.

[0070] Microgrid operator: The microgrid operator includes two intelligent agents, and a microgrid controller is configured on the grid node to realize data-driven demand response centralized regulation and control. The set of grid nodes is represented as The microgrid operator issues the demand response plan to each microgrid controller through the communication network to execute the demand response decision. The microgrid controller completes information interaction with the distributed power supply, energy storage, and controllable load through communication modes such as Ethernet, wireless network, and power carrier communication. The set of controllable loads controlled by the microgrid controller of the i th node is defined as

[0071] Source-grid-load-storage controllable resources: The source-grid-load-storage controllable resources mainly include distributed power supply, energy storage, and controllable load at each node of the grid. Since the controllable load is of various types, distributed, and has weak communication connection, the microgrid controller needs to aggregate a large number of controllable loads to participate in demand response.

[0072] The method includes:

[0073] Step 101: A two-stage robust optimization model of power grid-communication network dual-network operation uncertainty is established with the minimum power distribution network network loss and the weighted difference between microgrid operator income as the optimization objective.

[0074] In one embodiment, the power grid-communication network dual-network operation uncertainty two-stage robust optimization model in step 101 is represented as:

[0075]

[0076] It should be noted that for step 101, based on the above architecture, the present application proposes an optimization problem, which is a dual-network operation uncertainty two-stage robust optimization model, and can be represented as wherein is the optimization objective, indicating the minimization of the weighted difference between the power distribution network loss and the microgrid operator revenue, and the constraint conditions mainly consider the power distribution network branch flow constraint, the node voltage and line current safety constraint, and the controllable resource output constraint; x is a set of deterministic decision variables, which is composed of a set of controllable resource outputs; y is a set of uncertain variables, which is composed of each node uncertain factor α i,t , which is randomly taken in the interval [-1, 1].

[0077] For the set of deterministic decision variables and the set of uncertain variables, it should be noted that:

[0078] The present application considers the influence of the fluctuation of new energy output and the reliability of response equipment on the regulation and control decision, and performs closed-loop control on the controllable resource participating in demand response regulation. The distributed photovoltaic active power of node i at time period t is The charging and discharging power of the energy storage device is and and the energy storage device cannot charge and discharge at the same time; the curtailment of controllable load is The set of deterministic decision variables is x The set of uncertain variables is y = α t , wherein a t represents the α i,t set of all nodes at time period t.

[0079] Step 102, the microgrid operator constructs an agent agent max and an agent agent min to solve the maximization problem and the minimization problem, and designs an agent agent max and an agent agent min corresponding to the state space based on the power outage burst awareness to determine the emergency regulation preference set of the node controllable resource, and designs the action space and the reward function, wherein the agent agent max and the agent agent min both include a corresponding evaluation network, a target network and an experience replay pool.

[0080] For step 102, step 102 is a MDP modeling based on burst power outage awareness. The present application needs to be explained for step 101 two-network operation uncertainty two-stage robust optimization model Decoupling is performed by the tangent plane method, and the uncertain variable y that maximizes the weighted difference between power grid loss and benefit in the inner layer max model is solved first, and then the uncertain variable y is substituted into the outer layer min model to solve the decision variable x that minimizes the weighted difference between power grid loss and benefit, and finally the double-layer optimization model is converted into two single-layer maximum problems And Where (x * ,y * ) is the optimal solution of the model , the microgrid operator constructs an agent agent max and agent min to solve the maximum problem and the minimum problem

[0081] At the same time, in view of the problem that the demand response capability of part of controllable load resources is lost due to burst power outage under the coupling of power grid-communication network, the present application considers the load power protection priority, economic incentive level, demand response historical data and other information issued by the power grid dispatching center to determine the emergency control preference, designs the corresponding state space, action space and reward function, and performs emergency response to ensure the stable operation of power grid and reduce the risk of large-area power outage.

[0082] In specific implementation, step 102 includes the following steps:

[0083] Step 1021: Design the state space. The state space based on burst power outage awareness can be expressed as:

[0084]

[0085] Wherein, and are the demand response incentive prices paid by the power distribution network to the distributed photovoltaic, energy storage device and controllable load in the microgrid operator at period t, is the electricity price purchased by the energy storage device from the power distribution network at period t, and the specific price depends on the time-of-use electricity price; E(t-1)={E 1,t-1 ,...,E i,t-1 ,...,E N,t-1} is the node energy storage capacity set at period t-1; and are the reward outputs of the last iteration of another agent; sigma(t) is a burst power outage indicator variable, which is equal to 1 when the power distribution network bursts, and otherwise sigma(t) is equal to 0; is the emergency control preference set of controllable resources of node i, where Z = {DG, ES, CL} represents the specific power equipment type participating in emergency response at node i at time t, is the emergency control preference of node i at time t, which can be specifically represented as:

[0086]

[0087] wherein, is the set of controllable loads that still have demand response capability under sudden power failure; is the controllable load power protection priority issued by the distribution network at time t, which is divided into four priorities: critical equipment, important facilities, general commercial and residential electricity, and non-essential electricity. The higher the controllable load power protection priority, the smaller the value of ; is the price paid by the distribution network for emergency response at time t, which depends on the time-of-use price of the controllable resource type; represents the adjustable active power of the controllable resource type; is the historical average response performance index, which is represented as:

[0088]

[0089] wherein, respectively represent the average duration of response, response speed and response reliability of the power equipment demand response obtained according to historical experience information.

[0090] Step 1022: Design action space. The application defines the action space of agent max and agent min as Ψ max (t) = {Ψ 1,max (t),..., Ψ i,max (t),..., Ψ N,max (t)} and Ψ min (t) = {Ψ 1,min (t),..., Ψ i,min (t),..., Ψ N,min (t)}, respectively.

[0091] When the distribution network suddenly fails, i.e. σ(t) = 1 in the state space, the distribution network dispatching center determines the power protection priority of the equipment in the outage area and issues it to the microgrid operator, determines the emergency control preference according to formula (2), and simultaneously considers the network topology relationship after the fault. Under the premise that the emergency control response amount is greater than the power protection plan amount, the emergency response node set that can be connected to the outage node and still normally operate is constructed based on sorting Therefore, under the condition of sudden power failure, for action Ψ i,min (t), Ψ i,max (t) can be expressed as

[0092] Step 1023: design reward function. In order to improve the convergence speed of the algorithm, improve the accuracy of closed-loop feedback control, guarantee the planning requirements of emergency demand response of distribution network, and add a double DQN convergence closed-loop control item, the application compares the difference of the optimization objectives of two agents to determine whether to iterate. If the threshold is exceeded, feedback adjustment is performed through the closed-loop control item. Real-time reward function and is expressed as

[0093]

[0094] wherein, is the optimization objective of the optimization problem, that is, the weighted difference between the minimum distribution network loss and the microgrid operator revenue; v is the balance weight; W is a larger real number; is an event function, if the branch power flow constraint is met, then otherwise Similarly, if the controllable resource output constraint is met, then otherwise

[0095] For steps 103 to 105 described below, for the MDP model constructed in step 102, the application proposes a regulation and optimization based on double adversarial DQN algorithm. The microgrid operator constructs two DQN agents, agent min for solving the minimization problem, and agent max for solving the maximization problem. The agent structure is symmetrical, and the following steps are introduced taking agent min as an example.

[0096] agent min includes an evaluation network a target network and an experience replay pool Among them, the evaluation network realizes the mapping from the state-action combination to the state-action value function . The state-action value function represents the conditional expectation that the agent can obtain the sum of future rewards by selecting action Ψ min (t) in state

[0097] The agent learns and executes the microgrid operator demand response decision at each time period through the evaluation network; each controllable resource of the source, load and storage executes the decision and feeds back the updated information to the microgrid controller. In addition, the algorithm sets a target network​ and experience replay pool The correlation of sample data between consecutive time slots is reduced, so as to improve the evaluation of network optimization performance. Two agents continuously interact with the power distribution network environment, learn the optimal demand response control strategy, train the model combined with the output results of another agent, iteratively antagonize each other, improve the convergence speed of the algorithm, and obtain the optimal solution of demand response regulation. For details, see steps 103-105 below.

[0098] Step 103, initialize the number of iterations, initialize the iteration stop condition according to the reward function, and initialize the uncertain variables of the power-communication network dual-network operation uncertainty two-stage robust optimization model.

[0099] It should be noted that in the specific implementation, step 103 is the initialization of the confrontation, specifically: initialize the state space, the number of iterations k=0, and set the iteration stop condition as where e is the distance parameter; and the uncertain variables y of the power-communication network dual-network operation uncertainty two-stage robust optimization model k =0.

[0100] Step 104, through the agent agent max and the agent agent min determine the current observable state, input it into the evaluation network, and select the first action with the maximum state-action value in the current state-action value using the ∈-greedy strategy, while the agent agent max and the agent agent min executes the first action and calculates the reward.

[0101] It should be noted that in the specific implementation, step 104 is the solution of the agent confrontation, specifically: the initial value y k =0 of the uncertain variable in step 103 is substituted into At the same time, agent min inputs the current observable state into the evaluation network to obtain the state-action value function of different actions Ψ min (t) in the current period. On this basis, the action with the maximum state-action value in the current period is selected using the ∈-greedy strategy.

[0102] agent min executes the action Ψ min (t), calculates the reward value θ min (t), updates the relevant information, and transfers to the next state S min (t+1), and the action chain ζ=[S min (t), Ψ min (t), θ min(t), S min (t+1)] into the experience replay pool agent min randomly sample a set of experience data from the experience replay pool calculate the temporal difference error of the evaluation network in the current time slot, denoted as

[0103]

[0104] wherein, denotes the number of action chains extracted from the experience pool; h min (t) denotes the state-action value of the power grid master station after obtaining the reward updated by the target network, and its update formula is

[0105]

[0106] wherein, β min ∈[0,1] is a discount factor, used to measure the influence of future time slot rewards on the current state-action value. The greater the value, the greater the influence of future rewards on the current action policy, and the algorithm pays more attention to long-term benefits.

[0107] Further, the agent min updates the evaluation network parameters according to the temporal difference error, i.e. γ is the learning rate, and the greater the γ, the faster the parameter convergence, but too large γ will make the algorithm fall into a local optimum.

[0108] The target network is synchronized with the evaluation network every T' time periods, and remains unchanged for the next T' time periods, and the update formula is:

[0109]

[0110] wherein, tmodT' represents the remainder of t divided by T'.

[0111] find its initial optimal solution x k , and obtain

[0112] agent min input the optimal solution x k obtained in this iteration and the reward to the agent max . The agent max solves the process similar to the agent min , finds its optimal solution y k in this iteration, and obtains

[0113] Step 105: Determine whether the iteration stopping condition is met. If yes, output the optimal solution of the uncertainty model of the power grid-communication network operation during this period and send the control decision to the microgrid controller. Otherwise, update the iteration count and return to step 104.

[0114] It should be noted that in the specific implementation, step 105 is to determine convergence, specifically: comparison and Check if the iteration stopping condition is met. If not, it means the optimal solution has not yet been found. Let k = k + 1, return to step 104, and continue to the next round of adversarial iteration. If the condition is met, output the optimal solution (x) at this point. * ,y * )=(x k ,y k This refers to the control decision, which is then sent to the microgrid controller.

[0115] Step 106: After the source-grid-load-storage controllable resources respond to the regulation decision, calculate the network loss and revenue for this period, and generate the intelligent agent. max and intelligent agent min The action chain is stored in the experience replay pool, and experience data is extracted from the experience replay pool to calculate the agent. max and intelligent agent min The timing difference error.

[0116] It should be noted that in the specific implementation, step 106 involves calculating network losses and revenues. Specifically: after the source, grid, load, and storage controllable resources respond to the controllability decision, the network losses and revenues for this period are calculated; subsequently, an intelligent agent is generated. max and intelligent agent min The action chain is stored in the experience replay pool. Experience data is randomly extracted from the experience replay pool, and the agent is calculated separately. min and agent max Timing difference error φ min (t) and φ max (t).

[0117] The calculation process for timing difference error has already been explained in step 104 above, and will not be repeated here.

[0118] Step 107: Update the parameters of the evaluation network and the target network based on the time-series difference error, and determine whether the current time period t is greater than the total time period T. If so, end the optimization; otherwise, update the time period and return to step 103.

[0119] It should be noted that the agent is updated based on the time-series differential error. min and agent max Evaluation of network parameters and and if t>T, the optimization is ended, otherwise, t=t+1 and return to step 103.

[0120] The above is a demand response closed-loop control method provided in the embodiments of the present application, and the following is a demand response closed-loop control system provided in the embodiments of the present application.

[0121] Referring to Figure 5 The demand response closed-loop control system provided in the embodiments of the present application is applied to a technical architecture composed of a power distribution network dispatching center, a microgrid operator, and source-grid-load-storage controllable resources; wherein the microgrid operator regulates and controls the source-grid-load-storage controllable resources through a microgrid controller.

[0122] The system comprises:

[0123] The first construction unit 201 is configured to establish a power grid-communication network dual-network operation uncertainty two-stage robust optimization model with the minimum weighted difference between power grid loss and microgrid operator revenue as an optimization target.

[0124] The second construction unit 202 is configured to construct an agent agent max and an agent agent min to solve the maximization problem and the minimization problem, and design an agent agent max and an agent agent min corresponding to the state space based on power outage burst awareness to determine an emergency regulation preference set of node controllable resources, and design an action space and a reward function, wherein the agent agent max and the agent agent min both comprise a corresponding evaluation network, a target network, and an experience replay pool.

[0125] The initialization unit 203 is configured to initialize the number of iterations, and initialize the iteration stop condition according to the reward function, and the uncertain variables of the power grid-communication network dual-network operation uncertainty two-stage robust optimization model.

[0126] The first calculation unit 204 is configured to determine the current observable state through the agent agent max and the agent agent m i n , input the current observable state into the evaluation network, and select the first action with the maximum current state-action value by using an ε-greedy strategy, while the agent agent max and the agent agent min execute the first action and calculate the reward.

[0127] The judgment unit 205 is configured to judge whether the iteration stop condition is met, and if yes, output the optimal solution of the power grid-communication network double-network operation uncertainty model at this period, and send the control decision to the micro-grid controller, otherwise, update the iteration number and trigger the initialization module 203.

[0128] The second calculation unit 206 is configured to calculate the network loss and the income at this period after the source network load storage controllable resource responds to the control decision, and generate the action chain of the agent agent max and the agent agent min and store the action chain in the experience replay pool, and extract experience data from the experience replay pool to calculate the time difference error of the agent agent max and the agent agent min .

[0129] The updating unit 207 is configured to update the parameters of the evaluation network and the parameters of the target network according to the time difference error, and judge whether the current period t is greater than the total period T, if yes, end the optimization, otherwise, trigger the first calculation unit 204 after updating the period.

[0130] Further, the embodiment of the present application further provides a demand response closed-loop control device, the device comprising a processor and a memory:

[0131] The memory is configured to store program code and transmit the program code to the processor.

[0132] The processor is configured to execute the steps of the demand response closed-loop control method according to the instructions in the program code.

[0133] Further, the embodiment of the present application further provides a computer readable storage medium, which is configured to store program code, and the program code is configured to execute the demand response closed-loop control method.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0135] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of accomplishing functionalities that are either the same or similar to that of other embodiments of the application. Moreover, the terms "include", "have", and the like, are used in the detailed description and in the claims of this application essentially open- ended and are intended to encompass the items listed thereafter, equivalents thereof, as well as additional items not listed after the comma. Finally, terms of degree such as "substantially", "approximately", and the like, are used herein to convey an intended precision or range of values end points of which are known to be subject to, and can have been determined by, typical measurement and manufacturing tolerances as well as other factors or limitations that are expected to be within the scope of one having ordinary skill in the art. Such terms of degree are not intended to exclude the aforementioned absolute precisions and ranges.

[0136] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or", used to describe the relationship between associated objects, means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including single or multiple combinations of any combination. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0137] In several embodiments provided in the application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the above-described device embodiments are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0138] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0139] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0140] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods according to the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), a random access memory (English full name: Random Access Memory, English abbreviation: RAM), a magnetic disk or an optical disk, and various other media that can store program codes.

[0141] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A demand response closed loop control method, characterized by, Be applied to the technical architecture that is constituted by distribution network dispatching center, microgrid operator, source network load storage controllable resource, wherein, the microgrid operator controls the source network load storage controllable resource through microgrid controller; The method comprises: S1, with minimization distribution network loss and microgrid operator benefit weighted difference as optimization goal, establish grid-communication network dual-network operation uncertainty two-stage robust optimization model; S2, the microgrid operator builds an agent and the agent to solve the maximization problem and the minimization problem, and design the agent and the agent corresponding to the state space based on the blackout burst awareness to determine the emergency regulation preference set of the controllable resources of the node, and design the action space and the reward function, wherein the agent and the agent both include corresponding evaluation networks, target networks and experience replay pools; S3, initialize iteration number, and initialize iteration stop condition according to the reward function, and the uncertain variable of the grid-communication network dual-network operation uncertainty two-stage robust optimization model; S4, by the intelligent agent and the intelligent agent determining a current observable state and inputting to the evaluation network and adopting selecting a first action with the maximum state-action value of the current state, while the intelligent agent and the intelligent agent executing the first action and calculating a reward; S5, judge whether the iteration stop condition is met, if yes, output the optimal solution of the grid-communication network dual-network operation uncertainty two-stage robust optimization model at this period, and issue the control decision to the microgrid controller, otherwise update iteration number and return to step S4; S6、when the source network load storage controllable resource responds to the regulation and control decision, calculate the network loss and benefit of this period, and generate an agent and the action chain of the agent , and store them in the experience replay pool, and extract experience data from the experience replay pool to calculate the time difference error of the agent and the agent respectively; S7, update the parameters of the evaluation network and the parameters of the target network according to the time series difference error, and judge whether the current period t is greater than the total period T, if yes, end optimization, otherwise return to step S3 after updating the period.

2. The demand response closed loop control method of claim 1, wherein, The grid-communication network dual-network operation uncertainty two-stage robust optimization model is expressed as: ； Wherein, the constraint condition of the grid-communication network dual-network operation uncertainty two-stage robust optimization model includes: distribution network branch flow constraint, node voltage and line current safety constraint and controllable resource output constraint; wherein, is the optimization objective, representing the weighted difference between the minimization of the distribution network loss and the microgrid operator revenue, is the set of deterministic decision variables, consisting of the set of source-grid-load-storage controllable resource outputs, is the set of uncertain variables, consisting of the set of grid node uncertainty factors , taking values in the interval [-1, 1] arbitrarily.

3. The demand response closed loop control method of claim 1, wherein, The state space based on burst power outage sensing is expressed as: ； wherein, , and are respectively the demand response incentive price paid by the distribution grid to the distributed photovoltaic, energy storage device and controllable load in the microgrid operator, is the electricity price of the energy storage device purchasing electricity from the distribution grid, and the specific price depends on the time-of-use electricity price of the distribution grid; is the set of stored energy of the node energy storage device in the time period, and are respectively the reward output of the last iteration of another agent; is a sudden power failure indication variable, when the distribution grid suddenly loses power, , otherwise ; is the set of emergency regulation preferences of the node controllable resource, wherein represents the specific power equipment type participating in the emergency response at the node in the time period, is the emergency regulation preference of the node in the time period, and is specifically represented as: ； In the formula, A controllable load set that still retains demand response capability under sudden power failure; A power supply priority for the controllable load in the time period, which is divided into four priorities, i.e., critical equipment, important facilities, general commercial and residential power, and non-essential power, and the higher the power supply priority of the controllable load is, the smaller the value of the power supply priority is; A power supply priority for the controllable load in the time period, which is divided into four priorities, i.e., critical equipment, important facilities, general commercial and residential power, and non-essential power, and the higher the power supply priority of the controllable load is, the smaller the value of the power supply priority is; The smaller the value of the power supply priority is, the higher the power supply priority of the controllable load is; A price paid by the power distribution network for emergency response in the time period, which depends on the time-of-use price of the source, network, load, and storage controllable resource type; Indicates the adjustable active power of the source, network, load, and storage controllable resource type; Indicates the adjustable active power of the source, network, load, and storage controllable resource type; A historical average response performance index, which is represented as: ； In the formula, , , respectively represent the average duration response time, response speed and response reliability of the power equipment demand response obtained according to historical experience information.

4. The demand response closed loop control method of claim 1, wherein, the agent and the action space of the agent is represented as: ； ； The real-time reward function is expressed as: ； wherein, to optimize the objective of the problem, i.e., to minimize the weighted difference between the power grid loss and the microgrid operator revenue; to balance the weights; to be a larger real number; to be an event function, if the branch power flow constraint is satisfied, then , otherwise ; similarly, if the controllable resource output constraint is satisfied, then , otherwise .

5. The demand response closed loop control method of claim 4, wherein, The iteration stop condition is initialized according to the reward function, and the iteration stop condition is initialized according to the reward function. ； In the formula, is a distance parameter.

6. The demand response closed loop control method of claim 1, wherein, The time series difference error is expressed as: ； In the formula, represents the number of action chains extracted from the experience pool; is the first time period agent action space; represents the state-action value function; represents the evaluation network; represents the state-action value updated by the target network after the distribution network master station obtains the reward , and the update formula of the target network is: ； In the formula, is a discount factor, used to measure the influence of future time slot rewards on the current state-action value, the greater the value, the greater the influence of future rewards on the current action policy, and the algorithm pays more attention to long-term benefits; represents the reward value; represents the next state; represents the next action; represents the target network.

7. The demand response closed loop control method of claim 1, wherein, The update formula of the parameters of the evaluation network is: ； wherein represents divided by the remainder.

8. A demand response closed loop control system characterized by, Be applied to the technical architecture that is constituted by distribution network dispatching center, microgrid operator, source network load storage controllable resource, wherein, the microgrid operator controls the source network load storage controllable resource through microgrid controller; The system comprises: A first construction unit is configured to establish a grid-communication network dual-network operation uncertainty two-stage robust optimization model with minimization distribution network loss and microgrid operator benefit weighted difference as optimization goal; a second construction unit for constructing an agent by the micro-grid operator and the agent to solve the maximization problem and the minimization problem, and design the agent and the agent corresponding to the state space based on the blackout burst awareness to determine the emergency regulation preference set of the controllable resources of the node, and design the action space and the reward function, wherein the agent and the agent both include corresponding evaluation networks, target networks and experience replay pools; An initialization unit is configured to initialize iteration number, and initialize iteration stop condition according to the reward function, and the uncertain variable of the grid-communication network dual-network operation uncertainty two-stage robust optimization model; A first computing unit is configured to determine a current observable state of the agent and input the current observable state into the evaluation network, and adopt a policy to select a first action with a maximum state-action value for the current state, and the agent performs the first action and calculates a reward and the agent performs the first action and calculates a reward A judgment unit is configured to judge whether the iteration stop condition is met, if yes, output the optimal solution of the grid-communication network dual-network operation uncertainty two-stage robust optimization model at this period, and issue the control decision to the microgrid controller, otherwise update iteration number and trigger the first calculation unit; a second computing unit, configured to calculate the network loss and the benefit of the period when the source network load controllable resource responds to the regulation decision, and generate an agent and an action chain of the agent and store them in the experience replay pool, and extract experience data from the experience replay pool to calculate the time difference error of the agent and the agent respectively. An update unit is configured to update the parameters of the evaluation network and the parameters of the target network according to the time series difference error, and judge whether the current period t is greater than the total period T, if yes, end optimization, otherwise trigger the initialization unit after updating the period.

9. A demand response closed loop control device, characterized by, The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor. The processor is configured to execute the demand response closed-loop control method according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program code for executing the demand response closed-loop control method according to any one of claims 1-7.

Images