Power grid coordination control method and device based on deep reinforcement learning

Abstract

The invention relates to the technical field of power grid coordination control, and discloses a power grid coordination control method and device based on deep reinforcement learning. The method comprises the following steps: acquiring historical operation state information of the novel power system; constructing a state space, an action space and a reward function of the prevention and control model; constructing a state space, an action space and a reward function of the emergency control model; constructing a global reward function based on the reward function of the prevention control model and the reward function of the emergency control model; based on the historical operation state information and the global reward function, performing model training on the prevention control model and the emergency control model; and according to the real-time operation state information of the novel power system, generating a real-time control strategy by using the trained prevention control model and emergency control model, and adjusting the novel power system by using the real-time control strategy. According to the invention, the control flexibility of the power grid system is improved, the consistency and coordination of the prevention control stage and the emergency control stage are ensured, and the control effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power grid coordinated control, and in particular to a power grid coordinated control method and device based on deep reinforcement learning. Background Art

[0002] As the scale of new energy installations gradually expands, new energy units are gradually replacing thermal power units, which leads to a gradual reduction in synchronous generators that maintain voltage and frequency stability in the power system. The high proportion of photovoltaic and wind power access leads to a lack of active reserve capacity in the system. When the wind and solar power output fluctuates or the system fails, it is easy to cause active power imbalance and grid frequency exceeding the limit, which in turn causes a series of problems such as system instability, and even chain failures may occur, leading to serious consequences such as grid decoupling. Therefore, how to make wind, solar and energy storage actively participate in grid safety control can effectively improve the rapid response capability in emergency situations and ensure the safe and stable operation of the system. This is a difficult problem that needs to be solved in the development of new power systems. The current emergency control strategy of the power grid is often based on the traditional power system architecture, which takes more consideration of the characteristics of traditional power sources such as thermal power and hydropower, and considers less the participation of renewable energy such as wind power and photovoltaics in the new power system. For the new power system, wind turbines and photovoltaic power stations have the ability to continuously and smoothly control active power. They can participate in system emergency control by reserving active power. At the same time, energy storage stations can solve the problem of strong uncertainty in wind and solar output power. Therefore, when a power grid fails, it is urgent to propose an emergency control strategy that combines the technical performance advantages of wind, solar and storage with the characteristics of reinforcement learning such as "real-time decision-making and real-time control".

[0003] In power systems with a high proportion of renewable energy, traditional hydropower and thermal power account for a small proportion, and their active reserve capacity is significantly reduced compared with traditional power systems dominated by synchronous machines, and their risk resistance is weak. When a sudden fault occurs in the system, the active reserve capacity usually cannot meet the emergency control needs of the system. When the system power is insufficient, in order to protect the stability of the system, load shedding is generally adopted. Although this can quickly reduce the pressure on the power system, when the load is cut off, the power system will be exposed to high-risk instability after control. This instability will spread to cascade events, resulting in higher costs than excessive cut-off events. The existing control method for fixed reserve capacity of wind and solar power systems with a high proportion cannot simultaneously ensure the coexistence of reducing system risks and economy. Although wind and solar power generation systems can increase the system reserve capacity by active reserve, it is aimed at the working conditions when no fault occurs, and the margin of reserve capacity may not match the active power required for emergency control after the fault occurs. When the emergency control quantity does not match, it will cause many problems in the power grid, and even cause serious consequences such as chain failures and power grid decoupling. Therefore, how to coordinate the amount of wind and solar power curtailment with the emergency control required after a fault is an urgent problem that needs to be solved when considering wind and solar power output in emergency control. Summary of the invention

[0004] The present invention provides a power grid coordinated control method and device based on deep reinforcement learning, which improves the flexibility of power grid system control, ensures the consistency and coordination of the preventive control stage and the emergency control stage, and improves the control effect.

[0005] In order to solve the above technical problems, the present invention provides a power grid coordinated control method based on deep reinforcement learning, comprising:

[0006] Obtain historical operating status information of new power systems;

[0007] Construct the state space, action space and reward function of the preventive control model;

[0008] Construct the state space, action space, and reward function of the emergency control model;

[0009] Constructing a global reward function based on the reward function of the preventive control model and the reward function of the emergency control model;

[0010] Based on the historical operating state information and the global reward function, performing model training on the preventive control model and the emergency control model;

[0011] According to the real-time operation status information of the new power system, a real-time control strategy is generated by using the trained preventive control model and the emergency control model, and the new power system is adjusted by using the real-time control strategy.

[0012] Furthermore, the state space, action space and reward function of the preventive control model are constructed, including:

[0013] The state space of the preventive control model is as follows:

[0014] S prevent = {U i ,θ i ,δ i ,ω i ,ω WTi ,P WTi ,U PVi ,P PVi ,SOC(t),P ESS (t)}

[0015] In the formula, S prevent is the state space of the preventive control model; U i is the voltage amplitude of the i-th node; θ i is the phase angle of the i-th node; δ i is the power angle of the i-th generator; ω i is the rotor speed of the i-th generator; ωWTi is the rotor speed of the i-th fan; P WTi is the output power of the i-th fan; U PVi is the output voltage of the ith photovoltaic unit; P PVi is the output power of the i-th photovoltaic system; SOC(t) is the state of charge of the i-th energy storage system at time t; P ESS (t) is the charging and discharging power of the i-th energy storage system at time t;

[0016] The action space of the preventive control model is as follows:

[0017]

[0018] In the formula, A prevent is the action space of the preventive control model; ΔP Gi The output of the adjusted generator i; is the active power cut by the i-th wind turbine; is the active power reduced by the i-th photovoltaic unit; P ESS,dis,n is the discharge power of energy storage unit n; P ESS,ch,r is the charging power of energy storage unit r;

[0019] The reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, specifically:

[0020]

[0021] In the formula, R pre is the reward function of the preventive control model; F cost (t) is the cost operation function; is the preventive control constraint function; K is the penalty for non-convergence of the power flow; α and β are weight coefficients respectively; E imbalance It is the energy imbalance measure of the new power system; reserve Wind and solar active power reserve preset for new power systems.

[0022] Furthermore, the reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, including:

[0023] The cost operation function is specifically:

[0024]

[0025] In the formula, F cost (t) is the cost operation function; C G (t) is the generator output adjustment cost at time t; C WT (t) is the wind turbine output reduction cost at time t; C PV(t) is the photovoltaic output reduction cost at time t; C ESS (t) is the charge / discharge state adjustment cost of the energy storage unit at time t.

[0026] Furthermore, the reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, including:

[0027] The preventive control constraint function is determined based on a number of constraints and constraint coefficients corresponding to the constraints; wherein the constraints include power flow constraints, voltage and phase angle amplitude constraints at each node, generator output constraints, wind turbine and photovoltaic output reduction constraints, energy storage unit charging and discharging constraints, and line power limit constraints.

[0028] Furthermore, the constraints include power flow constraints, voltage and phase angle amplitude constraints of each node, generator output constraints, wind turbine and photovoltaic output reduction constraints, energy storage unit charging and discharging constraints, and line power limit constraints, including:

[0029] The power flow constraints are specifically:

[0030]

[0031] Where P G,i is the active power generated by node i; Q G,i is the reactive power of node i; P D,i is the active power of node i; Q D,i is the reactive power demand of node i; G ij is the conductance between node i and node j; B ij is the susceptance between node i and node j; U i is the voltage amplitude of the ith node; U j is the voltage amplitude of the ith node; N0 is the node set except the equilibrium node; N PQ is the set of PQ nodes; δ ij is the voltage phase angle difference between node i and node j; n is the total number of all nodes in the new power system;

[0032] The voltage and phase angle amplitude constraints of each node are specifically as follows:

[0033] 0.95u Ni ≤u i,t ≤1.05u Ni

[0034] θ i,min ≤θ i ≤θ i,max

[0035] In the formula, u i,t is the voltage of node i at time t; uNi is the rated voltage of node i; θ i is the node phase angle; θ i,max is the maximum phase angle of the node; θ i,min is the minimum phase angle of the node;

[0036] The generator output constraint is specifically:

[0037] P Gi,min ≤P Gi ≤P Gi,max i=1,2,…,N G

[0038] Where P Gi P is the operating power of the generator; Gi,min The minimum output power of the generator; P Gi,max Maximum output power for generator operation;

[0039] The wind turbine and photovoltaic output reduction constraints are specifically:

[0040]

[0041]

[0042] In the formula, is the active power cut by the i-th wind turbine; P WT,max is the maximum output power of the wind turbine; is the active power reduced by the i-th photovoltaic unit; P PV,max is the maximum output power of the photovoltaic unit;

[0043] The energy storage unit charging and discharging constraints are specifically:

[0044] 0≤P ESS,ch,r ≤P ESS,ch,max U ESS,ch

[0045] 0≤P ESS,dis,n ≤P ESS,dis,max U ESS,dis

[0046] U ESS,ch +U ESS,dis ≤1

[0047] Where P ESS,ch,r Charging power for energy storage unit; P ESS,ch,max is the maximum charging power of the energy storage unit; U ESS,ch It is the charging status flag of the energy storage unit; P ESS,dis,n is the discharge power of the energy storage unit; P ESS,dis,max is the maximum discharge power of the energy storage unit; U ESS,disIt is the discharge status flag of the energy storage unit;

[0048] The line power limit constraint is specifically:

[0049]

[0050] In the formula, is the power of line l; is the power limit of line l.

[0051] Furthermore, the state space, action space and reward function of the emergency control model are constructed, including:

[0052] The state space of the emergency control model is as follows:

[0053]

[0054] In the formula, S em is the state space of the emergency control model; U i is the voltage amplitude of the i-th node; θ i is the phase angle of the i-th node; δ i is the power angle of the i-th generator; ω i is the rotor speed of the i-th generator; ω WTi is the rotor speed of the i-th fan; P WTi is the output power of the i-th fan; U PVi is the output voltage of the ith photovoltaic unit; P PVi is the output power of the i-th photovoltaic unit; is the active power cut by the i-th wind turbine; is the active power cut by the i-th photovoltaic unit; ΔP dis is the actual power imbalance after a fault occurs in the power system;

[0055] The action space of the emergency control model is as follows:

[0056]

[0057] In the formula, A em is the action space of the emergency control model; is the active power removed by generator i; is the load shedding amount of node i; The active power adjusted upward or downward for the wind turbines involved in the regulation; The active power adjusted upward or downward by the photovoltaic units involved in the regulation;

[0058] The reward function of the emergency control model is as follows:

[0059]

[0060] In the formula, R em is the reward function of the emergency control model; P deficit The power gap that remains after wind, solar and storage participate in emergency control; C loadcut is the load shedding cost; C G Removal cost for the unit; is the emergency control constraint function; β1, β2 and β3 are weight coefficients respectively.

[0061] Furthermore, the reward function based on the preventive control model and the reward function of the emergency control model is used to construct a global reward function, which is specifically:

[0062] Based on the first adjustment coefficient and the second adjustment coefficient, dynamically adjusting the reward function of the preventive control model and the reward function of the emergency control model to form a global reward function;

[0063] Wherein, the global reward function is specifically:

[0064] R global =λ1R pre +λ2R em

[0065] In the formula, R global is the global function; λ1 is the first adjustment coefficient; λ2 is the second adjustment coefficient; R pre is the reward function of the preventive control model; R em is the reward function of the emergency control model.

[0066] Furthermore, based on the historical operating state information and the global reward function, the preventive control model and the emergency control model are trained as follows:

[0067] Taking maximizing the global reward function as the goal, the preventive control model and the emergency control model are trained using the historical operating state information, specifically:

[0068] Based on the historical operation status information, and the model strategies of the preventive control model and the emergency control model, generate a plurality of first samples;

[0069] Utilize each of the first samples to cyclically update the model Q value and calculate the corresponding loss function;

[0070] Using the gradient descent method, the network weights are updated according to the loss function;

[0071] When the change in the global reward function is within a preset range or the number of cycles reaches a preset update iteration amount, it is determined that the training of the preventive control model and the emergency control model is completed.

[0072] Furthermore, the generating of a plurality of first samples based on the historical operation status information and the model strategies of the preventive control model and the emergency control model is specifically:

[0073] Determine a first environmental state according to the historical operating state information, and select a first action using the first strategy of the preventive control model and the emergency control model;

[0074] Obtaining an environment feedback reward and a second environment state according to the first action;

[0075] The first environment state, the first action, the environment feedback reward and the second environment state are determined as a first sample in a sample experience library.

[0076] The present invention provides a power grid coordinated control method based on deep reinforcement learning, which obtains historical operating status information of a new power system; constructs the state space, action space and reward function of a preventive control model; constructs the state space, action space and reward function of an emergency control model; constructs a global reward function based on the reward function of the preventive control model and the reward function of the emergency control model; performs model training on the preventive control model and the emergency control model based on the historical operating status information and the global reward function; generates a real-time control strategy using the trained preventive control model and the emergency control model according to the real-time operating status information of the new power system, and adjusts the new power system using the real-time control strategy. The present invention improves the flexibility of power grid system control, ensures the consistency and coordination of the preventive control stage and the emergency control stage, and improves the control effect.

[0077] Accordingly, the present invention provides a power grid coordination control device based on deep reinforcement learning, comprising: an acquisition module, a first model construction module, a second model construction module, a model combination module, a model training module and a control module;

[0078] The acquisition module is used to acquire historical operation status information of the new power system;

[0079] The first model building module is used to build the state space, action space and reward function of the preventive control model;

[0080] The second model building module is used to build the state space, action space and reward function of the emergency control model;

[0081] The model combination module is used to construct a global reward function based on the reward function of the preventive control model and the reward function of the emergency control model;

[0082] The model training module is used to perform model training on the preventive control model and the emergency control model based on the historical operating state information and the global reward function;

[0083] The control module is used to generate a real-time control strategy based on the real-time operating status information of the new power system using the trained preventive control model and the emergency control model, and adjust the new power system using the real-time control strategy.

[0084] The present invention provides a power grid coordination control device based on deep reinforcement learning, which obtains historical operation status information of a new power system based on the organic combination of modules; constructs the state space, action space and reward function of a preventive control model; constructs the state space, action space and reward function of an emergency control model; constructs a global reward function based on the reward function of the preventive control model and the reward function of the emergency control model; performs model training on the preventive control model and the emergency control model based on the historical operation status information and the global reward function; generates a real-time control strategy using the trained preventive control model and the emergency control model according to the real-time operation status information of the new power system, and adjusts the new power system using the real-time control strategy. The present invention improves the flexibility of power grid system control, ensures the consistency and coordination of the preventive control stage and the emergency control stage, and improves the control effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A flowchart of an embodiment of a power grid coordinated control method based on deep reinforcement learning provided by the present invention;

[0086] Figure 2 A flowchart of another embodiment of a power grid coordinated control method based on deep reinforcement learning provided by the present invention;

[0087] Figure 3 A schematic structural diagram of an embodiment of a power grid coordination control device based on deep reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0090] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0091] Example 1

[0092] See also Figure 1 , is a flow chart of an embodiment of a power grid coordinated control method based on deep reinforcement learning provided by the present invention, the method includes steps 101 to 106, each step is specifically as follows:

[0093] Step 101: Obtain historical operating status information of the new power system.

[0094] In the first embodiment of the present invention, historical operating status information of the new power system is obtained for training a preventive control model and an emergency control model. After training, the preventive control model and the emergency control model can generate corresponding control strategies according to the current operating status information of the new power system to control the new power system. The historical operating status information of the new power system includes new energy units, synchronous generators, energy storage systems, transmission lines, etc.

[0095] In the first embodiment of the present invention, the preventive control process and the emergency control process of the new power system are modeled as Markov decision processes, and a preventive deep reinforcement learning agent and an emergency control deep reinforcement learning agent for the new power system with the participation of wind, solar and storage can be established respectively, so as to train the preventive control model and the emergency control model of the new power system. Among them, the agent includes a reward function, a state space, an action space, a target network and an estimation network, and finally a deep reinforcement learning agent model can be formed.

[0096] Step 102: Construct the state space, action space and reward function of the preventive control model.

[0097] Furthermore, in the first embodiment of the present invention, the state space, action space and reward function of the preventive control model are constructed, including:

[0098] The state space of the preventive control model is as follows:

[0099] S prevent = {U i ,θ i ,δ i ,ω i ,ωWTi ,P WTi ,U PVi ,P PVi ,SOC(t),P ESS (t)}

[0100] In the formula, S prevent is the state space of the preventive control model; U i is the voltage amplitude of the i-th node; θ i is the phase angle of the i-th node; δ i is the power angle of the i-th generator; ω i is the rotor speed of the i-th generator; ω WTi is the rotor speed of the i-th fan; P WTi is the output power of the i-th fan; U PVi is the output voltage of the ith photovoltaic unit; P PVi is the output power of the i-th photovoltaic system; SOC(t) is the state of charge of the i-th energy storage system at time t; P ESS (t) is the charging and discharging power of the i-th energy storage system at time t;

[0101] The action space of the preventive control model is as follows:

[0102]

[0103] In the formula, A prevent is the action space of the preventive control model; ΔP Gi The output of the adjusted generator i; is the active power cut by the i-th wind turbine; is the active power reduced by the i-th photovoltaic unit; P ESS,dis,n is the discharge power of energy storage unit n; P ESS,ch,r is the charging power of energy storage unit r;

[0104] The reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, specifically:

[0105]

[0106] In the formula, R pre is the reward function of the preventive control model; F cost (t) is the cost operation function; is the preventive control constraint function; K is the penalty for non-convergence of the power flow; α and β are weight coefficients respectively; E imbalance It is the energy imbalance measure of the new power system; reserve Wind and solar active power reserve preset for new power systems.

[0107] In the first embodiment of the present invention, the state space, action space and reward function of the preventive control model based on deep reinforcement learning are described above. The intelligent agent adjusts the wind and solar active reserve capacity, synchronous generator rotating reserve capacity and energy storage charging and discharging power by collecting the real-time operating status of the system and calculating the expected fault results under this state.

[0108] In the first embodiment of the present invention, the energy imbalance of the new power system in the reward function of the preventive control model is calculated by real-time monitoring of the load and power generation of the system, and the system energy imbalance is automatically calculated as a difference. The difference changes with the dynamic operation of the system and reflects the equilibrium state of the system in real time. From the setting of the reward function, it can be known that the system should reduce the system energy imbalance as much as possible. The preset wind and solar active reserve of the new power system in the reward function of the preventive control model refers to the adjustable power retained in normal operation to cope with sudden failures or load fluctuations. This value will be dynamically adjusted according to the system status, scheduling requirements and wind and solar power generation forecasts.

[0109] Furthermore, in the first embodiment of the present invention, the reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, including:

[0110] The cost operation function is specifically:

[0111]

[0112] In the formula, F cost (t) is the cost operation function; C G (t) is the generator output adjustment cost at time t; C WT (t) is the wind turbine output reduction cost at time t; C PV (t) is the photovoltaic output reduction cost at time t; C ESS (t) is the charge / discharge state adjustment cost of the energy storage unit at time t.

[0113] In the first embodiment of the present invention, the reward function is used to evaluate the quality of the agent's actions and guide the agent to adjust its parameters to maximize the expected reward, which is the key to the final control decision. The reward function of the preventive control agent includes an operating cost item and a constraint item. Among them, the operating cost item is set so that the smaller the system operating cost in the control period t, the greater the reward given to each agent in this period.

[0114] Furthermore, in the first embodiment of the present invention, the reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, including:

[0115] The preventive control constraint function is determined based on a number of constraints and constraint coefficients corresponding to the constraints; wherein the constraints include power flow constraints, voltage and phase angle amplitude constraints at each node, generator output constraints, wind turbine and photovoltaic output reduction constraints, energy storage unit charging and discharging constraints, and line power limit constraints.

[0116] In the first embodiment of the present invention, the preventive control constraint function needs to consider various constraints of the preventive control agent, and the preventive control constraint function is composed of various constraints and their corresponding constraint coefficients. Among them, the constraint coefficient can be determined by the judgment function The function of the judgment function is to judge whether the constraint is satisfied. When the constraint is satisfied, the function value of the judgment function is determined to be 1; when the constraint is not satisfied, the function value of the judgment function is determined to be 0.

[0117] Further, in the first embodiment of the present invention, the constraints include power flow constraints, voltage and phase angle amplitude constraints of each node, generator output constraints, wind turbine and photovoltaic output reduction constraints, energy storage unit charging and discharging constraints, and line power limit constraints, including:

[0118] The power flow constraints are specifically:

[0119]

[0120] Where P G,i is the active power generated by node i; Q G,i is the reactive power of node i; P D,i is the active power of node i; Q D,i is the reactive power demand of node i; G ij is the conductance between node i and node j; B ij is the susceptance between node i and node j; U i is the voltage amplitude of the ith node; U j is the voltage amplitude of the ith node; N0 is the node set except the equilibrium node; N PQ is the set of PQ nodes; δ ij is the voltage phase angle difference between node i and node j; n is the total number of all nodes in the new power system;

[0121] The voltage and phase angle amplitude constraints of each node are specifically as follows:

[0122] 0.95u Ni ≤u i,t ≤1.05u Ni

[0123] θ i,min ≤θ i ≤θ i,max

[0124] In the formula, u i,t is the voltage of node i at time t; u Ni is the rated voltage of node i; θ i is the node phase angle; θ i,max is the maximum phase angle of the node; θ i,min is the minimum phase angle of the node;

[0125] The generator output constraint is specifically:

[0126] P Gi,min ≤P Gi ≤P Gi,max i=1,2,…,N G

[0127] Where P Gi P is the operating power of the generator; Gi,min The minimum output power of the generator; P Gi,max Maximum output power for generator operation;

[0128] The wind turbine and photovoltaic output reduction constraints are specifically:

[0129]

[0130] In the formula, is the active power cut by the i-th wind turbine; P WT,max is the maximum output power of the wind turbine; is the active power reduced by the i-th photovoltaic unit; P PV,max is the maximum output power of the photovoltaic unit;

[0131] The energy storage unit charging and discharging constraints are specifically:

[0132] 0≤P ESS,ch,r ≤P ESS,ch,max U ESS,ch

[0133] 0≤P ESS,dis,n ≤P ESS,dis,max U ESS,dis

[0134] U ESS,ch +U ESS,dis ≤1

[0135] Where P ESS,ch,r Charging power for energy storage unit; P ESS,ch,max is the maximum charging power of the energy storage unit; U ESS,ch It is the charging status flag of the energy storage unit; P ESS,dis,n is the discharge power of the energy storage unit; P ESS,dis,max is the maximum discharge power of the energy storage unit; UESS,dis It is the discharge status flag of the energy storage unit;

[0136] The line power limit constraint is specifically:

[0137]

[0138] In the formula, is the power of line l; is the power limit of line l.

[0139] In the first embodiment of the present invention, U in the energy storage unit charge and discharge constraints ESS,ch It is the charging status flag of the energy storage unit, U ESS,dis It is the discharge status flag of the energy storage unit; its flag value is between 0 and 1. In the charging state: U ESS,ch =1,U ESS,dis =0, in the discharge state U ESS,dis =1,U ESS,ch =0.

[0140] In the first embodiment of the present invention, the preventive control model constructed above can output a preventive control plan through the set expected faults. If the system still suffers a certain expected fault due to a sudden serious fault or other reasons, causing a large amount of power imbalance in the system, in this case, emergency control is urgently needed to reduce the losses caused by the fault.

[0141] Step 103: Construct the state space, action space and reward function of the emergency control model.

[0142] Furthermore, in the first embodiment of the present invention, the state space, action space and reward function of the emergency control model are constructed, including:

[0143] The state space of the emergency control model is as follows:

[0144]

[0145] In the formula, S em is the state space of the emergency control model; U i is the voltage amplitude of the i-th node; θ i is the phase angle of the i-th node; δ i is the power angle of the i-th generator; ω i is the rotor speed of the i-th generator; ω WTi is the rotor speed of the i-th fan; P WTi is the output power of the i-th fan; U PVi is the output voltage of the ith photovoltaic unit; P PVi is the output power of the i-th photovoltaic unit; is the active power cut by the i-th wind turbine; is the active power cut by the i-th photovoltaic unit; ΔP dis is the actual power imbalance after a fault occurs in the power system;

[0146] The action space of the emergency control model is as follows:

[0147]

[0148] In the formula, A em is the action space of the emergency control model; is the active power removed by generator i; is the load shedding amount of node i; The active power adjusted upward or downward for the wind turbines involved in the regulation; The active power adjusted upward or downward by the photovoltaic units involved in the regulation;

[0149] The reward function of the emergency control model is as follows:

[0150]

[0151] In the formula, R em is the reward function of the emergency control model; P deficit The power gap that remains after wind, solar and storage participate in emergency control; C loadcut is the load shedding cost; C G Removal cost for the unit; is the emergency control constraint function; β1, β2 and β3 are weight coefficients respectively.

[0152] In the first embodiment of the present invention, the state space, action space and reward function of the emergency control model based on deep reinforcement learning are described above. The reward function of the emergency control model is based on P deficit , C loadcut , C G and Setting. Among them, P deficit is the power gap that remains after wind, solar and storage participate in emergency control; C loadcut is the load shedding cost. When there is a power gap in emergency control, the load shedding with the lowest cost should be selected. G is the unit removal cost. In case of emergency control, the unit with the lowest cost should be removed. is the emergency control constraint function.

[0153] Step 104: construct a global reward function based on the reward function of the preventive control model and the reward function of the emergency control model.

[0154] Furthermore, in the first embodiment of the present invention, a global reward function is constructed based on the reward function of the preventive control model and the reward function of the emergency control model, specifically:

[0155] Based on the first adjustment coefficient and the second adjustment coefficient, dynamically adjusting the reward function of the preventive control model and the reward function of the emergency control model to form a global reward function;

[0156] Wherein, the global reward function is specifically:

[0157] R global =λ1R pre +λ2R em

[0158] In the formula, R global is the global function; λ1 is the first adjustment coefficient; λ2 is the second adjustment coefficient; R pre is the reward function of the preventive control model; R em is the reward function of the emergency control model.

[0159] In the first embodiment of the present invention, a global reward function is introduced to ensure that both the preventive and emergency control agents focus on the global goal. The global reward function is provided with a first adjustment coefficient and a second adjustment coefficient, which can be dynamically adjusted to balance the contributions of the preventive control and emergency control stages according to system requirements.

[0160] Step 105: Based on the historical operating status information and the global reward function, the preventive control model and the emergency control model are trained.

[0161] Furthermore, in the first embodiment of the present invention, based on the historical operating state information and the global reward function, the preventive control model and the emergency control model are trained as follows:

[0162] Taking maximizing the global reward function as the goal, the preventive control model and the emergency control model are trained using the historical operating state information, specifically:

[0163] Based on the historical operation status information, and the model strategies of the preventive control model and the emergency control model, generate a plurality of first samples;

[0164] Utilize each of the first samples to cyclically update the model Q value and calculate the corresponding loss function;

[0165] Using the gradient descent method, the network weights are updated according to the loss function;

[0166] When the change in the global reward function is within a preset range or the number of cycles reaches a preset update iteration amount, it is determined that the training of the preventive control model and the emergency control model is completed.

[0167] Furthermore, in the first embodiment of the present invention, based on the historical operation status information and the model strategies of the preventive control model and the emergency control model, a plurality of first samples are generated, specifically:

[0168] Determine a first environmental state according to the historical operating state information, and select a first action using the first strategy of the preventive control model and the emergency control model;

[0169] Obtaining an environment feedback reward and a second environment state according to the first action;

[0170] The first environment state, the first action, the environment feedback reward and the second environment state are determined as a first sample in a sample experience library.

[0171] In the first embodiment of the present invention, the preventive control model and the emergency control model in the initial stage of training can be trained offline based on the deep Q network. The training goal is to find the optimal strategy to maximize the cumulative reward. The learning and exploration training process ends when the global reward converges.

[0172] In the first embodiment of the present invention, in the process of training the preventive control model and the emergency control model, the optimal strategies of the two agents are obtained by learning the deep Q network (DQN) algorithm with the goal of maximizing the cumulative reward, and the action value function Q value is iteratively updated using the estimation network and the target network. First, the environment and the model are initialized, the agent collects the initial state of the system, selects the action using the ε-greedy strategy, performs global exploration with a small probability ε, and selects the action with the most action value function using the greedy algorithm with a high probability 1-ε, and obtains the environmental feedback reward r t and the new state of the environment s t+1 At the same time, the experience replay mechanism is adopted to store [s t ,a t ,r t ,s t+1 ] to the sample experience library D, so as to make full use of the explored experience learning, and randomly extract the minimum batch samples from the sample experience library D for training, update the Q value and calculate the loss function, and use the gradient descent method to update the network weight w according to the loss function. The target network weight w is updated every I iterations. When the neural network parameters converge and can adapt to certain uncertain changes, all action value function values ​​can be obtained, that is, a stable control strategy is formed.

[0173] In the first embodiment of the present invention, the expression of the action value function in the strategy π is:

[0174]

[0175] In the formula, represents the expected function; S t is the state at time t; A t is the action at time t; R t+β+1 is the reward for the βth step after time t; γ β is the discount factor, which reflects the degree of concern about the impact of future rewards; s is the state; a is the action.

[0176] In the first embodiment of the present invention, the target network Q value is:

[0177]

[0178] In the formula, Q target is the target network Q value; w - is the parameter in the target network; Q(s t+1 ,a′;w - ) is the Q value of the target network fitting for the next state; r t+1 is the immediate reward obtained at time t+1; γ is the discount factor, which is used to balance the weights of immediate rewards and future rewards.

[0179] In the first embodiment of the present invention, the update iteration amount of the estimation network is:

[0180] ΔQ predict (s t ,a t )=α(Q(s t ,a t ;w - )-Q(s t ,a t ;w t ))

[0181] Where ΔQ predict (s t ,a t ) is the update iteration amount of the estimated network; Q(s t+1 ,a′;w - ) is the Q value of the target network fitting for the next state; w t is the network parameter of the estimated network; α is the learning rate; Q(s t ,a t ;w t ) is the agent at time t, based on the current strategy network parameter w t For state s tand action a t The estimated Q value of

[0182] In the first embodiment of the present invention, the loss function of the neural network is expressed as the square of the temporal difference error:

[0183]

[0184] Where, L oss is the loss function; Q(s t+1 ,a′;w - ) is the Q value of the target network fitting for the next state; Q(s t ,a t ;w t ) is the agent at time t, based on the current strategy network parameter w t For state s t and action a t The estimated Q value of t+1 is the immediate reward obtained at time t+1; γ is the discount factor, which is used to balance the weights of immediate rewards and future rewards.

[0185] In the first embodiment of the present invention, during the training of the preventive control model and the emergency control model, the ε-greedy strategy is used to determine the action selected for execution each time. The specific expression is as follows:

[0186]

[0187] Where π(α|s) is the probability that the agent takes action a in state s; q * (s,a) is the optimal action value function; ε is the probability of choosing a non-optimal action.

[0188] In the first embodiment of the present invention, after the training of the preventive control model and the emergency control model is completed, the intelligent body output action is adjusted in the preventive control stage, that is, the generator output, the energy storage charging and discharging state and the measures for reserving wind and solar active standby are adjusted so that the system reserves active standby capacity before the fault occurs, so that the intelligent body can use the ability of wind and solar storage to continuously and smoothly adjust the active power in the emergency control stage to quickly provide active power support for the power grid. If its active power is still insufficient, the intelligent body will give priority to cutting off the load node with the lowest cost. If its active power is excessive, the intelligent body will give priority to reducing the wind and solar output to maintain power balance. If it is still insufficient, it will choose to cut off the unit with the lowest cutting cost.

[0189] Step 106: Generate a real-time control strategy based on the real-time operating status information of the new power system using the trained preventive control model and emergency control model, and adjust the new power system using the real-time control strategy.

[0190] In the first embodiment of the present invention, after the training of the preventive control model and the emergency control model is completed, their intelligent bodies are set in the real-time scene of the power system, and their state space inputs come from the wide-area measurement information obtained by the system through the wide-area measurement system and the phasor measurement unit. When the real-time operating status information of the new power system is obtained, the preventive control model and the emergency control model work together to realize the online rapid generation of control strategies.

[0191] As an example of the first embodiment of the present invention, see Figure 2 , is a flow chart of another embodiment of the power grid coordinated control method based on deep reinforcement learning provided by the present invention. When the initial operation state s0 of the power system is obtained, according to the preset expected fault set F (F1, F2, ..., F n ), using the preventive control agent to output the preventive control plan; when a specific fault occurs in the power system operation, the power system operation state s1 is collected, and according to the specific fault F i , using the emergency control agent to output the emergency control plan. Among them, the preventive control agent and the emergency control agent work together in the power system, so that wind, solar and storage actively participate in the safety control of the power grid, effectively improve the rapid response capability in emergency situations, and ensure the safe and stable operation of the system.

[0192] The present invention analyzes the system operating status through an intelligent agent and selects and reserves appropriate active reserve to deal with power grid failures and load fluctuations. It can not only participate in preventive control (such as stable regulation of voltage and frequency), but also provide support to the power grid through rapid response in emergency situations, reduce the system's machine cutting and load shedding, and take into account its economy.

[0193] The present invention reduces the wind and solar power output before a fault occurs, so that the wind and solar power can participate in emergency control, and utilizes its fast and flexible power control capabilities to reserve a certain amount of active power to deal with grid failures, load fluctuations or other emergency situations. By reasonably arranging the wind and solar power reduction, the active power reserved by the wind and solar power can be quickly put into use in a short time during the emergency control stage to fill the power gap in time without directly shedding the load, which greatly improves the flexibility of coordinated control. This flexible scheduling method provides more options for the system.

[0194] The present invention continuously optimizes the reduction strategy through continuous interaction between the intelligent agent and the environment, so that the reserved active reserve can more accurately match the power demand after the fault. At the same time, the control strategy starts from the global goal and designs a global reward function to ensure that the preventive control and emergency control strategies maintain consistency and coordination at different stages, thereby avoiding conflicts and improving the control effect.

[0195] In summary, the first embodiment of the present invention provides a power grid coordinated control method based on deep reinforcement learning, which obtains historical operating status information of a new power system; constructs the state space, action space and reward function of a preventive control model; constructs the state space, action space and reward function of an emergency control model; constructs a global reward function based on the reward function of the preventive control model and the reward function of the emergency control model; performs model training on the preventive control model and the emergency control model based on the historical operating status information and the global reward function; generates a real-time control strategy using the trained preventive control model and the emergency control model according to the real-time operating status information of the new power system, and uses the real-time control strategy to adjust the new power system. The present invention improves the flexibility of power grid system control, ensures the consistency and coordination of the preventive control stage and the emergency control stage, and improves the control effect.

[0196] Example 2

[0197] See also Figure 3 , is a structural diagram of an embodiment of a power grid coordinated control device based on deep reinforcement learning provided by the present invention, the device includes an acquisition module 201, a first model construction module 202, a second model construction module 203, a model combination module 204, a model training module 205 and a control module 206;

[0198] The acquisition module 201 is used to acquire the historical operation status information of the new power system;

[0199] The first model building module 202 is used to build the state space, action space and reward function of the preventive control model;

[0200] The second model building module 203 is used to build the state space, action space and reward function of the emergency control model;

[0201] The model combination module 204 is used to construct a global reward function based on the reward function of the preventive control model and the reward function of the emergency control model;

[0202] The model training module 205 is used to perform model training on the preventive control model and the emergency control model based on the historical operating state information and the global reward function;

[0203] The control module 206 is used to generate a real-time control strategy based on the real-time operating status information of the new power system using the trained preventive control model and emergency control model, and adjust the new power system using the real-time control strategy.

[0204] Furthermore, in the second embodiment of the present invention, the state space, action space and reward function of the preventive control model are constructed, including:

[0205] The state space of the preventive control model is as follows:

[0206] S prevent = {U i ,θ i ,δ i ,ω i ,ω WTi ,P WTi ,U PVi ,P PVi ,SOC(t),P ESS (t)}

[0207] In the formula, S prevent is the state space of the preventive control model; U i is the voltage amplitude of the i-th node; θ i is the phase angle of the i-th node; δ i is the power angle of the i-th generator; ω i is the rotor speed of the i-th generator; ω WTi is the rotor speed of the i-th fan; P WTi is the output power of the i-th fan; U PVi is the output voltage of the ith photovoltaic unit; P PVi is the output power of the i-th photovoltaic system; SOC(t) is the state of charge of the i-th energy storage system at time t; P ESS (t) is the charging and discharging power of the i-th energy storage system at time t;

[0208] The action space of the preventive control model is as follows:

[0209]

[0210] In the formula, A prevent is the action space of the preventive control model; ΔP Gi The output of the adjusted generator i; is the active power cut by the i-th wind turbine; is the active power reduced by the i-th photovoltaic unit; P ESS,dis,n is the discharge power of energy storage unit n; P ESS,ch,r is the charging power of energy storage unit r;

[0211] The reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, specifically:

[0212]

[0213] In the formula, R pre is the reward function of the preventive control model; F cost (t) is the cost operation function; is the preventive control constraint function; K is the penalty for non-convergence of the power flow; α and β are weight coefficients respectively; E imbalance It is the energy imbalance measure of the new power system; reserve Wind and solar active power reserve preset for new power systems.

[0214] Furthermore, in the second embodiment of the present invention, the reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, including:

[0215] The cost operation function is specifically:

[0216]

[0217] In the formula, F cost (t) is the cost operation function; C G (t) is the generator output adjustment cost at time t; C WT (t) is the wind turbine output reduction cost at time t; C PV (t) is the photovoltaic output reduction cost at time t; C ESS (t) is the charge / discharge state adjustment cost of the energy storage unit at time t.

[0218] Furthermore, in the second embodiment of the present invention, the reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, including:

[0219] The preventive control constraint function is determined based on a number of constraints and constraint coefficients corresponding to the constraints; wherein the constraints include power flow constraints, voltage and phase angle amplitude constraints at each node, generator output constraints, wind turbine and photovoltaic output reduction constraints, energy storage unit charging and discharging constraints, and line power limit constraints.

[0220] Further, in the second embodiment of the present invention, the constraints include power flow constraints, voltage and phase angle amplitude constraints of each node, generator output constraints, wind turbine and photovoltaic output reduction constraints, energy storage unit charging and discharging constraints, and line power limit constraints, including:

[0221] The power flow constraints are specifically:

[0222]

[0223] Where P G,i is the active power generated by node i; Q G,i is the reactive power of node i; P D,i is the active power of node i; Q D,i is the reactive power demand of node i; G ij is the conductance between node i and node j; B ijis the susceptance between node i and node j; U i is the voltage amplitude of the ith node; U j is the voltage amplitude of the ith node; N0 is the node set except the equilibrium node; N PQ is the set of PQ nodes; ij is the voltage phase angle difference between node i and node j; n is the total number of all nodes in the new power system;

[0224] The voltage and phase angle amplitude constraints of each node are specifically as follows:

[0225] 0.95u Ni ≤u i,t ≤1.05u Ni

[0226] θ i,min ≤θ i ≤θ i,max

[0227] In the formula, u i,t is the voltage of node i at time t; u Ni is the rated voltage of node i; θ i is the node phase angle; θ i,max is the maximum phase angle of the node; θ i,min is the minimum phase angle of the node;

[0228] The generator output constraint is specifically:

[0229] P Gi,min ≤P Gi ≤P Gi,max i=1,2,…,N G

[0230] Where P Gi P is the operating power of the generator; Gi,min The minimum output power of the generator; P Gi,max Maximum output power for generator operation;

[0231] The wind turbine and photovoltaic output reduction constraints are specifically:

[0232]

[0233] In the formula, is the active power cut by the i-th wind turbine; P WT,max is the maximum output power of the wind turbine; is the active power reduced by the i-th photovoltaic unit; P PV,max is the maximum output power of the photovoltaic unit;

[0234] The energy storage unit charging and discharging constraints are specifically:

[0235] 0≤P ESS,ch,r ≤P ESS,ch,max U ESS,ch

[0236] 0≤P ESS,dis,n ≤P ESS,dis,max U ESS,dis

[0237] U ESS,ch +U ESS,dis ≤1

[0238] Where P ESS,ch,r Charging power for energy storage unit; P ESS,ch,max is the maximum charging power of the energy storage unit; U ESS,ch It is the charging status flag of the energy storage unit; P ESS,dis,n is the discharge power of the energy storage unit; P ESS,dis,max is the maximum discharge power of the energy storage unit; U ESS,dis It is the discharge status flag of the energy storage unit;

[0239] The line power limit constraint is specifically:

[0240]

[0241] In the formula, is the power of line l; is the power limit of line l.

[0242] Furthermore, in the second embodiment of the present invention, the state space, action space and reward function of the emergency control model are constructed, including:

[0243] The state space of the emergency control model is as follows:

[0244]

[0245] In the formula, S em is the state space of the emergency control model; U i is the voltage amplitude of the i-th node; θ i is the phase angle of the i-th node; δ i is the power angle of the i-th generator; ω i is the rotor speed of the i-th generator; ω WTi is the rotor speed of the i-th fan; P WTi is the output power of the i-th fan; U PVi is the output voltage of the ith photovoltaic unit; P PVi is the output power of the i-th photovoltaic unit; is the active power cut by the i-th wind turbine; is the active power cut by the i-th photovoltaic unit; ΔP dis is the actual power imbalance after a fault occurs in the power system;

[0246] The action space of the emergency control model is as follows:

[0247]

[0248] In the formula, A em is the action space of the emergency control model; is the active power removed by generator i; is the load shedding amount of node i; The active power adjusted upward or downward for the wind turbines involved in the regulation; The active power adjusted upward or downward by the photovoltaic units involved in the regulation;

[0249] The reward function of the emergency control model is as follows:

[0250]

[0251] In the formula, R em is the reward function of the emergency control model; P deficit The power gap that remains after wind, solar and storage participate in emergency control; C loadcut is the load shedding cost; C G Removal cost for the unit; is the emergency control constraint function; β1, β2 and β3 are weight coefficients respectively.

[0252] Furthermore, in the second embodiment of the present invention, a global reward function is constructed based on the reward function of the preventive control model and the reward function of the emergency control model, specifically:

[0253] Based on the first adjustment coefficient and the second adjustment coefficient, dynamically adjusting the reward function of the preventive control model and the reward function of the emergency control model to form a global reward function;

[0254] Wherein, the global reward function is specifically:

[0255] R global =λ1R pre +λ2R em

[0256] In the formula, R global is the global function; λ1 is the first adjustment coefficient; λ2 is the second adjustment coefficient; R pre is the reward function of the preventive control model; R em is the reward function of the emergency control model.

[0257] Furthermore, in the second embodiment of the present invention, based on the historical operating state information and the global reward function, the preventive control model and the emergency control model are trained as follows:

[0258] Taking maximizing the global reward function as the goal, the preventive control model and the emergency control model are trained using the historical operating state information, specifically:

[0259] Based on the historical operation status information, and the model strategies of the preventive control model and the emergency control model, generate a plurality of first samples;

[0260] Utilize each of the first samples to cyclically update the model Q value and calculate the corresponding loss function;

[0261] Using the gradient descent method, the network weights are updated according to the loss function;

[0262] When the change in the global reward function is within a preset range or the number of cycles reaches a preset update iteration amount, it is determined that the training of the preventive control model and the emergency control model is completed.

[0263] Furthermore, in the second embodiment of the present invention, based on the historical operation status information and the model strategies of the preventive control model and the emergency control model, a plurality of first samples are generated, specifically:

[0264] Determine a first environmental state according to the historical operating state information, and select a first action using the first strategy of the preventive control model and the emergency control model;

[0265] Obtaining an environment feedback reward and a second environment state according to the first action;

[0266] The first environment state, the first action, the environment feedback reward and the second environment state are determined as a first sample in a sample experience library.

[0267] In summary, the second embodiment of the present invention provides a power grid coordination control device based on deep reinforcement learning, which obtains historical operating status information of the new power system based on the organic combination of modules; constructs the state space, action space and reward function of the preventive control model; constructs the state space, action space and reward function of the emergency control model; constructs a global reward function based on the reward function of the preventive control model and the reward function of the emergency control model; performs model training on the preventive control model and the emergency control model based on the historical operating status information and the global reward function; generates a real-time control strategy using the trained preventive control model and the emergency control model according to the real-time operating status information of the new power system, and uses the real-time control strategy to adjust the new power system. The present invention improves the flexibility of power grid system control, ensures the consistency and coordination of the preventive control stage and the emergency control stage, and improves the control effect.

[0268] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A power grid coordination control method based on deep reinforcement learning, characterized in that: include: Obtain historical operating status information of new power systems; Construct the state space, action space and reward function of the preventive control model; Construct the state space, action space, and reward function of the emergency control model; Constructing a global reward function based on the reward function of the preventive control model and the reward function of the emergency control model; Based on the historical operating state information and the global reward function, performing model training on the preventive control model and the emergency control model; According to the real-time operation status information of the new power system, a real-time control strategy is generated by using the trained preventive control model and the emergency control model, and the new power system is adjusted by using the real-time control strategy.

2. The power grid coordinated control method based on deep reinforcement learning according to claim 1 is characterized in that: The state space, action space and reward function of the preventive control model are constructed, including: The state space of the preventive control model is as follows: S prevent ={U i ,θ i ,δ i ,ω i ,ω WTi ,P WTi ,U PVi ,P PVi ,SOC(t),P ESS (t)} In the formula, S prevent is the state space of the preventive control model; U i is the voltage amplitude of the i-th node; θ i is the phase angle of the i-th node; δ i is the power angle of the i-th generator; ω i is the rotor speed of the i-th generator; ω WTi is the rotor speed of the i-th fan; P WTi is the output power of the i-th fan; U PVi is the output voltage of the ith photovoltaic unit; P PVi is the output power of the i-th photovoltaic system; SOC(t) is the state of charge of the i-th energy storage system at time t; P ESS (t) is the charging and discharging power of the i-th energy storage system at time t; The action space of the preventive control model is as follows: In the formula, A prevent is the action space of the preventive control model; ΔP Gi The output of the adjusted generator i; is the active power cut by the i-th wind turbine; is the active power reduced by the i-th PV unit; P ESS,dis,n is the discharge power of energy storage unit n; P ESS,ch,r is the charging power of energy storage unit r; The reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, specifically: In the formula, R pre is the reward function of the preventive control model; F cost (t) is the cost operation function; is the preventive control constraint function; K is the penalty for non-convergence of the power flow; α and β are weight coefficients respectively; E imbalance It is the energy imbalance measure of the new power system; reserve Wind and solar active power reserve preset for new power systems.

3. The power grid coordinated control method based on deep reinforcement learning according to claim 2 is characterized in that: The reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, including: The cost operation function is specifically: In the formula, F cost (t) is the cost operation function; C G (t) is the generator output adjustment cost at time t; C WT (t) is the wind turbine output reduction cost at time t; C PV (t) is the photovoltaic output reduction cost at time t; C ESS (t) is the charge / discharge state adjustment cost of the energy storage unit at time t.

4. The power grid coordinated control method based on deep reinforcement learning according to claim 2 is characterized in that: The reward function of the preventive control model is determined by the cost operation function and the preventive control constraint function, including: The preventive control constraint function is determined based on a number of constraints and constraint coefficients corresponding to the constraints; wherein the constraints include power flow constraints, voltage and phase angle amplitude constraints at each node, generator output constraints, wind turbine and photovoltaic output reduction constraints, energy storage unit charging and discharging constraints, and line power limit constraints.

5. The power grid coordinated control method based on deep reinforcement learning according to claim 4 is characterized in that: The constraints include power flow constraints, voltage and phase angle amplitude constraints at each node, generator output constraints, wind turbine and photovoltaic output reduction constraints, energy storage unit charging and discharging constraints, and line power limit constraints, including: The power flow constraints are specifically: Where P G,i is the active power generated by node i; Q G,i is the reactive power of node i; P D,i is the active power of node i; Q D,i is the reactive power demand of node i; G ij is the conductance between node i and node j; B ij is the susceptance between node i and node j; U i is the voltage amplitude of the ith node; U j is the voltage amplitude of the ith node; N0 is the node set except the equilibrium node; N PQ is the set of PQ nodes; δ ij is the voltage phase angle difference between node i and node j; n is the total number of all nodes in the new power system; The voltage and phase angle amplitude constraints of each node are specifically as follows: 0.95u Ni ≤u i,t ≤1.05u Ni i i,min ≤θ i ≤θ i,max In the formula, u i,t is the voltage of node i at time t; u Ni is the rated voltage of node i; θ i is the node phase angle; θ i,max is the maximum phase angle of the node; θ i,min is the minimum phase angle of the node; The generator output constraint is specifically: P Gi,min ≤P Gi ≤P Gi,max i=1,2,…,N G Where P Gi P is the operating power of the generator; Gi,min The minimum output power of the generator; P Gi,max Maximum output power for generator operation; The wind turbine and photovoltaic output reduction constraints are specifically: In the formula, is the active power cut by the i-th wind turbine; P WT,max is the maximum output power of the wind turbine; is the active power reduced by the i-th PV unit; P PV,max is the maximum output power of the photovoltaic unit; The energy storage unit charging and discharging constraints are specifically: 0≤P ESS,ch,r ≤P ESS,ch,max IN ESS,ch 0≤P ESS,dis,n ≤P ESS,dis,max IN ESS,dis IN ESS,ch +U ESS,dis ≤1 Where P ESS,ch,r Charging power for energy storage unit; P ESS,ch,max is the maximum charging power of the energy storage unit; U ESS,ch It is the charging status flag of the energy storage unit; P ESS,dis,n is the discharge power of the energy storage unit; P ESS,dis,max is the maximum discharge power of the energy storage unit; U ESS,dis It is the discharge status flag of the energy storage unit; The line power limit constraint is specifically: In the formula, is the power of line l; is the power limit of line l.

6. The power grid coordinated control method based on deep reinforcement learning according to claim 1 is characterized in that: The state space, action space and reward function of the emergency control model are constructed, including: The state space of the emergency control model is as follows: In the formula, S em is the state space of the emergency control model; U i is the voltage amplitude of the i-th node; θ i is the phase angle of the i-th node; δ i is the power angle of the i-th generator; ω i is the rotor speed of the i-th generator; ω WTi is the rotor speed of the i-th fan; P WTi is the output power of the i-th fan; U PVi is the output voltage of the ith photovoltaic unit; P PVi is the output power of the i-th photovoltaic unit; is the active power cut by the i-th wind turbine; is the active power cut by the i-th photovoltaic unit; ΔP dis is the actual power imbalance after a fault occurs in the power system; The action space of the emergency control model is as follows: In the formula, A em is the action space of the emergency control model; is the active power removed by generator i; is the load shedding amount of node i; The active power adjusted upward or downward for the wind turbines involved in the regulation; The active power adjusted upward or downward by the photovoltaic units involved in the regulation; The reward function of the emergency control model is as follows: In the formula, R em is the reward function of the emergency control model; P deficit The power gap that remains after wind, solar and storage participate in emergency control; C loadcut is the load shedding cost; C G Removal cost for the unit; is the emergency control constraint function; β1, β2 and β3 are weight coefficients respectively.

7. The power grid coordinated control method based on deep reinforcement learning according to claim 1 is characterized in that: The reward function based on the preventive control model and the reward function of the emergency control model is used to construct a global reward function, which is specifically: Based on the first adjustment coefficient and the second adjustment coefficient, dynamically adjusting the reward function of the preventive control model and the reward function of the emergency control model to form a global reward function; The global reward function is specifically: R global =λ1R pre +λ2R em In the formula, R global is the global function; λ1 is the first adjustment coefficient; λ2 is the second adjustment coefficient; R pre is the reward function of the preventive control model; R em is the reward function of the emergency control model.

8. The power grid coordinated control method based on deep reinforcement learning according to claim 7 is characterized in that: The performing model training on the preventive control model and the emergency control model based on the historical operating state information and the global reward function is specifically as follows: Taking maximizing the global reward function as the goal, the preventive control model and the emergency control model are trained using the historical operating state information, specifically: Based on the historical operation status information, and the model strategies of the preventive control model and the emergency control model, generate a plurality of first samples; Utilize each of the first samples to cyclically update the model Q value and calculate the corresponding loss function; Using the gradient descent method, the network weights are updated according to the loss function; When the change in the global reward function is within a preset range or the number of cycles reaches a preset update iteration amount, it is determined that the training of the preventive control model and the emergency control model is completed.

9. The power grid coordinated control method based on deep reinforcement learning according to claim 8, characterized in that: The generating of a plurality of first samples based on the historical operation status information and the model strategies of the preventive control model and the emergency control model is specifically: Determine a first environmental state according to the historical operating state information, and select a first action using the first strategy of the preventive control model and the emergency control model; Obtaining an environment feedback reward and a second environment state according to the first action; The first environment state, the first action, the environment feedback reward and the second environment state are determined as a first sample in a sample experience library.

10. A power grid coordination control device based on deep reinforcement learning, characterized in that: include: An acquisition module, a first model building module, a second model building module, a model combination module, a model training module and a control module; The acquisition module is used to acquire historical operation status information of the new power system; The first model building module is used to build the state space, action space and reward function of the preventive control model; The second model building module is used to build the state space, action space and reward function of the emergency control model; The model combination module is used to construct a global reward function based on the reward function of the preventive control model and the reward function of the emergency control model; The model training module is used to perform model training on the preventive control model and the emergency control model based on the historical operating state information and the global reward function; The control module is used to generate a real-time control strategy based on the real-time operating status information of the new power system using the trained preventive control model and the emergency control model, and adjust the new power system using the real-time control strategy.

Images