Power system multi-target scheduling optimization control method and system based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based multi-target scheduling optimization method. The method comprises the following steps of obtaining historical voltage data of each node of a power system; randomly generating power supply and load disturbance, changing the voltage value of each node after solving through load flow calculation, and recording the current of each branch, the total network loss and the total reactive compensation capacity of the system; calculating the sensitivity vectors of each voltage node to the branch current, the total network loss of the system and the total reactive compensation amount by using a Monte Carlo method, and carrying out weight addition normalization to obtain a total weighted sensitivity vector which is the weight of each node; a node'weight 'is defined to be multiplied by a time sequence operation voltage deviation value of the node'weight' to be a node weighted voltage deviation value, and the node'weight 'is globally accumulated to be a system weighted voltage deviation value; introducing a system weighted voltage deviation value to train a near-end strategy to optimize a reinforcement learning agent; therefore, the voltage real-time optimization control strategy of the power system is realized. According to the invention, multi-target control of the power system can be effectively realized, and safe, reliable and economical operation of the power system is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-objective dispatch optimization of power systems, and in particular to a multi-objective dispatch optimization method and system based on reinforcement learning that considers weighted offset values ​​of voltage in sequential operation of power grids. Background Art

[0002] The penetration rate of distributed renewable energy is increasing during the construction of new power systems. However, with the growth of distributed power capacity, its many adverse effects on the power quality, safety and stability of the power system are gradually emerging.

[0003] The output of distributed power sources is constantly changing due to the intermittent and volatile nature of renewable energy, and the power output curve of the power source is difficult to synchronize with the load power. Therefore, after a large number of distributed power sources are connected to the new power system, the voltage limit and line loss problems faced by the power grid operation are becoming more serious, and the three-phase imbalance characteristics are more prominent. The harmonic content in the distribution network increases, and problems such as protection malfunction also occur frequently. The unbalanced operation of the system will cause a surge in the negative sequence component of the voltage, resulting in abnormal operation of the equipment and increased losses. In severe cases, it will threaten the safe and stable operation of the power system, and will also be accompanied by the island effect problem. In actual operation, distributed power sources will also have an adverse effect on the stability of the power system voltage, frequency, protection, harmonics, and rotor angle.

[0004] With the continuous increase in the installed capacity of new energy sources, more and more controllable power equipment has appeared in the new distribution network, and the control performance of controllable inverters, SVG and other equipment has become better and better. However, if an effective coordinated control method cannot be established between the compensation device and the distributed power source, the utilization rate of resources and the operating efficiency of the power system cannot be effectively improved, and problems such as overvoltage and increased network loss cannot be alleviated. Domestic and foreign scholars have conducted research in the field of coordinated optimization control between power system components and power electronic equipment. At present, common power electronic equipment control methods include: virtual synchronous generator technology (VSG), droop control, PQ control, etc. These technologies are relatively mature, widely used, simple and reliable, and have good local control effects, but lack the ability of overall coordinated optimization. Therefore, it is also necessary to combine some algorithms to achieve overall coordinated control optimization, such as: linear and nonlinear programming methods, dynamic programming methods, multi-objective particle swarm algorithms, genetic algorithms, etc. However, these algorithms are difficult to characterize the complex models of new active distribution networks and are prone to fall into local optimality. The rapid development of technologies such as artificial intelligence, big data analysis, and the Internet of Things provides effective methods for the optimization control of active distribution networks. Deep reinforcement learning can handle high-dimensional, large-scale, and uncertain control problems and has strong decision-making capabilities. At present, the application of power systems based on deep reinforcement learning has attracted widespread attention. However, the actual operation of power systems is highly complex and uncertain, closely related to climate conditions, economic environment, and human activities, and difficult to be represented by deterministic mathematical laws. Further research is needed to play a good role in actual control. Many scholars have applied genetic algorithms, fuzzy chance constrained programming, multi-objective particle swarm algorithms and other methods to the optimal scheduling of active distribution networks. These algorithms have high requirements for problem modeling, are not suitable for complex new power systems, and are prone to fall into local optimality. Summary of the invention

[0005] The purpose of the present invention is to solve the defects existing in the prior art and to provide a multi-objective scheduling optimization method and system based on reinforcement learning that takes into account the weighted offset value of the power grid timing operation voltage. The coordinated optimization of composite control objectives is achieved based on day-ahead optimization real-time control and deep reinforcement learning, which is conducive to building a new system that is safer, more reliable and more economical.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] The multi-objective dispatch optimization control method of power system based on reinforcement learning includes:

[0008] Obtain the power system topology and historical voltage data of each node in multiple scenarios, and build photovoltaic models, wind turbine models, and composite load models;

[0009] Randomly generate power system power node voltage and load disturbances, solve them through power flow calculation, record the voltage amplitude of each node that changes accordingly, record the current value of each branch, the real-time total network loss of the system, and the real-time total reactive power compensation of the system under this state;

[0010] The Monte Carlo method is used to analyze and obtain the sensitivity matrix of each voltage node to each current branch, the sensitivity vector to the real-time total network loss of the system, and the sensitivity vector to the real-time total reactive compensation of the system. The total weighted sensitivity vector is obtained by weighted addition and normalization, which is defined as the "weight" of each node.

[0011] The node weight is multiplied by its timing operation voltage offset value to define the node weighted voltage offset value, and the global accumulation is the system weighted voltage offset value in the current state of the system;

[0012] The system weighted voltage offset value is introduced to train the proximal strategy optimization reinforcement learning agent; the real-time optimization control of the power system voltage is carried out according to the trained intelligent optimization control model.

[0013] The present invention applies reinforcement learning algorithm to power system control and adopts proximal policy optimization (PPO) to train intelligent agents to achieve multi-objective scheduling optimization. At the same time, based on the system global weighted voltage offset value training, the obtained intelligent optimization control model can implement different degrees of precision control on different voltage nodes according to the "weights" corresponding to them.

[0014] Furthermore, the sensitivity matrix of the above voltage node to each current branch is as follows:

[0015]

[0016] Where α mn is the sensitivity of the nth node to the mth line in the circuit;

[0017] The sensitivity vector form of the voltage node to the real-time total network loss of the system is as follows:

[0018] [β s1 ,β s2 ,...,β sn ]

[0019] Where β sn is the sensitivity of the nth node to the total real-time network loss of the system;

[0020] The sensitivity vector form of the voltage node to the real-time total reactive compensation of the system is as follows:

[0021] [γ s1 ,γ s2 ,...,γ sn ]

[0022] Where γsn is the real-time total reactive power compensation sensitivity of the nth node to the system.

[0023] The above system weighted voltage offset value is obtained by the following steps:

[0024] Calculate the system's weighted voltage offset value δU with respect to current α , Weighted voltage offset value δU for loss β , Regarding the weighted voltage offset value δUγ of the system reactive power compensation:

[0025] δU α =α1δu1+α2δu2+α3δu3+...+α n δu n

[0026] δU β =β1δu1+β2δu2+β3δu3+...+β n δu n

[0027] δU γ =γ1δu1+γ2δu2+γ3δu3+...+γ n δu n

[0028] In the formula, δu n is the voltage offset value of the nth node,

[0029] α n is the “weight” of the branch current in the weighted voltage offset value of the nth node after processing,

[0030] β n is the “weight” of the circuit loss of the nth node after processing in the weighted voltage offset value,

[0031] γ n is the “weight” of the reactive power compensation of the system by the nth node after processing in the weighted voltage offset value;

[0032] Comprehensively control the priority of the target and set the weighted voltage coefficient of each part:

[0033] A+B+C=1

[0034] Where A, B, and C correspond to the weighted voltage coefficients of branch current, system loss, and reactive power compensation respectively;

[0035] Calculate the total weighted voltage offset value δU of the system:

[0036]

[0037] Among them, the state space of the reinforcement learning agent includes the state space of each phase voltage of each node, the switching and gear state of the controllable equipment, as follows:

[0038] S=[Sta u ,Sta C ,Sta O ,Sta G ,Sta S ]

[0039] Where S is the entire state space, which can be decomposed into five subsets including node voltage information and the states of various controllable devices.

[0040]

[0041]

[0042] In the formula, Sta u represents the state space voltage subset composed of each phase voltage, represents the first The node voltage of the phase, where It consists of three phases: a, b, and c, and the phase difference between the voltages of each phase is 120 degrees;

[0043] Sta C =[Sta c1 ,Sta c2 ,...,Sta ci ]

[0044] Sta O =[Sta o1 ,Sta o2 ,…,Sta oi ]

[0045] Sta G =[Sta g1 ,Sta g2 ,...,Sta gi ]

[0046] Sta S =[Sta s1 ,Sta s2 ,...,Sta si ]

[0047] In the formula, Sta C represents the state space capacitance subset composed of all capacitor switching states, Sta ci Indicates the switching status of each capacitor; Sta O represents the state space transformer subset consisting of all the on-load tap-changing transformer gear states, Sta oiRepresents the tap position status of each on-load tap-changing transformer; Sta G represents the state space generator subset consisting of all generator output states, Sta G Indicates the real-time active output of each generator set; Sta S Represents the state space circuit breaker subset consisting of all circuit breaker opening and closing states, Sta si Represents the opening and closing status of each circuit breaker.

[0048] Among them, the action space Act of the reinforcement learning agent includes Act c ,Act o ,Act G ,Act s Four sub-action sets:

[0049] Act=[Act c ,Act o ,Act G ,Act s ]

[0050] Act C =[Act c1 ,Act c2 ,…,Act ci ]

[0051] C i =[c i1 ,c i2 ]

[0052] In the formula, Act C Act represents the subset of the capacitor action space composed of all capacitor switching states. ci is the switching state of the capacitor labeled i, C i represents the set of actions that the ith capacitor can take, c i1 To remove the capacitor, c i2 To put in capacitor.

[0053] Act O =[Act o1 ,Act o2 ,…,Act oi ]

[0054] O i =[o i1 ,o i2 ,o i3 ,...,o ix ]

[0055] In the formula, Act OAct represents the subset of transformer action space composed of all on-load tap-changing transformer gear states. oi Represents the gear status of the on-load tap-changing transformer No. i, O i represents the set of actions that the ith transformer can take, o ix Indicates different connectors.

[0056] Act G =[Act g1 ,Act g2 ,…,Act gi ]

[0057] G i =[g i1 ,g i2 ]

[0058] In the formula, Act G Act represents the subset of the generator action space composed of all generator output states. gi Indicates the action of the i-th generator set, G i represents the set of actions that the i-th generator can take, g i1 In order to increase the active output of the generator by a gradient, g i2 To reduce the active output of the generator by a gradient.

[0059] Act S =[Act s1 ,Act s2 ,…,Act si ]

[0060] S i =[s i1 ,s i2 ]

[0061] In the formula, Act S Act represents the subset of the generator action space consisting of all circuit breaker opening and closing states. si Represents the action of circuit breaker No. i, S i represents the set of actions that the i-th circuit breaker can take, s i1 Indicates that the circuit breaker is opened, s i2 Indicates closing the circuit breaker.

[0062] The reward function of the reinforcement learning agent is constructed based on the system's global weighted voltage offset value as follows:

[0063] reward 1t =δU t_1 -δU t_2

[0064] That is, at time t, the reward is the reduction of the weighted voltage offset value before and after the action of the agent; where δU t_1 , δU t_2 Represents the voltage of each node before and after the intelligent agent action at time t;

[0065] And make the following constraints on the control voltage required by the model:

[0066]

[0067] In the formula is the voltage of each node;

[0068] When the voltage of a node exceeds the safe range, the agent will receive a reward 2t as follows:

[0069]

[0070] Where, M is 1000; U k is the per unit value of voltage at any node;

[0071] And set a reward for generator sets that exceed the output limit 3t , as follows:

[0072]

[0073] Where N is 5, P gen i is the real-time output value of any generator set, P gen j_min and P gen j_max They are the upper and lower limits of the generator set output respectively;

[0074] The global reward function of the reinforcement learning agent is as follows:

[0075] reward=reward 1t +reward 2t +reward 3t .

[0076] The training method of the above reinforcement learning agent is as follows:

[0077] During training, we first need to initialize the parameters of the two networks, “actor_net” and “critic_net” in the proximal policy optimization deep reinforcement learning algorithm, and set a buffer area to input the load power curve, the distributed power system power curve, the initial states of the capacitors, line switches and on-load voltage regulator taps, and the initial output of the generator set;

[0078] For any time, when the agent starts at time t k-1 Transition to time t kWhen the action sequence group is generated, the power flow calculation is performed to obtain the initial state of each node voltage before the agent takes action, and it is input into the two networks "actor_net" and "critic_net" respectively. "actor_net" makes a judgment to obtain the strategy; when the action sequence group is generated, the default load power consumption and photovoltaic system output power have not changed. The agent updates the state of the actionable equipment and performs power flow calculation again to obtain a new state. This state will also be returned to the agent, and the two states are simultaneously input into "critic_net" to obtain the corresponding reward value;

[0079] Vector (Stat k- , Sta k+ , R k , Act k ) will be placed in the cache area, where Sta k- Represents t k The state before the agent takes action at any moment, Sta k+ Represents t k The state of the agent after taking action at that moment, R k Represents t k The reward value obtained by the agent for taking an action at that moment, Act k For the action taken by the agent at this moment, after each trial, the agent will randomly extract L groups of vectors at that moment in the previous training process from the cache and store them in the L group (Stat k- , Sta k+ ) is put into "critic_net", the network will update the parameters and redefine the advantage function of its output based on the original output value of the network. k- , Act k ), it will be put into "actor_net", calculate the probability ratio of taking the action under the new and old strategies, update the network parameters, and generate a new strategy function;

[0080] In from t k to k+1 Within fifteen minutes, the system will maintain steady state S k , until reaching time t k+1 , the load power consumption of the system and the output power of the distributed power system change, and the power flow calculation will be performed again to obtain Act k The new state Sta before the agent takes action k+1- , continue to take new action group A k+1 , then the decision-making training process of the agent is related to time t k Consistent.

[0081] The present invention also provides a multi-objective dispatch optimization control system for a power system based on reinforcement learning, comprising:

[0082] An acquisition module is used to obtain the power system topology and the historical voltage data of each node in multiple scenarios;

[0083] The optimization control module is used to input voltage data into the trained intelligent optimization control model to obtain a real-time optimization control solution for the power system voltage;

[0084] The intelligent optimization control model is obtained by introducing the system weighted voltage offset value and training the proximal strategy optimization reinforcement learning agent;

[0085] The calculation method of the system weighted voltage offset value is as follows: the Monte Carlo method is used to analyze and obtain the sensitivity matrix of each voltage node to each current branch, the sensitivity vector to the real-time total network loss of the system, and the sensitivity vector to the real-time total reactive compensation of the system. The total weighted sensitivity vector is obtained by weighted addition and normalization, which is defined as the "weight" of each node; the node "weight" is multiplied by its timing operation voltage offset value to be defined as the node weighted voltage offset value, and the global accumulation is obtained.

[0086] Compared with the prior art, the present invention has the following advantages and effects:

[0087] (1) The present invention applies deep learning algorithm methods to power system voltage control. By using the PPO algorithm to train the intelligent agent, it can simultaneously complete the control of different power equipment. The model can better realize the scheduling optimization of composite control objectives and improve the control effect of the model. At the same time, it has efficient scheduling calculation capabilities and is suitable for composite objective control of large-scale power systems.

[0088] (2) The present invention constructs a deep learning agent reward function based on the system's global weighted voltage offset value. The model can implement different degrees of precision control on different voltage nodes according to the different "weights" corresponding to them. The strategy generation speed is faster, the optimal solution can be found in a shorter time, the agent strategy update process is accelerated, and a better optimization effect can be obtained under the constraint conditions.

[0089] (3) The present invention obtains an intelligent optimization control model through training. The model is based on historical power operation data, and is based on Markov decision and reinforcement learning processes, offline intelligent agents explore the environment based on behavioral strategies, and use policy gradient algorithms to train network parameters. Therefore, the voltage control scheme obtained based on this control module is a more excellent voltage control strategy, which can effectively improve power flow distribution, alleviate voltage over-limit, and smooth voltage fluctuations, thereby achieving real-time optimization control of the distribution network voltage under a high proportion of new energy access. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1This is a flow chart of a method for generating a weighted voltage offset value in Embodiment 1 of the present invention;

[0091] Figure 2 This is a flowchart of multi-objective optimization scheduling deep reinforcement learning agent training in Example 2 of the present invention;

[0092] Figure 3 is the bus voltage at each time point before optimization in Example 3 of the present invention

[0093] Figure 4 is the bus voltage at each time point after optimization in Embodiment 3 of the present invention;

[0094] Figure 5 This is a comparison diagram of network losses before and after optimization in Example 3 of the present invention. DETAILED DESCRIPTION

[0095] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0096] Embodiment 1

[0097] The system globally strengthens the acquisition of voltage offset value

[0098] The present invention constructs the concept of weighted voltage offset value, completes the calculation of weighted voltage offset value parameters of power system model, and uses weighted voltage offset value as reward to input into reinforcement learning distribution network model, and finally conducts data analysis on the control degree of node voltage under different "weight" positions under the reward setting. Figure 1 As shown, the specific steps are as follows:

[0099] Obtain the new power system topology and historical voltage data of each node in multiple scenarios, and build photovoltaic models, wind turbine models, and composite load models;

[0100] Randomly change the power node voltage and load to calculate the voltage amplitude of each node in different states, and record the current value of each branch, the real-time total network loss of the system, and the real-time total reactive compensation of the system under this state:

[0101] I=[I1,I2,...,I m ], P loss , Q total

[0102] I is defined as a one-dimensional current matrix containing m branches in the system, P loss Defined as the total network loss Q total Defined as the total real-time reactive power compensation of the system;

[0103] The Monte Carlo method is used to analyze and obtain the sensitivity matrix of each voltage node to each current branch, the sensitivity vector to the real-time total network loss of the system, and the sensitivity vector to the real-time total reactive power compensation of the system:

[0104] The sensitivity matrix of each voltage node to each branch current is as follows:

[0105]

[0106] Where α mn is the sensitivity of the nth node to the mth line in the line;

[0107] The grid loss sensitivity vector is in the following form:

[0108] [β s1 ,β s2 ,...,β sn ]

[0109] Where β sn The sensitivity of the nth node to the total real-time network loss of the system;

[0110] The reactive sensitivity vector form is as follows:

[0111] [γ s1 ,γ s2 ,...,γ sn ]

[0112] Where γ sn The nth node is the sensitivity of the system to the total reactive power compensation in real time;

[0113] Using principal component analysis, the current sensitivity matrix is ​​reduced to a one-dimensional vector, and the composite sensitivity vector of voltage to all line currents is obtained:

[0114] Call the PCA function to get the composite sensitivity of voltage to all line currents:

[0115] [α 01 ,α 02 ,...,α 0n ]

[0116] Where α 0n is the composite sensitivity of Node n to all branch currents in the system;

[0117] Normalize the three node voltage sensitivity vectors so that the sum of their internal elements is 1:

[0118] α1+α2+α3+...+α n =1

[0119] β1+β2+β3+...+βn =1

[0120] γ1+γ2+γ3+...+γ n =1

[0121] Where α n , β n , γ n They are the final “weight” parameters of the nth node based on sensitivity to branch current, system real-time total network loss, and system real-time total reactive power;

[0122] Calculate the system's weighted voltage offset value δU with respect to current α , weighted voltage offset value δU for loss β , Regarding the weighted voltage offset value δUγ of the system reactive power compensation:

[0123] δUα=α1δu1+α2δu2+α3δu3+...+αnδun

[0124] δUβ=β1δu1+β2δu2+β3δu3+...+βnδun

[0125] δUγ=γ1δu1+γ2δu2+γ3δu3+...+γnδun

[0126] δu n is the voltage offset value of Node n,

[0127] α n is the “weight” of the branch current in the weighted voltage offset value of Node n after processing,

[0128] β n is the “weight” of the circuit loss of Node n in the weighted voltage offset value after processing,

[0129] γ n is the “weight” of the reactive power compensation of the system by Node n in the weighted voltage offset value after processing;

[0130] Comprehensively control the priority of the target and set the weighted voltage coefficient of each part:

[0131] A+B+C=1

[0132] Where A, B, and C correspond to the weighted voltage coefficients of branch current, system loss, and reactive power compensation respectively;

[0133] Calculate the total weighted voltage offset value δU of the system:

[0134] δU=AδU α +BδU β +CδU γ

[0135] Since each voltage offset value is obtained by multiplying the node voltage by the weight of the node, δU can also be expressed as the sum of the node voltage multiplied by different coefficients:

[0136]

[0137] Embodiment 2

[0138] Multi-objective integrated control method

[0139] Construct a power system intelligent control constraint set that considers power flow constraints, generator output constraints, and node voltage constraints:

[0140]

[0141]

[0142] S cource-toal +S var-toal =S load-total +S line-total

[0143] Where m is the number of centralized power sources, n is the number of centralized point loads, S source-total is the total power output apparent power, S source-i is the apparent power of the ith generator set, S load-total is the apparent power sum of all loads, S load-i is the apparent power of the ith concentrated load group, S var-total is the apparent power sum of all reactive power compensation equipment, S line-total is the apparent power consumed by the system lines and.

[0144] 0.95U Kn ≤U k ≤1.05U kn

[0145] Where U k is the real-time voltage of any node, U Kn is the rated voltage of the node.

[0146] P gen j_min ≤P gen j ≤P gen j_max

[0147] Where P gen j is the real-time output of any generator, P gen j_min and P gen j_max They are the upper and lower limits of the generator set output respectively.

[0148] The state space of the deep enhanced intelligent agent, which includes the voltage of each phase of each node, the switching and gear status of the controllable equipment, is constructed as follows:

[0149] S=[Sta u ,Sta C ,Sta O ,Sta G ,Sta S ]

[0150] Where S is the entire state space, which can be decomposed into five subsets including node voltage information and the states of various controllable devices.

[0151]

[0152]

[0153] In the formula, Sta u represents the state space voltage subset composed of each phase voltage, represents the first The node voltage of the phase, where It consists of three phases: a, b, and c, and the phase difference between the voltages of each phase is 120 degrees;

[0154] Sta C =[Sta c1 ,Sta c2 ,...,Sta ci ]

[0155] Sta O =[Sta o1 ,Sta o2 ,…,Sta oi ]

[0156] Sta G =[Sta g1 ,Sta g2 ,...,Sta gi ]

[0157] Sta S =[Sta s1 ,Sta s2 ,...,Sta si ]

[0158] In the formula, Sta C represents the state space capacitance subset composed of all capacitor switching states, Sta ci Indicates the switching status of each capacitor; Sta O represents the state space transformer subset consisting of all the on-load tap-changing transformer gear states, Sta oi Represents the tap position status of each on-load tap-changing transformer; Sta Grepresents the state space generator subset consisting of all generator output states, Sta G Indicates the real-time active output of each generator set; Sta S Represents the state space circuit breaker subset consisting of all circuit breaker opening and closing states, Sta si Represents the opening and closing status of each circuit breaker.

[0159] Construct a discrete agent action space containing a complete controllable device, wherein the action space Act of the reinforcement learning agent includes Act c ,Act o ,Act G ,Act s Four sub-action sets:

[0160] Act=[Act c ,Act o ,Act G ,Act s ]

[0161] Act C =[Act c1 ,Act c2 ,…,Act ci ]

[0162] C i =[c i1 ,c i2 ]

[0163] In the formula, Act C Act represents the subset of the capacitor action space composed of all capacitor switching states. ci is the switching state of the capacitor labeled i, C i represents the set of actions that the ith capacitor can take, c i1 To remove the capacitor, c i2 To put in capacitor.

[0164] Act O =[Act o1 ,Act o2 ,…,Act oi ]

[0165] O i =[o i1 ,o i2 ,o i3 ,...,o ix ]

[0166] In the formula, Act O Act represents the subset of transformer action space composed of all on-load tap-changing transformer gear states.oi Represents the gear status of the on-load tap-changing transformer No. i, O i represents the set of actions that the ith transformer can take, o ix Indicates different connectors.

[0167] Act G =[Act g1 ,Act g2 ,…,Act gi ]

[0168] G i =[g i1 ,g i2 ]

[0169] In the formula, Act G Act represents the subset of the generator action space composed of all generator output states. gi Indicates the action of the i-th generator set, G i represents the set of actions that the i-th generator can take, g i1 In order to increase the active output of the generator by a gradient, g i2 To reduce the active output of the generator by a gradient.

[0170] Act S =[Act s1 ,Act s2 ,…,Act si ]

[0171] S i =[s i1 ,s i2 ]

[0172] In the formula, Act S Act represents the subset of the generator action space consisting of all circuit breaker opening and closing states. si Represents the action of circuit breaker No. i, S i represents the set of actions that the i-th circuit breaker can take, s i1 Indicates that the circuit breaker is opened, s i2 Indicates closing the circuit breaker.

[0173] Construct the reward function of the deep learning agent based on the system's global weighted voltage offset value:

[0174] reward 1t =δU t_1 -δU t_2

[0175] That is, at time t, the reward is the reduction of the weighted voltage offset value before and after the action of the agent; where δU t_1 , δUt_2 Represents the voltage of each Node before and after the intelligent agent acts at time t.

[0176] In addition, in order to ensure the voltage stability of the power system, the present invention imposes the following constraints on the control voltage required by the model:

[0177]

[0178] In the formula is the voltage of each Node.

[0179] When the voltage of a node exceeds the safe range, it will have a huge impact on the safety of the power system. Therefore, it is assumed that the agent will receive a huge penalty M in this state. At this time, the agent will receive a reward 2t as follows:

[0180]

[0181] M is set to 1000 in this model, where U k is the per-unit value of the voltage at any node.

[0182] In addition, a reward is set for generators that exceed their output limits. 3t , as follows:

[0183]

[0184] N is set to 5 in this model, where P gen i It is the real-time output value of any generator set.

[0185] reward=reward 1t +reward 2t +reward 3t

[0186] Reward is the global reward function of the agent's action;

[0187] The agent is trained multiple times based on a large number of scenarios, such as Figure 2 As shown in the figure: During training, it is first necessary to initialize the parameters of the two networks "actor_net" and "critic_net" in the proximal policy optimization (PPO) deep reinforcement learning algorithm, set a buffer area, input the load power curve, the distributed power system power curve, and the initial states of the capacitors, line switches and on-load voltage regulator taps and the initial output of the generator set.

[0188] For any time, when the agent starts at time t k-1 Transition to time t k, the real-time power output of the photovoltaic system and the real-time power consumption of the load in the distribution network system will change. Therefore, it is necessary to calculate the photovoltaic system power and load power corresponding to the time when the system is introduced, and perform flow calculation to obtain the initial state of each Node voltage composition before the agent takes action. The agent needs to input them into the two networks respectively, and "actor_net" makes a judgment to obtain the strategy. Here, the agent is allowed to complete the action in steps. The maximum step size of the action is set to 10, that is, the agent has ten choices to choose the action device and decide whether to complete the cut or cast. When the action sequence group is generated, the default load power consumption and photovoltaic system output power have not changed. The agent updates the state of the actionable device and performs flow calculation again to obtain a new state. The state will also be returned to the agent. The two states are simultaneously input into "critic_net" to obtain the corresponding reward value. Vector (Sta k- , Sta k+ , R k , Act k ) will be placed in the cache area, where Sta k- Represents t k The state before the agent takes action at any moment, Sta k+ Represents t k The state of the agent after taking action at that moment, R k Represents t k The reward value obtained by the agent for taking an action at that moment, Act k For the action taken by the agent at this moment, after each trial, the agent will randomly extract L groups of vectors at that moment in the previous training process from the cache and store them in the L group (Stat k- , Sta k+ ) is put into "critic_net", the network will update the parameters and redefine the advantage function of its output based on the original output value of the network. k- , Act k ), it will be put into "actor_net", calculate the probability ratio of taking the action under the new and old strategies, update the network parameters, and generate a new strategy function;

[0189] In from t k to k+1 Within fifteen minutes, the system will maintain steady state S k , until reaching time t k+1 , the load power consumption of the system and the output power of the distributed power system change, and the power flow calculation will be performed again to obtain Act k The new state Sta before the agent takes action k+1- , continue to take new action group A k+1 , then the decision-making training process of the agent is related to time tk The situation of a day will be divided into 96 points for decision making, and the distributed power system power and load power curves of that day will be trained multiple times.

[0190] Effect Example 3

[0191] The bus voltages at each time point before and after optimization are as follows: Figure 3 , Figure 4 As shown in the figure. Before optimization, due to the access of a large number of renewable energy sources to the power grid, many bus voltages had already exceeded the safety range of national standards, especially at noon and when electricity consumption was relatively high, the voltage per unit value reached a maximum of 1.16 and a minimum of 0.78, and the over-limit situation was very serious. After optimization, the voltage over-limit situation was successfully improved, and the per unit value of each bus voltage was stabilized between 0.95-1.05.

[0192] Comparison of network loss before and after optimization Figure 5 Before the control strategy was adopted, the maximum network loss and the average network loss were reduced by 46.67% and 25.45% respectively.

Claims

1. A multi-objective dispatch optimization control method for power systems based on reinforcement learning, characterized in that: The multi-objective dispatch optimization method of the power system comprises: Obtain the power system topology and historical voltage data of each node in multiple scenarios, and build photovoltaic models, wind turbine models, and composite load models; Randomly generate power system power node voltage and load disturbances, and obtain the voltage amplitude of each node after the change through power flow calculation and solution, and record the current value of each branch, the real-time total network loss of the system, and the real-time total reactive power compensation of the system under this state; The Monte Carlo method is used to analyze and obtain the sensitivity matrix of each voltage node to each current branch, the sensitivity vector to the real-time total network loss of the system, and the sensitivity vector to the real-time total reactive compensation of the system. The total weighted sensitivity vector is obtained by weighted addition and normalization, which is defined as the "weight" of each node. The node weight is multiplied by its timing operation voltage offset value to be defined as the node weighted voltage offset value. The global accumulation is the system weighted voltage offset value in the current state of the system. The system weighted voltage offset value is introduced to train the proximal strategy optimization reinforcement learning agent; the real-time optimization control of the power system voltage is carried out according to the trained intelligent optimization control model.

2. The multi-objective dispatch optimization control method for a power system according to claim 1, characterized in that: The sensitivity matrix of the voltage node to each current branch is as follows: Where α mn is the sensitivity of the nth node to the mth line in the circuit; The sensitivity vector form of the voltage node to the real-time total network loss of the system is as follows: [β s1 ,β s2 ,...,β sn ] Where β sn is the sensitivity of the nth node to the total real-time network loss of the system; The sensitivity vector form of the voltage node to the real-time total reactive compensation of the system is as follows: [c s1 ,c s2 ,...,c sn ] Where γ sn is the real-time total reactive power compensation sensitivity of the nth node to the system.

3. The multi-objective dispatch optimization control method for a power system according to claim 2, characterized in that: The system weighted voltage offset value is obtained by the following steps: Calculate the system's weighted voltage offset value δU with respect to current α , weighted voltage offset value δU for loss β , Regarding the weighted voltage offset value δUγ of the system reactive power compensation: δU α =α1δu1+α2δu2+α3δu3+...+α n you n δU β =β1δu1+β2δu2+β3δu3+...+β n you n δU γ =γ1δu1+γ2δu2+γ3δu3+...+γ n you n In the formula, δu n is the voltage offset value of the nth node, α n is the "weight" of the nth node in the weighted voltage offset value for the branch current after processing, β n is the "weight" of the circuit loss of the nth node after processing in the weighted voltage offset value, γ n is the "weight" of the reactive power compensation of the system by the nth node after processing in the weighted voltage offset value; Comprehensively control the priority of the target and set the weighted voltage coefficient of each part: A+B+C=1 Where A, B, and C correspond to the weighted voltage coefficients of branch current, system loss, and reactive power compensation respectively; Calculate the total weighted voltage offset value δU of the system:

4. The multi-objective dispatch optimization control method for a power system according to claim 3, characterized in that: The state space of the reinforcement learning agent includes the voltage of each phase of each node, the switching or gear state of the controllable equipment, as follows: S=[Sta u ,Is C ,Is O ,Is G ,Is S ] Where S is the entire state space, which is decomposed into five subsets including node voltage information and the states of various controllable devices: Is C =[Here c1 ,Is c2 ,...,Is ci ] Is O =[Here o1 ,Is o2 ,…,Is oi ] Mrs. G =[Here g1 ,Is g2 ,...,Is gi ] Is S =[Here s1 ,Is s2 ,...,Is si ] In the formula, Sta C represents the state space capacitance subset composed of all capacitor switching states, Sta ci Indicates the switching status of each capacitor; Sta O represents the state space transformer subset consisting of all the on-load tap-changing transformer gear states, Sta oi Represents the tap position status of each on-load tap-changing transformer; Sta G represents the state space generator subset consisting of all generator output states, Sta gi Indicates the real-time active output of each generator set; Sta S Represents the state space circuit breaker subset consisting of all circuit breaker opening and closing states, Sta si Represents the opening and closing status of each circuit breaker.

5. The multi-objective dispatch optimization control method for a power system according to claim 4, characterized in that: The action space Act of the reinforcement learning agent includes Act c ,Act o ,Act G ,Act s The four sub-action sets are represented as follows: Act=[Act c ,Act o ,Act G ,Act s ] Act C =[Act c1 ,Act c2 ,…,Act ci ] C i =[c i1 ,c i2 ] In the formula, Act C Act represents the subset of the capacitor action space composed of all capacitor switching states. ci is the switching state of the capacitor labeled i, C i represents the set of actions that the ith capacitor can take, c i1 To remove the capacitor, c i2 To put in capacitors; Act O =[Act o1 ,Act o2 ,…,Act oi ] The i =[o i1 ,o i2 ,o i3 ,...,o ix ] In the formula, Act O Act represents the subset of transformer action space composed of all on-load tap-changing transformer gear states. oi Represents the gear status of the on-load tap-changing transformer No. i, O i represents the set of actions that the ith transformer can take, o ix Indicates its different joints; Act G =[Act g1 ,Act g2 ,…,Act gi ] G i =[g i1 ,g i2 ] In the formula, Act G Act represents the subset of the generator action space composed of all generator output states. gi Indicates the action of the i-th generator set, G i represents the set of actions that the i-th generator can take, g i1 In order to increase the active output of the generator by a gradient, g i2 To reduce the active output of the generator by a gradient; Act S =[Act s1 ,Act s2 ,…,Act si ] S i =[s i1 ,s i2 ] In the formula, Act S Act represents the subset of the generator action space consisting of all circuit breaker opening and closing states. si Represents the action of circuit breaker No. i, S i represents the set of actions that the i-th circuit breaker can take, s i1 Indicates that the circuit breaker is opened, s i2 Indicates that the circuit breaker is closed.

6. The multi-objective dispatch optimization control method for a power system according to claim 5, characterized in that: The reward function of the reinforcement learning agent is constructed based on the global weighted voltage offset value of the system, as follows: reward 1t =δU t_1 -δU t_2 That is, at time t, the reward is the reduction of the weighted voltage offset value before and after the action of the agent; where δU t_1 , δU t_2 Represents the voltage of each node before and after the intelligent agent action at time t; And make the following constraints on the control voltage required by the model: In the formula is the voltage of each node; When the voltage of a node exceeds the safe range, the agent will receive a reward 2t as follows: Where, M is 1000; U k is the per unit value of voltage at any node; And set a reward for the generator set to exceed the output limit 3t , as follows: Where N is 5, P gen i is the real-time output value of any generator set, P gen j_min and P gen j_max They are the upper and lower limits of the generator set output respectively; The global reward function of the reinforcement learning agent is as follows: reward=reward 1t +reward 2t +reward 3t 。 7. The multi-objective dispatch optimization control method of the power system according to claim 6 is characterized in that: The training method of the reinforcement learning agent is as follows: During training, the parameters of the two networks "actor_net" and "critic_net" in the proximal policy optimization deep reinforcement learning algorithm are first initialized, and a buffer is set to input the load power curve, the distributed power system power curve, the initial states of the capacitors, line switches and on-load voltage regulator taps, and the initial output of the generator set; For any time, when the agent starts at time t k-1 Transition to time t k When the action sequence group is generated, the power flow calculation is performed to obtain the initial state of each node voltage before the agent takes action, and the initial state is input into the "actor_net" and "critic_net" networks respectively. The "actor_net" makes a judgment to obtain the strategy. When the action sequence group is generated, the default load power consumption and photovoltaic system output power have not changed. The agent updates the state of the actionable device and performs power flow calculation again to obtain a new state. The state will also be returned to the agent. The two states are simultaneously input into the "critic_net" to obtain the corresponding reward value. Vector (S k- , S k+ , R k , A k ) will be placed in the cache, where S k- Represents time t k The state space set before the agent's action, S k+ Represents time t k The state space set after the agent takes action, R k t k The reward value obtained by the agent's action at that moment, A k Then t k The action taken by the agent at that moment. After each trial, the agent will randomly extract L groups of vectors from the cache at that moment in the previous training process and store them in the L group (S k- , S k+ ) is put into "critic_net", the network will update the parameters and redefine the advantage function of its output based on the original output value of the network. k- , A k ), it will be put into "actor_net", calculate the probability ratio of taking the action under the new and old strategies, update the network parameters, and generate a new strategy function; In from t k to k+1 Within fifteen minutes, the system will maintain steady state S k , until reaching time t k+1 , the system load power consumption and distributed power system output power change, the power flow calculation will be performed again to the new state S k+1- , that is, t k+1 The initial state before the agent takes action at any moment, and continues to take new action group A k+1 , then the decision-making training process of the agent is related to time t k Consistent.

8. A multi-objective dispatch optimization control system for power system based on reinforcement learning, characterized in that: The multi-objective dispatch optimization control system of the power system comprises: An acquisition module is used to obtain the power system topology and the historical voltage data of each node in multiple scenarios; The optimization control module is used to input voltage data into the trained intelligent optimization control model to obtain a real-time optimization control solution for the power system voltage; The intelligent optimization control model is obtained by introducing a system weighted voltage offset value and training a proximal strategy optimization reinforcement learning agent; The method for calculating the weighted voltage offset value of the system is as follows: using the Monte Carlo method to analyze and obtain the sensitivity matrix of each voltage node to each current branch, the sensitivity vector to the real-time total network loss of the system, and the sensitivity vector to the real-time total reactive compensation of the system, weighted addition and normalization are performed to obtain the total weighted sensitivity vector, which is defined as the "weight" of each node; multiplying the node "weight" with its sequential operation voltage offset value is defined as the node weighted voltage offset value, and the global accumulation is obtained.

9. The multi-objective dispatch optimization control system of the power system according to claim 8, characterized in that: The system weighted voltage offset value is obtained by the following steps: Calculate the system's weighted voltage offset value δU with respect to current α , Weighted voltage offset value δU for loss β , Regarding the weighted voltage offset value δUγ of the system reactive power compensation: δU α =α1δu1+α2δ2+α3δu3+...+α n you n δU β =β1δu1+β2δu2+β3δu3+...+β n you n δU γ =γ1δu1+γ2δu2+γ3δu3+...+γ n you n In the formula, δu n is the voltage offset value of the nth node, α n is the "weight" of the nth node in the weighted voltage offset value for the branch current after processing, β n is the "weight" of the circuit loss of the nth node after processing in the weighted voltage offset value, γ n is the "weight" of the reactive power compensation of the system by the nth node after processing in the weighted voltage offset value; Comprehensively control the priority of the target and set the weighted voltage coefficient of each part: A+B+C=1 Where A, B, and C correspond to the weighted voltage coefficients of branch current, system loss, and reactive power compensation respectively; Calculate the total weighted voltage offset value δU of the system:

10. The multi-objective dispatch optimization control system of the power system according to claim 9, characterized in that: The state space of the reinforcement learning agent includes the state space of each phase voltage of each node, the switching and gear state of the controllable equipment, as follows: S=[Stau,StaC,StaO,StaG,StaS] Where S is the entire state space, which is decomposed into five subsets including node voltage information and the states of various controllable devices: Where Stau represents the state space voltage subset composed of each phase voltage, represents the first The node voltage of the phase, where It consists of three phases: a, b, and c, and the phase difference between the voltages of each phase is 120 degrees; StaC=[Stac1,Stac2,...,Staci] StaO=[Stao1, Stao2,…,Staoi] StaG=[Stag1,Stag2,...,Stag1 StaS=[Stas1, Stas2,...,Stasi] Wherein, StaC represents the state space capacitor subset composed of the switching states of all capacitors, Staci represents the switching state of each capacitor; StaO represents the state space transformer subset composed of the gear states of all on-load tap-changing transformers, Staoi represents the tap gear state of each on-load tap-changing transformer; StaG represents the state space generator subset composed of the output states of all generators, StaG represents the real-time active output of each generator set; StaS represents the state space circuit breaker subset composed of the opening and closing states of all circuit breakers, Stasi represents the opening and closing states of each circuit breaker; The action space Act of the reinforcement learning agent includes four sub-action sets Actc, Acto, ActG, and Acts, which are represented as follows: Act=[Actc,Acto,ActG,Acts] ActC=[Actc1,Actc2,…,Actci] Ci=[ci1,ci2] Where, ActC represents the subset of the capacitor action space formed by the switching states of all capacitors, Actci is the switching state of the capacitor labeled i, Ci represents the action set that can be taken by the i-th capacitor, ci1 is to cut off the capacitor, and ci2 is to put the capacitor in; Acto=[Acto1,Acto2,…,Actoi] Oi=[oi1,oi2,oi3,...,oix] Where, ActO represents the subset of transformer action space composed of the gear states of all on-load tap-changing transformers, Actoi represents the gear state of the i-th on-load tap-changing transformer, Oi represents the set of actions that can be taken by the i-th transformer, and oix represents its different joints; ActG=[Actg1,Actg2,…,Actgi] Gi=[gi1,gi2] Where, ActG represents the subset of the generator action space composed of all generator output states, Actgi represents the action of the i-th generator group, Gi represents the action set that the i-th generator group can take, gi1 is to increase the active output of the generator by one gradient, and gi2 is to reduce the active output of the generator by one gradient; ActS=[Acts1,Acts2,…,Actsi] Si=[si1,si2] Where, ActS represents the subset of the generator action space composed of the opening and closing states of all circuit breakers, Actsi represents the action of the i-th circuit breaker, Si represents the action set that the i-th circuit breaker can take, si1 represents opening the circuit breaker, and si2 represents closing the circuit breaker; The reward function of the reinforcement learning agent is constructed based on the global weighted voltage offset value of the system, as follows: reward 1t =δUt _1 -δUt _2 That is, at time t, the reward is the reduction of the weighted voltage offset value before and after the action of the agent; where δUt _1 , δUt _2 Represents the voltage of each node before and after the intelligent agent action at time t; And make the following constraints on the control voltage required by the model: In the formula is the voltage of each node; When the voltage of a node exceeds the safe range, the agent will receive a reward 2t as follows: Where, M is 1000; Uk is the per-unit value of any node voltage; And set a reward for the generator set to exceed the output limit 3t , as follows: Where N is 5, Pgeni is the real-time output value of any generator set, and Pgenj is _min and Pgenj _max They are the upper and lower limits of the generator set output respectively; The global reward function of the reinforcement learning agent is as follows: reward=reward 1t +reward 2t +reward 3t 。

Images