Power grid power flow convergence adjustment method based on PPO-HELM

Abstract

The invention discloses a power grid power flow convergence adjustment method based on PPO-HELM. The method comprises the following steps: firstly, introducing a voltage stability judgment index and a voltage collapse point index by using a power flow calculation method of HELM; secondly, after a power flow non-convergence state is generated by adjusting the power grid load level, voltage collapse point index values in the state are calculated through HELM and ranked from large to small, and first m corresponding load nodes are selected to participate in power flow adjustment according to the ranking result; and then according to the voltage stability judgment index, the voltage collapse point index and the selected load node, establishing a PPO algorithm environment based on HELM. And finally, constructing an objective function, building a PPO algorithm model, outputting a power flow convergence adjustment action, and carrying out training and testing. The number of load nodes needing capacitor switching operation is reduced, the active power and reactive power adjusting time is shortened, and the adjusting efficiency is improved.

Description

Technical Field

[0001] The invention belongs to the field of electric power information technology and relates to a power grid power flow convergence adjustment method based on PPO-HELM. Background Art

[0002] The convergence of power flow calculation is the key to analyzing the safety and stability of power systems. If the power flow calculation cannot converge, it may mean that the system is overloaded or has other potential problems. Power flow calculation is not only applicable to normal operation, but also can be used to evaluate the overall performance of the system. If the power flow calculation cannot converge, it means that there may be design problems in the system, which need to be optimized to ensure the stability and safety of the system.

[0003] The problem of non-convergence of power flow may be caused by two reasons: one is that there is no solution in the system itself, and the other is that the algorithm cannot effectively find a solution. Traditional power flow calculation methods are difficult to effectively solve the convergence problem when facing uncertain factors such as load fluctuations and renewable energy. The fully embedded power flow calculation method can avoid iterations, reduce computational complexity, and improve reliability. Traditional methods and machine learning algorithms (such as support vector regression (SVR) and ridge regression (RR) algorithms) have poor efficiency and real-time response in power flow adjustment and are difficult to adapt to dynamic changes in the power grid. Therefore, it is necessary to introduce advanced power flow calculation methods and deep reinforcement learning algorithms to improve the adaptability and convergence speed of power flow adjustment. Summary of the invention

[0004] In view of the above problems, a power grid flow convergence adjustment method based on HELM-PPO is proposed. This method combines the Holomorphic Embedding Load Flow Method (HELM) and the Proximal Policy Optimization (PPO) algorithm for the convergence adjustment of power grid flow. The HELM flow calculation method is adopted, and the voltage stability evaluation index is introduced. According to the voltage collapse point index, the load nodes participating in the flow adjustment are screened out, and the capacitor switching operation is performed on these nodes. In addition, the average value of the voltage stability judgment index of each node and the maximum value of the voltage collapse point index of each node are used to set the reward mechanism of the PPO algorithm. The IEEE 14-node system and the IEEE57-node system are used as examples to verify the effectiveness of the proposed method. The present invention proposes to combine the Holomorphic Embedding Method (HELM) with the Proximal Policy Optimization (PPO) deep reinforcement learning algorithm to deal with the power grid flow convergence problem. The voltage collapse point index and voltage stability index calculated by the HELM power flow calculation method can be used to evaluate whether the power flow solution of the system is in a stable state, and can guide the power grid power flow to adjust in a stable direction. The PPO algorithm makes full use of the advantages of deep learning and reinforcement learning algorithms. It can not only use neural networks to learn the complex nonlinear characteristics of the power grid, but also effectively deal with high-dimensional problems.

[0005] Specifically, the HELM method is used to calculate the power flow, and the voltage stability index L based on the power flow solution is introduced. ij , and the voltage stability index VSI based on sensitivity analysis HELM1i and Voltage Collapse Indicator VSI HELM2i . Calculate the VSI of each node in the power grid HELM1i , and find the average VSI of these nodes ave . Calculate the VSI of each node in the power grid HELM2i value, and find the maximum VSI among them max Secondly, calculate the voltage stability index VSI of each node HELM2i , and select the load nodes participating in the power flow adjustment based on this index, and select the generators whose initial active power is less than a certain value to join the power flow adjustment process, thereby reducing the number of nodes that need to participate in the adjustment, shortening the adjustment time of active power and reactive power, and improving the stability of the power grid. Finally, the PPO model is constructed and combined with VSI ave and VSI maxThe trained model can monitor the power grid flow data in real time, make intelligent adjustments based on real-time changes, and quickly respond to the impact of power flow fluctuations to ensure the continuous and stable operation of the power grid.

[0006] The method of the present invention specifically comprises:

[0007] (1) Introducing the voltage stability index VSI using the HELM power flow calculation method HELM1i and Voltage Collapse Indicator VSI HELM2i .

[0008] The voltage stability index VSI is introduced by calculating the sensitivity of each node voltage based on HELM. HELM1i and Voltage Collapse Indicator VSI HELM2i .

[0009] According to HELM, the sensitivity of each order of node voltage can be calculated. First, an embedded pure virtual function is constructed, as shown in formula (1).

[0010]

[0011] Where V i [n] represents the nth voltage component of the voltage at node i in the HELM power flow calculation, s n Represents the nth-order term of the frequency domain operator s.

[0012] Nonlinear sensitivity of voltage to injected power and As shown in formula (2):

[0013]

[0014] Through theoretical derivation and simulation analysis, a stability judgment index VSI is proposed HELM1i :

[0015]

[0016] Calculate the VSI of each node HELM1i value, and find the VSI of these nodes HELM1i The average value of VSI ave :

[0017]

[0018] Where: VSI HELM1i is the voltage stability index, node i is the bus number of the distribution network (i∈2~T), node j is the bus number of reactive power injection, and real refers to the real part of the corresponding complex number. Q j represents the reactive power of node j, P jrepresents the active power of node j, V i [n] represents the nth coefficient term of the voltage at node i. T represents the total number of nodes. The nth coefficient term representing the voltage at node i is used to find the partial derivative of the reactive power at node j; The nth coefficient term representing the voltage at node i is calculated with respect to the active power at node j. VSI HELM1i The smaller the value, the better the voltage stability of the system.

[0019] In order to determine the voltage stability of the system as a whole, an indicator VSI is added HELM2i :

[0020]

[0021] When VSI HELM2i <0, the power grid system is stable, otherwise, the voltage of the distribution network system has reached the voltage collapse point.

[0022] Calculate the VSI of each node HELM2i Values, and find the maximum value VSI max :

[0023]

[0024] VSI in step (1) HELM1i and VSI HELM2i The data output by the indicator is node data, so increase L ij Indicators to check the stability of each transmission line. ij The calculation formula is as follows:

[0025]

[0026] Where: X ij represents the reactance between node i and node j, R ij represents the resistance between node i and node j, U i The voltage amplitude of node i; when L ij <1, the power grid system is in a voltage stable state, L ij The smaller the index, the higher the voltage stability; when L ij When the index is close to 1, it means that the voltage stability of the system decreases and there is a risk of voltage collapse. ij When ≥1, the grid system voltage has lost stability and entered a voltage collapse state.

[0027] Calculate L for each line in the grid ij value, and select the maximum value as L max :

[0028]

[0029] (2) Select nodes involved in power flow adjustment;

[0030] After adjusting the grid load level to generate a non-convergent power flow state, HELM is used to calculate the VSI in this state. HELM2i value, and VSI HELM2i The values ​​are sorted from largest to smallest, and the first m VSIs are selected based on the sorting results. HELM2i The load nodes i corresponding to the value participate in the flow adjustment. Switching capacitors on and off at these load nodes to adjust reactive power can effectively adjust the node voltage level and optimize the grid voltage distribution. At the same time, the generators with small initial output and large adjustment margin are used for active power adjustment, so that they have flexible positive and negative power adjustment capabilities and achieve more effective flow distribution.

[0031] (3) Based on the voltage stability index VSI HELM1i , Voltage collapse index VSI HELM2i As well as the selected load nodes, the HELM-based PPO algorithm environment is established.

[0032] The deep reinforcement learning model of Proximal Policy Optimization (PPO) consists of an environment and an agent. The environment gives a state s t , the agent takes action a t And feed the result back to the environment, which then generates a new state s t+1 and reward R, and the experience tuple (s t ,a t ,s t+1 ,R) Deposit into experience pool Then, the experience pool Sampling is performed to train the agent. The agent learns the optimal strategy by interacting with the environment. This interaction process involves two key networks: the Actor network and the Critic network. The goal of the Actor network is to maximize the cumulative return of the strategy π, and the goal of the Critic network is to optimize the value network by minimizing the loss function. The Actor network contains a current strategy network π θ (a t |s t ) and the old policy network π θold (a t |s t ), while the Critic network includes the value network The PPO algorithm combines the advantages of deep learning and reinforcement learning, using deep neural networks to approximate value functions and policy functions, enabling intelligent agents to estimate these functions more accurately, thereby improving learning efficiency and performance.

[0033] First, we need to build the environment. Gi , reactive power Q of load S Li , VSI of each node HELM1i The mean of the values ​​and the VSI in each node HELM2i The maximum value of state s t :

[0034]

[0035] In the formula, Indicates the generator G i The active power, Indicates load L i Reactive power, VSI ave Indicates the VSI of each node HELM1i The mean of the values, VSI max Indicates the VSI in each node HELM2i Set the upper limit of the generator active power Gs max and lower limit Gs min And the upper limit of load reactive power Ls max and lower limit Ls min :

[0036]

[0037] In the formula, Represents the generator G i The upper limit of active power, Indicates the generator G i The lower limit of active power, Indicates load L i The upper limit of reactive power, Indicates load L i Lower limit of reactive power.

[0038] According to the nodes selected for power flow adjustment in step (2), power flow adjustment is achieved by adjusting the generator output or capacitor switching operation of these nodes. Action a t as follows:

[0039]

[0040] In the formula, Indicates the generator G i The change in active output, Indicates load L i The change in reactive power.

[0041] a tIt is a continuous space, and upper and lower limits are set for the action value space:

[0042]

[0043] In the formula, Indicates the maximum output of a single-action generator. Indicates the minimum output of a single-action generator; Indicates the maximum value of a single-action capacitor switching. Indicates the minimum value of single-action capacitor switching, ΔP minG is the lower limit of the active output of the generator, ΔP maxG is the upper limit of the active output of the generator; ΔQ minL is the minimum value of load reactive power, ΔQ maxL is the maximum value of the load reactive power.

[0044] The next state after taking the action is:

[0045] s t+1 ~f(s t+1 |s t ,a t ) (15)

[0046] Where: f is the state transfer function.

[0047] Reward mechanism setting: When the status is within the valid range, that is, when it does not exceed the preset upper and lower limits, according to VSI max and VSI ave To set the reward function.

[0048] The reward function is set as follows:

[0049] 1. VSI max When it is a positive number, according to VSI max Initial value VSI ini To set the reward function. The details are as follows:

[0050] a. If 0 <VSI max ≤(1.5*VSI ini ), when VSI max In this range, set the continuous reward R = (-4*VSI max ) / (3*VSI ini ).

[0051] b.VSI max >(1.5*VSI ini ), when VSI max If it is within this range, a penalty R=-5 is given.

[0052] 2. VSI max For non-positive numbers:

[0053] a. In VSI ave ≥ω 1 *VSI o In the case of 1 *VSI o ) / VSI ave , where VSI o For VSI ave The initial value of ω 1 is the coefficient.

[0054] b. In VSI ave <ω 1 *VSI o In this case, a reward R=5 is given.

[0055] (4) Construct the objective function, build the PPO algorithm model, and output the flow convergence adjustment action.

[0056] After the environment is created, the intelligent agent is built. The initial objective function of the PPO algorithm is As shown below:

[0057]

[0058] In the formula, ξ is the ratio of new and old strategies; is the advantage function estimate; θ is the parameter of the Actor network; E is the expectation.

[0059]

[0060] In the formula, π θ is the current strategy, π θold is the old strategy, θ is the parameter of the actor network, π θ (a t |s t ) is the current strategy in state s t Take action a t The probability of θold (a t |s t ) is the old strategy in state s t Take action a t The probability of; t is the time step; s t 、a t They are the states s observed at the tth time step. t and the action taken t .

[0061]

[0062] In the formula The target value of the state value network; V(s t ) is in state s t The state value function under r t is the reward obtained at the tth time step; γ is the discount factor; V(s t+1 ) is in state s t+1 The state value function under .

[0063] In order to limit the magnitude of policy updates, PPO introduces a clipping objective function as follows:

[0064]

[0065] Where clip(ξ,1-ε,1+ε) is the clipping function, which limits the size of the ratio of the new and old strategies ξ to the range of [1-ε,1+ε] to prevent drastic changes in the strategy.

[0066] Furthermore, in order to improve the performance of the strategy while maintaining the stability of the training algorithm, PPO combines the optimization of the value function and the entropy term. The final PPO objective function consists of three parts:

[0067]

[0068] In the formula, c 1 is the weight coefficient of the value function objective term, c 2 is the weight coefficient of the entropy objective term.

[0069] Value function objective term and entropy objective The calculation formula is as follows:

[0070]

[0071] H(π(·|s t ))=-π θ (a|s)logπ θ (a|s) (24)

[0072] In the formula Represents the parameters of the Critic network, For state s t The state value function under represents the target value of the state value network, H(π(·|s t )) is the entropy of the strategy, π θ (a t |s t ) is the current strategy in state s t Take action a tThe probability of , log represents logarithmic operation.

[0073] Loss function of the PPO algorithm The calculation formula is as follows:

[0074]

[0075] Use gradient descent algorithm to minimize the loss function Update the policy network parameters θ and value network parameters The calculation process is shown in the formula:

[0076]

[0077] Where α is the learning rate and β is the entropy regularization coefficient.

[0078] (5) Train and test the PPO algorithm model to obtain the results of power flow convergence adjustment.

[0079] When HELM is used for power flow calculation, a non-convergent power flow state can be generated by adjusting the load level in the power grid, which serves as a training sample for the PPO algorithm model.

[0080] Beneficial effects of the present invention: HELM is used to calculate the voltage stability index VSI of each node HELM2i , and according to VSI HELM2i The load nodes participating in the power flow adjustment are selected by the value, thereby reducing the number of load nodes that need to perform capacitor switching operations. At the same time, generators with initial active power less than a certain value are selected to participate in the adjustment. These two screening processes help shorten the active power and reactive power adjustment time and improve the adjustment efficiency. In addition, by using VSI max 、VSI ave The PPO algorithm's reward function is set based on the indicator, which can more comprehensively monitor the power grid status, promptly identify the situation where the power flow does not converge, and accelerate the convergence speed of reward accumulation, further improving the algorithm's adjustment accuracy and response speed. The trained model can adjust the power grid power flow from a non-convergent state to a convergent state, and further adjust the power grid to a more stable level. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is the PPO schematic diagram;

[0082] Figure 2 It is the PPO network structure diagram;

[0083] Figure 3 It is a training flow chart;

[0084] Figure 4 The cumulative reward for the PPO algorithm of the IEEE14-node system;

[0085] Figure 5(a) shows the IEEE 14-node system VSI HELM2i Comparison chart before and after value adjustment;

[0086] Figure 5(b) shows the IEEE 14-node system VSI HELM1i Comparison chart before and after value adjustment;

[0087] Figure 5(c) shows the IEEE 14-node system L ij Comparison chart before and after value adjustment. DETAILED DESCRIPTION

[0088] First, based on the HELM PPO environment, a node power grid model is established and the power grid flow data is initialized. Initialize network parameters ψ, φ and θ. In the process of the PPO algorithm, at the beginning of training, the HELM method is first used to analyze the current state s t The power flow calculation of the power grid model under the current state can be performed to obtain the power flow data of the power grid in the current state. Calculate the stability index and take action a t In order to determine the active power of the generator and the reactive power of the load involved in the power flow regulation, a t Get t+1 and reward R, then (s t , a t ,s t+1 , R) and store it in the experience pool , which is used for subsequent training. As the data in the experience pool is continuously updated, the network parameters will also be continuously adjusted and updated. The training process will continue until the stopping condition is met.

[0089] a. Reference Figure 1 , the present invention is carried out according to the following steps:

[0090] b. Step (1): Use the power flow calculation method based on holomorphic function embedding to build the IEEE14 power grid model. By using HELM and pandapower to calculate the power flow of the 14-node power grid model, the power flow data of each node and each branch obtained by the two methods are compared. The results show that the power flow calculation results of the two methods are consistent.

[0091] c. Step (2): Calculate VSI HELM2i Indicators are used to select load nodes that participate in power flow adjustment.

[0092] Calculate the VSI of each node based on the current non-convergence state of the power grid. HEIM2i Indicators and output VSI of all nodes HELM2i The calculation results are shown in Table 1:

[0093] Table 1 VSI of each branch of IEEE14 power grid modelHELM2i Value

[0094]

[0095]

[0096] According to the calculation results, the adjustment is made by adjusting the reactive power injection of the loads corresponding to node numbers 4 and 5. At the same time, all the generators in the system are selected to participate in the power flow adjustment.

[0097] d. Step (3): Establish a HELM-based PPO algorithm environment.

[0098] First, based on the calculation results, the reactive power of the loads on nodes 4 and 5 and the active power output of all generators are determined as adjustable variables. Then, the active power of the generators Reactive power of load VSI of each node HELM1i The mean of the values ​​and the VSI in each node HELM2i The maximum value of the current state s t , get the state s at the next moment t+1 . Using VSI-based max and VSI ave The reward R is determined by the calculation formula, and the size of the reward is used to evaluate the performance of the model.

[0099] e. Step (4): Build the PPO algorithm model.

[0100] refer to Figure 2 , the input of the Actor network is the state, and the output is the action probability distribution parameter. The input of the Critic network is the state, and the output is the target value of the state value network and the state value function Network update process reference for PPO deep reinforcement learning Figure 3 In the Actor network, the Actor is based on the current state s given by the environment. t Execution Strategy θ , calculate the new and old strategy ratio ξ and the clipping objective function And according to the strategy π θ Calculate Entropy Objective

[0101] In the Critic network, the target value based on the state value network and the current state value function Calculate the advantage function and value function objective term By clipping objective function Value function objective term and entropy objective The loss functions that make up PPO By adjusting the parameters θ and Perform gradient descent training updates to achieve the policy network π θ and value network training and optimization.

[0102] f. Step (5): Train and test the model to obtain the final adjustment result.

[0103] The flow convergence adjustment process is as follows:

[0104]

[0105]

[0106] This embodiment uses the Pytorch framework to build a neural network and uses the Adam optimizer for training. Adam is a commonly used adaptive gradient descent algorithm that can usually converge to better results in a shorter time when training a neural network. When using the PPO algorithm based on the Pytorch framework for training, the PPO algorithm parameter settings are shown in Table 2:

[0107] Table 2 PPO algorithm parameter settings

[0108]

[0109] Two experimental schemes are set up in the IEEE14 power grid model. In scheme 1, VSI is used max and VSI ave Indicators are used to set the reward mechanism of the algorithm. Then, in scheme 2, L max The reward mechanism of the indicator reset algorithm, the reward function is set as follows:

[0110] 1.L max <1:

[0111] Give reward R=5.

[0112] 2.L max ≥1, according to L max The initial value L ini The size sets the reward function. The details are as follows:

[0113] a.1 <L max ≤(1.3*L ini ), when L max When in this range, a reward of R = -2L is given max / (1.3*L ini ).

[0114] b. If L max >(1.3*L ini ), when L max When in this range, a reward R=-5 is given.

[0115] Solution 3: Based on the IEEE 14-bus system of pandapower, the convergence flag of the power flow calculation can be judged by the return value of the runpp function. When the power flow converges, R = 5, and when it does not converge, R = 0.

[0116] In the IEEE 14-node system, the total number of training steps k total is 35000 times, p total is 100, and the number of training rounds is 350. Cumulative rewards for the two experimental schemes are referenced Figure 4 The curve of Scheme 1-PPO-VSI shows that the PPO algorithm is used in Scheme 1 and VSI is introduced. max and VSI ave The cumulative reward curve obtained by the indicator. The curve of Scheme 2-PPO-L represents the use of the PPO algorithm in Scheme 2 and the introduction of L max The cumulative reward curve obtained by the indicator. The curve of Scheme 3-PPO represents the cumulative reward curve obtained by using the PPO algorithm in Scheme 3. Figure 4 According to the cumulative reward curve analysis, in Scheme 1, the reward gradually increases in the first 160 training rounds, stabilizes after 160 rounds, and the cumulative reward remains at a high level. In contrast, in Scheme 2, it can be observed from the curve that the reward curve fluctuates greatly in the first 60 rounds, stabilizes after 60 rounds, but the reward curve remains at a low level, which indicates that there are certain limitations in the reward setting in Scheme 1. In Scheme 3, the cumulative reward always remains at the 0 scale line, which indicates that during the trend adjustment process, the trend convergence has never occurred.

[0117] The above trained model is tested in the initial environment. The change curves of each indicator before and after the flow adjustment are respectively referred to Figure 5(a)-Figure 5(c) .from Figure 5(a)-Figure 5(c) As can be seen from Table 3, in Scheme 1, before adjustment, VSI max >0, the VSI of each node after adjustment HELM2i <0, then VSI max <0. This indicates that the power flow of the power grid has successfully converged after the power flow has been adjusted. Figure 5(a)-Figure 5(c) It can be seen that after adjustment, the VSI of each node HELM1i The values ​​of Figure 5(a)-Figure 5(c) The test results in Table 3 show that in Scheme 2, before adjustment, L max >1, after the flow adjustment, L maxIt has decreased but is still greater than 1. This shows that after adjustment, the power flow in Scheme 2 still does not converge and the adjustment fails. The above results show that the introduction of VSI max and VSI ave The power flow adjustment results after L max The result after.

[0118] Table 3 Values ​​of various indicators before and after adjustment of IEEE 14-bus system

[0119]

Claims

1. A power grid power flow convergence adjustment method based on PPO-HELM, characterized in that: The following steps are involved: Step 1: Use HELM's power flow calculation method to introduce the voltage stability index VSI HELM1i and Voltage Collapse Indicator VSI HELM2i ; Step 2: After adjusting the grid load level to generate a non-convergent power flow state, use HELM to calculate the VSI in this state HELM2i value, and VSI HELM2i The values ​​are sorted from largest to smallest, and the first m VSIs are selected based on the sorting results. HELM2i The load node i corresponding to the value participates in the power flow adjustment; Step 3: Based on the voltage stability index VSI HELM1i , Voltage collapse index VSI HELM2i and the selected load nodes, establishing the HELM-based PPO algorithm environment; Step 4: Construct the objective function, build the PPO algorithm model, and output the power flow convergence adjustment action; Step 5: Train and test the PPO algorithm model to obtain the result of power flow convergence adjustment.

2. The power grid power flow convergence adjustment method based on PPO-HELM according to claim 1, characterized in that: The specific implementation process of step 1 is as follows: According to HELM, the sensitivity of each order of grid node voltage is calculated, and the embedded pure virtual function V is constructed. i (s): Where V i [n] represents the nth voltage component of the voltage at node i in the HELM power flow calculation, s n represents the nth order term of the frequency domain operator s; Nonlinear sensitivity of voltage to injected power and As shown in formula (2): Propose stability judgment index VSI HELM1i : Calculate the VSI of each node HELM1i value, and find the VSI of these nodes HELM1i The average value of VSI ave ; Wherein node i is the bus number of the distribution network, node j is the bus number of reactive power injection, and real refers to the real part of the corresponding complex number; Q j represents the reactive power of node j, P j represents the active power of node j, V i [n] represents the nth coefficient term of the voltage at node i; T represents the total number of nodes; The nth coefficient term representing the voltage at node i is used to find the partial derivative of the reactive power at node j; The nth coefficient term representing the voltage at node i is calculated as the partial derivative of the active power at node j; Added indicator VSI HELM2i : When VSI HELM2i <0, the power grid system is stable, otherwise the voltage of the distribution network system has reached the voltage collapse point; Calculate the VSI of each node HELM2i Values, and find the maximum value VSI max ; VSI HELM1i and VSI HELM2i The data output by the indicator is node data, increasing L ij Indicators to check the stability of each transmission line, L ij The calculation formula is as follows: Where: X ij represents the reactance between node i and node j, R ij represents the resistance between node i and node j, U i The voltage amplitude of node i; when L ij <1, the power grid system is in a voltage stable state; when L ij When ≥1, the power grid system enters a voltage collapse state; Calculate L for each line in the grid ij value, and select the maximum value as L max .

3. The power grid power flow convergence adjustment method based on PPO-HELM according to claim 2, characterized in that: The HELM-based PPO algorithm environment is as follows: The deep reinforcement learning model of proximal policy optimization PPO includes an environment and an agent. The environment gives a state s t , the agent takes action a t And feed the result back to the environment, which then generates a new state s t+1 and reward R, and the experience tuple (s t ,a t ,s t+1 ,R) Deposit into experience pool Then the experience pool Perform sampling training on the agent. The agent learns the optimal strategy through interaction with the environment. The interaction process includes: Actor network and Critic network. The Actor network contains a current strategy network π θ (a t |s t ) and the old policy network π θold (a t |s t ), while the Critic network includes the value network The active power P of the generator Gi , reactive power Q of load S Li , VSI of each node HELM1i The mean of the values ​​and the VSI in each node HELM2i The maximum value of state s t : In the formula, For the generator G i The active power, is the load L i Reactive power; Set the upper limit of the generator active power Gs max and lower limit Gs min And the upper limit of load reactive power Ls max and lower limit Ls min : In the formula, Represents the generator G i The upper limit of active power, Indicates the generator G i The lower limit of active power, Indicates load L i The upper limit of reactive power, Indicates load L i Lower limit of reactive power; According to the nodes selected for power flow adjustment in step 2, power flow adjustment is achieved by adjusting the generator output or capacitor switching operation of these nodes. Action a t as follows: In the formula, For the generator G i The change in active output, is the load L i Change in reactive power; a t It is a continuous space, and upper and lower limits are set for the action value space: In the formula, Indicates the maximum output of a single-action generator. Indicates the minimum output of a single-action generator; Indicates the maximum value of a single-action capacitor switching. Indicates the minimum value of single-action capacitor switching, ΔP minG is the lower limit of the active output of the generator, ΔP maxG It is the upper limit of the active output of the generator; ΔQ minL is the minimum value of load reactive power, ΔQ maxL is the maximum value of the load reactive power; The next state after taking the action is: s t+1 ~f(s t+1 |s t ,a t ) (12) Where: f is the state transfer function; Reward mechanism setting: When the status is within the valid range, according to VSI max and VSI ave Set the reward function.

4. The power grid power flow convergence adjustment method based on PPO-HELM according to claim 3, characterized in that: The reward function is specifically set as follows: VSI max is a positive number, according to VSI max Initial value VSI ini Set up the reward function; it is as follows: a. If 0 <VSI max ≤(1.5*VSI ini ), when VSI max In this range, set the continuous reward R = (-4*VSI max ) / (3*VSI ini ); VSI max >(1.5*VSI ini ), when VSI max If it is within this range, a penalty R=-5 is given; VSI max is a non-positive number: In VSI ave ≥ω1*VSI o In the case of o ) / VSI ave ; Among them, VSI o For VSI ave The initial value of , ω1 is the coefficient; In VSI ave <ω1*VSI o In this case, a reward R=5 is given.

5. The power grid power flow convergence adjustment method based on PPO-HELM according to claim 4, characterized in that: The specific process of step 4 is as follows: After the environment is created, the intelligent agent is built; the initial objective function of the PPO algorithm As shown below: ξ is the ratio of new and old strategies; is the advantage function estimate; θ is the parameter of the Actor network; E is the expectation; PPO introduces a clipping objective function as follows: Where clip(ξ,1-ε,1+ε) is the clipping function, which limits the size of ξ to the range of [1-ε,1+ε]; The final PPO objective function consists of three parts: Where c1 is the weight coefficient of the value function objective term, and c2 is the weight coefficient of the entropy objective term; Value function objective term and entropy objective The calculation formula is as follows: H(π(·|s t ))=-π θ (a|s)logπ θ (a|s) (18) In the formula Represents the parameters of the Critic network, For state s t The state value function under represents the target value of the state value network, H(π(·|s t )) is the entropy of the strategy, π θ (a t |s t ) is the current strategy in state s t Take action a t The probability of logarithmic operation; the loss function of the PPO algorithm is 6. The power grid power flow convergence adjustment method based on PPO-HELM according to claim 5, characterized in that: In the step 5, it also includes, when using HELM to perform power flow calculation, generating a non-convergent power flow state by adjusting the load level in the power grid as a training sample for the PPO algorithm model.

Images