A power boosting method for power transmission corridor containing embedded direct current

Abstract

The application discloses a power transmission corridor power improvement method containing embedded direct current, first, historical tidal section files are collected to extract power grid state information; second, an intelligent agent model for power grid power transmission line power control is constructed, and samples are analyzed; then, a simulation environment reset process is constructed, actions output by the intelligent agent are written into a BPA format file used for tidal solution, and the intelligent agent-environment interaction process is formed together with the power grid state information collection process; subsequently, an intelligent agent training process based on reinforcement learning is implemented; finally, the training result is tested. The application adopts the method of reinforcement learning to realize the optimization decision of the problem, can avoid the difficulty problem caused by the nonlinear power grid tidal equation, and significantly improves the decision speed; the application can guarantee good initial decision ability, and the intelligent agent can be further optimized according to the change of the system tidal characteristics in actual operation.

Description

Technical Field

[0001] This invention belongs to the field of power system technology, and in particular relates to a method for power enhancement of a transmission corridor with embedded DC. Background Technology

[0002] The characteristics of AC / DC systems are complex, and the safety and stability of power grids are prominent issues. First, the increasingly complex grid structure of the receiving-end power grid, with its AC / DC hybrid configuration, raises the probability of system failures. Second, the centralized feeding of large-capacity, high-density ultra-high-voltage (UHVDC) power into the receiving-end grid, along with the large-scale integration of renewable energy sources, and the replacement of conventional generating units with UHVDC and renewable energy, leads to a decrease in system rotational inertia and a significant reduction in overall system regulation capacity, seriously threatening the safe and stable operation of the system. Third, the operating characteristics of DC transmission systems are highly sensitive to voltage, requiring the absorption of a large amount of reactive power during operation, making the dynamic processes of reactive power and voltage in the receiving-end grid more complex. The feeding of multiple DC lines into the receiving-end grid intensifies the interaction between AC and DC systems, complicates the dynamic processes of reactive power and voltage, expands the transmission range of regional power grid faults, and continuously increases the risks to power grid safety and stability, seriously affecting the power transmission security of key transmission corridors. Therefore, there is an urgent need to develop online intelligent control methods to effectively improve the transmission capacity of key corridors. Summary of the Invention

[0003] The purpose of this invention is to solve the problems mentioned in the background art and provide a method for improving the power of a power transmission corridor with embedded DC. By constructing an intelligent agent through reinforcement learning, the active power output of the generator sets on both sides of the power transmission corridor is adjusted online according to real-time power flow information, so as to improve the power transmission power of the power transmission corridor.

[0004] To achieve the objective of this invention, a method for enhancing the power output of a power transmission corridor with embedded DC transmission is disclosed, comprising the following steps:

[0005] Step 1: Collect historical power flow section files and extract power grid status information;

[0006] Step 2: Construct an intelligent agent model for power control of power transmission lines in the power grid, including control space, state space, and reward function, and analyze the samples;

[0007] Step 3: Construct a simulation environment reset process, write the actions output by the agent into a BPA format file for power flow solution, and combine it with the power grid state information acquisition process in Step 1 to form an agent-environment interaction process; then implement an agent training process based on reinforcement learning.

[0008] Step 4: Testing the training results, which involves selecting a test set from historical data, inputting it into the agent, and testing the agent's decision-making performance.

[0009] Furthermore, in step 1, the power grid status information includes the system network topology and generator load node location, the real-time active and reactive power output of each generator in the system, the real-time load of each node in the system, the system line impedance admittance information, and the line information included in the key transmission corridor. The power grid status information is used to constitute the control objectives and constraints.

[0010] Furthermore, the control objective is max P 输电走廊 The constraints are as follows:

[0011]

[0012]

[0013]

[0014]

[0015]

[0016]

[0017] in, and P represents the active and reactive power output of generator n on bus i. ij (y) and Q ij (y) represents the active and reactive power from bus i to bus j, V i The voltage amplitude of bus i represents the value of the bus, B represents the bus set, the superscript g indicates the generator, the superscript d indicates the grid load, and P represents the voltage amplitude of the bus. i g and It refers to the active power injection and reactive power injection of the generator on bus i, P i d and These are the active and reactive power of the load on bus i. and G represents the active and reactive power of the load m on bus i. i D is the set of generators on bus i. i It is the set of loads on bus i, B i It is the set of busbars that form branches with busbar i, g i It is the self-conductance of bus i, b i It is the self-susceptance of bus i; the safety constraint conditions are:

[0018]

[0019]

[0020]

[0021]

[0022]

[0023] in, and This indicates the upper and lower limits of the generator's active power. and G represents the upper and lower limits of reactive power of the generator, and V represents the set of generators. i min and V i max Indicates the upper and lower limits of the bus voltage amplitude. It is the upper limit of the apparent power of the transmission line, Ω L Ω represents the set of transmission lines. T G represents a set of transformers. ij It is the mutual conductance of bus i and bus j, θ i It is the phase angle of the bus voltage i, θ j Indicates the phase angle of bus voltage j, b ij It is the mutual susceptance of bus i and bus j, b ij0 It is the susceptance of the tie-line capacitor, P ij and Q ij V represents the active power and reactive power on line ij, respectively. i and V j These represent the voltage amplitudes of bus i and bus j, respectively.

[0024] Furthermore, in step 2, the agent model for power control of power transmission lines in the power grid is as follows:

[0025] The state space is the power flow state measurement information of the power grid, including bus voltage amplitude, voltage phase angle, line active power, line reactive power, transformer active power, and transformer reactive power;

[0026] The control space refers to the adjustable active power of the generator sets in the area where the power transmission corridor is located.

[0027]

[0028] The reward function is:

[0029]

[0030] Where α, β, and γ are coefficients, ΔP 输电走廊 It is the total increase in active power of the power transmission corridor. It refers to the total over-limit power of transmission lines in cases of power exceeding limits. It is the sum of all cases where the bus voltage exceeds the limit.

[0031] Furthermore, in step 3, the reinforcement learning-based agent uses the improved maximum entropy SAC reinforcement learning algorithm to solve the problem. To avoid overestimation, SAC adopts a double Actor network structure, with one Critic network configured for both the estimation network and the target network, i.e., a total of two Actor networks and two Critic networks. The Actor is responsible for learning the policy, i.e. how to choose actions, while the Critic is responsible for estimating the value function of the current policy.

[0032] Furthermore, the maximum entropy objective function comprises two parts: maximizing the expected reward and maximizing the policy entropy. SAC uses soft Q-learning as the training method for its Critic network, which extends standard Q-learning to a continuous action space and uses soft maximization operations to compute the target Q value. In addition, SAC introduces a parameter to balance the importance of reward and entropy, which is adaptive and can be adjusted according to the progress of training. This parameter controls the contribution of policy entropy, thereby affecting the balance between exploration and exploitation.

[0033] Furthermore, to improve the stability of the algorithm, SAC uses a dual Q-network and a delayed update strategy, with two Critic networks, each independently estimating the Q-value; the Actor's parameters are updated based on the estimates of the two Critic networks to reduce the variance caused by the error of a single Critic network.

[0034] Furthermore, the idea of ​​a truncated proximal optimization strategy algorithm is introduced to limit the control actions of the agent in each iteration within a preset range, thereby achieving the goal of safe learning.

[0035] Compared with existing technologies, the significant advancements of this invention are as follows: Compared with traditional optimization solutions, the use of reinforcement learning to achieve optimization decision-making for this problem avoids the difficulty in solving the problem caused by nonlinear power flow equations, and eliminates the iterative solution steps of power flow calculation and optimization process in the online decision-making process, significantly improving the decision-making speed; The intelligent agent training process designed in this invention, which interacts with the power flow solver, can be extensively pre-trained before the agent is put into actual operation, ensuring good initial decision-making ability, and can also be further optimized according to changes in the power flow characteristics of the system during actual operation.

[0036] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0038] Figure 1 These are schematic diagrams illustrating the changes in transmission power across the line section before and after unit output adjustment in two scenarios.

[0039] Figure 2 This is a schematic diagram of a power enhancement method for a power transmission corridor with embedded DC. Detailed Implementation

[0040] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] like Figure 2 As shown, a method for increasing the power of a transmission corridor with embedded DC includes the following steps:

[0042] Step 1: Collect historical power flow section files and extract power grid status information;

[0043] Step 2: Construct an intelligent agent model for power control of power transmission lines in the power grid, including control space, state space, and reward function, and analyze the samples;

[0044] Step 3: Construct a simulation environment reset process, write the actions output by the agent into a BPA format file for power flow solution, and combine it with the power grid state information acquisition process in Step 1 to form an agent-environment interaction process; then implement an agent training process based on reinforcement learning.

[0045] Step 4: Testing the training results, which involves selecting a test set from historical data, inputting it into the agent, and testing the agent's decision-making performance.

[0046] The specific steps are explained below:

[0047] 1) Step 1

[0048] Step 1 first requires the collection of historical power flow profile files, i.e., collecting historical power flow files generated by the power grid operation in BPA format. Further, key characteristics of the power grid operation are extracted from these files. These key characteristics specifically include the system network topology and generator load node locations, real-time active and reactive power output of each generator within the system, real-time load of each node in the system, system line impedance admittance information, and line information included in key transmission corridors. These characteristics are then used to construct the following control objectives and constraints:

[0049] Control objective: maxP 输电走廊

[0050] Constraints:

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] in, and P represents the active and reactive power output of generator n on bus i. ij (y) and Q ij (y) represents the active and reactive power from bus i to bus j, V i The voltage amplitude of bus i represents the value of the bus, B represents the bus set, the superscript g indicates the generator, the superscript d indicates the grid load, and P represents the voltage amplitude of the bus. i g and It refers to the active power injection and reactive power injection of the generator on bus i, P i d and These are the active and reactive power of the load on bus i. and G represents the active and reactive power of the load m on bus i. i D is the set of generators on bus i. i It is the set of loads on bus i, B i It is the set of busbars that form branches with busbar i, g i It is the self-conductance of bus i, b i It is the self-susceptance of bus i. The constraint condition of the safety constraint is:

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] in, and This indicates the upper and lower limits of the generator's active power. and G represents the upper and lower limits of reactive power of the generator, and V represents the set of generators. i min and V i max Indicates the upper and lower limits of the bus voltage amplitude. It is the upper limit of the apparent power of the transmission line, Ω L Ω represents the set of transmission lines. T G represents a set of transformers. ij It is the mutual conductance of bus i and bus j, θ i It is the phase angle of the bus voltage i, θ j Indicates the phase angle of bus voltage j, b ij It is the mutual susceptance of bus i and bus j, b ij0 It is the susceptance of the tie-line capacitor, P ij and Q ij V represents the active power and reactive power on line ij, respectively. i and V j These represent the voltage amplitudes of bus i and bus j, respectively.

[0064] 2) Step 2

[0065] Step 2 involves constructing a reinforcement learning model for improving power at critical cross sections. This includes a state space, control space, and reward function. The optimization problem described in Step 1 is transformed into the input and output of the agent during the training decision-making process, supporting subsequent actual training. The specific model is as follows:

[0066] State space: Power flow state measurement information of the power grid, including bus voltage amplitude, voltage phase angle, line active power, line reactive power, transformer active power, and transformer reactive power.

[0067] Control space: The active power of the generator sets can be adjusted in the area where the power transmission corridor is located.

[0068]

[0069] Reward function design:

[0070]

[0071] Where α, β, and γ are coefficients, ΔP 输电走廊 It is the total increase in active power of the power transmission corridor. It refers to the total over-limit power of transmission lines in cases of power exceeding limits. It is the sum of all cases where the bus voltage exceeds the limit.

[0072] 3) Step 3

[0073] Step 3 is the implementation of the actual reinforcement learning training process. First, it is necessary to construct a simulation environment reset process, that is, to write the actions output by the agent into a BPA format file used for power flow solution, which together with the key feature acquisition process in step 1 constitutes the agent-environment interaction process.

[0074] In terms of the actual reinforcement learning training process, this invention uses an improved maximum entropy (Soft Actor-Critic, SAC) reinforcement learning algorithm. Maximum entropy is a reinforcement learning algorithm that combines the ideas of maximum entropy policy optimization and soft Q-learning. This algorithm aims to simultaneously achieve efficient exploration, stable convergence, and adaptability in a continuous action space. SAC is an algorithm based on an Actor-Critic structure, which includes one Actor network and at least one Critic network. In this invention, to avoid overestimation problems, a double Actor network structure is adopted, with one Critic network configured for both the estimation network and the target network, thus containing a total of two Actor networks and two Critic networks. The Actor is responsible for learning the policy, i.e., how to choose actions, while the Critic is responsible for estimating the value function of the current policy. SAC is based on maximum entropy theory and attempts to maximize the uncertainty of the system. Compared with traditional deterministic policies, maximum entropy policies tend to explore unknown domains, thereby improving the robustness and generalization of learning. The maximum entropy objective function consists of two parts: maximizing the expected reward and maximizing the entropy of the policy. SAC uses soft Q-learning as its training method for its Critic network. This method extends standard Q-learning to a continuous action space and uses soft maximization to compute the target Q-value. This means it no longer requires selecting the action that maximizes the Q-value in every state, but instead considers a weighted average of all actions. Furthermore, SAC introduces a parameter to balance the importance of reward and entropy. This parameter is adaptive and can be adjusted according to the progress of training. This parameter controls the contribution of policy entropy, thus affecting the balance between exploration and exploitation. To improve the stability of the algorithm, SAC uses a dual Q-network and a delayed update strategy. This means there are two Critic networks, each independently estimating the Q-value. The Actor's parameters are updated based on the estimates of the two Critic networks to reduce variance caused by errors from a single Critic network. This invention also incorporates the idea of ​​a truncated proximal optimization policy algorithm, restricting the agent's control actions within a range for each iteration, thereby achieving safe learning. This improved maximum entropy algorithm achieves efficient exploration and stable convergence by combining a maximum entropy policy and soft Q-learning, and employing adaptive parameters and a dual Q-network. This makes it perform well in various reinforcement learning tasks with continuous action spaces.

[0075] 4) Step 4

[0076] Step 4 is to test the training results, which involves selecting a test set from historical data, inputting it into the agent, and testing the agent's decision-making performance. The following example illustrates the decision-making performance:

[0077] To verify the effectiveness of the deep reinforcement learning-based power enhancement control method for transmission corridors, a real-world model of the regional power grid was selected for analysis. A key north-south transmission corridor was chosen as the test object, comprising 12 AC lines and 3 DC lines. The basic control logic is as follows: the generator units in the northern region are selected to increase their power output, while the generator units in the southern region are selected to decrease their output, thereby gradually increasing the active power of the transmission corridor. Simultaneously, a system balancing node is selected outside the region.

[0078] The training process for a reinforcement learning agent is as follows:

[0079] Timestep 1 Line P increase 340.000 Line P increase%: 4.088

[0080] Timestep 2 Line P increase 626.500 Line P increase%: 7.533

[0081] Timestep 3Line P increase 982.300 Line P increase%: 11.811

[0082] Timestep 4 Line P increase 1341.900 Line P increase%: 16.135

[0083] Timestep 5 Line P increase 1747.400 Line P increase%: 21.010

[0084] Timestep 6 Line P increase 2130.700 Line P increase%: 25.619

[0085] Timestep 7 Failed! {'fail': "The trend doesn't stop"}

[0086] Log saved!

[0087] Timestep 1 Line P increase 401.600 Line P increase%: 4.829

[0088] Timestep 2 Line P increase 781.200 Line P increase%: 9.393

[0089] Timestep 3 Line P increase 1162.400 Line P increase%:13.976

[0090] Timestep 4 Line P increase 1600.600 Line P increase%:19.245

[0091] Timestep 5 Failed!{'fail': '潮流不收敛'}

[0092] Log saved!!

[0093] Timestep 1 Line P increase 475.000 Line P increase%:5.711

[0094] Timestep 2 Line P increase 963.100 Line P increase%:11.580

[0095] Timestep 3 Line P increase 1462.600 Line P increase%:17.586

[0096] Timestep 4 Line P increase 1952.700 Line P increase%:23.479

[0097] Timestep 5 Line P increase 2404.400 Line P increase%:28.910

[0098] Timestep 6 Line P increase 2870.900 Line P increase%:34.519

[0099] Timestep 7 Line P increase 3326.500 Line P increase%:39.997

[0100] Timestep 8 Line P increase 3793.400 Line P increase%:45.611

[0101] Timestep 9 Line P increase 4328.600 Line P increase%: 52.046

[0102] Timestep 10 Line P increase 4844.500 Line P increase%: 58.249

[0103] Timestep 11 Failed! {'fail': 'The trend continues unabated'}

[0104] Log saved!

[0105] Timestep 1 Line P increase 462.100 Line P increase%: 5.556

[0106] Timestep 2 Line P increase 944.900 Line P increase%: 11.361

[0107] Timestep 3 Line P increase 1646.600 Line P increase%: 19.798

[0108] Timestep 4 Line P increase 2252.300 Line P increase%: 27.081

[0109] Timestep 5 Line P increase 2916.500 Line P increase%: 35.067

[0110] The control agent was trained under two operating conditions: DC outage and full load in the embedded LCC (Liquid Cylinder Class) system. Constraints included generator output constraints, thermal stability constraints of the North-to-South power transmission line section, and power flow convergence constraints. The agents obtained under the two different scenarios were tested under the same load and generator output baseline. The changes in transmission power of each AC line on the North-to-South power transmission line section before and after output adjustment were shown in the figure. Figure 1 As shown in the figure. The results show that, by adjusting the unit output, regardless of whether there is embedded DC, the transmission power of each AC line on the North-to-South power transmission section has been significantly improved, and the improvement is particularly significant for lines 5 and 6, which had lower initial power.

[0111] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0112] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A power ramping method for power transmission corridors with embedded DC, characterized by, The method comprises the following steps: Step 1, collecting historical power flow section files and extracting power grid state information; Step 2, constructing an agent model for power grid transmission line power control, including a control space, a state space, a reward function, and analyzing samples; Step 3, constructing a simulation environment reset process, writing the action output by the agent into a BPA format file used for power flow solution, and combining the power grid state information collection process in step 1 to form an agent-environment interaction process; Then, implementing an agent training process based on reinforcement learning; Step 4, testing the training results, that is, selecting a test set from historical data and inputting it into the agent to test the decision-making effect of the agent; In step 1, the power grid state information includes system network topology, generator load node position, real-time active and reactive power output of each generator in the system, real-time load of each node in the system, line impedance admittance information of the system, and line information contained in the key transmission corridor. The power grid state information is used to form control objectives and constraint conditions; The control target is maxP 输电走廊 The constraint condition is as follows: where and P ij (y) and Q ij (y) represent the active and reactive power from bus i to bus j, V i represent the voltage magnitude of bus i, B represents the bus set, the superscript g represents the generator, and the superscript d represents the grid load, P i g and P i d and P and P i is the generator set on bus i, D i is the load set on bus i, B i is the bus set that forms a branch with bus i, g i is the self conductance of bus i, b i is the self susceptance of bus i; the constraint condition of the security constraint is: where, and represent the upper and lower limits of the generator active power, and represent the upper and lower limits of the generator reactive power, G represents the set of generators, is the upper limit of the apparent power of the transmission line, Ω L represents the set of transmission lines, Ω T represents the set of transformers, g ij is the mutual conductance of bus i and bus j, θ i is the phase angle of the voltage of bus i, θ j represents the phase angle of the voltage of bus j, b ij is the mutual susceptance of bus i and bus j, b ij0 is the susceptance of the tie-line capacitor, P ij and Q ij represent the active power and the reactive power on the line ij, respectively, V i and V j represent the voltage amplitudes of bus i and bus j, respectively; In step 2, the agent model for power grid transmission line power control is as follows: The state space is a vector composed of power grid power flow state quantity measurement information, including bus voltage amplitude, voltage phase angle, line active power, line reactive power, transformer active power, and transformer reactive power; The control space is the adjustable generator active power in the region where the transmission corridor is located: The reward function is Wherein, α, β, γ are coefficients, ΔP 输电走廊 is the total amount of active power transmission corridor upgrade, is the total amount of active power transmission line over-limit in the case of over-limit, is the total amount of bus voltage over-limit.

2. The power boost method for power transmission corridors with embedded DC according to claim 1, characterized in that, In step 3, the agent based on reinforcement learning is solved using an improved maximum entropy SAC reinforcement learning algorithm. The agent includes two Actor networks and two Critic networks. The Actor is responsible for learning the strategy, that is, how to select the action, and the Critic is responsible for estimating the value function of the current strategy.

3. The power boost method for power transmission corridors with embedded DC according to claim 2, characterized in that, The maximum entropy objective function includes two parts: maximizing the expected reward and maximizing the entropy of the strategy; SAC uses soft Q-learning as the training method for its Critic network. This method extends the standard Q-learning to continuous action space and uses a soft maximization operation to calculate the target Q value. In addition, SAC introduces a parameter to balance the importance of reward and entropy. The parameter is adaptive and can be adjusted according to the progress of training; The parameter controls the contribution of policy entropy, thereby affecting the balance between exploration and utilization.

4. The power boost method for power transmission corridors with embedded DC according to claim 2, characterized in that, To improve the stability of the algorithm, SAC uses a double Q network and a delayed update strategy. Two Critic networks independently estimate the Q value. The parameters of the Actor are updated according to the estimates of the two Critic networks to reduce the variance caused by the error of a single Critic network.

5. The power boost method for power transmission corridor with embedded DC according to claim 2, characterized in that, The idea of introducing a truncated proximal optimization strategy algorithm limits the control action of each iteration of the agent to a predetermined range, thereby achieving the purpose of safe learning.

Images