Reactive voltage regulation and control method and system for power system based on partition reinforcement learning

Abstract

The invention discloses an intelligent regulation and control method and system for reactive voltage of a power system based on partition reinforcement learning. The method comprises the following steps: automatically dividing control sub-partitions based on power grid topology analysis, and constructing a partition feature project; designing a partition collaborative reinforcement learning model, and realizing collaborative decision through a global agent framework; and a high-precision simulation interaction environment is constructed. The system can realize adaptive optimization control of the reactive voltage of the power system, effectively improve the voltage stability and reduce the network loss. The method solves the problem that a traditional optimal power flow method is difficult to calculate on large-scale node topology or consumes too long time, has high robustness and good expandability, and is suitable for a modern power system with high-proportion new energy access.

Description

Technical Field

[0001] This invention belongs to the field of power system automation and voltage control technology, specifically relating to a power system reactive power and voltage regulation method and system based on partitioned reinforcement learning. Background Technology

[0002] Reactive power balance and voltage stability in power systems are core aspects of ensuring the safe, high-quality, and economical operation of the power grid. With the continuous expansion of the power grid and the rapid increase in the penetration rate of fluctuating and intermittent distributed energy sources such as wind power and photovoltaics, the power flow distribution of the power grid is becoming increasingly complex, and voltage stability issues are becoming more and more prominent.

[0003] Currently, the reactive power voltage control methods widely used in engineering have the following limitations: Centralized OPF (Optimal Power Flow Function) based on mathematical models, while theoretically able to obtain the global optimal solution, suffers from excessive computation time when dealing with large-scale power grids containing thousands or even tens of thousands of nodes, failing to meet the speed requirements of real-time control. Furthermore, the accuracy of OPF solutions heavily depends on the accuracy of the power grid model, while errors in model parameters and changes in equipment states are unavoidable in actual systems, potentially leading to suboptimal results or even safety issues in practical applications. To reduce the problem's complexity, existing technologies have proposed zoned voltage control. These methods typically partition the grid based on sensitivity analysis or simple topology rules. However, these methods often suffer from static partitioning rules that cannot adapt to changes in power grid operation, potentially causing the partitioning results to fail under specific operating conditions. After partitioning, each zone typically uses a local controller for independent control, lacking an effective global coordination mechanism. Moreover, the selection of partition boundaries, dominant nodes, and controller parameter tuning largely depend on the experience of the operators.

[0004] In recent years, some studies have attempted to directly apply reinforcement learning to power grid control. However, without any dimensionality reduction, directly using the voltage of all nodes in the entire network and the reactive power output of all generators as the observation and action space of the agent will result in an extremely large dimensionality, leading to difficulties in agent training, slow convergence, and poor interpretability and reliability of the strategy.

[0005] Therefore, there is an urgent need in this field for a novel solution that combines the simplification of large system decomposition with intelligent algorithms that have global learning capabilities, and that is efficient, adaptive and robust. Summary of the Invention

[0006] Purpose of the invention: To address the problems existing in the prior art, this invention proposes a reactive power and voltage regulation method and system for power systems based on partitioned reinforcement learning, which is used to realize adaptive optimization regulation of reactive power and voltage in large-scale power systems, especially those with a high proportion of renewable energy connected to the grid.

[0007] Technical solution:

[0008] This invention proposes a reactive power and voltage regulation method for power systems based on partitioned reinforcement learning, comprising:

[0009] Based on the network topology data of the power system, several reactive voltage control sub-partitions are divided.

[0010] A feature vector is established for each control sub-region, including voltage level characteristics, reactive power resource characteristics, load level characteristics, and renewable energy output characteristics; the feature vectors are standardized, and the feature vectors of all control sub-regions are concatenated into a global state vector;

[0011] Construct a simulation environment that integrates a power flow calculation program, and set up an interface for receiving raw actions, a security protection mechanism for verifying raw actions, state feedback rules, and a reward function;

[0012] Based on the global state vector and the simulation environment, a centralized training-distributed execution training framework is established, and an agent is trained using samples generated from power flow sections in typical scenarios.

[0013] The trained agent is deployed to the actual power system, receives the feature vectors of each control sub-region, and outputs reactive power and voltage regulation commands.

[0014] Furthermore, the network topology data of the power system is derived from the power grid flow profile file, including: the connection relationship and static parameters of power grid components, and the dynamic operating status of the power grid in this profile. The operating status includes the active / reactive power output of each generator, the active / reactive power demand of each load node, the reactive power injection / absorption of each reactive power compensation device, and the voltage amplitude and phase angle of each bus.

[0015] Furthermore, the division into several reactive voltage control sub-regions includes:

[0016] Select the root node voltage level, traverse the buses of the root node voltage level of the entire network. If two or more buses can be connected through a power grid with a voltage level lower than that root node, divide all buses that can be connected through a power grid with a voltage level lower than that root node and their subordinate buses into the same initial sub-zone. If buses belonging to the same substation are divided into different initial sub-zones, merge all initial sub-zones containing buses of the same substation to obtain the control sub-zone.

[0017] Furthermore, the voltage level characteristics include the highest voltage value, the lowest voltage value, the average voltage value, the voltage of key hub nodes, the total number of buses in the current control sub-zone whose voltage exceeds the upper limit, and the total number of buses in the current control sub-zone whose voltage exceeds the lower limit.

[0018] The reactive power resource characteristics include the reactive power output of synchronous generators, the reactive power output of static var compensators and static synchronous compensators, the switching status and reactive power capacity of parallel capacitor banks and reactors;

[0019] The load level characteristics include the sum of active power and the sum of reactive power of all loads in the current control sub-zone;

[0020] The new energy output characteristics include the total active power output of all wind farms and photovoltaic power stations within the current control sub-region.

[0021] Furthermore, the security protection mechanism includes:

[0022] Receive original action Check if the action value exceeds its physical capability upper and lower limits. If it exceeds the limit, clamp it to the nearest limit and submit the original action through the safety protection mechanism to the power flow calculation program.

[0023] Furthermore, the state feedback rules include:

[0024] Regarding the current state and agent actions The power flow calculation program is invoked to simulate the physical response of the power grid and output the next state. ,state It includes the current reactive power output, maximum reactive power output, minimum reactive power output, number of zones with voltage exceeding the upper limit, number of zones with voltage exceeding the lower limit, maximum value of zone voltage exceeding the limit, number of zones with generator reactive power exceeding the upper limit, number of zones with generator reactive power exceeding the lower limit, and the value of zones with generator exceeding the limit.

[0025] Furthermore, the reward function includes:

[0026] (1) Voltage over-limit scoring: A piecewise linear interpolation method is used to assign corresponding scores to different voltage levels. The closer the voltage is to the preset threshold, the higher the reward; the further it deviates, the greater the penalty. Additional penalties are given for voltages that exceed the safety boundary.

[0027] (2) Generator reactive power limit score: The score is based on the relative proportion of generator reactive power output. When the reactive power output is in the middle range of the upper and lower limits, the reward is higher, and when it is close to or exceeds the limit, the penalty is imposed.

[0028] (3) Positive incentive: When all bus voltages in the zone are completely within the safe range and there is no voltage over-limit, a high positive incentive is given; when the number of over-limits decreases compared to the previous state, an improvement reward is given; when the reactive power output of all generators is within the normal range, an additional reward is given.

[0029] (4) Negative constraint: Consider the number of equipment operations, impose penalties on frequent equipment operations to prevent oscillations, and impose severe penalties when the power flow calculation does not converge.

[0030] Furthermore, the intelligent agent is a computational model comprising a policy network and a value network, and the Proximal Policy Optimization (PPO) algorithm is selected as the core training algorithm. The centralized training-distributed execution framework includes:

[0031] (1) Distributed execution: K independent replicas of the simulation environment are deployed simultaneously and run in parallel on multiple computing nodes or multiple CPU cores;

[0032] (2) Centralized training: Designate a central training node as the parameter update hub to maintain and update the agent containing the policy network and the value network;

[0033] The value network in the central training node updates the policy network by minimizing the error of the value function calculated by generalized advantage estimation. The policy network updates the policy by maximizing an objective function that includes pruning terms.

[0034] Furthermore, the training process includes:

[0035] (1) Initialize the policy network and value network parameters of the central training node and configure the parallel environment instance;

[0036] (2) Parallel environments run synchronously. Each environment interacts with the current strategy to generate experience data, which is then aggregated in the central buffer. The experience data is ( ),in The state observations of the simulation environment at time t. For the agent in state The original actions generated below, The reward value is calculated based on the reward function. To perform the action The state observations of the post-simulation environment at time t+1;

[0037] (3) Based on the accumulated experience data in the central buffer, the central training node performs the PPO update step and synchronizes the updated network parameters to each environment;

[0038] (4) Repeat steps (2) to (3) until the agent converges.

[0039] This invention also proposes a reactive power and voltage regulation system for power systems based on partitioned reinforcement learning, comprising:

[0040] The partitioning module is used to divide the network topology data of the power system into several reactive voltage control sub-partitions.

[0041] The feature engineering module is used to establish feature vectors for each control sub-region, including voltage level features, reactive power resource features, load level features, and renewable energy output features; the feature vectors are standardized, and the feature vectors of all control sub-regions are concatenated into a global state vector;

[0042] The simulation environment module is used to build a simulation environment that integrates a power flow calculation program, and sets up an interface for receiving raw actions, a security protection mechanism for verifying raw actions, state feedback rules, and a reward function.

[0043] The agent module is used to establish a centralized training-distributed execution training framework based on the global state vector and the simulation environment, train an agent using samples generated from power flow sections in typical scenarios, deploy the trained agent to the actual power system, receive feature vectors from each control sub-region, and output reactive power and voltage regulation commands.

[0044] Beneficial effects:

[0045] This invention partitions the power grid based on topology analysis, decomposing the large-scale global optimization problem into a collaborative decision-making problem for multiple sub-regions. This greatly reduces the observation and action space dimension of the reinforcement learning agent and solves the problem of insufficient computing power in traditional OPF.

[0046] This invention employs a centralized training-distributed execution hybrid architecture. During the training phase, multiple distributed environments collect data in parallel and converge to a central training node. The central training node performs centralized policy optimization and parameter updates based on global information, significantly improving the efficiency of training sample utilization and model convergence speed. Through extensive interaction with the simulation environment, the agent can learn strategies to cope with various complex working conditions, without relying on a precise physical model, and exhibits better adaptability and robustness to system changes.

[0047] The zoning rules based on different voltage levels proposed in this invention conform to the actual structure of the power grid, the zoning feature engineering can be easily obtained from existing SCADA systems, and the integrated safety verification mechanism ensures the safety of the control process, making the solution of this invention easy to implement in practical engineering. Attached Figure Description

[0048] Figure 1 This is a flowchart of the method of the present invention;

[0049] Figure 2 This is a flowchart illustrating step 1, grid partitioning, of the method of the present invention.

[0050] Figure 3 This is a diagram of a centralized training-distributed execution training framework for intelligent agents. Detailed Implementation

[0051] The invention will be further explained below with reference to the accompanying drawings and specific embodiments. This invention proposes a method and system for reactive power and voltage regulation in power systems based on partitioned reinforcement learning. Through topology partitioning and a simulation interactive environment, an intelligent agent capable of outputting voltage regulation commands for each partition is trained to achieve reactive power and voltage regulation in the power system. Specifically, the method of this invention includes the following steps:

[0052] Step 1: Power grid topology analysis and control sub-partitioning.

[0053] This step is fundamental to the method of this invention, and its goal is to intelligently decompose the vast interconnected power grid into several control sub-regions that are electrically closely linked, internally coupled, and weakly coupled to each other. This not only significantly reduces the complexity of subsequent agent learning but also lays the physical foundation for regional cooperative control. Specifically, as... Figure 2 As shown, the topology analysis in this step includes two levels:

[0054] First level: Initial partitioning based on electrical connectivity.

[0055] The power system network topology data originates from a pre-established power grid calculation model data file. This data file contains topology, equipment parameters, and power flow distribution information describing the steady-state operation of the power grid. The aforementioned data file can be a standard power flow data file conforming to the formats of common power system analysis software such as BPA, PSS / E, and PSASP. Specifically, it includes:

[0056] (1) The physical connection topology of the power grid, namely the connection relationship of components such as busbars, lines, transformers, and switches and their static parameters (such as resistance, reactance, turns ratio, rated capacity, etc.).

[0057] (2) The dynamic operating status of the power grid at this section mainly includes: the active and reactive power output of each generator, the active and reactive power demand of each load node, the reactive power injection / absorption of each reactive power compensation device (such as capacitors and reactors), and the voltage amplitude and phase angle of each bus (or can be calculated by power flow).

[0058] The first-level topology analysis includes the following process:

[0059] (1) Root Node Selection and Isolation: According to the preset voltage level priority rule, one voltage level is selected from the multiple voltage levels included in the power grid topology as the root node voltage level; wherein, the voltage level priority rule is configured to prioritize the selection of higher voltage levels, and in this embodiment, the root node voltage level is set to 500kV. Traverse the entire network and select all busbars of the root node voltage level (500 kV) as the initial root nodes, and temporarily disconnect all tie lines of 500 kV and above voltage levels connecting different 500 kV busbars logically. The purpose of this is to "island out" the ultra-high voltage main grid in the initial stage to obtain its lower-level electrical connection structure.

[0060] (2) Depth-first search (DFS) or breadth-first search (BFS): Starting from each isolated 500 kV root node, a graph traversal search is performed along the lines and transformers of the 230 kV, 110 kV, and other voltage levels below it. All buses that can be reached during the search process (regardless of voltage level) are assigned to the initial sub-region to which the root node belongs, ensuring the integrity of electrical connections within the region.

[0061] (3) Coupling identification and merging: During the traversal, if two or more different 500 kV root nodes are found to be electrically connected to each other through the connection of the lower-level power grid (such as a 230 kV network), it indicates that these 500 kV nodes are electrically tightly coupled and should not be separated. At this time, the system will automatically merge these 500 kV root nodes and all their subordinate buses into the same initial sub-partition.

[0062] For example, if it is found that two 500 kV busbars, A and B, are connected through a 230 kV power grid, then A, B, and all their subordinate 220 kV, 110 kV, and other lower-level busbars are classified into the same initial sub-region.

[0063] Second level: Secondary merger based on substation ownership.

[0064] The initial zoning at the first level may be too fragmented, especially when a 500 kV substation has multiple 500 kV busbars, which may be divided into different zones in the initial stage. From the perspective of actual power grid operation and management, equipment belonging to the same substation usually has uniformity in control.

[0065] The merging rules for the second level are as follows: Traverse all initial sub-partitions and identify all 500kV buses belonging to the same 500kV substation. Regardless of whether these buses were assigned to the same initial sub-partition in the first level, the system merges all their respective initial sub-partitions to form a single control sub-partition. The core rules for this stage are as follows:

[0066] (1) Construct an initial sub-partition mapping table to determine the affiliation of the 500 kV substations it contains.

[0067] (2) Traverse all initial sub-partition pairs (i, j). If two sub-partitions are mapped to the same 500 kV substation, mark them as to be merged.

[0068] (3) The Union-Find data structure is used to efficiently handle the merging logic: First, the Union-Find is initialized, with each partition being its own root node. Then, a double loop structure is used to traverse all possible partition pairs (i, j). For each pair of partitions, if they meet the above-mentioned substation affiliation condition, the union(i, j) operation is performed to merge the two into the same logical set in the Union-Find.

[0069] (4) Traverse all initial sub-partitions and query the root partition number of each partition using a disjoint-set data structure. Merge all bus data belonging to the same root partition to form a final control sub-partition.

[0070] In this embodiment, a merging operation is performed on the partitions containing busbars belonging to the same 500kV substation. For example, if partition A and partition C contain different busbars of the same "XX500kV substation", then partition A and partition C are merged. Through the above two-stage partitioning process, the provincial power grid is finally divided into 67 control sub-partitions, each with tight electrical coupling within its partition and clear connections between partitions.

[0071] Step 2: Partition feature engineering construction.

[0072] Feature engineering serves as a bridge connecting the physical power grid with artificial intelligence models. Each control sub-region is treated as a virtual device, and a sub-region feature engineering is constructed to characterize its operating status. The sub-region feature engineering includes one or more of the following features: voltage statistics of key nodes within the sub-region, total reactive power output of all reactive power sources within the sub-region, total active load of the sub-region, total reactive load of the sub-region, and total output of renewable energy generation within the sub-region.

[0073] This step constructs a feature vector for each control sub-partition that comprehensively and accurately represents its current operating state. The feature selection principle is that the selected features should be representative, observable, computable, and have low redundancy. For any control sub-partition i, its feature vector includes:

[0074] (1) Voltage level characteristics: Voltage amplitude of all 500 kV and key 230 kV busbars in the zone. Including: highest voltage value, lowest voltage value, average value, voltage of key hub nodes, total number of busbars in the zone that exceed the upper limit of voltage, and total number of busbars in the zone that exceed the lower limit of voltage.

[0075] (2) Reactive power resource characteristics: The sum and distribution of reactive power output of all controllable reactive power sources within the zone. This includes: the reactive power output of synchronous generators ( The reactive power output, switching status and reactive power capacity of static var compensators (SVC) and static synchronous compensators (STATCOM), as well as parallel capacitor banks and reactors.

[0076] (3) Load level characteristics: The sum of the active power of all loads within the zone ( ) and the sum of reactive power ( ).

[0077] (4) Characteristics of new energy output: The total active power output of all wind farms and photovoltaic power stations within the zone ( ) and its predicted value.

[0078] (5) Network structure characteristics: active and reactive power flow on the connection lines between partitions and adjacent partitions, used to sense the mutual influence between partitions.

[0079] Before proceeding to the next step, all features from step 2 are standardized (e.g., Z-score standardization or Min-Max normalization), and the feature vectors of each control sub-partition are concatenated into a global state vector to accelerate model training convergence in the subsequent steps and improve numerical stability.

[0080] In this embodiment, feature vectors are constructed for each of the 67 sub-regions. The feature engineering design fully considers the core requirements of voltage safety: the feature vector of each region mainly represents the voltage limit exceedance situation of that region, specifically defined as follows: statistically analyzing the voltage limit exceedance situation of all buses within the region; if 3 buses exceed the upper limit, the feature value is set to 3; if 2 buses exceed the lower limit, the feature value is set to -2; if there are no exceedances, the feature value is 0; if both upper and lower limit exceedances exist simultaneously, the feature value is set to 100000 as a special identifier. Finally, the feature vectors of the 67 regions are concatenated to form the global state vector. , as the observation input of the intelligent agent.

[0081] Step 3: Simulation environment construction.

[0082] A highly efficient simulation environment is crucial for training high-performance reinforcement learning agents. The simulation environment constructed in this invention is a power flow calculation software system, whose core components include a power flow calculation module, a state transition module, a reward function module, and a security verification module.

[0083] (1) Power flow calculation module, which adopts a self-developed high-performance DPF (power flow calculation) program or integrates commercial software (such as PSS®E, PSD-BPA, PSD-SCCPC, etc.) power flow calculation kernel. This module receives action commands from the intelligent agent (such as changing the generator terminal voltage or the switching state of the capacitor and reactor) and calculates the new system state.

[0084] (2) State transition module, based on the current state and agent actions By calling the power flow calculation module, the physical response of the power grid is simulated, and the next state is output. .state It includes the current reactive power output, maximum reactive power output, minimum reactive power output, number of zones exceeding the upper voltage limit, number of zones exceeding the lower voltage limit, maximum value of zone voltage exceeding the limit, number of zones exceeding the upper reactive power limit, number of zones exceeding the lower reactive power limit, and the limit value of zones.

[0085] (3) The reward function module is designed with a multi-level, fine-grained reward function, built around the two core objectives of voltage safety and equipment operation safety, specifically including:

[0086] (3.1) Voltage over-limit scoring: A piecewise linear interpolation method is used to assign corresponding scores to different voltage levels. The closer the voltage is to 1.0pu, the higher the reward. The further the voltage deviates, the greater the penalty. Special attention is paid to serious over-limit situations, and additional penalties are given for voltages that exceed the safety boundary.

[0087] (3.2) Generator reactive power limit exceedance score: The score is based on the relative proportion of generator reactive power output. When the reactive power output is in the middle range of the upper and lower limits, the reward is higher, and when it is close to or exceeds the limit, the penalty is imposed. The number of generator reactive power limit exceedances is counted for subsequent reward calculation.

[0088] (3.3) Positive incentive: When all bus voltages in the zone are completely within the safe range and there is no voltage over-limit, a high positive incentive is given; when the number of over-limits decreases compared to the previous state, an improvement reward is given; when the reactive power output of all generators is within the normal range, an additional reward is given.

[0089] (3.4) Negative constraint: Consider the number of equipment operations, impose penalties on frequent equipment actions to prevent action oscillations, and impose severe penalties when the power flow calculation does not converge.

[0090] (4) Security verification module, which verifies the original actions of the intelligent agent. Before being submitted to the power flow engine, this module performs verification, including checking the node voltage upper and lower limits, generator reactive power output limits, and transformer turns ratio adjustment range. It checks whether action values ​​(such as generator reactive power setpoints) exceed their physical capacity limits. If limits are exceeded, they are clamped to the nearest available limit.

[0091] This embodiment, based on the researched power grid model, integrates power flow simulation software into a Python environment to construct a simulation environment for reinforcement learning training. Specific configurations include:

[0092] Action Interface: Receives control actions output by the agent, including generator terminal voltage setpoint adjustment and capacitor switching commands; State Feedback: After performing power flow calculation, returns the new system state (feature vectors of each partition) and the comprehensive reward value; Reward Function Design: Adopts a multi-objective optimization reward mechanism; Safety Verification Mechanism: The environment has built-in multi-layer safety protection, automatically performs boundary verification on the agent's actions, and corrects the generator reactive power exceeding the limit to a safe range in real time if such situations are found.

[0093] Step 4: Agent training.

[0094] In this invention, an "intelligent agent" refers to a trainable computational model composed of a policy network (Actor) and a value network (Critic), which learns and optimizes decisions through interaction with the environment. Its core function is to output coordinated reactive power and voltage regulation commands for each control sub-region based on the input global power grid state vector, and through continuous trial and error optimization, ultimately learn a stable control strategy that can adapt to complex scenarios.

[0095] A centralized training-distributed execution framework is adopted to effectively improve data acquisition efficiency and model convergence speed. Its core is to achieve decoupling and coordination between simulation interaction and policy optimization through a data-parallel architecture, as detailed below:

[0096] (1) Training framework design:

[0097] Distributed environment instance: K independent replicas of the simulation environment are deployed simultaneously and run in parallel on multiple computing nodes or CPU cores to achieve large-scale, diverse synchronous acquisition of interactive data.

[0098] Central training node: Serves as the parameter update hub, maintaining and updating the global policy model, which includes the policy network (Actor) and the value network (Critic).

[0099] (2) Network structure and optimization mechanism:

[0100] The dual-network collaboration within the central training node achieves efficient and stable strategy optimization through the following mechanisms:

[0101] Critic: Based on generalized advantage estimation (GAE), it calculates time-series difference error and learns to accurately estimate the long-term returns of the system state by minimizing the value function loss, providing a stable and reliable value benchmark for policy optimization.

[0102] Policy Network (Actor): Following the constrained optimization mechanism of the PPO algorithm, it updates the policy by maximizing the objective function that includes the probability ratio pruning term. While promoting the improvement of policy performance, it limits the difference between the old and new policies and ensures the stability and controllability of the training process.

[0103] Collaborative optimization mechanism: The accurate state value estimation provided by the value network provides clear directional guidance for the gradient update of the policy network. The two form a closed-loop feedback during training, jointly driving the policy to converge toward the global optimal solution.

[0104] (3) Asynchronous data acquisition and centralized update process:

[0105] Asynchronous data acquisition: Each environment instance independently collects experience data tuples. ),in This represents the current state observation of the environment at time t (i.e., the power grid partition feature vector). To perform the action Then, the next state observation of the environment at time t+1; For the agent in state The original actions generated and verified by the system (i.e., the control command set for equipment such as generators and capacitors); To perform the action The immediate reward value after environmental feedback (determined by the reward function module based on (...) Calculate a scalar reward value This mechanism significantly increases the amount and diversity of data collected per unit time, which is beneficial for exploring various complex operating conditions of the power grid.

[0106] Centralized gradient update: Experience data collected from all environments is aggregated into a central experience buffer for batch gradient calculation. After the central training node updates the network parameters, the latest parameters are synchronized to all environment replicas, initiating the next round of data collection.

[0107] (4) Training execution process

[0108] ① Initialize the policy network and value network parameters of the central training node and configure the parallel environment instance;

[0109] ② Parallel environments run synchronously, with each environment interacting with the environment based on the current strategy to generate experience data and aggregate it into the central buffer.

[0110] ③ Based on accumulated experience, the central training node executes the PPO update step and synchronizes the updated network parameters to each environment replica;

[0111] ④ Repeat steps ② to ③ until the policy performance converges and stabilizes.

[0112] In this embodiment, 100 simulation environment instances are simultaneously run on a high-performance server, significantly improving training efficiency. The agent's input is a joint state vector composed of 67 sub-partition feature vectors; the output is a coordinated control command for all partitions, using a discrete action space design. The action value for each partition is {-1, 0, 1}, representing a certain percentage (e.g., 10%) of reducing, maintaining, or increasing the total reactive power output of that partition. For a single cross-section file, the output action is a 67-dimensional vector; when processing 100 cross-sections simultaneously, the output is a 100×67 two-dimensional action matrix.

[0113] Large-scale training samples are generated based on power flow sections in various typical scenarios, allowing the agent to interact and learn with the simulation environment for millions of steps. Through continuous trial and error, the agent gradually learns a coordinated control strategy for voltage stability and network loss optimization under various complex scenarios such as wind power fluctuations and load changes.

[0114] Step 5: Deployment and tidal flow section prediction.

[0115] After training, a stable agent is deployed to the actual power system. The agent receives power flow sections to be processed from the system (supporting single-section or multi-section batch processing), generates control strategies for each section through forward inference, and outputs decision instructions for each section to increase, decrease, or maintain reactive power output. Specifically, the process includes the following:

[0116] (1) Based on the initial power flow section, the intelligent agent outputs the first round of control actions to guide the switching of the capacitor bank equipment or the adjustment of the reactive power output of the generator in each zone;

[0117] (2) After executing the control action, the power flow calculation is performed again to generate a new system state; the updated system state is then input into the agent again to obtain the next round of optimization instructions;

[0118] (3) The loop continues until any of the following conditions are met:

[0119] All voltage over-limit issues are completely eliminated; the preset maximum number of running steps (e.g., 20 steps) is reached; the system state tends to stabilize, and further optimization is no longer significant.

[0120] (4) The intelligent agent outputs the final capacitive reactor action or generator terminal voltage adjustment value. This strategy can be directly applied to the automatic voltage control system of the actual power grid or used as a decision support basis for operators.

[0121] This invention achieves intelligent diagnosis and autonomous optimization of grid voltage problems through a closed-loop control mechanism of "sensing-decision-execution-verification," significantly improving the efficiency and reliability of voltage control. Through the specific implementation methods described above, this invention effectively solves the problems of low computational efficiency and poor adaptability of traditional methods in large-scale power grids, providing an intelligent reactive power and voltage control solution for modern power systems with a high proportion of renewable energy.

Claims

1. A reactive power and voltage regulation method for power systems based on partitioned reinforcement learning, characterized in that, include: Based on the network topology data of the power system, several reactive voltage control sub-partitions are divided. A feature vector is established for each control sub-region, including voltage level characteristics, reactive power resource characteristics, load level characteristics, and renewable energy output characteristics; the feature vectors are standardized, and the feature vectors of all control sub-regions are concatenated into a global state vector; Construct a simulation environment that integrates a power flow calculation program, and set up an interface for receiving raw actions, a security protection mechanism for verifying raw actions, state feedback rules, and a reward function; Based on the global state vector and the simulation environment, a centralized training-distributed execution training framework is established, and an agent is trained using samples generated from power flow sections in typical scenarios. The trained agent is deployed to the actual power system, receives the feature vectors of each control sub-region, and outputs reactive power and voltage regulation commands.

2. The power system reactive power and voltage regulation method according to claim 1, characterized in that, The network topology data of the power system comes from the power flow profile file of the power grid, including: the connection relationship and static parameters of the power grid components, and the dynamic operating status of the power grid in this profile. The operating status includes the active / reactive power output of each generator, the active / reactive power demand of each load node, the reactive power injection / absorption of each reactive power compensation device, and the voltage amplitude and phase angle of each bus.

3. The power system reactive power and voltage regulation method according to claim 2, characterized in that, The division into several reactive voltage control sub-regions includes: Select the root node voltage level, traverse the buses of the root node voltage level of the entire network. If two or more buses can be connected through a power grid with a voltage level lower than that root node, divide all buses that can be connected through a power grid with a voltage level lower than that root node and their subordinate buses into the same initial sub-zone. If buses belonging to the same substation are divided into different initial sub-zones, merge all initial sub-zones containing buses of the same substation to obtain the control sub-zone.

4. The power system reactive power and voltage regulation method according to claim 3, characterized in that, The voltage level characteristics include the highest voltage value, the lowest voltage value, the average voltage value, the voltage of key hub nodes, the total number of buses in the current control sub-zone whose voltage exceeds the upper limit, and the total number of buses in the current control sub-zone whose voltage exceeds the lower limit. The reactive power resource characteristics include the reactive power output of synchronous generators, the reactive power output of static var compensators and static synchronous compensators, the switching status and reactive power capacity of parallel capacitor banks and reactors; The load level characteristics include the sum of active power and the sum of reactive power of all loads in the current control sub-zone; The new energy output characteristics include the total active power output of all wind farms and photovoltaic power stations within the current control sub-region.

5. The power system reactive power voltage regulation method according to claim 4, characterized in that, The security protection mechanism includes: Receive original action Check if the action value exceeds its physical capability upper and lower limits. If it exceeds the limit, clamp it to the nearest limit and submit the original action through the safety protection mechanism to the power flow calculation program.

6. The power system reactive power voltage regulation method according to claim 5, characterized in that, The status feedback rules include: Regarding the current state and agent actions The power flow calculation program is invoked to simulate the physical response of the power grid and output the next state. ,state It includes the current reactive power output, maximum reactive power output, minimum reactive power output, number of zones with voltage exceeding the upper limit, number of zones with voltage exceeding the lower limit, maximum value of zone voltage exceeding the limit, number of zones with generator reactive power exceeding the upper limit, number of zones with generator reactive power exceeding the lower limit, and the value of zones with generator exceeding the limit.

7. The power system reactive power voltage regulation method according to claim 6, characterized in that, The reward function includes: (1) Voltage over-limit scoring: A piecewise linear interpolation method is used to assign corresponding scores to different voltage levels. The closer the voltage is to the preset threshold, the higher the reward; the further it deviates, the greater the penalty. Additional penalties are given for voltages that exceed the safety boundary. (2) Generator reactive power limit score: The score is based on the relative proportion of generator reactive power output. When the reactive power output is in the middle range of the upper and lower limits, the reward is higher, and when it is close to or exceeds the limit, the penalty is imposed. (3) Positive incentive: When all bus voltages in the zone are completely within the safe range and there is no voltage over-limit, a high positive incentive is given; when the number of over-limits decreases compared to the previous state, an improvement reward is given; when the reactive power output of all generators is within the normal range, an additional reward is given. (4) Negative constraint: Consider the number of equipment operations, impose penalties on frequent equipment operations to prevent oscillations, and impose severe penalties when the power flow calculation does not converge.

8. The power system reactive power voltage regulation method according to claim 7, characterized in that, The intelligent agent is a computational model comprising a policy network and a value network. The Proximal Policy Optimization (PPO) algorithm is selected as the core training algorithm. The centralized training-distributed execution framework includes: (1) Distributed execution: K independent replicas of the simulation environment are deployed simultaneously and run in parallel on multiple computing nodes; (2) Centralized training: A computing node is designated as the central training node to update the agent containing the policy network and the value network; the value network in the central training node is updated by minimizing the error of the value function calculated by the generalized advantage estimation, and the policy network is updated by maximizing an objective function containing pruning terms.

9. The power system reactive power and voltage regulation method according to claim 8, characterized in that, The training process includes: (1) Initialize the policy network and value network parameters of the central training node and configure the parallel environment instance; (2) Parallel environments run synchronously. Each environment interacts with the current strategy to generate experience data, which is then aggregated in the central buffer. The experience data is ( ),in The state observations of the simulation environment at time t. For the agent in state The original actions generated below, The reward value is calculated based on the reward function. To perform the action The state observations of the post-simulation environment at time t+1; (3) Based on the accumulated experience data in the central buffer, the central training node performs the PPO update step and synchronizes the updated network parameters to each environment; (4) Repeat steps (2) to (3) until the agent converges.

10. A reactive power and voltage regulation system for power systems based on partitioned reinforcement learning, characterized in that, include: The partitioning module is used to divide the network topology data of the power system into several reactive voltage control sub-partitions. The feature engineering module is used to establish feature vectors for each control sub-region, including voltage level features, reactive power resource features, load level features, and renewable energy output features; the feature vectors are standardized, and the feature vectors of all control sub-regions are concatenated into a global state vector; The simulation environment module is used to build a simulation environment that integrates a power flow calculation program, and sets up an interface for receiving raw actions, a security protection mechanism for verifying raw actions, state feedback rules, and a reward function. The agent module is used to establish a centralized training-distributed execution training framework based on the global state vector and the simulation environment, train an agent using samples generated from power flow sections in typical scenarios, deploy the trained agent to the actual power system, receive feature vectors from each control sub-region, and output reactive power and voltage regulation commands.

Images