Power system black-start partition decision-making method based on lightweight reinforcement learning

Abstract

The invention provides a power system black-start partition decision method based on lightweight reinforcement learning, and the method comprises the steps: constructing a black-start model of a power system containing multiple physical constraints, and formalizing the black-start model into a Markov decision process, the Markov decision process comprises a state space, an action space, a reward function and a transfer process; decomposing the power system into a plurality of sub-regions by adopting a dynamic partition recovery strategy; based on the Markov decision process, a depth deterministic strategy gradient algorithm is adopted to design and train a lightweight strategy network of deep reinforcement learning; enabling the intelligent agent to interact with the environment in each sub-region through the lightweight strategy network of deep reinforcement learning, and learning an optimal decision strategy; and according to the optimal decision strategy, performing partition power supply capability recovery on the power system. According to the invention, black-start rapid optimization decision in a complex power grid environment is realized.

Description

Technical Field

[0001] This invention relates to the field of power system emergency recovery technology, and in particular to a black-start partition decision-making method for power systems based on lightweight reinforcement learning. Background Technology

[0002] Black start of the power system is a core component of rebuilding the power supply network through self-starting units after a major power outage. The recovery sequence needs to be optimized under multiple constraints, such as unit start-up and shutdown power, node voltage limits, and network topology.

[0003] With the expansion of power grids and the integration of new energy sources, traditional mathematical programming-based methods are gradually becoming ineffective due to the exponential increase in computational complexity, while heuristic rules struggle to cope with dynamically changing power grid conditions. Currently, the closest existing technologies include: mixed-integer programming, which models black-start as an exact optimization problem, but computation in high-dimensional state spaces can take hours; heuristic rules, which set recovery orders based on empirical principles such as nearest neighbor priority or power priority, cannot handle complex scenarios such as voltage constraints and topological loops; and basic reinforcement learning, such as deep Q-networks (DQN), while approximating value functions through neural networks, does not integrate the radial topological constraints of the distribution network, and the high-dimensional state space leads to an exponential increase in training sample requirements, making convergence difficult in practical engineering applications.

[0004] In summary, the aforementioned existing technologies have the following drawbacks:

[0005] (1) The computational efficiency is low and the convergence speed is slow. Traditional mixed integer programming methods have extremely high computational complexity when dealing with large-scale systems, which cannot meet the real-time requirements of emergency recovery.

[0006] (2) The constraint processing mechanism is crude, and the heuristic rules rely on preset experience. It lacks the ability to adapt when facing dynamic changes in the power grid topology and multiple constraint coupling scenarios. For example, it is difficult to coordinate voltage over-limit and power balance problems at the same time. It often gets stuck in local optima, and the processing of voltage deviation and topology loops lacks systematicity. The decision feasibility is only 75%-85%.

[0007] (3) The state space explosion problem is significant. Existing reinforcement learning schemes have not effectively integrated the radial topology constraints unique to distribution networks. Furthermore, the dimensionality of the state space increases exponentially with the number of nodes, which makes the convergence speed during training extremely slow, and may even require the neural network to exceed 10. 6 It takes several iterations to achieve convergence. Summary of the Invention

[0008] In view of this, embodiments of the present invention provide a black-start partitioning decision-making method for power systems based on lightweight reinforcement learning. The aim is to overcome the bottlenecks of traditional methods in terms of real-time performance, constraint processing, and computational efficiency by using a lightweight reinforcement learning framework and partitioning recovery strategy, and to achieve rapid optimization decision-making for black starts in complex power grid environments.

[0009] According to a first aspect of the present invention, a black-start partitioning decision-making method for a power system based on lightweight reinforcement learning is provided, comprising: constructing a black-start model of a power system containing multiple physical constraints, and formalizing the black-start model as a Markov decision process, wherein the Markov decision process includes a state space, an action space, a reward function, and a transition process; decomposing the power system into multiple sub-regions using a dynamic partitioning recovery strategy; designing and training a lightweight policy network of deep reinforcement learning using a deep deterministic policy gradient algorithm based on the Markov decision process; enabling an agent to interact with the environment in each sub-region and learn the optimal decision strategy through the lightweight policy network of deep reinforcement learning; and performing partitioning recovery of power supply capability of the power system according to the optimal decision strategy.

[0010] Optionally, the objective function of the black-start model is:

[0011]

[0012] Among them, P l,i (t) represents the node load power, δ g (t) represents the energized state of the unit.

[0013] Optionally, the multiple physical constraints include node power balance, line power flow and topology constraints, unit operation constraints, and load input and system stability constraints.

[0014] Optionally, the dynamic partitioning recovery strategy achieves dynamic expansion through a dual criterion of topological neighborhood and power margin, thereby decomposing the power system into multiple sub-regions, specifically including:

[0015] The initial partitioning is centered on the black-boot unit, defining a neighborhood with a radius of R = 1. Subsequent partitioning expansion is based on the total startup power of the area to be restored, and must satisfy the following conditions:

[0016] ∑P CR,g ≤(1-δ)·P base

[0017] Where δ is the safety factor, P base This is the system's reference power.

[0018] The value of the safety factor δ is determined by the inertial time constant H of the power grid, and the calculation formula is as follows:

[0019]

[0020] Among them, T base H is the system reference time, which is obtained by actual measurement through the generator rotor motion equation, and κ is the adjustment factor.

[0021] Traditional units within the zone must retain a spin-off reserve capacity S. U,g and S D,g That is, satisfying:

[0022] |P g (t)-P g (t-1)∣≤S U,g +S D,g

[0023] During partition expansion, the next sub-region is selected based on the power margin priority principle, and the minimum distance between the node to be restored and the already restored region is calculated using the topological distance matrix D.

[0024] d j =min i∈已恢复区 D ij ,

[0025] Among them, D ij Let be the number of edges in the shortest path from node i to j, ensuring that partition expansion proceeds along the topological connectivity direction.

[0026] According to a second aspect of the present invention, a power system black-start partitioning decision-making device based on lightweight reinforcement learning is provided, comprising: a model building module, configured to construct a black-start model of a power system containing multiple physical constraints, and formalize the black-start model as a Markov decision process, wherein the Markov decision process includes a state space, an action space, a reward function, and a transition process; a decomposition module, configured to decompose the power system into multiple sub-regions using a dynamic partitioning recovery strategy; a reinforcement learning module, configured to design and train a lightweight policy network of deep reinforcement learning based on the Markov decision process using a deep deterministic policy gradient algorithm; an interactive learning module, configured to enable an agent to interact with the environment in each sub-region through the lightweight policy network of deep reinforcement learning, and learn the optimal decision-making strategy; and a power supply recovery module, configured to restore the power supply capability of the power system in partitions according to the optimal decision-making strategy.

[0027] Optionally, the objective function of the black-start model is:

[0028]

[0029] Among them, P l,i (t) represents the node load power, δ g (t) represents the energized state of the unit.

[0030] Optionally, the multiple physical constraints include node power balance, line power flow and topology constraints, unit operation constraints, and load input and system stability constraints.

[0031] Optionally, the dynamic partitioning recovery strategy achieves dynamic expansion through a dual criterion of topological neighborhood and power margin, thereby decomposing the power system into multiple sub-regions, specifically including:

[0032] The initial partitioning is centered on the black-boot unit, defining a neighborhood with a radius of R = 1. Subsequent partitioning expansion is based on the total startup power of the area to be restored, and must satisfy the following conditions:

[0033] ∑P CR,g ≤(1-δ)·P base

[0034] Where δ is the safety factor, P base This is the system's reference power.

[0035] The value of the safety factor δ is determined by the inertial time constant H of the power grid, and the calculation formula is as follows:

[0036]

[0037] Among them, T base H is the system reference time, which is obtained by actual measurement through the generator rotor motion equation, and κ is the adjustment factor.

[0038] Traditional units within the zone must retain a spin-off reserve capacity S. U,g and S D,g That is, satisfying:

[0039] |P g (t)-P g (t-1)∣≤S U,g +S D,g

[0040] During partition expansion, the next sub-region is selected based on the power margin priority principle, and the minimum distance between the node to be restored and the already restored region is calculated using the topological distance matrix D.

[0041] d j =min i∈已恢复区 D ij ,

[0042] Among them, D ij Let be the number of edges in the shortest path from node i to j, ensuring that partition expansion proceeds along the topological connectivity direction.

[0043] According to a third aspect of the present invention, an electronic device is provided, including a processor and a memory storing a program. The program includes instructions that, when executed by the processor, cause the processor to perform the steps performed by the method of the first aspect described above.

[0044] According to a fourth aspect of the present invention, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method of the first aspect described above.

[0045] This invention employs a lightweight reinforcement learning framework, particularly the Deep Deterministic Policy Gradient (DDPG) algorithm, to achieve rapid decision-making, meeting the high real-time requirements of the black-start process. Compared to traditional mixed-integer programming methods, this invention significantly reduces computational complexity and improves decision-making speed. Through a dynamic partitioned recovery strategy, this invention decomposes a large-scale power system into multiple sub-regions for successive optimization, effectively reducing the dimensionality of the state space, solving the state space explosion problem in traditional reinforcement learning, and accelerating training convergence. Through the agent's interactive learning with the environment within each sub-region, this invention can learn the optimal decision-making strategy, thereby guiding the power system to restore power supply capacity in different partitions, further improving the efficiency and effectiveness of system recovery. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0047] Figure 1 This is a flowchart illustrating the steps of a black-start partitioning decision-making method for power systems based on lightweight reinforcement learning, as described in this invention.

[0048] Figure 2 To and Figure 1 The overall flowchart of the black-start partitioning decision method for power systems based on lightweight reinforcement learning according to the present invention is shown below. Detailed Implementation

[0049] To provide a clearer understanding of the technical features, objectives, and effects of the embodiments of the present invention, specific implementation methods of the embodiments of the present invention will now be described with reference to the accompanying drawings.

[0050] In this document, “exemplary” means “serving as an example, illustration or description”, and any illustrations or implementations described herein as “exemplary” should not be construed as a more preferred or advantageous technical solution.

[0051] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0052] The specific implementation of the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0053] See Figure 1 This invention provides a black-start partitioning decision-making method for power systems based on lightweight reinforcement learning, comprising:

[0054] Step S1: Construct a black-start model of a power system containing multiple physical constraints, and formalize the black-start model into a Markov decision process, which includes a state space, action space, reward function, and transition process.

[0055] Step S2: Decompose the power system into multiple sub-regions using a dynamic partitioning recovery strategy;

[0056] Step S3: Based on the Markov decision process, a lightweight policy network for deep reinforcement learning is designed and trained using the deep deterministic policy gradient algorithm.

[0057] Step S4: The agent interacts with the environment in each sub-region through the lightweight policy network of deep reinforcement learning to learn the optimal decision-making policy;

[0058] Step S5: Based on the optimal decision-making strategy, restore the power supply capacity of the power system by partition.

[0059] Optionally, the objective function of the black-start model is:

[0060]

[0061] Among them, P l,i (t) represents the node load power, δ g (t) represents the energized state of the unit.

[0062] Optionally, the multiple physical constraints include node power balance, line power flow and topology constraints, unit operation constraints, and load input and system stability constraints.

[0063] Optionally, the dynamic partitioning recovery strategy achieves dynamic expansion through a dual criterion of topological neighborhood and power margin, thereby decomposing the power system into multiple sub-regions, specifically including:

[0064] The initial partitioning is centered on the black-boot unit, defining a neighborhood with a radius of R = 1. Subsequent partitioning expansion is based on the total startup power of the area to be restored, and must satisfy the following conditions:

[0065] ΣP CR,g ≤(1-δ)·P base

[0066] Where δ is the safety factor, P base This is the system's reference power.

[0067] The value of the safety factor δ is determined by the inertial time constant H of the power grid, and the calculation formula is as follows:

[0068]

[0069] Among them, T base H is the system reference time, which is obtained by actual measurement through the generator rotor motion equation, and κ is the adjustment factor.

[0070] Traditional units within the zone must retain a spin-off reserve capacity S. U,g and S D,g That is, satisfying:

[0071] |P g (t)-P g (t-1)∣≤S U,g +S D,g

[0072] During partition expansion, the next sub-region is selected based on the power margin priority principle, and the minimum distance between the node to be restored and the already restored region is calculated using the topological distance matrix D.

[0073] d j =min i∈已恢复区 D ij ,

[0074] Among them, D ij Let be the number of edges in the shortest path from node i to j, ensuring that partition expansion proceeds along the topological connectivity direction.

[0075] Specifically, the solution of the present invention is further described with reference to the following examples:

[0076] To address the problems existing in the prior art, the purpose of this invention is to design a lightweight reinforcement learning framework that improves decision-making speed through state-action space dimensionality reduction; to construct a constraint direct integration mechanism to ensure that decisions simultaneously satisfy multiple constraints such as topology, voltage, and power; and to propose a partitioned recovery strategy that decomposes large-scale systems into sub-regions for successive optimization, fundamentally reducing problem complexity.

[0077] The core of this invention lies in achieving efficient decision-making under complex constraints through the fusion of multi-level technologies. First, a black-start model incorporating multiple physical constraints is constructed and formalized as a Markov decision process, thus abstracting the black-start problem into a mathematical mapping of state space, action space, reward function, and transition process. The state space employs a representation combining binary vectors and continuous physical quantities, guiding the agent to quickly restore the power grid's supply capacity while ensuring system safety. Based on this Markov decision framework, a lightweight policy network using deep reinforcement learning is designed and trained, learning the optimal decision strategy through continuous interaction with the environment.

[0078] 1. Mathematical Modeling of Black-Start Model with Multiple Physical Constraints

[0079] (1) Node power balance

[0080] The net power P of the system in the state space must satisfy:

[0081] P = Σ g∈G P g (t)-∑ i∈N P l,i (t)

[0082] Among them, P g (t) represents the active power output of the thermal power unit, P l,i (t) represents the node load power. Verification is required after the action is executed.

[0083] ∑ g∈G (P g (t)-δ g (t)P CR,g )≥∑ i∈N P l,i (t)

[0084] Where, δ g (t) represents the energized state of the unit (1 indicates that it is energized), P CR,g This refers to the unit's starting power.

[0085] (2) Power flow and topology constraints

[0086] The action space is constrained by the topology matrix M, allowing only nodes connected to already restored regions to be restored. The elements of the topology matrix are defined as follows: The action space is constrained and filtered using the power grid topology matrix M, allowing only nodes connected to already restored regions to be selected.

[0087]

[0088] Where, m ij This represents the element in the i-th row and j-th column of the topology matrix, used to characterize the connectivity between nodes i and j. When nodes are directly connected via transmission lines, m... ij=1; otherwise, 0. This matrix is ​​used for constraint filtering in the action space, ensuring that the node to be restored is connected to the restored region (i.e., there exists m). ij =1 path), thus satisfying the power grid topology requirements. The legality judgment condition for the action, i.e., the node j to be restored must satisfy:

[0089] ∑ i∈已恢复节点 m ij ≥1

[0090] Linearization constraints are applied to power flow:

[0091]

[0092] Among them, P ij (t), Q ij (t) represents the active / reactive power of the line, g ij b ij U is the line admittance parameter. i (t), θ i (t) represents the node voltage magnitude and phase angle.

[0093] (3) Unit operation constraints

[0094] The output of thermal power units must meet the gradient rate K. R,g limit:

[0095] |P g (t)-P g (t-1)∣≤K R,g

[0096] (4) Load input and system stability constraints

[0097] The load input for a single operation must meet the frequency deviation limit:

[0098]

[0099] In the formula, Δf max For the maximum allowable frequency deviation, P N,g f is the rated power of the unit. d,g This is the frequency response value.

[0100] At the same time, the generator self-excitation constraints need to be verified:

[0101] ∑ i∈N ∑ j:(i,j)∈B Q sh,ij,i(t) ≤∑ g∈G K CB,g S N,g

[0102] Among them, Q sh,ij,i (t) represents the reactive power of the line to ground, K CB,gS is the unit's short-circuit ratio. N,g This is the rated capacity.

[0103] (5) Constraint fusion in partition expansion

[0104] The partitioned recovery strategy achieves dynamic expansion through a dual criterion of topological neighborhood and power margin. The initial partition is centered on the black-start unit, defining a neighborhood with a radius of R = 1 (i.e., directly connected nodes). Subsequent partition expansion is conditional on the total startup power of the area to be recovered satisfying the following:

[0105] ∑P CR,g ≤(1-δ)·P base

[0106] Where δ is the safety factor (typical value 0.2), P base This is the system reference power.

[0107] The value of the safety factor δ is determined by the inertial time constant H of the power grid (unit: seconds), and the calculation formula is:

[0108]

[0109] Wherein: T base is the system reference time; H is obtained through actual measurement using the generator rotor motion equation; κ is an adjustment factor, selected according to the complexity of the power grid topology: 0.1 for radial networks and 0.3 for multi-loop networks.

[0110] In addition, traditional units within the zone must retain a spin-up / spin-off reserve capacity S. U,g and S D,g That is, satisfying:

[0111] |P g (t)-P g (t-1)∣≤S U,g +S D,g

[0112] During partition expansion, the next sub-region is selected according to the "power margin priority" principle, and the minimum distance between the node to be restored and the already restored region is calculated using the topological distance matrix D.

[0113] d j =min i∈已恢复区 D ij ,

[0114] Among them, D ij Let be the number of edges in the shortest path from node i to j, ensuring that partition expansion proceeds along the topological connectivity direction.

[0115] (6) System objective function

[0116] Load loss and outage duration are important indicators for measuring the severity of power system faults. During unit startup, to ensure system stability during the recovery phase, it is necessary to quickly increase the system's generating capacity. Therefore, maximizing the sum of the rated output of the restored thermal power units throughout the entire time period is one of the optimization objectives. Simultaneously, to minimize outage losses, another optimization objective is to maximize the restored load power throughout the entire time period. Considering the above factors, the following objective function is established:

[0117]

[0118] 2. Markov decision chain modeling for unit startup

[0119] In the process of unit startup decision-making after a major power outage, the startup sequence directly affects the efficiency and safety of system recovery. This process is characterized by stages, temporal sequence, uncertainty, and dynamism. According to the Markov assumption, the decision outcome in a single time period depends only on the current state and is independent of past states and actions. In the startup process constructed above, the decision at the next moment depends only on the current state. Therefore, this problem satisfies the Markov assumption and can be modeled as a Markov decision chain, mainly including four parts: state space, action space, reward function, and state transition function.

[0120] In addition, to ensure that the decision results meet the system security requirements, a security verification module is set up in the environment.

[0121] (1) State space

[0122] The state space is a comprehensive depiction of the current operating state of the system. It must contain key information affecting the system recovery process, including the restored generating units and their output status within the current zone, node voltage amplitude and phase angle, the sum of the rated output of the restored generating units, and the restored load power for the time period. Because the system state continuously evolves during unit startup, the state space will dynamically change along with the process of unit commissioning, line restoration, and load connection. Therefore, the state space must be dynamically updated and evaluated to ensure that the state representation at each moment accurately reflects the current system operating characteristics and feasible recovery paths. A fixed-format state coding structure is used to represent this as follows:

[0123]

[0124] Among them, s t The state space at time t is a multi-dimensional vector. By combining discrete binary variables and continuous variables, it can completely describe the operating state of the power system units at startup, ensuring the comprehensiveness and accuracy of the state information and providing sufficient input features for deep reinforcement learning models. Considering that the dimensions of the feature vectors within each partition are inconsistent, their lengths are fixed to be consistent based on the maximum partition size.

[0125] (2) Action space

[0126] The action space refers to the set of all possible operations that an agent can perform at each decision moment. In the partition recovery model proposed in this invention, the action space refers to the set of all legal recovery operations that an agent can perform at each decision moment, mainly including the start-up decision of generator sets that meet the conditions within the partition and the recovery control of bus nodes, specifically represented as follows:

[0127] a t =[χ1,…,χ i ,ν1,…,ν i ]

[0128] in, v represents the node recovery factor. i This represents the unit recovery factor. Meanwhile, to ensure the physical rationality of the recovery process and system safety, the execution of actions must meet the following constraints: nodes can only transition from a recovered state to a recovered state, and the node to be recovered must have at least one connection to a recovered node; the recovery of a unit depends on the recovery of its host node, and the unit can only be started after the host node has recovered. Since the number of units and nodes in each partition varies, the dimensions of the state vector and action vector are inconsistent. Therefore, it is necessary to uniformly encode and standardize the states and actions based on the maximum number of units and nodes in a single partition across the entire system, ensuring they meet the fixed-length requirement for convenient input and output of deep reinforcement learning models. Specifically, the units and nodes within a partition are numbered according to a predetermined order, with insufficient numbers padded with zeros.

[0129] (3) State transition function

[0130] A state transition function describes the process by which a system transitions from its current state to the next state after performing a certain action. In this invention, during the black-start partition recovery process, based on the current recovery states of the generating units and nodes, the recovery actions given by the agent, and combined with the physical constraints and topology of the power grid, the start-up and shutdown states of each generating unit, the recovery states of the nodes, and the operating parameters of the system, including node voltage, node power factor, load allocation, and the state of the restored generating units, are updated to obtain the overall system state at the next moment.

[0131] (4) Reward function

[0132] The reward function is a feedback mechanism used in reinforcement learning to evaluate the quality of each action taken by an agent. As seen in the objective function of the black-start physics model above, the reward function needs to include the following key elements: maximizing the total rated output of restored thermal power units throughout the entire time period to improve the system's power generation capacity; maximizing the restored load power throughout the entire time period to reduce power outage losses; and considering the amount of load loss and the duration of the outage as important indicators of the severity of the accident. Furthermore, the reward function should also consider system stability and safety, avoiding overly rapid or unreasonable recovery operations to promote a smooth recovery of the system.

[0133] Therefore, a piecewise design is generally adopted to balance recovery efficiency and constraint satisfaction, and the specific expression is as follows:

[0134]

[0135] Among them, B T Let t be the power generation of the target node unit within time T. k Let ρ be the recovery time at step k, and λ be the penalty coefficients (determined through training, typical values ​​are ρ = 0.5 and λ = 100). This reward function indicates that when the agent selects a node with generators for recovery, it receives a certain reward; if the selected node does not have generators, it means that it cannot improve the system's power generation capacity, and a certain penalty is imposed. In addition, when the agent's action causes the system constraints to be unmet, that is, even though the agent selects a node with generators, it cannot meet the system constraints, a relatively large penalty is imposed.

[0136] The reward function of this invention adds a constraint penalty term to the original piecewise design, specifically detailing the penalties for recovery efficiency and system constraint violations, expressed as follows:

[0137]

[0138] Among them, voltage constraint is achieved through deviation penalty: when Time (U) n (for rated voltage), triggering a -λ penalty; power imbalance penalty: the topology must satisfy connectivity rules, which can be detected by the spanning tree algorithm.

[0139] (5) Security verification module

[0140] For decisions made by the agent, the current system state and the action the agent intends to perform are obtained from the environment. Then, a topology connectivity check is performed to ensure that the node to be restored has a valid connection with the restored area. Next, fast power flow calculations are used to verify whether the voltage, current, and line load are within safe ranges. Simultaneously, it is checked whether the unit startup meets technical constraints such as minimum start-up and shutdown time, ramp rate, and output limits. The system's spinning reserve capacity is calculated to ensure safety. If necessary, a simplified dynamic stability assessment is performed. Only actions that pass these multiple checks are confirmed as legitimate and executed; otherwise, the agent's action is corrected and penalized. Specifically:

[0141] 1) If the node selected by the agent for restoration is not effectively connected to the restored area, the restoration action of that node is canceled to avoid the restoration of isolated nodes and to provide a penalty.

[0142] 2) If the node where the selected unit is located has not yet recovered, the unit startup action will be forcibly canceled to ensure that the unit startup must be based on the node having recovered.

[0143] 3) Project actions that exceed the safety boundary into the nearest set of legal actions to ensure that all actions comply with physical and safety constraints.

[0144] The relevant results are fed back to the agent to guide it in learning safe and compliant recovery strategies, thereby ensuring the stability and security of the system during the black boot process.

[0145] 3. Detailed Process of Lightweight Reinforcement Learning Algorithm

[0146] This invention employs the Deep Deterministic Policy Gradient (DDPG) algorithm as a reinforcement learning framework. This algorithm is a deep reinforcement learning method based on an actor-critic architecture, specifically designed to solve optimal control problems in continuous action spaces. DDPG combines deterministic policy gradients with deep neural networks. It directly outputs deterministic continuous actions through the policy network (Actor), avoiding the curse of dimensionality caused by discretized action spaces. Simultaneously, it utilizes a value network (Critic) to estimate the value of state-action pairs, guiding the optimization of the policy network. The algorithm employs an experience replay mechanism and target network technology to enhance training stability and sample utilization efficiency. It uses the policy gradient to update the policy network parameters through gradient ascent, iteratively approximating the optimal policy. Since the state space structure of power system control tasks is relatively stable and the input dimension is limited, shallow structures can already meet the expressive power of policy fitting. To avoid overfitting and further reduce computational resource consumption, a simplified two-layer fully connected layer is used in the policy network to replace the deep architecture in the standard DDPG, thereby significantly reducing the number of model parameters and the computational complexity of forward propagation, improving training efficiency and deployment feasibility.

[0147] See Figure 2 The overall algorithm flow of this invention includes: first, initializing the black-start unit (χ) BS =1, P=0, U=U n Subsequently, in each round, the algorithm sequentially performs the following iterative process on the selected partitions: state-space encoding, action-space filtering, action execution, and environmental feedback reward calculation. The agent selects the optimal recovery operation based on the current system state, terminating when all units have recovered or the partition decision sequence in a round reaches the maximum number of steps in a single round, ultimately outputting the optimized recovery order and path. Finally, the recovery of each partition is expanded sequentially. This invention transforms engineering constraints into computable optimization objectives through mathematical modeling, and combines the adaptive capabilities of machine learning to achieve efficient solutions for black-start decisions.

[0148] Specifically, the steps of the algorithm proposed in this invention are as follows:

[0149] (1) Data preparation: Collect historical black start cases of power grids and build a dataset.

[0150] (2) Initialize the model and environment: Set the state space S, action space A, reward function R, and state transition function.

[0151] (3) Initialize two networks: Actor network μ(s|θ) μ The Critic network Q(s,a|θ) outputs a deterministic action policy. QEstimate the Q-value of the state-action pair.

[0152] (4) Initialize the corresponding target networks μ' and Q' of the Actor network and Critic network, respectively, and set the network structure and parameters for each network.

[0153] (5) Initialize the experience replay pool Used to store interaction trajectories (s) t ,a t ,r t ,s t+1 )

[0154] (6) Set up a noise exploration strategy Used to enhance the exploratory nature of strategies.

[0155] (7) For each training episode or each step t, perform the following steps:

[0156] (i) Interaction and Sampling: Based on the current state st, select an action through the Actor network, represented as:

[0157]

[0158] (ii) Perform action a t It feeds this back to the environment, and after passing through the security verification module, it obtains the next state s. t+1 Simultaneously calculate the value of the reward function and set (s) t ,a t ,r t ,s t+1 Store in the experience pool

[0159] (iii) In the early stages of training, the experience replay pool contains relatively little data, and immediate updates can lead to overfitting or ineffective learning. Therefore, a threshold is set, and training begins once the experience pool capacity reaches this threshold.

[0160] (8) Sampling and updating: A small sample of data is randomly sampled from the experience pool and denoted as:

[0161]

[0162] (9) Update the Critic network: For each sample i, calculate the target Q value (from the target network), expressed as:

[0163] y i =r i +γQ'(s′ i ,μ'(s′ i |θ μ' )|θ Q' )

[0164] Minimizing the mean squared error loss is expressed as:

[0165]

[0166] Update the Critic network parameters θ using gradient descent. Q .

[0167] (10) Update the Actor network: Maximize the Critic's rating of the Actor through the policy gradient, expressed as:

[0168]

[0169] Update the Actor network parameters θ using the gradient ascent method. μ .

[0170] (11) Soft update of the target network: using Polyak for smooth updates, represented as:

[0171] θ μ' ←θ μ +(1-τ)θ μ'

[0172] θ Q' ←θ Q +(1-τ)θ Q'

[0173] Where τ represents the update rate (usually set to 0.001).

[0174] (12) Terminate when all units are restored or the maximum number of steps in a single round is reached.

[0175] Compared with existing technologies, this invention exhibits significant advantages in multiple dimensions: In terms of real-time performance, relying on a lightweight reinforcement learning framework and state-action space dimensionality reduction technology, it completely overcomes the bottleneck of high computational complexity in traditional mixed-integer programming methods, improving the decision response speed from the previously long processing time that could not meet emergency recovery needs to a level that can adapt to the real-time requirements of black start. Regarding constraint handling, by directly integrating the radial topology constraints and voltage limits unique to distribution networks into the reward function and action selection mechanism, it changes the lack of systematic handling of complex constraints by heuristic rules, achieving precise coordination of multiple constraints such as voltage deviation and topological connectivity, and significantly improving the feasibility of decisions. In terms of convergence efficiency, by employing a dynamic partitioning recovery strategy based on topological neighborhood and power margin, the large-scale system is decomposed into sub-regions for successive optimization, effectively solving the problem of state space explosion in traditional reinforcement learning, and significantly improving the training convergence speed. From an engineering application perspective, by constructing a constraint direct integration mechanism that includes a security verification module, and combining a partitioned recovery strategy with a lightweight reinforcement learning algorithm, this approach achieves substantial optimization in key indicators such as grid recovery efficiency and power utilization compared to traditional deep Q-networks and other solutions, demonstrating stronger engineering applicability and technological advancement.

[0176] Alternatively, at the algorithm level, proximal policy optimization can be used instead of DDPG, which processes the continuous action space through policy gradients and adjusts the reward function to a cumulative reward form; the partitioning strategy can generate initial partitions based on the Louvain graph segmentation algorithm to improve the rationality of sub-region division; the constraint processing can transform voltage constraints into soft constraints, using voltage safety margin as the reward weight to enhance decision-making flexibility; a hybrid model can also be constructed, combining machine learning power flow models and reinforcement learning algorithms to further reduce online computation.

[0177] As another example, the present invention provides a black-start partitioning decision-making device for power systems based on lightweight reinforcement learning, comprising:

[0178] The model building module is used to construct a black-start model of a power system containing multiple physical constraints, and to formalize the black-start model into a Markov decision process, which includes a state space, an action space, a reward function, and a transition process.

[0179] The decomposition module is used to decompose the power system into multiple sub-regions using a dynamic partitioning recovery strategy.

[0180] The reinforcement learning module is used to design and train a lightweight policy network for deep reinforcement learning based on the Markov decision process using a deep deterministic policy gradient algorithm.

[0181] The interactive learning module is used to enable the agent to interact with the environment in each sub-region and learn the optimal decision-making strategy through the lightweight policy network of the deep reinforcement learning.

[0182] The power restoration module is used to restore the power supply capability of the power system in different zones according to the optimal decision strategy.

[0183] Optionally, the objective function of the black-start model is:

[0184]

[0185] Among them, P l,i (t) represents the node load power, δ g (t) represents the energized state of the unit.

[0186] Optionally, the multiple physical constraints include node power balance, line power flow and topology constraints, unit operation constraints, and load input and system stability constraints.

[0187] Optionally, the dynamic partitioning recovery strategy achieves dynamic expansion through a dual criterion of topological neighborhood and power margin, thereby decomposing the power system into multiple sub-regions, specifically including:

[0188] The initial partitioning is centered on the black-boot unit, defining a neighborhood with a radius of R = 1. Subsequent partitioning expansion is based on the total startup power of the area to be restored, and must satisfy the following conditions:

[0189] ∑P CR,g ≤(1-δ)·P base

[0190] Where δ is the safety factor, P base This is the system's reference power.

[0191] The value of the safety factor δ is determined by the inertial time constant H of the power grid, and the calculation formula is as follows:

[0192]

[0193] Among them, T base H is the system reference time, which is obtained by actual measurement through the generator rotor motion equation, and κ is the adjustment factor.

[0194] Traditional units within the zone must retain a spin-off reserve capacity S. U,g and S D,g That is, satisfying:

[0195] |P g (t)-P g (t-1)∣≤S U,g +S D,g

[0196] During partition expansion, the next sub-region is selected based on the power margin priority principle, and the minimum distance between the node to be restored and the already restored region is calculated using the topological distance matrix D.

[0197] d j =min i∈已恢复区 D ij ,

[0198] Among them, D ij Let be the number of edges in the shortest path from node i to j, ensuring that partition expansion proceeds along the topological connectivity direction.

[0199] The apparatus in this embodiment is used to implement the corresponding methods in the foregoing multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0200] As another example, an electronic device is provided, which will now be described as an example of a hardware device that can be applied to various aspects of the invention, serving as a server or client of the invention. The term "electronic device" is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0201] The electronic device may include a processor, a communications interface, memory, and a communications bus.

[0202] The processor, communication interface, and memory communicate with each other via a communication bus. The communication interface is used to communicate with other electronic devices or servers.

[0203] The processor is used to execute programs, specifically the relevant steps in the above method embodiments.

[0204] Specifically, the program may include program code, which includes computer operation instructions.

[0205] The processor may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in a smart device may be of the same type, such as one or more CPUs; or they may be of different types, such as one or more CPUs and one or more ASICs.

[0206] Memory is used to store programs. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0207] When executed by a processor, the program is used to enable electronic devices to perform a black-start partitioning decision method for power systems based on lightweight reinforcement learning, as described in this invention.

[0208] Furthermore, the specific implementation of each step in the program can be found in the corresponding descriptions of the steps and units in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0209] An exemplary embodiment of the present invention also provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the methods of the various embodiments of the present invention. The corresponding process descriptions in the foregoing method embodiments can be referred to, and will not be repeated here.

[0210] The methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0211] Specific embodiments of the invention have now been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0212] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0213] Finally, it should be noted that the above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A power system black-start partitioning decision method based on lightweight reinforcement learning, characterized in that, The method comprises the following steps: A black start model of a power system containing multiple physical constraints is constructed, and the black start model is formalized as a Markov decision process, the Markov decision process comprising a state space, an action space, a reward function and a transition process; The power system is decomposed into multiple sub-regions by using a dynamic zoning recovery strategy; Based on the Markov decision process, a deep reinforcement learning lightweight policy network is designed and trained by using a deep deterministic policy gradient algorithm; An agent interacts with the environment in each sub-region by using the deep reinforcement learning lightweight policy network, and learns an optimal decision policy; According to the optimal decision policy, the power system is recovered in a zoning manner.

2. The method of claim 1, wherein, The objective function of the black start model is: Wherein, P l,i (t) is the node load power, δ g (t) is the unit energized state.

3. The method of claim 1, wherein, The multiple physical constraints include node power balance, line power flow and topology constraints, unit operation constraints, and load input and system stability constraints.

4. The method of claim 1, wherein, The dynamic zoning recovery strategy realizes dynamic expansion by using a topology neighborhood and a power margin dual criterion to decompose the power system into multiple sub-regions, specifically comprising: The initial zoning takes the black start unit as the center, and a neighborhood with a radius R=1 is drawn, and the subsequent zoning expansion condition is that the total startup power of the region to be recovered needs to satisfy: ∑P CR,g ≤(1-δ)·P base Wherein, δ is the safety factor, P base is the system reference power; The value of the safety factor δ is determined by the inertia time constant H of the power grid, and the calculation formula is: where T base is the system reference time, H is obtained by the generator rotor motion equation, and κ is an adjustment factor. The traditional units in the partition need to reserve up / down rotating standby capacity S U,g and S D,g i.e. meet: | P g (t)-P g (t-1) | ≤ S U,g + S D,g When zoning expansion, the next sub-region is selected according to the power margin priority principle, and the minimum distance between the node to be recovered and the recovered region is calculated by using a topology distance matrix D. where D ij is the number of shortest path edges from node i to j, ensuring that the partition expansion proceeds in a topologically connected direction.

5. A power system black-start partitioning decision device based on lightweight reinforcement learning, characterized in that, The method comprises the following steps: A model construction module is configured to construct a black start model of a power system containing multiple physical constraints, and formalize the black start model as a Markov decision process, the Markov decision process comprising a state space, an action space, a reward function and a transition process; A decomposition module is configured to decompose the power system into multiple sub-regions by using a dynamic zoning recovery strategy; A reinforcement learning module is configured to design and train a deep reinforcement learning lightweight policy network based on the Markov decision process by using a deep deterministic policy gradient algorithm; An interactive learning module is configured to make an agent interact with the environment in each sub-region by using the deep reinforcement learning lightweight policy network, and learn an optimal decision policy; A power recovery module is configured to recover the power system in a zoning manner according to the optimal decision policy.

6. The apparatus of claim 1, wherein, The objective function of the black start model is: Wherein, P l,i (t) is the node load power, δ g (t) is the unit power-on state.

7. The apparatus of claim 1, wherein, The multiple physical constraints include node power balance, line power flow and topology constraints, unit operation constraints, and load input and system stability constraints.

8. The apparatus of claim 1, wherein, The dynamic zoning recovery strategy realizes dynamic expansion by using a topology neighborhood and a power margin dual criterion to decompose the power system into multiple sub-regions, specifically comprising: The initial zoning takes the black start unit as the center, and a neighborhood with a radius R=1 is drawn, and the subsequent zoning expansion condition is that the total startup power of the region to be recovered needs to satisfy: ∑P CR,g ≤(1-δ)·P base wherein δ is a safety factor, P base is the system base power; The value of the safety factor δ is determined by the inertia time constant H of the power grid, and the calculation formula is: where T base is the system reference time, H is obtained by the generator rotor motion equation, and κ is an adjustment factor. The traditional units in the partition need to reserve up / down rotating standby capacity S U,g and S D,g i.e. meet: |P g (t)-P g (t-1) |≤S U,g +S D,g When zoning expansion, the next sub-region is selected according to the power margin priority principle, and the minimum distance between the node to be recovered and the recovered region is calculated by using a topology distance matrix D. where D ij is the number of shortest path edges from node i to j, ensuring that the partition expansion proceeds in a topologically connected direction.

9. An electronic device, comprising: The method comprises the following steps: A processor; A memory for storing programs; The program comprises instructions which, when executed by the processor, cause the processor to perform the steps of the method of any of claims 1-4.

10. A computer storage medium, characterized in that, A computer program product comprising a computer readable medium having stored thereon the computer program, the program being configured such that, when executed by a processor, the processor is caused to perform the method of any of claims 1-4.

Images