A transient voltage stability control method fusing expert experience and agent exploration

Abstract

The application relates to the technical field of power system control, and discloses a transient voltage stability control method fusing expert experience and agent exploration, which establishes a large-interference transient voltage stability emergency control mathematical model, clearly defines control targets and various constraint conditions, and determines emergency control measures taking load shedding as the core; the emergency control problem is modeled as a Markov decision process, a state space, an action space and a reward function are designed by taking into account system transient stability and control economy; an emergency control model based on a SACGAIL fusion framework is constructed, the SAC algorithm is improved to adapt to single-step decision, the GAIL framework is introduced to fuse expert experience, and a behavior cloning pre-training and dynamic expert expansion mechanism is used to optimize model training; an optimal control strategy is obtained through offline training and is deployed to a power grid dispatching control center, and optimal load shedding control actions are output under large-interference faults. The application effectively improves the reliability and efficiency of transient voltage stability emergency control of a power system.

Description

Technical Field

[0001] This invention relates to the field of power system control technology, specifically to a transient voltage stabilization control method that integrates expert experience with intelligent agent exploration. Background Technology

[0002] Against the backdrop of a global green and low-carbon energy transition, developing new energy sources has become an inevitable path for countries to address environmental issues and the energy crisis. According to the International Energy Agency's "World Energy Outlook 2025" report, renewable energy currently accounts for about one-third of global electricity generation, and this figure is projected to reach more than half by 2035 and over two-thirds by 2050. Currently, wind power, solar power, and other new energy sources are experiencing rapid growth, and high-voltage direct current transmission projects are continuously being put into operation, forming the world's largest and most complex power system.

[0003] This large power system has high dimensionality, strong uncertainty and nonlinearity. In addition, the power electronic equipment has weak disturbance rejection and weak reactive power support. After AC or DC faults, the risk of transient voltage instability in the system is aggravated. In extreme cases, it may lead to voltage collapse and system disconnection, which poses a severe challenge to the safe and stable operation of the power grid.

[0004] Emergency control, as the second line of defense for power grid security, is of paramount importance in terms of reliability. Stability control technology based on pre-defined strategy tables is the most widely used due to its high maturity and low cost, employing an "offline pre-decision-online matching" mechanism. However, with the expansion and increasing complexity of the power grid, it faces challenges such as difficulty in constructing power grid mechanism models, high simulation resource consumption, and difficulties in covering all complex disturbance scenarios due to decision-making based on a limited set of anticipated faults, as well as the susceptibility to strategy mismatch.

[0005] With the widespread adoption of PMU and WAMS, dynamic response-driven emergency control methods incorporating AI technology are rapidly developing. Shallow neural networks, deep learning, and DRL methods have been applied to voltage stability assessment and control. DRL, in particular, does not require precise mathematical models, is suitable for high-dimensional and nonlinear control problems, and is widely used in transient stability emergency control scenarios. However, DRL suffers from low training sample efficiency, slow convergence, and a tendency to get trapped in local optima. Furthermore, its reward function design relies on human preferences and does not fully utilize expert experience, limiting its application potential.

[0006] The GAIL method in imitation learning can learn expert policies from environmental trajectories, improving learning efficiency and safety. It has been preliminarily applied to power system optimization and dispatching, but research in the field of transient voltage stability emergency control is limited. Moreover, GAIL alone cannot consider the physical effects of control actions and may violate physical constraints. Therefore, it is necessary to combine expert guidance and environmental feedback to construct a hybrid reward signal to guide the agent in optimizing its strategy. Summary of the Invention

[0007] To overcome the shortcomings of the existing technology, the present invention aims to provide a transient voltage stability control method that integrates expert experience and agent exploration, so as to solve the technical problems of low sample efficiency, slow convergence, easy getting trapped in local optima, and the reward function design relying on human preferences and not making full use of power system expert experience and historical excellent strategies, resulting in limited strategy optimization effect and reliability.

[0008] This invention is achieved through the following technical solution: This invention provides a transient voltage stabilization control method that integrates expert experience and agent exploration, comprising: A mathematical model for emergency control of transient voltage stability under large disturbances was established, the control objectives, dynamic constraints, static constraints and inequality constraints were clarified, and the emergency control measures with load shedding as the core were determined. The core emergency control measures are modeled as Markov decision processes, and the state space, action space and reward function of the Markov decision process are designed. The reward function comprehensively reflects the system's transient stability requirements and control economy objectives. An emergency control model based on the SAC-GAIL fusion framework is constructed using Markov decision process. The SAC algorithm is improved to adapt to single-step decision-making. The GAIL framework is introduced to realize the fusion of expert experience. Behavioral cloning pre-training and dynamic expert expansion mechanism are used to optimize model training. The emergency control model of the SAC-GAIL fusion framework is trained offline to obtain and save the optimal control strategy; The emergency control model of the SAC-GAIL fusion framework, which was trained offline, was deployed to the power grid dispatch and control center. The power grid status was monitored in real time during the online application phase. After a major disturbance fault was detected, the model output the optimal load shedding control action to complete the emergency control of transient voltage stability of the power system.

[0009] The preferred mathematical model for emergency control of large-disturbance transient voltage stability takes minimizing the total control cost as its objective function, and the objective function expression is as follows:

[0010] Where w represents the cost vector of emergency control measures, and a represents the action vector of emergency control measures; The dynamic constraint expression is as follows:

[0011] Where x represents the state variable describing the dynamic characteristics of the system, y represents the algebraic variable of the system, and d represents the external disturbance. This constraint includes the synchronous generator rotor motion equation, the excitation system equation, and the equipment dynamic characteristic equation. The static constraint expression is as follows: 0 = g(x, y, a, d) The static constraints include power network equations, power flow equations, and equipment static characteristic equations. The inequality constraint expression is as follows: 0 <h(x,y,a,d) The inequality constraints include system stability constraints and emergency control measure control quantity constraints.

[0012] Preferably, the system stability constraints include transient voltage stability constraints and power angle stability constraints; The transient voltage stability constraints are as follows:

[0013] in, express t Time Node i The voltage amplitude; Indicates the fault clearing time; Indicates the number of nodes; The power angle stability constraint is as follows:

[0014] in, express t Time of the first i The power angle of the generator; This indicates the number of generators.

[0015] Preferably, the state-space expression is as follows:

[0016] in, and These represent the voltage magnitude vector and phase angle vector at each node of the system, respectively. and These represent the generator power angle vector and the rotor angular velocity vector, respectively. The state vector is composed of system measurement data within a 0.2-second time window after the fault is cleared. The action space is as follows:

[0017] in, Indicates the first i The load shedding amount at each load node, with the action value being a continuous variable. This represents the maximum amount of material that can be removed from this node. Indicates the number of load nodes that can be shelved; The reward function is as follows:

[0018] The voltage stability reward expression is as follows:

[0019] The voltage-steady final-state reward expression is as follows:

[0020] The expression for the power angle stability reward is as follows:

[0021] The expression for the cost control reward is as follows:

[0022] Where R vol For voltage stability bonus, R angle For the purpose of improving the stability of the angle, R load To control costs, incentives are offered; terminal This is a reward for the final state of voltage stability; T This refers to the duration of the transient simulation. c 1. c 2. and c 3 represents the weighting coefficients for low voltage penalty, overvoltage penalty, and control cost, respectively.

[0023] Preferably, the improvement process of the SAC algorithm includes setting the discount factor γ to 0 and calculating the Q value through the Critic network, as shown in the following formula:

[0024] in, R hybrid It is a hybrid reward, obtained by weighted fusion of environmental rewards and GAIL imitation rewards; The update process of the SAC algorithm includes Critic network update, Actor network update, and adaptive adjustment of entropy coefficient. The Actor network solves the gradient backpropagation problem through reparameterization sampling, and the entropy coefficient α is automatically adjusted through an optimization problem with entropy constraints.

[0025] Preferably, the GAIL framework uses a discriminator and the SAC Actor network for adversarial training. The discriminator uses a double-buffer structure to construct training data and is trained using a binary cross-entropy loss function to distinguish between expert data and agent-generated data. The hybrid reward expression is as follows:

[0026]

[0027] in, λ represents the GAIL reward, k is the scaling factor, and λ is the fusion coefficient, which gradually decreases from an initial high value to 0 as training progresses.

[0028] Preferably, behavior cloning pre-training utilizes an expert dataset to optimize the Actor network by minimizing the mean square error between the agent's actions and the expert's actions; The dynamic expert expansion mechanism adds state-action pairs whose environmental reward R(s,a) exceeds a predetermined threshold ξ to the expert dataset after each round of interaction, thereby expanding the expert data.

[0029] Preferably, offline training includes the following process: S1, initialize the GAIL discriminator network, SAC Actor-Critic network parameters and experience replay buffer, and load the steady-state and transient simulation files of the power system simulation environment; S2, use the expert dataset to perform BC pre-training on the Actor network of SAC to obtain the initial security policy; S3 configures the initial operation mode of the power grid and large disturbance fault scenarios. The intelligent agent interacts with the simulation environment and collects status, action and reward data. S4. Select high-performance interactive samples to expand the expert dataset, and train the GAIL discriminator network and the SAC Actor-Critic network alternately. S5 determines whether the model performance has converged or reached the preset maximum number of training iterations. If not, it returns to S3 to reconfigure the scene and continue training. If it does, it saves the optimal control strategy.

[0030] Preferably, the specific process of the online application phase includes: L1 deploys the optimal control strategy model, which has been trained and saved offline, to the power grid dispatch and control center and establishes a communication connection with the wide-area measurement system. L2 continuously monitors the power grid's operating status information in real time through the wide-area measurement system; L3 determines in real time whether a major disturbance fault has occurred in the power grid; if no fault has occurred, it continues to monitor; if a fault is detected, it activates the deployed control strategy model. L4 acquires real-time grid status information after fault clearing, and the control strategy model calculates and outputs the optimal emergency control action command within milliseconds. L5 sends the control action command to the power grid actuator to perform load shedding operations in order to quickly restore and maintain the transient voltage stability of the power system.

[0031] Furthermore, the control strategy model is a SAC-GAIL fusion framework model obtained through BC pre-training, dynamic expert expansion, and SAC and GAIL co-optimization training. Its input is the system measurement data within a 0.2-second time window after fault clearance, including the voltage amplitude, phase angle, generator power angle, and rotor angular velocity of each node. The emergency control action command is the load shedding amount of each load shedding node, and the actuator is a circuit breaker. By quickly cutting off the power deficit of the specified load node, the power flow pressure of the transmission section is relieved.

[0032] Compared with the prior art, the present invention has the following beneficial technical effects: This invention provides a transient voltage stabilization control method that integrates expert experience and agent exploration. It constructs a collaborative training mechanism combining SAC maximum entropy optimization and GAIL expert imitation. SAC serves as the basic optimizer, ensuring the exploratory nature and convergence stability of the policy. GAIL acts as an expert knowledge injection pathway, guiding the agent's behavior to approximate the expert policy distribution, effectively balancing experience inheritance and autonomous optimization. Simultaneously, a hybrid reward mechanism and a dual-buffer architecture are designed. The hybrid reward mechanism dynamically integrates GAIL imitation rewards and environmental rewards, balancing imitation learning and autonomous optimization objectives through adjustable weight coefficients. The dual-buffer structure utilizes a recent interaction buffer to quickly capture policy updates and combines it with an experience replay buffer to reuse historical data, jointly ensuring the accuracy of the discrimination signal and training stability. Furthermore, behavioral cloning pre-training is introduced, using expert data to safely initialize the policy network, reducing initial exploration risks and accelerating convergence. A dynamic expert expansion mechanism is also employed, online screening of high-reward samples to enrich the expert dataset, enabling the discriminator to continuously learn and improve the policy, effectively alleviating expert data scarcity, enhancing model generalization ability and control performance, and improving the reliability and efficiency of transient voltage stabilization emergency control. Attached Figure Description

[0033] Figure 1 This is a flowchart of the transient voltage stabilization control method that integrates expert experience and intelligent agent exploration in an embodiment of the present invention; Figure 2 This is the main training flowchart of the transient voltage stability emergency control model based on the SAC-GAIL fusion framework in this embodiment of the invention; Figure 3 This is a diagram of the emergency control framework for stabilizing transient voltage under large disturbances in an embodiment of the present invention. Figure 4 This is a structural diagram of the CSEE-VS system in an embodiment of the present invention; Figure 5 This is a graph showing the reward function and discriminator probability curves during the training process in an embodiment of the present invention; Figure 6This is a comparison chart of the training reward functions of different algorithms in the embodiments of the present invention; Figure 7 This is a diagram showing the voltage response of the key bus in the system before and after applying control under a typical fault scenario in an embodiment of the present invention. Figure 8 This is a diagram showing the voltage response of the key busbar of the system before and after pre-planned control under extreme operating conditions in an embodiment of the present invention. Figure 9 This is a diagram showing the voltage response of the key bus in the system after SAC-GAIL control under extreme conditions in an embodiment of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] The purpose of this invention is to provide a transient voltage stabilization control method that integrates expert experience and agent exploration, in order to solve the technical problems in the prior art, such as low sample efficiency, slow convergence, easy getting trapped in local optima, and the reward function design relying on human preferences and not making full use of power system expert experience and historical excellent strategies, resulting in limited strategy optimization effect and reliability.

[0036] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 In one embodiment of the present invention, a transient voltage stabilization control method integrating expert experience and agent exploration is provided, comprising: Step 1: Establish a mathematical model for emergency control of transient voltage stability under large disturbances, clarify the control objectives, dynamic constraints, static constraints and inequality constraints, and determine the emergency control measures with load shedding as the core. Specifically, the emergency control problem of power systems is a high-dimensional, highly nonlinear optimal control problem. The objective function is to minimize the total control cost, and the constraints include static and transient security constraints. The resulting emergency control optimization model is as follows: (1) (2) (3) (4) In the formula, Represents the cost vector of emergency control measures; Represents the action vector of emergency control measures; State variables that describe the dynamic characteristics of a system; Represents the algebraic variables of the system; Equation (2) represents the external disturbance. Equation (3) represents the dynamic constraints of the system, which include a set of differential equations describing the dynamic characteristics of the system, such as the synchronous generator rotor motion equation, the excitation system equation, and the equipment dynamic characteristic equation; Equation (4) represents the static constraints of the system, which include a set of algebraic equations of the system, such as the power network equation, the power flow equation, and the equipment static characteristic equation; Equation (5) represents the inequality constraints of the system, including system stability constraints and emergency control measure control quantity constraints.

[0037] According to the practical criteria for large disturbance voltage stability, during the transient process after a power system is disturbed, the load node voltage should recover to above 0.8 pu within a certain time (defined as 5 seconds in this invention) after the fault is cleared. Combined with system overvoltage requirements, the voltage amplitude should not exceed the rated voltage upper limit of 1.2 pu. Therefore, the system transient voltage stability constraints are defined as follows: (5) In the formula, express t Time Node i The voltage amplitude; Indicates the fault clearing time; Indicates the number of nodes.

[0038] During emergency voltage stabilization control, measures such as reactive power compensation or load reduction can alter the system power flow distribution, potentially leading to power imbalance and power angle instability. Therefore, a comprehensive voltage stabilization control strategy must consider both voltage and power angle stability. The power angle stability constraint is defined as follows, using a relative power angle difference of no more than 180° between key units: (6) In the formula, express t Time of the first i The power angle of the generator; This indicates the number of generators.

[0039] In summary, the emergency control problem for large-disturbance transient voltage stability is essentially a complex decision-making problem in a high-dimensional, nonlinear dynamic system environment, satisfying multiple stringent stability constraints. Its mathematical model is characterized by strong nonlinearity, high dimensionality, and uncertainty, specifically manifested in: the power system's differential-algebraic equations describing complex electromechanical transient processes; the strong nonlinear effects of emergency control measures; and the introduction of combinatorial optimization challenges. Furthermore, the randomness of the system's operating point, fault scenarios, and the coupling effects between stability constraints make traditional optimization methods based on linearized models or enumeration search difficult to solve effectively and apply online, facing bottlenecks such as the "curse of dimensionality" and low computational efficiency.

[0040] DRL offers a novel approach to solving such complex decision-making problems. DRL agents do not rely on precise mathematical models of the system; instead, they learn optimal control strategies directly from data through repeated interaction with power system simulation environments. To transform the voltage emergency control problem into a DRL-solvable paradigm, it first needs to be modeled as a Markov Decision Process (MDP).

[0041] Step 2: Model the core emergency control measures as a Markov decision process, and design the state space, action space and reward function of the Markov decision process. The reward function comprehensively reflects the system's transient stability requirements and control economy objectives. Specifically, MDP modeling is performed for the transient voltage stability emergency control problem. DRL is a machine learning method based on MDP as its theoretical framework. Its core lies in the agent learning the optimal strategy that maximizes cumulative rewards through interaction with the environment. This method is essentially a data-driven approximate dynamic optimization technique. To construct a DRL model, three core elements need to be clarified: the state space perceived by the agent, the action space that can be executed, and the reward function that guides the learning direction. The state transition is controlled by the mathematical model of emergency control (1)-(4) and is realized by transient simulation in the power system environment (PSD-BPA in this invention), from which the agent learns the system dynamics. Through this formal modeling, the agent can learn a mapping strategy from the system state to the control action.

[0042] Specifically, the emergency plan table that establishes the mapping relationship between "fault scenarios and control measures" under the grid contingency plan control mode is a single-step decision-making process. To effectively utilize the expert experience of grid contingency plan control, this invention adopts the same single-step decision-making DRL training paradigm as contingency plan control. At a single decision moment after fault clearance, a complete, one-step global emergency control strategy is generated based on the system state. The learning objective is to learn an optimal strategy. The goal is to maximize the expected reward of the agent's actions after fault clearance. Therefore, the design of the agent's state space, action space, and reward function must all revolve around the core objective of achieving the final control effect of a single-step decision.

[0043] 1) State-space design In the single-step control paradigm, the state space needs to provide the agent with a multi-dimensional representation of the system's operating state after fault clearance, in order to support its single-step control decisions. The state vector is composed of system measurement data within a specific time window after fault clearance (0.2 seconds in this invention), aiming to characterize the initial transient characteristics of the system after fault disturbance. The voltage amplitude and phase angle of each node in the system, the generator power angle and rotor angular velocity are selected as the state vector, as shown in Equation (7). By integrating the above information, the state space can effectively reflect the impact of the fault scenario on the system, thereby providing the necessary perception basis for the agent to generate control strategies.

[0044] (7) In the formula, and These represent the voltage magnitude vector and phase angle vector at each node of the system, respectively. and These represent the generator power angle vector and the rotor angular velocity vector, respectively.

[0045] 2) Motion space design The action space defines the set of control instructions that an agent can execute at the decision-making moment. In emergency voltage stabilization control, load shedding, as a widely used and effective direct control method, essentially improves the reactive power balance of the system by reducing the power deficit in local areas, alleviating the power flow pressure on the transmission section, preventing further voltage drops, and promoting its recovery. This invention uses load shedding as the core control method, and the action space is defined as shown in equation (8): (8) In the formula, Indicates the first i The load shedding amount at each load node, with the action value being a continuous variable. This represents the maximum amount of material that can be removed from this node. This indicates the number of load nodes that can be switched.

[0046] 3) Reward function design The reward function is designed to comprehensively reflect the transient stability requirements and control economic objectives of the power system, quantifying the dynamic response and control costs of the system during transient processes. The reward function is calculated after each simulation round to evaluate the overall control effectiveness of the implemented control strategy.

[0047] The transient stability reward is evaluated based on the entire simulation trajectory. The voltage stability reward promotes the rapid recovery of the voltage to the safe range after a fault. Equation (9) penalizes the portion of the bus voltage that is below 0.8 pu and above 1.1 pu during the recovery process (leaving a certain safety margin relative to the overvoltage requirement of 1.2 pu). Equation (10) only gives the final state reward when all bus voltages eventually recover to above 0.8 pu and no overvoltage occurs during the entire recovery process. The power angle stability reward and the control cost reward are defined by Equations (11) and (12), respectively. The total reward is the sum of these components, as shown in Equation (13).

[0048] (9) (10) (11) (12) (13) In the formula, T This refers to the duration of the transient simulation. c 1. c 2. and c 3 represents the weighting coefficients for low voltage penalty, overvoltage penalty, and control cost, respectively. β vol The final state reward for successfully restoring voltage stability is set to a large positive number. β angle To penalize power angle instability, a large negative number is set. All electrical quantities in the equations are expressed in per-unit values. By reasonably setting the weighting coefficients of each item, the DRL agent can be guided to maintain the transient stability of the system while minimizing the control cost, thereby achieving efficient and reliable emergency voltage control.

[0049] Step 3: Construct an emergency control model based on the SAC-GAIL fusion framework through Markov decision process, improve the SAC algorithm to adapt to single-step decision-making, introduce the GAIL framework to realize the fusion of expert experience, and optimize model training by using behavior cloning pre-training and dynamic expert expansion mechanism. Specifically, an emergency control model for transient voltage stability in power systems based on the SAC-GAIL fusion framework is proposed. This framework fully utilizes the optimization capabilities of the SAC algorithm in continuous control, while introducing expert demonstration data through the GAIL mechanism, enabling the agent to generate control strategies that both meet system stability requirements and approximate expert behavior patterns.

[0050] To address the single-step decision-making characteristics in emergency control of transient voltage stability in power systems, this invention specifically improves the SAC algorithm, making it more suitable for single-step control scenarios. The original SAC algorithm, as an offline policy algorithm based on maximum entropy reinforcement learning, explores and utilizes entropy regularization balance, and employs a dual... Q The network architecture mitigates the overvaluation problem. The algorithm employs a collaborative architecture of one random policy network (Actor) and two value networks (Critic). The parameters of the two Critic networks are as follows: and Used for independently estimating the value of state-action pairs Q The smaller value is taken as the objective to guide the Actor's policy optimization; the Actor network parameters are... θ It takes the real-time system state vector as input and outputs a random policy distribution in the continuous action space. It interacts with the environment through the maximum entropy objective and achieves stable policy updates under the guidance of Critic value.

[0051] In the single-step control paradigm, the agent performs an action only at a single decision point after the fault is cleared, without considering long-term cumulative rewards. Therefore, the discount factor... γ Setting it to 0 means Q Value updates no longer depend on the value estimate of the next state, thus eliminating the need for the target network and its soft update mechanism. The Critic network is calculated using equation (14). Q This simplification avoids errors introduced by estimating the value of future states, thus improving training stability.

[0052] (14) In the formula, R hybrid It is a hybrid reward, derived from a weighted fusion of environmental rewards and GAIL imitation rewards.

[0053] The improved SAC update paradigm is as follows: 1) Critic Network Update The dual-Critic network estimates the state-action value by minimizing the mean squared error loss, and its update method is as follows: (15) (16) In the formula, This serves as an experience replay buffer, storing data collected in the past by the strategy. This represents the learning rate of the dual-Critic network.

[0054] 2) Actor Network Update The Actor network addresses the issue of gradients failing to propagate back to the policy network parameters through action reparameterization. The loss function of the Actor network is derived from the KL divergence, and its update method is as follows: (17) (18) (19) In the formula, This represents the entropy regularization term, which quantifies the policy. In state The uncertainty of choosing an action encourages strategies to maintain a certain degree of randomness; This is the entropy coefficient, used to control the weight of the entropy term; This indicates the action after reparameterized sampling; μ θ and σ θ Let these represent the mean and variance of the policy network output, respectively. To conform to the standard Gaussian distribution Sampling noise; This represents the learning rate of the Actor network.

[0055] 3) Adaptive adjustment of entropy coefficient In the SAC algorithm, the entropy coefficient Instead of fixing the hyperparameters, the goal of reinforcement learning is rewritten as an optimization problem with entropy constraints to achieve automatic adjustment of the entropy regularization term. This means maximizing the expected reward while ensuring the mean entropy is greater than the target value. The update method is as follows: (20) (twenty one) In the formula, The target entropy is usually set as a negative value of the action space dimension; The learning rate represents the entropy coefficient. This mechanism enables the algorithm to dynamically adjust the exploration intensity based on the difference between the policy entropy and the target entropy, in order to maintain a proper balance between exploration and exploitation.

[0056] To integrate the experience and historically successful control strategies of power system operation experts into the learning process of the agent, this invention introduces the GAIL framework. Through an adversarial training mechanism, it guides the agent to generate action data distributions that approximate the expert data distribution. GAIL originates from the idea of ​​generative adversarial networks (GANs), achieving imitation learning through adversarial game between the discriminator and the generator. Specifically, the discriminator... D ψThe goal is to distinguish between expert data and agent-generated data, and output... This represents the probability that a state-action pair belongs to the expert data; the generator, i.e., the Actor network of SAC, attempts to generate actions that can "deceive" the discriminator, thereby causing the agent's policy to gradually converge to the expert policy distribution.

[0057] In the SAC-GAIL algorithm implementation of this invention, the training data for the discriminator is constructed using a double-buffer structure: a recent interaction buffer. Stores recent interaction data of the intelligent agent, representing data generated under the latest policy, and serves as an experience replay buffer. This allows the system to accumulate state-action data generated by historical strategies. By prioritizing sampling of recent data, the discriminator can better adapt to distribution changes during strategy updates, while fully reusing diverse past experiences.

[0058] The discriminator is trained based on the binary cross-entropy loss function, and is optimized by maximizing the log-likelihood difference between expert data and agent-generated data. This ensures that the distribution of all state-action pairs in the environment is consistent with the expert policy. The update method is as follows: (twenty two) (twenty three) (twenty four) In the formula, ψ Indicates the discriminator network parameters; D E Represents the expert dataset, N E The number of its state-action pair samples; η ψ This represents the learning rate of the discriminator.

[0059] Through iterative updates, the discriminator gradually improves its discrimination ability, forcing the agent's strategy to generate data more similar to expert behavior, thereby achieving the goal of imitation learning. To balance expert imitation and environmental feedback, this invention designs a hybrid reward mechanism, merging GAIL rewards and environmental rewards into a hybrid reward. The GAIL reward is linearized and scaled to the same order of magnitude as the environmental reward to facilitate reward fusion. Specifically, the hybrid reward... The definition is as follows: (25) (26) In the formula, The GAIL reward measures the similarity between the agent's actions and the expert's behavior. A high reward is given when the agent's behavior is similar to the expert's behavior, and a low reward is given otherwise. The environmental reward is represented by equations (9)-(13), which are used to evaluate the overall control effect of the control action. As a scaling factor, the GAIL reward value is adjusted to be on the same order of magnitude as the environment reward, thus avoiding training instability caused by differences in reward scale. This is the fusion coefficient, used to balance the goals of imitation learning and reinforcement learning, so that it decreases from an initial high value as training progresses. λ The value is gradually reduced from 0 to 0. This design constructs a learning curriculum: in the early stages of training, the agent focuses on imitating expert demonstrations (…). λ (High value), as training progresses, gradually shift attention to maximizing environmental rewards ( λ (Low value).

[0060] To mitigate control risks that may arise from random exploration in the early stages of reinforcement learning and to accelerate convergence, this invention first employs behavior cloning (BC) to pre-train the Actor network. In this stage, expert behavior is directly mimicked through supervised learning using an expert dataset. The optimization objective is to minimize the mean squared error between the agent's actions and the expert's actions. (27) In the formula, π θ ( s ) indicates that the policy network is in state The action to be output; This indicates the actual control actions performed by the expert in this state.

[0061] By pre-training with BC, the Actor network can obtain an initial policy that is close to that of an expert. This helps to reduce the training variance in the early exploration phase of reinforcement learning and provides a high-starting-point initial solution for subsequent adversarial training.

[0062] To address the limitations of limited expert data scale and insufficient scenario coverage, this invention designs a dynamic expert expansion mechanism. This mechanism continuously incorporates high-quality interaction data generated by the agent into the expert dataset during reinforcement learning training, thereby expanding and enhancing the expert data. Specifically, after each round of interaction, the system evaluates the performance of the agent's control strategy. If the environmental reward for a certain state-action pair is... Exceeding the predetermined threshold Then add the interaction sample to the expert dataset. As shown in equation (28): (28) The dynamic expert expansion mechanism draws on the idea of ​​bootstrapping learning, continuously feeding high-performance samples explored by the agent back to the expert set. This effectively alleviates the problem of scarce initial expert data and significantly improves the diversity and scale of training data. More importantly, this mechanism guides the discriminator's learning objective from imitating fixed initial expert behavior to gradually evolving into identifying generalized superior behaviors that lead to high rewards. This constructs an implicit curriculum that guides the agent to gradually learn better strategies, helping to improve control performance and generalization performance under complex and variable conditions.

[0063] The main training process of the transient voltage stability emergency control model based on the SAC-GAIL fusion framework established in this invention is as follows: Figure 2 As shown. The overall training process integrates BC pre-training, dynamic expert augmentation, and collaborative optimization of SAC and GAIL, aiming to achieve efficient and stable policy learning. It mainly includes the following two stages: 1) Initialization and BC pre-training phase: Initialize the network parameters and experience replay buffer, and use the expert dataset to perform BC pre-training on the SAC Actor network to obtain a safe initial policy; 2) Main training loop: The agent interacts with the environment to collect transfer samples, selects high-performance samples to expand the expert set, and alternately trains the GAIL discriminator network and the SAC Actor-Critic network.

[0064] The pseudocode for the detailed training process of the algorithm is shown below: SAC-GAIL fusion algorithm Input: Power system simulation environment, expert dataset Number of BC pre-training iterations and learning rate SAC policy network and learning rate SAC Value Network , and learning rate GAIL discriminator network and learning rate Total number of training rounds Empty experience replay buffer and recent interaction buffer Recent interaction buffer capacity Entropy regularization coefficient GAIL reward scaling factor k Reward fusion coefficient Experts expand reward threshold

[0065] 1. / / Phase 1: Behavioral Cloning Pre-training 2.for i =1 to do 3. From Randomly sampled small batches of data

[0066] 4. Calculate the behavioral cloning loss:

[0067] 5. Update the policy network:

[0068] 6. end for 7. / / Phase Two: Main Training Loop 8.for j =1 to do 9. / / 1. Environmental Interaction and Data Collection 10. Set the system operating mode and fault scenarios, simulate until the fault is cleared, and observe the current system status.

[0069] 11. Output control actions according to the strategy:

[0070] 12. Perform actions in a simulation environment. Run transient simulations and obtain environmental rewards.

[0071] 13. Storing and transferring samples: ,

[0072] 14.if then 15. From Remove the oldest sample / / First-In-First-Out (FIFO) management 16.end if 17. / / 2. Dynamic Expert Expansion 18.if then 19. Add the corresponding interaction sample to the expert set:

[0073] 20.end if 21. / / 3. GAIL Discriminator Update 22. From the expert group Sampling small batches of expert data:

[0074] 23. From the buffer and Sampling small batches of interactive data:

[0075] 24. Calculate the discriminator loss:

[0076] 25. Update the discriminator network:

[0077] 26. / / 4. SAC agent update 27. From Sample transfer in small batches

[0078] 28. Calculate GAIL rewards:

[0079] 29. Calculate mixed rewards:

[0080] 30. / / Update the Critic network 31. Calculate Critic loss:

[0081] 32. Update the Critic network:

[0082] 33. / / Update the Actor network 34. Through reparameterized sampling action:

[0083] 35. Calculate the policy loss:

[0084] 36. Update the Actor network:

[0085] 37. / / Update entropy regularization coefficients 38. Calculate Loss:

[0086] 39. Update the entropy coefficient:

[0087] 40.end for Step 4: Train the emergency control model of the SAC-GAIL fusion framework offline to obtain and save the optimal control strategy; Specifically, the SAC-GAIL-based transient voltage stability emergency control method is divided into an offline training phase and an online application phase. The offline phase is used to train the agent to interact extensively with the power grid simulation environment, repeatedly exploring and optimizing, and learning the dynamic mechanisms of the power grid under the guidance of expert data, thereby gradually acquiring an effective control strategy. The main steps of the offline training phase are as follows: S1: System Initialization Initial parameters for the GAIL discriminator network and the SAC Actor-Critic network are randomly generated, and the steady-state and transient simulation files of BPA are loaded to complete the system initialization.

[0088] S2: Behavioral Cloning Pre-training Based on the power grid emergency control plan or excellent control examples, the SAC Actor network is pre-trained using BC to improve the initial policy performance of the agent.

[0089] S3: Training Scenario Configuration For a single training round, configure the initial operating mode of the power grid in the BPA environment, such as load level and generator output, and preset large disturbance fault scenarios.

[0090] S4: Interaction and Training Loop This step is the core process of agent learning, and it includes a complete interactive loop: 1) State awareness: The intelligent agent obtains the state variables of the power grid after the fault is cleared, such as node voltage and generator power angle, in real time from the BPA simulation environment through the self-developed BPA API interface.

[0091] 2) Decision-making and output: The SAC-GAIL agent analyzes the current state and outputs specific emergency control action instructions.

[0092] 3) Execution and feedback: Control actions are sent to the BPA simulation environment via API for execution. The simulation program advances to the final state, and the simulation environment then feeds back the new state of the power grid after the action is executed. The environment reward is calculated as feedback based on the voltage recovery effect and control cost.

[0093] 4) Policy optimization: The SAC-GAIL agent uses the collected state, action and reward data to expand the high-quality interaction data generated by the agent to the expert dataset, and trains the GAIL discriminator and SAC's Actor-Critic network in small batches.

[0094] S5: Convergence Test Determine whether the agent's performance has converged or whether the preset maximum number of training iterations has been reached. If the conditions are not met, return to S2 to configure a new scenario and continue learning; if the conditions are met, proceed to S5.

[0095] S6: Policy Saving Save and archive the parameters of the trained and high-performing SAC-GAIL agent network for use in the online application phase.

[0096] Step 5: Deploy the emergency control model of the SAC-GAIL fusion framework, which has been trained offline, to the power grid dispatch and control center. Through the online application phase, monitor the power grid status in real time. After detecting a large disturbance fault, output the optimal load shedding control action through the model to complete the emergency control of transient voltage stability of the power system.

[0097] Specifically, in the online application phase, the fully trained agent is connected to the power grid control center. Through a wide-area measurement system (WAMS), it monitors the power grid status in real time. When a major disturbance is detected, the agent is activated, inputting real-time response information after the fault is cleared. The agent then outputs emergency control actions to restore the system to stability. The main steps of the online application phase are as follows: L1: Agent Deployment The SAC-GAIL intelligent agent model, which has been fully trained and stored offline, is deployed to the power grid dispatch and control center to connect it with the wide-area measurement system.

[0098] L2: Real-time status monitoring The intelligent agent continuously monitors the real-time operation data of the power grid through a wide-area measurement system.

[0099] L3: High Interference Fault Detection The system continuously monitors for major disturbances such as short circuits. If no fault is detected, monitoring continues; if a fault is detected, the deployed SAC-GAIL agent is activated.

[0100] L4: Real-time perception and decision-making The activated agent acquires real-time power grid status information after the fault is cleared and calculates the optimal emergency control measures within milliseconds.

[0101] L5: Control Execution and System Recovery The control commands output by the intelligent agent are sent to actuators such as circuit breakers to carry out load shedding operations, ultimately guiding the grid voltage to return to stability.

[0102] Because the offline training phase requires extensive simulations, it is time-consuming. However, once the agent is fully trained, the online application phase only requires converting the measurement information into the state variables needed by the agent to quickly output real-time control measures. The emergency control framework for power grid large-disturbance transient voltage stability proposed in this invention is as follows: Figure 3 As shown.

[0103] Example Simulation analysis was performed using the new power system standard example (CSEE-VS) published by the Chinese Society for Electrical Engineering. Its main grid structure is as follows: Figure 4 As shown in the figure, this example is a DC receiving-end system with a high proportion of renewable energy. The system uses a 500kV main grid with a total of 66 nodes and a base capacity of 100MVA. The total active load is 4905MW, and the reactive load is 1647Mvar. The load model is set to 30% induction motors and 70% constant impedance loads. All loads are connected to the 220kV bus and then to the 500kV bus via a three-winding transformer. The installed capacities of renewable energy units and conventional units are 3600MW and 2640MW, respectively. The system contains one DC line with a rated input power of 1600MW. This example shows that renewable energy accounts for more than 50% of the output, comprehensively reflecting the instability characteristics such as voltage collapse and sustained low voltage under electromechanical transient scales.

[0104] Taking into account factors such as load level and renewable energy penetration rate, multiple power grid operation modes are simulated. System load levels are randomly initialized to 95%–105%, induction motor ratio to 30%–50%, renewable energy penetration rate to 40%–60%, and DC system transmission power to 80%–100%, generating various basic operation modes. Based on these operation modes, multiple fault scenarios are simulated. At t=1s, three-phase permanent short-circuit faults are randomly set at 1%–99% of lines B03-B05, B07-B10, or B08-B11, and the faults are cleared 0.05s–0.2s after the disturbance to fully simulate transient voltage instability. Each 500kV bus, some 220kV buses, and conventional generating units are selected as observation points, and measurement data 0.2s after fault clearing is used to construct the state space. Buses B1-B12 are all loaded, denoted as L1-L12. As an emergency load shedding control measure, the maximum controllable quantities at each load shedding point are shown in Table 1. During agent training, random initialization of operating modes and fault scenarios enhances sample diversity, thereby improving the agent's adaptability to the environment and strengthening its generalization ability. In each round of interactive training, the simulation termination time is set to 5 seconds after fault clearance.

[0105] Table 1 Maximum controllable quantities at each load shedding point

[0106] This invention utilizes a multilayer perceptron to construct a neural network. The structural settings of each neural network are shown in Table 2. During training, the parameters of the Actor-Critic network and the discriminator network are updated in each round. The hyperparameters of the proposed SAC-GAIL algorithm are shown in Table 3. The hardware platform used for simulation and verification is: CPU Intel(R) Core(TM) i7-14700KF @3.40 ​​GHz, RAM 64.0 GB, GPU NVIDIA GeForce RTX 4090 @55.9 GB. The training code for the proposed SAC-GAIL algorithm is written in Python. The power system transient simulation simulator uses BPA, and the simulation step size is set to 0.01s.

[0107] Table 2. Neural network structures for the SAC-GAIL algorithm

[0108] Table 3 Hyperparameters of the SAC-GAIL algorithm

[0109] Based on the above settings, the agent is trained using the SAC-GAIL algorithm. First, 50 typical scenarios are constructed for the selected operating modes and faults. A genetic algorithm (GA) is used to optimize the control in each scenario, obtaining the optimal control measures for each scenario as a pre-planned control table, generating 50 state-action pairs to form an expert dataset. Then, these expert demonstrations are used to pre-train the Actor network using BC (Browser-Guided Interpreter) algorithms.

[0110] In the main training process of SAC-GAIL, the reward fusion coefficient is... λ It decays linearly to 0 as training progresses. Figure 5 (a) shows the training curves for different rewards, which were processed with a moving average over 100 rounds. Experimental results show that after approximately 3000 rounds of training, the reward curves exhibit a stable convergence trend. The average environmental reward value eventually stabilizes around 50, indicating that the agent has successfully learned an effective control policy. R gail As training progresses, the convergence gradually approaches zero, reflecting a shift from expert behavior imitation to environment-feedback-driven optimization. Accordingly, Figure 5 As shown in (b), the discriminator's probability of distinguishing between expert samples and agent-generated samples gradually approaches 0.5, indicating that the discriminator can no longer easily distinguish between the two types of sample sources. These results suggest that the agent's strategy has reached an expert level, and the adversarial training has reached a stable equilibrium.

[0111] To verify the effectiveness of the proposed SAC-GAIL algorithm in voltage emergency control tasks, two mainstream deep reinforcement learning algorithms, SAC and PPO, were selected as baselines for comparison. To ensure the reliability of the results, each algorithm was trained independently five times under the same simulation environment, with different random seeds used for initialization each time. Figure 6 This diagram illustrates the changes in environmental rewards for each algorithm during training. The numerical sequences have all undergone a moving average process with a window size of 100 rounds. The solid line represents the average trend of reward values ​​under different random seeds, while the shaded area represents the fluctuation range of the reward, visually reflecting the stability of the algorithm training.

[0112] Analysis of the training curves reveals that, in terms of convergence speed, the SAC-GAIL algorithm reaches a stable state after approximately 3000 training rounds. In contrast, the SAC and PPO algorithms require approximately 5700 and 5500 rounds, respectively, to reach a similar stable state. This indicates that the GAIL mechanism, by introducing expert demonstration data, effectively guides the agent's exploration process, significantly improving training efficiency. Regarding convergence reward values, the average environmental rewards for the three algorithms stabilize at around 51, 47, and 38, respectively, in the later stages of training. In terms of fluctuation, SAC-GAIL exhibits the narrowest reward distribution range, indicating the most consistent and stable policy performance under different random seeds. Higher final rewards and smaller fluctuations suggest that SAC-GAIL has learned a policy with better control and more robust performance. In summary, SAC-GAIL significantly outperforms the comparative algorithms in training efficiency, final performance, and robustness, fully demonstrating the effectiveness of the fusion framework.

[0113] After training, three typical fault scenarios from the training set were selected to verify the effectiveness of the learned control strategy, and the voltage response of the key bus in the system was obtained as follows: Figure 7 As shown in Table 4, the corresponding emergency control strategies are as follows. The results indicate that after the agent applies emergency control measures, the system voltage quickly recovers to the allowable range and remains stable, verifying the effectiveness of the control strategies learned by the agent.

[0114] Table 4 Emergency Control Strategies for Typical Fault Scenarios

[0115] Testing was conducted on a contingency-based emergency control mismatch scenario to further verify the adaptability of the proposed control strategy under extreme conditions. The scale of renewable energy generation and the level of DC power supply are key factors affecting the voltage stability of the receiving-end power system. Therefore, for the typical fault scenarios already listed in the contingency plan, the penetration level of renewable energy generation was increased to 70%, and the DC system transmission power was increased to 1800MW. A three-phase permanent short circuit was set on one of the double-circuit lines B03-B05, with the fault lasting 0.1s before clearing. Subsequently, the contingency-based emergency control strategy was implemented, and the voltage response of the key bus in the system before and after control was as follows: Figure 8 As shown in Table 5, the corresponding control strategies are as follows. The results indicate that under the modified operating conditions, pre-planned control cannot restore the system to stability.

[0116] For this contingency-based mismatch scenario, a trained agent is applied to apply emergency control to obtain the system's critical bus voltage response, such as... Figure 9 As shown in Table 5, the corresponding control strategies are also presented. The results demonstrate that the proposed response-driven SAC-GAIL model can effectively solve the pre-planned control mismatch problem and enhance the system's second line of defense.

[0117] Overall, the proposed method can balance accuracy and efficiency when dealing with emergency control problems of voltage stability in power systems. It demonstrates good comprehensive performance while achieving efficient decision-making and can be fully adapted to the response-driven control mode of "real-time decision-making and real-time control", providing reliable support for the safe and stable operation of power systems with a high proportion of new energy access.

[0118] Table 5. Emergency Control Strategies for Contingency Plans and SAC-GAIL under Extreme Operating Conditions

[0119] This embodiment constructs a collaborative training mechanism combining maximum entropy optimization using SAC and expert imitation using GAIL. In this framework, SAC serves as the basic optimizer, ensuring the exploratory nature and convergence stability of the policy; GAIL acts as an expert knowledge injection pathway, guiding the agent's behavior to approximate the expert policy distribution. This framework effectively balances the relationship between experience inheritance and autonomous optimization. This embodiment designs a hybrid reward mechanism and a dual-buffer architecture to synergistically improve training efficiency and policy quality. The hybrid reward mechanism dynamically integrates GAIL imitation rewards and environment rewards, balancing imitation learning and autonomous optimization objectives through adjustable weight coefficients. The dual-buffer structure rapidly captures policy updates through a recent interaction buffer and reuses historical data through an experience replay buffer, jointly ensuring the accuracy of the discrimination signal and training stability. This embodiment introduces behavioral cloning pre-training, using expert data to safely initialize the policy network, reducing initial exploration risks and accelerating convergence. Simultaneously, dynamic expert augmentation is employed, online screening of high-reward samples to enrich the expert dataset, enabling the discriminator to continuously learn and improve policies, effectively alleviating expert data scarcity and enhancing model generalization ability and control performance.

[0120] In summary, this invention provides a transient voltage stabilization control method that integrates expert experience and agent exploration. It constructs a collaborative training mechanism combining SAC maximum entropy optimization and GAIL expert imitation. SAC serves as the basic optimizer, ensuring the exploratory nature and convergence stability of the policy. GAIL acts as an expert knowledge injection pathway, guiding the agent's behavior to approximate the expert policy distribution, effectively balancing experience inheritance and autonomous optimization. Simultaneously, a hybrid reward mechanism and a dual-buffer architecture are designed. The hybrid reward mechanism dynamically integrates GAIL imitation rewards and environmental rewards, balancing imitation learning and autonomous optimization objectives through adjustable weight coefficients. The dual-buffer structure utilizes a recent interaction buffer to quickly capture policy updates and combines it with an experience replay buffer to reuse historical data, jointly ensuring the accuracy of the discrimination signal and training stability. Furthermore, behavioral cloning pre-training is introduced, using expert data to safely initialize the policy network, reducing initial exploration risks and accelerating convergence. A dynamic expert expansion mechanism is also employed, online screening of high-reward samples to enrich the expert dataset, enabling the discriminator to continuously learn and improve the policy, effectively alleviating expert data scarcity, enhancing model generalization ability and control performance, and improving the reliability and efficiency of transient voltage stabilization emergency control.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A transient voltage stabilization control method integrating expert experience and agent exploration, characterized in that, include: A mathematical model for emergency control of transient voltage stability under large disturbances was established, the control objectives, dynamic constraints, static constraints and inequality constraints were clarified, and the emergency control measures with load shedding as the core were determined. The core emergency control measures are modeled as Markov decision processes, and the state space, action space and reward function of the Markov decision process are designed. The reward function comprehensively reflects the system's transient stability requirements and control economy objectives. An emergency control model based on the SAC-GAIL fusion framework is constructed using Markov decision process. The SAC algorithm is improved to adapt to single-step decision-making. The GAIL framework is introduced to realize the fusion of expert experience. Behavioral cloning pre-training and dynamic expert expansion mechanism are used to optimize model training. The emergency control model of the SAC-GAIL fusion framework is trained offline to obtain and save the optimal control strategy; The emergency control model of the SAC-GAIL fusion framework, which was trained offline, was deployed to the power grid dispatch and control center. The power grid status was monitored in real time during the online application phase. After a major disturbance fault was detected, the model output the optimal load shedding control action to complete the emergency control of transient voltage stability of the power system.

2. The transient voltage stabilization control method integrating expert experience and agent exploration according to claim 1, characterized in that, The mathematical model for emergency control of large-disturbance transient voltage stability takes minimizing the total control cost as its objective function, and the objective function expression is as follows: Where w represents the cost vector of emergency control measures, and a represents the action vector of emergency control measures; The dynamic constraint expression is as follows: Where x represents the state variable describing the dynamic characteristics of the system, y represents the algebraic variable of the system, and d represents the external disturbance. This constraint includes the synchronous generator rotor motion equation, the excitation system equation, and the equipment dynamic characteristic equation. The static constraint expression is as follows: 0 = g(x, y, a, d) The static constraints include power network equations, power flow equations, and equipment static characteristic equations. The inequality constraint expression is as follows: 0 <h(x,y,a,d) The inequality constraints include system stability constraints and emergency control measure control quantity constraints.

3. The transient voltage stabilization control method integrating expert experience and agent exploration according to claim 1, characterized in that, The system stability constraints include transient voltage stability constraints and power angle stability constraints; The transient voltage stability constraints are as follows: in, express t Time Node i The voltage amplitude; Indicates the fault clearing time; Indicates the number of nodes; The power angle stability constraint is as follows: in, express t Time of the first i The power angle of the generator; This indicates the number of generators.

4. The transient voltage stabilization control method integrating expert experience and agent exploration according to claim 1, characterized in that, The state-space expression is as follows: in, and These represent the voltage magnitude vector and phase angle vector at each node of the system, respectively. and These represent the generator power angle vector and the rotor angular velocity vector, respectively. The state vector is composed of system measurement data within a 0.2-second time window after the fault is cleared. The action space is as follows: in, Indicates the first i The load shedding amount at each load node, with the action value being a continuous variable. This represents the maximum amount of material that can be removed from this node. Indicates the number of load nodes that can be shelved; The reward function is as follows: The voltage stability reward expression is as follows: The voltage-steady final-state reward expression is as follows: The expression for the power angle stability reward is as follows: The expression for the cost control reward is as follows: Where R vol For voltage stability bonus, R angle For the purpose of improving the stability of the angle, R load To control costs, incentives are offered; terminal This is a reward for the final state of voltage stability; T This refers to the duration of the transient simulation. c 1. c 2. and c 3 represents the weighting coefficients for undervoltage penalty, overvoltage penalty, and control cost, respectively.

5. The transient voltage stabilization control method integrating expert experience and agent exploration according to claim 1, characterized in that, The improvement process of the SAC algorithm includes setting the discount factor γ to 0 and calculating the Q value through the Critic network, as shown in the following formula: in, R hybrid It is a hybrid reward, obtained by weighted fusion of environmental rewards and GAIL imitation rewards; The update process of the SAC algorithm includes Critic network update, Actor network update, and adaptive adjustment of entropy coefficient. The Actor network solves the gradient backpropagation problem through reparameterization sampling, and the entropy coefficient α is automatically adjusted through an optimization problem with entropy constraints.

6. The transient voltage stabilization control method integrating expert experience and agent exploration according to claim 1, characterized in that, The GAIL framework uses a discriminator to perform adversarial training against the SAC Actor network. The discriminator uses a double-buffer structure to construct training data and is trained using a binary cross-entropy loss function to distinguish between expert data and agent-generated data. The hybrid reward expression is as follows: in, λ represents the GAIL reward, k is the scaling factor, and λ is the fusion coefficient, which gradually decreases from an initial high value to 0 as training progresses.

7. The transient voltage stabilization control method integrating expert experience and agent exploration according to claim 1, characterized in that, The behavior clone pre-training utilizes an expert dataset to optimize the Actor network by minimizing the mean square error between the agent's actions and the expert's actions; The dynamic expert expansion mechanism adds state-action pairs whose environmental reward R(s,a) exceeds a predetermined threshold ξ to the expert dataset after each round of interaction, thereby expanding the expert data.

8. The transient voltage stabilization control method integrating expert experience and agent exploration according to claim 1, characterized in that, The offline training includes the following process: S1, initialize the GAIL discriminator network, SAC Actor-Critic network parameters and experience replay buffer, and load the steady-state and transient simulation files of the power system simulation environment; S2, use the expert dataset to perform BC pre-training on the Actor network of SAC to obtain the initial security policy; S3 configures the initial operation mode of the power grid and large disturbance fault scenarios. The intelligent agent interacts with the simulation environment and collects status, action and reward data. S4. Select high-performance interactive samples to expand the expert dataset, and train the GAIL discriminator network and the SAC Actor-Critic network alternately. S5 determines whether the model performance has converged or reached the preset maximum number of training iterations. If not, it returns to S3 to reconfigure the scene and continue training. If it does, it saves the optimal control strategy.

9. The transient voltage stabilization control method integrating expert experience and agent exploration according to claim 1, characterized in that, The specific process of the online application phase includes: L1 deploys the optimal control strategy model, which has been trained and saved offline, to the power grid dispatch and control center and establishes a communication connection with the wide-area measurement system. L2 continuously monitors the power grid's operating status information in real time through the wide-area measurement system; L3 determines in real time whether a major disturbance fault has occurred in the power grid; if no fault has occurred, it continues to monitor; if a fault is detected, it activates the deployed control strategy model. L4 acquires real-time grid status information after fault clearing, and the control strategy model calculates and outputs the optimal emergency control action command within milliseconds. L5 sends the control action command to the power grid actuator to perform load shedding operations in order to quickly restore and maintain the transient voltage stability of the power system.

10. The transient voltage stabilization control method integrating expert experience and agent exploration according to claim 9, characterized in that, The control strategy model is a SAC-GAIL fusion framework model obtained through BC pre-training, dynamic expert expansion, and SAC and GAIL co-optimization training. Its input is the system measurement data within a 0.2-second time window after fault clearance, including the voltage amplitude, phase angle, generator power angle, and rotor angular velocity of each node. The emergency control action command is the load shedding amount of each load shedding node, and the actuator is a circuit breaker. By quickly cutting off the power deficit of the specified load node, the power flow pressure of the transmission section is relieved.

Images