Quick-closing valve control method for transient stability of power system based on deep reinforcement learning

Abstract

The invention belongs to the technical field of electric power systems, and particularly relates to a quick-closing valve control method for transient stability of an electric power system based on deep reinforcement learning. Comprising the following steps: establishing a Markov decision model based on power system transient stability emergency control; constructing a quick-closing valve decision framework based on a near-end strategy optimization algorithm; constructing a deep reinforcement learning model for emergency quick-closing valve control; by introducing the deep reinforcement learning method, a more accurate and efficient quick-closing valve control strategy can be formulated, so that when the power system breaks down, the stability of the system is recovered more quickly, and the loss caused by instability is reduced.

Description

Technical Field

[0001] This invention belongs to the field of power system technology, specifically relating to a fast-closing valve control method for transient stability of power systems based on deep reinforcement learning. Background Technology

[0002] With the continuous development and maturation of power grid technology, the protection and control strategies of power systems are also evolving. Traditional generator tripping and load shedding, as emergency measures, can indeed maintain system stability in certain situations, but their long reconnection times may lead to reduced power system efficiency or increased operating costs. Meanwhile, although modern power grids are actively introducing and deploying renewable energy on a large scale to meet the needs of sustainable development and environmental protection, the role of steam turbines in the power system remains indispensable. As traditional large-scale power generation equipment, steam turbines provide stable baseload power and necessary system inertia to the power grid, which is crucial for maintaining the transient stability of the grid and responding to emergencies. Therefore, to ensure the stable operation of the power grid more efficiently and economically, fast-closing steam valves, as an effective control strategy, can quickly adjust the output of steam turbines to respond to the real-time demand and changes in the power grid. This can reduce unnecessary unit downtime, reduce the impact on units, avoid the impact of excessive generator tripping on system stability, and improve the flexibility and resilience of the power system.

[0003] Setting the optimal control parameters for the fast-closing steam valve is a crucial step in the fast-closing control process. It determines the speed and depth of turbine unloading and affects the power angle oscillation amplitude and convergence time. Traditionally, fast-closing steam valve control relies on several preset parameters provided by the turbine manufacturer, without considering various uncertainties and variations in actual operation. This can lead to under- or over-adjustment due to parameter mismatch, threatening the stability of the power system.

[0004] Application content

[0005] To address the above problems, this invention proposes a fast steam valve closing control method for transient stability of power systems based on deep reinforcement learning, comprising the following steps:

[0006] Step 1: Establish a Markov decision model based on the transient stability emergency control of the power system; the power system stability control process is analogous to the Markov decision process, which consists of a 5-tuple including S, A, P, R, γ, where S is the state space, A is the action space, P is the transition probability, R is the reward function, and γ is the reward discount factor used to calculate the cumulative reward.

[0007] Step 2: Construct a fast valve closing decision framework based on a near-end policy optimization algorithm. The decision framework includes a power system environment and a fast valve closing decision agent. The power system environment is used to simulate a real power system to reflect the complexity and uncertainty of the actual power system. The fast valve closing decision agent is a deep reinforcement learning agent with an Actor-Critic architecture, composed of an Actor neural network and a Critic neural network. The power system environment is built based on the PSCAD and EMTDC simulation platforms, and data interaction with the fast valve closing decision agent is realized through an interface module.

[0008] Step 3: Construct a deep reinforcement learning model for emergency fast-closing valve control; define the state space, action space, and reward function of the deep reinforcement learning model. The setting of the state space, action space, and reward function is used to ensure the training efficiency, convergence speed, control accuracy, and stability of the model; wherein, the observation data of the state space is obtained through the PMU synchronous phasor measurement device in the power system environment.

[0009] Step 4: Based on the fast valve closing decision framework from Step 2 and the deep reinforcement learning model from Step 3, train the Actor network and the Critic network; during training, an experience playback mechanism is used to store the experience data generated by the agent's interaction with the environment, including s t ,a t ,r t ,s t+1 The network parameters are updated based on empirical data from batch sampling, and the synchronization update cycle between the target network and the current network is set to improve training stability.

[0010] Step 5: Extract the real-time electrical quantities of the power system as input to the trained strategy model. The strategy model outputs the fast-closing steam valve control strategy command and sends the command to the power system for execution. During the execution process, a control strategy effectiveness verification module is set up. If the verification finds that the system stability does not meet the standard after the strategy is executed, the model retraining process is triggered.

[0011] Preferably, in step 1, the interaction between the agent and the environment during the Markov decision-making process is described by a set of time series: at time t, the environment is in state s. t The intelligent agent provides a control strategy a based on the environmental state. t After the environment accepts the stabilization strategy given by the agent, its state transitions to s. t+1 At the same time, a reward signal r is output to the agent. t+1 The transition probability P is defined as P(s) t+1 |s t ,a t ), indicating that in state s t Next, execute action at Then transition to state s t+1 The probability of.

[0012] Preferably, in step 2, the power system environment simulation includes load level, generator output, new energy grid connection status, and fault events; the fault events include line short-circuit faults and transformer faults, and the type, location, and duration of the simulated faults can be flexibly configured.

[0013] Preferably, in step 2, the Actor neural network is constructed from a fully connected layer neural network. It takes the environmental observation state as input, fits the probability distribution characteristics of the output action through nonlinear mapping of the hidden layer, and then outputs the corresponding action probability vector through the output layer neural network. The hidden layer of the Actor neural network uses the ReLU activation function, and the output layer uses the Softmax activation function. The strategy output by the Actor neural network is represented as follows:

[0014]

[0015] Where π(a|s) is the probability of taking action a given state s; f(a,s;θ) is the network's output score for state s and action a, with the parameter θ; a' represents all possible actions; the Actor network ultimately selects the action with the highest probability as the output.

[0016] Preferably, in step 2, the Critic neural network is constructed from a fully connected layer neural network, taking the environmental observation state as input and outputting a state value estimate V(s). t ), combining the actual immediate reward r and the value estimate V(s) of the next state. t+1 Obtain the advantage function A(s) t ,a t The feedback is then fed back to the Actor network to guide it in selecting a better action; the hidden layers of the Critic neural network use the ReLU activation function, and the output layer uses a linear activation function; the advantage function is expressed as:

[0017] A(s t ,a t )≈r+γV(s t+1 )-V(s t )

[0018] Where r is in state s t Take action a t The immediate reward obtained, γ is the discount factor for future rewards, V(s) t+1 ) is the next state s t+1 Value estimate.

[0019] Preferably, the proximal policy optimization algorithm introduces a pruning mechanism to simplify the calculation process and maintain the stability of policy updates by controlling the change in the ratio of new to old policies; the objective function of the proximal policy optimization algorithm is expressed as:

[0020]

[0021] Where r(θ)=π(a|s) / π(a|s) is the probability ratio between the old and new policies, the clip function is the clipping function, and ε is a positive number used to define the clipping threshold, controlling the change between the old and new policies within [1-ε,1+ε]. The near-end policy optimization algorithm also adopts an adaptive learning rate adjustment mechanism, which dynamically adjusts the learning rate according to the change of the loss function during the training process.

[0022] Preferably, in step 3, the observed state S in the state space includes the relative power angle matrix δ of the generator and the generator terminal energy W within the corresponding time [T,T)∪[T,T]∪…∪[T,T] after the fault and after stabilization; the observed state S is expressed as:

[0023] S k =[δ k W k ]

[0024]

[0025] Where δ represents the relative power angle of the lower synchronizing machine n at time t in the k-th step; W represents the terminal energy of the lower synchronizing machine n at time t in the k-th step; the relative power angle is calculated with reference to the power angle of a certain reference generator in the system.

[0026] Preferably, in step 3, the fast-closing valve decision quantity in the action space includes selecting the target unit for fast-closing control and the fast-closing valve control parameters, wherein the fast-closing valve control parameters only consider the cylinder stall time T; the Actor network outputs action a, which corresponds to decision b at the environment receiving layer through action-decision mapping, wherein:

[0027]

[0028] b represents the i-th unit selected for fast shutdown control in step k, with a total of i units set. b represents the fast shutdown parameter of the unit participating in control in step k. The value range of the cylinder stagnation time T is 0.5s to 3s, and it is divided into multiple selectable levels through discretization.

[0029] Preferably, in step 3, the reward function comprises two parts: the power system transient stability index (TSI) and the energy reward function, specifically expressed as follows:

[0030]

[0031] Where p is the penalty value for instability after stabilization, ranging from 100 to 500; λ is the reward coefficient for the decision effect of fast valve closing, ranging from 0.6 to 0.8; λ is the energy function penalty coefficient, ranging from 0.2 to 0.4; ΔW is the absolute value of the difference between the sum of terminal energies W(S) and the set energy reference value W; when TSI>1 / 3, the system is judged to be stable, otherwise it is judged to be unstable; Δδ is the maximum power angle difference of the generator; P and Q are the active power and reactive power generated by the i-th generator, respectively; U and θ are the voltage amplitude and phase angle of the i-th terminal bus, respectively; the energy reference value W is determined according to the average terminal energy under normal operating conditions of the power system. Preferably, in step 4, the neuron structure of both the Critic and Actor networks is set to [32,32] during training, the total number of training rounds of the neural network is set to 2000, the learning rate is 0.0061459, the advantage estimation parameter Lambda is 0.9, the discount factor gamma is 0.99, and the clipping parameter clip range is 0.3.

[0032] Preferably, in step 5, the real-time decision-making electrical quantities include generator terminal voltage, current, active power, reactive power, and generator power angle, all of which are collected in real time by a PMU synchronous phasor measurement device.

[0033] Beneficial effects

[0034] Strong real-time control decision-making: The PMU synchronous phasor measurement device collects state data at high frequency and combines it with the trained deep reinforcement learning model to quickly output the fast-closing steam valve control strategy. No complex iterative calculation is required. It can quickly respond to power system transient faults, effectively shorten the fault handling time, and improve the system transient stability margin.

[0035] Excellent adaptability and robustness: The power system environment built based on PSCAD and EMTDC can simulate various load levels, new energy grid connection scenarios and fault types. The trained model can adapt to the complexity and uncertainty of the power system and output the optimal control strategy for different fault scenarios, avoiding the problem of poor adaptability of traditional fixed control strategies.

[0036] High control precision and stability: Through the alternating iterative training mechanism of the Actor-Critic dual network, combined with optimization methods such as experience playback and synchronous update of the target network, the model training stability and convergence accuracy are improved, and the output fast-closing valve control parameters are more accurate, which can effectively control the energy fluctuation at the generator end, reduce the system power angle difference, and ensure the stable operation of the system.

[0037] Improved ease of operation and maintenance: The system includes a control strategy effectiveness verification module and a model retraining trigger mechanism, which can automatically adapt to changes in the operating status of the power system without the need for frequent manual adjustments to control parameters, thus reducing operation and maintenance costs. At the same time, the reward function incorporates transient stability indicators and energy constraints to achieve a balance between control effectiveness and operational economy.

[0038] High compatibility: Through the interface module, it realizes efficient data interaction between deep reinforcement learning agents and existing power system simulation platforms and PMU measurement systems. It can be seamlessly integrated into the existing power system stability control system without large-scale modification of the existing system, and has high engineering application value. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0040] Figure 1 This is a diagram of the Markov decision process based on the emergency control of power system transient stability in this invention;

[0041] Figure 2 This is a diagram of the fast-closing valve decision framework constructed based on the near-end strategy optimization algorithm in this invention;

[0042] Figure 3 This is a 3-machine, 9-node computational example diagram used in the demonstration.

[0043] Figure 4 This is an average reward graph obtained from an example demonstration;

[0044] Figure 5 This is a power angle curve of a generator without a stabilization strategy, obtained from an example demonstration.

[0045] Figure 6 This is a generator power angle curve obtained from an example demonstration using a stabilization strategy. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the present invention is not limited to the following embodiments, and specific implementation methods can be determined according to the technical solutions of the present invention and the actual situation. In order to avoid obscuring the essence of the present invention, well-known methods, processes, flows, components, and circuits are not described in detail.

[0047] The fast steam valve closing control method for transient stability of power systems based on deep reinforcement learning proposed in this invention includes:

[0048] 1) Markov decision processes can be used to model reinforcement learning problems, establishing Markov decision models based on transient stability emergency control of power systems, such as... Figure 1 As shown.

[0049] In Markov decision-making processes, the interaction between the agent and the environment can be described by a set of time series: at time t, the environment is in state s. t The intelligent agent provides a control strategy a based on the environmental state. t After the environment accepts the stabilization strategy given by the agent, its state transitions to s. t+1 As a feedback environment, it outputs a reward signal r to the agent. t+1 .

[0050] 2) Construct a fast-closing valve decision framework based on a proximal policy optimization algorithm, such as... Figure 2 As shown. The upper part of the framework is the power system environment that requires the decision to close the steam valve quickly. It simulates the real power system, including various factors such as load level, generator output, new energy grid connection status, and fault events, aiming to reflect the complexity and uncertainty of the actual power system as realistically as possible. The lower part of the framework is a deep reinforcement learning agent with an Actor-Critic architecture. The Actor network is constructed by a fully connected layer neural network. It takes the environmental observation state as input, fits the probability distribution characteristics of the output action through the nonlinear mapping of the hidden layer, and then outputs the corresponding action probability vector by the output layer neural network. The policy output by the Actor network can be represented by Equation (1).

[0051]

[0052] Where, π θ (a|s) represents the probability of taking action a given state s; f θ (a,s; θ) represents the network's output score for state s and action a, with parameters denoted by θ; a′ represents all possible actions. The Actor network ultimately selects the action with the highest probability as the output.

[0053] Similarly, the Critic network is constructed from fully connected layer neural networks, taking the environmental observation state as input and outputting a state value estimate V(s). t ), combining the actual immediate reward r and the value estimate V(s) of the next state. t+1 Obtain the advantage function A(s) t ,a t This feedback is then sent to the Actor network to guide it in selecting better actions.

[0054] A(s t ,at )≈r+γV(s t+1 )-V(s t (2)

[0055] Where r is in state s t Take action a t The immediate reward obtained, γ is the discount factor for future rewards, V(s) t+1 ) is the next state s t+1 Value estimate.

[0056] 3) The deep reinforcement learning algorithm used is the proximal policy optimization algorithm. It introduces a pruning mechanism to simplify the calculation process and maintain the stability of policy updates by controlling the change range of the ratio of new and old policies.

[0057]

[0058] in, It represents the probability ratio between the old and new strategies. The clip function is the clipping function, and ε is a positive number used to define the clipping threshold. The change between the old and new strategies is controlled within [1-ε, 1+ε], which ensures a certain level of policy update safety while reducing the implementation difficulty and computational cost, and improving the practicality and universality of the algorithm.

[0059] 4) In modeling the fast-closing valve control problem as a deep reinforcement learning problem, the settings of the agent's state space, action space, and reward function directly determine the performance of the deep reinforcement learning model. The design of elements such as state, reward, and action is introduced below.

[0060] State space:

[0061] The agent's input observation state S k It should include key information on the stability of the reaction power angle and the decision to close the steam valve quickly. This article selects the corresponding time [T] after the fault and after the stabilization. a ,T fv1 )∪[T fv1 ,T o1 ]∪…∪[T fvk ,T ok The relative power angle matrix δ of the generator within ] k and generator port energy W k The observed state S is the object of observation for the agent. k It is represented by equation (4).

[0062] S k =[δ k W k (4)

[0063]

[0064]

[0065] in This represents the relative power angle of the synchrotron n at time t in the k-th step. Let represent the port energy of the synchronizer n at time t in the k-th step.

[0066] Action space:

[0067] The decision-making process for fast-closing steam valves includes selecting the target generator for fast-closing control and the control parameters. The agent is trained to adjust the participating generators and parameters to determine the optimal solution through interaction with the grid environment. This is because fast-closing steam valve control has transient characteristics—the fast-closing steam valve speed T... A The main factor affecting the initial swing angle is the speed of the first swing, and the faster the initial swing, the more stable the swing. The cylinder stagnation time T... B The main factor affecting the power angle is the backswing; therefore, the fast-closing parameters in the motion space only consider the cylinder slump time T. B The Actor network outputs action a. k Decision b is mapped to action-decision mapping at the environment interface layer. k .

[0068]

[0069] in This indicates that the i-th generating unit selected in step k participates in the fast shutdown control, with a total of ig generating units available for fast shutdown control. This represents the fast-shutdown parameters of the unit involved in the control at step k.

[0070] Reward function:

[0071] The design reward function consists of two parts: the power system transient stability index (TSI) and the energy reward function, as shown in equation (9).

[0072]

[0073] ΔW=|W(S t )-W ref | (10)

[0074]

[0075]

[0076] Where p is the penalty value for instability after stabilization, and λ TSI λ is the reward coefficient for the decision to quickly close the valve. W Let ΔW be the energy function penalty coefficient, and ΔW be the sum of the port energies W(S). t) and the set energy reference value W ref The absolute value of the difference; when TSI > 1 / 3, the system is considered stable; otherwise, it is considered unstable; Δδ max P represents the maximum power angle difference of the generator. i and Q i U represents the active power and reactive power generated by the i-th generator, respectively. i and θ i These represent the voltage amplitude and phase angle of the terminal bus i, respectively. The energy reference value W is determined based on the average terminal energy under normal operating conditions of the power system.

[0077] During training, an experience playback mechanism is used to store the experiential data generated by the agent's interaction with the environment, including s t ,a t ,r t ,s t+1 The network parameters are updated based on the empirical data from batch sampling. At the same time, the synchronization update cycle between the target network and the current network is set to improve training stability. During training, the neuron structure of the Critic and Actor networks is set to [32,32]. The total number of training rounds of the neural network is set to 2000, the learning rate is 0.0061459, the advantage estimation parameter Lambda is 0.9, the discount factor gamma is 0.99, and the clipping parameter cliprange is 0.3.

[0078] In step 5, the real-time decision-making electrical quantities include generator terminal voltage, current, active power, reactive power, and generator power angle, all of which are collected in real time by the PMU synchronous phasor measurement device.

[0079] Example Demonstration

[0080] To demonstrate the effectiveness of the application, a framework was constructed as follows: Figure 3 The 3-machine, 9-node system shown has the following settings:

[0081] 1) Specific settings for expected fault conditions: A three-phase short circuit occurs at 50% of line 8-9 and lasts for 0.261s before the fault is cleared; a three-phase short circuit occurs at 50% of line 5-7 and lasts for 0.332s before the fault is cleared.

[0082] 2) Units equipped with quick-closing steam valve function: G2, G3.

[0083] 3) Agent parameter settings: The neuron structure of both the Critic and Actor networks is [32, 32]. The total number of training rounds of the agent's neural network is set to 2000, the learning rate is 0.0061459, the advantage estimation parameter (Lambda) is 0.9, the discount factor (gamma) is 0.99, and the clip range parameter (clip range) is 0.3.

[0084] Figure 4 The graph shows the average reward curve obtained by the agent during the training phase. In the early stages of training (the first 250 training rounds), the reward value of the agent remained at a low level. However, as the number of training rounds increased and the agent continued to try and fail, the reward value eventually stabilized, indicating that the agent had found a suitable fast valve closing control strategy at this point.

[0085] The trained agent, for the operating condition (a three-phase short circuit occurs at 50% of line 8-9, lasting 0.261s, and the fault is cleared), proposes a control strategy: selecting unit G2 for fast valve closing control (TA = 0.3s, TB = 1s). To demonstrate the effectiveness and rationality of the agent's control strategy, [further details are needed]. Figure 5 It can be seen that when the quick-closing valve control is not adopted, after the three-phase short-circuit fault is cleared, the generator's relative power angle gradually increases over time until it loses synchronism. Through... Figure 6 It can be seen that after adopting the quick-closing steam valve control, the power angle of each unit fluctuates and eventually returns to a stable state.

[0086] Those skilled in the art should understand that the above embodiments are merely illustrative of the content of this disclosure and do not limit its scope. The system capacity, voltage, line parameters, etc., shown may vary depending on the specific circumstances of the power electronic grid-connected generator set and its grid connection. Based on this disclosure, those skilled in the art can make other changes or adjustments, and these changes still fall within the scope of this disclosure.

Claims

1. A fast steam valve closing control method for transient stability of a power system based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Establish a Markov decision model based on emergency control of power system transient stability; Step 2: Construct a fast valve closing decision framework based on a near-end policy optimization algorithm. The decision framework includes the power system environment and the fast valve closing decision agent. Step 3: Construct a deep reinforcement learning model for emergency fast-closing valve control to define the state space, action space, and reward function of the deep reinforcement learning model. The observation data of the state space is acquired by the PMU synchronization phasor measurement device in the power system environment. Step 4: Based on the fast valve closing decision framework in Step 2 and the deep reinforcement learning model in Step 3, train the Actor network and the Critic network. Step 5: Extract the real-time electrical quantities of the power system as input to the trained strategy model. The strategy model outputs the fast-closing steam valve control strategy command and sends the command to the power system for execution. During the execution process, a control strategy effectiveness verification module is set up. If the verification finds that the system stability does not meet the standard after the strategy is executed, the model retraining process is triggered.

2. The fast steam valve closing control method for transient stability of power systems based on deep reinforcement learning according to claim 1, characterized in that, In step 1, the interaction between the agent and the environment in the Markov decision-making process is described by a set of time series: at time t, the environment is in state s. t The intelligent agent provides a control strategy a based on the environmental state. t After the environment accepts the stabilization strategy given by the agent, its state transitions to s. t+1 At the same time, a reward signal r is output to the agent. t+1 The transition probability P is defined as P(s) t+1 |s t ,a t ), indicating that in state s t Next, execute action a t Then transition to state s t+1 The probability of.

3. The fast steam valve closing control method for transient stability of power systems based on deep reinforcement learning according to claim 1, characterized in that, In step 2, the power system environment simulation includes load level, generator output, new energy grid connection status, and fault events; the fault events include line short circuit faults and transformer faults, and the type, location, and duration of the simulated faults can be flexibly configured.

4. The fast steam valve closing control method for transient stability of power systems based on deep reinforcement learning according to claim 1, characterized in that, In step 2, the Actor neural network is constructed from fully connected layers. Taking the environmental observation state as input, it fits the probability distribution characteristics of the output action through nonlinear mapping of the hidden layers, and then outputs the corresponding action probability vector through the output layer neural network. The hidden layers of the Actor neural network use the ReLU activation function, and the output layer uses the Softmax activation function. The policy output by the Actor neural network is represented as follows: Where π(a|s) is the probability of taking action a given state s; f(a,s;θ) is the network's output score for state s and action a, with the parameter θ; a' represents all possible actions; the Actor network ultimately selects the action with the highest probability as the output.

5. The fast steam valve closing control method for transient stability of power systems based on deep reinforcement learning according to claim 1, characterized in that, In step 2, the Critic neural network is constructed from a fully connected layer neural network, taking the environmental observation state as input and outputting a state value estimate V(s). t ), combining the actual immediate reward r and the value estimate V(s) of the next state. t+1 Obtain the advantage function A(s) t ,a t The feedback is then fed back to the Actor network to guide it in selecting a better action; the hidden layers of the Critic neural network use the ReLU activation function, and the output layer uses a linear activation function; the advantage function is expressed as: A(s t ,a t )≈r+γV(s t+1 )-V(s t ) Where r is in state s t Take action a t The immediate reward obtained, γ is the discount factor for future rewards, V(s) t+1 ) is the next state s t+1 Value estimate.

6. The fast steam valve closing control method for transient stability of power systems based on deep reinforcement learning according to claim 1, characterized in that, In step 2, the near-end policy optimization algorithm introduces a pruning mechanism to simplify the calculation process and maintain the stability of policy updates by controlling the change in the ratio of new to old policies; the objective function of the near-end policy optimization algorithm is expressed as: Where r(θ)=π(a|s) / π(a|s) is the probability ratio between the old and new policies, the clip function is the clipping function, and ε is a positive number used to define the clipping threshold, controlling the change between the old and new policies within [1-ε,1+ε]. The near-end policy optimization algorithm also adopts an adaptive learning rate adjustment mechanism, which dynamically adjusts the learning rate according to the change of the loss function during the training process.

7. The fast steam valve closing control method for transient stability of power systems based on deep reinforcement learning according to claim 1, characterized in that, In step 3, the observed state S in the state space includes the relative power angle matrix δ of the generator and the generator terminal energy W within the corresponding time [T,T)∪[T,T]∪…∪[T,T] after the fault and after stabilization; the observed state S is expressed as: S k =[δ k ,W k ] Where δ represents the relative power angle of the lower synchronizing machine n at time t in the k-th step; W represents the terminal energy of the lower synchronizing machine n at time t in the k-th step; the relative power angle is calculated with reference to the power angle of a certain reference generator in the system.

8. The fast steam valve closing control method for transient stability of power systems based on deep reinforcement learning according to claim 1, characterized in that, In step 3, the fast-closing valve decision quantity in the action space includes selecting the target unit for fast-closing control and the fast-closing valve control parameters, wherein the fast-closing valve control parameters only consider the cylinder stall time T; the Actor network outputs action a, which corresponds to decision b at the environment receiving layer through action-decision mapping, wherein: b represents the i-th unit selected for fast shutdown control in step k, with a total of i units set. b represents the fast shutdown parameter of the unit participating in control in step k. The cylinder stagnation time T ranges from 0.5s to 3s and is divided into multiple selectable levels through discretization.

9. The fast steam valve closing control method for transient stability of power systems based on deep reinforcement learning according to claim 1, characterized in that, In step 3, the reward function comprises two parts: the power system transient stability index (TSI) and the energy reward function, specifically expressed as follows: Where p is the penalty value for instability after stabilization, ranging from 100 to 500; λ is the reward coefficient for the decision effect of fast valve closing, ranging from 0.6 to 0.8; λ is the energy function penalty coefficient, ranging from 0.2 to 0.4; ΔW is the absolute value of the difference between the sum of terminal energies W(S) and the set energy reference value W; when TSI>1 / 3, the system is judged to be stable, otherwise it is judged to be unstable; Δδ is the maximum power angle difference of the generator; P and Q are the active power and reactive power generated by the i-th generator, respectively; U and θ are the voltage amplitude and phase angle of the i-th terminal bus, respectively; the energy reference value W is determined according to the average terminal energy under normal operating conditions of the power system.

10. The fast steam valve closing control method for transient stability of power systems based on deep reinforcement learning according to claim 1, characterized in that, In step 4, during the training process, an experience replay mechanism is used to store the experiential data generated by the interaction between the agent and the environment, including s. t ,a t ,r t ,s t+1 The network parameters are updated based on the empirical data from batch sampling. At the same time, the synchronization update cycle between the target network and the current network is set to improve training stability. During training, the neuron structure of the Critic and Actor networks is set to [32,32]. The total number of training rounds of the neural network is set to 2000, the learning rate is 0.0061459, the advantage estimation parameter Lambda is 0.9, the discount factor gamma is 0.99, and the clipping parameter clip range is 0.3.

Images