Receiving end power grid response driving load shedding control method considering transient voltage recovery demand

Abstract

The invention provides a receiving-end power grid load shedding control method considering a transient voltage recovery demand, belongs to response driving type load shedding control, and solves the technical problem of insufficient system voltage regulation capability caused by large-scale access of new energy to a receiving-end power grid and mass replacement of a traditional fossil energy synchronous unit. According to the technical scheme, the method comprises the following steps: (1) establishing a Markov sequential decision process model for transient voltage control; (2) designing states, actions and rewards in the Markov process; (3) establishing a reinforcement learning model based on the Double DQN; (4) performing off-line centralized learning and training of corresponding agents; and (5) forming a distributed self-adaptive load shedding control method. The method has the beneficial effects that the load shedding controller can be comprehensively deployed at each load station, and a closed-loop distributed load shedding control strategy of'monitoring-judgment-decision-control 'is realized, so that the overall voltage recovery of a power grid is promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of response-driven load shedding control, and in particular to a receiving-end power grid response-driven load shedding control method taking into account transient voltage recovery requirements. Background Art

[0002] Transient voltage stability is one of the key issues affecting the safe operation of the receiving-end power grid at large load centers. The development of the national economy has promoted the continuous increase of electricity load, especially the rapid growth of induction motor loads such as commercial residential air conditioners and industrial loads, and the induction motor load accounts for more than 60% of the total load. The induction motor load slows down or even stalls during the voltage drop, resulting in a significant increase in dynamic reactive power consumption, which is easy to form a positive feedback voltage deterioration process, causing transient voltage instability and even inducing a major power outage. In order to prevent the induction motor load from stalling and disconnecting from the grid after a fault, and to avoid transient voltage safety damage accidents, the North American Electric Reliability Council has stipulated the "voltage threshold-recovery time" segmented transient voltage recovery criterion (TVRC) in its IEEE 1547-2020 reliability guidelines.

[0003] At present, the receiving power grid has eliminated a large number of local fossil energy synchronous units due to the acceptance of remote clean power feed-in, which has caused serious hollowing out of the load center and a significant decrease in the voltage support capacity of the load center power supply. In the case of insufficient power supply voltage support capacity and lack of dynamic reactive power compensation equipment, load reduction control has become a necessary measure to prevent a sharp drop in voltage and avoid transient voltage instability. The existing load reduction control schemes are generally divided into two modes: event-driven and response-driven. Among them, the response-driven load reduction scheme does not require specific matching of operating scenarios and faults, but makes load reduction decisions based on the dynamic change process of voltage to promote voltage recovery. The response-driven load reduction scheme can be further divided into centralized and decentralized types. The decentralized load reduction control scheme only uses the locally monitored voltage information to make on-site load reduction decisions, which is easier to implement under current engineering conditions than the centralized scheme. Summary of the invention

[0004] The purpose of the present invention is to provide a receiving-end power grid response-driven load shedding control method considering the transient voltage recovery demand, and propose a receiving-end power grid adaptive load shedding transient voltage control strategy based on Double DQN response-driven. First, a Markov sequential model of transient voltage load shedding control is established, and the transient voltage evolution trajectory information in the power grid is extracted as the state in combination with TVRC. The load shedding amount and the delay between rounds are selected as actions, and the reward function is designed according to the control effect. Secondly, the load shedding agent and the delay agent based on DQN representation are designed according to the action, and together constitute the Double DQN training structure. The agent is trained offline and centralized through the state-action-reward time series data set generated by the Markov model during the interaction with the environment to learn the optimal load shedding control strategy. Then, the trained agent is deployed at each bus in the system to jointly constitute a decentralized load shedding control strategy, which can make real-time adaptive decisions on the load shedding position, load shedding amount, load shedding time and load shedding rounds according to the current transient voltage severity, and promote voltage recovery to meet TVRC requirements at a better load shedding cost.

[0005] The inventive concept of the present invention is: modeling transient voltage control as a Markov sequential process, extracting the severity of transient voltage as the environmental state quantity, selecting the load reduction amount and the time delay between rounds as actions, and designing a reward function based on the control effect. Secondly, designing a load reduction agent and a delay agent based on DQN representation according to the action, and jointly forming a Double DQN training structure. The agent is trained offline and centrally through the state-action-reward time series data set generated based on the Markov model during the interaction with the environment to learn the optimal load reduction control strategy. Then, the trained agent is deployed at each bus in the system to jointly form a distributed load reduction control strategy, which can adaptively decide the load reduction position, load reduction amount, load reduction time and load reduction rounds in real time according to the current transient voltage severity, and realize a closed-loop distributed load reduction control strategy of "monitoring-judgment-decision-control" to promote the overall voltage recovery of the power grid.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a receiving-end power grid response driven load reduction control method considering the transient voltage recovery demand, comprising the following steps:

[0007] (1) Modeling transient voltage control as a Markov sequential decision process model;

[0008] (2) Based on the transient voltage recovery requirement (TVRC) of the power grid and the real-time voltage evolution trajectory of the load nodes, the environmental state quantity in the Markov sequential decision-making process is extracted;

[0009] (3) According to the factors that affect voltage recovery, the load shedding amount and the time delay between rounds are designed as actions in the Markov sequential decision process;

[0010] (4) Design load reduction agents and time delay agents based on actions; and design differentiated reward functions;

[0011] (5) The agent is trained offline using a state-action-reward dataset obtained from a Markov model during its interaction with the environment;

[0012] (6) The trained load shedding agents and time delay agents are deployed at load sites, forming a decentralized load shedding control structure that can adaptively decide the load shedding location, load shedding amount, and load shedding time, and promote voltage recovery with a smaller load shedding amount to meet transient voltage recovery requirements.

[0013] The specific implementation method of step (1) consists of (2), (3) and (4).

[0014] As a further optimization scheme of a receiving-end power grid response driven load reduction control method considering transient voltage recovery requirements provided by the present invention, in the step (2), an environmental state quantity is extracted;

[0015] According to the TVRC requirements of the power grid and the real-time voltage evolution trajectory of the load node, the negative voltage amplitude deviation ΔV is extracted. m and recovery time deviation ΔT r , voltage deviation ΔV m It is expressed as:

[0016] ΔV m (t k )=V th -V m (t k ) (1)

[0017] Where: ΔV m (t k ) indicates that the current load node is at t k Voltage deviation at the moment, V th Indicates that at the current t k The voltage threshold that needs to be restored at different times according to TVRC requirements, V m (t k ) is the current load node at t k Voltage value at the moment;

[0018] Voltage recovery time deviation ΔT r Calculated by the following formula:

[0019]

[0020] ΔT r (t k )=min{t k +T es (tk )-t c -T req , T r，max} (3)

[0021] In formula (2): T es (t k ) is at t k The estimated time required for the voltage to recover to the corresponding voltage safety threshold at the moment t is Δt, which is the time step. k At this moment, if the voltage shows an upward trend, T es (t k ) is a finite positive value. If the voltage is decreasing, T es (t k ) is positive infinity, which is used to indicate that the estimated voltage trajectory is difficult to recover. In formula (3): ΔT r (t k ) is t k Moment recovery time deviation, t k represents the current time, t c Indicates the short-circuit fault clearing time, T req Indicates that the voltage recovers to V after the fault th The maximum allowable recovery time, T r,max Indicates ΔT r The maximum value of can be taken as the time scale of the transient voltage recovery problem of concern. It should be noted that TVRC has a segmented characteristic. The ΔV m and ΔT r Should be based on the V at the corresponding time th and T req to calculate.

[0022] ΔV extracted in combination with TVRC requirements m (t k ) and ΔT r (t k ) can be used to characterize the severity of the current voltage evolution state. The necessary condition for triggering load reduction control action is: ΔV m (t k )>0 and ΔT r (t k )>0. When the load node bus needs to perform load reduction control action, ΔV is extracted according to the evolution characteristics of each load bus voltage trajectory. m and ΔT r The two-dimensional state is used as the environmental state quantity s t :

[0023] s t = {ΔV m (t),ΔT r (t)} (4)

[0024] As a further optimization scheme of a receiving-end power grid response driven load reduction control method considering transient voltage recovery requirements provided by the present invention, in the step (3), the load reduction amount and the time delay between rounds are designed as actions in a Markov sequential decision process according to factors affecting voltage recovery, including:

[0025] The load shedding amount in each round and the delay between rounds (affecting the load shedding action time) will affect the transient voltage load shedding control effect. In addition, considering the discreteness of the load outgoing line of the substation, the load shedding amount is set as a discrete amount. The load shedding amount and the delay between rounds are selected as the decision action a respectively. Ct and a Tt , and its corresponding discrete action space A Cd and A Td Respectively expressed as:

[0026] A Cd = {ΔP d1 ,ΔP d2 ,…,Δp dm} (5)

[0027] A Td ={ΔT d1 ,ΔT d2 ,…,ΔT dn} (6)

[0028] Where: ΔP d1 ,ΔP d2 ,…,ΔP dm Indicates the action sequence consisting of the percentage of load reduction at the load station, ΔT d1 ,ΔT d2 ,…,ΔT dn represents the action sequence consisting of the time interval between load shedding rounds, and m and n represent the action space A Cd and A Td Dimension.

[0029] According to the selection of load shedding action, two agents, load shedding agent and time delay agent, are designed. The agents can make different action responses according to the environment, and the agents can reach the optimal level after learning and training. The load shedding agent is used to decide the load shedding amount according to environmental changes, and the time delay agent is used to decide the time interval of each round of load shedding action. The agents select actions and respond through the value estimation network. The value estimation network Q of the load shedding agent and the time delay agent C and Q T Used to establish the state-action value mapping relationship. The value estimation network is a multi-layer neural network, and the input is the transient voltage evolution state ΔV of the load bus extracted based on TVRC. m and ΔT rThe two-dimensional state information is fed forward by multiple hidden layer neurons to obtain an output value, which is the value of each discrete action in this state. Its dimension is consistent with the dimension of the defined action space. The neural network can evaluate the pros and cons of different actions under the current transient voltage severity.

[0030] As a further optimization scheme of a receiving-end power grid response driven load reduction control method considering transient voltage recovery requirements provided by the present invention, in the step (4), a load reduction agent and a time delay agent are designed according to the action, and a differentiated reward function is designed:

[0031] The load shedding control implemented in the present invention aims to minimize the load shedding amount under the premise that the voltage recovery of the load bus after the disturbance meets the TVRC requirements. Therefore, rewards are set based on the voltage recovery effect and the size of the load shedding loss.

[0032] For the load shedding agent, the reward function r is designed according to the load bus voltage recovery effect. Ct The expression is as follows:

[0033]

[0034] Where: α 1 and α 2 is a positive constant, U t is the load bus voltage at time t, U 0 is the initial value of the bus voltage before the disturbance.

[0035] For the time-delay agent, the reward function r is designed according to the load shedding loss. Tt The expression is as follows:

[0036]

[0037] Where: β 1 and β 2 is a positive constant, ΔP Ct The current round of load reduction at the load station.

[0038] The reward function r designed above Ct and r Tt By introducing conditional statements, it is possible to clearly define whether the load bus status after the agent's action is implemented meets the TVRC requirements and give targeted rewards. The value of action implementation can be effectively evaluated in the dynamic interaction between the load reduction agent and the time delay agent and the power grid environment, and can be used to guide the agent to perform reinforcement learning to optimize the action strategy.

[0039] As a further optimization scheme of a receiving-end power grid response driven load reduction control method considering transient voltage recovery requirements provided by the present invention, in the step (5), a gradient descent algorithm is used to train the intelligent agent offline to further improve the control performance:

[0040] Offline centralized training is performed based on the "state-action-reward-next state" experience quadruple generated from the interaction between the load shedding agent and the time delay agent and the power grid environment.

[0041] (5-1) Construction of the dataset

[0042] The intelligent agent selects the load shedding control action to respond to the grid environment according to the transient voltage evolution state perceived at the location, and obtains corresponding rewards by observing the control effect under the action implementation. The dynamic process is described by Markov modeling, and t ,A t ,R t ,S t+1 >Sequential data set consisting of four tuples. According to the transient voltage evolution trajectory of the load station, combined with TVRC, the corresponding transient voltage state - voltage amplitude deviation ΔV is extracted m and recovery time deviation ΔT r As state S t , the two agents make action decisions on the load reduction amount and the delay between rounds according to the voltage state. t , and obtain the corresponding reward R from the power grid environment t and the next state S to which it is transferred t+1 , which constitutes the load shedding agent and the delay agent t ,A t ,R t ,S t+1 >The four-tuple data set is stored in the load shedding experience pool and the delay experience pool as training samples,

[0043] When the agent interacts with the power grid to collect the experience quadruple dataset, action a Ct and a Tt Use the ε-greedy algorithm to select, the expression is:

[0044]

[0045] Where: ε C and ε T is the exploration rate of the load shedding agent and the delay agent, ω C and ω T Q C and Q T The weight parameter, a C and a T The actions that the load reduction agent and the delay agent can choose in the corresponding action space respectively.

[0046] (5-2) Training process

[0047] ​​A loss function is established and batch data is selected from the experience pool for agent training to obtain load shedding agents and time delay agents with maximized rewards. The above process is based on the interactively acquired quadruple data set of each load station, and unified offline centralized reinforcement learning training of load shedding agents and time delay agents is performed. The time-difference-based gradient descent algorithm is used to continuously increase the rewards obtained by the agent's actions, so that the agent can choose the optimal value action strategy in response to the environmental state.

[0048] The intelligent agent is trained to achieve the optimal state-action value of the Q network, so that the intelligent agent can respond to the state decision and make the best action. For the Markov process, the optimal Bellman equation is used to characterize Q. C and Q T The maximum expected value of the comprehensive estimation network and the target network, the state-action value function formed by the optimal Bellman equation based on Double DQN can be expressed as:

[0049]

[0050] Where: Expresses expectation; R Ct and R Tt is the reward of the agent that has not been observed at the current moment, which can be calculated by the reward function in the next state; C and γ T are the discount factors of the corresponding agents; maxQ C (S t+1 ,a C ;ω C ) is the next state S t+1 The corresponding Q C The corresponding maximum action value, maxQ T (S t+1 ,a T ;ω T ) is the next state S t+1 Q T The corresponding maximum action value.

[0051] Monte Carlo methods can be used to approximate the expectation of a random variable, so the quadruple obtained from the time series process can be used t ,a t ,r t ,s t+1 >To express the expectation, the above formulas (11) and (12) are approximately:

[0052] Q C (s t ,a Ct ;ω C )≈r Ct +γ C ​maxQ C (s t+1 ,a C ;ω C )(13)

[0053] Q T (s t ,a Tt ;ω T )≈r Tt +γ T maxQ T (s t+1 ,a T ;ω T )(14)

[0054] Where: maxQ C (s t+1 ,a C ;ω C ) is the next state s t+1 The corresponding Q C The corresponding maximum action value, maxQ T (s t+1 ,a T ;ω T ) is the next state s t+1 Q T The corresponding maximum action value.

[0055] Combining equations (13) and (14), the right side of the equation is used as the learning target, that is, as the label quantity of the reinforcement learning process, for Q C and Q T Establish loss functions separately and The expression is as follows:

[0056]

[0057] Where: y Ct and Tt Representative Q C and Q T The target value of the network at time t is composed of the target network and the real observed reward, and T is the end time of the round.

[0058] Using the gradient descent algorithm, estimate the network weight parameter ω C and ω T The update formula can be expressed as:

[0059]

[0060] Where: represents gradient calculation; α C and α Tare the learning rates of the load shedding agent and the delay agent, respectively.

[0061] The intelligent agent explores the action space through equations (9) to (10), accumulates the four-tuple data in the Markov process to build an experience pool. When the data samples stored in the experience pool reach the predetermined storage capacity, the intelligent agent randomly extracts small batches of sample data (mini-batch) from the experience pool for offline centralized training, and updates the weight parameters of the intelligent agent's estimation network and target network through equations (13)-(22) to minimize the loss function and make the state-action value estimation network approach the optimal.

[0062] As a further optimization scheme of the receiving-end power grid response-driven load reduction control method considering the transient voltage recovery demand provided by the present invention, the load sites of the trained intelligent agents are comprehensively deployed to form a distributed adaptive load reduction control strategy, which can respond to local voltage information to perform closed-loop load reduction actions and adaptively decide on the load reduction position, load reduction time and load reduction amount on-site, forming a transient voltage load reduction control strategy in a closed-loop manner of "monitoring-judgment-decision-control", thereby promoting the overall voltage recovery of the power grid.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] (1) The present invention specifically designs a receiving-end power grid response driven load reduction control method that takes into account the transient voltage recovery requirements, so that the voltage recovery satisfies TVRC and a decentralized adaptive load reduction strategy that can minimize the load reduction cost is adopted. The controller at each load station can respond to the local voltage information to perform adaptive load reduction actions to achieve overall voltage recovery of the power grid after the disturbance.

[0065] (2) This paper models the transient voltage control process as a Markov decision process, solves it through the DDQN algorithm, and designs the load shedding agent and the time delay agent to learn the optimal control method in the process of interacting with the environment. An adaptive load shedding transient voltage control method based on D2DQN is obtained. This strategy can adaptively decide the load shedding location, load shedding amount, load shedding rounds and load shedding time according to the severity of the transient voltage evolution trajectory in the power grid, and respond to TVRC with a smaller load shedding cost.

[0066] (3) The transient voltage load reduction control structure based on D2DQN designed in this paper adopts the execution mode of offline centralized training-online decentralized control, which enables the intelligent agent to effectively take into account the global control performance of the system while performing real-time autonomous control based on local information.

[0067] (4) The distributed load shedding control method proposed in the present invention is composed of a load shedding agent and a time delay agent installed and deployed at each load site. It can make real-time adaptive decisions on load shedding factors according to the severity of the transient voltage evolution trajectory of the load node, so as to make the voltage recovery meet the TVRC requirements with a smaller load shedding requirement, thereby promoting the voltage recovery of the entire power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0069] Figure 1 The present invention provides an overall implementation flow chart of a receiving-end power grid response driven load reduction control method considering transient voltage recovery requirements.

[0070] Figure 2 It is a schematic diagram of feedback signal extraction based on TVRC requirements and real-time voltage change status in the present invention.

[0071] Figure 3 Schematic diagram of the offline centralized training part of the interaction between the intelligent agent and the environment designed in the present invention.

[0072] Figure 4 This is a structural diagram of the distributed real-time adaptive load reduction control of the power grid in the present invention.

[0073] Figure 5 This is a structural diagram of the Nordic system used to verify the proposed load reduction control strategy in the present invention.

[0074] Figure 6 For the present invention Figure 5 The bus voltage cluster curve diagrams when no load reduction action is implemented under four basic fault scenarios of the power grid are shown;

[0075] Among them, (a) is the bus voltage cluster curve diagram of fault scenario 1 when no load reduction control is applied; (b) is the bus voltage cluster curve diagram of fault scenario 2 when no load reduction control is applied; (c) is the bus voltage cluster curve diagram of fault scenario 3 when no load reduction control is applied; (d) is the bus voltage cluster curve diagram of fault scenario 4 when no load reduction control is applied.

[0076] Figure 7 It is a bus voltage cluster curve diagram under the load reduction control method proposed in the present invention;

[0077] Among them, (a) is the bus voltage cluster curve after load shedding control is applied in fault scenario 1; (b) is the bus voltage cluster curve after load shedding control is applied in fault scenario 2; (c) is the bus voltage cluster curve after load shedding control is applied in fault scenario 3; (d) is the bus voltage cluster curve after load shedding control is applied in fault scenario 4. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0079] Example 1

[0080] In order to implement a response-driven decentralized load reduction control strategy for a receiving-end power grid to achieve voltage recovery control, this embodiment provides a technical solution, which is a response-driven load reduction control method for a receiving-end power grid considering transient voltage recovery requirements, comprising the following steps:

[0081] (1) Modeling transient voltage control as a Markov sequential decision process model;

[0082] (2) Based on the transient voltage recovery requirement (TVRC) of the power grid and the real-time voltage evolution trajectory of the load nodes, the environmental state quantity in the Markov sequential decision-making process is extracted;

[0083] (3) According to the factors that affect voltage recovery, the load shedding amount and the time delay between rounds are designed as actions in the Markov sequential decision process;

[0084] (4) Design load reduction agents and time delay agents based on actions, and design differentiated reward functions;

[0085] (5) The agent is trained offline using a state-action-reward dataset generated by a Markov model during its interaction with the environment;

[0086] (5) The trained load shedding agents and time delay agents are deployed at load sites, forming a decentralized load shedding control structure that can adaptively decide the load shedding location, load shedding amount, and load shedding time, and promote voltage recovery with a smaller load shedding amount to meet transient voltage recovery requirements.

[0087] Specifically, the specific implementation method of step (1) consists of (2), (3) and (4). In step (2), based on the transient voltage recovery requirements of the power grid and the real-time voltage evolution trajectory of the load node, the environmental state quantity in the Markov sequential decision process is extracted:

[0088] Attach Figure 2In the above, taking the voltage trajectory 2 where the voltage is delayed and restored to the safe range as an example, according to the TVRC requirements of the power grid and the real-time voltage evolution trajectory of the load node, the negative voltage amplitude deviation ΔV is extracted. m and recovery time deviation ΔT r , voltage deviation ΔV m It is expressed as:

[0089] ΔV m (t k )=V th1 -V m (t k ) (1)

[0090] Where: ΔV m (t k ) indicates that the current load node is at t k Voltage deviation at the moment, V th1 Indicates that at the current t k The voltage threshold that needs to be restored at different times according to TVRC requirements, V m (t k ) is the current load node at t k Voltage value at the moment;

[0091] Voltage recovery time deviation ΔT r Calculated by the following formula:

[0092]

[0093] ΔT r (t k )=min{t k +T es (t k )-t c -T req1 , T r，max} (3)

[0094] In formula (2): T es (t k ) is at t k The estimated time required for the voltage to recover to the corresponding voltage safety threshold at the moment t is Δt, which is a time step of 0.02s. This formula shows that at t k At this moment, if the voltage shows an upward trend, T es (t k ) is a finite positive value. If the voltage is decreasing, T es (t k ) is positive infinity, which is used to indicate that the estimated voltage trajectory is difficult to recover. In formula (3): ΔT r (t k ) is t k Moment recovery time deviation, tc Indicates the short-circuit fault clearing time, t k represents the current time, T req Indicates that the voltage recovers to V after the fault th The maximum allowable recovery time, T r,max Indicates ΔT r The maximum value of can be taken as the time scale of the transient voltage recovery problem of concern. It should be noted that TVRC has a segmented characteristic. The ΔV m and ΔT r Should be based on the V at the corresponding time th and T req to calculate.

[0095] ΔV extracted in combination with TVRC requirements m (t k ) and ΔT r (t k ) can be used to characterize the severity of the current voltage evolution state. The necessary condition for triggering load reduction control action is: ΔV m (t k )>0 and ΔT r (t k )>0. When the load node bus needs to perform load reduction control action, ΔV is extracted according to the evolution characteristics of each load bus voltage trajectory. m and ΔT r The two-dimensional state is used as the environmental state quantity s t :

[0096] s t = {ΔV m (t),ΔT r (t)} (4)

[0097] Specifically, in step (3), according to the factors affecting voltage recovery, the load shedding amount and the time delay between rounds are designed as actions in the Markov sequential decision process, including:

[0098] The load shedding amount in each round and the delay between rounds (affecting the load shedding action time) will affect the transient voltage load shedding control effect. In addition, considering the discreteness of the load outgoing line of the substation, the load shedding amount is set as a discrete amount. The load shedding amount and the delay between rounds are selected as the decision action a respectively. Ct and a Tt , and its corresponding discrete action space A Cd and A Td Respectively expressed as:

[0099] A Cd = {ΔP d1 ,ΔP d2 ,…,Δp dm} (5)

[0100] A Td ={ΔT d1 ,ΔT d2 ,…,ΔT dn} (6)

[0101] Where: ΔP d1 ,ΔP d2 ,…,ΔP dm Indicates the action sequence consisting of the percentage of load reduction at the load station, ΔT d1 ,ΔT d2 ,…,ΔT dn represents the action sequence consisting of the time interval between load shedding rounds, and m and n represent the action space A Cd and A Td Dimension.

[0102] According to the selection of load shedding action, two agents, load shedding agent and time delay agent, are designed. The agents can make different action responses according to the environment, and the agents can reach the optimal level after learning and training. The load shedding agent is used to decide the load shedding amount according to environmental changes, and the time delay agent is used to decide the time interval of each round of load shedding action. The agents select actions and respond through the value estimation network. The value estimation network Q of the load shedding agent and the time delay agent C and Q T Used to establish the state-action value mapping relationship. The value estimation network is a multi-layer neural network, and the input is the transient voltage evolution state ΔV of the load bus extracted based on TVRC. m and ΔT r The two-dimensional state information is fed forward by multiple hidden layer neurons to obtain an output value, which is the value of each discrete action in this state. Its dimension is consistent with the dimension of the defined action space. The neural network can evaluate the pros and cons of different actions under the current transient voltage severity.

[0103] Specifically, in step (4), a load reduction agent and a delay agent are designed according to the action, and a differentiated reward function is designed:

[0104] The load shedding control implemented in this paper aims to minimize the load shedding amount under the premise that the voltage recovery of the load bus after the disturbance meets the TVRC requirements. Therefore, rewards are set based on the voltage recovery effect and the size of the load shedding loss.

[0105] For the load shedding agent, the reward function r is designed according to the load bus voltage recovery effect. Ct The expression is as follows:

[0106]

[0107] Where: α 1and α 2 Both are set to 1, U t is the load bus voltage at time t, U 0 is the initial value of the bus voltage before the disturbance.

[0108] For the time-delay agent, the reward function r is designed according to the load shedding loss. Tt The expression is as follows:

[0109]

[0110] Where: β 1 =1,β 2 =10,ΔP Ct The current round of load reduction at the load station.

[0111] The reward function r designed above Ct and r Tt By introducing conditional statements, it is possible to clearly define whether the load bus status after the agent's action is implemented meets the TVRC requirements and give targeted rewards. The value of action implementation can be effectively evaluated in the dynamic interaction between the load reduction agent and the time delay agent and the power grid environment, and can be used to guide the agent to perform reinforcement learning to optimize the action strategy.

[0112] Specifically, in step (5), the gradient descent algorithm is used to train the agent offline to further improve the control performance:

[0113] Offline centralized training is performed based on the "state-action-reward-next state" experience quadruple generated from the interaction between the load shedding agent and the time delay agent and the power grid environment.

[0114] (5-1) Construction of the dataset

[0115] The intelligent agent selects the load shedding control action to respond to the grid environment according to the transient voltage evolution state perceived at the location, and obtains corresponding rewards by observing the control effect under the action implementation. The dynamic process is described by Markov modeling, and t ,A t ,R t ,S t+1 >Sequential data set consisting of four tuples. According to the transient voltage evolution trajectory of the load station, combined with TVRC, the corresponding transient voltage state - voltage amplitude deviation ΔV is extracted m and recovery time deviation ΔT r As state S t , the two agents make action decisions on the load reduction amount and the delay between rounds according to the voltage state. t , and obtain the corresponding reward R from the power grid environment t and the next state S to which it is transferred t+1 ​, which constitutes the load shedding agent and the delay agent t ,A t ,R t ,S t+1 >The four-tuple data set is stored in the load shedding experience pool and the delay experience pool as training samples,

[0116] When the agent interacts with the power grid to collect the experience quadruple dataset, action a Ct and a Tt Use the ε-greedy algorithm to select, the expression is:

[0117]

[0118] Where: ε C and ε T is the exploration rate ε of the load shedding agent and the delay agent C =0.3,ε T =0.3;ω C and ω T Q C and Q T The weight parameter of C and a T The actions that the load reduction agent and the delay agent can choose in the corresponding action space respectively.

[0119] (5-2) Training process

[0120] A loss function is established and batch data is selected from the experience pool for agent training to obtain load shedding agents and time delay agents with maximized rewards. The above process is based on the interactively acquired quadruple data set of each load station, and unified offline centralized reinforcement learning training of load shedding agents and time delay agents is performed. The time-difference-based gradient descent algorithm is used to continuously increase the rewards obtained by the agent's actions, so that the agent can choose the optimal value action strategy in response to the environmental state.

[0121] The intelligent agent is trained to achieve the optimal state-action value of the Q network, so that the intelligent agent can respond to the state decision and make the best action. For the Markov process, the optimal Bellman equation is used to characterize Q. C and Q T The maximum expected value of the comprehensive estimation network and the target network, the state-action value function formed by the optimal Bellman equation based on Double DQN can be expressed as:

[0122]

[0123] Where: Expresses expectation; R Ct and R Tt ​is the reward of the agent that has not been observed at the current moment, which can be calculated by the reward function in the next state; C and γ T are the discount factors of the corresponding agents; maxQ C (S t+1 ,a C ;ω C ) is the next state S t+1 The corresponding Q C The corresponding maximum action value, maxQ T (S t+1 ,a T ;ω T ) is the next state S t+1 Q T The corresponding maximum action value.

[0124] Monte Carlo methods can be used to approximate the expectation of a random variable, so the quadruple obtained from the time series process can be used t ,a t ,r t ,s t+1 >To express the expectation, the above formulas (11) and (12) are approximately:

[0125] Q C (s t ,a Ct ;ω C )≈r Ct +γ C maxQ C (s t+1 ,a C ;ω C )(13)

[0126] Q T (s t ,a Tt ;ω T )≈r Tt +γ T maxQ T (s t+1 ,a T ;ω T )(14)

[0127] Where: maxQ C (s t+1 ,a C ;ω C ) is the next state s t+1 The corresponding Q C The corresponding maximum action value, maxQ T (s t+1 ,a T ;ω T ​) is the next state s t+1 Q T The corresponding maximum action value.

[0128] Combining equations (13) and (14), the right side of the equation is used as the learning target, that is, as the label quantity of the reinforcement learning process, for Q C and Q T Establish loss functions separately and The expression is as follows:

[0129]

[0130] Where: y Ct and Tt Representative Q C and Q T The target value of the network at time t is composed of the target network and the real observed reward, and T is the end time of the round.

[0131] Using the gradient descent algorithm, estimate the network weight parameter ω C and ω T The update formula can be expressed as:

[0132]

[0133] Where: represents gradient calculation; α C and α T are the learning rates of the load shedding agent and the delay agent, α C =0.001,α T =0.001.

[0134] The intelligent agent explores the action space through equations (9)-(10) and accumulates the four-tuple data in the Markov process to build an experience pool. When the data samples stored in the experience pool reach the predetermined storage capacity, the intelligent agent randomly extracts small batches of sample data (mini-batch) from the experience pool for offline centralized training. The intelligent agent updates the estimated network and target network weight parameters through equations (13)-(22) to minimize the loss function and make the state-action value estimation network approach the optimal.

[0135] Specifically, in step (6), each load station where the trained intelligent agent is deployed comprehensively forms a distributed adaptive load reduction control strategy, which responds to local voltage information to perform closed-loop load reduction actions to adaptively decide the load reduction position, load reduction time and load reduction amount locally, forming a transient voltage load reduction control strategy in a closed-loop manner of "monitoring-judgment-decision-control", thereby promoting the overall voltage recovery of the power grid. The distributed load reduction control structure is shown in the attached figure. Figure 4 shown.

[0136] In order to fully verify the effectiveness of the load reduction control strategy proposed in this embodiment, the attached Figure 5 The Nordic power grid in Northern Europe is simulated and analyzed as shown in the figure. A CLM load model is established for 11 load stations in the heavy-load Central area for simulation. Specifically, three-phase short-circuit faults at a, b, c and d are simulated as basic fault scenarios 1, 2, 3 and 4. The voltage curves before and after transient control are shown in the attached figure. Figure 6 and attached Figure 7 shown.

[0137] It can be seen that the bus voltage trajectory when load reduction control is not implemented in the four scenarios exceeds the voltage recovery safety range specified by TVRC, while the voltage trajectory after the deployed trained intelligent agent applies load reduction control can effectively meet the TVRC requirements. The total load reduction in the four scenarios is 805MW.

[0138] In summary, the simulation results further verify the adequacy and effectiveness of the load reduction control method proposed in this embodiment, which is of great significance for improving the voltage safety level of the receiving-end power grid and ensuring the safe and stable operation of the system.

[0139] Example 2

[0140] To further verify the superiority of the method of this embodiment, the method proposed in this embodiment is compared with the traditional distributed load reduction control method. The specific load reduction control method is as follows:

[0141] (1) Method 1: Adaptive load shedding transient voltage control method based on Double DQN response drive proposed in this embodiment;

[0142] (2) Method 2: Traditional distributed load shedding control strategy, the load shedding control start thresholds are 0.9 pu, 0.8 pu, and 0.7 pu, the load shedding amount in each round is 10%, and the delay between rounds is 0.2 s.

[0143] Table 1 shows the control effects and load reduction of the two control methods under fault scenarios 1-4. The results show that after method 1 is controlled, all scenarios meet the TVRC requirements, while after method 4 is controlled, the voltage limit is particularly serious, and the four fault scenarios cannot meet the TVRC requirements; and in all scenarios, method 1 can promote voltage recovery at a lower load reduction cost, and the overall load reduction is 69.7% smaller than that of method 2. Figure 6 , Figure 7 The simulation results in Table 1 show that the load shedding control strategy proposed in this embodiment can respond to TVRC with a smaller load shedding cost and promote the recovery of the voltage level when adaptively performing transient voltage load shedding control, which verifies the superiority of the adaptive load shedding control strategy proposed in this embodiment.

[0144] Table 1 Control effect and load reduction under different control methods

[0145] Tab.1Control performance and load shedding amount with different control methods in fault scenario 1-4

[0146]

[0147]

[0148] Note: ΔP All Indicates the load reduction amount (MW) of each fault scenario; ΣΔP All Indicates all fault scenarios ΔP All The sum of P erf_ind Represents the transient voltage control effect index, 1 means that TVRC is met, and 0 means that TVRC is not met.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A receiving-end power grid response driven load reduction control method considering transient voltage recovery requirements, characterized in that: The following steps are involved: (1) Modeling transient voltage control as a Markov sequential decision process model; (2) Based on the transient voltage recovery requirements (TVRC) of the power grid and the real-time voltage evolution trajectory of the load nodes, the environmental state quantity in the Markov sequential decision-making process is extracted; (3) According to the factors that affect voltage recovery, the load shedding amount and the time delay between rounds are designed as actions in the Markov sequential decision process; (4) Design load reduction agents and time delay agents based on actions, and design differentiated reward functions; (5) The agent is trained offline using a state-action-reward dataset generated by a Markov model during its interaction with the environment; (6) The trained load shedding agents and time delay agents are deployed at load sites to form a decentralized load shedding control structure, which adaptively decides the load shedding location, load shedding amount, and load shedding time to minimize the load shedding amount and promote voltage recovery to meet transient voltage recovery requirements.

2. The receiving-end power grid response driven load reduction control method considering transient voltage recovery requirements according to claim 1 is characterized in that: In the step (2), based on the transient voltage recovery requirements of the power grid and the real-time voltage evolution trajectory of the load node, the environmental state quantity in the Markov sequential decision process is extracted, including: According to the TVRC requirements of the power grid and the real-time voltage evolution trajectory of the load node, the voltage amplitude deviation ΔV is extracted m and recovery time deviation ΔT r : Voltage deviation ΔV m It is expressed as: ΔV m (t k )=V th -V m (t k ) (1) Where: ΔV m (t k ) indicates that the current load node is at t k Voltage deviation at the moment, V th Indicates the voltage threshold to be restored at different time periods according to TVRC requirements, V m (t k ) is the current load node at t k Voltage value at the moment; Voltage recovery time deviation ΔT r Calculated by the following formula: ΔT r (t k )=min{t k +T es (t k )-t c -T req ,T r，max } (3) In formula (2): T es (t k ) is at t k The time required for the estimated voltage to recover to the corresponding voltage safety threshold at time t k At this moment, if the voltage shows an upward trend, T es (t k ) is a finite positive value. If the voltage is decreasing, T es (t k ) is positive infinity, which is used to indicate that the estimated voltage trajectory is difficult to recover, Δt is the time step, and in formula (3): ΔT r (t k ) is t k Moment recovery time deviation, t k represents the current time, t c Indicates the short-circuit fault clearing time, T req Indicates that the voltage returns to V after a fault th The maximum allowable recovery time, T r,max Indicates ΔT r The maximum value is taken as the time scale of the transient voltage recovery problem of concern. TVRC has a segmented characteristic. The ΔV m and ΔT r Should be based on the V at the corresponding time th and T req to calculate; ΔV extracted in combination with TVRC requirements m (t k ) and ΔT r (t k ) is used to characterize the severity of the current voltage evolution state. The necessary condition for triggering load reduction control action is: ΔV m (t k )>0 and ΔT r (t k )>0, when the load node bus performs load reduction control, according to the evolution characteristics of each load bus voltage trajectory, ΔV m and ΔT r The two-dimensional state is used as the environmental state quantity s t : s t ={ΔV m (t),ΔT r (t)} (4)。 3. The receiving-end power grid response driven load reduction control method considering transient voltage recovery requirements according to claim 1 is characterized in that: In step (3), the load shedding amount and the time delay between rounds are designed as actions in the Markov sequential decision process according to the factors affecting voltage recovery: The load shedding amount in each round and the time delay between rounds will affect the transient voltage load shedding control effect. In addition, considering that the load outgoing line of the substation is discrete, the load shedding amount is set as a discrete amount, and the load shedding amount and the time delay between rounds are selected as the decision action a respectively. Ct and a Tt , and its corresponding discrete action space A Cd and A Td Respectively expressed as: A Cd ={ΔP d1 ,ΔP d2 ,…,Δp dm } (5) A Td ={ΔT d1 ,ΔT d2 ,…,ΔT dn } (6) Where: ΔP d1 ,ΔP d2 ,…,ΔP dm Indicates the action sequence consisting of the percentage of load reduction at the load station, ΔT d1 ,ΔT d2 ,…,ΔT dn represents the action sequence consisting of the time interval between load shedding rounds, and m and n represent the action space A Cd and A Td Dimension.

4. The receiving-end power grid response driven load reduction control method considering transient voltage recovery requirements according to claim 1 is characterized in that: In the step (5), two agents, a load shedding agent and a time delay agent, are designed according to the selection of the load shedding action: The agent makes different action responses according to the environment through the value estimation network. The agent reaches the optimal state after learning and training. The load shedding agent is used to decide the load shedding amount according to environmental changes. The delay agent is used to decide the time interval of each round of load shedding action. The value estimation network Q of the load shedding agent and the delay agent C and Q T Used to establish the state-action value mapping relationship. The value estimation network is a multi-layer neural network. The input is the transient voltage evolution state ΔV of the load bus extracted based on TVRC. m and ΔT r The two-dimensional state information is fed forward by multiple hidden layer neurons to obtain an output value, which is the value of each discrete action in this state. Its dimension is consistent with the dimension of the defined action space. The neural network is used to evaluate the pros and cons of different actions under the current transient voltage severity. The reward function is set for the intelligent agent according to the control objectives and control performance pursued by transient voltage load shedding control. Load shedding control means that the load shedding amount is minimized under the premise that the voltage recovery of the load bus after the disturbance meets the TVRC requirements. The rewards are set based on the voltage recovery effect and the load shedding loss size respectively. For the load shedding agent, the reward function r is designed according to the load bus voltage recovery effect. Ct The expression is as follows: Where: α1 and α2 are positive constants, U t is the load bus voltage at time t, U0 is the initial value of the bus voltage before the disturbance; For the time-delay agent, the reward function r is designed according to the load shedding loss. Tt The expression is as follows: Where: β1 and β2 are positive constants, ΔP Ct The current round of load reduction of the load station; The reward function r designed above Ct and r Tt Conditional statements are introduced to clearly define whether the load bus status meets the TVRC requirements after the agent's action is implemented, and targeted rewards are given. The value of action implementation is effectively evaluated in the dynamic interaction between the load reduction agent and the time delay agent and the power grid environment, which is used to guide the agent to perform reinforcement learning to optimize the action strategy.

5. The receiving-end power grid response driven load reduction control method considering transient voltage recovery requirements according to claim 1 is characterized in that: In the step (5), offline centralized training is performed based on the "state-action-reward-next state" experience quadruple generated in the state of the load shedding agent and the delay agent interacting with the power grid environment, including: The first step is to construct a data set. The intelligent agent selects load shedding control actions to respond to the grid environment based on the transient voltage evolution state it perceives at its location, and observes the control effect of the action to obtain corresponding rewards. This dynamic process is described by Markov modeling, and we can get t ,A t ,R t ,S t+1 >The sequential data set composed of four tuples, based on the transient voltage evolution trajectory of the load station, combined with TVRC, extracts the corresponding transient voltage state-voltage amplitude deviation ΔV m and recovery time deviation ΔT r As state S t , the two agents make action decisions on the load reduction amount and the delay between rounds according to the voltage state. t , and obtain the corresponding reward R from the power grid environment t and the next state S to which it is transferred t+1 , which constitutes the load shedding agent and the delay agent t ,A t ,R t ,S t+1 >Quadruple data sets are stored in the load shedding experience pool and the delay experience pool as training samples;​​ When the agent interacts with the power grid to collect the experience quadruple dataset, action a Ct and a Tt Use the ε-greedy algorithm to select, the expression is: Where: ε C and ε T is the exploration rate of the load shedding agent and the delay agent, ω C and ω T Q C and Q T The weight parameter, a C and a T The actions that the load shedding agent and the delay agent can choose in the corresponding action space respectively; Secondly, a loss function is established and batch data is selected from the experience pool for agent training to obtain load shedding agents and time delay agents with optimal action values. The above process is based on the four-tuple data set of each load station obtained interactively, and unified offline centralized reinforcement learning training of load shedding agents and time delay agents is carried out. The time-difference-based gradient descent algorithm is used to continuously increase the rewards obtained by the agent's actions, so that the agent can choose the optimal value action strategy in response to the environmental state. The intelligent agent is trained to make the best action in response to the state, facing the Markov process, and characterizes Q through the optimal Bellman equation. C and Q T The maximum expected value of the comprehensive estimation network and the target network, the state-action value function formed by the optimal Bellman equation based on DoubleDQN is expressed as: Where: Expresses expectation; R Ct and R Tt is the reward of the agent that has not been observed at the current moment, which is calculated by the reward function in the next state; C and γ T are the discount factors of the corresponding agents; maxQ C (S t+1 ,a C ;ω C ) is the next state S t+1 The corresponding Q C The corresponding maximum action value, maxQ T (S t+1 ,a T ;ω T ) is the next state S t+1 Q T The corresponding maximum action value; Monte Carlo methods are used to approximate the expectation of a random variable, using the quadruple obtained from the time series process t ,a t ,r t ,s t+1 >To express the expectation, the above formulas (11) and (12) are expressed as:​ Q C (s t ,a Ct Oh C )≈r Ct +g C max Q C (s t+1 ,a C Oh C ) (13) Q T (s t ,a Tt Oh T )≈r Tt +g T max Q T (s t+1 ,a T Oh T ) (14) Where: maxQ C (s t+1 ,a C ;ω C ) is the next state s t+1 The corresponding Q C The corresponding maximum action value, maxQ T (s t+1 ,a T ;ω T ) is the next state s t+1 Q T The corresponding maximum action value; Combining equations (13) and (14), the right side of the equation is used as the learning target, that is, as the label quantity of the reinforcement learning process, for Q C and Q T Establish loss functions separately and The expression is as follows: Where: y Ct and Tt Representative Q C and Q T The target value of the network at time t, which is composed of the target network and the real observed reward, and T is the end time of the round; Using the gradient descent algorithm, estimate the network weight parameter ω C and ω T The update formula is expressed as: Where: represents gradient calculation; α C and α T are the learning rates of the load shedding agent and the time delay agent, respectively; The optimal state-action value function training of the intelligent agent based on the Double DQN algorithm explores the action space through equations (9) to (10), accumulates the four-tuple data in the Markov process to build an experience pool, and when the data samples stored in the experience pool reach the predetermined storage capacity, the intelligent agent randomly extracts a small batch of sample data mini-batch from the experience pool for offline centralized training, and updates the weight parameters of the estimated network and target network of the intelligent agent through equations (13) to (22) to minimize the loss function and make the state-action value estimation network approach the optimal.

6. The receiving-end power grid response driven load reduction control method considering transient voltage recovery requirements according to claim 1 is characterized in that: In the step (5), the load shedding agent and the time delay agent after intensive training are deployed at each load shedding load station. The agent adaptively decides the load shedding position, load shedding time and load shedding amount on site according to the transient voltage trajectory evolution state monitored by the load station, forming a transient voltage load shedding control strategy in a "monitoring-judgment-decision-control" closed-loop manner. The agent performs centralized training parameter update based on the global experience data set, so that the decentralized control takes into account the overall load shedding optimization control performance on the basis of local efficient autonomous load shedding control.

Images