A low-voltage load shedding emergency control method for power grid based on graph deep reinforcement learning
Through the graph deep reinforcement learning method, the topological features of the power system are extracted using the FastGCN-D3QN model, which solves the problem of poor generalization performance of deep reinforcement learning in grid control, and achieves a more reliable and economical low-voltage load reduction control in grid.
Patent Information
- Application Number
- CN202210284930.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-03-22
AI Technical Summary
When the existing technology deals with the power grid control problem of frequent topology, the model generalization performance of the deep reinforcement learning method is poor, resulting in low accuracy of control strategies and inability to effectively capture the state characteristics of the power grid system, affecting the stable economic operation of the power grid.
Using the low-voltage load reduction emergency control method of power grid based on graph deep reinforcement learning, the low-voltage load reduction emergency control model is extracted through the FastGCN-D3QN graph deep reinforcement learning model, combined with the graph convolutional neural network and deep reinforcement learning framework, the topological features of the power system are extracted, and the low-voltage load reduction emergency control model is established. , provide more reliable control strategies.
In the power system scenario where topology changes frequently, the adaptability of the model and the accuracy of the control strategy are improved, and a more reliable and economical grid low-voltage load reduction solution is provided, with stronger adaptability and better control strategy.
Smart Images

Figure CN114865638B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of safe operation and control of power systems, and in particular to a low-voltage load shedding emergency control method for power grids based on graph deep reinforcement learning. Background Art
[0002] Current research on low-voltage load reduction algorithms includes traditional methods, adaptive methods, computer-aided algorithms, and deep reinforcement learning methods.
[0003] The traditional method is a static method that cuts off a fixed number of loads each time. It may over-cut or under-cut when dealing with emergencies, and cannot provide the optimal load shedding solution. The adaptive method dynamically cuts off the load based on the disturbance level of the system. Computer-aided algorithms include artificial neural networks, heuristic algorithms, particle swarm optimization algorithms, fuzzy control, etc. In this type of method, the low-voltage load shedding problem is defined as an optimal decision problem, which can provide a better load shedding solution. However, the above three methods are not very adaptable to the uncertainty of the system. When dealing with nonlinear and high-dimensional problems, the accuracy of the control strategy is significantly reduced. In comparison, the deep reinforcement learning method combines the state perception ability of deep learning with the decision-making ability of reinforcement learning, and has strong adaptability to system uncertainty. However, its shortcomings are that the generalization performance of the model is poor when dealing with power grid control problems with frequent topological changes, and the accuracy of the control strategy is low. The reason is that the topological structure of the power grid represents the relationship between nodes in the form of a graph. Compared with structured text and image data, the topological structure of the graph is complex. Each node has edges related to other nodes in the graph, and each node has a varying number of adjacent nodes. Traditional deep learning methods are not suitable for such non-Euclidean spatial data and cannot capture changes in system state characteristics well, resulting in a significant reduction in the accuracy of the model control strategy, which is unacceptable for the goal of stable and economic operation of the power grid. Summary of the invention
[0004] The present invention mainly solves the technical problems existing in the prior art; it provides a low-voltage load reduction emergency control method for a power grid based on graph deep reinforcement learning, which provides a solution to the perception and decision-making problems in power system scenarios with frequently changing topologies, and can provide a more reliable and economical control strategy.
[0005] The above technical problems of the present invention are mainly solved by the following technical solutions:
[0006] A method for emergency control of low-voltage load shedding in a power grid based on graph deep reinforcement learning, characterized in that it comprises the following steps:
[0007] Step 1: Determine the scale of the power system calculation and set the power system topology change scenario; design the low-voltage load reduction action node, low-voltage load reduction action strategy and low-voltage load reduction strategy objective function;
[0008] Step 2: Collect node state observation value sample sets at multiple historical moments of the power system topology to generate a node feature matrix X and a power system topology graph G;
[0009] Step 3, preprocess the node feature matrix to obtain normalized sample feature data; introduce the FastGCN-D3QN graph deep reinforcement learning model, input the preprocessed node feature matrix into the FastGCN-D3QN graph deep reinforcement learning model to extract features and obtain effective sample features; further train through the reinforcement learning process of the FastGCN-D3QN graph deep reinforcement learning model to establish a low-voltage load reduction emergency control model for the power system;
[0010] Step 4, the multi-time node state observation values of the power grid to be tested collected in real time, including the node voltage amplitude and load proportion at the operating time t and the previous m-1 times, are used to construct a real-time node feature matrix X through step 2, and further processed through the data preprocessing process described in step 3 to obtain real-time normalized sample feature data X', and further predicted through the trained power system low-voltage load reduction emergency control model to obtain the load adjustment amount corresponding to each action bus node, so as to form a low-voltage load reduction emergency control plan for the power grid to be predicted.
[0011] Preferably, the power system calculation example scale in step 1 is n;
[0012] The topology change scenario described in step 1 is:
[0013] First, construct the power system topology consisting of vertex set V and edge set E, where v i represents the ith node v in the power system topology i , v i ∈V,i=1,2,…,n;
[0014] Secondly, multiple types of system voltage disturbances and topology change scenarios with multiple line disconnections are set. Short-circuit faults are set for important load bus nodes and other bus nodes in the example, and one connecting line between the short-circuit point bus and the adjacent line and one or two of the remaining connecting lines are disconnected.
[0015] The designed low voltage load reduction action node in step 1 is:
[0016] The low voltage load shedding action node is defined as the important load bus node of the system;
[0017] The low voltage load reduction action strategy designed in step 1 is:
[0018] The load shedding action at each action node is a low-voltage load shedding action in rounds to keep the load ratio within the specified range;
[0019] The objective function of the low voltage load reduction strategy designed in step 1 is:
[0020] Set voltage recovery standard time T st And voltage recovery standard value U st , and based on the goal of achieving transient voltage stability with minimum load shedding loss, the objective function of the low-voltage load shedding strategy is established:
[0021]
[0022] If t c <t<t c +T st
[0023] In the formula, is the voltage amplitude at the ith node of the power system topology at time t and restore the standard value U st The difference, It refers to the load reduction of the ith node in the power system topology at time t, and a and b are the cumulative voltage deviations of each node respectively. and the accumulated load reduction of each node The corresponding coefficient, t c Indicates the time to clear the fault.
[0024] Preferably, the node state observation value sample set of the power system topology at multiple historical moments in step 2 is specifically defined as:
[0025] For the i-th node in the power system topology, the collected observation value samples Including the node voltage amplitude and load proportion at sampling time t and its previous m-1 historical moments, expressed as:
[0026]
[0027] In the formula, and They represent the node voltage amplitude of the ith node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t, respectively. and They represent the load proportions of the ith node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t respectively;
[0028] For the system topology, its node state observation value sample set Obs G It includes the node voltage amplitude and load proportion of each node in the topology at sampling time t and its previous m-1 historical moments, expressed as:
[0029]
[0030] In the formula, and They represent the node voltage amplitudes of the first node, the second node, and the nth node in the power system topology at time m-1 before sampling time t,
[0031] and They represent the node voltage amplitudes of the first node, the second node, and the nth node in the power system topology at time m-2 before sampling time t, respectively.
[0032] and They represent the node voltage amplitudes of the first node, the second node and the nth node of the power system topology at sampling time t respectively;
[0033] and They represent the load proportions of the first node, the second node, and the nth node in the power system topology at time m-1 before sampling time t,
[0034] and They represent the load proportions of the first node, the second node, and the nth node in the power system topology at time m-2 before sampling time t, respectively.
[0035] and They represent the load proportions of the first node, the second node, and the nth node in the power system topology at sampling time t respectively;
[0036] Step 2 generates the node feature matrix X, specifically:
[0037] Based on the node state observation value sample set Obs G , establish the node feature matrix X, the observation dimension is n*2m, which is specifically expressed as follows:
[0038]
[0039] In the formula, and They represent the node voltage amplitudes of the first node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t, respectively.
[0040] and They represent the node voltage amplitude of the second node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t, respectively.
[0041] and They represent the node voltage amplitude of the nth node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t respectively;
[0042] and They represent the load proportions of the first node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and time t, respectively.
[0043] and They represent the load proportions of the second node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t, respectively.
[0044] and They represent the load proportions of the nth node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t respectively;
[0045] Step 2 generates the power system topology diagram G, specifically:
[0046] Based on the adjacency matrix A of the power system topology, a power system topology graph G containing a vertex set V and an edge set E is generated:
[0047] The adjacency matrix A is used to represent the connection relationship between nodes in the power topology. If the node v i and node v j On the same line, it is represented as 1, otherwise it is represented as 0, which is represented as follows:
[0048]
[0049] In the formula, v i 、v j They represent the ith and jth nodes in the power system topology, respectively, and v i ,v j ∈V,i,j=1,2,…,n,p is the row index of the adjacency matrix A,q is the column index of the adjacency matrix A,p,q=0,1,…,n-1,A p,q represents the element in the p+1th row and q+1th column of the adjacency matrix A;
[0050] Create a power system topology graph G from a given adjacency matrix A.
[0051] Preferably, the data preprocessing in step 3 includes:
[0052] The node feature matrix X is linearly normalized row by row and mapped to the interval [0,1], so as to convert the node observation feature values to the same dimension and obtain the preprocessed node feature matrix X'. The specific normalization method is as follows:
[0053]
[0054] Where f is the row index of the node feature matrix X, g is the column index of the feature matrix X, and x' f,g is the element in the f+1th row and g+1th column of the preprocessed node feature matrix X', min(X f ) and max(X f ) are the minimum and maximum values of the f+1th row of the node feature matrix X before normalization;
[0055] The FastGCN-D3QN graph deep reinforcement learning model described in step 3:
[0056] It is composed of the improved FastGCN graph convolutional neural network, the deep convolutional neural network part of D3QN deep reinforcement learning, and the reinforcement learning framework of D3QN deep reinforcement learning in cascade order;
[0057] The improved FastGCN graph convolutional neural network is used to extract power topology features from non-Euclidean spatial data;
[0058] Use D3QN's deep convolutional neural network to further process the observation value characteristics of each node;
[0059] The reinforcement learning framework based on D3QN deep reinforcement learning is used to train the model and establish an emergency control model for low-voltage load shedding in the power system.
[0060] Step 3 uses FastGCN-D3QN graph deep reinforcement learning to train and establish a low-voltage load reduction emergency control model for the power system, specifically:
[0061] Step 3.1: Input the preprocessed node feature matrix X' into the FastGCN-D3QN graph deep reinforcement learning model, use the improved FastGCN graph convolutional neural network to learn the topological structure and vertex attribute information of non-Euclidean spatial data types in the power system topology graph G, sample and aggregate the features of each node; use the deep convolutional neural network in the D3QN deep reinforcement learning to further extract the features of each node:
[0062] The number of sampling network layers L is set according to the topological scale of the power system. Node feature sampling is performed using the mini-batch method. First, the nodes are divided into multiple batches, and each layer is sampled using the layer-wise sampling method according to the degree of the vertex. l vertices;
[0063] Considering that the power system topology is relatively sparse (the number of edges |E| is less than |V| 2 ), the shared neighbor set will make the adjacent vertices in the mini-batch more sparse, and the model effect is not good. Here we propose an improved importance sampling method - pre-traversing the nodes to be sampled in each layer. Specifically, first, according to the actual scale of the example topology and the relationship between nodes, define the minimum acceptable value of the correlation between samples in each adjacent layer Degree min ; Each vertex v i According to the probability distribution q(v i ) for sampling:
[0064]
[0065] In the formula, A=A+I N Add the identity matrix I to the adjacency matrix A of the undirected graph G N , v i represents the i-th node in the power system topology. Each layer is based on the probability distribution q(v i ) The number of samples is t l The last layer is not sampled to form a sampling sample combination, and the correlation between the sampling samples of each adjacent layer in the combination is analyzed. If the correlation is lower than Degree min , then eliminate the combination; repeat this step multiple times to form an optimized pre-selected sample set;
[0066] In the actual sampling process, a group of sampling sample combinations are randomly selected from the pre-selected sample set for node feature collection. The principle of collection is still to select a sampling sample combination for each batch, and each layer is based on q(v i ) The number of samples is t l The nodes of the last layer are not sampled;
[0067] In the basic GCN method, each convolutional layer passes through the adjacency matrix A and the feature matrix h (l) Multiply them together to get the summary of each vertex's neighbor features, and then multiply them by the parameter matrix W (l) Add activation function σ to perform a nonlinear transformation to obtain the matrix h of aggregated adjacent vertex features. (l+1) , for the improved FastGCN graph convolutional neural network, its propagation rule is expressed in integral form:
[0068] h (l+1) (v i )=∫A(v i ,v j )h (l) (v j )W (l)dP(v j ),h (l+1) =σ(h (l+1) (v i ))
[0069] In the formula, v i and v i They represent the i-th and j-th nodes in the power system topology, j≠i, i,j=1,2,…,n, h (l) is the activation unit matrix of layer l, h (l) is the activation unit matrix of layer l, h (0) =X,h (l+1) (v i ) represents the node v at the lth layer i The characteristic matrix, h (l) (v j ) represents the node v at the lth layer j The characteristic matrix, P(v j ) is the probability distribution of nodes. In this method, the Monte Carlo method is used to sample each layer t l Vertices To approximate the integral, it can be expressed as:
[0070]
[0071] Generate feature vector representation of each node
[0072]
[0073] In the formula, v i represents the ith node v in the power system topology i , i=1,2,…,n;
[0074] The deep convolutional neural network in D3QN deep reinforcement learning is used to further extract the features of each node, and two different sets of parameters θ and θ' are selected for current state action selection and optimal strategy evaluation respectively:
[0075] Here, the reward function r is defined based on the low-voltage load reduction strategy objective function established in step 1.4. t :
[0076]
[0077] In the formula, system collapse means that the system voltage does not recover to the target voltage value within the specified time, and -c is the huge penalty value when the system collapses;
[0078] First, based on the parameter θ, the current value neural network is used to approximate the value function Q(s) of the current state of the topology. t ,a;θ t), select the current state action, where Q means the given low voltage load reduction strategy π, the power system state s at time t t Execute low-voltage load reduction control action a and get reward value r t , the state-action value function corresponding to the state-action pair (s, a);
[0079] Next, find the action a' corresponding to the maximum Q value max , based on another parameter θ', the target value neural network is used to calculate the parameter update Q value function and evaluate the optimal strategy a' max :
[0080] a' max = argmaxQ(s t+1 ,a;θ t )
[0081] Y t =R t+1 +γQ(s t+1 ,argmaxQ(s t+1 ,a;θ t );θ′ t )
[0082] In the formula, is the cumulative reward value starting from time t, γ∈[0,1] is the discount factor;
[0083] Step 3.2: Use the Q value obtained in step 3.1 as the input data of reinforcement learning, introduce the competitive network structure to divide the Q network into two branches, one of which is only related to the power system topology state s and has nothing to do with the low-voltage load reduction control action a to be executed, called the state value function V(s; θ0, β), where θ0 represents the convolutional network parameters of the common part, and β represents the network parameters of the state value function part; the other branch is related to both the system state s and the low-voltage load reduction action a, called the action advantage function A(s, a; θ0, α), where α represents the network parameters of the action advantage function part. The two parts are aggregated to obtain the Q value of the corresponding action:
[0084] Q(s,a;θ0,α,β)=V(s;θ0,β)+A(s,a;θ0,α)
[0085] Through the above process, the output value of the main neural network is approximated to the target Q value, and then the optimal low-voltage load shedding emergency control strategy is obtained by selecting the action with the maximum Q value.
[0086] Compared with the existing technology, the present invention can model non-Euclidean spatial data and capture the spatiotemporal connection relationship of the data, providing a solution to the perception and decision-making problems of power system scenarios with frequently changing topologies. When dealing with power system scenarios with frequently changing topologies, the present invention is more adaptable and can provide more reliable and economical control strategies. BRIEF DESCRIPTION OF THE DRAWINGS
[0087] Figure 1 : is a schematic diagram of the improved FastGCN graph convolutional neural network structure adopted in the present invention.
[0088] Figure 2 : It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION
[0089] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.
[0090] A method for emergency control of low-voltage load shedding in a power grid based on graph deep reinforcement learning, characterized in that it comprises the following steps:
[0091] Step 1: Determine the scale of the power system calculation and set the power system topology change scenario; design the low-voltage load reduction action node, low-voltage load reduction action strategy and low-voltage load reduction strategy objective function;
[0092] The scale of the power system example in step 1 is n=500;
[0093] The topology change scenario described in step 1 is:
[0094] First, construct the power system topology consisting of vertex set V and edge set E, where v i represents the ith node v in the power system topology i , v i ∈V,i=1,2,…,500;
[0095] Secondly, multiple types of system voltage disturbances and topology change scenarios with multiple line disconnections are set. Short-circuit faults are set for important load bus nodes and other bus nodes in the example, and one connecting line between the short-circuit point bus and the adjacent line and one or two of the remaining connecting lines are disconnected.
[0096] The designed low voltage load reduction action node in step 1 is:
[0097] The low voltage load shedding action node is defined as the important load bus node of the system;
[0098] The low voltage load reduction action strategy designed in step 1 is:
[0099] The load shedding action at each action node is a low-voltage load shedding action in rounds to keep the load ratio at each node within the [0.7:1] range, including: 0 (no load shedding), 1 (5% load shedding) and 2 (10% load shedding);
[0100] The objective function of the low voltage load reduction strategy designed in step 1 is:
[0101] According to the State Grid Corporation of China's safety and stability calculation technical specifications, the voltage recovery standard time is set in seconds T st =10 and voltage recovery standard value U st =0.8, and based on the goal of achieving transient voltage stability with minimum load shedding loss, the objective function of the low-voltage load shedding strategy is established:
[0102]
[0103] If t c <t<t c +10
[0104] In the formula, is the voltage amplitude at the ith node of the power system topology at time t and restore the standard value U st =0.8 difference, It refers to the load reduction of the ith node in the power system topology at time t, and a and b are the cumulative voltage deviations of each node respectively. and the accumulated load reduction of each node The corresponding coefficient, t c Indicates the time to clear the fault.
[0105] Step 2: Collect node state observation value sample sets at multiple historical moments of the power system topology to generate a node feature matrix X and a power system topology graph G;
[0106] The node state observation value sample set of the power system topology at multiple historical moments in step 2 is specifically defined as:
[0107] For the i-th node in the power system topology, the collected observation value samples Including the node voltage amplitude and load proportion at sampling time t and its previous four historical moments, expressed as:
[0108]
[0109] In the formula, and They represent the node voltage amplitude of the ith node in the power system topology at 4, 3, 2, 1 moments before sampling time t and at sampling time t, respectively. and They represent the load proportions of the ith node in the power system topology at the 4, 3, 2, 1 moments before sampling time t and at sampling time t respectively;
[0110] For the system topology, its node state observation value sample set Obs G Including the node voltage amplitude and load proportion of each node in the topology at the sampling time t and its previous 4 historical times, expressed as:
[0111]
[0112] In the formula, and They represent the node voltage amplitudes of the first node, the second node, and the 500th node in the power system topology at 4 moments before the sampling time t,
[0113] and They represent the node voltage amplitudes of the first node, the second node, and the 500th node in the power system topology at three moments before the sampling time t,
[0114] and They represent the node voltage amplitudes of the first node, the second node, and the 500th node in the power system topology at two moments before the sampling time t,
[0115] and They represent the node voltage amplitudes of the first node, the second node, and the 500th node in the power system topology at one moment before the sampling time t,
[0116] and They represent the node voltage amplitudes of the 1st node, the 2nd node and the 500th node of the power system topology at sampling time t respectively;
[0117] and They represent the load proportions of the first, second and 500th nodes in the power system topology four moments before sampling time t,
[0118] and They represent the load proportions of the first, second and 500th nodes in the power system topology at the three moments before sampling time t,
[0119] and They represent the load proportions of the first, second and 500th nodes in the power system topology at two moments before sampling time t,
[0120] and They represent the load proportions of the first node, the second node, and the 500th node in the power system topology at one moment before the sampling time t,
[0121] and They represent the load proportions of the first node, the second node, and the 500th node in the power system topology at sampling time t respectively;
[0122] Step 2 generates the node feature matrix X, specifically:
[0123] Based on the node state observation value sample set Obs G , establish the node feature matrix X, the observation dimension is 500*10, and the specific expression is as follows:
[0124]
[0125] In the formula, and They represent the node voltage amplitude of the first node of the power system topology at 4, 3, 2, 1 moments before sampling time t and at sampling time t, respectively.
[0126] and They represent the node voltage amplitudes of the second node in the power system topology at 4, 3, 2, 1 moments before sampling time t and at sampling time t, respectively.
[0127] and They represent the node voltage amplitude of the 500th node in the power system topology at 4, 3, 2, 1 moments before sampling time t and at sampling time t respectively;
[0128] and They represent the load proportion of the first node in the power system topology at the 4th, 3rd, 2nd, 1st moments before sampling time t and sampling time t respectively.
[0129] and They represent the load proportions of the second node in the power system topology at the 4th, 3rd, 2nd, 1st moments before sampling time t and sampling time t, respectively.
[0130] and They represent the load proportions of the 500th node in the power system topology at the 4, 3, 2, 1 moments before sampling time t and at sampling time t respectively;
[0131] Step 2 generates the power system topology diagram G, specifically:
[0132] Based on the adjacency matrix A of the power system topology, a power system topology graph G containing a vertex set V and an edge set E is generated:
[0133] The adjacency matrix A is used to represent the connection relationship between nodes in the power topology. If the node v i and node v j On the same line, it is represented as 1, otherwise it is represented as 0, which is represented as follows:
[0134]
[0135] In the formula, v i 、v j They represent the ith and jth nodes in the power system topology, respectively, and v i ,v j ∈V,i,j=1,2,…,500, p is the row index of the adjacency matrix A, q is the column index of the adjacency matrix A, p,q=0,1,…,499, A p,q represents the element in the p+1th row and q+1th column of the adjacency matrix A;
[0136] Create a power system topology graph G from a given adjacency matrix A.
[0137] Step 3, preprocess the node feature matrix to obtain normalized sample feature data; introduce the FastGCN-D3QN graph deep reinforcement learning model, input the preprocessed node feature matrix into the FastGCN-D3QN graph deep reinforcement learning model to extract features and obtain effective sample features; further train through the reinforcement learning process of the FastGCN-D3QN graph deep reinforcement learning model to establish a low-voltage load reduction emergency control model for the power system;
[0138] The data preprocessing described in step 3 includes the following specific methods:
[0139] The node feature matrix X is linearly normalized row by row and mapped to the interval [0,1], so as to convert the node observation feature values to the same dimension and obtain the preprocessed node feature matrix X'. The specific normalization method is as follows:
[0140]
[0141] Where f is the row index of the node feature matrix X, g is the column index of the feature matrix X, and x' f,g is the element in the f+1th row and g+1th column of the preprocessed node feature matrix X', min(X f ) and max(X f) are the minimum and maximum values of the f+1th row of the node feature matrix X before normalization;
[0142] The FastGCN-D3QN graph deep reinforcement learning model described in step 3:
[0143] It is composed of the improved FastGCN graph convolutional neural network, the deep convolutional neural network part of D3QN deep reinforcement learning, and the reinforcement learning framework of D3QN deep reinforcement learning in cascade order;
[0144] The improved FastGCN graph convolutional neural network is used to extract power topology features from non-Euclidean spatial data;
[0145] Use D3QN's deep convolutional neural network to further process the observation value characteristics of each node;
[0146] The reinforcement learning framework based on D3QN deep reinforcement learning is used to train the model and establish an emergency control model for low-voltage load shedding in the power system.
[0147] Step 3 uses FastGCN-D3QN graph deep reinforcement learning to train and establish a low-voltage load reduction emergency control model for the power system, specifically:
[0148] Step 3.1: Input the preprocessed node feature matrix X' into the FastGCN-D3QN graph deep reinforcement learning model, use the improved FastGCN graph convolutional neural network to learn the topological structure and vertex attribute information of non-Euclidean spatial data types in the power system topology graph G, sample and aggregate the features of each node; use the deep convolutional neural network in the D3QN deep reinforcement learning to further extract the features of each node:
[0149] According to the node scale n=500 of the power system, the number of sampling network layers L=3 is set, and node feature sampling is performed through mini-batch method. First, the nodes are divided into multiple batches, and batch=60 is set. Then, 60 nodes are sampled from 500 nodes in the system topology each time. The 60 nodes correspond to the number of node feature vectors output by the last layer.
[0150] Each layer uses layer-wise sampling according to the vertex degree to sample t l Considering that the power system topology is relatively sparse (the number of edges |E| is less than |V| 2 ), the shared neighbor set will make the adjacent vertices in the mini-batch more sparse, and the model effect is not good. Here we propose an improved importance sampling method - pre-traversing the nodes to be sampled in each layer. Specifically, first, according to the actual scale of the example topology and the relationship between nodes, define the minimum acceptable value of the correlation between samples in each adjacent layer Degreemin ; Each vertex v i According to the probability distribution q(v i ) for sampling:
[0151]
[0152] In the formula, A=A+I N Add the identity matrix I to the adjacency matrix A of the undirected graph G N , v i Represents the ith node in the power system topology. The first layer extracts t1=6 nodes from 60 nodes, the second layer extracts t2=15 nodes from 60 nodes, and the last layer does not perform sampling, that is, the 6 nodes in the first layer are used to approximate the vectors of the 15 nodes in the second layer. These 6 nodes are used as the shared neighbor set of the 15 nodes in the second layer. The 15 nodes in the second layer are used as the shared neighbor set of the last layer to approximate the vectors of the 60 nodes in the entire batch, forming a sampling sample combination, and analyzing the correlation between the sampling samples of each adjacent layer in the combination. If the correlation is lower than Degree min , then eliminate the combination; repeat this step multiple times to form an optimized pre-selected sample set;
[0153] In the actual sampling process, a group of sampling sample combinations are randomly selected from the optimized pre-selected sample set for node feature collection. The principle of collection is still to select t1=6 nodes from 60 nodes in the first layer, t2=15 nodes from 60 nodes in the second layer, and no sampling is performed in the last layer.
[0154] In the basic GCN method, each convolutional layer passes through the adjacency matrix A and the feature matrix h (l) Multiply them together to get the summary of each vertex's neighbor features, and then multiply them by the parameter matrix W (l) Add activation function σ to perform a nonlinear transformation to obtain the matrix h of aggregated adjacent vertex features. (l+1) , for the improved FastGCN graph convolutional neural network, its propagation rule is expressed in integral form:
[0155] h (l+1) (v i )=∫A(v i ,v j )h (l) (v j )W (l) dP(v j ),h (l+1) =σ(h (l+1) (v i ))
[0156] In the formula, vi and v i They represent the i-th and j-th nodes in the power system topology, j≠i, i,j=1,2,…,500, h (l) is the activation unit matrix of layer l, h (0) =X,h (l+1) (v i ) represents the node v at the lth layer i The characteristic matrix, h (l) (v j ) represents the node v at the lth layer j The characteristic matrix, P(v j ) is the probability distribution of nodes. Using the Monte Carlo method, we sample t at each layer. l Vertices To approximate the integral, it can be expressed as:
[0157]
[0158] For the first layer, sample t1 = 6 vertices To approximate the integral, it can be expressed as:
[0159]
[0160] For the second layer, sample t2 = 15 vertices To approximate the integral, it can be expressed as:
[0161]
[0162] Generate feature vector representation of each node
[0163]
[0164] In the formula, v i represents the ith node v in the power system topology i , i=1,2,…,500;
[0165] The deep convolutional neural network in D3QN deep reinforcement learning is used to further extract the features of each node, and two different sets of parameters θ and θ' are selected for current state action selection and optimal strategy evaluation respectively:
[0166] Here, the reward function r is defined based on the load ratio interval [0.7:1] defined in step 1 and the low-voltage load reduction strategy objective function. t , and through multiple experiments determine the optimal combination of reward function coefficients a = 26, b = 100, c = 1000:
[0167]
[0168]
[0169] In the formula, system crash means that the system voltage does not recover to the target voltage value within the specified time, and -1000 is the huge penalty value when the system crashes. represents the load proportion of the ith node in the power system topology at sampling time t;
[0170] First, based on the parameter θ, the current value neural network is used to approximate the value function Q(s) of the current state of the topology. t ,a;θ t ), select the current state action, where Q means the power system state s at time t given the low voltage load reduction strategy t t Execute low-voltage load reduction control action a and get reward value r t , the state-action value function corresponding to the state-action pair (s, a);
[0171] Next, find the action a' corresponding to the maximum Q value max , based on another parameter θ', the target value neural network is used to calculate the parameter update Q value function and evaluate the optimal strategy a' max :
[0172] a' max = argmaxQ(s t+1 ,a;θ t )
[0173] Y t =R t+1 +γQ(s t+1 ,argmaxQ(s t+1 ,a;θ t );θ′ t )
[0174] In the formula, is the cumulative reward value starting from time t, γ∈[0,1] is the discount factor;
[0175] Step 3.2: Use the Q value obtained in step 3.1 as the input data of reinforcement learning, introduce the competitive network structure to divide the Q network into two branches, one of which is only related to the power system topology state s and has nothing to do with the low-voltage load reduction control action a to be executed, called the state value function V(s; θ0, β), where θ0 represents the convolutional network parameters of the common part, and β represents the network parameters of the state value function part; the other branch is related to both the system state s and the low-voltage load reduction action a, called the action advantage function A(s, a; θ0, α), where α represents the network parameters of the action advantage function part. The two parts are aggregated to obtain the Q value of the corresponding action:
[0176] Q(s,a;θ0,α,β)=V(s;θ0,β)+A(s,a;θ0,α)
[0177] Through the above process, the output value of the main neural network is approximated to the target Q value, and then the optimal low-voltage load shedding emergency control strategy is obtained by selecting the action with the maximum Q value.
[0178] Step 4, the multi-time node state observation values of the power grid to be tested, including the node voltage amplitude and load proportion at the operating time t and the four previous times, are collected in real time, and the real-time node feature matrix X is constructed through step 2. The real-time normalized sample feature data X' is further processed through the data preprocessing process described in step 3, and the load adjustment amount corresponding to each action bus node is further predicted through the trained power system low-voltage load reduction emergency control model, so as to form a low-voltage load reduction emergency control plan for the power grid to be predicted.
[0179] In order to verify the reliability and effectiveness of this method, under the same topology change scenario, this method and the traditional deep reinforcement learning method are used to train the low-voltage load reduction model, and the results are compared through online testing. The model trained by this method is more adaptable to system state changes when dealing with power system scenarios with frequent topology changes, and can provide more reliable and economical control strategies.
[0180] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.
Claims
1. A low-voltage load shedding emergency control method for a power grid based on graph deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Determine the scale of the power system calculation and set the power system topology change scenario; design the low-voltage load reduction action node, low-voltage load reduction action strategy and low-voltage load reduction strategy objective function; Step 2: Collect node state observation value sample sets at multiple historical moments of the power system topology to generate a node feature matrix X and a power system topology graph G; Step 3, preprocess the node feature matrix to obtain normalized sample feature data; introduce the FastGCN-D3QN graph deep reinforcement learning model, input the preprocessed node feature matrix into the FastGCN-D3QN graph deep reinforcement learning model to extract features and obtain effective sample features; further train through the reinforcement learning process of the FastGCN-D3QN graph deep reinforcement learning model to establish a low-voltage load reduction emergency control model for the power system; Step 4, the multi-time node state observation values of the power grid to be tested collected in real time, including the node voltage amplitude and load proportion at the operating time t and the previous m-1 times, are used to construct a real-time node feature matrix X through step 2, and further processed through the data preprocessing process described in step 3 to obtain real-time normalized sample feature data X', and further predicted through the trained power system low-voltage load reduction emergency control model to obtain the load adjustment amount corresponding to each action bus node, so as to form a low-voltage load reduction emergency control scheme for the power grid to be predicted; The data preprocessing described in step 3 includes the following specific methods: The node feature matrix X is linearly normalized row by row and mapped to the interval [0,1], so as to convert the node observation feature values to the same dimension and obtain the preprocessed node feature matrix X'. The specific normalization method is as follows: Where f is the row index of the node feature matrix X, g is the column index of the feature matrix X, and x' f,g is the element in the f+1th row and g+1th column of the preprocessed node feature matrix X', min(X f ) and max(X f ) are the minimum and maximum values of the f+1th row of the node feature matrix X before normalization; The FastGCN-D3QN graph deep reinforcement learning model described in step 3: It is composed of the improved FastGCN graph convolutional neural network, the deep convolutional neural network part of D3QN deep reinforcement learning, and the reinforcement learning framework of D3QN deep reinforcement learning in cascade order; The improved FastGCN graph convolutional neural network is used to extract power topology features from non-Euclidean spatial data; Use D3QN's deep convolutional neural network to further process the observation value characteristics of each node; The reinforcement learning framework based on D3QN deep reinforcement learning is used to train the model and establish an emergency control model for low-voltage load shedding in the power system. Step 3 uses FastGCN-D3QN graph deep reinforcement learning to train and establish a low-voltage load reduction emergency control model for the power system, specifically: Step 3.1: Input the preprocessed node feature matrix X' into the FastGCN-D3QN graph deep reinforcement learning model, use the improved FastGCN graph convolutional neural network to learn the topological structure and vertex attribute information of non-Euclidean spatial data types in the power system topology graph G, sample and aggregate the features of each node; use the deep convolutional neural network in the D3QN deep reinforcement learning to further extract the features of each node: The number of sampling network layers L is set according to the topological scale of the power system; node feature sampling is performed using a mini-batch method. First, the nodes are divided into multiple batches, and each layer is sampled using a layer-wise sampling method according to the degree of the vertex. l vertices; Considering that the power system topology is relatively sparse (the number of edges |E| is less than |V| 2 ), the shared neighbor set will make the adjacent vertices in the mini-batch more sparse, and the model effect is not good. Here, an improved importance sampling method is proposed - pre-traversing the nodes to be sampled in each layer; specifically, first, according to the actual scale of the example topology and the relationship between nodes, the minimum acceptable value of the correlation between samples in each adjacent layer is defined. min ; Each vertex v i According to the probability distribution q(v i ) for sampling: In the formula, A=A+I N Add the identity matrix I to the adjacency matrix A of the undirected graph G N , v i Represents the i-th node in the power system topology; each layer is based on the probability distribution q(v i ) The number of samples is t l The last layer is not sampled to form a sampling sample combination, and the correlation between the sampling samples of each adjacent layer in the combination is analyzed. If the correlation is lower than Degree min , then eliminate the combination; repeat this step multiple times to form an optimized pre-selected sample set; In the actual sampling process, a group of sampling sample combinations are randomly selected from the pre-selected sample set for node feature collection. The principle of collection is still to select a sampling sample combination for each batch, and each layer is based on q(v i ) The number of samples is t l The nodes of the last layer are not sampled; In the basic GCN method, each convolutional layer passes through the adjacency matrix A and the feature matrix h (l) Multiply them together to get the summary of each vertex's neighbor features, and then multiply them by the parameter matrix W (l) Add activation function σ to perform a nonlinear transformation to get the matrix h of aggregated adjacent vertex features (l+1) , for the improved FastGCN graph convolutional neural network, its propagation rule is expressed in integral form: h (l+1) (v i )=∫A(v i ,v j )h (l) (v j )W (l) dP(v j ),h (l+1) =σ(h (l+1) (v i )) In the formula, v i and v i They represent the i-th and j-th nodes in the power system topology, j≠i, i,j=1,2,…,n, h (l) is the activation unit matrix of layer l, h (l) is the activation unit matrix of layer l, h (0) =X,h (l+1) (v i ) represents the l-th layer node v i The characteristic matrix, h (l) (v j ) represents the l-th layer node v j The characteristic matrix, P(v j ) is the probability distribution of nodes. In this method, the Monte Carlo method is used to sample each layer t l Vertices To approximate the integral, it can be expressed as: Generate feature vector representation of each node In the formula, v i represents the ith node v in the power system topology i , i=1,2,…,n; The deep convolutional neural network in D3QN deep reinforcement learning is used to further extract the features of each node, and two different sets of parameters θ and θ' are selected for current state action selection and optimal strategy evaluation respectively: Here, the reward function r is defined based on the low-voltage load reduction strategy objective function established in step 1.
4. t : In the formula, system collapse means that the system voltage does not recover to the target voltage value within the specified time, and -c is the huge penalty value when the system collapses; First, based on the parameter θ, the current value neural network is used to approximate the value function Q(s) of the current state of the topology. t ,a;θ t ), select the current state action, where Q means the given low-voltage load reduction strategy π, the power system state s at time t t Execute low-voltage load reduction control action a and get reward value r t , the state-action value function corresponding to the state-action pair (s, a); Next, find the action a' corresponding to the maximum Q value max , based on another parameter θ', the target value neural network is used to calculate the parameter update Q value function and evaluate the optimal strategy a' max : a' max argmaxQ(s). t+1 ,a6θ t ) Y t =R t+1 +γQ(s t+1 ,argmaxQ(s t+1 ,a;θ t );θ' t ) In the formula, is the cumulative reward value starting from time t, γ∈[0,1] is the discount factor; Step 3.2: Use the Q value obtained in step 3.1 as the input data of reinforcement learning, introduce the competitive network structure to divide the Q network into two branches, one of which is only related to the power system topology state s and has nothing to do with the low-voltage load reduction control action a to be executed, called the state value function V(s; θ0, β), where θ0 represents the convolutional network parameters of the common part, and β represents the network parameters of the state value function part; the other branch is related to both the system state s and the low-voltage load reduction action a, called the action advantage function A(s, a; θ0, α), where α represents the network parameters of the action advantage function part. The two parts are aggregated to obtain the Q value of the corresponding action: Q(s,a;θ0,α,β)=V(s;θ0,β)+A(s,a;θ0,α) Through the above process, the output value of the main neural network is approximated to the target Q value, and then the optimal low-voltage load shedding emergency control strategy is obtained by selecting the action with the maximum Q value.
2. The method for emergency control of low-voltage load shedding in a power grid based on graph deep reinforcement learning according to claim 1 is characterized in that: The scale of the power system example in step 1 is n; The topology change scenario described in step 1 is: First, construct the power system topology consisting of vertex set V and edge set E, where v i represents the ith node v in the power system topology i , v i ∈V,i=1,2,…,n; Secondly, multiple types of system voltage disturbances and topology change scenarios with multiple line disconnections are set. Short-circuit faults are set for important load bus nodes and other bus nodes in the example, and one connecting line between the short-circuit point bus and the adjacent line and one or two of the remaining connecting lines are disconnected. The designed low voltage load reduction action node in step 1 is: The low voltage load shedding action node is defined as the important load bus node of the system; The low voltage load reduction action strategy designed in step 1 is: The load shedding action at each action node is a low-voltage load shedding action in rounds to keep the load ratio within the specified range; The objective function of the low voltage load reduction strategy designed in step 1 is: Set voltage recovery standard time T st And voltage recovery standard value U st , and based on the goal of achieving transient voltage stability with minimum load shedding loss, the objective function of the low-voltage load shedding strategy is established: If t c <t<t c +T st In the formula, is the voltage amplitude U at the ith node of the power system topology at time t vi (t) and the recovery standard value U st The difference, It refers to the load reduction of the ith node in the power system topology at time t, and a and b are the cumulative voltage deviations of each node respectively. and the accumulated load reduction of each node The corresponding coefficient, t c Refers to the time to clear the fault.
3. The method for emergency control of low-voltage load shedding in a power grid based on graph deep reinforcement learning according to claim 1, characterized in that: The node state observation value sample set of the power system topology at multiple historical moments in step 2 is specifically defined as: For the i-th node in the power system topology, the collected observation value samples Including the node voltage amplitude and load proportion at sampling time t and its previous m-1 historical moments, expressed as: In the formula, and They represent the node voltage amplitude of the ith node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t, respectively. and They represent the load proportions of the ith node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t respectively; For the system topology, its node state observation value sample set Obs G It includes the node voltage amplitude and load proportion of each node in the topology at sampling time t and its previous m-1 historical moments, expressed as: In the formula, and They represent the node voltage amplitudes of the first node, the second node, and the nth node in the power system topology at time m-1 before sampling time t, and They represent the node voltage amplitudes of the first node, the second node, and the nth node in the power system topology at time m-2 before sampling time t, respectively. and They represent the node voltage amplitudes of the first node, the second node and the nth node of the power system topology at sampling time t respectively; and They represent the load proportions of the first node, the second node, and the nth node in the power system topology at time m-1 before sampling time t, and They represent the load proportions of the first node, the second node, and the nth node in the power system topology at time m-2 before sampling time t, respectively. and They represent the load proportions of the first node, the second node, and the nth node in the power system topology at sampling time t respectively; Step 2 generates the node feature matrix X, specifically: Based on the node state observation value sample set Obs G , establish the node feature matrix X, the observation dimension is n*2m, which is specifically expressed as follows: In the formula, and They represent the node voltage amplitudes of the first node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t, respectively. and They represent the node voltage amplitude of the second node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t, respectively. and They represent the node voltage amplitude of the nth node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t respectively; and They represent the load proportions of the first node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and time t, respectively. and They represent the load proportions of the second node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t, respectively. and They represent the load proportions of the nth node in the power system topology at time m-1 before sampling time t, time m-2 before sampling time t, and sampling time t respectively; Step 2 generates the power system topology diagram G, specifically: Based on the adjacency matrix A of the power system topology, a power system topology graph G containing a vertex set V and an edge set E is generated: The adjacency matrix A is used to represent the connection relationship between nodes in the power topology. If the node v i and node v j On the same line, it is represented as 1, otherwise it is represented as 0, which is represented as follows: In the formula, v i 、v j They represent the ith and jth nodes in the power system topology, respectively, and v i ,v j ∈V,i,j=1,2,…,n,p is the row index of the adjacency matrix A,q is the column index of the adjacency matrix A,p,q=0,1,…,n-1,A p,q represents the element in the p+1th row and q+1th column of the adjacency matrix A; Create a power system topology graph G from a given adjacency matrix A.
Citation Information
Patent Citations
False data injection attack recognition method based on line topology analysis and power flow characteristics
CN113904786A
Deep reinforcement learning traffic signal control method combined with state prediction
CN113963555A
