A method for extracting emergency control strategies for power systems using deep reinforcement learning
By using a weighted tilt decision tree model of deep Q learning network and information gain ratio in the power system, lightweight and interpretable control strategies are extracted, and the black box and difficult interaction of artificial intelligence algorithms in the power system is solved. The effectiveness and explanatory ability of emergency control of the power system is realized.
Patent Information
- Application Number
- CN202111188349.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-12
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-10-12
AI Technical Summary
The existing power system protection and control mechanism cannot adapt to unknown changes in the power system, and the application of artificial intelligence algorithms in power systems is limited by their black box nature and difficulty in interactivity, especially in key decision-making scenarios. Strategy.
A method for extracting the emergency control strategy of deep reinforcement learning of power system is proposed. By introducing historical moment feature data of the power system node model, using the weighted tilt decision tree model of deep Q learning network and information gain ratio, the optimization training is lightweight and explainable The control strategy of the power system, combined with stochastic gradient descent and quasi-Newtonian algorithm, generates a model for emergency control of the power system.
It realizes the conversion of complex deep reinforcement learning models into lightweight and interpretable control strategies, provides good control performance, and solves the black box problem in the application of artificial intelligence technology in power systems.
Smart Images

Figure CN114004282B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the intersection field of artificial intelligence and power systems, and in particular relates to a method for extracting emergency control strategies for power systems through deep reinforcement learning. Background Art
[0002] Some major power outages that occurred around the world, such as the 2003 blackout in the United States and Canada, have caused huge social and economic losses, warning us that we urgently need to build a safer and more reliable power system. However, the current power system protection and control mechanisms are designed based on some typical scenarios in offline conditions and cannot adapt to unknown changes in the power system. At the same time, with the development of artificial intelligence (AI) technology in natural language processing technology, computer vision, autonomous driving and other fields, these technologies have also been successfully applied in power systems, such as load forecasting and new energy forecasting, ice thickness identification of transmission lines, and fast charging guidance for electric vehicles. Artificial intelligence algorithms represented by deep learning (DL) can make it easier to cope with unknown changes in the power system because of their powerful feature extraction and nonlinear mapping capabilities.
[0003] In recent years, the application of deep reinforcement learning (DRL) in autonomous driving, games, etc. has verified its advantages in solving sequential decision-making problems, including power system control problems. Many scholars have tried to solve power system preventive control, emergency control, recovery control and other problems based on DRL, and achieved good results.
[0004] However, the black-box nature and difficulty of interaction of artificial intelligence algorithms limit their application in practical scenarios, especially when it comes to some key decisions. Therefore, scholars at home and abroad have tried to use existing AI models based on concepts such as imitation learning and knowledge distillation to build a more lightweight and explainable decision-making model. Specifically, some scholars have proposed reinforcement learning strategy extraction methods based on decision trees and their variants. However, these works have only verified their feasibility in some simple game scenarios, and the research and results of related work in power system control problems have not yet been carried out. Summary of the invention
[0005] In order to solve the deficiencies of the above-mentioned background technology, the present invention proposes a method for extracting emergency control strategies of power systems through deep reinforcement learning.
[0006] The specific technical solution of the present invention is a method for extracting emergency control strategies of power systems through deep reinforcement learning, which specifically includes the following steps.
[0007] Step 1: Introduce the characteristic data of multiple historical moments of the power system node model to construct observation data;
[0008] Step 2: Introduce the deep Q learning network model, further input multiple groups of observation data into the deep Q learning network model in sequence, predict and obtain the load reduction action, and further use the stochastic gradient descent optimization algorithm for optimization training to obtain the deep reinforcement learning model for emergency control of the power system;
[0009] Step 3: Generate a data set under a specific fault scenario based on the trained deep Q learning network model;
[0010] Step 4: Under each non-leaf node in the weighted tilted decision tree model of the information gain ratio, the state-action pair data in the data set under each non-leaf node is input into the weighted tilted decision tree model of the information gain ratio, and the minimum value of the model objective function is solved by the quasi-Newton algorithm, and the optimal parameters of the model under the node are obtained, and the data set under the node is divided into a left subset and a right subset, and a left child node and a right child node are constructed, and the above steps are repeated until the algorithm termination condition is met;
[0011] Step 5: Set the strategy fidelity index, strategy actual control performance index, and model complexity index to evaluate the model performance under different hyperparameters, so as to select the optimal model according to the task requirements for emergency control of the power system;
[0012] Preferably, the observation data in step 1 is specifically defined as:
[0013] X t =[u t ,u t+1 ,...,u t+L-1 ] T
[0014] u t+l ={data t+l,p,j |1≤p≤P,1≤j≤J},l∈[0,L-1]
[0015] Among them, X t represents the tth group of observation data, t represents the starting time of the tth group of observation data, and L is a positive integer, which is the length of the observation data window. t+k represents the lth group of observation data in the tth group of observation data, that is, the observation data at the t+lth moment in the multi-node model of the power system; data t+l,p,j It represents the characteristic data of the jth type of the p-th bus node in the l-th step observation data in the t-th group of observation data, that is, the characteristic data of the j-th type of the p-th power system node at the t+l-th time in the power system multi-node model, P represents the number of power system nodes, and J is the number of node characteristics.
[0016] Preferably, the predicted load reduction action in step 2 is composed of a combination of load shedding percentages of the power system bus nodes, and each bus node has two load shedding modes, which respectively define no action and load shedding of 20% of the total load on the bus node;
[0017] The number of load shedding actions predicted by the deep Q-learning network model includes 2 H types, H is the number of controllable nodes; further, the actions contained in the action are sorted and numbered, that is, the action set is defined as:
[0018] Y=[0,1,...,y,...,2 H-1 ],y∈Ν
[0019] As a preferred method, the specific steps of generating the data set involved in step 3 are as follows: after the deep Q learning network model training is completed, for the set fault scenario, the characteristic quantity x of the power system from time t to time t+L-1 is calculated. t Rolling input DQN decision model, the decision model selects the best action y in the action set Y t , and record the model input and output data of each step, construct state-action pairs, (x t ,y t ) to complete the generation of the labeled data set. The state-action pair data set described in step 3 can be expressed as:
[0020] S={(x1,y1),(x2,y2),(x i ,y i ),...,(x N ,y N )}
[0021] Among them, (x i ,y i ) represents the i-th state-action pair in the state-action pair dataset, x i Represents the power system state quantity of the i-th state-action pair in the state-action pair data set, y i represents the control action of the i-th state-action pair in the state-action pair dataset, and N represents the number of state-action pairs in the state-action pair dataset;
[0022] Preferably, the step 4 is as follows:
[0023] Step 4.1: Set the information gain ratio of the weighted tilted decision tree model. For each non-leaf node, the input condition is the training data set S, (x i ,y i)∈S,i=1,2,3,...,M≤N, where M is the number of samples in the data set under the current node and N is the total number of samples; the maximum depth of the model is set to D and the depth of the current node is d;
[0024] The training data set under the root node is the data set S generated in step 3, and the training data sets under other non-leaf nodes are the left subsets S' obtained by dividing the training set of their parent nodes. L , right subset S' R ;
[0025] Step 4.2: Create a model root node G based on the data set S, and set the current node depth d = 0;
[0026] Step 4.3: If the current node depth d> the maximum depth D of the model, then the node G is set as a leaf node, and its label is the corresponding label k with the largest number of samples in the data set S; otherwise, go to step 4.4;
[0027] Step 4.4: If all samples in the data set S belong to the same category k, then node G is set as a leaf node with a label of k. Otherwise, go to step 4.5;
[0028] Step 4.5: Initialize the parameter θ at the current node of the model using a univariate decision tree to obtain the initial value θ0;
[0029] Step 4.6: Solve the minimum value of the model objective function based on the quasi-Newton algorithm and the initial value θ0, and obtain the optimal model parameter θ best ;
[0030]
[0031] Among them, L(θ) is the model objective function, λ is the L2 regularization term coefficient, θ is the parameter to be trained at each node of the model, ||θ|| 2 is the binary norm of θ, H(S) is the empirical entropy of sample set S, and H(S|θ) is the conditional empirical entropy of sample set S under θ;
[0032]
[0033] Among them, K is the total number of sample categories; k represents the label of the kth sample in the sample; |S k | is the number of samples of the kth class in the sample set S, |S| is the total number of samples in the sample set S;
[0034]
[0035] Among them, W L is the sum of the weights of all samples belonging to the left subset, W R is the sum of the weights of all samples belonging to the right subset, H Lis the weighted information entropy of the left child node, H R is the weighted information entropy of the right child node, M is the total number of samples in the sample set S under this node, and θ is the parameter to be trained under each node of the model;
[0036]
[0037] in, For the sample (x i ,y i ) belongs to the weight of the left child node, S L Each sample is associated with a set of weight information belonging to the left child node.
[0038]
[0039] in, For the sample (x i ,y i ) belongs to the weight of the right child node, S R Each sample is associated with a set of weight information belonging to the right child node.
[0040]
[0041] Among them, K is the total number of sample categories; k represents the label of the kth sample in the sample, is the sum of the weights of samples belonging to the left subset under k categories in the sample set, W L It is the sum of the weights of all samples in the sample set S belonging to the left subset.
[0042]
[0043] Among them, K is the total number of sample categories; k represents the label of the kth sample in the sample, is the sum of the weights of samples belonging to the right subset under k categories in the sample set; W R is the sum of the weights of all samples in the sample set S belonging to the right subset;
[0044]
[0045]
[0046] Among them, (x i ,y i ) represents the i-th sample in the sample set S, S L is the set of weight information of each sample belonging to the left subset, S R is the set of weight information of each sample belonging to the right subset;
[0047]
[0048]
[0049] in, is the weight of the i-th sample belonging to the left child node, is the weight of the i-th sample belonging to the right child node, σ(·) is the sigmoid function;
[0050]
[0051] Among them, K is the total number of sample categories; k represents the label of the kth sample in the sample, It is the sum of the weights of samples belonging to the left subset under k categories in the sample set.
[0052]
[0053] Among them, K is the total number of sample categories; k represents the label of the kth sample in the sample, It is the sum of the weights of samples belonging to the right subset under k categories in the sample set.
[0054] Step 4.7: Based on the current model parameters θ best Combine L(θ) to calculate the objective function value L0 at this time;
[0055] Step 4.8: Randomly initialize the parameter θ and repeat C times, where C is the model hyperparameter, to obtain the initial value of the parameter θ'0;
[0056] Step 4.9: Solve the equation L(θ) based on the quasi-Newton algorithm and the initial value θ'0 to obtain the optimal model parameter θ' best ;
[0057] Step 4.10: Based on the model parameters θ' obtained by the current solution best Calculate the objective function value L'0 at this time;
[0058] Step 4.11: If the objective function value L'0<L0, then the optimal model parameter θ best =θ' best , otherwise go to step 4.12;
[0059] Step 4.12: Based on the optimal parameter θ best Get the left subset and the right subset, S' L ,S' R ;in,
[0060]
[0061]
[0062] Among them, (x i ,y i) represents the i-th sample in the sample set S, is the weight of the i-th sample belonging to the left child node, S L is the set of weight information of each sample association belonging to the left subset, is the weight of the i-th sample belonging to the right child node, S R is the set of weight information of each sample belonging to the right subset;
[0063] Step 4.13: Construct the left child node G L , and let the training set under this node be S' L , d=d+1, go to step 4.3;
[0064] Step 4.14: Construct the right child node G R , and let the training set under this node be S' R , d=d+1, go to step 4.3;
[0065] Preferably, the step 5 is specifically as follows:
[0066] Step 5.1: The strategy fidelity index in step 5 is strategy fidelity, which means the decision matching degree between the weighted tilted decision tree model based on the information gain ratio and the deep reinforcement learning strategy. The calculation formula is:
[0067]
[0068] Among them, y and are the outputs of deep reinforcement learning and weighted tilted decision tree models based on information gain ratio under the same input x, respectively. N is the total number of samples, and I(·) is the indicative function.
[0069] Step 5.2: The actual control performance index of the strategy in step 5 is the actual control performance of the strategy, which means the average return obtained in each round when the weighted tilted decision tree strategy based on information gain ratio is applied to the actual control scenario. When the deep reinforcement learning model is applied to the actual control scenario, the average return obtained in each round is recorded as: r e Correspondingly, the average return obtained by the weighted tilted decision tree based on the information gain ratio in the corresponding scenario is recorded as: r' e Therefore, R e =r' e -r e >0 means that the control performance of the weighted tilted decision tree model based on information gain ratio is better than that of the deep reinforcement learning model, and vice versa.
[0070] Step 5.3: The model complexity index described in step 5 is the model complexity, which is measured by the number of model parameters or model depth. From the perspective of model interpretability and interactivity, we need to find a weighted tilted decision tree strategy based on information gain ratio with the smallest model complexity possible.
[0071] Step 5.4: After comprehensively considering the indicator results from step 5.1 to step 5.3, select the weighted tilted decision tree model with the optimal information gain ratio according to actual needs;
[0072] The advantage of the present invention is that it can well extract the complex deep reinforcement learning model into a lightweight control strategy with certain interpretability, and provide good control performance, solving the problem that artificial intelligence technology is difficult to apply in practice due to its black box nature. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] Figure 1 : It is the workflow diagram of emergency control strategy extraction by deep reinforcement learning for power system;
[0074] Figure 2 : It is the IEEE39 node system topology diagram;
[0075] Figure 3 : This is the algorithm pseudo code diagram of the weighted tilted decision tree model based on information gain ratio; DETAILED DESCRIPTION
[0076] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0077] This embodiment is based on the low-voltage load shedding problem on the IEEE39-node system, and extracts strategies for the deep reinforcement learning agent used for low-voltage load shedding. The complex deep reinforcement learning strategy is converted into a lighter strategy in the form of a weighted tilted decision tree model with a certain interpretable information gain ratio. The effectiveness and advancement of the proposed method are evaluated through three indicators: strategy fidelity, actual control performance of the strategy, and model complexity.
[0078] Combine the following Figures 1 to 3 The embodiments of the present invention are described in detail in the following. Figure 1 The specific implementation of the present invention is a method for extracting emergency control strategies through deep reinforcement learning for power systems, which specifically includes the following steps:
[0079] Step 1: As attached Figure 1 As shown in step 1, the characteristic data of multiple historical moments of the power system multi-node model are introduced to construct the observation data.
[0080] The observation data described in step 1 is specifically defined as:
[0081] X t =[u t ,u t+1 ,...,u t+L-1 ] T
[0082] u t+l ={data t+l,p,j |1≤p≤P,1≤j≤J},l∈[0,L-1]
[0083] Among them, X t represents the tth group of observation data, t represents the starting time of the tth group of observation data, and L is a positive integer, which is the length of the observation data window. t+k represents the lth group of observation data in the tth group of observation data, that is, the observation data at the t+lth moment in the multi-node model of the power system; data t+l,p,j It represents the characteristic data of the jth type of the p-th bus node in the l-th step observation data in the t-th group of observation data, that is, the characteristic data of the j-th type of the p-th power system node at the t+l-th time in the power system multi-node model, P represents the number of power system nodes, and J is the number of node characteristics.
[0084] like Figure 2 As shown, the observations of the intelligent agent are set as: the per unit voltage of the high and low voltage sides of buses 4, 7, 8, and 18, and the load margin of buses 4 and 7, P = 4, J 母线4 =J 母线7 =3, J 母线8 =J 母线18 =2. To capture the trend of feature changes, the feature quantities of the last five simulation steps are superimposed as the final input quantity of the agent, x. That is: P = 4, J 母线4 =J 母线7 =3, J 母线8 =J 母线18 =2, L = 5, then, x t =[u t-4 ,u t-3 ,...u t ].
[0085] Step 2: As attached Figure 1 As shown in step 2, a deep Q learning network model is introduced, and multiple groups of observation data are further input into the deep Q learning network model in sequence to predict and obtain load reduction actions. The stochastic gradient descent optimization algorithm is further used for optimization training to obtain a deep reinforcement learning model for emergency control of the power system.
[0086] The predicted load reduction action in step 2 is composed of a combination of load shedding percentages at the power system bus nodes. Each bus node has two load reduction modes, which respectively define no action and load shedding of 20% of the total load on the bus node.
[0087] The number of load shedding actions predicted by the deep Q-learning network model includes 2 H types, H is the number of controllable nodes; further, the actions contained in the action are sorted and numbered, that is, the action set is defined as:
[0088] Y=[0,1,...,y,...,2 H-1 ],y∈Ν
[0089] Here, the operable buses are set to buses 4 and 7, that is, H = 2, so: Y = [0, 1, 2, 3].
[0090] Step 3: As attached Figure 1 As shown in step 3, based on the trained deep Q learning network model, a data set is generated under a specific fault scenario.
[0091] The specific steps for generating the data set involved in step 3 are as follows: After the deep Q learning network model is trained, for the set fault scenario, the characteristic quantity x of the power system from time t to time t+4 is calculated. t Rolling input DQN decision model, the decision model selects the best action y in the action set [0,1,2,3] t , and record the model input and output data of each step, construct state-action pairs, (x t ,y t ) to complete the generation of the labeled data set. The state-action pair data set described in step 3 can be expressed as:
[0092] S={(x1,y1),(x2,y2),(x i ,y i ),...,(x N ,y N )}
[0093] Among them, (x i ,y i ) represents the i-th state-action pair in the state-action pair dataset, x i Represents the power system state quantity of the i-th state-action pair in the state-action pair data set, y i represents the control action of the ith state-action pair in the state-action pair data set, and N represents the number of state-action pairs in the state-action pair data set. Here, N=4836.
[0094] Step 4: As attached Figure 1As shown in step 4, under each non-leaf node in the weighted tilted decision tree model of information gain ratio, the state-action pair data in the data set under each non-leaf node is input into the weighted tilted decision tree model of information gain ratio, and the minimum value of the model objective function is solved by the quasi-Newton algorithm, and the optimal parameters of the model under the node are obtained, and the data set under the node is divided into a left subset and a right subset, and a left child node and a right child node are constructed, and the above steps are repeated until the algorithm termination condition is met;
[0095] As attached Figure 3 As shown in the pseudo code, step 4 is as follows:
[0096] Step 4.1: Set the information gain ratio of the weighted tilted decision tree model. For each non-leaf node, the input condition is the training data set S, (x i ,y i )∈S,i=1,2,3,...,M≤N, where M is the number of samples in the data set under the current node, N is the total number of samples, here N=4836; set the maximum depth of the model to D, here it is set to D∈{3,4,5,6,7,8}, and the current node depth is d;
[0097] The training data set under the root node is the data set S generated in step 3, and the training data sets under other non-leaf nodes are the left subsets S' obtained by dividing the training set of their parent nodes. L , right subset S' R ;
[0098] Step 4.2: Create a model root node G based on the data set S, and set the current node depth d = 0;
[0099] Step 4.3: If the current node depth d> the maximum depth D of the model, then the node G is set as a leaf node, and its label is the corresponding label k with the largest number of samples in the data set S; otherwise, go to step 4.4;
[0100] Step 4.4: If all samples in the data set S belong to the same category k, then node G is set as a leaf node with a label of k. Otherwise, go to step 4.5;
[0101] Step 4.5: Initialize the parameter θ at the current node of the model using a univariate decision tree to obtain the initial value θ0;
[0102] Step 4.6: Solve the minimum value of the model objective function based on the quasi-Newton algorithm and the initial value θ0, and obtain the optimal model parameter θ best ;
[0103]
[0104] Among them, L(θ) is the model objective function, λ is the L2 regularization term coefficient, which is set here as: λ = 0.0001; θ is the parameter to be trained at each node of the model, ||θ|| 2 is the binary norm of θ, H(S) is the empirical entropy of sample set S, and H(S|θ) is the conditional empirical entropy of sample set S under θ;
[0105]
[0106] Where K is the total number of sample categories, here, K = 4; k represents the label of the kth sample in the sample; |S k | is the number of samples of the kth class in the sample set S, |S| is the total number of samples in the sample set S;
[0107]
[0108] Among them, W L is the sum of the weights of all samples belonging to the left subset, W R is the sum of the weights of all samples belonging to the right subset, H L is the weighted information entropy of the left child node, H R is the weighted information entropy of the right child node, M is the total number of samples in the sample set S under this node, and θ is the parameter to be trained under each node of the model;
[0109]
[0110] in, For the sample (x i ,y i ) belongs to the weight of the left child node, S L Each sample is associated with a set of weight information belonging to the left child node.
[0111]
[0112] in, For the sample (x i ,y i ) belongs to the weight of the right child node, S R Each sample is associated with a set of weight information belonging to the right child node.
[0113]
[0114] Where K is the total number of sample categories, here, K = 4; k represents the label of the kth sample in the sample, is the sum of the weights of samples belonging to the left subset under k categories in the sample set, W L It is the sum of the weights of all samples in the sample set S belonging to the left subset.
[0115]
[0116] Where K is the total number of sample categories, here, K = 4; k represents the label of the kth sample in the sample, is the sum of the weights of samples belonging to the right subset under k categories in the sample set; W R is the sum of the weights of all samples in the sample set S belonging to the right subset;
[0117]
[0118]
[0119] Among them, (x i ,y i ) represents the i-th sample in the sample set S, S L is the set of weight information of each sample belonging to the left subset, S R is the set of weight information of each sample belonging to the right subset;
[0120]
[0121]
[0122] in, is the weight of the i-th sample belonging to the left child node, is the weight of the i-th sample belonging to the right child node, σ(·) is the sigmoid function;
[0123]
[0124] Where K is the total number of sample categories, here, K = 4; k represents the label of the kth sample in the sample, It is the sum of the weights of samples belonging to the left subset under k categories in the sample set.
[0125]
[0126] Where K is the total number of sample categories, here, K = 4; k represents the label of the kth sample in the sample, It is the sum of the weights of samples belonging to the right subset under k categories in the sample set.
[0127] Step 4.7: Based on the current model parameters θ best Combine L(θ) to calculate the objective function value L0 at this time;
[0128] Step 4.8: Randomly initialize the parameter θ and repeat C times, where C is the model hyperparameter, which is set to C=3 here, and obtain the initial value of the parameter θ'0;
[0129] Step 4.9: Solve the equation L(θ) based on the quasi-Newton algorithm and the initial value θ'0 to obtain the optimal model parameter θ' best ;
[0130] Step 4.10: Based on the model parameters θ' obtained by the current solution best Calculate the objective function value L'0 at this time;
[0131] Step 4.11: If the objective function value L'0<L0, then the optimal model parameter θ best =θ' best , otherwise go to step 4.12;
[0132] Step 4.12: Based on the optimal parameter θ best Get the left subset and the right subset, S' L ,S' R ;in,
[0133]
[0134]
[0135] Among them, (x i ,y i ) represents the i-th sample in the sample set S, is the weight of the i-th sample belonging to the left child node, S L is the set of weight information of each sample association belonging to the left subset, is the weight of the i-th sample belonging to the right child node, S R is the set of weight information of each sample belonging to the right subset;
[0136] Step 4.13: Construct the left child node G L , and let the training set under this node be S' L , d=d+1, go to step 4.3;
[0137] Step 4.14: Construct the right child node G R , and let the training set under this node be S' R , d=d+1, go to step 4.3;
[0138] Step 5: As attached Figure 1 As shown in step 5, set the strategy fidelity index, strategy actual control performance index, and model complexity index to evaluate the model performance under different hyperparameters, so as to select the optimal model according to actual needs;
[0139] The step 5 is specifically as follows:
[0140] Step 5.1: The strategy fidelity index in step 5 is strategy fidelity, which means the decision matching degree between the weighted tilted decision tree model based on the information gain ratio and the deep reinforcement learning strategy. The calculation formula is:
[0141]
[0142] Among them, y and are the outputs of deep reinforcement learning and weighted tilted decision tree models based on information gain ratio under the same input x, respectively. N is the total number of samples, and I(·) is the indicative function.
[0143] Step 5.2: The actual control performance index of the strategy in step 5 is the actual control performance of the strategy, which means the average return obtained in each round when the weighted tilted decision tree strategy based on information gain ratio is applied to the actual control scenario. When the deep reinforcement learning model is applied to the actual control scenario, the average return obtained in each round is recorded as: r e Correspondingly, the average return obtained by the weighted tilted decision tree based on the information gain ratio in the corresponding scenario is recorded as: r' e Therefore, R e =r' e -r e >0 means that the control performance of the weighted tilted decision tree model based on information gain ratio is better than that of the deep reinforcement learning model, and vice versa.
[0144] Step 5.3: The model complexity index described in step 5 is the model complexity, which is measured by the number of model parameters or model depth. From the perspective of model interpretability and interactivity, we need to find a weighted tilted decision tree strategy based on information gain ratio with the smallest model complexity possible.
[0145] Step 5.4: After comprehensively considering the indicator results from step 5.1 to step 5.3, select the weighted tilted decision tree model with the optimal information gain ratio according to actual needs;
[0146] Here, the results of each indicator are:
[0147]
[0148] Therefore, when the policy fidelity is not much different from the actual control performance of the policy, we choose a model with lower model complexity, that is, a weighted tilted decision tree model based on information gain ratio with a model depth of 3.
[0149] It should be understood that parts not elaborated in detail in this specification belong to the prior art.
[0150] Although the specific embodiments of the present invention are described above in conjunction with the accompanying drawings, it should be understood by those skilled in the art that these are only examples, and that various variations or modifications may be made to these embodiments without departing from the principles and essence of the present invention. The scope of the present invention is limited only by the appended claims.
Claims
1. A method for extracting emergency control strategies for power systems using deep reinforcement learning. It is characterized in that The steps include: Step 1: Introduce the characteristic data of multiple historical moments of the power system node model to construct observation data; Step 2: Introduce the deep Q learning network model, further input multiple groups of observation data into the deep Q learning network model in sequence, predict and obtain the load reduction action, and further use the stochastic gradient descent optimization algorithm for optimization training to obtain the deep reinforcement learning model for emergency control of the power system; Step 3: Generate a data set under a specific fault scenario based on the trained deep Q learning network model; Step 4: Under each non-leaf node in the weighted tilted decision tree model of the information gain ratio, the state-action pair data in the data set under each non-leaf node is input into the weighted tilted decision tree model of the information gain ratio, and the minimum value of the model objective function is solved by the quasi-Newton algorithm, and the optimal parameters of the model under the node are obtained, and the data set under the node is divided into a left subset and a right subset, and a left child node and a right child node are constructed, and the above steps are repeated until the algorithm termination condition is met; Step 5: Set the strategy fidelity index, strategy actual control performance index, and model complexity index to evaluate the model performance under different hyperparameters, so as to select the optimal model according to the task requirements for emergency control of the power system; The step 4 is specifically as follows: Step 4.1: Set the information gain ratio of the weighted tilted decision tree model. For each non-leaf node, the input condition is the training data set S, (x i ,y i )∈S,i=1,2,3,...,M≤N, where M is the number of samples in the data set under the current node and N is the total number of samples; the maximum depth of the model is set to D and the depth of the current node is d; The training data set under the root node is the data set S generated in step 3, and the training data sets under other non-leaf nodes are the left subsets S' obtained by dividing the training set of their parent nodes. L , right subset S' R ; Step 4.2: Create a model root node G based on the data set S, and set the current node depth d = 0; Step 4.3: If the current node depth d> the maximum depth D of the model, then the node G is set as a leaf node, and its label is the corresponding label k with the largest number of samples in the data set S; otherwise, go to step 4.4; Step 4.4: If all samples in the data set S belong to the same category k, then node G is set as a leaf node with a label of k; otherwise, go to step 4.5; Step 4.5: Initialize the parameter θ at the current node of the model using a univariate decision tree to obtain the initial value θ 0 ; Step 4.6: Based on the quasi-Newton algorithm and the initial value θ 0 Solve the minimum value of the model objective function and obtain the optimal model parameter θ best ; Among them, L(θ) is the model objective function, λ is the L2 regularization term coefficient, θ is the parameter to be trained at each node of the model, ||θ|| 2 is the binary norm of θ, H(S) is the empirical entropy of sample set S, and H(S|θ) is the conditional empirical entropy of sample set S under θ; Among them, K is the total number of sample categories; k represents the label of the kth sample in the sample; |S k | is the number of samples of the kth class in the sample set S, |S| is the total number of samples in the sample set S; Among them, W L is the sum of the weights of all samples belonging to the left subset, W R is the sum of the weights of all samples belonging to the right subset, H L is the weighted information entropy of the left child node, H R is the weighted information entropy of the right child node, M is the total number of samples in the sample set S under this node, and θ is the parameter to be trained under each node of the model; in, For the sample (x i ,y i ) belongs to the weight of the left child node, S L Each sample is associated with a set of weight information belonging to the left child node; in, For the sample (x i ,y i ) belongs to the weight of the right child node, S R Each sample is associated with a set of weight information belonging to the right child node; Among them, K is the total number of sample categories; k represents the label of the kth sample in the sample, is the sum of the weights of samples belonging to the left subset under k categories in the sample set, W L is the sum of the weights of all samples in the sample set S belonging to the left subset; Among them, K is the total number of sample categories; k represents the label of the kth sample in the sample, is the sum of the weights of samples belonging to the right subset under k categories in the sample set; W R is the sum of the weights of all samples in the sample set S belonging to the right subset; Among them, (x i ,y i ) represents the i-th sample in the sample set S, S L is the set of weight information of each sample belonging to the left subset, S R is the set of weight information of each sample belonging to the right subset; in, is the weight of the i-th sample belonging to the left child node, is the weight of the i-th sample belonging to the right child node, σ(·) is the sigmoid function; Among them, K is the total number of sample categories; k represents the label of the kth sample in the sample, is the sum of the weights of samples belonging to the left subset under k categories in the sample set; Among them, K is the total number of sample categories; k represents the label of the kth sample in the sample, is the sum of the weights of samples belonging to the right subset under k categories in the sample set; Step 4.7: Based on the current model parameters θ best Combined with L(θ) to calculate the objective function value L at this time 0 ; Step 4.8: Randomly initialize the parameter θ and repeat C times, where C is the model hyperparameter, to get the initial value of the parameter θ 0 '; Step 4.9: Based on the quasi-Newton algorithm and the initial value θ' 0 Solve the formula L(θ) to get the optimal model parameter θ' best ; Step 4.10: Based on the model parameters θ' obtained by the current solution best Calculate the objective function value L' at this time 0 ; Step 4.11: If the objective function value L' 0 <L 0 , then the optimal model parameter θ best =θ' best , otherwise go to step 4.12; Step 4.12: Based on the optimal parameter θ best Get the left subset and the right subset, S' L ,S' R ;in, Among them, (x i ,y i ) represents the i-th sample in the sample set S, is the weight of the i-th sample belonging to the left child node, S L is the set of weight information of each sample association belonging to the left subset, is the weight of the i-th sample belonging to the right child node, S R is the set of weight information of each sample belonging to the right subset; Step 4.13: Construct the left child node G L , and let the training set under this node be S' L , d=d+1, go to step 4.3; Step 4.14: Construct the right child node G R , and let the training set under this node be S' R , d=d+1, go to step 4.
3.
2. The method for extracting emergency control strategies for power systems using deep reinforcement learning according to claim 1, It is characterized in that The observation data described in step 1 is specifically defined as: X t =[u t ,u t+1 ,...,u t+L-1 ] T u t+l ={data t+l,p,j |1≤p≤P,1≤j≤J},l∈[0,L-1] Among them, X t represents the tth group of observation data, t represents the starting time of the tth group of observation data, L is a positive integer, and is the length of the observation data window; u t+k represents the lth group of observation data in the tth group of observation data, that is, the observation data at the t+lth moment in the multi-node model of the power system; data t+l,p,j It represents the characteristic data of the jth type of the p-th bus node in the l-th step observation data in the t-th group of observation data, that is, the characteristic data of the j-th type of the p-th power system node at the t+l-th time in the power system multi-node model, P represents the number of power system nodes, and J is the number of node characteristics.
3. The method for extracting emergency control strategies for power systems using deep reinforcement learning according to claim 1, It is characterized in that The predicted load reduction action in step 2 is composed of a combination of load shedding percentages at the power system bus nodes. Each bus node has two load shedding modes, which respectively define no action and 20% of the total load on the bus node. The number of load shedding actions predicted by the deep Q-learning network model includes 2 H types, H is the number of controllable nodes; further, the actions contained in the action are sorted and numbered, that is, the action set is defined as: And=[0,1,...,y,...,2 H-1 ], and∈N。 4. The method for extracting emergency control strategies for power systems using deep reinforcement learning according to claim 1, It is characterized in that The specific steps for generating the data set involved in step 3 are as follows: After the deep Q learning network model is trained, for the set fault scenario, the characteristic quantity x of the power system from time t to time t+L-1 is calculated. t Rolling input DQN decision model, the decision model selects the best action y in the action set Y t , and record the model input and output data of each step, construct state-action pairs, (x t ,y t ) to complete the generation of the labeled data set; the state-action pair data set described in step 3 can be expressed as: S={(x 1 ,and 1 ),(x 2 ,and 2 ),(x i ,and i ),...,(x N ,and N )} Among them, (x i ,y i ) represents the i-th state-action pair in the state-action pair dataset, x i Represents the power system state quantity of the i-th state-action pair in the state-action pair data set, y i represents the control action of the i-th state-action pair in the state-action pair dataset, and N represents the number of state-action pairs in the state-action pair dataset.
5. The method for extracting emergency control strategies for power systems using deep reinforcement learning according to claim 1, It is characterized in that The step 5 is specifically as follows: Step 5.1: The strategy fidelity index in step 5 is strategy fidelity, which means the decision matching degree between the weighted tilted decision tree model based on the information gain ratio and the deep reinforcement learning strategy. The calculation formula is: Among them, y and are the outputs of deep reinforcement learning and weighted tilted decision tree models based on information gain ratio under the same input x, respectively. N is the total number of samples, and I(·) is the indicative function. Step 5.2: The actual control performance index of the strategy in step 5 is the actual control performance of the strategy, which means that when the weighted tilted decision tree strategy based on information gain ratio is applied to the actual control scenario, the average return obtained in each round; when the deep reinforcement learning model is applied to the actual control scenario, the average return obtained in each round is recorded as: r e , correspondingly, the average return obtained by the weighted tilted decision tree based on the information gain ratio in the corresponding scenario is recorded as: r e '; Therefore, R e =r e '-r e >0 means that the control performance of the weighted tilted decision tree model based on information gain ratio is better than that of the deep reinforcement learning model, and vice versa; Step 5.3: The model complexity index described in step 5 is model complexity, which is measured by the number of model parameters or model depth. From the perspective of model interpretability and interactivity, we need to find a weighted tilted decision tree strategy based on information gain ratio with the smallest possible model complexity. Step 5.4: After comprehensively considering the indicator results of steps 5.1 to 5.3, select the weighted tilted decision tree model with the optimal information gain ratio according to actual needs.
Citation Information
Patent Citations
Power grid emergency control method based on expert system and deep reverse reinforcement learning
CN111179121A
Decision tree generation method for understanding deep learning model decision mechanism
CN111553389A
Power system high-dimensional static security domain boundary fitting method
CN115688572A
