Power system regulation and control method and device based on reinforcement learning and data driving

Abstract

The invention provides an electric power system regulation and control method and device based on reinforcement learning and data driving, and the method comprises the steps: constructing a target variable set and a control variable set based on an operation target and a schedulable variable of an electric power system; constructing an instant reward set, and constructing an action value function through a target network of a preset deep reinforcement learning algorithm; iteratively updating the action value function through an evaluation network of a preset deep reinforcement learning algorithm until the action value function meets a preset training requirement; constructing an initial knowledge base; and performing clustering division on the initial knowledge base through a knowledge extraction algorithm to obtain a plurality of subclass knowledge matrixes, adjusting the control variable set, and regulating and controlling the power system. According to the method, the problem of poor regulation and control accuracy of the power system caused by the defects of state space dimension explosion, insufficient data quality and expandability, poor system regulation and control interpretability, poor generalization ability and the like in the prior art is solved. The intelligent scheduling efficiency and accuracy of the power system can be improved.

Description

Technical Field

[0001] The present invention relates to the field of power system regulation, and in particular to a power system regulation method and device based on reinforcement learning and data driving. Background Art

[0002] With the widespread access of new energy sources (such as solar energy and wind energy) to the power system, the power load has increased dramatically, and the grid structure has become complex. In order to better adapt to the limitations of grid operation and the complexity of grid structure brought about by the large-scale access of new energy sources, many intelligent power system operation optimization strategies have been proposed.

[0003] At present, the traditional existing technology integrates structured data with unstructured semantic knowledge adversarially to construct a multimodal heterogeneous knowledge graph, builds a knowledge graph anomaly detection model corresponding to the typical fault mapping based on observational learning theory to judge the fault state, and uses the reinforcement learning framework to adjust the fault detection strategy in real time; through the technical solution of evaluating the expected effects of different fault handling solutions, the operation of the power grid is optimized; there are still some problems with the existing technology: on the one hand, the design of the state space and action space needs to fully consider the complexity and uncertainty of the power system to ensure that the intelligent agent can learn effective strategies; on the other hand, the definition of the reward function needs to accurately reflect the operation goals and constraints of the power system to guide the intelligent agent to learn in the right direction; and due to the limitations of technology, equipment and network conditions, it is often difficult to ensure the real-time and accuracy of the power system control data. In summary, the existing technology still has defects such as state space dimension explosion, insufficient data quality and scalability, poor interpretability and generalization ability of system control, resulting in the problem of poor accuracy of power system control. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a power system control method and device based on reinforcement learning and data-driven, which can effectively deal with the problem of state space dimensional explosion, improve data quality and utilization, and enhance the real-time and scalability of system control.

[0005] To achieve the above objectives, an embodiment of the present invention provides a power system control method based on reinforcement learning and data-driven, comprising:

[0006] Based on the operation objectives and dispatchable variables of the power system, construct the target variable set and the control variable set;

[0007] Build an instant reward set based on the output level of the power system;

[0008] Based on the instant reward set, construct an action value function through a target network of a preset deep reinforcement learning algorithm;

[0009] Iteratively updating the action value function through an evaluation network of a preset deep reinforcement learning algorithm until the action value function meets the preset training requirements, thereby obtaining the optimal control amount and the optimal scheduling knowledge;

[0010] Building an initial knowledge base based on the optimal control quantity, optimal scheduling knowledge, target variable set and control variable set;

[0011] Clustering the initial knowledge base using a knowledge extraction algorithm to obtain several sub-class knowledge matrices;

[0012] Based on the several sub-category knowledge matrices, a set of control variables is adjusted and the power system is regulated.

[0013] The embodiment of the present invention proposes a power system control method based on reinforcement learning and data-driven, which uses multi-source heterogeneous data such as the operation objectives and dispatchable variables of the power system to construct a target variable set and a control variable set to construct a system simulation image, improve data utilization and adaptability, and ensure the accuracy of system control in different data scenarios; based on the instant reward and cumulative reward mechanism of reinforcement learning, the technical and economic performance of the power system operation is comprehensively evaluated to ensure the effectiveness and optimization of intelligent scheduling decisions; the action value function is abstracted by reinforcement learning technology, and the state space dimension explosion and data quality problems are effectively solved; by constructing a knowledge base and a knowledge extraction algorithm, the interpretability of system control is enhanced, so that each decision can be traced and understood, and the transparency and reliability of the decision are improved; the data is abstracted into a knowledge matrix through knowledge transfer technology and the control variable set is adjusted to control the system, so as to improve the generalization ability of system control in different scenarios and goals, so that existing knowledge can be effectively applied and migrated in new scenarios; thus, the present invention can effectively improve the efficiency, accuracy and reliability of intelligent scheduling of the power system, and realize the intelligent operation, scheduling and management of the power system.

[0014] Furthermore, the target variable set and the control variable set are constructed based on the operation target and dispatchable variables of the power system, including:

[0015] Construct a target variable set based on the load of each node in the power system, the voltage of the load node, the network loss rate and the load rate of the transmission unit;

[0016] Based on the dispatchable variables corresponding to the target variable set in the power system, a control variable set is constructed; the dispatchable variables corresponding to the target variable set include: the active power of the generator, the node voltage of the generator, the reactive power of the reactive compensation device and the switching of the components.

[0017] Furthermore, the instant reward set is constructed based on the output level of the power system, including:

[0018] Set the initial state based on the historical operation data of the power system;

[0019] Based on the initial state, the output level of the power system is discretized into a number of actions to obtain an action set;

[0020] Constructing convergence performance evaluation, penalty for exceeding limits, and loss of exploration efficiency based on the action set and the initial state;

[0021] Based on the convergence performance evaluation, the penalty for exceeding the limit and the loss of exploration efficiency, an immediate reward set is constructed.

[0022] Furthermore, the step of constructing an action value function based on the instant reward set by presetting a target network of a deep reinforcement learning algorithm includes:

[0023] Obtain the training parameters of the target network of the preset deep reinforcement learning algorithm;

[0024] Construct an action-value function using the immediate reward set, the parameters to be trained of the target network, and the initial state.

[0025] Furthermore, the action value function is iteratively updated by the evaluation network of the preset deep reinforcement learning algorithm until the action value function meets the preset training requirements, and the optimal control amount and optimal scheduling knowledge are obtained, including:

[0026] The estimated network parameters are updated through an evaluation network of a preset deep reinforcement learning algorithm to obtain an updated evaluation network;

[0027] Assigning the updated evaluation network weight value to the target network to obtain an updated target network;

[0028] Based on the updated target network, the action value function is iteratively updated until the action value function converges to obtain the optimal control amount and the optimal scheduling knowledge.

[0029] Furthermore, the initial knowledge base is constructed based on the optimal control quantity, optimal scheduling knowledge, target variable set and control variable set, including:

[0030] Constructing column vectors corresponding to the optimal control quantity, optimal scheduling knowledge, target variable set and control variable set to obtain a column vector set;

[0031] The column vector sets are merged to obtain an initial knowledge base.

[0032] Furthermore, the initial knowledge base is clustered and divided by the knowledge extraction algorithm to obtain several sub-class knowledge matrices, including:

[0033] Clustering and dividing the elements of each column vector in the initial knowledge base by a preset clustering algorithm to construct a plurality of sub-matrices;

[0034] The knowledge extraction algorithm is used to extract knowledge from the plurality of sub-matrices to obtain a plurality of sub-class knowledge matrices.

[0035] Furthermore, the step of extracting knowledge from the plurality of sub-matrices by a knowledge extraction algorithm to obtain a plurality of sub-class knowledge matrices further includes:

[0036] Based on the target variable set corresponding to each subclass knowledge matrix, the first cluster center and the relative standard deviation of the target set are calculated;

[0037] Based on the control variable set corresponding to each subclass knowledge matrix, the second cluster center and the relative standard deviation of the control set are calculated;

[0038] Based on the first cluster center, the target set relative standard deviation, the second cluster center and the control set relative standard deviation, knowledge extraction is performed on the plurality of sub-matrices through a knowledge extraction algorithm to obtain a plurality of sub-class knowledge matrices.

[0039] Furthermore, based on the plurality of sub-category knowledge matrices, the control variable set is adjusted and the power system is regulated, including:

[0040] Based on several sub-class knowledge matrices, the control variable set is adjusted through the optimal control quantity and optimal dispatch knowledge to obtain the power system adjustment strategy;

[0041] Based on the power system adjustment strategy, the dispatchable variables of the power system are regulated; wherein the dispatchable variables include: the active power of the generator, the node voltage of the generator, the reactive power of the reactive compensation device and the switching of the components.

[0042] The embodiment of the present invention also provides a power system control device based on reinforcement learning and data-driven, comprising: a variable set construction module, a reward set construction module, an action value function construction module, an action value function iterative update module, a knowledge base construction module, a knowledge matrix partitioning module and a power system control module;

[0043] The variable set building module is used to build a target variable set and a control variable set based on the operation target and dispatchable variables of the power system;

[0044] The reward set building module is used to build an instant reward set based on the output level of the power system;

[0045] The action value function construction module is used to construct an action value function based on the instant reward set through a target network of a preset deep reinforcement learning algorithm;

[0046] The action value function iterative update module is used to iteratively update the action value function through an evaluation network of a preset deep reinforcement learning algorithm until the action value function meets the preset training requirements and obtains the optimal control amount and the optimal scheduling knowledge;

[0047] The knowledge base construction module is used to construct an initial knowledge base based on the optimal control quantity, optimal scheduling knowledge, target variable set and control variable set;

[0048] The knowledge matrix partitioning module is used to perform clustering partitioning on the initial knowledge base through a knowledge extraction algorithm to obtain a plurality of sub-class knowledge matrices;

[0049] The power system control module is used to adjust the control variable set and control the power system based on the plurality of sub-category knowledge matrices.

[0050] The embodiment of the present invention proposes a power system control device based on reinforcement learning and data-driven. Through the variable set construction module, multi-source heterogeneous data such as the operation objectives and dispatchable variables of the power system are used to construct the target variable set and the control variable set to construct the system simulation image, thereby improving the data utilization and adaptability, and ensuring the accuracy of the system control in different data scenarios; the reward set construction module is based on the instant reward and cumulative reward mechanism of reinforcement learning to comprehensively evaluate the technical and economic performance of the power system operation, thereby ensuring the effectiveness and optimization of the intelligent dispatching decision; the action value function is abstracted by the reinforcement learning technology through the action value function construction module and the action value function iterative update module, and the action value function is effectively solved. The problem of state space dimension explosion and data quality is solved; the knowledge base construction module enhances the interpretability of system control by constructing a knowledge base and a knowledge extraction algorithm, so that each decision can be traced and understood, and the transparency and reliability of the decision are improved; the knowledge matrix division module and the power system control module also abstract the data into the knowledge matrix through the knowledge transfer technology and adjust the control variable set to control the system, thereby improving the generalization ability of system control under different scenarios and goals, so that the existing knowledge can be effectively applied and transferred in new scenarios; thus, the present invention can effectively improve the efficiency, accuracy and reliability of intelligent dispatching of the power system, and realize the intelligent operation, dispatching and management of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the steps of a power system control method based on reinforcement learning and data-driven provided in an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a power dispatching model architecture of a power system control method based on reinforcement learning and data-driven provided in a certain embodiment of the present invention;

[0053] Figure 3A flowchart of learning and training steps of a power system control method based on reinforcement learning and data-driven provided in an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of the module structure of a power system control device based on reinforcement learning and data-driven is provided for a certain embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Reinforcement learning technology, with the help of the dynamic interaction and learning mechanism of intelligent agents in a simulated environment, can gradually approach the optimal decision-making plan in the process of continuous trial and error correction. In the complex system of the power system, reinforcement learning is applied to efficient energy scheduling, accurate load forecasting and complex power grid control tasks, and can solve complex power system problems. By constructing a reasonable state space, action space and reward function, the intelligent agent can learn how to adjust the power generation output and optimize the power grid structure according to the real-time operating status of the power grid. In this embodiment, a power system control method and device based on reinforcement learning and data-driven are proposed to achieve safe, economical and reliable operation of the power system.

[0057] Example 1

[0058] See also Figure 1 , Figure 1 A schematic diagram of the steps of a power system control method based on reinforcement learning and data-driven is provided in an embodiment of the present invention. Figure 1 As shown, the embodiment of the present invention proposes a power system control method based on reinforcement learning and data-driven, including steps 101 to 107, each step is specifically as follows:

[0059] Step 101, constructing a target variable set and a control variable set based on the operation target and dispatchable variables of the power system;

[0060] As an example of this embodiment, a target variable set is constructed based on the load of each node in the power system, the voltage of the load node, the network loss rate and the load rate of the transmission unit; a control variable set is constructed based on the dispatchable variables corresponding to the target variable set in the power system; the dispatchable variables corresponding to the target variable set include: the active power of the generator, the node voltage of the generator, the reactive power of the reactive compensation device and the switching of the components. A specific implementation method is as follows: construct a target variable set based on the load of each node in the power system, the voltage of the load node, the network loss rate and the load rate of the transmission unit, etc. ps , the target variable set O ps It is a set of variables that reflect the system operation objectives or technical and economic performance; according to the active power of the generator in the power system, the node voltage of the generator, the reactive power of the reactive compensation equipment, the switching of the components, etc., the control variable set C is constructed. ps ; Control variable set C ps It is a set of variables that can be used for power system dispatch and can change the value of the target variable.

[0061] Step 102, constructing an instant reward set based on the output level of the power system;

[0062] As an example of this embodiment, according to the historical operation data of the power system, an initial state is set; based on the initial state, the output level of the power system is discretized into a number of actions to obtain an action set; based on the action set and the initial state, a convergence performance evaluation, a penalty for exceeding the limit, and a loss of exploration efficiency are constructed; based on the convergence performance evaluation, the penalty for exceeding the limit, and the loss of exploration efficiency, an immediate reward set is constructed. A specific implementation method: the state set S (initial state) is composed of the demand vector {P L} and the power supply vector {P G}; Each generator can be discretized into p actions at the current output level. If there are l generators in the system, the total number of optional actions in the action set is k = p × l; the immediate reward r consists of three components, namely r = r 1 +r 2 +r 3 , where r 1 It is a convergent performance evaluation, that is, it is necessary to give positive rewards to excellent behaviors and negative rewards to bad behaviors. Specifically, r 1 The calculation formula is as follows:

[0063]

[0064] In the formula, k 1 and c 1 are proportional factor and proportional adjustment factor respectively; f(P Gi ) represents the operating cost of the first generator.

[0065] r 2 It represents the penalty for exceeding the limit, that is, it is necessary to continuously obtain the power flow information of the power grid during the exploration of the environment; if the power flow does not meet the given operation constraints, the behavior that does not meet the power grid operation constraints should be punished to reduce the probability of making similar behaviors.

[0066] r 3 Represents the loss of exploration efficiency. In order to prevent the agent from repeatedly exploring around the local optimal solution, it is necessary to punish each adjustment action performed by the agent. 3 Normally a constant, but when the action chosen is to keep the generator output constant, r 3 =0.

[0067] It is worth mentioning that the state set S in the reinforcement learning method usually comes from the target variable set O ps ; Since the agent can form the final optimal scheduling decision through multiple steps, if the optimal control With the initial control C ps (i) If the deviation between them is large, the number of action steps required for the agent to explore the optimal control will increase. When defining the action set, coarse adjustment actions with larger step sizes and fine adjustment actions with smaller step sizes can be defined at the same time. The knowledge learning process ensures both high scheduling accuracy and high exploration efficiency.

[0068] Step 103, constructing an action value function based on the instant reward set through a target network of a preset deep reinforcement learning algorithm;

[0069] As an example of this embodiment, the parameters to be trained of the target network of the preset deep reinforcement learning algorithm are obtained; the action value function is constructed through the immediate reward set, the parameters to be trained of the target network and the initial state. A specific implementation method is that for deep Q learning, the main function of the neural network is to fit the Q table. Depending on the complexity of the problem, the neural network structure can be composed of 2-5 fully connected layers. In order to solve the problem of overestimation of the action value by the neural network, the estimation network and the target network are used to jointly complete the iterative update of the action value function Q during the training of the intelligent agent. The estimation network is used to fit the mapping relationship from state to Q value, and the target network generates Q value for constructing the learning goal. Specifically, the target network Q value update formula is:

[0070] Q=r t+1 +γmax(s t+1 ,a t ,ω - )

[0071] In the formula, ω - represents the parameters of the target network to be trained, Q(s t+1 ,a t,ω-) is the Q value of the next state fitted by the target network.

[0072] Step 104, iteratively updating the action value function through an evaluation network of a preset deep reinforcement learning algorithm until the action value function meets the preset training requirements, thereby obtaining an optimal control amount and optimal scheduling knowledge;

[0073] As an example of this embodiment, the estimated network parameters are updated through the evaluation network of the preset deep reinforcement learning algorithm to obtain an updated evaluation network; the weight value of the updated evaluation network is assigned to the target network to obtain an updated target network; based on the updated target network, the action value function is iteratively updated until the action value function converges to obtain the optimal control amount and the optimal scheduling knowledge. A specific implementation method is that the target network structure is the same as the estimated network structure. After several weight assignment steps, the target network is updated to the latest estimated network parameters, and the parameters are kept unchanged within the interval, so that the entire iterative process is more stable. The update amount of the Q value in the estimated network is as follows:

[0074] ΔQ(s t ,a t )=α[Q(s t ,a t ,ω - )-Q(s t ,a t ,ω t )]

[0075] In the formula, ω t To estimate the network parameters of the network, α is the learning rate of the neural network.

[0076] When the neural network converges, the target O can be obtained. ps (i) The corresponding optimal control C ps* (i) and optimal scheduling knowledge It takes the form of a deep neural network.

[0077] Step 105, constructing an initial knowledge base based on the optimal control quantity, optimal scheduling knowledge, target variable set and control variable set;

[0078] As an example of this embodiment, the column vectors corresponding to the optimal control quantity, optimal scheduling knowledge, target variable set and control variable set are constructed to obtain a column vector set; the column vector set is merged to obtain an initial knowledge base. ps The optimal control cumulative reward of (i) is denoted as G ps* (i), define three column vectors as follows:

[0079] O ps,vector =[O ps(1)O ps (2)...O ps (n)] T

[0080] C ps,vector =[C ps (1)C ps (2)...C ps (n)] T

[0081] G ps,vector =[G ps (1) G ps (2)...G ps (n)] T

[0082] Among them, ps (i) and C ps (i) is itself a vector, then the generalized n×3 original knowledge base matrix K ps,ori It can be defined as:

[0083]

[0084] Step 106, clustering the initial knowledge base using a knowledge extraction algorithm to obtain a number of sub-class knowledge matrices;

[0085] As an example of this embodiment, the elements of each column vector in the initial knowledge base are clustered and divided by a preset clustering algorithm to construct several sub-matrices; knowledge extraction is performed on the several sub-matrices by a knowledge extraction algorithm to obtain several sub-class knowledge matrices.

[0086] A specific explanation is that in the original knowledge matrix, each row O ps Each vector has a corresponding (i) The optimal control is associated with each specific goal. Therefore, the original knowledge can only handle a specific problem, but not a class of problems. Through the clustering method, the original knowledge matrix K ps,ori The first column (i.e., vector O ps ) is divided into m categories, based on which the original knowledge matrix K ps,ori Divided into m sub-matrices, namely:

[0087]

[0088] Each submatrix corresponds to a knowledge subclass, and the elements in each subclass have similar information. ps,ori The definition of , its j-th sub-matrix can be written as:

[0089]

[0090] The number of rows (the number of target control pairs included) is β j , we can get:

[0091]

[0092] As another example of this embodiment, based on the target variable set corresponding to each subclass knowledge matrix, the first cluster center and the relative standard deviation of the target set are calculated; based on the control variable set corresponding to each subclass knowledge matrix, the second cluster center and the relative standard deviation of the control set are calculated; based on the first cluster center, the relative standard deviation of the target set, the second cluster center and the relative standard deviation of the control set, knowledge extraction is performed on the several sub-matrices through a knowledge extraction algorithm to obtain several sub-class knowledge matrices.

[0093] A specific implementation method is as follows: for the target set O j ps,vector , define the vector j of the jth knowledge subclass and the cluster center of the set (First cluster center) and target set relative standard deviation for:

[0094]

[0095] Similarly, for the control set Defines the vector and cluster center of the jth subclass (Second cluster center) and control set relative standard deviation

[0096] For the control set That is, the vector j of the jth subclass, the mean of all control values ​​is defined as

[0097]

[0098] Maximum relative deviation of returns among subclasses for:

[0099]

[0100] After extraction, the final knowledge matrix K is formed ps :

[0101]

[0102] Where K ps From the original K ps Beta j OK, K j ps,ori The line is reduced to 1:

[0103]

[0104] The knowledge validity evaluation index of knowledge subclass j (or submatrix j) is Defined as a triple:

[0105]

[0106] For a knowledge subcategory of a given classification, Smaller ternaries indicate better knowledge composition or shallower knowledge compression, while larger ones indicate better knowledge composition or shallower knowledge compression. and smaller and This means that the control group is adaptable to a wider range of objective requirements, and the control has a tendency to converge to the target. In addition, the relatively small and relatively large and This means that this group of control groups is very sensitive to the target, and the control has a divergent trend towards the target. In the process of control implementation, it may face a large fluctuation in control errors. For different sub-classification schemes, it is obvious that when the original knowledge base K ps,ori When the extraction process is deeply compressed (retaining fewer knowledge subcategories), the three sets of values ​​in the knowledge validity evaluation index should be larger. On the contrary, if subcategory compression is not performed, that is, m = n, then all knowledge validity evaluation indicators K of the original knowledge base are ps,ori All are 0, according to the knowledge validity evaluation index It can balance the contradiction between knowledge simplification and control effect in progressive knowledge extraction. For the divided knowledge subcategories, It can be used to diagnose and compare the quality differences of each knowledge subcategory and to apply knowledge to any new specified goal. ps,sprc , in the knowledge matrix Q ps The closest to it can be found in the first column of At this time, control The loss of income is not greater than

[0107] Step 107: Based on the plurality of sub-category knowledge matrices, adjust the control variable set and regulate the power system.

[0108] As an example of this embodiment, based on several sub-class knowledge matrices, the control variable set is adjusted through the optimal control quantity and the optimal scheduling knowledge to obtain the power system adjustment strategy; based on the power system adjustment strategy, the dispatchable variables of the power system are regulated; wherein the dispatchable variables include: the active power of the generator, the node voltage of the generator, the reactive power of the reactive compensation device, and the switching of components.

[0109] As an implementable method of this embodiment, according to the optimal control and optimal dispatching knowledge, the control variable set, such as the active power of the generator, the node voltage of the generator, the reactive power of the reactive compensation device and the switching of the components, is adjusted to achieve safe, economical and reliable operation of the power system. For example, when the load increase is detected, the intelligent agent increases the active power of the generator or adjusts the reactive power of the reactive compensation device according to the learned strategy to meet the load demand and maintain voltage stability. At the same time, through real-time data processing and dynamic adjustment mechanism, it quickly responds to changes in the power grid system. The agent continuously obtains the power grid flow information in the process of exploring the environment. If the flow does not meet the given operation constraints, the behavior that does not meet the power grid operation constraints is punished, thereby adjusting the control strategy to ensure the real-time and stability of the system; in addition, through knowledge extraction and application, the contradiction between knowledge simplification and ensuring control effect in knowledge extraction is balanced according to the knowledge validity evaluation index. For the new specified target, the closest target is found in the knowledge matrix, and corresponding control measures are taken to ensure that the loss of income is not greater than a certain threshold, so as to achieve precise regulation of the power grid system.

[0110] The embodiment of the present invention proposes a power system control method based on reinforcement learning and data-driven, which uses multi-source heterogeneous data such as the operation objectives and dispatchable variables of the power system to construct a target variable set and a control variable set to construct a system simulation image, improve data utilization and adaptability, and ensure the accuracy of system control in different data scenarios; based on the instant reward and cumulative reward mechanism of reinforcement learning, the technical and economic performance of the power system operation is comprehensively evaluated to ensure the effectiveness and optimization of intelligent scheduling decisions; the action value function is abstracted by reinforcement learning technology, and the state space dimension explosion and data quality problems are effectively solved; by constructing a knowledge base and a knowledge extraction algorithm, the interpretability of system control is enhanced, so that each decision can be traced and understood, and the transparency and reliability of the decision are improved; the data is abstracted into a knowledge matrix through knowledge transfer technology and the control variable set is adjusted to control the system, so as to improve the generalization ability of system control in different scenarios and goals, so that existing knowledge can be effectively applied and migrated in new scenarios; thus, the present invention can effectively improve the efficiency, accuracy and reliability of intelligent scheduling of the power system, and realize the intelligent operation, scheduling and management of the power system.

[0111] Example 2

[0112] See also Figure 4 , Figure 4 A schematic diagram of the module structure of a power system control device based on reinforcement learning and data-driven is provided in one embodiment of the present invention. Figure 4 As shown, the embodiment of the present invention proposes a power system control device based on reinforcement learning and data-driven, including:

[0113] Variable set construction module 401, reward set construction module 402, action value function construction module 403, action value function iterative update module 404, knowledge base construction module 405, knowledge matrix partition module 406 and power system control module 407;

[0114] The variable set construction module 401 is used to construct a target variable set and a control variable set based on the operation target and dispatchable variables of the power system;

[0115] The reward set building module 402 is used to build an instant reward set based on the output level of the power system;

[0116] The action value function construction module 403 is used to construct an action value function based on the instant reward set through a target network of a preset deep reinforcement learning algorithm;

[0117] The action value function iterative update module 404 is used to iteratively update the action value function through the evaluation network of the preset deep reinforcement learning algorithm until the action value function meets the preset training requirements and obtains the optimal control amount and the optimal scheduling knowledge;

[0118] The knowledge base construction module 405 is used to construct an initial knowledge base based on the optimal control quantity, optimal scheduling knowledge, target variable set and control variable set;

[0119] The knowledge matrix partitioning module 406 is used to perform clustering partitioning on the initial knowledge base through a knowledge extraction algorithm to obtain a plurality of sub-class knowledge matrices;

[0120] The power system control module 407 is used to adjust the control variable set and control the power system based on the plurality of sub-category knowledge matrices.

[0121] The embodiment of the present invention proposes a power system control device based on reinforcement learning and data-driven. Through the variable set construction module, multi-source heterogeneous data such as the operation objectives and dispatchable variables of the power system are used to construct the target variable set and the control variable set to construct the system simulation image, thereby improving the data utilization and adaptability, and ensuring the accuracy of the system control in different data scenarios; the reward set construction module is based on the instant reward and cumulative reward mechanism of reinforcement learning to comprehensively evaluate the technical and economic performance of the power system operation, thereby ensuring the effectiveness and optimization of the intelligent dispatching decision; the action value function is abstracted by the reinforcement learning technology through the action value function construction module and the action value function iterative update module, and the action value function is effectively solved. The problem of state space dimension explosion and data quality is solved; the knowledge base construction module enhances the interpretability of system control by constructing a knowledge base and a knowledge extraction algorithm, so that each decision can be traced and understood, and the transparency and reliability of the decision are improved; the knowledge matrix division module and the power system control module also abstract the data into the knowledge matrix through the knowledge transfer technology and adjust the control variable set to control the system, thereby improving the generalization ability of system control under different scenarios and goals, so that the existing knowledge can be effectively applied and transferred in new scenarios; thus, the present invention can effectively improve the efficiency, accuracy and reliability of intelligent dispatching of the power system, and realize the intelligent operation, dispatching and management of the power system.

[0122] Example 3

[0123] See also Figure 2 , Figure 2 A schematic diagram of a power dispatching model architecture of a power system control method based on reinforcement learning and data-driven is provided in a certain embodiment of the present invention; Figure 2 As shown in FIG. 1 , a power dispatch model architecture (AIDLM) for a power system control method based on reinforcement learning and data-driven is constructed; specifically, the target variable set O ps It is a set of variables that reflect the system operation objectives or technical and economic performance; the control variable set C ps It is a set of variables that can be used for power system dispatching and can change the value of the target variable; environment E ps It covers all the operating variables obtained by sensing in the actual power system, and can also establish a mathematical model to reflect the correlation between all operating variables; it combines multi-source heterogeneous data to improve data quality and utilization, constructs the environment from historical data, extracts target vectors and control vectors, and forms an initial "target-control" simulation image; the cumulative reward G ps* It is an objective variable and corresponding control for evaluating technical and economic performance; since income evaluation requires environmental E psTherefore, various constraints can be tested in this module. In addition, violations and their impact on control values ​​can be corrected. The exploration reward for a single-step action on the environment is called an immediate reward r. In each episode, multiple control actions will be selected to maintain good economic and technical performance of the power system. Since the intelligent power system scheduling problem needs to meet multiple operating goals in time series and the optimal decision of multiple power generation control variables, the optimal scheduling knowledge under a single goal can be expressed by Q value. Finally, the original knowledge formed by "target control" is processed through knowledge extraction to obtain a more refined knowledge expression (knowledge repository) K ps Through the above description, the power dispatch model architecture (AIDLM) is constructed.

[0124] As an example of this embodiment, see Figure 3 , Figure 3 A flow chart of learning and training steps of a power system control method based on reinforcement learning and data-driven is provided for an embodiment of the present invention; after the power dispatch model architecture (AIDLM) is constructed, for general reinforcement learning problems involving a large number of states and continuous actions, Markov chains, ε-greedy, Monte Carlo decision trees and other methods are usually used to select feasible actions. If the environment is relatively complex, the state set S may face the problem of dimensionality explosion. At this time, the action evaluation Q value under each state and action can be obtained by fitting a neural network with a parameter of ω, providing method support for the accumulation and application of complex intelligent dispatching knowledge. In this way, when the neural network reaches convergence, a certain goal can be achieved by comparing the values ​​of the neural network output between different actions. ps (i); the action value-based reinforcement learning method outputs the values ​​of all possible actions in each iterative step. In addition, when the knowledge of the intelligent agent is used for optimal scheduling, the convergence process of the optimization decision can be observed; and the power system has a relatively complete data acquisition system, including supervisory control and data acquisition (SCADA), which records the operation of the system. These data reflect the relationship between variables such as environment, target and control in the past operation of the actual power system; for each episode of learning process, a learning training step is proposed. Assuming that the historical data set {data(i)} contains the operating data of n scenarios, the environment E can be constructed from the data of scenario i ps (i) and obtain the environment E ps (i) State S t , we can also extract the target vector O from {data(i)} ps (i) and the control vector C ps(i) Form the initial “target control”; during the entire training process of the agent, AIDLM sets the initial state of the agent to the initial control given by each set of historical data (selecting operations based on the state); fits a neural network with a parameter ω to obtain each state and action a 1 The action evaluation Q value under the condition is: ps The optimal scheduling knowledge of (i) needs to be updated through multiple episodes; in each episode, by fitting the action and the environment, multiple control actions are selected to adjust the environmental state S t Update and calculate reward r, store environment state S t , Q value and reward r, so that the system's operating state gradually converges or the power flow exceeds the limit. If the system's operating state does not converge or the power flow does not exceed the limit, it is judged that the maximum control time has been reached. If not, the next control action is updated and re-learned until the maximum control time is reached. Through the above description and step explanation, the power dispatch model architecture (AIDLM) is data-driven.

[0125] The embodiment of the present invention proposes a power system control method based on reinforcement learning and data-driven. By means of state compression and abstraction, the problem of state space dimension expansion is effectively resolved, and the efficiency of learning and decision-making is improved; by using power system simulation mirror and multi-source heterogeneous data, the quality and utilization of data are improved, and the model is ensured to have good adaptability and accuracy in different data scenarios; by adopting the mechanism of real-time data processing and dynamic adjustment, the real-time and scalability of the model are enhanced, so that it can quickly respond and adapt to the expansion of the system; by constructing a knowledge base and a knowledge extraction method, the interpretability of the model is enhanced, so that each step of the decision can be traced and understood, and the transparency and reliability of the decision are improved; in addition, by using deep reinforcement learning and multi-step control optimization methods, a dynamic mapping relationship between target variables and control variables is established, and online learning and optimization of the optimal control strategy are realized; finally, by constructing a knowledge base and knowledge transfer technology, the generalization ability of the model in different scenarios and goals is improved, so that the existing knowledge can be effectively applied and transferred in new scenarios; in general, the technical solution proposed in the embodiment of the present invention greatly improves the efficiency, accuracy and reliability of intelligent dispatching of the power system, and realizes the intelligent operation and management of the power system.

[0126] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0127] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0128] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

Claims

1. A power system control method based on reinforcement learning and data-driven, characterized in that: include: Based on the operation objectives and dispatchable variables of the power system, construct the target variable set and the control variable set; Build an instant reward set based on the output level of the power system; Based on the instant reward set, construct an action value function through a target network of a preset deep reinforcement learning algorithm; Iteratively updating the action value function through an evaluation network of a preset deep reinforcement learning algorithm until the action value function meets the preset training requirements, thereby obtaining the optimal control amount and the optimal scheduling knowledge; Building an initial knowledge base based on the optimal control quantity, optimal scheduling knowledge, target variable set and control variable set; Clustering the initial knowledge base using a knowledge extraction algorithm to obtain several sub-class knowledge matrices; Based on the several sub-category knowledge matrices, a set of control variables is adjusted and the power system is regulated.

2. The power system control method based on reinforcement learning and data-driven according to claim 1, characterized in that: The target variable set and the control variable set are constructed based on the operation target and dispatchable variables of the power system, including: Construct a target variable set based on the load of each node in the power system, the voltage of the load node, the network loss rate and the load rate of the transmission unit; Based on the dispatchable variables corresponding to the target variable set in the power system, a control variable set is constructed; the dispatchable variables corresponding to the target variable set include: the active power of the generator, the node voltage of the generator, the reactive power of the reactive compensation device and the switching of the components.

3. The power system control method based on reinforcement learning and data-driven according to claim 1, characterized in that: Based on the output level of the power system, an instant reward set is constructed, including: Set the initial state based on the historical operation data of the power system; Based on the initial state, the output level of the power system is discretized into a number of actions to obtain an action set; Constructing convergence performance evaluation, penalty for exceeding limits, and loss of exploration efficiency based on the action set and the initial state; Based on the convergence performance evaluation, the penalty for exceeding the limit and the loss of exploration efficiency, an immediate reward set is constructed.

4. The power system control method based on reinforcement learning and data-driven according to claim 3, characterized in that: The step of constructing an action value function based on the instant reward set by presetting a target network of a deep reinforcement learning algorithm includes: Obtain the training parameters of the target network of the preset deep reinforcement learning algorithm; Construct an action-value function using the immediate reward set, the parameters to be trained of the target network, and the initial state.

5. The power system control method based on reinforcement learning and data-driven according to claim 4, characterized in that: The action value function is iteratively updated by the evaluation network of the preset deep reinforcement learning algorithm until the action value function meets the preset training requirements to obtain the optimal control amount and the optimal scheduling knowledge, including: The estimated network parameters are updated through an evaluation network of a preset deep reinforcement learning algorithm to obtain an updated evaluation network; Assigning the updated evaluation network weight value to the target network to obtain an updated target network; Based on the updated target network, the action value function is iteratively updated until the action value function converges to obtain the optimal control amount and the optimal scheduling knowledge.

6. The power system control method based on reinforcement learning and data-driven according to claim 1, characterized in that: The initial knowledge base is constructed based on the optimal control quantity, the optimal scheduling knowledge, the target variable set and the control variable set, including: Constructing column vectors corresponding to the optimal control quantity, optimal scheduling knowledge, target variable set and control variable set to obtain a column vector set; The column vector sets are merged to obtain an initial knowledge base.

7. The power system control method based on reinforcement learning and data-driven according to claim 6, characterized in that: The initial knowledge base is clustered and divided by the knowledge extraction algorithm to obtain several sub-class knowledge matrices, including: Clustering and dividing the elements of each column vector in the initial knowledge base by a preset clustering algorithm to construct a plurality of sub-matrices; The knowledge extraction algorithm is used to extract knowledge from the plurality of sub-matrices to obtain a plurality of sub-class knowledge matrices.

8. The power system control method based on reinforcement learning and data-driven according to claim 7, characterized in that: The step of extracting knowledge from the plurality of sub-matrices by using a knowledge extraction algorithm to obtain a plurality of sub-class knowledge matrices further includes: Based on the target variable set corresponding to each subclass knowledge matrix, the first cluster center and the relative standard deviation of the target set are calculated; Based on the control variable set corresponding to each subclass knowledge matrix, the second cluster center and the relative standard deviation of the control set are calculated; Based on the first cluster center, the target set relative standard deviation, the second cluster center and the control set relative standard deviation, knowledge extraction is performed on the plurality of sub-matrices through a knowledge extraction algorithm to obtain a plurality of sub-class knowledge matrices.

9. The power system control method based on reinforcement learning and data-driven according to claim 8, characterized in that: Based on the several sub-category knowledge matrices, the control variable set is adjusted and the power system is regulated, including: Based on several sub-class knowledge matrices, the control variable set is adjusted through the optimal control quantity and optimal dispatch knowledge to obtain the power system adjustment strategy; Based on the power system adjustment strategy, the dispatchable variables of the power system are regulated; wherein the dispatchable variables include: the active power of the generator, the node voltage of the generator, the reactive power of the reactive compensation device and the switching of the components.

10. A power system control device based on reinforcement learning and data-driven, characterized in that: Executing a power system control method based on reinforcement learning and data-driven according to any one of claims 1 to 9, comprising: Variable set construction module, reward set construction module, action value function construction module, action value function iterative update module, knowledge base construction module, knowledge matrix partition module and power system control module; The variable set building module is used to build a target variable set and a control variable set based on the operation target and dispatchable variables of the power system; The reward set building module is used to build an instant reward set based on the output level of the power system; The action value function construction module is used to construct an action value function based on the instant reward set through a target network of a preset deep reinforcement learning algorithm; The action value function iterative update module is used to iteratively update the action value function through an evaluation network of a preset deep reinforcement learning algorithm until the action value function meets the preset training requirements and obtains the optimal control amount and the optimal scheduling knowledge; The knowledge base construction module is used to construct an initial knowledge base based on the optimal control quantity, optimal scheduling knowledge, target variable set and control variable set; The knowledge matrix partitioning module is used to perform clustering partitioning on the initial knowledge base through a knowledge extraction algorithm to obtain a plurality of sub-class knowledge matrices; The power system control module is used to adjust the control variable set and control the power system based on the plurality of sub-category knowledge matrices.

Images