A safe power dispatching method based on deep reinforcement learning
Through deep reinforcement learning methods, combined with Markov decision-making process and expert rule base, the power scheduling solution is optimized, and the safety constraints of the power grid under new energy and peak loads are solved, the safety and scheduling performance of the power grid are improved, and the power is realized The system is stable and responds quickly.
Patent Information
- Application Number
- CN202411372731.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-29
AI Technical Summary
The existing power scheduling methods have shortcomings in terms of efficiency and robustness, especially under the rapid development of new energy and peak load conditions, it is difficult to meet the instantaneous safety constraints of the power grid, which may lead to grid collapse and economic losses.
Adopt a safe power scheduling method based on deep reinforcement learning, through Markov decision-making process and adversarial learning training scheduling model, combined with the expert rule base interactive tuning to train the main expert model, optimize the power scheduling scheme, and consider the safety of voltage, current, frequency, etc. Cost, improve the safety and scheduling performance of the power grid.
It improves the optimization effect of the power scheduling plan, enhances the safety and robustness of the power grid, can balance supply and demand during peak periods, quickly detect and isolate faults, formulate emergency plans, and ensure the stable operation of the power system.
Smart Images

Figure CN118886689B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of electric power dispatching, and specifically relates to a safe electric power dispatching method based on deep reinforcement learning. Background Art
[0002] Power dispatching is an effective management method that aims to ensure the safe and stable operation of the power grid, reliable power supply, and the orderly progress of various types of power production work. The specific work content of power dispatching includes judging the safety and economic operation status of the power grid based on the data information fed back by various information collection devices or the information provided by monitoring personnel, combined with the actual operation parameters of the power grid (such as voltage, current, frequency, load, etc.), comprehensively considering the development of various production work. The dispatching center issues operating instructions through telephone or automatic system, directing on-site operators or automatic control systems to make adjustments, such as adjusting generator output, load distribution, and switching capacitors and reactors, so as to ensure the continuous safe and stable operation of the power grid.
[0003] The rapid development of new energy and the continuous growth of peak load have brought new challenges to the dispatching capacity of the power generation side. New energy refers to renewable energy such as wind power and photovoltaic power, which can be obtained through natural capabilities and has the characteristics of intermittent and randomness. Not every region has the ability to build a wind power plant or a photovoltaic plant, so it is necessary to interconnect the power grid quickly and on a large scale. With the increase of various flexible devices or electronic devices, the regulation scale of the entire power system has become larger, and the uncertainty and volatility have also increased. The instantaneous safety constraints at every moment need to be met in the grid dispatching. Once a constraint is not met, the entire power grid may collapse, causing large-scale power outages and huge economic losses. Therefore, the entire power system needs to have efficient and robust dispatching capabilities.
[0004] In recent years, deep reinforcement learning algorithms have been widely used in the field of power dispatching. Power dispatching through deep reinforcement learning is a research direction with great application value and prospects. Reinforcement learning has been applied to economic dispatch problems involving virtual power plants, facilitating the rapid decomposition of dispatch instructions for virtual power plants. This technology can also be applied to the self-dispatching of wind farms and energy storage systems to achieve faster dispatching decisions. Summary of the invention
[0005] In view of the above-mentioned deficiencies in the prior art, the safe power dispatching method based on deep reinforcement learning provided by the present invention solves the problems of poor efficiency and robustness of the existing power dispatching methods.
[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a safe power dispatching method based on deep reinforcement learning, comprising the following steps:
[0007] S1, train the scheduling model based on adversarial learning using Markov decision process;
[0008] S2, training the main expert model based on interactive tuning of the expert rule base;
[0009] S3. Output the final power dispatching plan based on the dispatching model optimized by the main expert model.
[0010] Further: S1 specifically includes:
[0011] The action space, state space and reward function are defined through the Markov decision process, the behavior of the scheduling model and the adversarial model in the virtual simulation power grid environment is simulated, the network parameters of the scheduling model and the adversarial model are updated through the Rainbow DQN algorithm, and the scheduling model is trained through adversarial learning;
[0012] The strategy defined by the Markov decision process is as follows: based on the observed data in the virtual simulation power grid environment, a power dispatching plan is generated through a dispatching model or an adversarial model;
[0013] The observation space defined by the Markov decision process includes the grid information of node voltage value, line current value, generator output power and load power demand, and the state space The specific expression is:
[0014]
[0015] In the formula, represents the voltage value of all nodes, Indicates the current value of all lines, represents the output power of all generators, Represents the power demand of all loads;
[0016] The action space defined by the Markov decision process includes all permutations and combinations of power dispatch actions. The specific expression is:
[0017]
[0018] In the formula, Indicates the adjustment value of the generator output power, Indicates the switch operation status, Indicates load adjustment;
[0019] The reward function defined by the Markov decision process is specifically based on the goals of power grid operation, including power supply reliability, economic benefits and robustness. The reward function R t The specific expression is:
[0020]
[0021]
[0022]
[0023]
[0024]
[0025] In the formula, Indicates power supply reliability, used to measure voltage stability, frequency deviation, represents the voltage of the ith node, represents the rated voltage of each node in the power grid, N represents the total number of nodes in the power grid, represents the current of the jth line, represents the maximum allowable current of the jth line, M represents the total number of lines in the power grid, Represents economic benefits, used to measure the cost of power generation, is the cost coefficient of the kth generator, G represents the total number of generators, represents the adjustment value of the output power of the kth generator, Represents environmental benefits and is used to measure pollutant emissions. is the emission coefficient of the kth generator, Represents robustness, which is used to measure the ability to resist failure. is the impact assessment value of the lth fault, F represents the total number of faults, represents the weight of the first indicator, represents the weight of the second indicator, represents the weight of the third indicator, Indicates the weight of the fourth indicator.
[0026] Further: The expression for updating the network parameters of the scheduling model and the adversarial model through the Rainbow DQN algorithm is specifically:
[0027]
[0028] In the formula, For having Atomic vector The projection on represents the number of atoms on the discrete support, is the target distribution, is the KL divergence, is the probability of time difference error, To adjust right The control parameters of the degree of influence.
[0029] Further: In S1, the security cost of the Markov decision process according to Status settings;
[0030] Security costs The specific expression is:
[0031]
[0032]
[0033]
[0034]
[0035] In the formula, represents the first weight coefficient, represents the second weight coefficient, represents the third weight coefficient, Indicates the voltage safety cost, which is used to reflect the degree to which the voltage deviates from the rated value. Indicates the allowable voltage deviation range, Indicates the current safety cost, which is used to reflect whether the line current exceeds the limit of safe operation. Represents the frequency security cost, which is used to measure the frequency deviation of the power grid. Indicates the current grid frequency. Indicates the rated frequency of the power grid, Indicates the allowable frequency deviation range, Represents the maximum value function.
[0036] The beneficial effect of the above further scheme is: according to the characteristics of the power system, the scheduling model is trained by the Rainbow DQN algorithm using adversarial learning and expert rule base interactive tuning to further improve the optimization effect of the power scheduling scheme.
[0037] Furthermore, the objective function of adversarial learning training for the scheduling model is specifically as follows:
[0038]
[0039] In the formula, Indicates the current state. represents the actions of the scheduling model, represents disturbance, It indicates the state to which the system transfers after executing the scheduling and disturbance. represents the strategy of the scheduling model, represents the strategy of adversarial model, and All are obtained through the state space, Acquired through action space;
[0040] Indicates that the adversarial model tries to find the most unfavorable perturbation , It means that the scheduling model learns how to adjust the scheduling strategy under the most unfavorable scenario generated by the adversarial model. Represents the average of all possible state transitions.
[0041] Further: S2 comprises the following sub-steps:
[0042] S21, setting a master expert model and a slave expert model, the master expert model includes basic knowledge and preliminary optimization capabilities of power dispatching, and the slave expert model is generated based on a copy of the master expert model;
[0043] S22, initializing the master expert model and the slave expert model;
[0044] S23. Optimize the output of the slave expert model through the expert rule base, and then train the master expert model.
[0045] The beneficial effect of the above further scheme is: the present invention adopts an interactive tuning method based on the expert rule base constructed by power experts to train the main expert model, each slave expert model corresponds to a power expert, and the main expert model is used for unified updating and management, further improving the optimization effect of the power dispatching scheme.
[0046] Further: S23 includes the following sub-steps:
[0047] S231, outputting an initial power dispatching plan through a dispatching model, inputting the initial power dispatching plan into an expert model, and obtaining an adjusted power dispatching plan;
[0048] S232, performing fault business analysis on the adjusted power dispatching plan through an expert rule base, and generating feedback results of the adjusted power dispatching plan;
[0049] S233, sending the feedback result to the slave expert model through the virtual simulation power grid environment, and updating the model parameters of the slave expert model according to the feedback result;
[0050] S234. Feedback the model parameters updated from the expert model to the main expert model according to a preset period to train the main expert model.
[0051] The beneficial effects of the above further scheme are: using the expert rule base to interactively tune and adjust the master expert model to make it more consistent with the professional judgment of power experts. Using environmental feedback to adjust the slave expert model to make it more consistent with the actual situation of power grid operation. Regularly feeding back the updates of the slave expert model to the master expert model to ensure that the master expert model can gradually absorb the knowledge and experience of various power experts.
[0052] Further: In S234, the expression of the loss function Loss for training the main expert model is specifically:
[0053]
[0054] In the formula, is the expert confidence score of the expert rule base for the i-th adjusted power dispatching plan, is the predicted score of the ith adjusted power dispatch plan from the expert model, and N is the number of dispatch plans.
[0055] Further: S3 includes the following sub-steps:
[0056] S31, inputting the observed values in the virtual simulation power grid environment into the dispatching model to generate a power dispatching plan, wherein the observed values include node voltage, line current, generator output power and load demand;
[0057] S32, sending the power dispatching plan to the main expert model to generate an optimized power dispatching plan;
[0058] S33, performing fault business analysis on the optimized power dispatching plan through an expert rule base, and generating feedback results of the optimized power dispatching plan;
[0059] S34. Determine whether the optimized power dispatching plan is qualified according to the feedback results of the optimized power dispatching plan. If so, use the optimized power dispatching plan as the final power dispatching plan; if not, optimize the main expert model according to the feedback results of the optimized power dispatching plan and return to S32.
[0060] The beneficial effects of the present invention are:
[0061] (1) The present invention provides a safe power dispatching method based on deep reinforcement learning. By learning the knowledge related to power dispatching and combining the characteristics and disadvantages of deep reinforcement learning, the present invention can effectively deal with various safety issues in power dispatching. Based on the Rainbow DQN algorithm, the present invention proposes two novel training methods, namely, training the adversarial model and the dispatching model based on adversarial learning to perform safety constraint assurance learning, and training the main expert model based on the interactive tuning of the expert rule base. The present invention proposes a safe power dispatching algorithm based on deep reinforcement learning. The safety cost of the Markov decision process Taking voltage safety cost, current safety cost and frequency safety cost into consideration improves the safety of power grid operation. Theoretical analysis shows that this method has potential advantages in terms of safety and dispatch performance.
[0062] (2) The purpose of this invention is to expand the problem into safe power dispatching, including load management, fault detection and isolation, emergency plans, etc., based on various safety standards and rules in power dispatching and in combination with the characteristics of the power grid system, and finally establish a safe power dispatching algorithm based on deep reinforcement learning. It is used to balance supply and demand and reduce the pressure of peak load on the system through demand response, load transfer and peak-to-valley filling during peak power consumption periods; it is also used to quickly detect and isolate fault areas, prevent fault expansion, and reduce the impact on the power system; it is also used to formulate emergency plans to ensure rapid response and power restoration in the event of an emergency. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 This is a flow chart of the safe power dispatching method based on deep reinforcement learning of the present invention.
[0064] Figure 2 Schematic diagram of adversarial learning training based on Markov decision process.
[0065] Figure 3 Schematic diagram of master-slave expert model training.
[0066] Figure 4 Generate and adjust flow charts for power dispatch plans. DETAILED DESCRIPTION
[0067] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.
[0068] like Figure 1As shown, in one embodiment of the present invention, a safe power dispatching method based on deep reinforcement learning includes the following steps:
[0069] S1, train the scheduling model based on adversarial learning using Markov decision process;
[0070] S2, training the main expert model based on interactive tuning of the expert rule base;
[0071] S3. Output the final power dispatching plan based on the dispatching model optimized by the main expert model.
[0072] The S1 is specifically:
[0073] The action space, state space and reward function are defined through the Markov decision process, the behavior of the scheduling model and the adversarial model in the virtual simulation power grid environment is simulated, the network parameters of the scheduling model and the adversarial model are updated through the Rainbow DQN algorithm, and the scheduling model is trained through adversarial learning;
[0074] The strategy defined by the Markov decision process is as follows: based on the observed data in the virtual simulation power grid environment, a power dispatching plan is generated through a dispatching model or an adversarial model;
[0075] The observation space defined by the Markov decision process includes the grid information of node voltage value, line current value, generator output power and load power demand, and the state space The specific expression is:
[0076]
[0077] In the formula, represents the voltage value of all nodes, Indicates the current value of all lines, represents the output power of all generators, Represents the power demand of all loads;
[0078] The action space defined by the Markov decision process includes all permutations and combinations of power dispatch actions. The specific expression is:
[0079]
[0080] In the formula, Indicates the adjustment value of the generator output power, Indicates switch operation status, such as line on / off, Indicates load adjustment;
[0081] The reward function defined by the Markov decision process is specifically based on the goals of power grid operation, including power supply reliability, economic benefits and robustness. The reward function R t The specific expression is:
[0082]
[0083]
[0084]
[0085]
[0086]
[0087] In the formula, Indicates power supply reliability, used to measure voltage stability, frequency deviation, represents the voltage of the ith node, It represents the rated voltage of each node in the power grid. It is a fixed value and represents the normal operating voltage of the power grid. N represents the total number of nodes in the power grid. represents the current of the jth line, represents the maximum allowable current of the jth line, M represents the total number of lines in the power grid, Represents economic benefits, used to measure the cost of power generation, is the cost coefficient of the kth generator, G represents the total number of generators, represents the adjustment value of the output power of the kth generator, Represents environmental benefits and is used to measure pollutant emissions. is the emission coefficient of the kth generator, Represents robustness, which is used to measure the ability to resist failure. is the impact assessment value of the lth fault, F represents the total number of faults, represents the weight of the first indicator, represents the weight of the second indicator, represents the weight of the third indicator, Indicates the weight of the fourth indicator. The indicator weight can be adjusted according to actual needs.
[0088] like Figure 2 As shown, in this embodiment, the task of training the scheduling model and the adversarial model is modeled as a reinforcement learning problem, namely, a Markov decision process. The key components of reinforcement learning, such as the strategy, action space, observation space and reward function defined by the Markov decision process, are as follows: The strategy refers to the power scheduling plan generated by the scheduling model or the adversarial model based on the observation of the model in the virtual simulation power grid environment; To counter the strategy, is the scheduling strategy, is the action output by the adversarial strategy, The action output by the scheduling strategy, The current state, action, reward, and next state of the interaction between the adversarial strategy and the environment. , , , ) is the current state, action, reward, and next state of the interaction between the dispatching strategy and the environment; the action space contains all possible combinations of power dispatching actions; the observation space includes grid information such as node voltage, line current, generator output power, load demand, etc.; the reward function is based on the goals of grid operation, such as power supply reliability: ensuring that the grid does not fail during the power supply process, such as overload, undervoltage, etc.; economic benefits: minimizing power generation costs and optimizing the output of generator sets; environmental benefits: minimizing pollution emissions; robustness: being able to withstand a certain degree of disturbance and attack and maintain normal operation.
[0089] The expression for updating the network parameters of the scheduling model and the adversarial model through the Rainbow DQN algorithm is as follows:
[0090]
[0091] In the formula, For having Atomic vector The projection on represents the number of atoms on the discrete support, is the target distribution, is the KL divergence, is the probability of time difference error, To adjust right In this embodiment, atoms are discrete points used to represent the distribution of rewards, through which the algorithm can approximate the distribution of rewards, rather than just the expected rewards.
[0092] In this embodiment, the Rainbow DQN algorithm is used to train the scheduling model according to the characteristics of the power system by using adversarial learning and interactive tuning of the expert rule base to further improve the optimization effect of the power scheduling scheme. The Rainbow DQN algorithm combines six independent improvements of the DQN (Deep Q Learning) algorithm by the reinforcement learning community and achieves good results.
[0093] In S1, the security cost of the Markov decision process in S1 according to Status settings;
[0094] Security costs The specific expression is:
[0095]
[0096]
[0097]
[0098]
[0099] In the formula, represents the first weight coefficient, represents the second weight coefficient, It represents the third weight coefficient, which reflects the contribution of each safety indicator to the total safety cost. It represents the voltage safety cost, which is used to reflect the degree to which the voltage deviates from the rated value and has a direct impact on the stability of power grid operation. Indicates the allowable voltage deviation range, Indicates the current safety cost, which is used to reflect whether the line current exceeds the limit of safe operation. Excessive current may cause line damage. Indicates the frequency safety cost, which is used to measure the frequency deviation of the power grid. Frequency instability will affect the normal operation of the equipment. Indicates the current grid frequency. Indicates the rated frequency of the power grid, Indicates the allowable frequency deviation range, Represents the maximum value function.
[0100] In this embodiment, the security cost It is related to the St+1 state. For example, if a transmission line is suddenly interrupted at time t+1, the cost will be very high. Therefore, the worst-case training can be performed in advance. Adversarial learning is usually used to learn the safety constraints for the worst-case of N-1 line interruption. Through this adversarial training, the dispatching model can learn how to optimize the dispatching strategy in various worst-case scenarios, so as to master the safety measures for extreme situations such as N-1 line interruption in advance. The ultimate goal is to enable power dispatching to maintain safe operation and meet the safety requirements of the power grid in the most unfavorable conditions. The objective function of adversarial learning training for the dispatching model and the adversarial model can be expressed as the following Minimax optimization problem:
[0101]
[0102] In the formula, is the current state; To schedule the actions of the model; Actions against the model, i.e. generated perturbations, such as line interruptions; The state that the system transfers to after executing the scheduling and disturbance; The strategy for scheduling the model; To combat the model, and All are obtained through the state space, Obtained through action space.
[0103] Maximize the section Indicates that the adversarial model tries to find the most unfavorable perturbation , for example, simulating N-1 outage scenarios to find the line outage combination that has the greatest impact on grid security. This is partly to foresee the worst-case scenario in advance.
[0104] Minimize the part It means that the dispatch model learns how to adjust the dispatch strategy under the most unfavorable scenario generated by the adversarial model to minimize the safety cost of the system, that is, to ensure the safe operation of the power grid under the most unfavorable circumstances.
[0105] expect It represents the average of all possible state transitions, comprehensively considering different perturbations and scheduling combinations to evaluate the overall performance.
[0106] The S2 comprises the following sub-steps:
[0107] S21, setting a master expert model and a slave expert model, the master expert model includes basic knowledge and preliminary optimization capabilities of power dispatching, and the slave expert model is generated based on a copy of the master expert model;
[0108] S22, initializing the master expert model and the slave expert model. In this embodiment, the scheduling model is used to initialize the expert model to ensure that the expert model and the scheduling model have the same basic knowledge, which reduces problems that may be caused by knowledge mismatch, such as hallucinations and inconsistencies.
[0109] S23. Optimize the output of the slave expert model through the expert rule base, and then train the master expert model.
[0110] like Figure 3 As shown, in this embodiment, the present invention adopts an interactive tuning method based on the expert rule base constructed by power experts to train the main expert model. Each slave expert model corresponds to an electric power expert. The main expert model is used for unified updating and management to further improve the optimization effect of the power dispatching solution.
[0111] The S23 comprises the following sub-steps:
[0112] S231, outputting an initial power dispatching plan through a dispatching model, inputting the initial power dispatching plan into an expert model, and obtaining an adjusted power dispatching plan;
[0113] S232, performing fault business analysis on the adjusted power dispatching plan through an expert rule base, and generating feedback results of the adjusted power dispatching plan;
[0114] Among them, the expert rule base is set up by power experts, which includes various characteristic information of the power grid, such as: the actual operating parameters, operating environment, warning information and faulty equipment information of the power grid when a power grid fault occurs. The expert rule base is used to perform fault business analysis on the power dispatching plan, and judge the abnormal power grid operating parameters, abnormal environment, abnormal warning information and abnormal equipment when the fault occurs. When any abnormal information causes a power grid equipment failure, it is predicted whether the power dispatching plan will cause a power grid failure.
[0115] S233, sending the feedback result to the slave expert model through the virtual simulation power grid environment, and updating the model parameters of the slave expert model according to the feedback result;
[0116] S234. Feedback the model parameters updated from the expert model to the main expert model according to a preset period to train the main expert model.
[0117] In this embodiment, the master expert model is interactively tuned using the expert rule base to make it more consistent with the professional judgment of power experts. The slave expert model is adjusted using environmental feedback to make it more consistent with the actual situation of power grid operation. The updates of the slave expert model are regularly fed back to the master expert model to ensure that the master expert model can gradually absorb the knowledge and experience of various power experts.
[0118] In S234, the expression of the loss function Loss for training the main expert model is specifically:
[0119]
[0120] In the formula, is the expert confidence score of the expert rule base for the i-th adjusted power dispatching plan, is the predicted score of the ith adjusted power dispatch plan from the expert model, and N is the number of dispatch plans.
[0121] In this embodiment, the training goal of the expert model is to maximize the consistency between the confidence selected by the expert rule base and the prediction of the expert model, which is achieved by minimizing the relevant loss function.
[0122] The S3 comprises the following sub-steps:
[0123] S31, inputting the observed values in the virtual simulation power grid environment into the dispatching model to generate a power dispatching plan, wherein the observed values include node voltage, line current, generator output power and load demand;
[0124] S32, sending the power dispatching plan to the main expert model to generate an optimized power dispatching plan;
[0125] S33, performing fault business analysis on the optimized power dispatching plan through an expert rule base, and generating feedback results of the optimized power dispatching plan;
[0126] S34. Determine whether the optimized power dispatching plan is qualified according to the feedback results of the optimized power dispatching plan. If so, use the optimized power dispatching plan as the final power dispatching plan; if not, optimize the main expert model according to the feedback results of the optimized power dispatching plan and return to S32.
[0127] In this embodiment, the power dispatching scheme outputted by the expert model optimization dispatching model is adopted, and the optimized power dispatching scheme is subjected to fault business analysis through the expert rule base. If qualified, the scheme becomes the final power dispatching scheme. If unqualified, the expert model is further trained and optimized according to the feedback results to improve its performance in future tasks.
[0128] The beneficial effects of the present invention are as follows: the present invention provides a safe power dispatching method based on deep reinforcement learning, which effectively addresses various safety issues in power dispatching by learning relevant knowledge about power dispatching and combining the characteristics and disadvantages of deep reinforcement learning. The present invention is based on the Rainbow DQN algorithm and proposes two novel training methods, namely, training the adversarial model and the dispatching model based on adversarial learning to perform safety constraint assurance learning, and training the main expert model based on the interactive tuning of the expert rule base. The present invention proposes a safe power dispatching algorithm based on deep reinforcement learning, and the safety cost of the Markov decision process Taking voltage safety cost, current safety cost and frequency safety cost into consideration improves the safety of power grid operation. Theoretical analysis shows that this method has potential advantages in terms of safety and dispatch performance.
[0129] The purpose of this invention is to expand the problem into safe power dispatching, including load management, fault detection and isolation, emergency plans, etc., based on various safety standards and rules in power dispatching, combined with the characteristics of the power grid system, and finally establish a safe power dispatching algorithm based on deep reinforcement learning. It is used to balance supply and demand and reduce the pressure of peak load on the system through demand response, load transfer and peak-to-valley filling during peak power consumption; it is also used to quickly detect and isolate fault areas, prevent fault expansion, and reduce the impact on the power system; it is also used to formulate emergency plans to ensure rapid response and power restoration in the event of an emergency.
[0130] In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used only for descriptive purposes, and cannot be understood as indicating or implying the relative importance or the number of implicitly specified technical features. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of the features.
Claims
1. A safe power dispatching method based on deep reinforcement learning, characterized in that: The following steps are involved: S1, train the scheduling model based on adversarial learning using Markov decision process; S2, training the main expert model based on interactive tuning of the expert rule base; S3, outputting the final power dispatching plan according to the main expert model optimization dispatching model; The S1 is specifically: The action space, state space and reward function are defined through the Markov decision process, the behavior of the scheduling model and the adversarial model in the virtual simulation power grid environment is simulated, the network parameters of the scheduling model and the adversarial model are updated through the Rainbow DQN algorithm, and the scheduling model is trained through adversarial learning; The strategy defined by the Markov decision process is as follows: based on the observed data in the virtual simulation power grid environment, a power dispatching plan is generated through a dispatching model or an adversarial model; The observation space defined by the Markov decision process includes the grid information of node voltage value, line current value, generator output power and load power demand, and the state space The specific expression is: ; In the formula, represents the voltage value of all nodes, Indicates the current value of all lines, represents the output power of all generators, Represents the power demand of all loads; The action space defined by the Markov decision process includes all permutations and combinations of power dispatch actions. The specific expression is: ; In the formula, Indicates the adjustment value of the generator output power, Indicates the switch operation status, Indicates load adjustment; The reward function defined by the Markov decision process is specifically based on the goals of power grid operation, including power supply reliability, economic benefits and robustness. The reward function R t The specific expression is: ; ; ; ; ; In the formula, Indicates power supply reliability, used to measure voltage stability, frequency deviation, represents the voltage of the ith node, represents the rated voltage of each node in the power grid, N represents the total number of nodes in the power grid, represents the current of the jth line, represents the maximum allowable current of the jth line, M represents the total number of lines in the power grid, Represents economic benefits, used to measure the cost of power generation, is the cost coefficient of the kth generator, G represents the total number of generators, represents the adjustment value of the output power of the kth generator, Represents environmental benefits and is used to measure pollutant emissions. is the emission coefficient of the kth generator, Represents robustness, which is used to measure the ability to resist failure. is the impact assessment value of the lth fault, F represents the total number of faults, represents the weight of the first indicator, represents the weight of the second indicator, represents the weight of the third indicator, represents the weight of the fourth indicator; The expression for updating the network parameters of the scheduling model and the adversarial model through the Rainbow DQN algorithm is as follows: ; In the formula, For having Atomic vector The projection on represents the number of atoms on the discrete support, is the target distribution, is the KL divergence, is the probability of time difference error, To adjust right The control parameters of the degree of influence; In S1, the security cost of the Markov decision process according to Status settings; Security costs The specific expression is: ; ; ; ; In the formula, represents the first weight coefficient, represents the second weight coefficient, represents the third weight coefficient, Indicates the voltage safety cost, which is used to reflect the degree to which the voltage deviates from the rated value. Indicates the allowable voltage deviation range, Indicates the current safety cost, which is used to reflect whether the line current exceeds the limit of safe operation. Represents the frequency security cost, which is used to measure the frequency deviation of the power grid. Indicates the current grid frequency. Indicates the rated frequency of the power grid, Indicates the allowable frequency deviation range, represents the maximum value function; The objective function of adversarial learning training for the scheduling model is as follows: ; In the formula, Indicates the current state. represents the actions of the scheduling model, represents disturbance, It indicates the state to which the system transfers after executing the scheduling and disturbance. represents the strategy of the scheduling model, represents the strategy of adversarial model, and All are obtained through the state space, Acquired through action space; Indicates that the adversarial model tries to find the most unfavorable perturbation , It means that the scheduling model learns how to adjust the scheduling strategy under the most unfavorable scenario generated by the adversarial model. Represents the average of all possible state transitions; The S2 comprises the following sub-steps: S21, setting a master expert model and a slave expert model, the master expert model includes basic knowledge and preliminary optimization capabilities of power dispatching, and the slave expert model is generated based on a copy of the master expert model; S22, initializing the master expert model and the slave expert model; S23, optimizing the output of the slave expert model through the expert rule base, and then training the master expert model; The S23 comprises the following sub-steps: S231, outputting an initial power dispatching plan through the dispatching model, inputting the initial power dispatching plan into the expert model, and obtaining an adjusted power dispatching plan; S232, performing fault business analysis on the adjusted power dispatching plan through an expert rule base, and generating feedback results of the adjusted power dispatching plan; S233, sending the feedback result to the slave expert model through the virtual simulation power grid environment, and updating the model parameters of the slave expert model according to the feedback result; S234, feeding back the model parameters updated from the expert model to the main expert model according to a preset period, and training the main expert model; The S3 comprises the following sub-steps: S31, inputting the observed values in the virtual simulation power grid environment into the dispatching model to generate a power dispatching plan, wherein the observed values include node voltage, line current, generator output power and load demand; S32, sending the power dispatching plan to the main expert model to generate an optimized power dispatching plan; S33, performing fault business analysis on the optimized power dispatching plan through the expert rule base, and generating feedback results of the optimized power dispatching plan; S34. Determine whether the optimized power dispatching plan is qualified according to the feedback results of the optimized power dispatching plan. If so, use the optimized power dispatching plan as the final power dispatching plan; if not, optimize the main expert model according to the feedback results of the optimized power dispatching plan and return to S32.
2. The safe power dispatching method based on deep reinforcement learning according to claim 1 is characterized in that: In S234, the expression of the loss function Loss for training the main expert model is specifically: ; In the formula, is the expert confidence score of the expert rule base for the i-th adjusted power dispatching plan, is the predicted score of the ith adjusted power dispatch plan from the expert model, and N is the number of dispatch plans.
Citation Information
Patent Citations
Adaptive self-learning method for regulation and control operation decision aided analysis expert model
CN108074037A
Power grid emergency control method based on expert system and deep reverse reinforcement learning
CN111179121A
Power grid real-time adaptive decision-making method based on deep reinforcement learning
CN114217524A
Intelligent replenishment decision-making method based on reinforcement learning and imitation learning and storage medium
CN114781724A
Unit commitment deep reinforcement learning solving method
CN117117989A
