Power grid operation risk control strategy generation system and method

Abstract

The invention discloses a power grid operation risk control strategy generation system and method, which can learn a unit output decision for ensuring safe, stable and economical operation of a power grid based on a power grid power flow out-of-limit self-adaptive adjustment agent of out-of-limit reward function and load balance post-processing guide training, and improve the phenomena of low efficiency and insufficiency of traditional adjustment depending on artificial experience. The training effect of the power grid out-of-limit self-adaptive adjustment intelligent agent is further improved by realizing high-level simulation of the real environment of the power grid, and adjustment treatment based on artificial experience can be gradually replaced by continuous iterative learning and training in the later period and can be directly applied to the actual production environment of the power grid.

Description

Technical Field

[0001] The present invention belongs to the technical field of power systems, and specifically relates to a system and method for generating a power grid operation risk control strategy, and also relates to a system and method for generating a power grid operation risk control strategy based on hybrid enhancement of deep reinforcement learning and boundary knowledge. Background Art

[0002] As the operating characteristics and control modes of power systems become increasingly complex, the operation modes of power grids become more diverse, the objects of dispatching and control grow exponentially, and the uncertainty on both the source and load sides increases, making it more difficult for controllers to handle power grid faults. A more automated and intelligent approach is needed to handle power grid faults.

[0003] At present, in the power grid control business mentioned in the patent publication number "CN110502604B", it is generally based on conventional manual processing, and the conventional manual processing method has the disadvantages of misjudgment, missed judgment and low processing efficiency.

[0004] Therefore, in order to adapt to the complex and uncertain power grid environment, seek the optimal long-term control objectives, achieve the safety and economy of power grid operation, and make real-time disposal decisions after a power grid fault occurs, it is necessary to propose a power grid over-limit adaptive decision-making method based on reinforcement learning, build a decision-making intelligent agent, learn the optimal decision in an offline power grid simulation environment, and apply the learned strategy in the real power grid environment, so as to solve the shortcomings of conventional manual disposal methods such as misjudgment, missed judgment and low disposal efficiency. Summary of the invention

[0005] In order to solve the defects existing in the prior art, the present invention proposes a system and method for generating a risk control strategy for power grid operation. The power grid current over-limit adaptive adjustment intelligent agent trained based on the over-limit reward function and load balance post-processing guidance can learn to ensure the unit output decision of the safe, stable and economical operation of the power grid, and improve the inefficiency and insufficiency of the traditional adjustment based on manual experience. By realizing a high degree of simulation of the real environment of the power grid, the training effect of the power grid over-limit adaptive adjustment intelligent agent is further improved. In the later stage, through continuous iterative learning and training, it can gradually replace the adjustment and disposal of manual experience and be directly applied to the actual production environment of the power grid.

[0006] The present invention utilizes the following technical solutions.

[0007] A method for generating a power grid operation risk control strategy, comprising:

[0008] Step 1: Construct a power grid power flow over-limit adaptive adjustment model based on deep reinforcement learning;

[0009] Step 2: Build an integrated framework for offline training and online application.

[0010] Preferably, step 1 specifically includes:

[0011] Step 1-1: Construct the action space and state space of the power flow over-limit elimination agent;

[0012] Step 1-2: Construct a power flow over-limit reward function;

[0013] Step 1-3: Processing is performed after the output and load are balanced;

[0014] Step 1-4: Build a power grid simulation operation reinforcement learning environment.

[0015] Preferably, in step 1-1, the action space of the power flow over-limit elimination agent is constructed, that is, the action space of the agent is determined as shown in the following formula (1):

[0016] A={P units} (1)

[0017] Among them, P units It is the active output of each unit at the next moment generated based on the current power grid status.

[0018] Preferably, in step 1-1, the state space of the power flow over-limit elimination agent is constructed, that is, the state space shown in the following formula (2) is constructed for the agent:

[0019] S={P units , Q units , U lines , I lines , P loads} (2)

[0020] Among them, P units is the active output of each unit in the power grid at the current moment, Q units is the reactive power output of each unit in the power grid at the current moment, U lines is the voltage value of each line in the power grid at the current moment, I lines is the current value of each line in the power grid at the current moment, P loads It is the load forecast value in the power grid at the next moment.

[0021] Preferably, in step 1-2, a power flow over-limit reward function as shown in the following formula (3) is constructed:

[0022]

[0023] in, is the actual current carrying capacity of the nth line in the power grid flow at the current moment, is the upper limit of the current carrying capacity of the nth line in the power grid flow, is the lower limit of the current carrying capacity of the nth line in the power grid flow, N is the total number of power grid lines, max() is the maximum value function, and R is the reward value of the safety target.

[0024] Preferably, in steps 1-3, after the intelligent agent generates a decision through the deep neural network, the value is normalized to a range of [-1, 1], which needs to be combined with the adjustable range of the unit and then mapped to a specific unit output value.

[0025] Preferably, in the post-processing process, the sum of the predicted load values ​​at the next moment is taken as the adjustment target, that is, as the sum of the outputs of each unit, and the adjustment amount is allocated according to the set ratio based on the adjustable range of each unit at this moment, so as to ultimately achieve that the sum of the unit outputs is equal to the sum of the predicted load values.

[0026] Preferably, steps 1-4 specifically include performing flow calculations on the power grid operation simulation environment to determine the voltage and phase angle of each node in the power system and the flow of each transmission line. When performing the flow calculations, the interactive service module realizes the integration of the interactive environment and the flow calculation module. The interactive service module is a Java-based software program that is responsible for data communication and transmission of control instructions with the reinforcement learning agent.

[0027] Preferably, the functions of the interactive service module are as follows:

[0028] Receive and parse the action request of the intelligent agent, and update the unit and load data in the power system model according to the control variables in the request;

[0029] Call the power flow calculation module to solve the voltage of each node and the power flow of each line according to the structural parameters and load conditions in the power system model, and determine whether the power system meets the constraint conditions;

[0030] Generate a new QS file and save the power flow calculation results in the QS file;

[0031] Relevant data of units, lines, and busbar equipment are extracted from the QS file and returned to the intelligent agent as the power grid status for learning and decision-making.

[0032] Preferably, step 2 specifically includes:

[0033] Step 2-1: After reading the grid information at this time, the simulation environment initializes the agent state space and action space to adapt to the grid structure, and constructs the actor and critic network based on the deep neural network;

[0034] Step 2-2: Determine the time period of training data, total number of training steps, number of training rounds, and the number of steps included in each round;

[0035] Step 2-3: In the first 1000 training steps, which is the exploration phase, the agent requests the state information at a random time from the interactive environment service program at the beginning of each round;

[0036] Step 2-4: The interactive environment service program reads the QS file at the current moment and the QS file at the next moment, parses the data therein and returns the observation state data required by the agent;

[0037] Step 2-5: After receiving the observed state data, the agent uses it as input and uses the actor network to obtain the unit adjustment output value in the interval [-1,1];

[0038] Step 2-6: Post-process the output value obtained in step 2-5. During the training process, the actual load value at the next moment is directly used as the load prediction value at the next moment to achieve the balance adjustment between the unit output and the predicted load value, and obtain the actual output value of the unit;

[0039] Step 2-7: The agent returns the actual output value of the unit to the interactive environment service program. After receiving the actual output value of the unit, the interactive environment service program directly replaces it with the output value of the corresponding unit in the QS file at the next moment, and performs power flow calculation. Then, the QS file generated after the power flow calculation and the data extracted from the QS file at the next moment are returned to the agent as the observation state at the next moment, and the over-limit information after the power flow calculation is returned at the same time;

[0040] Step 2-8: After receiving the data, the agent first calculates the reward value of this action based on the limit-crossing information, and stores it together with the current moment observation state of this action, the current action, and the next moment observation state as a piece of data in the training data cache for subsequent learning;

[0041] Step 2-9: At this time, the number of training steps for the next round of the agent increases by 1, and the total number of training steps increases by 1, and the operation returns to step 2-5; when the number of round steps reaches 12, this round ends, the total number of rounds increases by 1, and a new round of training begins. When the total number of training steps reaches the parameters set in step 2-2, the agent training is completed, the training parameters are saved, and the program is exited to end the training;

[0042] Step 2-10: After 1000 steps of training are completed, it is the application stage. In addition to following the process of steps 2-3 to 2-9 at the beginning of each round, the initial moment is selected as much as possible when the historical data exceeds the limit. At this time, after each training step interaction, it is also necessary to randomly read n data from the training data cache, and use the gradient descent method to update the critic network parameters and the actor network parameters to optimize the parameters of the intelligent agent and generate the best strategy.

[0043] After the model completes offline training based on actual historical power grid data, the intelligent agent has mastered the generation of unit output regulation strategies that are safe, within limits, and balance power supply with predicted loads.

[0044] Preferably, the method for generating a power grid operation risk control strategy further includes: placing the trained intelligent agent in an online application, and the online decision-making process is as follows:

[0045] The agent reads the parameters saved after training and uses them to initialize itself, and is ready to make decisions;

[0046] If the operator finds that the power grid is out of limit at this time, after taking corresponding actions, the production environment will generate a new QS file. At this time, the interactive environment service program deployed in the application environment will extract the out-of-limit QS file at the previous moment and the latest QS file data and return them to the intelligent agent as the observation state;

[0047] After receiving the observed state data, the agent uses it as input and uses the actor network to obtain the unit adjustment output value in the interval [-1,1]. After post-processing, the actual adjustment value of the unit is obtained and returned to the interactive environment service program. This is the adjustment strategy generated for the over-limit situation.

[0048] After the interactive environment service program receives the actual output value of the unit, it directly replaces it with the output value of the corresponding unit in the QS file at the next moment, and performs power flow calculation to obtain the QS file of the power grid status after the intelligent agent decision is adopted. By comparing it with the QS file formed after manual operation in the production environment, it can be seen from the effect that the power grid after adjustment by the intelligent agent is in a safe and limit-free state.

[0049] A power grid operation risk control strategy generation system, comprising:

[0050] A construction module, which is used to construct a power grid power flow over-limit adaptive adjustment model based on deep reinforcement learning;

[0051] The building module is used to build an integrated framework for offline training and online applications.

[0052] The beneficial effects of the present invention are that, compared with the prior art, the technical effects of the present invention include:

[0053] The power grid current over-limit adaptive adjustment agent based on over-limit reward function and load balance post-processing guidance training can learn to ensure the safe, stable and economical operation of the power grid, and improve the inefficiency and insufficiency of traditional adjustment based on manual experience. By achieving a high degree of simulation of the real environment of the power grid, the training effect of the power grid over-limit adaptive adjustment agent is further improved. In the later stage, through continuous iterative learning and training, it can gradually replace the adjustment and disposal of manual experience and be directly applied to the actual production environment of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of the method for generating a power grid operation risk control strategy in the present invention;

[0055] Figure 2 It is a schematic diagram of the architecture of the power grid operation risk control strategy generation system described in the present invention;

[0056] Figure 3 It is a schematic diagram of post-processing of the unit output in the present invention;

[0057] Figure 4 It is a schematic diagram of the integration of the interactive environment and the flow calculation module in the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely expressed in combination with the drawings in the embodiments of the present invention. The embodiments expressed in this application are only part of the embodiments of the present invention, not all of them. According to the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative labor are all within the protection scope of the present invention.

[0059] like Figure 1 As shown, a method for generating a power grid operation risk control strategy according to the present invention includes:

[0060] In the complex and ever-changing new power system, power grid failures occur from time to time, resulting in cross-section and equipment over-limit situations, which poses a great safety hazard to the stable operation of the power grid. How to adapt to the new power system with such strong uncertainty and adjust the unit output in time when a failure occurs to eliminate the over-limit state is an urgent problem to be solved in the current dispatching field.

[0061] The present invention proposes a grid over-limit adaptive adjustment model based on reinforcement learning. First, a decision-making intelligent agent is constructed, the state space and action space of the intelligent agent are analyzed and established, the optimization objectives are analyzed and disassembled, and a reward function that considers the impact of over-limit is constructed. At the same time, in order to achieve the goal of economic operation, the unit output post-processing is innovatively proposed and introduced to achieve the linear constraint condition of the balance between the unit output and the grid load. Finally, the flow calculation is introduced to build a complete and accurate simulation environment for simulating the operation of the grid. In an offline environment, the intelligent agent is trained based on the real historical grid operation data, and the relevant parameters are saved after achieving satisfactory results. The trained intelligent agent is then put into online applications, facing the real grid operation data, making decisions in real time, eliminating the over-limit state that occurs when the grid fails, and achieving the goal of economic operation of the grid on the basis of ensuring the safe and stable operation of the grid. The method for generating a grid operation risk control strategy is as follows:

[0062] Step 1: Construct a power grid power flow over-limit adaptive adjustment model based on deep reinforcement learning;

[0063] The present invention constructs an adaptive adjustment model for power grid flow over-limit based on deep reinforcement learning. According to the actual over-limit scenario generated when the power grid fails, the state space and action space of the intelligent agent are constructed, the interaction rules between the intelligent agent and the operating environment are formulated, and the reward function is constructed to guide the intelligent agent to continuously learn and explore the unit adjustment action when the power grid exceeds the limit. Combined with post-processing, the optimal decision-making plan is formed. Constructing an adaptive adjustment model for power grid flow over-limit based on deep reinforcement learning is also called modeling.

[0064] In a preferred but non-limiting embodiment of the present invention, step 1 specifically comprises:

[0065] Step 1-1: Construct the action space and state space of the power flow over-limit elimination agent;

[0066] In the problem of power grid regulation, the action space of the power flow over-limit elimination agent generally includes different types of actions such as unit active and reactive power regulation, bus voltage regulation, line disconnection, etc. These actions will have different degrees of impact on the power grid state, so these impacts need to be quantified and evaluated during the modeling process.

[0067] For example, adjusting the generator output can change the power output of the generator, thus affecting the operation of the entire power system. Specifically, increasing the generator output can improve the system's active power supply capacity, but it will also increase the system's reactive power loss; reducing the generator output can reduce the system's reactive power loss, but it will also reduce the system's active power supply capacity. Therefore, in the modeling process, it is necessary to calculate and analyze the impact of changes in generator output on the system's active and reactive power balances.

[0068] Adjusting the transformer gear can change the voltage ratio at both ends of the transformer, thus affecting the operation of electrical equipment. Specifically, increasing the transformer gear can increase the voltage level on the low-voltage side of the transformer, but it will also increase the current level on the high-voltage side of the transformer; lowering the transformer gear can reduce the current level on the high-voltage side of the transformer, but it will also reduce the voltage level on the low-voltage side of the transformer. Therefore, in the modeling process, it is necessary to calculate and analyze the impact of the change in transformer gear on the voltage and current at both ends of the transformer.

[0069] Opening a line can cut off the current transmission on the line, thus affecting the nodes at both ends of the line. Specifically, opening a line can reduce the active and reactive losses on the line, but it will also change the impedance relationship between the nodes at both ends of the line; closing a line can increase the active and reactive losses on the line, but it will also restore the impedance relationship between the nodes at both ends of the line. Therefore, in the modeling process, it is necessary to calculate and analyze the impact of line opening on the voltage, current, and impedance of the nodes at both ends of the line.

[0070] In the modeling process, it is necessary to determine the actions that the agent can take based on the actual situation of the power grid and incorporate these actions into the action space. The size and complexity of the action space depends on the type, quantity, and scope of the devices controlled by the agent. Generally speaking, the larger and more complex the action space, the more difficult it is to solve the agent control problem; the smaller and simpler the action space, the easier it is to solve the agent control problem. Therefore, in the modeling process, it is necessary to select an appropriate action space based on the difficulty and goal of the agent control problem.

[0071] In a preferred but non-limiting embodiment of the present invention, in step 1-1, by analyzing the optimization target of the agent decision in the system of the present invention, it can be found that the main adjustment object is the active power output of the unit, so the action space of the power flow over-limit elimination agent is finally constructed, that is, the action space of the agent is determined as shown in the following formula (1):

[0072] A={P units} (1)

[0073] Among them, P units It is the active output of each unit at the next moment generated based on the current power grid status.

[0074] In a preferred but non-limiting embodiment of the present invention, in step 1-1, the state space mainly refers to a useful data set that the agent can observe through the environmental interaction process. Similarly, in combination with the power grid operation environment of this article, the state space of the power flow over-limit elimination agent can be constructed, that is, the state space shown in the following formula (2) is constructed for the agent:

[0075] S={P units , Q units , U lines , I lines , P loads} (2)

[0076] Among them, P units is the active output of each unit in the power grid at the current moment, Q units is the reactive power output of each unit in the power grid at the current moment, U lines is the voltage value of each line in the power grid at the current moment, I linesis the current value of each line in the power grid at the current moment, P loads It is the load forecast value in the power grid at the next moment.

[0077] Step 1-2: Construct a power flow over-limit reward function;

[0078] The present invention uses reinforcement learning technology, takes safe, stable and economical operation of the power grid as the optimization goal, generates the initial output decision plan of the unit through a deep neural network, and forms an actual adjustment strategy in combination with the post-processing conditions of the power grid constraints.

[0079] For the application of reinforcement learning, in addition to building the environment, training decision-making agents is also an important part. The key to how to train agents lies in the guidance of the reward and punishment mechanism. Therefore, in order for the agents to generate effective decisions, it is necessary to construct the corresponding agent reward function.

[0080] In a preferred but non-limiting embodiment of the present invention, in step 1-2, first, with safety as the training goal, the line equipment and the power grid section are controlled not to exceed the limit, and the power flow exceeding limit reward function shown in the following formula (3) is constructed:

[0081]

[0082] in, is the actual current carrying capacity of the nth line in the power grid flow at the current moment, is the upper limit of the current carrying capacity of the nth line in the power grid flow, is the lower limit of the current carrying capacity of the nth line in the power grid flow, N is the total number of power grid lines, max() is the maximum value function, and R is the reward value of the safety target.

[0083] When the actual current carrying capacity of a line exceeds the upper limit of the current carrying capacity of this line, it means that the line is out of limit. In the experiments and applications of the present invention, the sum of the over-limit current carrying capacities is within 1000, so the over-limit current carrying capacity is divided by 10 (this parameter can be dynamically adjusted according to the actual situation of the current power grid environment when in use) to obtain an over-limit risk score calculated in percentage. Subtract this value from 100 and limit it to a non-negative number to obtain the reward value of the safety target. The higher the reward value, the safer the line is at this time, thereby motivating the intelligent agent to learn actions that can achieve the reward value, thereby training the best decision with safety as the goal.

[0084] Step 1-3: Processing is performed after the output and load are balanced;

[0085] For pure neural networks, even if it is a well-trained network, after inputting new data, there is no guarantee of its output value. There is great uncertainty in the range. At the same time, the role of the neural network is not to obtain a value that is completely unbiased with the sample output after the data is input. This cannot be directly applied in power grid scenarios where the specific adjustment amount of each unit needs to be determined and the adjustment amount still has an effective range. Therefore, it is necessary to perform quantitative post-processing operations on the output of the neural network.

[0086] Considering that it is difficult to achieve convergence by simply letting the intelligent agent learn load balance, which is almost an equality constraint, after the intelligent agent gives an action, the difference between the predicted load and the adjustable space of each unit is calculated to adjust the decision output. Therefore, the present invention needs to add post-processing to the traditional reinforcement learning process. The overall process of the method of the present invention is as follows Figure 1 shown.

[0087] In the system implemented by the present invention, in a preferred but non-restrictive implementation manner of the present invention, in steps 1-3, after the intelligent agent generates a decision through the deep neural network, the value is in the range of [-1,1] after normalization, which needs to be combined with the adjustable range of the unit and then mapped to a specific unit output value. This is the post-processing operation designed by the present invention.

[0088] In a preferred but non-limiting embodiment of the present invention, Figure 3 As shown, in order to achieve economical regulation of power grid operation, in the post-processing process, the sum of the predicted load values ​​at the next moment is taken as the regulation target, that is, as the sum of the outputs of each unit, and according to the adjustable range of each unit at this moment, the regulation amount is allocated according to the set ratio, and finally the sum of the unit output is equal to the sum of the predicted load values.

[0089] Step 1-4: Build a power grid simulation operation reinforcement learning environment.

[0090] In the application of reinforcement learning, the construction of a reinforcement learning environment is essential, which is equivalent to the basis of all training and learning. Without a reinforcement learning environment, the learning of the intelligent agent will be meaningless, so the construction of a reinforcement learning environment is the top priority of reinforcement learning, which is also an important feature that distinguishes reinforcement learning from other traditional machine learning. Therefore, it is necessary to combine the state space and action space of the intelligent agent, reward function, post-processing, and the characteristics of the actual power grid environment to build a reinforcement learning environment.

[0091] In order to build a reinforcement learning environment, an interactive environment is needed that can simulate the operating state of the power system and make corresponding adjustments according to the actions of the intelligent agent. In order to simulate the characteristics of the actual power grid environment, power flow calculation needs to be introduced. Power flow calculation is a basic calculation method in power system analysis, which is used to calculate the voltage amplitude and phase angle of each node and the power flow of each branch under steady-state operating conditions. The purpose of power flow calculation is to determine whether the power system can meet various operating constraints under given operating conditions, as well as the operating status of each node and branch. The results of power flow calculation can provide an important basis for the operation and planning of the power system. For example, the power transmission capacity of the power system can be determined by power flow calculation, thereby providing a reference for the expansion planning of the power system. In addition, power flow calculation can also be used for safety analysis of the power system. By analyzing the results of power flow calculation, it is determined whether there are safety hazards in the power system. Power flow calculation is the basis of power system analysis and optimization, and it can provide important decision-making basis for intelligent agents.

[0092] In a preferred but non-limiting embodiment of the present invention, steps 1-4 specifically include: In the environment of the present invention, it is necessary to perform power flow calculation on the power grid operation simulation environment to determine the voltage and phase angle of each node in the power system and the power flow of each transmission line, and when performing power flow calculation, it is necessary to consider multiple factors, such as transmission line impedance, node injection power, node type, etc. Therefore, an interactive service module is developed to realize the integration of the interactive environment and the power flow calculation module. The interactive service module is a Java-based software program, which is responsible for data communication and control instruction transmission with the reinforcement learning agent.

[0093] In a preferred but non-limiting embodiment of the present invention, the functions of the interactive service module are as follows:

[0094] Receive and analyze the action requests of the intelligent agent, and update the data of units, loads, etc. in the power system model according to the control variables in the request (such as generator output, bus voltage, transformer ratio, etc.); these data are important factors affecting the operating status of the power system, and by adjusting them, the power system can be optimally controlled;

[0095] Call the power flow calculation module to solve the voltage of each node and the power flow of each line according to the structural parameters and load conditions in the power system model, and determine whether the power system meets the constraints (such as bus voltage range, line power limit, etc.); power flow calculation is the basis of power system analysis and optimization, which can reflect the operating status and performance indicators of the power system, as well as the effect of the intelligent agent's actions;

[0096] Generate a new QS file and save the power flow calculation results in the QS file for subsequent analysis and optimization; QS file is a file format used to store power system data and results, which can easily read and process data;

[0097] Extract relevant data of equipment such as units, lines, and buses (such as node voltage, line power, etc.) from the QS file and return it to the agent as the power grid state for learning and decision-making. These data are environmental information observed by the reinforcement learning agent, which can help the agent evaluate the reward or penalty in the current state and choose the next action according to the strategy.

[0098] In this way, the integration of the interactive environment and the power flow calculation module is realized, providing a reliable platform for the power grid fault handling system based on reinforcement learning, and constructing a simulation environment for the operation of reinforcement learning agents, such as Figure 4 shown.

[0099] Step 2: Build an integrated framework for offline training and online application.

[0100] Based on deep reinforcement learning technology, the present invention constructs an integrated power grid flow over-limit adaptive decision-making system framework that can realize offline training and online application.

[0101] In a preferred but non-limiting embodiment of the present invention, step 2 specifically includes: in the offline training part, collecting and using historical power grid information QS file data. At the same time, considering that the input state is continuous power grid data and the output is also continuous unit output adjustment, the DDPG algorithm with good application effect in reinforcement learning is used to train the intelligent agent. The training process of this system is as follows:

[0102] Step 2-1: After reading the grid information at this time, the simulation environment initializes the agent state space and action space to adapt to the grid structure, and constructs the actor and critic network based on the deep neural network;

[0103] Step 2-2: Determine the training data time period, total number of training steps, number of training rounds, number of steps in a round, and other agent training condition parameters;

[0104] Step 2-3: In the first 1000 training steps, which is the exploration phase, the agent requests the state information at a random time from the interactive environment service program at the beginning of each round;

[0105] Step 2-4: The interactive environment service program reads the QS file at the current moment and the QS file at the next moment, parses the data therein and returns the observation state data required by the agent;

[0106] Step 2-5: After receiving the observed state data, the agent uses it as input and uses the actor network to obtain the unit adjustment output value in the interval [-1,1];

[0107] Step 2-6: Post-process the output value obtained in step 2-5. During the training process, the actual load value at the next moment is directly used as the load prediction value at the next moment to achieve the balance adjustment between the unit output and the predicted load value, and obtain the actual output value of the unit;

[0108] Step 2-7: The agent returns the actual output value of the unit to the interactive environment service program. After receiving the actual output value of the unit, the interactive environment service program directly replaces it with the output value of the corresponding unit in the QS file at the next moment, and performs power flow calculation. Then, the QS file generated after the power flow calculation and the data extracted from the QS file at the next moment are returned to the agent as the observation state at the next moment, and the over-limit information after the power flow calculation is returned at the same time;

[0109] Step 2-8: After receiving the data, the agent first calculates the reward value of this action based on the limit-crossing information, and stores it together with the current moment observation state of this action, the current action, and the next moment observation state as a piece of data in the training data cache for subsequent learning;

[0110] Step 2-9: At this time, the number of training steps for the next round of the agent increases by 1, and the total number of training steps increases by 1, and the operation returns to step 2-5; when the number of round steps reaches 12, this round ends, the total number of rounds increases by 1, and a new round of training begins. When the total number of training steps reaches the parameters set in step 2-2, the agent training is completed, the training parameters can be saved, and the program can be exited to end the training;

[0111] Step 2-10: After 1000 steps of training are completed, it is the application stage. In addition to following the process of steps 2-3 to 2-9 at the beginning of each round, the initial moment is selected as much as possible when the historical data exceeds the limit. At this time, after each training step interaction, it is also necessary to randomly read n data from the training data cache, and use the gradient descent method to update the critic network parameters and the actor network parameters to optimize the parameters of the intelligent agent and generate the best strategy.

[0112] After the model completes offline training based on actual historical power grid data, the intelligent agent has mastered the generation of unit output regulation strategies that are safe, within limits, and balance power supply with predicted loads.

[0113] In a preferred but non-restrictive implementation of the present invention, the method for generating a grid operation risk control strategy also includes: and because there is historical over-limit data in the training data, the exploration and learning application of this data is deliberately increased during training, so when the intelligent agent faces over-limit data again, it can quickly generate an adjustment plan to reduce the over-limit situation, which will also be applied in an online environment. Because grid regulation operations are very demanding and require very high safety, control is not currently handed over to artificial intelligence programs. Therefore, the present invention will use real-time grid operation data for simulation and then compare it with the actual situation. The trained intelligent agent is placed in an online application, and its online decision-making process is as follows:

[0114] The agent reads the parameters saved after training and uses them to initialize itself, and is ready to make decisions;

[0115] If the operator finds that the power grid is out of limit at this time, after taking corresponding actions, the production environment will generate a new QS file. At this time, the interactive environment service program deployed in the application environment will extract the out-of-limit QS file at the previous moment and the latest QS file data and return them to the intelligent agent as the observation state;

[0116] After receiving the observed state data, the agent uses it as input and uses the actor network to obtain the unit adjustment output value in the interval [-1,1]. After post-processing, the actual adjustment value of the unit is obtained and returned to the interactive environment service program. This is the adjustment strategy generated for the over-limit situation.

[0117] After the interactive environment service program receives the actual output value of the unit, it directly replaces it with the output value of the corresponding unit in the QS file at the next moment, and performs power flow calculation to obtain the QS file of the power grid status after the intelligent agent decision is adopted. By comparing it with the QS file formed after manual operation in the production environment, it can be seen from the effect that the power grid after adjustment by the intelligent agent is in a safe and limit-free state.

[0118] like Figure 2 As shown, a power grid operation risk control strategy generation system according to the present invention comprises:

[0119] A construction module, which is used to construct a power grid power flow over-limit adaptive adjustment model based on deep reinforcement learning;

[0120] The building module is used to build an integrated framework for offline training and online applications.

[0121] The beneficial effects of the present invention are that, compared with the prior art, the technical effects of the present invention include:

[0122] The power grid current over-limit adaptive adjustment agent based on over-limit reward function and load balance post-processing guidance training can learn to ensure the safe, stable and economical operation of the power grid, and improve the inefficiency and insufficiency of traditional adjustment based on manual experience. By achieving a high degree of simulation of the real environment of the power grid, the training effect of the power grid over-limit adaptive adjustment agent is further improved. In the later stage, through continuous iterative learning and training, it can gradually replace the adjustment and disposal of manual experience and be directly applied to the actual production environment of the power grid.

[0123] The present disclosure can be a system, a method and / or a computer program product. The computer program product can include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement each aspect of the present disclosure.

[0124] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the above. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (such as a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0125] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0126] The computer program instructions for performing the operations of the present disclosure can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer-readable program instructions can be executed entirely on the controller computer, partially on the controller computer, as a separate software package, partially on the controller computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the controller computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (such as connecting through the Internet using an Internet service provider). In some embodiments, each aspect of the present disclosure is implemented by utilizing state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), which can execute computer-readable program instructions.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be edited or replaced by equivalents, and any editing or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for generating a power grid operation risk control strategy, characterized in that: include: Step 1: Construct a power grid power flow over-limit adaptive adjustment model based on deep reinforcement learning; Step 2: Build an integrated framework for offline training and online application.

2. The method for generating a power grid operation risk control strategy according to claim 1, characterized in that: Step 1 specifically includes: Step 1-1: Construct the action space and state space of the power flow over-limit elimination agent; Step 1-2: Construct a power flow over-limit reward function; Step 1-3: Processing is performed after the output and load are balanced; Step 1-4: Build a power grid simulation operation reinforcement learning environment.

3. The method for generating a power grid operation risk control strategy according to claim 2, characterized in that: In step 1-1, the action space of the power flow over-limit elimination agent is constructed, that is, the action space of the agent is determined as shown in the following formula (1): A={P units } (1) Among them, P units It is the active output of each unit at the next moment generated based on the current power grid status; In step 1-1, the state space of the power flow over-limit elimination agent is constructed, that is, the state space shown in the following formula (2) is constructed for the agent: S={P units ,Q units ,U lines ,I lines ,P loads } (2) Among them, P units is the active output of each unit in the power grid at the current moment, Q units is the reactive power output of each unit in the power grid at the current moment, U lines is the voltage value of each line in the power grid at the current moment, I lines is the current value of each line in the power grid at the current moment, P loads It is the load forecast value in the power grid at the next moment.

4. The method for generating a power grid operation risk control strategy according to claim 3, characterized in that: In step 1-2, the power flow over-limit reward function is constructed as shown in the following formula (3): in, is the actual current carrying capacity of the nth line in the power grid flow at the current moment, is the upper limit of the current carrying capacity of the nth line in the power grid flow, is the lower limit of the current carrying capacity of the nth line in the power grid flow, N is the total number of power grid lines, max() is the maximum value function, and R is the reward value of the safety target.

5. The method for generating a power grid operation risk control strategy according to claim 4, characterized in that: In steps 1-3, after the agent generates a decision through the deep neural network, the value is normalized to the range of [-1,1] and needs to be combined with the adjustable range of the unit and then mapped to the specific unit output value.

6. The method for generating a power grid operation risk control strategy according to claim 5, characterized in that: In the post-processing process, the sum of the predicted load values ​​at the next moment is taken as the adjustment target, that is, the sum of the outputs of each unit, and the adjustment amount is allocated according to the set ratio based on the adjustable range of each unit at this moment, so as to finally achieve the sum of the unit outputs equal to the sum of the predicted load values.

7. The method for generating a power grid operation risk control strategy according to claim 6, characterized in that: Steps 1-4 specifically include performing power flow calculation on the power grid operation simulation environment to determine the voltage and phase angle of each node in the power system and the power flow of each transmission line. When performing power flow calculation, the interactive service module realizes the integration of the interactive environment and the power flow calculation module. The interactive service module is a Java-based software program that is responsible for data communication and control command transmission with the reinforcement learning agent. The functions of the interactive service module are as follows: Receive and parse the action request of the intelligent agent, and update the unit and load data in the power system model according to the control variables in the request; Call the power flow calculation module to solve the voltage of each node and the power flow of each line according to the structural parameters and load conditions in the power system model, and determine whether the power system meets the constraint conditions; Generate a new QS file and save the power flow calculation results in the QS file; Relevant data of units, lines, and busbar equipment are extracted from the QS file and returned to the intelligent agent as the power grid status for learning and decision-making.

8. The method for generating a power grid operation risk control strategy according to claim 7, characterized in that: Step 2 specifically includes: Step 2-1: After reading the grid information at this time, the simulation environment initializes the agent state space and action space to adapt to the grid structure, and constructs the actor and critic network based on the deep neural network; Step 2-2: Determine the time period of training data, total number of training steps, number of training rounds, and the number of steps included in each round; Step 2-3: In the first 1000 training steps, which is the exploration phase, the agent requests the state information at a random time from the interactive environment service program at the beginning of each round; Step 2-4: The interactive environment service program reads the QS file at the current moment and the QS file at the next moment, parses the data therein and returns the observation state data required by the agent; Step 2-5: After receiving the observed state data, the agent uses it as input and uses the actor network to obtain the unit adjustment output value in the interval [-1,1]; Step 2-6: Post-process the output value obtained in step 2-5. During the training process, the actual load value at the next moment is directly used as the load prediction value at the next moment to achieve the balance adjustment between the unit output and the predicted load value, and obtain the actual output value of the unit; Step 2-7: The agent returns the actual output value of the unit to the interactive environment service program. After receiving the actual output value of the unit, the interactive environment service program directly replaces it with the output value of the corresponding unit in the QS file at the next moment, and performs power flow calculation. Then, the QS file generated after the power flow calculation and the data extracted from the QS file at the next moment are returned to the agent as the observation state at the next moment, and the over-limit information after the power flow calculation is returned at the same time; Step 2-8: After receiving the data, the agent first calculates the reward value of this action based on the limit-crossing information, and stores it together with the current moment observation state of this action, the current action, and the next moment observation state as a piece of data in the training data cache for subsequent learning; Step 2-9: At this time, the number of training steps for the next round of the agent increases by 1, and the total number of training steps increases by 1, and the operation returns to step 2-5; when the number of round steps reaches 12, this round ends, the total number of rounds increases by 1, and a new round of training begins. When the total number of training steps reaches the parameters set in step 2-2, the agent training is completed, the training parameters are saved, and the program is exited to end the training; Step 2-10: After 1000 steps of training are completed, it is the application stage. In addition to following the process of steps 2-3 to 2-9 at the beginning of each round, the initial moment is selected as much as possible when the historical data exceeds the limit. At this time, after each training step interaction, it is also necessary to randomly read n data from the training data cache, and use the gradient descent method to update the critic network parameters and the actor network parameters to optimize the parameters of the intelligent agent and generate the best strategy.

9. The method for generating a power grid operation risk control strategy according to claim 8, characterized in that: The method for generating a power grid operation risk control strategy also includes: placing the trained intelligent agent in an online application, and the online decision-making process is as follows: The agent reads the parameters saved after training and uses them to initialize itself, and is ready to make decisions; If the operator finds that the power grid is out of limit at this time, after taking corresponding actions, the production environment will generate a new QS file. At this time, the interactive environment service program deployed in the application environment will extract the out-of-limit QS file at the previous moment and the latest QS file data and return them to the intelligent agent as the observation state; After receiving the observed state data, the agent uses it as input and uses the actor network to obtain the unit adjustment output value in the interval [-1,1]. After post-processing, the actual adjustment value of the unit is obtained and returned to the interactive environment service program. This is the adjustment strategy generated for the over-limit situation. After the interactive environment service program receives the actual output value of the unit, it directly replaces it with the output value of the corresponding unit in the QS file at the next moment, and performs power flow calculation to obtain the QS file of the power grid status after the intelligent agent decision is adopted. By comparing it with the QS file formed after manual operation in the production environment, it can be seen from the effect that the power grid after adjustment by the intelligent agent is in a safe and limit-free state.

10. A power grid operation risk control strategy generation system, characterized in that: include: A construction module, which is used to construct a power grid power flow over-limit adaptive adjustment model based on deep reinforcement learning; The building module is used to build an integrated framework for offline training and online applications.

Images