A method and device for automatic operation of power system based on physical information and deep reinforcement learning

Abstract

This invention discloses a method and apparatus for automated operation of power systems based on physical information and deep reinforcement learning. The method sets the neural network structure, parameters, state space, and selectable action space required for reinforcement learning training, constructing a deep Q-learning agent that predicts automatic actions by minimizing the frequency and severity of line faults. Then, it directly applies the physical information of the power system as known constraints for training, reducing training data and accelerating training time. Next, it designs a reward function for optimization, training the selection of different actions to achieve the maximum reward, thus finding the optimal automated operation. This invention combines physical information features with the DQN algorithm to optimize the control of the topology of various transmission lines and substations in the power system, balancing economy and effectiveness. This invention enables real-time power system stability control with relatively little data.

Description

Technical Field

[0001] The present invention belongs to the technical field of power system operation optimization, and specifically relates to a power system automation operation method and device based on physical information and deep reinforcement learning. Background Art

[0002] As economic and social development increase the demand for electricity, the large-scale access of wind power and intermittent renewable energy has put the power grid under pressure to supply electricity. Power outages are highly destructive to the power system, disastrous for the safe and stable operation of the power grid, and the cost of corrective maintenance is extremely high. As the complexity of the power grid increases, manual operators are unable to effectively monitor and manage real-time problems. Strong demand growth, limited power generation capacity, and complex real-time operations provide favorable conditions for the use of intelligent algorithms to participate in power system decision-making.

[0003] Reinforcement learning algorithms are performed by an agent in an environment that maximizes certain rewards based on state parameters, then takes actions, updates state information, and so on. It is not explicitly told what the agent should do, but performs semi-random actions to learn what will produce the most ideal results. New states and rewards are given to the agent to indicate whether the behavior is beneficial or harmful. Based on the reward, the agent will modify its action strategy.

[0004] There have been some studies that apply reinforcement learning to power systems because large amounts of state data can be processed quickly. Complex decision factors also need to be considered before any action is taken on the power grid. Reinforcement learning is best suited for this type of scenario because the best action is determined through experiments and training. For example, the learning agent will iterate each possible action in the action space and determine the best action plan. The structure of the agent is also such that if the power grid is not facing any serious danger, no action will be taken. Based on the good characteristics of reinforcement learning used in power systems, how to develop it to tap its potential to adapt to complex and changing power grid conditions is an urgent problem to be solved. Summary of the invention

[0005] In order to address the deficiencies in the prior art, the present invention provides an automated operation method and device for an electric power system based on physical information and deep reinforcement learning, which saves computing power with less sacrifice in decision-making accuracy and calculation speed, and effectively replaces manual operators to perform electric power system operations, including substation connection topology and opening and closing of transmission lines.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] The present invention provides a method for automatic operation of a power system based on physical information and deep reinforcement learning, comprising:

[0008] With the power system grid state data as input, the power system operation action as output, and the power system physical information as condition, a power system automation operation physical information neural network is constructed;

[0009] The physical information neural network of power system automation operation is constructed by deep reinforcement learning training. During the training process, the optimal reward function is used to select the optimal operation action decision;

[0010] Based on the real-time power system grid status data and the trained power system automation operation physical information neural network, the real-time power system automation operation action decision is obtained.

[0011] Furthermore, the power system grid state data includes: generator state, load state, line start node state, line end node state and line characteristic state data of each node, as well as observation time series data.

[0012] Furthermore, based on the pypownet platform, the real-time steady-state operation of the power system is simulated and the power flow calculation is performed to obtain the power system grid status data, and the grid status data is filtered through the Observation class of pypownet.

[0013] Furthermore, the power system operation action includes:

[0014] Open or close transmission lines and reconfigure substation connection topology.

[0015] Furthermore, the power system operation action includes:

[0016] Opening transmission lines and reconfiguring substation connection topology.

[0017] Furthermore, the power system physical information includes:

[0018] The power system transmission lines have a thermal threshold, exceeding which will increase the penalty value of training;

[0019] The power system topology network updates the power system grid status information online in real time as the power system operates, and then makes decisions on the next round of operation actions.

[0020] Furthermore, the reward function is as follows:

[0021] reward = αN load +βN gen +γL use +δC action

[0022] Among them, reward is the reward value, α, β, γ, δ are sub-reward coefficients, N loadis the number of loads to be turned off, N gen is the number of generator sets shut down, L use is the line capacity usage, C action is the action cost; N load 、N gen and L use By obtaining the grid status data, the action cost C action It is the sum of the cost of node splitting and the cost of line state switching.

[0023] Furthermore, during the training process, if the reward reaches a preset reward threshold, or an illegal operation is performed, or the number of line cuts exceeds a preset value, the training is terminated; the illegal operation refers to connecting a preset unavailable power transmission line.

[0024] Furthermore, a memory replay session is added to the training process.

[0025] Furthermore, during the training process, a prediction step is added to determine whether an operation is required, specifically:

[0026] When there is no decrease in the past N reward values, no operation action is performed.

[0027] The present invention also provides an electric power system automatic operation device based on physical information and deep reinforcement learning, comprising:

[0028] A construction module is used to construct a power system automation operation physical information neural network with power system grid state data as input, power system operation actions as output, and power system physical information as a condition;

[0029] A training module is used to train the physical information neural network of power system automation operation constructed by deep reinforcement learning. During the training process, the optimal operation action decision is selected by using the optimized reward function;

[0030] as well as,

[0031] The execution module is used to obtain real-time power system automation operation action decisions based on real-time power system grid status data and the trained power system automation operation physical information neural network.

[0032] Furthermore, the training module designs a reward function as follows:

[0033] reward = αN load +βN gen +γL use +δC action

[0034] Among them, reward is the reward value, α, β, γ, δ are sub-reward coefficients, N loadis the number of loads to be turned off, N gen is the number of generator sets shut down, L use is the line capacity usage, C action is the action cost; N load 、N gen and L use By obtaining the grid status data, the action cost C action It is the sum of the cost of node splitting and the cost of line state switching.

[0035] The beneficial effects achieved by the present invention are:

[0036] 1) The present invention combines physical information characteristics and optimizes the topology of each power transmission line and substation in the power system based on the DQN algorithm, taking into account both economy and effectiveness. The present invention can perform stable control of the power system in real time with less data.

[0037] 2) Due to the addition of critical condition prediction and memory playback process, the cumulative reward increases dramatically, and better results can be achieved on the basis of the proposed DQN control algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the power system deep reinforcement learning network optimization method based on physical information of the present invention;

[0039] Figure 2 A schematic diagram of the relationship between whether to reduce the action space and the end event in an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the relationship between whether to use PINN and an end event in an embodiment of the present invention;

[0041] Figure 4 It is a schematic diagram of the gradient change of whether PINN is used in the embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0043] like Figure 1 As shown, the present invention provides a method for automatic operation of a power system based on physical information and deep reinforcement learning, and the specific steps include:

[0044] Step 1) Set the neural network structure, parameters, state space, and optional action space required for reinforcement learning training.

[0045] In the embodiment of the present invention, considering that the industrial power system directly collects power grid data to train and control the power grid, the real-time steady-state operation and power flow calculation of the power system are simulated based on the pypownet platform, and the deep Q learning (DQN) algorithm is developed and fine-tuned. Physical information is added as a restriction, and its performance is tested using deep Q learning. The power grid state data can be divided into the generator state, load state, line starting node state, line end node state, and line characteristic state of each node, such as line current, power generation, power consumption, line thermal limit, etc. The specific state refers to the 36 state data provided by the pypownet.environment.Observation class, as well as the observation time series data. For the IEEE14-node power system used, there are 538 complete power grid state data, which can be reduced to a size of 392 or 224 by filtering out non-essential parameters. This is passed to the input layer of the neural network, and then an array with the same size as the action space is output. These are implemented through the Observation class of pypownet. Most importantly, the simulator tracks when catastrophic failures occur, called end events, which occur when a line exceeds its current limits for a long period of time, or when too many outages occur.

[0046] There are four main components of the power system: power plants, power users, transmission lines, and substations. Substations are intermediate nodes where multiple lines are connected. The environment runs in discrete time steps and allows the agent to take two types of actions: turning on or off a specific transmission line or reconfiguring the substation connection topology. Each node in the output layer is associated with a unique action, which represents the predicted reward that will be obtained by taking that specific action at the current time. For the IEEE14-node power system, there are 5 generators, 11 loads, and 20 lines, so there are 20 line starting connection states, 20 end connection states, and 20 line on-off states. The total action space is N A = 76 elements, but this can be reduced to 56 by restricting the line to only allow one end to be disconnected, and a single action is represented by an integer A∈[0,N A ) identifier, which maps to any configurable substation or switchable transmission line.

[0047] Step 2) Use the physical information of the power system as known constraints for training to reduce training data and speed up training time.

[0048] In the embodiment of the present invention, traditional machine learning and supervised learning cannot participate in the decision-making of the intelligent agent in real time, and a physical information neural network (PINN) is introduced to participate in a reinforcement learning, which uses potential physical rules to learn faster, use less data and execute more accurately.

[0049] The following physical information conditions are used as presets: (1) The transmission line has a thermal threshold, and exceeding the thermal threshold will increase the training penalty value; (2) The power system topology network continuously updates the status information online in real time as the operation takes place before the next round of actions.

[0050] Step 3) Design a reward function for optimization, train the system to select different actions to achieve the maximum reward, and find the optimal automated operation.

[0051] A reward function approach is adopted, where the agent learns from its experience and provides necessary feedback. The reward is set as the sum of four factors with scaling factors: the number of loads turned off N load 、Number of generator sets to be shut down gen 、Line capacity usage L use 、Action cost C action .

[0052] reward = αN load +βN gen +γL use +δC action

[0053] Where α, β, γ, and δ are all coefficients of sub-rewards. The shutdown state of each node and the line capacity usage at this time can be known through the state space. The action cost is defined as the sum of the cost of node splitting and the cost of line state switching, that is, the sum of the numbers 1 in the action vector, because they represent the activation of the switch.

[0054] Depending on the state information, there can be multiple sub-rewards, which are either positive or negative. Rewards such as load shedding or output reduction are negative because they are a parameter that the agent must minimize. On the other hand, line cutting is a positive reward, meaning that the system remains stable after the line cutting, which is a valid behavior, because reducing the line cutting is considered a better action for the agent.

[0055] In addition, three types of end-of-training signs are set, namely, power flow no longer converges, violations (such as opening a faulty line), and excessive generator or load cutoff times.

[0056] Once the reward reaches a certain threshold, or if an illegal action is attempted (such as reconnecting a broken line that is not available in the preset), or if too many line cuts occur, the corresponding end flag is triggered to indicate the end event to the agent. Once it ends, the agent is given a large negative reward and training is restarted at the next time step. The agent is also heavily penalized at the end of each game, which allows us to force the agent to avoid making wrong actions in the future.

[0057] Step 4) Perform debugging and fine-tune the comparison by reducing the action space, adding memory playback, adding the process of checking action prediction, etc. to achieve better results.

[0058] Testing has shown that a common cause of simulated power grid failures is too many disconnected loads. To avoid this and potentially reduce training time, the action space is reduced by removing the ability to disconnect lines. A Memory Replay module is added to DQN to make the neural network more stable.

[0059] Before the action begins, add a prediction module to check whether the action is needed and whether the reward changes continuously at the beginning. The check is necessary at the beginning because when the action is not needed, the agent will continue to perform actions. The specific prediction is that when the past N rewards have not decreased, the agent will choose to do nothing.

[0060] Example

[0061] The test case is constructed with IEEE14-node model data, and the power system status information data including 36 lists of each information type is used as the reinforcement learning environment for operation simulation, such as line current, power generation, power consumption, sudden weather disasters, etc. The reinforcement learning optimization goal is to consider factors such as voltage crossing, current thermal threshold and transmission line status, and try to avoid the power system operation entering the end state.

[0062] The action space includes opening or closing specific transmission lines and reconfiguring substation connection topology. The complete action space has N A = 76 elements, but can be reduced to 56. A single action is represented by an integer A∈[0,N A ) identifier, which maps to any configurable substation or switchable transmission line.

[0063] The neural network structure used to train the DQN agent is shown in Table 1, and the parameters of the test set and validation set used for simulation are shown in Table 2:

[0064] Table 1 Neural network structure used in DQN agent training Model: "sequential_14"

[0065]

[0066] Number of parameters: 38,357

[0067] Number of training parameters: 38,357

[0068] Number of untrained parameters: 0

[0069] Table 2 Parameters of test set and validation set

[0070]

[0071] The agent was trained 100,000 times on a local machine. Testing showed that a common cause of power grid failures is too many disconnected loads. To avoid this, and potentially reduce training time, the action space was reduced by removing the ability to disconnect lines. This reduced the action space to 56 actions instead of the original 76, which only reduced the training time per iteration by 0.3%, but significantly improved the overall performance of the agent. For a plot of the end event versus time step, see Figure 2 .Depend on Figure 2 It can be seen that the reduced action space ends training earlier than the full action space, helping the agent to train faster and reduce the number of training ends.

[0072] One of the biggest improvements in training neural networks came from memory replay. Memory replay is when the agent’s experience at each time step is stored in a dataset and reused multiple times to train the agent. This helps the agent learn faster because the results of a specific action are known. The incorporation of memory replay increased the final cumulative reward by about 68.6%.

[0073] After implementing the above functions, train and test the agent. Exploration and exploitation are managed by a decay factor. For long training of 100,000 iterations, exploration is simply set to 99.99%, then tested with 500 exploitation iterations, and the final cumulative reward is tracked. Most sub-rewards are scaled to similar orders of magnitude, except for the game end penalty. Since game end events are relatively rare for well-behaved agents, it is necessary to scale the penalties so that they are significant when trained to a neural network. This means scaling the game end reward to a large negative value. By default, the scaling factor is 1.

[0074] To finally test the efficacy of PINN, its performance was tested against a basic non-PINN version. The success or failure of an action is defined by whether the entire process ends as a result of it. Therefore, for the non-PINN agent, all sub-rewards except the game-ending penalty were removed, allowing a fair comparison of the performance of PINN and non-PINN by tracking game-ending events over time. Figure 3 In Figure 2, the total number of game-endings during 3000 iterations is shown, where each agent has no pre-training and exploration is linearly reduced from 100% to 0% over the first 2000 iterations. It can be seen that the PINN agent learns to avoid game-ending events earlier than the non-PINN agent, resulting in fewer total occurrences.

[0075] Figure 4It shows how the frequency of closure varies with and without the physics-based solution, with non-PINNs always having a higher closure rate.

[0076] As the number of iterations increases, the number of intermediate iterations before the PINN solution reaches a plateau. According to these indicators, the PINN agent is less likely to end than the non-PINN agent, which shows that the proposed method successfully demonstrates the superiority of the reinforcement learning strategy based on physical information.

[0077] Before Memory Replay, the cumulative reward value after 100,000 training times was -460,458.2, and after adding Memory Replay, it increased to -144,610.1. Adding the pre-judgment process also increased the total by 10%.

[0078] The present invention combines the characteristics of physical information and optimizes the topology of each transmission line and substation of the power system based on the DQN algorithm, taking into account both economy and effectiveness. The present invention can perform stable control of the power system in real time with less data. Due to the addition of checking critical condition prediction and implementing memory playback process, the cumulative reward increases sharply, and better effects can be achieved on the basis of the proposed DQN control algorithm.

[0079] Another embodiment of the present invention provides a power system automation operation device based on physical information and deep reinforcement learning, including:

[0080] A construction module is used to construct a power system automation operation physical information neural network with power system grid state data as input, power system operation actions as output, and power system physical information as a condition;

[0081] A training module is used to train the physical information neural network of power system automation operation constructed by deep reinforcement learning. During the training process, the optimal operation action decision is selected by using the optimized reward function;

[0082] as well as,

[0083] The execution module is used to obtain real-time power system automation operation action decisions based on real-time power system grid status data and the trained power system automation operation physical information neural network.

[0084] In the embodiment of the present invention, the training module designs the reward function as follows:

[0085] reward = αN load +βN gen +γL use +δC action

[0086] Among them, reward is the reward value, α, β, γ, δ are sub-reward coefficients, N load is the number of loads to be turned off, N gen is the number of generator sets shut down, L use is the line capacity usage, C action is the action cost; N load 、N gen and L use By obtaining the grid status data, the action cost C action It is the sum of the cost of node splitting and the cost of line state switching.

[0087] It is worth pointing out that the device embodiment corresponds to the above-mentioned method embodiment, and the implementation methods of the above-mentioned method embodiments are all applicable to the device embodiment and can achieve the same or similar technical effects, so they will not be repeated here.

[0088] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0089] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for automatic operation of a power system based on physical information and deep reinforcement learning, characterized in that: include: With the power system grid state data as input, the power system operation action as output, and the power system physical information as condition, a power system automation operation physical information neural network is constructed; The power system operation actions include: opening or closing the transmission line, and reconfiguring the substation connection topology; the power system physical information includes: the power system transmission line has a thermal threshold, exceeding the thermal threshold will increase the penalty value of training; and the power system topology network updates the power system grid status information online in real time with the power system operation actions, and then makes the next round of operation action decisions; The physical information neural network of power system automation operation constructed by deep reinforcement learning training is used. During the training process, the optimal operation action decision is selected by using the optimized reward function; the reward function is as follows: reward=αN load +βN gen +γL use +δC action ; Among them, reward is the reward value, α, β, γ, δ are sub-reward coefficients, N load is the number of loads to be turned off, N gen is the number of generator sets shut down, L use is the line capacity usage, C action is the action cost; N load 、N gen and L use By obtaining the grid status data, the action cost C action It is the sum of the cost of node splitting and the cost of line state switching; During the training process, if the reward reaches the preset reward threshold, or an illegal operation is performed, or the number of line cuts exceeds the preset value, the training is terminated; the illegal operation refers to connecting the preset unavailable power transmission line; and during the training process, a memory replay link and a pre-judgment link of whether an operation action is required are added; the pre-judgment link of whether an operation action is required is specifically: when the past N reward values ​​have not decreased, the operation action is no longer performed; Based on the real-time power system grid status data and the trained power system automation operation physical information neural network, the real-time power system automation operation action decision is obtained.

2. The method for automatic operation of a power system based on physical information and deep reinforcement learning according to claim 1, characterized in that: The power system grid state data includes: generator state, load state, line start node state, line end node state and line characteristic state data of each node, as well as observation time series data.

3. The method for automatic operation of a power system based on physical information and deep reinforcement learning according to claim 2, characterized in that: Based on the pypownet platform, the real-time steady-state operation of the power system is simulated and the power flow calculation is performed to obtain the power system grid status data, and the grid status data is filtered through the pypownet's Observation class.

4. An automatic operation device for a power system based on physical information and deep reinforcement learning, characterized in that: The device is used to implement the power system automation operation method based on physical information and deep reinforcement learning as described in any one of claims 1 to 3, comprising: A construction module is used to construct a power system automation operation physical information neural network with power system grid state data as input, power system operation actions as output, and power system physical information as a condition; the power system operation actions include: opening or closing transmission lines, and reconfiguring substation connection topology; the power system physical information includes: power system transmission lines have thermal thresholds, exceeding the thermal threshold will increase the penalty value of training; and the power system topology network updates the power system grid state information online in real time with the power system operation actions, and then makes the next round of operation action decisions; The training module is used to train the physical information neural network of power system automation operation constructed by deep reinforcement learning. During the training process, the optimal operation action decision is selected by using the optimized reward function, and the reward function is as follows: reward=αN load +βN gen +γL use +δC action ; Among them, reward is the reward value, α, β, γ, δ are sub-reward coefficients, N load is the number of loads to be turned off, N gen is the number of generator sets shut down, L use is the line capacity usage, C action is the action cost; N load 、N gen and L use By obtaining the grid status data, the action cost C action It is the sum of the cost of node splitting and the cost of line state switching; During the training process of the training module, if the reward reaches a preset reward threshold, or an illegal operation is performed, or the number of line cuts exceeds a preset value, the training is terminated; the illegal operation refers to connecting a preset unavailable power transmission line; and during the training process, a memory replay link and a pre-judgment link of whether an operation action is required are added; the pre-judgment link of whether an operation action is required is specifically: when the past N reward values ​​have not decreased, the operation action is no longer performed; as well as, The execution module is used to obtain real-time power system automation operation action decisions based on real-time power system grid status data and the trained power system automation operation physical information neural network.

Images