Power Grid Flow Adjustment Method Based on Attention Mechanism and Deep Reinforcement Learning
Through the method based on attention mechanism and deep reinforcement learning, the problem that grid trend adjustment depends on expert experience is solved, and automated and efficient grid trend adjustment is achieved, which is suitable for real-time and random power flows in complex large-scale power grids. Distribution changes.
Patent Information
- Application Number
- CN202210601241.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-30
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-05-30
Smart Images

Figure CN114970351B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of smart grid power flow adjustment, in particular to the technical field of smart grid power flow adjustment based on artificial intelligence technology. Background Art
[0002] In order to meet the growing power load demand of society and industry, the construction scale of my country's power system is constantly expanding. With the advancement of my country's industrial informatization process, the requirements for automation and intelligence of the power system are gradually increasing. However, traditional power analysis and control technologies are inefficient and costly in contemporary large-scale power grids, and they urgently need to be transformed into smart grids. Artificial intelligence is a key supporting technology for the construction of new power systems and a key path to help power grids develop towards high quality.
[0003] Grid flow adjustment is a key task in the daily operation and control of the power grid. It controls the distribution of power flow in the power grid by dispatching the power of the generators in the whole grid, so that the power transmission can reach the expected state under the changing power load. The typical manifestation of the power flow state of the power grid is the power of the grid section. The section is a collection of certain lines in the power system, which usually represents the power exchange between different regions. The degree of automation of this task reflects the intelligence level of the power grid and also determines the safety and stability of the power grid operation.
[0004] Although some power flow simulation and analysis technologies can be used to help grid dispatchers adjust power flow, in reality, the efficient completion of this task usually relies on the long-term experience of dispatch experts. In the current complex large-scale power grid with new energy structures, it is difficult to meet the real-time dispatching needs and the random distribution changes of power flow. Artificial intelligence technology represented by deep reinforcement learning can automatically learn and solve sequential decision problems, and is a new technology for realizing power flow adjustment. Summary of the invention
[0005] In order to overcome the shortcomings of existing power grid flow methods such as reliance on expert experience, low efficiency, and limited generalization ability, the present invention proposes a power grid flow adjustment method based on attention mechanism and deep reinforcement learning that does not require human intervention, is intelligent, and has strong generalization ability.
[0006] The method of the present invention interacts with the power system simulation environment to train the intelligent agent to learn the optimal generator dispatching strategy under different power flow environments and adjustment targets. At the same time, the attention mechanism is introduced to model the implicit relationship between the target to be adjusted and other electrical characteristics, thereby improving the performance of the intelligent agent.
[0007] The power grid flow adjustment method based on attention mechanism and deep reinforcement learning includes the following steps:
[0008] 1) Construct a reinforcement learning interactive environment based on the Markov decision process and the power flow simulation program, and build a neural network model as an intelligent agent based on the attention mechanism;
[0009] 2) Randomly initialize the power flow state of the power grid, randomly select the power grid line or section that needs to be adjusted, and randomly set the target power within a reasonable range;
[0010] 3) The data collected by the agent and the environment are stored in the experience pool, and the dual competitive deep Q network (D3QN) algorithm is used to train and update the parameters of the agent;
[0011] 4) Use the trained agent to output the scheduling sequence for power flow adjustment under given power flow and objectives.
[0012] The reinforcement learning interactive environment construction process in step 1) is as follows: Based on the power flow adjustment problem, the basic elements of reinforcement learning are defined, including observation state, reward function, action value, etc. The observation state includes the electrical characteristics (active power, reactive power values) and position characteristics (discrete number) of the entire network branch and bus node, as well as the target electrical characteristics and position characteristics of the branch or section to be adjusted; the reward function is defined as the distance function between the current power flow adjustment target value and the current value; the action value is defined as the discrete adjustment amount of the generator. For G adjustable generators, when each generator has C adjustment measures, the size of the action space is C×G.
[0013] Furthermore, the process of constructing a neural network model as an intelligent agent based on the attention mechanism in step 1) is as follows:
[0014] (1.1) Set W learnable parameters for each observation state, and after multiplying with the original value, each observation state can obtain an embedding vector of length W. Concatenate the embedding vectors corresponding to the active power, reactive power, and position features of the same electrical object (branch or bus node) to obtain an embedding vector of length 3W. Superimpose different embedding vectors to obtain an observation feature input matrix of dimension (M+N, 3W). The number of rows M represents the number of different electrical objects, and N represents the number of preset objects to be adjusted. When the number of objects to be adjusted is less than N, all the insufficient rows are set to 0;
[0015] (1.2) Construct the attention mechanism module, and input the query vector matrix Q, key vector matrix K, and value vector matrix V through three linear transformation matrices of dimension (3W, W). Then, after the attention operation, the weighted representation of the value vector matrix is output. The operation formula is as follows:
[0016]
[0017] The softmax(t) function formula is t represents a vector.
[0018] (1.3) Set T groups of attention mechanism modules in parallel, input the observed feature matrix into each group of attention mechanism modules respectively, learn the embedded vector expression in different implicit spaces, and obtain the output matrix with dimension size (M+N,W) respectively
[0019] (1.4) The T output matrices are concatenated to obtain a matrix of dimension (M+N, T·W), which is then passed through an information extraction matrix of dimension (T·W,O), flattened into a one-dimensional vector, and then passed through several layers of fully connected network layers and ReLU activation functions to obtain the action value estimation vector.
[0020] The agent training process described in step 3) is specifically as follows:
[0021] (3.1) The agent obtains observation variables from the interactive environment, inputs them into the model to obtain the estimated action value, selects the action with the largest estimated value, executes the corresponding generator dispatch strategy, calculates the power flow through the power simulation program in the interactive environment, performs state transfer to obtain new observation variables, and brings the result of power flow adjustment into the reward function to obtain the reward value;
[0022] (3.2) The observed variables, actions, reward values, updated observed variables and other information in each interaction process are stored in the experience pool in the form of tuples;
[0023] (3.3) If the number of tuples in the experience pool exceeds 1024, 1024 pieces of data are randomly extracted from it, and the gradients of the estimated network and the target network in D3QN are calculated based on the batch stochastic gradient descent method and the parameters are updated. The target network uses the soft update method to delay the update of parameters, and the formula is:
[0024] θ′ i+1 =(1-ε)θ′ i +εθ i (2)
[0025] where θ i Represents the parameters of the estimated network at the current moment, θ′ i Represents the parameters of the target network at the current moment, θ′ i+1 Represents the parameters of the target network at the next moment (i.e. after the update), ε is the update coefficient, requiring 0<ε<<1, so that the update of the target network is relatively smooth
[0026] (3.4) Repeat steps (3.1) to (3.4) until the maximum preset number of times.
[0027] The state transfer process in step (3.1) is as follows: modify the active output parameters of the corresponding generator in the adjustment strategy, perform power flow calculation simulation, output a new power flow distribution result file, and the interactive environment reads relevant parameters from the result file to update the branch active power, branch reactive power, bus active power, and bus reactive power in the observation state.
[0028] In step 4), the specific process of using the trained intelligent agent to output the scheduling sequence under the given power flow and target for power flow adjustment is as follows: only use the estimated network in D3QN to load the trained model weights, input the current power flow state into the model in the form of observed variables, obtain the value evaluation of each generator adjustment action, select the strategy with the largest estimated value to execute, re-run the power flow calculation, and verify whether the power flow adjustment reaches the target. If not, input the updated power flow state again and repeat the above behavior until the power flow adjustment target is reached, the adjustment is successful, or the maximum number of attempts is reached, and the closest adjustment plan is obtained.
[0029] The present invention constructs a deep reinforcement learning model based on the attention mechanism, which can automatically learn and solve the power grid flow adjustment task. First, a reinforcement learning interactive environment is constructed based on the Markov decision process and the power flow simulation program, and a neural network model is constructed as an intelligent agent based on the attention mechanism; then the power grid flow state is randomly initialized, the power grid line or section that needs to be adjusted is randomly selected, and the target power within a reasonable range is randomly set; then the intelligent agent interacts with the environment to collect data and store it in the experience pool, and the dual-competitive deep Q network algorithm is used to train and update the intelligent agent parameters.
[0030] The advantages of the present invention are that the generator scheduling strategy is learned by the intelligent agent's autonomous interaction with the environment, thereby realizing the automation of section power control, which has stronger adaptability than the manual adjustment method based on expert knowledge; the attention mechanism is introduced to model and capture the relationship between the adjustment target and the global electrical characteristics, thereby improving the accuracy of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is the overall flow chart of the present invention.
[0032] Figure 2 It is a framework diagram of the model training of the present invention. DETAILED DESCRIPTION
[0033] The present invention will be further described with reference to the accompanying drawings:
[0034] The overall flow chart of the power grid flow adjustment method based on attention mechanism and deep reinforcement learning is shown in Figure 1 , the method comprises the following steps:
[0035] 1) Construct a reinforcement learning interactive environment based on the Markov decision process and the power flow simulation program, and build a neural network model as an intelligent agent based on the attention mechanism;
[0036] 2) Randomly initialize the power flow state of the power grid, randomly select the power grid line or section that needs to be adjusted, and randomly set the target power within a reasonable range;
[0037] 3) The data collected by the agent and the environment are stored in the experience pool, and the dual competitive deep Q network (D3QN) algorithm is used to train and update the parameters of the agent;
[0038] 4) Use the trained agent to output the scheduling sequence for power flow adjustment under given power flow and objectives.
[0039] The reinforcement learning interactive environment construction process in step 1) is as follows: Based on the power flow adjustment problem, the basic elements of reinforcement learning are defined, including observation state, reward function, action value, etc. The observation state includes the electrical characteristics (active power, reactive power values) and position characteristics (discrete number) of the entire network branch and bus node, as well as the target electrical characteristics and position characteristics of the branch or section to be adjusted; the reward function is defined as the distance function between the current power flow adjustment target value and the current value; the action value is defined as the discrete adjustment amount of the generator, for N G adjustable generators, when each generator has C adjustment measures, the size of the action space is C·N G .
[0040] Furthermore, the reward function R gives negative rewards according to the difference between the target power and the target line / section:
[0041] R=-|P c -P goal | (1)
[0042] Among them, P c Represents the current power value of the target line / section, P goal Represents its target power value.
[0043] In the power flow adjustment task, the section power is adjusted by modifying the effective flag or active output of the generator to form a discrete action space. Specifically, for N in the power grid G There are four adjustable generators. There are four possible adjustment behaviors for a generator, which represent turning off the generator, adjusting to 1 / 4 output, adjusting to 1 / 2 output, and adjusting back to the original output. The size of the action space is: 4·N G , action A is represented as a specific action selected from the action space:
[0044] A=a i ,i∈[0,4·N G ] (2)
[0045] Furthermore, the process of building a neural network model as an agent based on the attention mechanism in step 1) is as follows:
[0046] (1.1) Set W learnable parameters for each observation state, and after multiplying with the original value, each observation state can obtain an embedding vector of length W. W can be flexibly set according to the scale of power grid parameters and computing resources, such as W = 8. Concatenate the embedding vectors corresponding to the active power, reactive power, and position features of the same electrical object (branch or bus node) to obtain an embedding vector of length 3W. Superimpose different embedding vectors to obtain an observation feature input matrix with a dimension size of (M+N, 3W). The number of rows M represents the number of different electrical objects, and N represents the number of preset objects to be adjusted. When the number of objects to be adjusted is less than N, all the insufficient rows are set to 0;
[0047] (1.2) Construct the attention mechanism module. The input is respectively transformed through three linear transformation matrices of size (3W, W) to obtain the query vector matrix Q, key vector matrix K, and value vector matrix V. Then, after the attention operation, the weighted representation of the value vector matrix is output. The operation formula is as follows:
[0048]
[0049] The softmax(t) function formula is t represents a vector.
[0050] (1.3) Set T groups of attention mechanism modules in parallel, input the observed feature matrix into each group of attention mechanism modules respectively, learn the embedded vector expression in different implicit spaces, and obtain the output matrix with dimension size (M+N,W) respectively
[0051] (1.4) The T output matrices are concatenated to obtain a matrix of dimension (M+N, T·W), which is then passed through an information extraction matrix of dimension (T·W,O), flattened into a one-dimensional vector, and then passed through several layers of fully connected network layers and ReLU activation functions to obtain the action value estimation vector.
[0052] The agent training process of step 3) can be referred to Figure 2 , specifically:
[0053] (3.1) The agent obtains observation variables from the interactive environment, inputs them into the model to obtain the estimated action value, selects the action with the largest estimated value, executes the corresponding generator dispatch strategy, and uses the power simulation program in the interactive environment to calculate the power flow to obtain the updated observation variables. The reward value is obtained by bringing the result of the power flow adjustment into the reward function;
[0054] (3.2) The observed variables, actions, reward values, updated observed variables and other information in each interaction process are stored in the experience pool in the form of tuples;
[0055] (3.3) If the number of tuples in the experience pool exceeds 1024, 1024 pieces of data are randomly extracted from it, and the gradients of the estimated network and the target network in D3QN are calculated based on the batch stochastic gradient descent method and the parameters are updated. The target network uses the soft update method to delay the update of parameters, and the formula is:
[0056] θ′ i+1 =(1-ε)θ′ i +εθ i (4)
[0057] where θ i Represents the parameters of the estimated network at the current moment, θ′ i Represents the parameters of the target network at the current moment, θ′ i+1 Represents the parameters of the target network at the next moment (i.e. after the update), ε is the update coefficient, requiring 0<ε<<1, so that the update of the target network is relatively smooth
[0058] (3.4) Repeat steps (3.1) to (3.4) until the maximum preset number of times.
[0059] The state transfer process in step (3.1) is as follows: modify the active output parameters of the corresponding generator in the adjustment strategy, perform power flow calculation simulation, output a new power flow distribution result file, and the interactive environment reads relevant parameters from the result file to update the branch active power, branch reactive power, bus active power, and bus reactive power in the observation state.
[0060] In step 4), the specific process of using the trained intelligent agent to output the scheduling sequence under the given power flow and target for power flow adjustment is as follows: only use the estimated network in D3QN to load the trained model weights, input the current power flow state into the model in the form of observed variables, obtain the value evaluation of each generator adjustment action, select the strategy with the largest estimated value to execute, re-run the power flow calculation, and verify whether the power flow adjustment reaches the target. If not, input the updated power flow state again and repeat the above behavior until the power flow adjustment target is reached, the adjustment is successful, or the maximum number of attempts is reached, and the closest adjustment plan is obtained.
[0061] The present invention proposes a power grid flow adjustment method based on attention mechanism and deep reinforcement learning, which has the advantages of: using deep reinforcement method to solve complex power grid flow adjustment problems, without the need to model complex power systems, learning generator scheduling strategies through autonomous interaction between intelligent agents and the environment, and realizing the automation of section power control. Compared with the manual adjustment method based on expert knowledge, it has stronger adaptability; introducing attention mechanism, modeling and capturing the relationship between adjustment targets and global electrical characteristics, and improving the accuracy of the algorithm.
[0062] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.
Claims
1. A power grid flow adjustment method based on attention mechanism and deep reinforcement learning, characterized by The basic steps are as follows: 1) A reinforcement learning interactive environment is constructed based on a Markov decision process and a power flow simulation program, and a neural network model is constructed as an intelligent agent based on an attention mechanism; the reinforcement learning interactive environment construction process is as follows: based on the power flow adjustment problem, basic elements of reinforcement learning are defined, including observation state, reward function, and action value; the observation state includes electrical characteristics and position characteristics of the entire network branch and busbar node, and target electrical characteristics and position characteristics of the branch or section to be adjusted; The electrical characteristics of the bus node include active power and reactive power values; the location characteristics are discrete numbers; the reward function is defined as a distance function between the current power flow adjustment target value and the current value; the action value is defined as the discrete adjustment amount of the generator. For G adjustable generators, when each generator has C adjustment measures, the action space size is C×G; The process of constructing a neural network model as an intelligent agent based on the attention mechanism is as follows: (1.1) Set W learnable parameters for each observation state. After multiplying with the original value, each observation state can obtain an embedding vector of length W; concatenate the embedding vectors corresponding to the active power, reactive power, and position features of the same electrical object, i.e., the branch or bus node, to obtain an embedding vector of length 3W. Superimpose different embedding vectors to obtain an observation feature input matrix of dimension (M+N, 3W). The number of rows M represents the number of different electrical objects, and N represents the number of preset objects to be adjusted. When the number of objects to be adjusted is less than N, all the insufficient rows are set to 0. (1.2) Construct an attention mechanism module. The input is transformed into a query vector matrix Q, a key vector matrix K, and a value vector matrix V through three linear transformation matrices of dimension (3W, W). Then, after the attention operation, the weighted representation of the value vector matrix is output. The operation formula is as follows: The softmax(t) function formula is t represents a vector; (1.3) Set T groups of attention mechanism modules in parallel, input the observed feature matrix into each group of attention mechanism modules respectively, learn the embedding vector expression under different implicit spaces, and obtain output matrices with dimensions of (M+N,W) respectively; (1.4) The T output matrices are concatenated to obtain a matrix of dimension (M+N, T·W), which is then flattened into a one-dimensional vector after passing through an information extraction matrix of dimension (T·W,O) and then passed through several layers of fully connected network layers and ReLU activation functions to obtain the action value estimation vector; 2) Randomly initialize the power flow state of the power grid, randomly select the power grid line or section that needs to be adjusted, and randomly set the target power within a reasonable range; 3) The data collected by the agent and the environment are stored in the experience pool, and the dual competitive deep Q network D3QN algorithm is used to train and update the parameters of the agent; 4) Use the trained agent to output the scheduling sequence for power flow adjustment under given power flow and objectives.
2. The method for adjusting power flow of a power grid according to claim 1, characterized in that The agent training process of step 3) is as follows: (3.1) The agent obtains observation variables from the interactive environment, inputs them into the model to obtain the estimated action value, selects the action with the largest estimated value, executes the corresponding generator dispatch strategy, and uses the power simulation program in the interactive environment to calculate the power flow, performs state transfer to obtain new observation variables, and then uses the results of the power flow adjustment to obtain the reward value. (3.2) The observed variables, actions, reward values, and updated observed variable information in each interaction process are stored in the experience pool in the form of tuples; (3.3) If the number of tuples in the experience pool exceeds 1024, 1024 pieces of data are randomly extracted from it, and the gradients of the estimated network and the target network in D3QN are calculated based on the batch stochastic gradient descent method and the parameters are updated; The target network uses soft update to delay parameter update, and the formula is: θ′ i+1 =(1-ε)θ′ i +eth i (2) where θ i Represents the parameters of the estimated network at the current moment, θ′ i Represents the parameters of the target network at the current moment, θ′ i+1 Represents the parameters of the target network after the next update, ε is the update coefficient, requiring 0<ε<<1, so that the update of the target network is relatively smooth (3.4) Repeat steps (3.1) to (3.4) until the maximum preset number of times; The state transfer process in step (3.1) is as follows: modify the active output parameters of the corresponding generator in the adjustment strategy, perform power flow calculation simulation, output a new power flow distribution result file, and the interactive environment reads relevant parameters from the result file to update the branch active power, branch reactive power, bus active power, and bus reactive power in the observation state.
3. The method for adjusting power flow of a power grid according to claim 1, characterized in that In step 4), the specific process of using the trained intelligent agent to output the scheduling sequence under the given power flow and target for power flow adjustment is as follows: only use the estimated network in D3QN to load the trained model weights, input the current power flow state into the model in the form of observed variables, obtain the value evaluation of each generator adjustment action, select the strategy with the largest estimated value to execute, re-run the power flow calculation, and verify whether the power flow adjustment reaches the target. If not, input the updated power flow state again and repeat the power flow calculation until the power flow adjustment target is reached, the adjustment is successful, or the maximum number of attempts is reached, and the closest adjustment plan is obtained.
