Power grid thermal stability over-limit control method and system based on multi-agent comparative reinforcement learning

Abstract

A method for controlling thermal over-limit stability of power grids based on multi-agent comparative reinforcement learning is as follows: first, a series of thermal over-limit data sets of power grids are constructed based on the normal operation data of typical daily power grids, and the embedded representation vectors of power grid nodes are calculated using a graph attention neural network; then, a multi-agent reinforcement learning framework for generator control is constructed based on a value decomposition deep Q network based on a collaborative architecture. The framework fits the value of each action of each generator node according to the representation characteristics of the power grid graph; then, a comparative learning model is constructed to better distinguish the feature representations of different generator nodes; finally, a generator action execution module is constructed. Each generator selects the action with the largest expected value through the agent network according to the representation vector of its own node to control the power grid, so that the power grid recovers from thermal over-limit stability to normal state. The present invention also includes a power grid thermal over-limit stability control system based on multi-agent comparative reinforcement learning.

Description

Technical Field

[0001] The present invention belongs to the field of power grid control and multi-agent deep reinforcement learning, and relates to a power grid multi-node control method. Background Art

[0002] Due to numerous equations and inequalities, such as rated load and multi-fuel effects, power systems exhibit non-convex input-output characteristics, often preventing direct solutions using fast mathematical programming methods. With the rapid expansion of power systems and the growing imbalance between power demand and generation, issues such as their safety, economics, and vulnerability have become increasingly critical. As the scale of power grids continues to expand, centralized grid control is becoming increasingly difficult, and distributed control methods based on multi-agent systems have become a research focus. Centralized control relies on a central controller to optimize power, voltage, frequency, and load consumption, controlling various nodes in the grid. However, system communication is cumbersome, data processing is complex, and the system's poor robustness can easily lead to system-wide collapse. Distributed control based on multi-agent systems utilizes local observation and control by adjacent agents to replace a central controller. This approach offers advantages such as a simple structure, robustness, and the ability to effectively avoid single points of failure, resulting in enhanced security.

[0003] As a common algorithm for solving complex, high-dimensional decision-making problems today, deep reinforcement learning can extract features from input observations and output decision actions in an end-to-end manner. This learning model has already demonstrated excellent performance in many tasks based on electronic games and autonomous driving. Power grid optimization research has also demonstrated the impressive optimization capabilities of reinforcement learning in power grid environments. However, a multi-agent deep reinforcement learning-based multi-node control method for power grid thermal stability over-limits requires simultaneous training of different strategies for different agent nodes. Furthermore, identifying which nodes have taken correct actions during training, correcting incorrect actions, and assigning different "trust levels" to different nodes for different power grid states are also pressing issues to be addressed. Summary of the Invention

[0004] To solve the above problems, the present invention provides a power grid thermal stability over-limit control method and system based on multi-agent comparative reinforcement learning.

[0005] This invention considers the graph network structure characteristics of power grid data, models power grid data as isomorphic graph data, and adds a comparative learning module to the existing reinforcement learning multi-agent deep Q network based on the value decomposition of the collaborative architecture to promote the model to learn a more discriminative multi-node control method. The technical solution of this invention is:

[0006] The power grid thermal stability over-limit control method based on multi-agent comparative reinforcement learning includes the following steps:

[0007] Step 1. Construct a sample data set of power grid thermal stability exceeding limit;

[0008] The data set was constructed using Pandapower, a Python power system simulation tool. A grid simulation model was first established based on the electrical characteristic parameters and topology of the grid. The active power output of different generators in the grid was then randomly modified based on typical daily data, and the load active power was modified to maintain a power generation-consumption balance. Finally, the data was filtered through power flow calculations: samples with converged power flow calculations and thermal stability exceeding limits were added to the data set. After multiple iterations, the data set could be used to train and verify a reinforcement learning multi-agent model for multi-node thermal stability control in power grids.

[0009] Step 2. Calculate the grid node embedding matrix;

[0010] A power grid graph state can be defined as s = (A, H), where A is the adjacency matrix containing N power grid nodes, It is the original feature matrix composed of N node original features, and then the graph attention neural network is used to calculate the node features between the nodes of the power grid graph:

[0011]

[0012]

[0013]

[0014] Where f is a function that calculates the correlation between the feature vectors of two nodes. g is the linear transformation weight matrix that can be trained, N i is the set of node i itself and its first-order neighbors, σ(·) is an activation function used to generate the final node feature representation of the power grid graph

[0015] Step 3. Construct a value decomposition power grid multi-agent control model;

[0016] A value decomposition deep Q network based on a collaborative architecture is used to build a multi-agent reinforcement learning model for generator scheduling, and the grid graph nodes generated in step 2 are used to represent the features. As model input, the agent in this architecture uses a deep neural network with parameters θ To estimate the specific state of grid node i The value of action a is defined as the expected reward value after executing the action under the current grid state. The agent selects the action with the highest reward value to regulate the grid each time. Then all nodes calculate Q iInput a credit distribution network to calculate the final expected reward value Q tot =mixer(Q1,Q1,...,Q n ). Then the gradient of the neural network is updated through the method of temporal difference.

[0017] Step 4. Build a contrastive learning model;

[0018] Take the multi-agent control model based on value decomposition in step 3 as the backbone model, including the agent network and the credit assignment network mixer; embed the contrastive learning model into the credit assignment network. Define the credit assignment for node i at time t as:

[0019]

[0020] So for each node i we define the credit allocation vector of the time series as:

[0021]

[0022] Where T is the time length of a power grid regulation. In order to make the credit allocation vector To be distinguishable, we introduce the potential identity representation of each node agent i As an intermediate variable, it is used to approximate the identities of different node agents and achieve the distinguishability. To this end, an auxiliary loss function based on contrastive learning is designed and optimized using gradient descent:

[0023]

[0024] Where D is the experience replay pool unique to reinforcement learning, and sim(·) is a function that measures the similarity between two vectors.

[0025] Step 5. Execute generator actions based on the deep learning model;

[0026] After step 4, a more regular credit allocation module is obtained, which can better guide the collaborative regulation of the power grid by the generator nodes. When a set of power grid thermal stability limit data is input, each generator obtains the representation vector according to its own node. The intelligent agent network selects the action with the largest expected value to regulate the power grid and restore it to normal state.

[0027] A system for implementing a power grid thermal stability over-limit control method based on multi-agent comparative reinforcement learning as described in the present invention is characterized by including: a power grid control data construction module, a power grid node embedded feature calculation module, a value decomposition power grid multi-agent control model, a comparative learning module, and a power grid node credit allocation evaluation module.

[0028] A method for regulating power grid thermal stability over-limit based on multi-agent comparative reinforcement learning: first, a series of power grid operation data sets are constructed based on typical days, and the embedded representation vectors of power grid nodes are calculated using a graph attention neural network. Then, a multi-agent reinforcement learning framework for generator scheduling is constructed based on a value decomposition deep Q network based on a collaborative architecture. The framework fits the value of each action of each generator node according to the representation characteristics of the power grid graph. A comparative learning model is then constructed to better distinguish the feature representations of different generator nodes. Finally, a generator action execution module is constructed. Each generator selects the action with the largest expected value through the agent network according to the representation vector of its own node to regulate the power grid, so that the power grid recovers from thermal stability over-limit to normal state. The present invention also includes a power grid thermal stability over-limit regulation system based on multi-agent comparative reinforcement learning.

[0029] The advantages of this invention are: it considers the graph network structure characteristics of power grid data and uses a graph attention neural network to aggregate and extract graph node representation features of power grid data. It then utilizes a multi-agent reinforcement learning algorithm and a contrastive learning module to establish heterogeneous generator action control. Finally, it uses a value decomposition deep Q network learned through reinforcement learning to make decisions. The strategies learned by the agents can effectively and simultaneously adjust multiple generators to achieve thermal stability in the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of generating power grid thermal stability over-limit data according to the present invention.

[0031] Figure 2 It is a general schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is explained and described clearly and completely below in conjunction with the accompanying drawings.

[0033] The power grid thermal stability over-limit control method based on multi-agent comparative reinforcement learning includes the following steps:

[0034] Step 1. Construct a grid temporary stability sample dataset;

[0035] In order to verify the effectiveness of the present invention, it is necessary to construct a graph-structured multi-section over-limit dataset of a power grid, which includes information such as the electrical characteristic parameters of different power grid systems, the topological structure related to the power grid, and the output load of each node under different states. Among them, the electrical characteristic parameters are a series of characteristics inherent in the power grid, which are used in the flow calculation process; the busbars in the power grid topology correspond to the nodes in the graph structure, and the loads connected to the busbars, the voltage of the units and other information are regarded as the accompanying attributes of the nodes. The transmission lines or transformers connecting the two busbars in the power grid topology correspond to the edges in the graph structure; since there are few thermal stability over-limit data in the actual power grid operation, this will lead to an imbalance in the sample analogy of the dataset, so the python power system simulation tool Pandapower is used to construct the dataset; first, a power grid simulation model is established based on the electrical characteristic parameters and topological structure of the power grid, and then the active power output of different generators in the power grid is randomly modified based on the data of a typical day, and the load active power is modified to maintain the balance between power generation and power consumption, and finally the data is filtered through the flow calculation: the samples with convergence of the flow calculation are added to the dataset; after multiple iterations, the dataset can be used to train and verify the reinforcement learning multi-agent model for multi-node control of the power grid ( Figure 1 ).

[0036] Step 2. Calculate the grid node embedding matrix;

[0037] A power grid graph state can be defined as s = (A, H), where A is the adjacency matrix containing N power grid nodes, It is the original feature matrix composed of N node original features, and then the graph attention neural network is used to calculate the node features between the nodes of the power grid graph:

[0038]

[0039]

[0040]

[0041] Where f is a function that calculates the correlation between the feature vectors of two nodes. g is the linear transformation weight matrix that can be trained, N i is the set of node i itself and its first-order neighbors, σ(·) is an activation function used to generate the final node feature representation of the power grid graph

[0042] Step 3. Construct a value decomposition power grid multi-agent control model;

[0043] A deep Q-network based on collaborative architecture is used to build a multi-agent reinforcement learning model for generator scheduling, and the grid graph nodes generated in step 2 are used to represent the features. As model input, the agent in this architecture uses a deep neural network with parameters θ To estimate the specific state of grid node i The value of action a is defined as the expected reward value after executing the action under the current grid state. The agent selects the action with the highest reward value to regulate the grid each time. Then all nodes calculate Q i Input a credit distribution network to calculate the final expected reward value Q tot =mixer(Q1,Q1,...,Q n ). Then the gradient of the neural network is updated by temporal difference. The complete formula of temporal difference is:

[0044]

[0045] Where D is the experience replay pool unique to reinforcement learning, r is the reward value at the current moment, and γ is the preset attenuation coefficient, which is a hyperparameter. is the expected total value at the next moment.

[0046] Step 4. Build a contrastive learning model;

[0047] Take the multi-agent control model based on value decomposition in step 3 as the backbone model, including the agent network and credit assignment network mixer; embed the contrastive learning model into the credit assignment network ( Figure 2 ). Define the credit allocation for node i at time t as:

[0048]

[0049] So for each node i we define the credit allocation vector of the time series as:

[0050]

[0051] Where T is the time length of a power grid regulation. In order to make the credit allocation vector To be distinguishable, we introduce the potential identity representation of each node agent i As an intermediate variable to approximate ( Figure 2 ), to achieve the distinguishability of the identities of different node agents. To this end, an auxiliary loss function based on contrastive learning is designed and optimized using gradient descent:

[0052]

[0053] Where D is the experience replay pool unique to reinforcement learning, sim is a function that measures the similarity between two vectors. When this method is actually used, the inner product of the two vectors is directly used to replace the sim function ( Figure 2 ). It is worth noting that the model in step 4 does not change the original structure of any backbone network. So the total loss function is defined as:

[0054] L tot =L TD +αL CIA #(8)

[0055] The coefficient of the α auxiliary loss function is a hyperparameter.

[0056] Step 5. Execute generator actions based on the deep learning model;

[0057] After step 4, a more regular credit allocation module is obtained, which can better guide the collaborative regulation of the power grid by the generator nodes. When a set of power grid thermal stability limit data is input, each generator obtains the representation vector according to its own node. The intelligent agent network selects the action with the largest expected value to regulate the power grid and restore it to normal state.

[0058] The system for implementing the power grid thermal over-limit control method based on multi-agent comparative reinforcement learning of the present invention comprises: a power grid thermal over-limit data construction module, a power grid node embedded feature calculation module, a value decomposition power grid multi-agent control model, a comparative learning module, and a generator action execution module. Each of these modules sequentially encompasses steps 1 through 5 of the present method.

[0059] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A power grid thermal stability over-limit control method based on multi-agent comparative reinforcement learning includes the following steps: Step 1. Construct a sample data set of power grid thermal stability exceeding limit; Step 2. Calculate the grid node embedding matrix; specifically including: A power grid state is defined as s = (A, H), where A is the adjacency matrix containing N power grid nodes, It is the original feature matrix composed of N node original features, and then the graph attention neural network is used to calculate the node features between the nodes of the power grid graph: Where f is a function that calculates the correlation between the feature vectors of two nodes; W g is the linear transformation weight matrix that can be trained, N i is the set of node i itself and its first-order neighbors, σ(·) is an activation function used to generate the final node feature representation of the power grid graph Step 3. Construct a value decomposition multi-agent control model for power grids. Specifically, it includes: using a value decomposition deep Q network based on a collaborative architecture to construct a multi-agent reinforcement learning model for generator scheduling, and using the grid graph nodes generated in step 2 to represent features. As model input, the agent in this architecture uses a deep neural network with parameters θ To estimate the specific state of grid node i The value of action a is defined as the expected reward value after executing the action under the current power grid state. The agent selects the action with the highest reward value to regulate the power grid each time. Then all nodes calculate Q i Input a credit distribution network to calculate the final expected reward value Q tot = mixer(Q1,Q1,...,Q n ) ; then the gradient of the neural network is updated by temporal difference; Step 4. Build a contrastive learning model; Step 5. Execute generator actions according to the deep learning model.

2. The method for controlling power grid thermal stability over-limit based on multi-agent comparative reinforcement learning according to claim 1, characterized in that: Step 1 specifically includes: using Python's power system simulation tool Pandapower to build a dataset; first, establishing a power grid simulation model based on the electrical characteristic parameters and topology of the power grid; then randomly modifying the active power output of different generators in the power grid based on typical daily data, modifying the load active power to maintain the power generation-consumption balance; and finally, filtering the data through power flow calculation: adding samples with converged power flow calculations and thermal stability exceeding the limit to the dataset; after multiple iterations, the dataset can be used to train and verify a reinforcement learning multi-agent model for multi-node thermal stability exceeding the limit control of power grids.

3. The method for controlling power grid thermal stability over-limit based on multi-agent comparative reinforcement learning according to claim 1, characterized in that: Step 4 specifically includes: taking the multi-agent control model based on value decomposition in step 3 as the backbone model, including the agent network and the credit allocation network mixer; embed the contrastive learning model into the credit allocation network and define the credit allocation for node i at time t as: So for each node i, the credit allocation vector of the timing is defined as: Where T is the time length of a power grid regulation; in order to make the credit allocation vector Becomes distinguishable, introducing the potential identity representation of each node agent i As an intermediate variable, it is used to approximate the identities of different node agents to achieve the distinguishability. For this purpose, an auxiliary loss function based on contrastive learning is designed and optimized using gradient descent: Where D is the experience replay pool unique to reinforcement learning, and sim(·) is a function that measures the similarity between two vectors.

4. The method for controlling power grid thermal stability over-limit based on multi-agent comparative reinforcement learning according to claim 1, characterized in that: Step 5 specifically includes: after step 4, a more regular credit allocation module is obtained, so as to better guide the collaborative regulation of the power grid by the generator nodes; when a set of power grid thermal stability limit data is input, each generator obtains the representation vector according to its own node The intelligent agent network selects the action with the largest expected value to regulate the power grid and restore it to normal state.

5. A system for implementing the power grid thermal stability over-limit control method based on multi-agent comparative reinforcement learning as claimed in claim 1, characterized in that: include: Power grid thermal stability over-limit data construction module, power grid node embedded feature calculation module, value decomposition power grid multi-agent control model, comparative learning module, and generator action execution module.

Images