A multi-agent reinforcement learning power grid energy management method and system

Abstract

The present invention discloses a multi-agent reinforcement learning power grid energy management method and system, which takes the state of the current agent as input and calls the execution strategy to output control actions; the training process of the execution strategy is as follows: constructing agent-level graph topology and bus-level graph topology; constructing a hierarchical graph attention model, inputting the agent-level graph topology and bus-level graph topology into the hierarchical graph attention model to output a global state value, evaluating the state of a large-scale energy management environment and control actions based on the global state value, and using the global state value in an existing multi-agent reinforcement learning algorithm to train the execution strategy of the agent; the power grid energy management method and system enable the agent to learn a cooperative strategy by introducing a paradigm of centralized training and decentralized execution; designing a hierarchical graph attention architecture, reducing the dimensionality of the high-dimensional state space and effectively extracting features, making training more efficient.

Description

Technical Field

[0001] The present invention relates to the technical field of power grid management, and in particular to a multi-agent reinforcement learning power grid energy management method and system. Background Art

[0002] 1) Grid energy management

[0003] The increasing shortage of fossil fuels and growing environmental awareness have made the adoption of solar photovoltaic (PV) power generation a key trend in renewable energy development. In recent years, an increasing number of PV systems have been integrated into distribution networks due to their low-carbon, clean, and economical advantages. However, the growing popularity of PV systems has posed significant challenges to grid voltage stability. Among these challenges, voltage fluctuations in distribution networks caused by current reverse flow and renewable energy instability are receiving increasing attention. These distributed renewable energy sources convert solar energy into electricity and input it into the grid. Rapid variations in solar energy can affect the power input to the grid, leading to unstable voltage fluctuations. Therefore, optimizing the utilization of existing controllable resources in the grid to ensure safe and reliable grid operation, reduce energy waste, and increase the acceptance of renewable energy has attracted widespread attention. Previous research has shown that using inverters to control PV power conversion can alleviate this problem. In addition, various energy storage and energy demand response methods have also been proposed as voltage regulation methods. Therefore, a comprehensive solution is needed to coordinate the control of these resources to ensure stable operation of the entire power system when PV penetration is high. This is the concept of grid energy management.

[0004] Compared to traditional power grids, smart grid energy management integrates not only renewable energy (such as photovoltaic power generation) but also flexible loads (such as air conditioners and electric water heaters) and advanced measurement and control technologies, enabling more efficient management and control of electricity and various other energy sources. Furthermore, as a future power technology energy system, grid energy management leverages advanced sensing and measurement technologies, equipment technologies, and control and decision-making methods to achieve reliable, secure, economical, and efficient management and operation of the grid. Energy management is a core issue in smart grid operations. Smart grid energy management requires flexible integration of large-scale distributed and renewable energy sources to ensure safe and stable grid operation while providing high-quality electricity to end users. Grid energy management, as defined here, aims to schedule, manage, make decisions, and control renewable energy generation units, flexible loads, and storage units within the grid, ultimately ensuring safe grid operation while meeting user energy needs.

[0005] In grid energy management, buildings located at specific nodes in the distribution network are considered intelligent agents that need to control the charge and discharge rates, or energy conversion rates, of multiple components, such as photovoltaics and batteries. Because the electricity consumed or generated passes through the distribution network and causes voltage fluctuations, the goal of grid energy management is to control these components within the building to keep the voltage within a safe range while meeting the energy needs of the building's users. Grid energy management can be formulated as a cooperative task, as all intelligent agents share a common goal: stabilizing the voltage at every node in the entire distribution network. Previous studies have applied deep reinforcement learning to grid energy management, using independent PPO algorithms and rule-based control to stabilize voltage.

[0006] 2) Multi-agent reinforcement learning

[0007] In reinforcement learning, an agent interacts with its environment, performing actions and obtaining rewards based on the state of the environment. This process involves continuous trial and error, iteratively optimizing the agent's strategy based on the information gained. Reinforcement learning algorithms require continuous interaction with the environment and use the data generated from these interactions for training to optimize their strategies. The environments used in reinforcement learning are often uncertain. This differs significantly from traditional machine learning algorithms in several ways: First, the training data is not independent and identically distributed (IID): The data for most machine learning algorithms can be roughly assumed to be IID, otherwise convergence problems will arise. However, the data generated by the interaction between the agent and the environment has strong temporal correlations, making it impossible to completely decouple them at the data level and violating the IID property. Consequently, reinforcement learning training is unstable. The agent's behavior also affects the subsequent data distribution. Second, there is no "correct" behavior and no immediate feedback: supervised learning uses specific sample labels, while reinforcement learning lacks similar strong supervisory signals, typically only a single signal based on a reward function. The delayed reward problem in reinforcement learning scenarios is that the agent cannot receive immediate feedback from a single sample, requiring continuous trial and error and balancing the weights of short-term and long-term rewards. Third, it has a superhuman upper limit: Traditional machine learning algorithms, especially supervised learning, rely on manually labeled data, and the performance upper limit of models trained from this data is the upper limit of the model (human) that produced the data. Reinforcement learning, on the other hand, can start from scratch and continuously interact with the environment, unconstrained by human prior knowledge, enabling superhuman performance in some tasks.

[0008] When multiple agents interact with the environment simultaneously, the entire system becomes a multi-agent system. Each agent's goal remains to maximize expected cumulative reward, but the global state of the environment is dependent on the combined actions of all agents. Therefore, compared to reinforcement learning, multi-agent reinforcement learning also needs to consider the impact of combined actions during the learning process, which presents a greater challenge. Multi-agent reinforcement learning, based on reinforcement learning theory, studies how multiple agents in a multi-agent system select strategies based on their respective goals to maximize cumulative reward. Through environmental perception and information exchange, it aims to identify optimal strategies for collaborative, competitive, and other behaviors among multiple agents. It is used to handle multi-objective, high-dimensional planning, or partially observable sequential decision-making tasks, providing a viable technical solution for large-scale, real-world time-series tasks. Compared to single-agent reinforcement learning, multi-agent reinforcement learning optimizes the strategies of multiple agents to achieve swarm intelligence. This approach can be widely applied in scenarios such as smart cities, military defense, finance and economics, and social governance.

[0009] Traditional control techniques for large, complex, and nonlinear systems in power grids are insufficient for real-time decision-making, especially in systems with high renewable energy penetration. Therefore, the use of deep reinforcement learning algorithms has emerged as a potential and effective approach to alleviate these difficulties. Multi-agent reinforcement learning, in a data-driven and model-free setting, does not require precise environmental modeling and, compared to traditional methods, is applicable to situations with high photovoltaic penetration. Furthermore, the use of multi-agent reinforcement learning in power grids has the potential to reduce costs and is considered to have plug-and-play capabilities.

[0010] The above approaches have the following drawbacks: 1) The distribution network is a complex nonlinear system. If an agent takes an action, it can have a ripple effect on the voltages of all nodes within the network. Previous work has used agents to learn to cooperate only based on their own observations during training and execution. This limits their capabilities and makes it difficult to stabilize the voltages of all nodes. 2) Large-scale multi-agent systems result in a large state space. Real-world distribution networks have hundreds or thousands of households. Directly learning a centralized agent system during training requires handling a vast state space and high-dimensional environment, which leads to serious scalability and efficiency issues. 3) The complex topology of the distribution network. In a distribution network, each node is connected to several other nodes, forming a tree-like structure. The voltage at each node is affected by all other nodes, but the degree of influence decreases with distance. 4) In grid energy management, each building is considered an agent. Building types vary, and different types of buildings have different energy demands. For example, restaurants typically have higher energy demands at noon when people are dining. This suggests that policies should pay more attention to the state of restaurants than to offices at noon. Previous work did not distinguish between different classes of agents, resulting in limited performance. Summary of the Invention

[0011] Based on the technical problems existing in the background technology, the present invention proposes a multi-agent reinforcement learning power grid energy management method and system, introduces the paradigm of centralized training and decentralized execution, so that the agents can learn cooperative strategies; designs a hierarchical graph attention architecture, reduces the dimensionality of the high-dimensional state space and effectively extracts features, making training more efficient, and when the agents make decisions, they can pay more attention to the distribution network topology and the categories of surrounding agents.

[0012] The present invention proposes a multi-agent reinforcement learning method for power grid energy management, which takes the current state of the agent as input and calls the execution strategy to output the control action;

[0013] The training process of the execution strategy is as follows:

[0014] Constructing agent-level graph topology G 1 (V 1 ,D 1 ) and bus-level graph topology G 2 (V 2 ,D 2 ), where G represents the graph topology, V represents the set of all nodes in the graph, and D represents the adjacency matrix;

[0015] Construct a hierarchical graph attention model to convert the agent-level graph topology G 1 (V 1 ,D 1) and bus-level graph topology G 2 (V 2 ,D 2 ) is input into the hierarchical graph attention model to output the global state value, based on which the state of the large-scale energy management environment and the control action are evaluated, and the global state value is used in the existing multi-agent reinforcement learning algorithm to train the execution strategy of the agent.

[0016] Furthermore, the training process of the execution strategy shares the global information in the same centralized Critic, and during the use of the execution strategy, each agent can only access local information during execution.

[0017] Furthermore, the hierarchical graph attention model includes an agent-level graph attention module, an aggregation module, a bus-level graph attention module, a first readout layer, a second readout layer, and a linear layer;

[0018] The input of the agent-level graph attention module is the agent-level graph topology G 1 (V 1 ,D 1 ), the output of the agent-level graph attention module is connected to the input of the aggregation module and the input of the first readout layer respectively, the aggregation module, the bus-level graph attention module, and the second readout layer are connected in sequence, and the output of the first readout layer and the output of the second readout layer are connected to the input of the linear layer;

[0019] The agent-level graph attention module extracts the agent-level graph topology G 1 (V 1 ,D 1 ), obtain agent-level features, which are used as input to the aggregation module, and the features output by the first readout layer and the second readout layer are cascaded to aggregate features, and the aggregated features output the global state value through the linear layer.

[0020] Furthermore, the aggregation module is used to aggregate agent-level nodes into bus classes, specifically: clustering and organizing agent-level features, aggregating and embedding the organized agent-level features into the bus to obtain bus-level embedded features, and inputting the bus-level embedded features into the bus-level graph attention module to convert them into new bus representations.

[0021] Furthermore, graph contrast learning is applied to the hierarchical graph attention model, specifically: in the input agent-level graph topology G 1 (V 1 ,D 1 ), given the graph topology G at the agent level 1 (V i ,D 1 ) and a graph data Z consisting of observations of all nodes in a training sample batch of size Nq , the remaining N-1 graph data in the training sample batch are Z k express;

[0022] For graph data Z q Perform the graph data augmentation operations of observation mask and deletion of some edges respectively to obtain two related graphs as positive pairs and For the remaining graph Z k Randomly perform observation masking and delete some edges to enhance the graph data, and obtain

[0023] Using graph data Z q and Z k Normalized temperature scaled cross entropy loss l n :

[0024]

[0025] Among them, τ represents the temperature parameter, N represents the size of the training sample batch, and W s represents the learnable parameters.

[0026] Furthermore, the agent-level graph topology G 1 (V 1 ,D 1 )The specific construction process is as follows:

[0027] Each agent is modeled as a node, and the node set V 1 Contains all agents in the environment;

[0028] All nodes of the same bus are connected to each other to form a complete graph, and the first agent adjacency matrix is ​​obtained

[0029] Connect the nodes on adjacent buses so that all nodes on the two adjacent buses form a complete bipartite graph, and obtain the second agent adjacency matrix

[0030] The first agent adjacency matrix and the second agent adjacency matrix As a union, we get the final agent adjacency matrix of the agent-level graph topology

[0031] Furthermore, the first agent adjacency matrix The definition is as follows:

[0032]

[0033] Where b(i) and b(j) represent the buses where nodes i and j are located respectively;

[0034] Second agent adjacency matrix The definition is as follows:

[0035]

[0036] Final agent adjacency matrix The definition is as follows:

[0037]

[0038] Here, ∪ represents a union.

[0039] Furthermore, the bus-level graph topology G 2 (V 2 ,D 2 )The specific construction process is as follows:

[0040] The agents from the same bus are considered as a cluster, each bus is modeled as a node, and the node set V 2 It consists of all buses in the distribution network;

[0041] If two buses are connected to each other in a real power grid, then 2 Connect the corresponding nodes in the final bus adjacency matrix

[0042] Terminal bus adjacency matrix The definition is as follows:

[0043]

[0044] A multi-agent reinforcement learning power grid energy management system that takes the current state of the agent as input and calls the execution strategy to output control actions;

[0045] The training process of the execution strategy is as follows:

[0046] Constructing agent-level graph topology G 1 (V 1 ,D 1 ) and bus-level graph topology G 2 (V 2 ,D 2 ), where G represents the graph topology, V represents the set of all nodes in the graph, and D represents the adjacency matrix;

[0047] Construct a hierarchical graph attention model to convert the agent-level graph topology G 1 (V 1 ,D 1 ) and bus-level graph topology G 2 (V 2 ,D 2) is input into the hierarchical graph attention model to output the global state value, based on which the state of the large-scale energy management environment and the control action are evaluated, and the global state value is used in the existing multi-agent reinforcement learning algorithm to train the execution strategy of the agent.

[0048] A computer-readable storage medium stores a plurality of programs, which are used to be called by a processor and execute the multi-agent reinforcement learning power grid energy management method as described above.

[0049] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0050] The advantages of a multi-agent reinforcement learning grid energy management method and system provided by the present invention are: a multi-agent reinforcement learning grid energy management method and system provided in the structure of the present invention; 1) the multi-agent reinforcement learning paradigm of centralized training and decentralized execution is applied to the existing grid energy management system. 2) the entire distribution network is modeled as an agent-level topology and a bus-level topology. Based on these topological structures, a hierarchical graph attention model is constructed to extract the correlation between agents and between agents and the grid. 3) Two graph enhancement methods and graph comparative learning that take into account RL characteristics are designed as an auxiliary task in the reinforcement learning training process to improve representation learning from the grid graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of the architecture of the multi-agent reinforcement learning grid energy management method;

[0052] Figure 2 It is a structural diagram of the bus, intelligent agent, and controllable component actions;

[0053] Figure 3 This is a schematic diagram of the topological structure of the power grid layered graph, where the nodes enclosed by red dashed lines are a bus and all the intelligent agents on it;

[0054] Figure 4 Schematic diagram of the layered graph attention model; DETAILED DESCRIPTION

[0055] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0056] like Figures 1 to 4 As shown in FIG, the present invention proposes a multi-agent reinforcement learning power grid energy management method, which takes the current state of the agent as input and calls the execution strategy to output the control action, such as Figure 2 As shown, controlling the actions of controllable components;

[0057] The training process of the execution strategy is as follows:

[0058] Constructing agent-level graph topology G 1 (V 1 ,D 1 ) and bus-level graph topology G 2 (V 2 ,D 2 ), where G represents the graph topology, V represents the set of all nodes in the graph, and D represents the adjacency matrix;

[0059] Construct a hierarchical graph attention model to convert the agent-level graph topology G 1 (V 1 ,D 1 ) and bus-level graph topology G 2 (V 2 ,D 2 ) is input into the hierarchical graph attention model to output the global state value, based on which the large-scale energy management environment and control actions are evaluated, and then used in the existing multi-agent reinforcement learning algorithm to train the execution strategy of the agent.

[0060] During the reinforcement learning training process, a paradigm of centralized training and decentralized execution is introduced to enable intelligent agents to learn cooperative strategies. A hierarchical graph attention model is designed to reduce the dimensionality of high-dimensional state spaces and effectively extract features, making training more efficient. When intelligent agents make decisions, they can pay more attention to the distribution network topology and the categories of surrounding intelligent agents.

[0061] The global status value is used to evaluate the current status of the large-scale energy management environment and the quality of the executed actions. For example, if the global status value is high, it means that the entire power grid is currently under good control.

[0062] The design of this embodiment mainly includes the following three parts: 1) Applying the multi-agent reinforcement learning paradigm of centralized training and decentralized execution to the existing power grid energy management system. 2) Modeling the entire distribution network as agent-level and bus-level topologies. Based on these topologies, a hierarchical graph attention model is constructed to extract correlations between agents and between agents and the power grid. 3) Two graph enhancement methods and graph contrastive learning that take into account the characteristics of RL are designed as auxiliary tasks in the reinforcement learning training process to improve the representation learning from the power grid graph.

[0063] The system structure of the reinforcement learning grid energy management method is as follows:

[0064] (a1) Centralized Critic: Estimates the current global value based on the current status of all agents;

[0065] (a2) Decentralized Actor: Make action decisions based on the current state of each agent;

[0066] (a3) Hierarchical graph attention model: extracts representations based on two-layer graph topology and agent state information;

[0067] (a4) Graph contrast learning module: Two graph augmentation methods and temperature-scaled cross entropy loss are used as auxiliary tasks for graph contrast learning.

[0068] The following are specific instructions:

[0069] A) Centralized Training and Decentralized Execution

[0070] To enable agents to better learn cooperative strategies, this paper applies centralized training and decentralized execution to power grid energy management. Centralized Training and Decentralized Execution (CTDE) is a classic paradigm in multi-agent reinforcement learning. It assumes that global information is only available during training, and each agent can only access local information during execution to achieve decentralized execution. In this embodiment, power grid energy management can be viewed as a cooperative task because all agents have a common goal, which is to stabilize the voltage of every bus in the entire distribution network. If each agent can only obtain local information on its bus during the training phase, the agent will not consider the global situation, making it difficult to control the voltage within a safe range and ensure service quality. One reason is that if only local observations are considered, the environment is non-stationary, and then the behavior of an agent will actually affect the voltage changes of the entire distribution network.

[0071] Therefore, the CTDE paradigm is applied to the problem of power grid energy management. In CTDE, the information of the agents is shared during the training phase, that is, the training process of the execution strategy shares the global information in the same centralized Critic. During the execution phase and when evaluating the performance of the algorithm, the agents are only allowed to make decisions based on their local observations, that is, during the use of the execution strategy, each agent can only access local information during execution. The advantages of CTDE in power grid energy management are two-fold. On the one hand, the centralized training process can motivate multiple agents to learn cooperative strategies by perceiving a more comprehensive environment. On the other hand, the execution process is completely decentralized, and complete information is not required during the training phase, which ensures the efficiency and flexibility of online energy management. In addition, due to commercial settings and user privacy regulations, the paradigm of centralized training and centralized execution is not applicable to this task.

[0072] B) Grid topology modeling

[0073] In order to capture the correlation between agents and consider the unique characteristics of the power grid, a two-level graph topology structure is constructed, namely the agent-level graph topology G 1 (V 1 ,D 1 ) and bus-level graph topology G 2 (V 2 ,D 2 ), where G represents the graph topology, V represents the set of all nodes in the graph, and D represents the adjacency matrix.

[0074] (B1) Agent-level graph topology G 1 (V 1 ,D 1 )

[0075] Each agent is modeled as a node. The node set V 1 Including all agents in the environment, two types of operations are designed to connect edges.

[0076] The first is the operation of nodes on the same bus, where all nodes on the same bus are connected to each other, and the nodes on the same bus form a complete graph, obtaining the first agent adjacency matrix The first edge connection operation is defined as follows:

[0077]

[0078] Where b(i) and b(j) represent the buses where nodes i and j are located, respectively.

[0079] The second operation is to connect the nodes on adjacent buses. If i and j are connected in the distribution network, all nodes on i will be connected to the nodes on j. Unlike the first operation, this operation makes all nodes on the two adjacent buses form a complete bipartite graph, and the second agent adjacency matrix is ​​obtained. The second edge connection operation is defined as follows:

[0080]

[0081] The first agent adjacency matrix and the second agent adjacency matrix As a union, we get the final agent adjacency matrix of the agent-level graph topology Final agent adjacency matrix The definition is as follows:

[0082]

[0083] Here, ∪ represents a union.

[0084] In summary, the first operation models the relationships of all agents on the same bus, and the second operation models the relationships of different agents on adjacent buses.

[0085] (B2) Bus-level graph topology G 2 (V 2 ,D 2 )

[0086] Agents from the same bus are considered as a cluster, so each bus is modeled as a node, and the node set V 2 It consists of all buses in the distribution network. If two buses are connected to each other in a real power grid, then G 2 In the IEEE ...

[0087]

[0088] exist Figure 3 An example of a graph topology is visualized in , where the nodes enclosed by a red dashed line are a bus and all the agents on it.

[0089] C) Construction of Hierarchical Graph Attention Model

[0090] like Figure 4 As shown in the figure, in order to better extract the relationship between intelligent agents from the power grid topology and provide suitable representation for downstream power grid energy management tasks, the graph attention network GAT is introduced, and a hierarchical graph attention model architecture is designed based on it.

[0091] The hierarchical graph attention model includes an agent-level graph attention module, an aggregation module, a bus-level graph attention module, a first readout layer, a second readout layer, and a linear layer. The input of the agent-level graph attention module is the agent-level graph topology G. 1 (V 1 ,D 1 ), the output of the agent-level graph attention module is connected to the input of the aggregation module and the input of the first readout layer respectively, the aggregation module, the bus-level graph attention module, and the second readout layer are connected in sequence, and the output of the first readout layer and the output of the second readout layer are connected to the input of the linear layer.

[0092] First, using the agent-level graph topology mentioned above, representations are extracted from the observations of the agent through the agent-level graph attention module. The implementation of the original graph attention module is essentially flat because it only relies on the edges of the graph to propagate information. The network structure designed in this embodiment is intended to define a strategy in grid energy management that allows two or more graph attention modules to be used in a hierarchical manner to extract appropriate representations from the multi-layer structure of the grid. Formally, given the input features (agent-level features extracted by the agent-level graph attention module), that is, the output of the upper network, an attempt is made to define a strategy to output new dense graph features. The new graph features contain fewer nodes and fewer node connections. This new graph feature can then be used as the input of another bus-level graph attention module. Therefore, the aggregation module is mainly designed to aggregate agent-level features into bus classes, such as Figure 4 As shown in the upper part, this also means that the agent-level features are converted into bus-level embedded features through the aggregation module. The bus-level features are then converted into new bus representations in the bus-level graph attention module using the bus-level graph topology. The structure of the bus-level graph attention module is similar to the design of the agent-level attention module, with the main difference being the inserted graph topology.

[0093] In order to reduce the dimension of representation and discard redundant information, this embodiment uses the readout layer to aggregate node features to form an output layer with a fixed size representation. The pooling operation in the readout layer mainly extracts the basic information of the state into a potential representation while discarding redundant information. Figure 4 As shown, a permutation-invariant readout layer is applied to extract an integrated representation of the agent. A readout layer is applied after each agent-level graph attention module or bus-level graph attention module. The features output by the first and second readout layers are then concatenated to aggregate the features. The final representation is input into a linear function to predict the global state value of the MAPPO algorithm or MAA2C ​​algorithm. The above readout layer generally refers to the first and second readout layers, and the main structure of the two readout layers is the same.

[0094] D) Image contrast learning auxiliary task

[0095] In large-scale energy management environments, observations from hundreds of agents result in high-dimensional model inputs, making it difficult to learn representations using reinforcement learning objectives, which rely solely on rewards from the environment. Graph contrastive learning has demonstrated its effectiveness in graph prediction tasks but has not been explored in reinforcement learning tasks, primarily due to the different nature of the problem. Inspired by graph contrastive learning, which has been used in graph prediction tasks, and image contrastive learning, which has been used in pixel-based reinforcement learning settings, we design a graph contrastive learning objective as an auxiliary task in reinforcement learning, primarily to stimulate the algorithm to learn better representations from high-dimensional and diverse observation inputs.

[0096] In order to apply graph contrastive learning to multi-agent reinforcement learning, we first introduce the enhancement operations that should be performed on the graph. Graph enhancement methods include:

[0097] (i) Observation masking. For this operation, the algorithm randomly selects some agents and masks a specific proportion of their observations. The observation masking operation forces the model to recover the masked agent observations using information from the unmasked portions. The underlying assumption is that missing node attributes do not significantly impact model performance.

[0098] (ii) Deleting some edges. This operation aims to eliminate the connectivity in the agent-level graph topology by randomly discarding a certain proportion of edges. It shows that the semantics of the agent-level graph topology is robust to the variance of the edge connection pattern. This paper also follows the independent and identical distribution to delete edges. Specifically, given the graph topology G 1 and a graph data Z consisting of observations of all nodes in a training batch of size N being executed q , the remaining N-1 graph data in the training sample batch are Z k Indicates; Z q The above graph data (observation mask and deletion of some edges) will be enhanced to obtain two related graphs as positive pairs and For the remaining graph data Z k Randomly perform observation masking and delete some edges to enhance the graph data, and obtain Then, using the graph data Z q and Z k The normalized temperature scaled cross entropy loss (NT Xent) is as follows:

[0099]

[0100] In this embodiment, bilinear product is used to evaluate the similarity of paired instances, τ represents the temperature parameter, N represents the size of the training sample batch, and W s represents the learnable parameters, the target l n It will be used as an auxiliary task and optimized together with the reinforcement learning objective.

[0101] At this point, the entire system framework has been built. The complete multi-agent hierarchical graph attention reinforcement learning system architecture for power grid energy management is as follows: Figure 1 and 2 As shown in the figure, each agent has a policy and shares the same critic. The agent uses the policy to make decisions based on its own observations. During the training phase, the critic predicts the global state based on the observations of all agents and updates its parameters using reinforcement learning loss and graph contrast loss. The policy is updated using the corresponding reinforcement learning loss and the critic's predictions. During the execution phase, only the policy is used, not the critic, and only the agent's local observations are used to make action decisions.

[0102] Advantages of this embodiment:

[0103] Simple reinforcement learning-based power grid energy management systems suffer from difficulties in learning cooperative strategies, excessively large state spaces, and unutilized topological information, making them difficult to deploy in large-scale smart grids. To address this problem, we applied centralized training with decentralized execution and designed a multi-agent hierarchical graph attention model architecture to form a new reinforcement learning-based power grid energy management system. This system effectively reduces voltage over-limit rates and the number of overvoltage and undervoltage events while meeting various user energy needs, thereby enhancing the stability and security of the entire power grid.

[0104] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A multi-agent reinforcement learning power grid energy management method, characterized in that: Taking the current state of the agent as input, call the execution strategy to output the control action; The training process of the execution strategy is as follows: Building agent-level graph topology and bus-level graph topology ,in, represents the graph topology, represents the set of all nodes in the graph, represents the adjacency matrix; Construct a hierarchical graph attention model to convert the graph topology of the agent level and bus-level graph topology The global state value is input into the hierarchical graph attention model to output the global state value. The state of the large-scale energy management environment and the control action are evaluated based on the global state value. The global state value is then used in the existing multi-agent reinforcement learning algorithm to train the execution strategy of the agent. Among them, graph contrast learning is used as an auxiliary task in the reinforcement learning training process and applied to the hierarchical graph attention model. Specifically: Graph topology at the input agent level Previously, given the graph topology at the agent level and size A graph data consisting of observations of all nodes is being executed in the training sample batch , the rest of the training sample batch Graph data for express; Graph data Perform the graph data augmentation operations of observation mask and deletion of some edges respectively to obtain two related graphs as positive pairs and , for the remaining graph data Randomly perform observation masking and delete some edges to enhance the graph data, and obtain ; Leveraging graph data and Normalized temperature-scaled cross entropy loss : in, represents the temperature parameter, represents the size of the training sample batch, represents the learnable parameters.

2. The multi-agent reinforcement learning power grid energy management method according to claim 1, characterized in that: The training process of the execution strategy shares the global information in the same centralized Critic. During the use of the execution strategy, each agent can only access local information during execution.

3. The multi-agent reinforcement learning power grid energy management method according to claim 1, characterized in that: The hierarchical graph attention model includes an agent-level graph attention module, an aggregation module, a bus-level graph attention module, a first readout layer, a second readout layer, and a linear layer; The input of the agent-level graph attention module is the agent-level graph topology The output of the agent-level graph attention module is connected to the input of the aggregation module and the input of the first readout layer respectively. The aggregation module, the bus-level graph attention module, and the second readout layer are connected in sequence. The output of the first readout layer and the output of the second readout layer are connected to the input of the linear layer. The agent-level graph attention module extracts the agent-level graph topology , the agent-level features are obtained, and the agent-level features are used as the input of the aggregation module. The features output by the first readout layer and the second readout layer are cascaded to aggregate the features, and the aggregated features output the global state value through the linear layer.

4. The multi-agent reinforcement learning power grid energy management method according to claim 3, characterized in that: The aggregation module is used to aggregate agent-level nodes into bus classes, specifically by clustering and organizing agent-level features, aggregating and embedding the organized agent-level features into the bus to obtain bus-level embedded features, and inputting the bus-level embedded features into the bus-level graph attention module to convert them into new bus representations.

5. The multi-agent reinforcement learning power grid energy management method according to claim 1, characterized in that: The agent-level graph topology The specific construction process is as follows: Each agent is modeled as a node, and the node set Contains all agents in the environment; All nodes of the same bus are connected to each other to form a complete graph, and the first agent adjacency matrix is ​​obtained ; Connect the nodes on adjacent buses so that all nodes on the two adjacent buses form a complete bipartite graph, and obtain the second agent adjacency matrix ; The first agent adjacency matrix and the second agent adjacency matrix As a union, we get the final agent adjacency matrix of the agent-level graph topology .

6. The multi-agent reinforcement learning power grid energy management method according to claim 5, characterized in that: First agent adjacency matrix The definition is as follows: in, and Represents nodes respectively and The bus it is on; Second agent adjacency matrix The definition is as follows: Final agent adjacency matrix The definition is as follows: in, Represents a union.

7. The multi-agent reinforcement learning power grid energy management method according to claim 1, characterized in that: The bus-level graph topology The specific construction process is as follows: The agents from the same bus are considered as a cluster, each bus is modeled as a node, and the node set It consists of all buses in the distribution network; If two buses are connected to each other in a real power grid, then Connect the corresponding nodes in the final bus adjacency matrix ; Terminal bus adjacency matrix The definition is as follows:

8. A computer system comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of programs, which are used to be called by the processor and execute the multi-agent reinforcement learning grid energy management method as described in any one of claims 1 to 7.

Images