Metareinforcement learning multi-agent system for complex environment

Abstract

The invention belongs to the technical field of power systems, and more particularly relates to a complex environment-oriented meta reinforcement learning multi-agent system, which learns a group of universal strategy parameters from a large number of simulated power grid scenes through a meta learning module, and takes the strategy parameters as initialization parameters of agents, so as to improve the stability of the multi-agent system. The intelligent agent module is helped to quickly adapt to a new environment; based on the multi-agent cooperation module management and coordination of cooperation and competition relationships among all node agents in the power grid, each agent module can dynamically adjust the behavior strategy of the agent module according to the environment change, and the flexibility and robustness of the whole system are improved; and a high-fidelity power grid simulation environment is constructed through the environment simulation module, so that abundant training scenes are provided for the meta-learning module, and the generalization ability of the strategy in a complex power grid environment is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of power systems, and more specifically, to a meta-reinforcement learning multi-agent system for complex environments. Background Art

[0002] The power grid system is an extremely complex network system, which includes a large number of power plants, substations, transmission lines and various power-consuming equipment. With the continuous development of smart grid technology, the power grid system is evolving towards a more distributed and interconnected direction. This not only improves the flexibility and reliability of the power grid, but also brings new technical challenges.

[0003] In modern power grid systems, the configuration, status, and connection relationships of various power assets are constantly changing. For example, the widespread access to distributed renewable energy, the deployment of electric vehicle charging facilities, and the application of user-side energy storage systems have all led to the continuous evolution of the power grid topology. At the same time, due to environmental factors and equipment failures, the power grid system often faces sudden state changes. These dynamic power grid environments have brought huge challenges to power grid operation and maintenance management.

[0004] Traditional centralized power grid management and control methods are difficult to effectively cope with the dynamic changes in the complex environment of the power grid. The power grid system often needs to rely on a lot of manual intervention and experience to maintain the topology of assets, detect and handle faults, etc. This is not only inefficient, but also prone to unstable power grid operation due to human errors. Therefore, it is urgent to develop new intelligent control technologies to achieve adaptive management of power grid systems.

[0005] In recent years, distributed power grid control methods based on multi-agent systems (MAS) have attracted widespread attention. In this method, each node or device in the power grid is modeled as a collaborative intelligent agent, which completes tasks such as optimal scheduling and fault handling of the power grid through local information exchange and self-organization coordination. However, most existing MAS methods rely on pre-designed rules and strategies, which are difficult to adapt to the complex and changing power grid environment. Summary of the invention

[0006] The present invention provides a meta-reinforcement learning multi-agent system for complex environments, which aims to solve the technical problem that most existing MAS methods rely on pre-designed rules and strategies and are difficult to adapt to complex and changeable power grid environments.

[0007] Meta-reinforcement learning multi-agent system for complex environments, including agent module, meta-learning module, multi-agent collaboration module and environment simulation module;

[0008] The intelligent agent module includes multiple agents for executing decision-making and behavior control; each intelligent agent coordinates and interacts through communication and information sharing;

[0009] The meta-learning module provides a gradient-based optimization algorithm for rapid learning and migration of the strategy and value functions of the agent module;

[0010] The multi-agent collaboration module is used to manage and coordinate the cooperation and competition relationship between multiple agents;

[0011] The environment simulation module simulates the actual environment based on the constructed simulation environment to provide suitable training and verification scenarios for the meta-learning module.

[0012] The present invention learns a set of common strategy parameters from a large number of simulated power grid scenarios through a meta-learning module, and uses the strategy parameters as the initialization parameters of the intelligent agent to help the intelligent agent module quickly adapt to the new environment; then based on the multi-agent collaboration module, the cooperation and competition relationship between the intelligent agents at each node in the power grid is managed and coordinated, so that each intelligent agent module can dynamically adjust its own behavior strategy according to environmental changes, thereby improving the flexibility and robustness of the entire system; and through the environmental simulation module, a high-fidelity power grid simulation environment is constructed to provide a rich training scenario for the meta-learning module, thereby increasing the generalization ability of the strategy in a complex power grid environment.

[0013] Preferably, each of the intelligent agent modules includes a decision-making module and a behavior execution module;

[0014] The decision module is responsible for generating the optimal action decision using a meta-reinforcement learning algorithm according to the current environment state and goal;

[0015] The behavior execution module is responsible for converting the decision into actual action output and feeding the result back to the decision module.

[0016] Preferably, the decision module constructs a deep neural network as a policy function approximator based on the deep reinforcement learning method, and uses the MAML algorithm to learn a set of common policy network parameters to improve the rapid learning ability of the intelligent agent in a complex environment; when the input is the current environmental observation, the output is the probability distribution of each optional action, and the intelligent agent determines the final action based on the distribution with the highest probability.

[0017] Preferably, the intelligent agents exchange information through broadcast and unicast, share environmental information and behavioral decisions, and adopt a communication mechanism based on local areas.

[0018] Preferably, the meta-learning module includes a meta-learning trainer and a meta-strategy generator;

[0019] The meta-learning trainer is responsible for the acute meta-level training and optimization of the policy network of the agent decision module based on the MAML algorithm;

[0020] The meta-strategy generation is responsible for generating an initial strategy suitable for the current environment for each agent based on the trained strategy parameters.

[0021] Preferably, the training steps of the meta-learning trainer are as follows:

[0022] A. Randomly sample several tasks from a set of simulated environments as training sets;

[0023] B. For each task, use the MAML algorithm to update the strategy parameters and obtain the optimal strategy for the task;

[0024] C. Calculate the gradient of the policy parameters for the entire task set and use the gradient to update the initial policy parameters;

[0025] D. Repeat steps A to C until the meta-strategy converges and obtains general strategy parameters suitable for complex environments.

[0026] Preferably, the multi-agent module includes a collaboration mechanism unit, a cooperation incentive unit, a dispute resolution unit, a self-organization coordination unit, and a learning optimization unit;

[0027] The collaboration mechanism unit adopts a distributed collaboration mechanism. Each intelligent agent module has independent decision-making and behavior execution capabilities. Through broadcast and unicast communication methods, intelligent agent modules exchange information, share environmental information and behavior decisions, and improve collaboration efficiency through local area communication mechanisms.

[0028] The cooperation incentive unit adopts a cooperation incentive mechanism based on global rewards. When the behaviors between the intelligent agent modules produce synergistic effects, positive feedback and rewards are given; when the behaviors between the intelligent agent modules produce conflicts, negative feedback and rewards are given.

[0029] The dispute resolution unit adopts a resolution mechanism based on game theory, using the local information and expected benefits of each agent to reach a compromise through game strategies, and introduces the role of arbitrator to mediate and arbitrate disputes;

[0030] The self-organizing coordination unit uses a self-organizing method to dynamically divide roles and assign tasks between each intelligent agent; and the intelligent agent autonomously adjusts its own behavior strategy according to the current environmental status and task requirements; and through an adaptive coordination mechanism, the flexibility and robustness of the entire system are improved;

[0031] The learning and optimization unit utilizes the reinforcement learning method to continuously optimize the cooperation strategy of the intelligent agent module; adjusts the decision and behavior of the intelligent agent module according to the feedback penalty signal; and forms a stable and efficient cooperation mode between multiple intelligent agents through multiple rounds of iterations.

[0032] The beneficial effects of the present invention include:

[0033] The present invention learns a set of common strategy parameters from a large number of simulated power grid scenarios through a meta-learning module, and uses the strategy parameters as the initialization parameters of the intelligent agent to help the intelligent agent module quickly adapt to the new environment; then based on the multi-agent collaboration module, the cooperation and competition relationship between the intelligent agents at each node in the power grid is managed and coordinated, so that each intelligent agent module can dynamically adjust its own behavior strategy according to environmental changes, thereby improving the flexibility and robustness of the entire system; and through the environmental simulation module, a high-fidelity power grid simulation environment is constructed to provide a rich training scenario for the meta-learning module, thereby increasing the generalization ability of the strategy in a complex power grid environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 A system architecture block diagram provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0038] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0039] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0040] See also Figure 1 As shown, the best embodiment of the present invention is further described;

[0041] Meta-reinforcement learning multi-agent system for complex environments, including agent module, meta-learning module, multi-agent collaboration module and environment simulation module;

[0042] The intelligent agent module includes multiple intelligent agents, and each power grid node or device corresponds to an intelligent agent;

[0043] Each of the intelligent agent modules includes a decision-making module and a behavior execution module;

[0044] Decision module:

[0045] Using deep reinforcement learning methods, a deep neural network is constructed as a policy function approximator;

[0046] Network structure design

[0047] A multi-layer fully connected neural network structure is used as the policy function approximator;

[0048] The input layer accepts the current power grid environment state observation x;

[0049] The hidden layer uses the ReLU activation function to gradually extract high-level features;

[0050] The output layer uses a softmax activation function to output the probability distribution π(a|x) of each optional control action a;

[0051] Network parameter initialization

[0052] The Xavier initialization method is used to randomly initialize the network parameters; ensuring that the initial parameter θ conforms to the distribution of zero mean and unit variance; this helps to speed up the convergence of network training;

[0053] Loss function design

[0054] The policy gradient method is used, with the cumulative discounted return R as the optimization target;

[0055] Loss function L = -log(π(a|x))R;

[0056] Where π(a|x) is the probability of taking action a in the current state, and R is the cumulative discounted return of the action;

[0057] Optimization Algorithm

[0058] Use gradient-based optimization algorithms, such as the Adam optimizer;

[0059] Calculate the gradient of the network parameter θ with respect to the loss function L through back propagation

[0060] Update the parameter θ according to the gradient descent to minimize the loss function L;

[0061] Network training

[0062] Use the large amount of training data provided by the power grid simulation environment to train the network;

[0063] Use mini-batch gradient descent to improve training efficiency;

[0064] Dynamically adjust the learning rate during training to improve convergence;

[0065] Strategy Output

[0066] The trained policy network can output the probability distribution π(a|x) of each optional action a;

[0067] The agent takes the action a* with the highest probability as the final control decision.

[0068] Use the MAML (Model-Agnostic Meta-Learning) algorithm to learn general policy network parameters;

[0069] The input is the current grid environment state observation x, and the output is the probability distribution π(a|x) of each optional control action a;

[0070] The agent determines the final control decision based on the action a* with the highest probability;

[0071] The update rules of the MAML algorithm are as follows:

[0072] Randomly select N tasks from a set of simulated power grid environments as training sets;

[0073] For each task, the MAML algorithm is used to perform the following updates:

[0074] Based on the current policy network parameters θ, calculate the loss function L of the task task (θ);

[0075] Then perform a gradient descent update on the parameter θ to obtain the new parameter θ':

[0076]

[0077] Calculate the parameters λ' of all tasks, the gradient of the initial parameters λ, and use the gradient to update the initial parameters λ:

[0078]

[0079] Where: λ is the parameter of the policy network, α represents the local update step size; β is the meta-update step size; is the task loss function; λ' represents the new parameter value of the policy network parameter of the agent decision module after local update on a single task

[0080] Through this MAML training method, a set of common strategy network parameters λ can be learned, which enables the intelligent agent to have the ability to quickly adapt in a complex and changing power grid environment.

[0081] Behavior execution module: converts the control action a* output by the decision module into actual power grid equipment control instructions, and feeds the control results back to the decision module for strategy update; the details are as follows:

[0082] The control action a* output by the decision module is a discrete action selection. This abstract action selection needs to be mapped to an actual power grid equipment control instruction. An action-instruction mapping table can be established to correspond each action to a specific control instruction. According to the action-instruction mapping table, the corresponding control instruction is found from a*, and the control instruction is converted into a standard format that the power grid equipment can recognize and execute. For example, a GOOSE or SV message that complies with the IEC 61850 standard can be generated.

[0083] Establish a communication interface with the power grid equipment and send the generated control instructions to the equipment; industrial communication protocols such as Modbus, IEC 61850, etc. can be used to ensure that the instructions can be accurately transmitted to the corresponding power grid equipment;

[0084] Receive the feedback results after the power grid equipment executes the instructions, and feed back the feedback information, such as equipment status changes, actual control effects, etc., to the decision-making module. The decision-making module can use this feedback to update its own strategy network parameters and improve control performance.

[0085] Multi-agent collaboration module

[0086] Adopting a distributed collaboration mechanism, each intelligent agent node has independent decision-making and execution capabilities;

[0087] Agents exchange local information through broadcast and unicast, and share environmental status and behavior decisions. Broadcast is used to publish environmental status and behavior decisions for reference by nearby points; unicast is used for targeted information interaction and coordination; information exchange follows a communication mechanism based on local areas.

[0088] Design a collaborative incentive mechanism based on game theory, giving positive rewards when the agent's behavior produces a synergistic effect, and negative penalties when it does not;

[0089] The reward and punishment functions are as follows:

[0090]

[0091] Where: r i is the individual reward of agent module i; r ij is the collaboration reward between agents i and j; λ is the collaboration factor;

[0092] The introduction of a self-organizing and coordinating mechanism enables intelligent agents to dynamically adjust according to environmental changes. Intelligent agents can independently determine their own roles in the system and adjust their decision-making strategies to adapt to new environments and collaboration requirements. Self-organizing and coordinating can be based on game theory and achieve global optimization through local interactions.

[0093] Through the above-mentioned distributed collaboration mechanism, local information exchange, collaborative incentive mechanism and self-organization coordination, the multi-agent system can exert a synergistic effect and improve the adaptability of the entire power grid system.

[0094] Specifically:

[0095] The distributed collaboration mechanism ensures the robustness and scalability of the system; local information exchange ensures effective collaboration between agents; the game theory incentive mechanism promotes win-win cooperation between agents; and self-organization and coordination enable the system to autonomously adapt to environmental changes. This multi-agent collaboration mode provides effective technical support for realizing intelligent control in complex power grid environments.

[0096] Meta-learning module

[0097] Contains a meta-learning trainer and a meta-strategy generator;

[0098] The meta-learning trainer uses the MAML algorithm to learn common policy parameters from a large number of power grid simulation scenarios;

[0099] Randomly sample N tasks from a set of simulated power grid environments as training sets;

[0100] For each task, use the MAML algorithm to update the strategy parameters and obtain the optimal strategy for the task;

[0101] Calculate the gradient of the policy parameters for the entire task set and use the gradient to update the initial policy parameters θ;

[0102] Repeat the above steps until the meta-strategy converges.

[0103] The meta-strategy generator generates a suitable initial strategy for each agent node based on the trained general strategy parameters θ.

[0104] Environmental simulation module

[0105] Use physical simulation engines, such as Modelica, to build high-fidelity digital models of power grid systems;

[0106] Simulate the topological structure, line parameters, load characteristics, power generation equipment and other physical characteristics of the power grid;

[0107] Take various physical quantities of the power grid system, such as voltage, current, active / reactive power, etc., as simulation output;

[0108] Environmental random factors

[0109] Introduce various random factors into the basic physical model of power grid;

[0110] Such as equipment failure, fluctuations in new energy output, changes in meteorological conditions, etc.;

[0111] Ensure that the simulation environment can cover complex and changing power grid operation scenarios;

[0112] Data collection and annotation

[0113] Collect a large amount of power grid operation data in a high-fidelity power grid simulation environment;

[0114] Including environmental state observation (such as voltage, power, etc.) and final control effect;

[0115] Label the collected data for subsequent supervised learning;

[0116] Correspondence between simulation environment and actual power grid

[0117] Establish the correspondence between the simulation environment and the actual power grid;

[0118] The high correlation between simulation data and real power grid can be ensured through parameter mapping, modeling formulas, etc.

[0119] Improve the transferability of intelligent agent strategies trained in simulation environments to actual power grids;

[0120] Environment simulation and agent training

[0121] The intelligent agent decision-making module is trained using the constructed high-fidelity power grid simulation environment;

[0122] Input the collected power grid operation data into the decision-making module for reinforcement learning;

[0123] The agent continuously optimizes its decision-making strategy in the simulation environment;

[0124] Through the above method, the environment simulation module can provide the intelligent agent decision-making module with a simulation training environment that is highly close to reality.

[0125] The high-fidelity power grid model built based on the physical simulation engine can accurately simulate the various physical characteristics of the power grid system.

[0126] Introduce random factors, enrich power grid operation scenarios, and enhance the diversity of training data.

[0127] Establish the correspondence between the simulation environment and the actual power grid to improve the mobility of the strategy in reality.

[0128] Collect a large amount of simulation data to provide training samples for the reinforcement learning of intelligent agents.

[0129] In this way, the intelligent agent can continuously optimize its decision-making strategy in a simulation environment close to reality, so that it can quickly adapt to the complex and changing operating environment in the actual power grid.

[0130] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. Meta-reinforcement learning multi-agent system for complex environments, characterized by: Includes agent module, meta-learning module, multi-agent collaboration module and environment simulation module; The intelligent agent module includes multiple agents for executing decision-making and behavior control; each intelligent agent coordinates and interacts through communication and information sharing; The meta-learning module provides a gradient-based optimization algorithm for rapid learning and migration of the strategy and value functions of the agent module; The multi-agent collaboration module is used to manage and coordinate the cooperation and competition relationship between multiple agents; The environment simulation module simulates the actual environment based on the constructed simulation environment to provide suitable training and verification scenarios for the meta-learning module.

2. The reinforcement learning multi-agent system for complex environments according to claim 1, characterized in that: Each of the intelligent agent modules includes a decision-making module and a behavior execution module; The decision module is responsible for generating the optimal action decision using a meta-reinforcement learning algorithm according to the current environment state and goal; The behavior execution module is responsible for converting the decision into actual action output and feeding the result back to the decision module.

3. The reinforcement learning multi-agent system for complex environments according to claim 2, characterized in that: The decision-making module constructs a deep neural network as a policy function approximator based on the deep reinforcement learning method, and uses the MAML algorithm to learn a set of common policy network parameters to improve the rapid learning ability of the intelligent agent in complex environments; when the input is the current environmental observation, the output is the probability distribution of each optional action, and the intelligent agent determines the final action based on the distribution with the highest probability.

4. The reinforcement learning multi-agent system for complex environments according to claim 1, characterized in that: The intelligent agents exchange information through broadcast and unicast, share environmental information and behavior decisions, and adopt a communication mechanism based on local areas.

5. The complex environment oriented reinforcement learning multi-agent system according to claim 1, characterized in that: The meta-learning module includes a meta-learning trainer and a meta-strategy generator; The meta-learning trainer is responsible for the acute meta-level training and optimization of the policy network of the agent decision module based on the MAML algorithm; The meta-strategy generation is responsible for generating an initial strategy suitable for the current environment for each agent based on the trained strategy parameters.

6. The reinforcement learning multi-agent system for complex environments according to claim 5, characterized in that: The training steps of the meta-learning trainer are as follows: A. Randomly sample several tasks from a set of simulated environments as training sets; B. For each task, use the MAML algorithm to update the strategy parameters and obtain the optimal strategy for the task; C. Calculate the gradient of the policy parameters for the entire task set and use the gradient to update the initial policy parameters; D. Repeat steps A to C until the meta-strategy converges and obtains general strategy parameters suitable for complex environments.

7. The complex environment oriented reinforcement learning agent system according to claim 1, characterized in that: The multi-agent module includes a collaboration mechanism unit, a cooperation incentive unit, a dispute resolution unit, a self-organization coordination unit, and a learning optimization unit; The collaboration mechanism unit adopts a distributed collaboration mechanism. Each intelligent agent module has independent decision-making and behavior execution capabilities. Through broadcast and unicast communication methods, intelligent agent modules exchange information, share environmental information and behavior decisions, and improve collaboration efficiency through local area communication mechanisms. The cooperation incentive unit adopts a cooperation incentive mechanism based on global rewards. When the behaviors between the intelligent agent modules produce synergistic effects, positive feedback and rewards are given; when the behaviors between the intelligent agent modules produce conflicts, negative feedback and rewards are given. The dispute resolution unit adopts a resolution mechanism based on game theory, using the local information and expected benefits of each agent to reach a compromise through game strategies, and introduces the role of arbitrator to mediate and arbitrate disputes; The self-organizing coordination unit uses a self-organizing method to dynamically divide roles and assign tasks between each intelligent agent; According to the current environment status and task requirements, the intelligent agent can autonomously adjust its own behavior strategy; and through the adaptive coordination mechanism, the flexibility and robustness of the entire system can be improved; The learning and optimization unit utilizes the reinforcement learning method to continuously optimize the cooperation strategy of the intelligent agent module; adjusts the decision and behavior of the intelligent agent module according to the feedback penalty signal; and forms a stable and efficient cooperation mode between multiple intelligent agents through multiple rounds of iterations.

Images