Distributed micro-grid group collaborative multi-agent random consistent game method
By combining reinforcement learning, graph theory and consensus algorithms in distributed systems, the problem of collaborative control and resource allocation consistency of multiple agents in a random environment is solved, and efficient and robust Distributed control, suitable for the collaborative operation of distributed micronet groups.
Patent Information
- Application Number
- CN202411402366.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2025-01-10
AI Technical Summary
In distributed systems, the problem of multi-agent consistency has always been a hot field of research, and the prior art is difficult to achieve efficient, robust and scalable collaborative control and resource allocation in random environments.
The multi-agent random consistent game method of distributed micronet group collaboration is adopted, combined with reinforcement learning, graph theory and consensus algorithms, and the consistency of collaborative control and resource allocation is achieved through information exchange and strategy learning among multiple agents.
It realizes efficient, robust and scalable distributed control, which provides strong support for the coordinated operation of distributed micronet groups, can achieve consistent decision-making results in a random environment, and improves the stability and performance of the system.
Smart Images
Figure CN119292060A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multi-agent consistency technology, and in particular to a multi-agent random consistency game method for distributed microgrid group collaboration. Background Art
[0002] In distributed systems, the problem of multi-agent consistency has always been a hot research area. The essence of this problem lies in how to make multiple agents in the system reach consistent decision results during the game process. In practical applications, multi-agent consistency is of great significance for solving problems such as collaboration, resource allocation, and routing selection.
[0003] In order to solve the problem of multi-agent consistency, scholars have proposed many distributed game methods. One of the methods is a distributed game method based on reinforcement learning algorithm 4. Reinforcement learning is a machine learning algorithm that solves the optimal decision-making strategy through continuous trial and error and adjustment. In multi-agent games, each agent learns and optimizes its own strategy by observing the dense environment and interacting with other agents. Through the iterative learning process, a consistent decision result is finally achieved. Summary of the invention
[0004] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a multi-agent random consensus game method for distributed microgrid group collaboration.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] A multi-agent random consensus game method for distributed microgrid group collaboration, including information exchange, strategy learning and coordination among multiple agents, to achieve collaborative control and resource allocation consistency in a random environment. This method combines multiple technical means such as reinforcement learning, graph theory, consensus algorithm, etc. to achieve efficient, robust and scalable distributed control.
[0007] S1 Multi-agent Reinforcement Learning:
[0008] ① Technical foundation: Reinforcement learning is a machine learning algorithm that optimizes strategies through trial and error and feedback. In a multi-agent system, each agent learns the optimal strategy by observing the environment, interacting with other agents, and learning the optimal strategy.
[0009] ② Application strategy: Single-agent algorithms such as Q(λ), SARSA, and hierarchical Q can be introduced, and gradually developed into multi-agent algorithms such as DWoLF-PHC(λ), PDWoLF-PHC(λ), etc. to adapt to the complex environment of distributed microgrids
[0010] S2 Graph Theory and Network Modeling:
[0011] ① Modeling method: Each node in the microgrid group (such as distributed energy, energy storage equipment, load, etc.) is regarded as a node in the graph, and the connection relationship between the nodes (such as power transmission lines, communication links, etc.) is regarded as an edge.
[0012] ②Analysis method: Use graph theory tools to analyze the mutual influence and dependency between nodes, construct the network topology of the system, and provide a basis for subsequent collaborative control.
[0013] S3 consensus algorithm:
[0014] ① Algorithm selection: Use distributed consensus algorithms such as Raft and Paxos to ensure that multiple agents can reach consistent decision-making results during the collaborative process.
[0015] ② Implementation mechanism: Through multiple rounds of information exchange and coordination, each agent gradually adjusts its strategy according to its own observation data and the decision results of other agents until a consensus is reached.
[0016] S4 random consistent game:
[0017] ① Game model: The coordinated control problem of distributed microgrid groups is modeled as a random consensus game problem, in which each intelligent agent is a participant in the game, and they optimize their own benefits (such as reducing power generation costs, improving the utilization rate of renewable energy, etc.) through strategy selection and adjustment.
[0018] ②Solution method: Combine reinforcement learning and consensus algorithm, and gradually approach the optimal collaborative control strategy through iterative learning and negotiation process.
[0019] Preferably, the implementation steps are:
[0020] S1 system modeling: According to the actual situation of the distributed microgrid group, build the system's network topology structure and the physical characteristics model of each node.
[0021] S2 algorithm design: Select appropriate reinforcement learning algorithms, graph theory tools and consensus algorithms, and design specific implementation plans for multi-agent random consensus game methods.
[0022] S3 simulation verification: Build a distributed microgrid cluster model in a simulation environment, and apply the designed multi-agent randomized consensus game method to conduct simulation experiments to verify its effectiveness and performance.
[0023] S4 optimization and adjustment: Optimize and adjust the algorithm according to the simulation results to improve its effect and stability in practical applications.
[0024] S5 actual deployment: Deploy the optimized algorithm to the actual distributed microgrid control system to achieve collaborative control and resource allocation.
[0025] Preferably, the system modeling part includes physical modeling and network modeling:
[0026] Physical modeling: According to the actual situation of the distributed microgrid group, establish the physical characteristic model of each node, including distributed energy (such as solar energy, wind energy, etc.), energy storage equipment, load, etc.
[0027] Network modeling: Each node is regarded as a node in the graph, and the connection relationship between nodes (such as power transmission lines, communication links, etc.) is regarded as an edge to build the network topology of the system.
[0028] Game model design: Model the collaborative control problem as a random consensus game problem, and define the payoff function and strategy space of each agent.
[0029] Preferably, the implementation of the algorithm design includes:
[0030] Reinforcement learning algorithm selection: Select reinforcement learning algorithms suitable for multi-agent environments, such as Q-learning, SARSA, Deep Q-Network (DQN), etc., and consider their extension to multi-agent environments, such as Multi-Agent Reinforcement Learning (MARL).
[0031] Consensus algorithm integration: Integrate distributed consensus algorithms (such as Raft, Paxos, etc.) into the reinforcement learning framework to ensure that multiple agents can reach consistent decision results in the collaborative process. Design an information exchange mechanism so that each agent can share key information (such as observation data, decision results, etc.) to support the operation of the consensus algorithm.
[0032] Stochastic consensus game solution: Combine reinforcement learning and consensus algorithms to design a solution algorithm for multi-agent stochastic consensus games. Through iterative learning and negotiation, each agent gradually adjusts its strategy based on its own observation data, the decision results of other agents, and the output of the consensus algorithm until a consistent decision result is reached.
[0033] Preferably, the simulation verification and optimization include:
[0034] Simulation environment construction: Build a model of the distributed microgrid group in the simulation software, including the physical characteristics of each node, network topology, etc. Configure simulation parameters, such as temperature, light speed, wind speed in the next 24 hours, the purchase and sale price of the large power grid, and the upper limit of the power that the tie line can withstand.
[0035] Algorithm implementation and testing: Implement the designed multi-agent random consensus game algorithm into the simulation environment. Conduct multiple simulation experiments to observe the performance of the algorithm in different scenarios, including convergence speed, consistency, system stability, etc.
[0036] Algorithm optimization: Optimize and adjust the algorithm based on the simulation results, such as adjusting the learning rate, improving the efficiency of the consensus algorithm, etc. Repeat the simulation verification process until the algorithm performance meets the requirements.
[0037] Preferably, the actual deployment includes:
[0038] System deployment: Deploy the optimized algorithm to the actual distributed microgrid control system. Configure system parameters to ensure that each node can communicate and work together normally.
[0039] Operation monitoring: Real-time monitoring of the system's operating status, including the power output, energy storage status, load conditions, etc. of each node. Timely detection and handling of abnormal situations to ensure stable operation of the system.
[0040] Performance evaluation: Regularly evaluate the system's performance indicators, such as renewable energy utilization, power generation costs, power supply and demand balance, etc. Optimize and adjust the system based on the evaluation results to further improve its performance and stability.
[0041] Compared with the prior art, the present invention has the following beneficial effects:
[0042] The multi-agent random consensus game method for distributed microgrid group collaboration proposed in this invention is a technical means with broad application prospects. By comprehensively using multiple technical means such as reinforcement learning, graph theory, and consensus algorithm, this method can achieve efficient, robust, and scalable distributed control, providing strong support for the collaborative operation of distributed microgrid groups. In the future, with the continuous development and improvement of technology, this method is expected to be widely used and promoted in more fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 This is a flowchart of a multi-agent random consensus game method for distributed microgrid group collaboration proposed by the present invention.
[0044] Figure 2 This is the demand analysis and system design block diagram of the present invention.
[0045] Figure 3 This is a block diagram of the algorithm design and implementation in the present invention.
[0046] Figure 4 This is a simulation verification and optimization block diagram of the present invention.
[0047] Figure 5 This is a block diagram of the actual deployment and operation of the present invention. DETAILED DESCRIPTION
[0048] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0049] Reference Figure 1-5 , a multi-agent random consensus game method for distributed microgrid group collaboration, including information exchange, strategy learning and coordination among multiple agents to achieve collaborative control and resource allocation consistency in a random environment. This method combines multiple technical means such as reinforcement learning, graph theory, and consensus algorithm to achieve efficient, robust and scalable distributed control.
[0050] S1 Multi-agent Reinforcement Learning:
[0051] ① Technical foundation: Reinforcement learning is a machine learning algorithm that optimizes strategies through trial and error and feedback. In a multi-agent system, each agent learns the optimal strategy by observing the environment, interacting with other agents, and learning the optimal strategy.
[0052] ② Application strategy: Single-agent algorithms such as Q(λ), SARSA, and hierarchical Q can be introduced, and gradually developed into multi-agent algorithms such as DWoLF-PHC(λ), PDWoLF-PHC(λ), etc. to adapt to the complex environment of distributed microgrids
[0053] S2 Graph Theory and Network Modeling:
[0054] ① Modeling method: Each node in the microgrid group (such as distributed energy, energy storage equipment, load, etc.) is regarded as a node in the graph, and the connection relationship between the nodes (such as power transmission lines, communication links, etc.) is regarded as an edge.
[0055] ②Analysis method: Use graph theory tools to analyze the mutual influence and dependency between nodes, construct the network topology of the system, and provide a basis for subsequent collaborative control.
[0056] S3 consensus algorithm:
[0057] ① Algorithm selection: Use distributed consensus algorithms such as Raft and Paxos to ensure that multiple agents can reach consistent decision-making results during the collaborative process.
[0058] ② Implementation mechanism: Through multiple rounds of information exchange and coordination, each agent gradually adjusts its strategy according to its own observation data and the decision results of other agents until a consensus is reached.
[0059] S4 random consistent game:
[0060] ① Game model: The coordinated control problem of distributed microgrid groups is modeled as a random consensus game problem, in which each intelligent agent is a participant in the game, and they optimize their own benefits (such as reducing power generation costs, improving the utilization rate of renewable energy, etc.) through strategy selection and adjustment.
[0061] ②Solution method: Combine reinforcement learning and consensus algorithm, and gradually approach the optimal collaborative control strategy through iterative learning and negotiation process.
[0062] The implementation steps are:
[0063] S1 system modeling: According to the actual situation of the distributed microgrid group, build the system's network topology structure and the physical characteristics model of each node.
[0064] S2 algorithm design: Select appropriate reinforcement learning algorithms, graph theory tools and consensus algorithms, and design specific implementation plans for multi-agent random consensus game methods.
[0065] S3 simulation verification: Build a distributed microgrid cluster model in a simulation environment, and apply the designed multi-agent randomized consensus game method to conduct simulation experiments to verify its effectiveness and performance.
[0066] S4 optimization and adjustment: Optimize and adjust the algorithm according to the simulation results to improve its effect and stability in practical applications.
[0067] S5 actual deployment: Deploy the optimized algorithm to the actual distributed microgrid control system to achieve collaborative control and resource allocation.
[0068] The system modeling part includes physical modeling and network modeling:
[0069] Physical modeling: According to the actual situation of the distributed microgrid group, establish the physical characteristic model of each node, including distributed energy (such as solar energy, wind energy, etc.), energy storage equipment, load, etc.
[0070] Network modeling: Each node is regarded as a node in the graph, and the connection relationship between nodes (such as power transmission lines, communication links, etc.) is regarded as an edge to build the network topology of the system.
[0071] Game model design: Model the collaborative control problem as a random consensus game problem, and define the payoff function and strategy space of each agent.
[0072] The implementation of the algorithm design includes:
[0073] Reinforcement learning algorithm selection: Select reinforcement learning algorithms suitable for multi-agent environments, such as Q-learning, SARSA, Deep Q-Network (DQN), etc., and consider their extension to multi-agent environments, such as Multi-Agent Reinforcement Learning (MARL).
[0074] Consensus algorithm integration: Integrate distributed consensus algorithms (such as Raft, Paxos, etc.) into the reinforcement learning framework to ensure that multiple agents can reach consistent decision results in the collaborative process. Design an information exchange mechanism so that each agent can share key information (such as observation data, decision results, etc.) to support the operation of the consensus algorithm.
[0075] Stochastic consensus game solution: Combine reinforcement learning and consensus algorithms to design a solution algorithm for multi-agent stochastic consensus games. Through iterative learning and negotiation, each agent gradually adjusts its strategy based on its own observation data, the decision results of other agents, and the output of the consensus algorithm until a consistent decision result is reached.
[0076] The simulation verification and optimization include:
[0077] Simulation environment construction: Build a model of the distributed microgrid group in the simulation software, including the physical characteristics of each node, network topology, etc. Configure simulation parameters, such as temperature, light speed, wind speed in the next 24 hours, the purchase and sale price of the large power grid, and the upper limit of the power that the tie line can withstand.
[0078] Algorithm implementation and testing: Implement the designed multi-agent random consensus game algorithm into the simulation environment. Conduct multiple simulation experiments to observe the performance of the algorithm in different scenarios, including convergence speed, consistency, system stability, etc.
[0079] Algorithm optimization: Optimize and adjust the algorithm based on the simulation results, such as adjusting the learning rate, improving the efficiency of the consensus algorithm, etc. Repeat the simulation verification process until the algorithm performance meets the requirements.
[0080] The actual deployment includes:
[0081] System deployment: Deploy the optimized algorithm to the actual distributed microgrid control system. Configure system parameters to ensure that each node can communicate and work together normally.
[0082] Operation monitoring: Real-time monitoring of the system's operating status, including the power output, energy storage status, load conditions, etc. of each node. Timely detection and handling of abnormal situations to ensure stable operation of the system.
[0083] Performance evaluation: Regularly evaluate the system's performance indicators, such as renewable energy utilization, power generation costs, power supply and demand balance, etc. Optimize and adjust the system based on the evaluation results to further improve its performance and stability.
[0084] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A multi-agent random consensus game method for distributed microgrid group collaboration, including information exchange, strategy learning and coordination among multiple agents to achieve collaborative control and resource allocation consistency in a random environment. This method combines multiple technical means such as reinforcement learning, graph theory, consensus algorithm, etc. to achieve efficient, robust and scalable distributed control; It is characterized by The distributed microgrid group coordinated multi-agent random consensus game method comprises the following steps: S1 Multi-agent Reinforcement Learning: ① Technical foundation: Reinforcement learning is a machine learning algorithm that optimizes strategies through trial and error and feedback. In a multi-agent system, each agent learns the optimal strategy by observing the environment and interacting with other agents; ② Application strategy: Introduce Q(λ), SARSA, and hierarchical Q single-agent algorithms, and gradually develop them into multi-agent algorithms such as DWoLF-PHC(λ) and PDWoLF-PHC(λ) to adapt to the complex environment of distributed microgrids; S2 Graph Theory and Network Modeling: ① Modeling method: each node in the microgrid group is regarded as a node in the graph, and the connection relationship between nodes is regarded as an edge; ②Analysis method: Use graph theory tools to analyze the mutual influence and dependency between nodes, build the network topology of the system, and provide a basis for subsequent collaborative control; S3 consensus algorithm: ① Algorithm selection: Use distributed consensus algorithms such as Raft and Paxos to ensure that multiple agents can reach consistent decision results during the collaborative process; ② Implementation mechanism: Through multiple rounds of information exchange and coordination, each agent gradually adjusts its strategy based on its own observation data and the decision results of other agents until a consensus is reached; S4 random consistent game: ① Game model: The coordinated control problem of distributed microgrids is modeled as a stochastic consensus game problem, in which each agent is a participant in the game, and they optimize their own benefits through strategy selection and adjustment; ②Solution method: Combining reinforcement learning and consensus algorithm, through iterative learning and negotiation process, gradually approach the optimal collaborative control strategy.
2. A multi-agent random consensus game method for distributed microgrid group collaboration according to claim 1, characterized in that The implementation steps are: S1 system modeling: According to the actual situation of the distributed microgrid group, the network topology structure of the system and the physical characteristic model of each node are constructed; S2 Algorithm Design: Select appropriate reinforcement learning algorithms, graph theory tools and consensus algorithms, and design specific implementation plans for multi-agent random consensus game methods; S3 Simulation Verification: Build a distributed microgrid cluster model in a simulation environment, and apply the designed multi-agent random consensus game method to conduct simulation experiments to verify its effectiveness and performance; S4 optimization and adjustment: optimize and adjust the algorithm according to the simulation results to improve its effect and stability in practical applications; S5 actual deployment: Deploy the optimized algorithm to the actual distributed microgrid control system to achieve collaborative control and resource allocation.
3. A multi-agent random consensus game method for distributed microgrid group collaboration according to claim 2, characterized in that The system modeling part includes physical modeling and network modeling: Physical modeling: According to the actual situation of the distributed microgrid group, establish the physical characteristic model of each node, including distributed energy, energy storage equipment, and load; Network modeling: each node is regarded as a node in the graph, and the connection relationship between nodes is regarded as an edge, so as to build the network topology of the system; Game model design: Model the collaborative control problem as a random consensus game problem, and define the payoff function and strategy space of each agent.
4. A multi-agent random consensus game method for distributed microgrid group collaboration according to claim 2, characterized in that The implementation of the algorithm design includes: Reinforcement learning algorithm selection: Select a reinforcement learning algorithm suitable for a multi-agent environment and consider its extension to multi-agent environments; Consensus algorithm integration: Integrate the distributed consensus algorithm into the reinforcement learning framework to ensure that multiple agents can reach consistent decision-making results in the collaborative process, and design an information exchange mechanism so that each agent can share key information to support the operation of the consensus algorithm; Solving random consensus games: Combining reinforcement learning and consensus algorithms, we design a solution algorithm for multi-agent random consensus games. Through iterative learning and negotiation, each agent gradually adjusts its strategy based on its own observation data, the decision results of other agents, and the output of the consensus algorithm until a consistent decision result is reached.
5. The multi-agent random consensus game method for distributed microgrid group collaboration according to claim 2 is characterized in that The simulation Verification and optimization include: Simulation environment construction: Build a model of the distributed microgrid group in the simulation software, including the physical characteristics of each node and the network topology; configure simulation parameters, including: temperature, light speed, wind speed in the next 24 hours, the purchase and sale electricity price of the large power grid, and the upper limit of the power that the tie line can withstand; Algorithm implementation and testing: Implement the designed multi-agent random consensus game algorithm into the simulation environment, conduct multiple simulation experiments, and observe the performance of the algorithm in different scenarios, including convergence speed, consistency, and system stability; Algorithm optimization: Optimize and adjust the algorithm based on the simulation results, including adjusting the learning rate, improving the efficiency of the consensus algorithm, and repeating the simulation verification process until the algorithm performance meets the requirements.
6. The multi-agent random consensus game method for distributed microgrid group collaboration according to claim 2, characterized in that: The actual deployment includes: System deployment: deploy the optimized algorithm to the actual distributed microgrid control system, configure system parameters, and ensure that each node can communicate and work together normally; Operation monitoring: Real-time monitoring of the system's operating status, including the power output, energy storage status, and load conditions of each node, timely detection and handling of abnormal situations to ensure stable operation of the system; Performance evaluation: Regularly evaluate the system's performance indicators, such as renewable energy utilization, power generation costs, and power supply and demand balance, and optimize and adjust the system based on the evaluation results to further improve its performance and stability.