Heterogeneous multi-agent cooperative comprehensive park electrical energy scheduling method and system

Abstract

The invention belongs to the technical field of intelligent power grid energy management, and particularly relates to a heterogeneous multi-agent cooperative comprehensive park electric energy scheduling method and system, and the method comprises the steps: customizing a corresponding heterogeneous agent through analyzing the characteristics and operation conditions of each energy system; utilizing a causal relationship graph and a conditional mutual information quantification method to establish a model of situation-dependent causal relationship between the intelligent agents; performing causal intervention collection and condition mutual information calculation between the agents; measuring a causal relationship among the agents to construct an internal reward mechanism of each agent; designing a total reward function of each agent based on the internal reward mechanism of each agent in combination with the direct reward parameter of each agent in the environment; performing strategy optimization based on the total reward function of each agent, and outputting a corresponding scheduling strategy; and executing a corresponding control action according to the scheduling strategy. The cooperative work capability among multiple agents is enhanced, and the overall optimization of park electric energy scheduling is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart grid energy management, and specifically relates to a comprehensive park electric energy dispatching method and system for heterogeneous multi-agent collaboration. Background Art

[0002] With the continuous adjustment of my country's industrial structure and the rapid progress in the field of smart buildings, the construction scale of parks, as important places of energy consumption, continues to expand, and the demand for electric energy has also risen sharply. The park's power energy system is a complex and large system, covering a variety of energy components such as photovoltaic power generation systems, energy storage devices, municipal grid access points, and electric vehicle charging piles, which together constitute a microgrid system. At present, how to manage this diversified energy system in an efficient, economical and environmentally friendly way, while ensuring that the electricity demand of users within the park is met and the comfort of electricity consumption is maintained, has become a core challenge that needs to be solved urgently.

[0003] Traditional campus power energy system dispatch strategies mainly rely on fixed models and algorithms, which make it difficult to fully consider the dynamic interactions between subsystems and the real-time changing working environment. Faced with complex and changing energy demands, this static dispatch mode often fails to ensure the maximization of overall energy utilization efficiency, and may even lead to energy waste and increased operating costs. Especially in energy systems with heterogeneous characteristics such as photovoltaic power generation, energy storage, and electric vehicle charging piles, the traditional dispatch method is even more inadequate due to the interdependence of the operating status and decision-making process of these components. Summary of the invention

[0004] In view of the complex and changeable energy demand faced by the traditional park power energy system scheduling strategy, this static scheduling mode is often difficult to ensure the maximization of the overall energy utilization efficiency, and may even cause energy waste and increased operating costs. The present invention provides a comprehensive park power energy scheduling method and system with heterogeneous multi-agent collaboration.

[0005] In a first aspect, the technical solution of the present invention provides a comprehensive park power energy dispatching method for heterogeneous multi-agent collaboration, comprising the following steps: Customize corresponding heterogeneous intelligent agents by analyzing the characteristics and operating conditions of each energy system; Using causal relationship graphs and conditional mutual information quantification methods, a model of context-dependent causal relationships between agents is established; that is, a multi-agent causal graph model; Carry out causal intervention collection and conditional mutual information calculation between intelligent agents; Measure the causal relationship between each agent and build the intrinsic reward mechanism of each agent; Design the total reward function for each agent based on the intrinsic reward mechanism of each agent and the direct benefit parameters of each agent in the environment; Optimize the strategy based on the total reward function of each agent and output the corresponding scheduling strategy; According to the scheduling strategy, corresponding control actions are executed to realize the scheduling of the electric energy of the entire park.

[0006] As a preferred embodiment of the technical solution of the present invention, the steps of customizing corresponding heterogeneous intelligent agents by analyzing the characteristics and operating conditions of each energy system include: Constructing a photovoltaic power generation system agent; the photovoltaic power generation system agent generates electricity according to the intensity and time of sunshine; maximizing the power generation by setting the state space, action space and environmental reward function of the photovoltaic power generation system agent; Constructing an energy storage system intelligent agent; the energy storage system intelligent agent optimizes the charging and discharging strategy according to the peak and valley electricity prices of the power grid, load demand and the charging state of the battery; setting the state space, action space and environmental reward function of the energy storage system intelligent agent; Construct an intelligent agent for a car charging pile system; the intelligent agent for the car charging pile system determines the source of electricity and the charging or discharging operation according to the charging demand of the electric vehicle, the electricity price of the power grid and the charging status of the energy storage battery; and sets the state space, action space and environmental reward function of the intelligent agent for the car charging pile system.

[0007] As a preferred embodiment of the technical solution of the present invention, the state space of the photovoltaic system intelligent agent is:

[0008] Action space of photovoltaic system agent To adjust the power generation of the photovoltaic system according to the real-time solar radiation intensity and weather conditions to maximize power output;

[0009] Environmental reward function for photovoltaic system agents It is based on the amount of electricity it generates and its reliance on the grid;

[0010] In the formula, is the solar radiation intensity at the current time t, The current power generation of the photovoltaic system, Light exposure time; Indicates the possible shielding of solar radiation, 0 means complete shielding, and 1 means no shielding; is the grid electricity price at time t; is the minimum output power of photovoltaic power generation, is the maximum rated power of the photovoltaic system, is the output power of photovoltaic power generation; is the power generated by the photovoltaic system at time t.

[0011] As a preferred embodiment of the technical solution of the present invention, in the multi-agent causal graph model, the node is used to represent the state of each agent i at time t ,action and the state at time t+1 ; The state node describes the agent's perception of the current environment, and the action node describes the agent's decision-making behavior; the edge is used to represent the causal relationship between multiple heterogeneous agents, that is, the state or action of a certain agent affects the state or action of other agents, and the direction of the edge indicates the flow of the causal relationship.

[0012] As a preferred embodiment of the technical solution of the present invention, the steps of collecting causal interventions and calculating conditional mutual information between intelligent agents include: Quantify the degree to which the action of one system depends on the state of other systems by calculating conditional mutual information; Current status of the photovoltaic system Next, its action Next state of energy storage system The conditional mutual information formula of the causal influence intensity is as follows:

[0013] Among them, the left side of the formula is the conditional mutual information The approximate value of Represents a neural network.

[0014] As a preferred embodiment of the technical solution of the present invention, the step of measuring the causal relationship between the agents and constructing the intrinsic reward mechanism of each agent includes: constructing the intrinsic reward mechanism based on the calculated causal influence strength, specifically including: When the action of the photovoltaic system has a positive causal impact on the energy storage system and the charging pile system and the causal impact intensity is greater than the first set value, the photovoltaic system is given an intrinsic reward to maximize the incentives of the photovoltaic system to other systems; When the action of the energy storage system has a positive impact on the demand satisfaction rate of the charging pile system and the causal influence intensity is greater than the second set value, an intrinsic reward is allocated to the energy storage system to select the charging and discharging strategy that is most beneficial to the charging pile system; When the behavior of the charging pile system has a positive causal impact on the load management of the power grid or energy storage system, it provides intrinsic rewards for the charging pile system to optimize its support for the system load.

[0015] As a preferred embodiment of the technical solution of the present invention, the step of designing the total reward function of each agent based on the intrinsic reward mechanism of each agent combined with the direct benefit parameter of each agent in the environment specifically includes: The total reward function of the photovoltaic system:

[0016] The total reward function of the energy storage system:

[0017] The total reward function of the energy storage system:

[0018] in, , and Respectively represent the direct benefits of each system in the environment; In order to adjust the weight coefficient of causal influence rewards, the reference values ​​are between [0-1] and can be adjusted according to the actual degree of collaboration between the systems.

[0019] As a preferred embodiment of the technical solution of the present invention, the step of performing strategy optimization based on the total reward function of each agent and outputting the corresponding scheduling strategy includes: Design the system total reward function based on the total reward function of each agent; And considering the electricity expenditure, this patent introduces it as a penalty term into the overall optimization objective function; By maximizing the total reward of each agent, the PPO objective function is used to perform policy optimization and output the optimized scheduling strategy.

[0020] System total reward function .

[0021] As a preferred embodiment of the technical solution of the present invention, by maximizing the total reward of each agent, the overall optimization objective function of the system is :

[0022]

[0023] Electricity cost of the whole system As shown in the above formula. Specifically, It represents the cost of electricity purchased by the energy storage system from the grid minus the revenue from electricity sold by the energy storage system to the grid; Represents the revenue from the electricity sold by the PV system to the grid; It represents the cost of the charging station purchasing electricity from the grid minus the income from selling electricity to the grid; Indicates the real-time electricity price.

[0024] For each agent i, the objective function of PPO is used for optimization; Strategy update objective function:

[0025] Where: is the total reward of all agents at time step t, π represents the policy function of different systems; T is the maximum number of time steps, γ is the discount factor used to control the decay degree of future rewards, and the value range is 0≤γ≤1; represents the strategy update objective function, is the probability ratio of the current strategy to the old strategy, that is, the current strategy in state Next select action The ratio of the probability of choosing this action to the probability of choosing this action by the old strategy; is the advantage estimate at time step t, For the shear operation, is a hyperparameter used to control the magnitude of the policy update. In a second aspect, the technical solution of the present invention provides a comprehensive park power energy dispatching system for heterogeneous multi-agent collaboration, including a heterogeneous agent construction module, wherein the heterogeneous agent construction module is connected to an agent causal graph model module, an intrinsic reward design module, and a total reward calculation and strategy optimization module; the agent causal graph model module is connected to the intrinsic reward design module through a causal relationship sampling and conditional mutual information calculation module; Heterogeneous agent building module, which is used to customize corresponding heterogeneous agents by analyzing the characteristics and operating conditions of each energy system; The agent causal graph model module is used to build a model of context-dependent causal relationships between agents using causal relationship graphs and conditional mutual information quantification methods; that is, a multi-agent causal graph model; Causal relationship sampling and conditional mutual information calculation module, used for causal intervention collection and conditional mutual information calculation between intelligent agents; The intrinsic reward design module is used to measure the causal relationship between each intelligent agent and build the intrinsic reward mechanism of each intelligent agent; The total reward calculation and strategy optimization module is used to design the total reward function of each agent based on the intrinsic reward mechanism of each agent and the direct benefit parameters of each agent in the environment; optimize the strategy based on the total reward function of each agent and output the corresponding scheduling strategy.

[0026] As a preferred embodiment of the technical solution of the present invention, the system also includes a scheduling execution module, which is used to execute corresponding control actions according to the scheduling strategy to realize the scheduling of electric energy in the entire park.

[0027] As a preferred embodiment of the technical solution of the present invention, a heterogeneous agent construction module is specifically used to construct a photovoltaic power generation system agent; the photovoltaic power generation system agent generates electricity through sunshine intensity and time; the power generation is maximized by setting the state space, action space and environmental reward function of the photovoltaic power generation system agent; an energy storage system agent is constructed; the energy storage system agent optimizes the charging and discharging strategy according to the peak and valley electricity prices of the power grid, load demand and the charging status of the battery; the state space, action space and environmental reward function of the energy storage system agent are set; a car charging pile system agent is constructed; the car charging pile system agent determines the source of electricity and the charging or discharging operation according to the charging needs of the electric vehicle, the power price of the power grid and the charging status of the energy storage battery; the state space, action space and environmental reward function of the car charging pile system agent are set.

[0028] As a preferred embodiment of the technical solution of the present invention, the state space of the photovoltaic system intelligent agent is:

[0029] Action space of photovoltaic system agent To adjust the power generation of the photovoltaic system according to the real-time solar radiation intensity and weather conditions to maximize power output;

[0030] Environmental reward function for photovoltaic system agents It is based on the amount of electricity it generates and its reliance on the grid;

[0031] In the formula, is the solar radiation intensity at the current time t, The current power generation of the photovoltaic system, Light exposure time; Indicates the possible shielding of solar radiation, 0 means complete shielding, and 1 means no shielding; is the grid electricity price at time t; is the minimum output power of photovoltaic power generation, is the maximum rated power of the photovoltaic system, is the output power of photovoltaic power generation; is the power generated by the photovoltaic system at time t.

[0032] As a preferred embodiment of the technical solution of the present invention, in the multi-agent causal graph model, the node is used to represent the state of each agent i at time t ,action and the state at time t+1 ; The state node describes the agent's perception of the current environment, and the action node describes the agent's decision-making behavior; the edge is used to represent the causal relationship between multiple heterogeneous agents, that is, the state or action of a certain agent affects the state or action of other agents, and the direction of the edge indicates the flow of the causal relationship.

[0033] As a preferred embodiment of the technical solution of the present invention, the causal relationship sampling and conditional mutual information calculation module quantifies the degree of dependence of the action of a system on the state of other systems by calculating conditional mutual information; Current status of the photovoltaic system Next, its action Next state of energy storage system The conditional mutual information formula of the causal influence intensity is as follows:

[0034] Among them, the left side of the formula is the conditional mutual information The approximate value of Represents a neural network.

[0035] As a preferred embodiment of the technical solution of the present invention, an intrinsic reward design module constructs an intrinsic reward mechanism based on the calculated causal influence intensity. The intrinsic reward mechanism specifically includes: when the action of the photovoltaic system has a positive causal impact on the energy storage system and the charging pile system and the causal influence intensity is greater than a first set value, an intrinsic reward is given to the photovoltaic system to maximize the incentives of the photovoltaic system to other systems; when the action of the energy storage system has a positive impact on the demand satisfaction rate of the charging pile system and the causal influence intensity is greater than a second set value, an intrinsic reward is allocated to the energy storage system to select the charging and discharging strategy that is most beneficial to the charging pile system; when the behavior of the charging pile system has a positive causal impact on the load management of the power grid or the energy storage system, an intrinsic reward is provided to the charging pile system to optimize its support for the system load.

[0036] As a preferred embodiment of the technical solution of the present invention, the total reward function of each agent designed by the total reward calculation and strategy optimization module includes: The total reward function of the photovoltaic system:

[0037] The total reward function of the energy storage system:

[0038] The total reward function of the energy storage system:

[0039] in, , and Respectively represent the direct benefits of each system in the environment; In order to adjust the weight coefficient of causal influence rewards, the reference values ​​are between [0-1] and can be adjusted according to the actual degree of collaboration between the systems.

[0040] As a preferred embodiment of the technical solution of the present invention, the total reward calculation and strategy optimization module designs the system total reward function based on the total reward function of each agent; by maximizing the total reward of each agent, the PPO objective function is used to perform strategy optimization and output the optimized scheduling strategy.

[0041] System total reward function .

[0042] As a preferred embodiment of the technical solution of the present invention, the total reward calculation and strategy optimization module maximizes the total reward of each agent and takes into account the electricity cost, and introduces it as a penalty term into the overall optimization objective function. In the step of using the PPO objective function to optimize the strategy and output the optimized scheduling strategy, the overall optimization objective function of the system is :

[0043]

[0044] is the electricity cost of the entire system. Specifically, It represents the cost of electricity purchased by the energy storage system from the grid minus the revenue from electricity sold by the energy storage system to the grid; Represents the revenue from the electricity sold by the PV system to the grid; It represents the cost of the charging station purchasing electricity from the grid minus the income from selling electricity to the grid; Indicates the real-time electricity price.

[0045] For each agent i, the objective function of PPO is used for optimization; Strategy update objective function:

[0046] Where: is the total reward of all agents at time step t, π represents the policy function of different systems; T is the maximum number of time steps, γ is the discount factor used to control the decay degree of future rewards, and the value range is 0≤γ≤1; represents the strategy update objective function, is the probability ratio of the current strategy to the old strategy, that is, the current strategy in state Next select action The ratio of the probability of choosing this action to the probability of choosing this action by the old strategy; is the advantage estimate at time step t, For the shear operation, is a hyperparameter used to control the magnitude of the policy update.

[0047] It can be seen from the above technical solutions that the present invention has the following advantages: the present invention customizes the corresponding heterogeneous agents by analyzing the characteristics and operating conditions of each energy system; at the same time, the context-dependent causal relationship between agents is clarified by using the causal relationship map and the conditional mutual information quantification method. This not only allows each agent to make the best judgment based on its own state, but also prompts them to take actions that are conducive to improving the overall system efficiency. Ultimately, by considering this causal interaction as part of the internal incentive mechanism, the collaborative working ability between members within the multi-agent system is enhanced, and the overall optimization of the park's power energy scheduling is achieved. By optimizing the strategies of the three agents in parallel, the system executes the corresponding control actions, thereby effectively improving the collaborative efficiency between the photovoltaic power generation system, the energy storage system, and the vehicle charging pile system, and ultimately achieving the collaborative optimization of the energy management system and maximizing the overall benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0050] Figure 2 It is a schematic diagram of the multi-agent causal graph model.

[0051] Figure 3 is a schematic block diagram of a system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention proposes a heterogeneous multi-agent collaboration (SCIC) strategy based on context-dependent causal influence, aiming to optimize the dispatch efficiency of the modern integrated park power energy system. Aiming at the inherent heterogeneous characteristics of different energy systems such as photovoltaic, energy storage, and electric vehicle charging piles, special heterogeneous agents are designed, and the causal influence map between these agents is established to promote effective cooperation between them. Specifically, the present invention customizes the corresponding heterogeneous agent model by analyzing the characteristics and operating conditions of each energy system; at the same time, the context-dependent causal relationship between agents is clarified by using the causal relationship map and the conditional mutual information quantification method. This not only allows each agent to make the best judgment based on its own state, but also prompts them to take actions that are conducive to improving the overall system efficiency. Finally, by considering this causal interaction as part of the internal incentive mechanism, the collaborative working ability between members in the multi-agent system is enhanced, and the overall optimization of the park power energy dispatch is achieved. In order to enable people in the field of this technology to better understand the technical solution in the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work should fall within the scope of protection of the present invention.

[0053] like Figure 1 As shown, an embodiment of the present invention provides a comprehensive park power energy dispatching method for heterogeneous multi-agent collaboration, comprising the following steps: Step 1: Customize the corresponding heterogeneous intelligent agents by analyzing the characteristics and operating conditions of each energy system; The embodiments of the present invention focus on the heterogeneous characteristics between different energy systems such as photovoltaics, energy storage, and electric vehicle charging piles, and construct special heterogeneous agents to represent these systems. At the same time, factors such as load and municipal electricity prices are regarded as the state of the environment. Specifically, the functions and goals of each agent are as follows: (1) Photovoltaic power generation system agent: The photovoltaic system agent generates electricity according to the intensity and time of sunshine, with the goal of maximizing power generation and minimizing dependence on the power grid.

[0054] ① State space: including but not limited to The solar radiation intensity at the current time t, in watts per square meter (W / m2); The current power generation of the photovoltaic system, in kilowatts (kW); Light exposure time; Weather conditions, indicating possible shielding of solar radiation, are expressed in weather indexes with a value range of [0,1], where 0 indicates complete shielding and 1 indicates no shielding; is the grid electricity price at time t.

[0055] Therefore, the state space of the photovoltaic system agent is: (1) ②Action space: Action space of the photovoltaic system agent It is defined as: adjusting the power generation of the photovoltaic system according to the real-time solar radiation intensity and weather conditions to maximize the power output. Can be determined by the agent, whose maximum power is the rated power of the photovoltaic system , the action space includes the adjustment range of the generated power.

[0056] (2) in The minimum output power of photovoltaic power generation, usually zero, indicating no power generation. is the maximum rated power of the photovoltaic system.

[0057] ③Environmental reward function: The environmental reward function of the photovoltaic system agent, Based on the amount of electricity generated and the degree of dependence on the grid, the reward function takes into account the benefits of reducing grid dependence and is associated with the grid electricity price to encourage the agent to increase power generation as much as possible and reduce grid power purchases when the grid electricity price is high.

[0058] (3) in, is the grid electricity price at time t, The electricity generated by the photovoltaic system at time t, the reward value increases as the photovoltaic power generation increases, and a higher reward can be obtained by maximizing the power generation.

[0059] (2) Energy storage system agent: The task of the energy storage system agent is to optimize the charging and discharging strategy according to the peak and valley electricity prices of the power grid, load demand, and the charging status of the battery. The energy storage system can not only store the excess power of the photovoltaic power generation system, but also purchase electricity for storage when the power grid electricity price is low, and sell the stored electricity back to the power grid when the electricity price is high, thereby maximizing the economic benefits of the system.

[0060] ① State space: including but not limited to : The current battery power in kilowatt-hours (kWh); : The charging rate of the battery in kilowatts (kW). : The discharge rate of the battery in kilowatts (kW); : The power purchase price of the power grid at time t, in yuan / kilowatt-hour (yuan / kWh); The grid electricity price at time t, in yuan / kilowatt-hour (yuan / kWh); the current load demand , in kilowatts (kW). Therefore, the state space of the energy storage system agent is: (4) ②Action space: the action space of the energy storage system agent Defined as the rate control range for battery charging and discharging, the action space includes: (5) in It is the maximum charging and discharging capacity of the energy storage system.

[0061] ③Environmental reward function: Environmental reward function of the energy storage system agent It mainly includes two parts: the economic benefits of municipal electricity purchase and sales while meeting load demand.

[0062] - (6) in, is the grid (purchase) electricity price at time t; is the grid (sale) electricity price at time t, is the amount of electricity purchased by the energy storage system from the grid at time t. is the amount of electricity sold by the energy storage system to the grid at time t, is the current load demand.

[0063] The intelligent agent of the automobile charging pile system is responsible for determining the source of electricity and the operation of charging or discharging according to the charging demand of electric vehicles, the electricity price of the power grid and the charging status of the energy storage battery. The intelligent agent of the charging pile has a two-way function: when the electricity price is low, it charges from the power grid or the energy storage system; when the electricity price is high, it sells electricity back to the power grid. ① State space: including but not limited to : Current charging power of the electric vehicle, in kilowatts (kW); : The current discharge power of the electric vehicle, in kilowatts (kW). : The remaining power of the electric vehicle's current battery, in kilowatt-hours (kWh). : The price of electricity purchased from the power grid, in yuan per kilowatt-hour (yuan / kWh). : The electricity selling price of the power grid, in yuan / kilowatt-hour (yuan / kWh).

[0064] Therefore, the state space of the electric vehicle charging pile agent is: (7) ②Action space: Action space of the intelligent agent in the car charging pile system The following control variables are included: Charging action: The charging pile decides whether to provide power to the electric vehicle and controls the charging power. , so that it does not exceed the maximum charging capacity of the charging pile .

[0065] (8) Discharge action: The charging pile decides whether to feed the stored power back to the grid or energy storage system to control the discharge power. , and also cannot exceed its maximum discharge capacity.

[0066] (9) ②Environmental reward function: Environmental reward function of the intelligent agent of the car charging pile system It also mainly includes two main parts: the economic benefits of municipal electricity purchase and sales.

[0067] (10) in, is the grid (purchase) electricity price at time t; is the grid (sale) electricity price at time t, It is the amount of electricity purchased by the electric vehicle charging pile system from the power grid at time t. The amount of electricity sold to the grid at time t.

[0068] Step 2: Use the causal relationship graph and conditional mutual information quantification method to establish a model of context-dependent causal relationships between agents; that is, a multi-agent causal graph model; By modeling the causal relationship between agents, it helps agents consider the behavior of other agents and their overall impact on the system during decision-making. This modeling method relies on causal graphs to represent and reason about causal dependencies, thereby guiding the decision-making process of each agent in a multi-agent system.

[0069] The schematic diagram of the multi-agent causal graph model is as follows Figure 2 As shown in the multi-agent causal graph model of the integrated park power energy system, the nodes are used to represent the state of each power system agent i at time t. ,action and the state at time t+1 Among them, the state node describes the agent's perception of the current environment, and the action node describes the agent's decision-making behavior. In addition, the edge is used to represent the causal relationship between multiple heterogeneous agents, that is, the state or action of a certain agent affects the state or action of other agents, and the direction of the edge indicates the flow of the causal relationship. For example: the power output of a photovoltaic power generation system Influencing the charging and discharging decisions of energy storage systems wait.

[0070] Step 3: Collect causal interventions and calculate conditional mutual information between agents; By constructing a causal graph, the potential causal relationship between various intelligent agents is clarified, which helps to identify which agents' actions and states may have causal influences, so that in the subsequent causal intervention sampling and conditional mutual information calculation modules, the impact of these key nodes on other systems can be analyzed intensively.

[0071] Causal intervention sampling: In order to measure the impact of an agent's actions on the states of other agents, this patent intervenes in the actions of the agent to observe its causal effects under specific contextual dependencies. For example, the actions of a photovoltaic power generation system can be Causal intervention is performed by artificially setting the strategy of power output and recording the next time step state of the energy storage system and charging pile system in this scenario ( and )’s impact.

[0072] Conditional mutual information calculation: After causal intervention, conditional mutual information is calculated to quantify the degree to which the action of a system depends on the state of other systems. For example: the current state of a photovoltaic system Next, its action Next state of energy storage system The conditional mutual information formula of the causal influence intensity is as follows: (11) The left side of the formula is the conditional mutual information An approximate value, which is used to assess the current state of the photovoltaic system Next, its action Next state of energy storage system The strength of the causal influence. Represents a neural network, such as the MINE model, with parameters Control is used to estimate the approximate value of the conditional mutual information. By adjusting , neural network It is possible to optimize the estimation of conditional mutual information.

[0073] Step 4: Measure the causal relationship between each agent and build the intrinsic reward mechanism of each agent; Based on the calculated causal influence strength, an intrinsic reward mechanism is constructed to encourage each system to choose actions that have a positive impact on other systems during the collaborative optimization process: (1) Intrinsic rewards of photovoltaic power generation system: When the action of the photovoltaic system has a significant positive causal impact on the energy storage system and the charging pile system (such as >0 and >0), the PV system is given an intrinsic reward. This intrinsic reward is intended to encourage the PV system to optimize its output strategy in a specific situation to maximize its incentives to other systems.

[0074] (2) Intrinsic rewards of energy storage system: When the action of the energy storage system has a significant positive impact on the demand satisfaction rate of the charging pile system (e.g. >0), an intrinsic reward is allocated to the energy storage system to encourage it to choose a charging and discharging strategy that is beneficial to the charging pile system at the appropriate time.

[0075] (3) Intrinsic rewards for the charging pile system: When the behavior of the charging pile system has a positive causal impact on the load management of the power grid or energy storage system, intrinsic rewards are provided to the charging pile system to optimize its support for the system load.

[0076] Step 5: Design the total reward function of each agent based on the intrinsic reward mechanism of each agent and the direct benefit parameters of each agent in the environment; The total reward of each agent system includes the environmental reward (the operating income of the system itself) and the intrinsic reward of causal influence. The calculation formula is as follows: (1) Total reward for photovoltaic system: (12) (2) Total rewards for energy storage systems: (13) (3) Total rewards for energy storage systems: (14) in, , and Respectively represent the direct benefits of each system in the environment; To adjust the weight coefficient of causal influence rewards.

[0077] Step 6: Optimize the strategy based on the total reward function of each agent and output the corresponding scheduling strategy; In order to achieve the best synergy among multiple agents in the entire system, the overall reward function should consider the causal relationship between photovoltaic power generation, energy storage and charging pile systems. It can be defined as the weighted sum of the total rewards of each system: (15) Specifically, the overall reward function comprehensively considers the environmental benefits and causal impacts of the three systems, and promotes effective collaboration among agents through causal impact rewards.

[0078] The overall optimization objective function of the system. It combines the optimization objectives of each agent, that is, each agent optimizes its strategy by maximizing the total reward of its strategy: (16) In this objective function, It is the total reward of all agents (photovoltaic system, energy storage system and charging pile system) at time step t, which combines environmental rewards and causal influence rewards. Specifically, π represents the policy function of different systems; T: the maximum number of time steps, which indicates the time range for optimizing the objective function calculation, that is, the cumulative benefits of the agent in a time interval; γ: the discount factor, which controls the degree of attenuation of future rewards. The value range of the discount factor γ is 0≤γ≤1. By maximizing , which can achieve the coordinated optimization of the entire system.

[0079] (17) Electricity cost of the whole system As shown in formula (17). Specifically, It represents the cost of electricity purchased by the energy storage system from the grid minus the revenue from electricity sold by the energy storage system to the grid; Represents the revenue from the electricity sold by the PV system to the grid; It represents the cost of the charging station purchasing electricity from the grid minus the income from selling electricity to the grid; Indicates the real-time electricity price.

[0080] For each agent i, the objective function of PPO is used for optimization: (18) Where: represents the policy update objective function (used to optimize the performance of the current policy), is the probability ratio of the current strategy to the old strategy, that is, the current strategy in state Next select action The ratio of the probability of choosing this action to the probability of choosing this action by the old strategy; is the advantage estimate at time step t, measuring the current action superiority relative to the average; This is to prevent shearing operations introduced by large updates. is a hyperparameter used to control the magnitude of policy updates. By limiting the magnitude of updates, we can avoid excessive policy updates and ensure the stability of the training process.

[0081] Step 7: Execute corresponding control actions according to the scheduling strategy to achieve the scheduling of the electric energy of the entire park.

[0082] By optimizing the strategies of the three intelligent agents in parallel, the system performs corresponding control actions, thereby effectively improving the collaborative efficiency between the photovoltaic power generation system, the energy storage system and the vehicle charging pile system, and ultimately achieving coordinated optimization of the energy management system and maximizing the overall benefits.

[0083] like Figure 3 As shown, an embodiment of the present invention provides a comprehensive park power energy dispatching system for heterogeneous multi-agent collaboration, including a heterogeneous agent construction module, wherein the heterogeneous agent construction module is connected to an agent causal graph model module, an intrinsic reward design module, and a total reward calculation and strategy optimization module; the agent causal graph model module is connected to the intrinsic reward design module via a causal relationship sampling and conditional mutual information calculation module; Heterogeneous agent building module, which is used to customize corresponding heterogeneous agents by analyzing the characteristics and operating conditions of each energy system; In the heterogeneous agent building module, the present invention pays special attention to the heterogeneous characteristics between different energy systems such as photovoltaics, energy storage, and electric vehicle charging piles, and builds a special heterogeneous agent to represent these systems. At the same time, factors such as load and municipal electricity prices are regarded as the state of the environment.

[0084] The agent causal graph model module is used to build a model of context-dependent causal relationships between agents using causal relationship graphs and conditional mutual information quantification methods; that is, a multi-agent causal graph model; The purpose of the agent causal graph model module is to help agents consider the behavior of other agents and their overall impact on the system during decision-making by modeling the causal relationship between agents. It relies on the causal graph to represent and reason about causal dependencies, thereby guiding the decision-making process of each agent in the multi-agent system.

[0085] The causal relationship sampling and conditional mutual information calculation module is used to collect causal interventions and calculate conditional mutual information between intelligent agents. By constructing a causal graph, the potential causal relationships between various intelligent agents are clarified, helping to identify which intelligent agents' actions and states may have causal influences, so that in the subsequent causal intervention sampling and conditional mutual information calculation modules, the impact of these key nodes on other systems can be analyzed intensively.

[0086] The intrinsic reward design module is used to measure the causal relationship between each intelligent agent and build the intrinsic reward mechanism of each intelligent agent; Based on the calculated causal influence strength, an intrinsic reward mechanism is constructed to encourage each system to choose actions that have a positive impact on other systems during the collaborative optimization process.

[0087] The total reward calculation and strategy optimization module is used to design the total reward function of each agent based on the intrinsic reward mechanism of each agent and the direct benefit parameters of each agent in the environment; optimize the strategy based on the total reward function of each agent and output the corresponding scheduling strategy.

[0088] The system also includes a scheduling execution module, which is used to execute corresponding control actions according to the scheduling strategy to realize the scheduling of the electric energy of the entire park.

[0089] It should be noted that the heterogeneous agent building module is specifically used to build a photovoltaic power generation system agent; the photovoltaic power generation system agent generates electricity through sunshine intensity and time; maximizes electricity generation by setting the state space, action space and environmental reward function of the photovoltaic power generation system agent; builds an energy storage system agent; the energy storage system agent optimizes the charging and discharging strategy according to the peak and valley electricity prices of the power grid, load demand and the charging status of the battery; sets the state space, action space and environmental reward function of the energy storage system agent; builds a car charging pile system agent; the car charging pile system agent determines the source of electricity and the charging or discharging operation according to the charging needs of the electric vehicle, the power price of the power grid and the charging status of the energy storage battery; sets the state space, action space and environmental reward function of the car charging pile system agent.

[0090] In the multi-agent causal graph model, a node is used to represent the state of each agent i at time t. ,action and the state at time t+1 ; The state node describes the agent's perception of the current environment, and the action node describes the agent's decision-making behavior; the edge is used to represent the causal relationship between multiple heterogeneous agents, that is, the state or action of a certain agent affects the state or action of other agents, and the direction of the edge indicates the flow of the causal relationship.

[0091] The intrinsic reward design module constructs an intrinsic reward mechanism based on the calculated causal influence intensity. The intrinsic reward mechanism specifically includes: when the action of the photovoltaic system has a positive causal impact on the energy storage system and the charging pile system and the causal influence intensity is greater than the first set value, the photovoltaic system is given an intrinsic reward to maximize the incentives of the photovoltaic system to other systems; when the action of the energy storage system has a positive impact on the demand satisfaction rate of the charging pile system and the causal influence intensity is greater than the second set value, the energy storage system is allocated an intrinsic reward to select the charging and discharging strategy that is most beneficial to the charging pile system; when the behavior of the charging pile system has a positive causal impact on the load management of the power grid or the energy storage system, the charging pile system is provided with an intrinsic reward to optimize its support for the system load.

[0092] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0093] An embodiment of the integrated park power energy dispatching system for heterogeneous multi-agent collaboration provided in an embodiment of the present invention belongs to the same inventive concept as the integrated park power energy dispatching method for heterogeneous multi-agent collaboration in the above-mentioned embodiments. For details not described in detail in the embodiment of the integrated park power energy dispatching system for heterogeneous multi-agent collaboration, reference can be made to the embodiment of the integrated park power energy dispatching method for heterogeneous multi-agent collaboration mentioned above.

[0094] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A comprehensive park power energy dispatching method with heterogeneous multi-agent collaboration, characterized in that: The steps include: Customize corresponding heterogeneous intelligent agents by analyzing the characteristics and operating conditions of each energy system; Using causal relationship graphs and conditional mutual information quantification methods, a model of context-dependent causal relationships between agents is established; that is, a multi-agent causal graph model; Carry out causal intervention collection and conditional mutual information calculation between intelligent agents; Measure the causal relationship between each agent and build the intrinsic reward mechanism of each agent; Design the total reward function for each agent based on the intrinsic reward mechanism of each agent and the direct benefit parameters of each agent in the environment; Optimize the strategy based on the total reward function of each agent and output the corresponding scheduling strategy; According to the scheduling strategy, corresponding control actions are executed to realize the scheduling of the electric energy of the entire park.

2. The method for comprehensive park power energy dispatching based on heterogeneous multi-agent collaboration according to claim 1 is characterized in that: The steps of customizing the corresponding heterogeneous intelligent agents by analyzing the characteristics and operating conditions of each energy system include: Constructing a photovoltaic power generation system agent; the photovoltaic power generation system agent generates electricity according to the intensity and time of sunshine; maximizing the power generation by setting the state space, action space and environmental reward function of the photovoltaic power generation system agent; Constructing an energy storage system intelligent agent; the energy storage system intelligent agent optimizes the charging and discharging strategy according to the peak and valley electricity prices of the power grid, load demand and the charging state of the battery; setting the state space, action space and environmental reward function of the energy storage system intelligent agent; Construct an intelligent agent for a car charging pile system; the intelligent agent for the car charging pile system determines the source of electricity and the charging or discharging operation according to the charging demand of the electric vehicle, the electricity price of the power grid and the charging status of the energy storage battery; and sets the state space, action space and environmental reward function of the intelligent agent for the car charging pile system.

3. The method for comprehensive park power energy dispatching based on heterogeneous multi-agent collaboration according to claim 2 is characterized in that: The state space of the photovoltaic system agent is: Action space of photovoltaic system agent To adjust the power generation of the photovoltaic system according to the real-time solar radiation intensity and weather conditions to maximize power output; Environmental reward function for photovoltaic system agents It is based on the amount of electricity it generates and its reliance on the grid; In the formula, is the solar radiation intensity at the current time t, The current power generation of the photovoltaic system, Light exposure time; Indicates the possible shielding of solar radiation, 0 means complete shielding, and 1 means no shielding; is the grid electricity price at time t; is the minimum output power of photovoltaic power generation, is the maximum rated power of the photovoltaic system, is the output power of photovoltaic power generation; is the power generated by the photovoltaic system at time t.

4. The method for comprehensive park power energy dispatching based on heterogeneous multi-agent collaboration according to claim 3 is characterized in that: In the multi-agent causal graph model, a node is used to represent the state of each agent i at time t. ,action and the state at time t+1 ; The state node describes the agent's perception of the current environment, and the action node describes the agent's decision-making behavior; Edges are used to represent the causal relationship between multiple heterogeneous agents, that is, the state or action of a certain agent affects the state or action of other agents, and the direction of the edge indicates the flow of the causal relationship.

5. The method for comprehensive park power energy dispatching based on heterogeneous multi-agent collaboration according to claim 4 is characterized in that: The steps of measuring the causal relationship between each agent and constructing the intrinsic reward mechanism of each agent include: constructing the intrinsic reward mechanism based on the calculated causal influence strength, specifically including: When the action of the photovoltaic system has a positive causal impact on the energy storage system and the charging pile system and the causal impact intensity is greater than the first set value, the photovoltaic system is given an intrinsic reward to maximize the incentives of the photovoltaic system to other systems; When the action of the energy storage system has a positive impact on the demand satisfaction rate of the charging pile system and the causal influence intensity is greater than the second set value, an intrinsic reward is allocated to the energy storage system to select the charging and discharging strategy that is most beneficial to the charging pile system; When the behavior of the charging pile system has a positive causal impact on the load management of the power grid or energy storage system, it provides intrinsic rewards for the charging pile system to optimize its support for the system load.

6. The method for comprehensive park power energy dispatching based on heterogeneous multi-agent collaboration according to claim 5 is characterized in that: The steps of designing the total reward function of each agent based on the intrinsic reward mechanism of each agent and the direct benefit parameters of each agent in the environment specifically include: The total reward function of the photovoltaic system: The total reward function of the energy storage system: The total reward function of the energy storage system: in, , and Respectively represent the direct benefits of each system in the environment; To adjust the weight coefficient of causal influence rewards.

7. The method for comprehensive park power energy dispatching based on heterogeneous multi-agent collaboration according to claim 6 is characterized in that: The steps of optimizing the strategy based on the total reward function of each agent and outputting the corresponding scheduling strategy include: Design the system total reward function based on the total reward function of each agent; By maximizing the total reward of each agent, the PPO objective function is used to perform policy optimization and output the optimized scheduling strategy.

8. The method for comprehensive park power energy dispatching based on heterogeneous multi-agent collaboration according to claim 7 is characterized in that: By maximizing the total reward of each agent, the overall optimization objective function of the system is used to optimize the strategy and output the optimized scheduling strategy using the objective function of PPO. : For each agent i, the objective function of PPO is used for optimization; Strategy update objective function: Where: is the total reward of all agents at time step t, π represents the policy function of different systems; T is the maximum number of time steps, γ is the discount factor used to control the decay degree of future rewards, and the value range is 0≤γ≤1; represents the strategy update objective function, is the probability ratio of the current strategy to the old strategy, that is, the current strategy in state Next select action The ratio of the probability of choosing this action to the probability of choosing this action by the old strategy; is the advantage estimate at time step t, For the shear operation, is a hyperparameter used to control the magnitude of the policy update, It is the electricity cost of the entire system.

9. A comprehensive park power energy dispatching system with heterogeneous multi-agent collaboration, characterized in that: It includes a heterogeneous agent building module, wherein the heterogeneous agent building module is connected to an agent causal graph model module, an intrinsic reward design module, and a total reward calculation and strategy optimization module; The agent causal graph model module is connected to the intrinsic reward design module through the causal relationship sampling and conditional mutual information calculation module; Heterogeneous agent building module, which is used to customize corresponding heterogeneous agents by analyzing the characteristics and operating conditions of each energy system; The agent causal graph model module is used to build a model of context-dependent causal relationships between agents using causal relationship graphs and conditional mutual information quantification methods; that is, a multi-agent causal graph model; Causal relationship sampling and conditional mutual information calculation module, used for causal intervention collection and conditional mutual information calculation between intelligent agents; The intrinsic reward design module is used to measure the causal relationship between each agent and build the intrinsic reward mechanism of each agent; The total reward calculation and strategy optimization module is used to design the total reward function of each agent based on the intrinsic reward mechanism of each agent and the direct benefit parameters of each agent in the environment; optimize the strategy based on the total reward function of each agent and output the corresponding scheduling strategy.

10. The method for comprehensive park power energy dispatching based on heterogeneous multi-agent collaboration according to claim 9 is characterized in that: The system also includes a scheduling execution module, which is used to execute corresponding control actions according to the scheduling strategy to realize the scheduling of the electric energy of the entire park.

Images