A distributed multi-agent microgrid energy management method based on Q-Learning algorithm
By applying the Q-Learning algorithm in the microgrid, each agent optimizes the strategy to solve the reliability and economic problems of the microgrid, achieving more efficient energy management, improving the benefits and user costs of distributed energy, and reducing the cost of the best The dependence of the main grid.
Patent Information
- Application Number
- CN202210131686.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-14
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
After the microgrid faces a large number of distributed power supplies and energy storage connections, the variety of energy types, two-way trends and renewable energy uncertainty lead to reliability, stability and economic problems, and there are benefits distribution and costs between energy suppliers and users. Optimization Challenge.
Using the distributed multi-agent microgrid energy management method based on the Q-Learning algorithm, the state space and action space are constructed through the environmental data acquisition of wind power farms and photovoltaic power farms, and the action selection strategy and market pricing strategy are designed. Each agent is based on Instant reward updates the Q table and optimizes its own strategies to maximize profits and minimize costs.
It improves the benefits of distributed energy agents, reduces the cost of user agents, reduces the dependence of microgrids on the main grid, and improves the overall benefits and reliability of microgrids.
Smart Images

Figure CN114498750B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of microgrid energy management, and in particular to a distributed multi-agent microgrid energy management method based on a Q-Learning algorithm. Background Art
[0002] In recent years, the global policy focus on environmental issues has been to increase low-carbon energy. Compared with traditional power grids, microgrids can make extensive use of renewable energy and distributed energy, guide consumers to participate in the operation of the power system, and achieve optimal energy management. Microgrids are independent controllable systems composed of a variety of distributed power sources, energy storage, and loads, which can flexibly switch between grid-connected mode and island mode. If the energy of a microgrid is managed, the reliability, stability, and economy of its power supply can be effectively improved. Therefore, the energy management strategy of microgrids is one of the core issues in the field of microgrid research.
[0003] At present, the energy management methods of microgrids can be divided into two categories, namely centralized methods and distributed methods. The centralized method has the advantages of simple and efficient operation and rapid response, but it also has certain limitations. The central controller needs to process a large amount of information. Once a failure occurs, the entire microgrid system will collapse. In addition, once the centralized control framework of the microgrid is established, it cannot be changed. If new distributed power sources or loads are connected, it must be torn down and rebuilt. Because each subsystem has a certain degree of independence and can work together using the communication network within the system, the distributed method has become a research hotspot in recent years.
[0004] Due to the access to a large number of distributed power sources, energy storage and loads, the energy management of microgrids faces a series of challenges, including diverse energy types, bidirectional power flow, uncertainty of renewable energy, etc., which causes problems in the reliability, stability and economy of microgrids. At the same time, there are also problems of benefit distribution and cost optimization between energy suppliers and users. In order to solve the above problems, the present invention proposes a distributed multi-agent microgrid energy management method based on Q-Learning algorithm. Summary of the invention
[0005] The purpose of the present invention is to provide a distributed multi-agent microgrid energy management method based on the Q-Learning algorithm. The distributed power sources and users in the microgrid use the Q-Learning algorithm to optimize their own strategies to seek maximum profit and minimum cost respectively.
[0006] To achieve the above purpose, the technical solution provided by the present invention is:
[0007] A distributed multi-agent microgrid energy management method based on Q-Learning algorithm, characterized by comprising the following steps:
[0008] Step 1, obtaining three sets of data, namely, the ambient wind speed of the wind farm, the ambient temperature of the photovoltaic power plant, and the light intensity: recording the data of the ambient wind speed, ambient temperature, and light intensity from the data acquisition system of the power plant at a certain time interval, and using them as input data of the wind power generation system and the photovoltaic power generation system;
[0009] Step 2: Perform sensitivity analysis on learning parameters and select appropriate learning parameters;
[0010] Step 3: Construct the state space and action space based on Q-Learning to initialize the Q table as the learning model of each agent;
[0011] Step 4, based on the data obtained in step 1, calculate the output power of the wind power generation system and the photovoltaic power generation system, and use it as the output power data of the renewable energy agent;
[0012] Step 5: Design an action selection strategy, and each agent selects an action based on this strategy: the renewable energy agent selects a bid according to the designed strategy, other distributed energy agents select output power and bids according to the designed strategy, and the user agent selects load reduction according to the designed strategy;
[0013] Step 6: Based on the quotations and output powers of the distributed energy agents obtained in Steps 4 and 5, multiple quotation blocks are formed, and the structure of each quotation block is {output power, quotation}. At the same time, after the user agent chooses to reduce the load, the remaining load demand is obtained, and the quotation block and the remaining load demand are submitted to the upper-level energy management agent;
[0014] Step 7, designing a market pricing strategy. The energy management agent receives the quotation block from the distributed energy agent and the remaining load demand from the user agent, and specifies the market electricity price according to the designed pricing strategy while ensuring the balance between supply and demand.
[0015] Step 8: Calculate the immediate reward of each agent: the immediate reward of the renewable energy agent is the profit from electricity sales, the immediate reward of other distributed energy agents needs to consider the power generation cost in addition to the profit from electricity sales, and the immediate reward of the user agent is the total cost of meeting the load demand;
[0016] Step 9: Based on the immediate reward obtained in step 8, each agent updates the Q table according to formula (16);
[0017] Step 10, determine whether the Q table converges. If not, return to step 4; if converged, the algorithm ends.
[0018] Based on the above scheme, the microgrid is a grid-connected microgrid composed of a wind power generation system, a photovoltaic power generation system, a micro gas turbine, a fuel cell, a diesel generator, an energy storage system and a load that can be reduced, and each component in the microgrid is regarded as an intelligent entity with learning ability and can change its own strategy according to the environment.
[0019] Based on the above scheme, the parameter α∈(0,1] described in step 2 represents the learning rate. The larger the α, the more greedy the agent is, and it cares more about immediate rewards when choosing actions. γ∈(0,1] represents the discount rate. The larger the γ, the more the agent cares about future rewards.
[0020] Based on the above scheme, the formula for calculating the output power of the wind power generation system and the photovoltaic power generation system in step 4 is as follows:
[0021] 1) The output power of the wind power generation system is as follows:
[0022]
[0023] Among them, P wt It represents the output power of the wind power generation system under a certain wind speed. represents the rated power of the wind power generation system; v represents the ambient wind speed, v ci 、v r 、v co They respectively represent the cut-in wind speed, rated wind speed and cut-out wind speed when the fan is running.
[0024] 2) The output power of the photovoltaic power generation system is as follows:
[0025] P pv =FF×V×I (2)
[0026] T c =T a +s×(N OT -20) / 0.8 (3)
[0027] I=s×[I SC +K I ×(T c -25)] (4)
[0028] V=V oc -K v ×T c (5)
[0029]
[0030] Among them, T a is the ambient temperature, T c represents the temperature of the photovoltaic cell, s represents the light intensity, NOT is the standard operating temperature of photovoltaic cells, I SC 、V oc Respectively represent the short-circuit current and open-circuit voltage of the photovoltaic cell, I MPP 、V MPP They represent the maximum power point current and maximum power point voltage of the photovoltaic cell, FF represents the fill factor, and K I , K v are the current coefficient and the voltage coefficient.
[0031] Based on the above scheme, the action selection strategy described in step 5 is the ε-greedy strategy, and its steps are as follows:
[0032] 1) Select the initial value of ε∈(0,1];
[0033] 2) Select a random number rand∈(0,1];
[0034] 3) If rand < ε, then a random action is selected from the action set in that state, otherwise, the action with the highest Q value is selected in the action set.
[0035] Based on the above scheme, the method for the energy management agent in step 7 to determine the market electricity price according to the market pricing strategy is as follows:
[0036] The energy management agent receives the load demand L submitted by the user agent and the output power data P of the distributed energy agent. i and quote p i The formed quotation block (P i ,p i ), and these quotation blocks are divided into price p i Sort from low to high, then buy from low to high until the load demand is met. The price corresponding to the last quotation block is the market electricity price MCP. If the energy provided by the distributed energy agent is greater than the load demand, the remaining energy will be sold to the main grid; if the energy provided by the distributed energy agent cannot meet the load demand, the energy management agent will make up for it from the main grid.
[0037] The energy management agent needs to ensure the balance between supply and demand, as shown in formula (7):
[0038]
[0039] Among them, P demand,t Represents the load requirement submitted by the user agent, They represent the energy sold by each distributed energy agent to the microgrid, P ess,t represents the charging / discharging power of the energy storage agent, P maingrid,t Represents energy purchased from the main grid.
[0040] Based on the above scheme, the method for calculating the agent's immediate reward in step 8 is as follows:
[0041] Assume that the total demand of a user agent at time t is LD i,t , the market price is MCP t The load that the agent chooses to reduce is LC i,t , therefore, the total cost of the user agent is the sum of the remaining load demand cost and the penalty cost of reducing the load:
[0042] Cost i,t =MCP t ×(LD i,t -LC i,t )+PF×LC i,t (8)
[0043] Among them, PF is the penalty coefficient, and the reduction amount LC i,t The following constraints must be met:
[0044] 0≤LC i,t ≤LD i,t (9)
[0045] The calculation of the incentive for renewable energy agents is shown in formula (10)-formula (11):
[0046]
[0047]
[0048] Among them, MP t is the electricity purchase price of the main power grid, and Energy sold to microgrids by renewable energy agents, and Energy sold by renewable energy agents to the main grid.
[0049] The rewards of other distributed agents are defined as the profit from their sale of electricity minus the cost of generation, as shown in Formula (12)-Formula (14):
[0050]
[0051]
[0052]
[0053] Cost diesel,t Cost mt,t and Cost fc,t is the power generation cost of each agent, and is the energy sold by each agent to the microgrid, and It is the energy sold by each agent to the main grid, MP t It is the price of electricity purchased by the main power grid.
[0054] The reward of the energy storage agent is defined as the profit from electricity sales minus the charging cost, as shown in formula (15):
[0055] Reward ess,t =MCP t ×P dch,t -MCP t ×P ch,t (15)
[0056] Where P dch,t is the discharge power of the energy storage agent, P ch,t is the charging power of the energy storage agent.
[0057] Based on the above solution, the updated Q table described in step 9 is shown in formula (16):
[0058]
[0059] Where α∈(0,1] is the learning rate, γ∈(0,1] is the discount rate, and r t+1 is the agent in state s at time t t Take action a t Instant rewards received. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] The present invention has the following accompanying drawings:
[0061] Figure 1 It is a flow chart of a distributed multi-agent microgrid energy management method based on Q-Learning algorithm described in the present invention;
[0062] Figure 2 is a structural schematic diagram of the microgrid of the present invention;
[0063] Figure 3 is the daily cost variation of the user agents in the microgrid under the four configurations;
[0064] Figure 4 is the daily profit variation of the wind power system agent in the microgrid under four configurations;
[0065] Figure 5 is the daily profit variation of the fuel cell agent in the microgrid under four configurations;
[0066] Figure 6is the daily profit variation of the energy storage agent in the microgrid under the four configurations;
[0067] Figure 7 It is the power interaction between the microgrid and the main grid under four configurations. DETAILED DESCRIPTION
[0068] The following is combined with Figure 1-7 The present invention is described in further detail.
[0069] The flow chart of the present invention is as follows Figure 1 As shown,
[0070] Step 1, obtaining three sets of data, namely, the ambient wind speed of the wind farm, the ambient temperature of the photovoltaic power plant, and the light intensity: recording the data of the ambient wind speed, ambient temperature, and light intensity from the data acquisition system of the power plant at a certain time interval, and using them as input data of the wind power generation system and the photovoltaic power generation system;
[0071] Step 2: Perform sensitivity analysis on learning parameters and select appropriate learning parameters;
[0072] Step 3: Construct the state space and action space based on Q-Learning to initialize the Q table as the learning model of each agent;
[0073] Step 4, based on the data obtained in step 1, calculate the output power of the wind power generation system and the photovoltaic power generation system, and use it as the output power data of the renewable energy agent;
[0074] Step 5: Design an action selection strategy, and each agent selects an action based on this strategy: the renewable energy agent selects a bid according to the designed strategy, other distributed energy agents select output power and bids according to the designed strategy, and the user agent selects load reduction according to the designed strategy;
[0075] Step 6: Based on the quotations and output powers of the distributed energy agents obtained in Steps 4 and 5, multiple quotation blocks are formed, and the structure of each quotation block is {output power, quotation}. At the same time, after the user agent chooses to reduce the load, the remaining load demand is obtained, and the quotation block and the remaining load demand are submitted to the upper-level energy management agent;
[0076] Step 7, designing a market pricing strategy. The energy management agent receives the quotation block from the distributed energy agent and the remaining load demand from the user agent, and specifies the market electricity price according to the designed pricing strategy while ensuring the balance between supply and demand.
[0077] Step 8: Calculate the immediate reward of each agent: the immediate reward of the renewable energy agent is the profit from electricity sales, the immediate reward of other distributed energy agents needs to consider the power generation cost in addition to the profit from electricity sales, and the immediate reward of the user agent is the total cost of meeting the load demand;
[0078] Step 9: Based on the immediate reward obtained in step 8, each agent updates the Q table according to formula (16);
[0079] Step 10, determine whether the Q table converges. If not, return to step 4; if converged, the algorithm ends.
[0080] In order to demonstrate the superiority of the distributed multi-agent microgrid energy management method based on the Q-Learning algorithm proposed in the present invention in seeking the optimal strategy for each distributed power source and user agent, the present invention simulates the behavior of the agent within one month under four configurations. The four configurations are:
[0081] 1) All agents have no learning ability (Nobody Learn, NL for short);
[0082] 2) Only user agents have learning capabilities (Customers Learn, hereinafter referred to as CL);
[0083] 3) Only distributed energy agents have learning capabilities (DER Learn, hereinafter referred to as DL);
[0084] 4) All agents have the ability to learn (ALL Learn, hereinafter referred to as AL).
[0085] Based on the above scheme, the microgrid is a grid-connected microgrid composed of a wind power generation system, a photovoltaic power generation system, a micro gas turbine, a fuel cell, a diesel generator, an energy storage system and a load that can be reduced. Each component in the microgrid is regarded as an intelligent entity with learning ability and can change its own strategy according to the environment. The structural diagram of the microgrid is as follows: Figure 2 shown.
[0086] Based on the above scheme, the parameter α∈(0,1] described in step 2 represents the learning rate. The larger the α, the more greedy the agent is, and it cares more about immediate rewards when choosing actions. γ∈(0,1] represents the discount rate. The larger the γ, the more the agent cares about future rewards.
[0087] Based on the above scheme, the formula for calculating the output power of the wind power generation system and the photovoltaic power generation system in step 4 is as follows:
[0088] 1) The output power of the wind power generation system is as follows:
[0089]
[0090] Among them, P wt It represents the output power of the wind power generation system under a certain wind speed. represents the rated power of the wind power generation system; v represents the ambient wind speed, v ci 、v r 、v co They respectively represent the cut-in wind speed, rated wind speed and cut-out wind speed when the fan is running.
[0091] 2) The output power of the photovoltaic power generation system is as follows:
[0092] P pv =FF×V×I (2)
[0093] T c =T a +s×(N OT -20) / 0.8 (3)
[0094] I=s×[I SC +K I ×(T c -25)] (4)
[0095] V=V oc -K v ×T c (5)
[0096]
[0097] Among them, T a is the ambient temperature, T c represents the temperature of the photovoltaic cell, s represents the light intensity, N OT is the standard operating temperature of photovoltaic cells, I SC 、V oc Respectively represent the short-circuit current and open-circuit voltage of the photovoltaic cell, I MPP 、V MPP They represent the maximum power point current and maximum power point voltage of the photovoltaic cell, FF represents the fill factor, and K I , K v are the current coefficient and the voltage coefficient.
[0098] Based on the above scheme, the action selection strategy described in step 5 is the ε-greedy strategy, and its steps are as follows:
[0099] 1) Select the initial value of ε∈(0,1];
[0100] 2) Select a random number rand∈(0,1];
[0101] 3) If rand < ε, then a random action is selected from the action set in that state, otherwise, the action with the highest Q value is selected in the action set.
[0102] Based on the above scheme, the method for the energy management agent in step 7 to determine the market electricity price according to the market pricing strategy is as follows:
[0103] The energy management agent receives the load demand L submitted by the user agent and the output power data P of the distributed energy agent. i and quote p i The formed quotation block (P i ,p i ), and these quotation blocks are divided into price p i Sort from low to high, then buy from low to high until the load demand is met. The price corresponding to the last quotation block is the market electricity price MCP. If the energy provided by the distributed energy agent is greater than the load demand, the remaining energy will be sold to the main grid; if the energy provided by the distributed energy agent cannot meet the load demand, the energy management agent will supplement it from the main grid. The power interaction between the microgrid and the main grid is as follows Figure 7 shown.
[0104] The energy management agent needs to ensure the balance between supply and demand, as shown in formula (7):
[0105]
[0106] Among them, P demand,t Represents the load requirement submitted by the user agent, They represent the energy sold by each distributed energy agent to the microgrid, P ess,t represents the charging / discharging power of the energy storage agent, P maingrid,t Represents energy purchased from the main grid.
[0107] Based on the above scheme, the method for calculating the agent's instant reward in step 8 is as follows:
[0108] Assume that the total demand of a user agent at time t is LD i,t , the market price is MCP t The load that the agent chooses to reduce is LC i,t , therefore, the total cost of the user agent is the sum of the remaining load demand cost and the penalty cost of reducing the load:
[0109] Cost i,t =MCP t ×(LD i,t -LC i,t )+PF×LC i,t (8)
[0110] Among them, PF is the penalty coefficient, and the reduction amount LC i,t The following constraints must be met:
[0111] 0≤LC i,t ≤LD i,t (9) Figure 3 The daily cost changes of the user agent under four configurations.
[0112] The calculation of the incentive for renewable energy agents is shown in formula (10)-formula (11):
[0113]
[0114]
[0115] Among them, MP t is the electricity purchase price of the main power grid, and Energy sold to microgrids by renewable energy agents, and The energy sold by the renewable energy agent to the main grid. The present invention selects the daily profit of the wind power generation system as a reference for the renewable energy agent, such as Figure 4 shown.
[0116] The rewards of other distributed agents are defined as the profit from their sale of electricity minus the cost of generation, as shown in Formula (12)-Formula (14):
[0117]
[0118]
[0119]
[0120] Cost diesel,t Cost mt,t and Cost fc,t is the power generation cost of each agent, and is the energy sold by each agent to the microgrid, and It is the energy sold by each agent to the main grid, MP t is the main grid electricity purchase price. The present invention selects the daily profit of fuel cells as a reference for other distributed energy agents, such as Figure 5 shown.
[0121] The reward of the energy storage agent is defined as the profit from electricity sales minus the charging cost, as shown in formula (15):
[0122] Rewardess,t =MCP t ×P dch,t -MCP t ×P ch,t (15)
[0123] Where P dch,t is the discharge power of the energy storage agent, P ch,t is the charging power of the energy storage agent. Figure 6 Shown is the change in daily profit of the energy storage agent under four configurations.
[0124] Based on the above solution, the updated Q table described in step 9 is shown in formula (16):
[0125]
[0126] Where α∈(0,1] is the learning rate, γ∈(0,1] is the discount rate, and r t+1 is the agent in state s at time t t Take action a t Instant rewards received.
[0127] Combined with case analysis
[0128] The simulation environment of this experiment is MATLAB R2021a. The behavior of the agent in one month is simulated under four configurations. The four configurations are:
[0129] 1) All agents have no learning ability (Nobody Learn, NL for short);
[0130] 2) Only user agents have learning capabilities (Customers Learn, hereinafter referred to as CL);
[0131] 3) Only distributed energy agents have learning capabilities (DER Learn, hereinafter referred to as DL);
[0132] 4) All agents have the ability to learn (ALL Learn, hereinafter referred to as AL).
[0133] Take renewable energy agents as an example, Figure 4As shown, the average profit of the wind power system agent under the DL configuration is at least 20% higher than that of NL and CL, and the average profit under the AL configuration is 357.9¢ and 1178.3¢ higher than that of NL and CL. Because the user agent learned to reduce consumption when the market electricity price was high, the average profit of the wind power system agent under the AL configuration decreased by 14% compared with the DL configuration. When the wind power system agent had no learning ability, the user agent under the CL configuration reduced demand, resulting in a decrease in the average profit of the wind power system agent. Due to the uncertainty of wind, the average profit of the wind power system agent fluctuated significantly every day, but under the DL and AL configurations, it learned to explore the best strategy, and the fluctuation was significantly lower than that of NL and CL.
[0134] Beneficial effects of the present invention:
[0135] It can be seen from the experimental results shown in the accompanying drawings that the distributed multi-agent microgrid energy management method based on the Q-Learning algorithm proposed in the present invention has an obvious effect on the strategy optimization of each agent, greatly improves the benefits of distributed energy agents, reduces the costs of user agents, and also reduces the dependence of the microgrid on the main grid, thereby improving the overall benefits of the microgrid.
[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
[0137] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.
Claims
1. A distributed multi-agent microgrid energy management method based on Q-Learning algorithm, It is characterized in that The following steps are involved: Step 1, obtaining three sets of data, namely, the ambient wind speed of the wind farm, the ambient temperature of the photovoltaic power plant, and the light intensity: recording the data of the ambient wind speed, ambient temperature, and light intensity from the data acquisition system of the power plant at a certain time interval, and using them as input data of the wind power generation system and the photovoltaic power generation system; Step 2: Perform sensitivity analysis on learning parameters and select appropriate learning parameters; Step 3: Construct the state space and action space based on Q-Learning to initialize the Q table as the learning model of each agent; Step 4, based on the data obtained in step 1, calculate the output power of the wind power generation system and the photovoltaic power generation system, and use it as the output power data of the renewable energy agent; Step 5: Design an action selection strategy, and each agent selects an action based on this strategy: the renewable energy agent selects a bid according to the designed strategy, other distributed energy agents select output power and bids according to the designed strategy, and the user agent selects load reduction according to the designed strategy; Step 6: Based on the quotations and output powers of the distributed energy agents obtained in Steps 4 and 5, multiple quotation blocks are formed, and the structure of each quotation block is {output power, quotation}. At the same time, after the user agent chooses to reduce the load, the remaining load demand is obtained, and the quotation block and the remaining load demand are submitted to the upper-level energy management agent; Step 7, designing a market pricing strategy. The energy management agent receives the quotation block from the distributed energy agent and the remaining load demand from the user agent, and specifies the market electricity price according to the designed pricing strategy while ensuring the balance between supply and demand. Step 8: Calculate the immediate reward of each agent: the immediate reward of the renewable energy agent is the profit from electricity sales, the immediate reward of other distributed energy agents needs to consider the power generation cost in addition to the profit from electricity sales, and the immediate reward of the user agent is the total cost of meeting the load demand; Step 9: Based on the immediate reward obtained in step 8, each agent updates the Q table according to formula (16); Step 10, determine whether the Q table converges. If not, return to step 4; if converged, the algorithm ends.
2. A distributed multi-agent microgrid energy management method based on Q-Learning algorithm as claimed in claim 1, It is characterized in that The microgrid is a grid-connected microgrid composed of a wind power generation system, a photovoltaic power generation system, a micro gas turbine, a fuel cell, a diesel generator, an energy storage system and a load that can be reduced, and each component in the microgrid is regarded as an intelligent entity with learning ability and can change its own strategy according to the environment.
3. A distributed multi-agent microgrid energy management method based on Q-Learning algorithm as claimed in claim 1, It is characterized in that The parameter α∈(0,1] described in step 2 represents the learning rate. The larger the α, the more greedy the agent is, and it cares more about the immediate reward when choosing an action. γ∈(0,1] represents the discount rate. The larger the γ, the more the agent cares about future rewards.
4. A distributed multi-agent microgrid energy management method based on Q-Learning algorithm as claimed in claim 1, It is characterized in that The formula for calculating the output power of the wind power generation system and the photovoltaic power generation system in step 4 is as follows: 1) The output power of the wind power generation system is as follows: Among them, P wt It represents the output power of the wind power generation system under a certain wind speed. represents the rated power of the wind power generation system; v represents the ambient wind speed, v ci 、v r 、v co Respectively represent the cut-in wind speed, rated wind speed and cut-out wind speed when the fan is running; 2) The output power of the photovoltaic power generation system is as follows: P pv =FF×V×I (2) T c =T a +s×(N OT -20) / 0.8 (3) I=s×[I SC +K I ×(T c -25)] (4) V=V oc -K v ×T c (5) Among them, T a is the ambient temperature, T c represents the temperature of the photovoltaic cell, s represents the light intensity, N OT is the standard operating temperature of photovoltaic cells, I SC 、V oc Respectively represent the short-circuit current and open-circuit voltage of the photovoltaic cell, I MPP 、V MPP They represent the maximum power point current and maximum power point voltage of the photovoltaic cell, FF represents the fill factor, and K I , K v are the current coefficient and the voltage coefficient.
5. A distributed multi-agent microgrid energy management method based on Q-Learning algorithm as claimed in claim 1, It is characterized in that The action selection strategy described in step 5 is the ε-greedy strategy, and its steps are as follows: 1) Select the initial value of ε∈(0,1]; 2) Select a random number rand∈(0,1]; 3) If rand < ε, then a random action is selected from the action set in that state, otherwise, the action with the highest Q value is selected in the action set.
6. A distributed multi-agent microgrid energy management method based on Q-Learning algorithm as claimed in claim 1, It is characterized in that The method for the energy management agent in step 7 to determine the market electricity price according to the market pricing strategy is as follows: The energy management agent receives the load demand submitted by the user agent and the output power data P of the distributed energy agent. i and quote p i The formed quotation block (P i ,p i ), and these quotation blocks are divided into price p i Sort from low to high, then buy from low to high until the load demand is met. The price corresponding to the last quotation block is the market electricity price MCP. If the energy provided by the distributed energy agent is greater than the load demand, the remaining energy will be sold to the main grid; if the energy provided by the distributed energy agent cannot meet the load demand, the energy management agent will make up for it from the main grid. The energy management agent needs to ensure the balance between supply and demand, as shown in formula (7): Among them, P demand,t Represents the load requirement submitted by the user agent, They represent the energy sold by each distributed energy agent to the microgrid, P ess,t represents the charging / discharging power of the energy storage agent, P maingrid,t Represents energy purchased from the main grid.
7. A distributed multi-agent microgrid energy management method based on Q-Learning algorithm as claimed in claim 1, It is characterized in that The method for calculating the agent's instant reward in step 8 is as follows: Assume that the total demand of a user agent at time t is LD i,t , the market price is MCP t The load that the agent chooses to reduce is LC i,t , therefore, the total cost of the user agent is the sum of the remaining load demand cost and the penalty cost of reducing load: Cost i,t =MCP t ×(LD i,t -LC i,t )+PF×LC i,t (8) Among them, PF is the penalty coefficient, and the reduction amount LC i,t The following constraints must be met: 0≤LC i,t ≤LD i,t (9) The calculation of the incentive for renewable energy agents is shown in formula (10)-formula (11): Among them, MP t is the electricity purchase price of the main power grid, and Energy sold to microgrids by renewable energy agents, and Energy sold by renewable energy agents to the main grid. The rewards of other distributed agents are defined as the profit from their sale of electricity minus the cost of generation, as shown in Formula (12)-Formula (14): Cost diesel,t Cost mt,t and Cost fc,t is the power generation cost of each agent, and is the energy sold by each agent to the microgrid, and It is the energy sold by each agent to the main grid, MP t It is the price of electricity purchased by the main power grid. The reward of the energy storage agent is defined as the profit from electricity sales minus the charging cost, as shown in formula (15): Reward ess,t =MCP t ×P dch,t -MCP t ×P ch,t (15) Where P dch,t is the discharge power of the energy storage agent, P ch,t is the charging power of the energy storage agent.
8. A distributed multi-agent microgrid energy management method based on Q-Learning algorithm as claimed in claim 1, It is characterized in that The updated Q table described in step 9 is shown in formula (16): Where α∈(0,1] is the learning rate, γ∈(0,1] is the discount rate, and r t+1 is the agent in state s at time t t Take action a t Instant rewards received.
