A micro-grid energy transaction method and system based on multi-agent deep reinforcement learning

Abstract

The application provides a micro-grid energy transaction method and system based on multi-agent deep reinforcement learning, comprising: constructing a hierarchical micro-grid controller agent according to the demand of micro-grid participating in community market P2P energy transaction; wherein, the FDS controller is used for flexible demand scheduling in the micro-grid, and the ET controller is used for energy transaction between the micro-grid and other micro-grids; a hierarchical multi-agent deep neural network model is constructed, the optimal strategy learning task is decomposed into two sub-sequence tasks, and the neural network model is trained in combination with the priority experience replay and importance sampling mechanism; the trained model is combined with the intermediate market pricing strategy to control the micro-grid to complete the P2P energy transaction. The application can not only support the micro-grid to make autonomous quotation and quantity when participating in the energy transaction, ensure the new energy to be traded preferentially, and maximize the P2P transaction income. Meanwhile, the calculation complexity is reduced, and the efficiency of the agent making the optimal transaction sequence decision is improved.

Description

Technical Field

[0001] This invention relates to the field of new energy science and technology, and in particular to a microgrid energy trading method and system based on multi-agent deep reinforcement learning. Background Technology

[0002] The increasing prevalence of distributed energy resources and emerging Internet of Things (IoT) technologies are driving the development of modern power systems. Interconnection of multiple microgrids (MGs) helps achieve distributed energy supply and demand balance. However, in traditional electricity market transactions, the main grid pre-sets energy prices, and microgrids, as price takers, cannot independently determine energy prices and trading volumes, nor can they directly trade with other microgrids, significantly hindering local energy sharing among microgrids. Furthermore, this centralized pricing method, lacking attractiveness, fails to fully leverage the advantages of distributed energy resources.

[0003] Peer-to-peer (P2P) energy trading offers a coordinated and comprehensive market paradigm. It leverages the diverse renewable energy generation and load demands across multiple microgrids to achieve complementary benefits of distributed renewable energy among heterogeneous microgrids (such as industrial, commercial, and residential microgrids), and to construct a real-time and dynamic energy supply and demand management system. Furthermore, by introducing autonomous pricing strategies into the P2P trading market, directly reflecting the local energy supply and demand situation of the microgrid, efficient and flexible demand dispatch and energy trading can be achieved in an information-impaired market, ensuring the efficient utilization of renewable energy and achieving higher social welfare by balancing the interests of buyers and sellers in the microgrid. Therefore, constructing a P2P-based microgrid energy trading market framework to achieve efficient energy dispatch and trading strategies is crucial.

[0004] Deep reinforcement learning offers a novel perspective for solving optimal decision-making and long-term reward maximization problems, and many deep reinforcement learning models have been successfully applied to P2P energy trading. However, these methods still have limitations in constructing a P2P-based microgrid energy trading market framework for the following reasons: First, the internal flexible demand scheduling and external energy trading of a microgrid are coupled and interactive, requiring simultaneous consideration when modeling microgrid trading behavior; separate mathematical modeling is not feasible. Second, existing energy pricing strategies do not support microgrids autonomously setting energy prices and trading volumes, making it impossible for existing methods to guarantee the priority sharing of more cost-effective local renewable energy in the P2P market, hindering the efficient utilization of renewable energy and reducing economic benefits. Finally, most existing multi-agent-based deep reinforcement learning algorithms employ centralized training to overcome environmental imbalance problems. However, centralized training requires exploring a joint action space encompassing all microgrid trading behaviors, leading to computational overload issues for deep reinforcement learning algorithms based on complex neural network structures.

[0005] In peer-to-peer (P2P) energy trading between microgrids, flexible demand dispatch within the microgrid and external energy trading are coupled and interactive. The lack of a coordinated modeling mechanism prevents globally optimal trading decisions, thus reducing the market returns of the microgrid trading platform. Furthermore, existing P2P pricing methods cannot provide microgrids with autonomous bidding and quantity reporting capabilities, nor can they guarantee priority trading of cost-effective renewable energy sources. These issues limit the application of the microgrid P2P energy trading framework in the distributed renewable energy trading market. Summary of the Invention

[0006] This invention proposes a novel P2P energy trading framework based on multi-agent deep reinforcement learning, aiming to maximize trading profits among multiple heterogeneous microgrids. Specifically, a novel hierarchical structure based on microgrid controller agents is designed to model the coupled interaction between flexible demand scheduling and autonomous bidding in microgrids; an improved mid-market rate (IMMR) pricing strategy is proposed, allowing microgrids to autonomously bid and submit quantities to incentivize their active participation in local energy trading; to reduce computational complexity, a hierarchical multi-agent DQN algorithm is proposed, decomposing the optimal policy learning workload into two sub-task sequences and dividing the larger neural network into two smaller hierarchical neural networks with the same reward, thereby improving action exploration efficiency in the sequence learning task while maintaining operational flexibility.

[0007] The above-mentioned technical problems of the present invention are mainly solved by the following technical solutions:

[0008] A microgrid energy trading method based on multi-agent deep reinforcement learning.

[0009] An FDS controller for flexible load demand scheduling within a microgrid and an ET controller for energy trading between the microgrid and other microgrids are constructed, wherein the output data of the FDS controller is the input data of the ET controller.

[0010] The FDS controller and ET controller are trained using empirical data.

[0011] Using a trained controller, a microgrid is controlled to complete P2P energy trading based on an improved intermediate market pricing strategy.

[0012] The above-mentioned microgrid energy trading method based on multi-agent deep reinforcement learning,

[0013] Each FDS controller and each ET controller can obtain the current market environment state to generate their respective private state at the current moment; and each FDS controller obtains a flexible load scheduling action strategy based on the obtained private state and inputs the action strategy into the ET controller.

[0014] Each ET controller obtains an energy trading action strategy based on the private state it acquires and the action strategy input by the FDS controller, selects an action to execute, and returns an immediate reward and the next moment's state.

[0015] In the aforementioned microgrid energy trading method based on multi-agent deep reinforcement learning, flexible loads include...

[0016] Highly flexible loads are represented by the following model:

[0017]

[0018]

[0019]

[0020] For a moderately flexible load, the model is represented as follows:

[0021]

[0022] Low-flexibility loads are represented by the following model:

[0023]

[0024] in, Indicates indoor temperature. and These represent the minimum and maximum acceptable temperatures, respectively. For air heating efficiency. For thermal resistance, For heat capacity, For HVAC power demand, To the maximum demand; and These represent the charging and discharging efficiencies, respectively. and These represent the charging and discharging energy of the EV, respectively.

[0025] In the aforementioned microgrid energy trading method based on multi-agent deep reinforcement learning, for medium-flexibility loads, the technical constraints of EVs are related to the battery's capacity constraints and maximum charge / discharge constraints, as shown below:

[0026]

[0027]

[0028]

[0029] binary variables Indicates the charging status of the EV. This is a schedulable flag for EVs. The maximum depth of discharge, At maximum charging state, For the maximum capacity of EV batteries, and These represent the maximum charging and discharging energy, respectively. , and These represent the electrical energy required by EF, DW, and SA, respectively. This refers to the load operation cycle.

[0030] In the aforementioned microgrid energy trading method based on multi-agent deep reinforcement learning, the definition is... For flexible load sets, Indicates the first The first microgrid One flexible load; among which This indicates the electrical energy required to meet the load. The time interval for deferring loads without penalty; each flexible load should be within the allowed scheduling window. Completed within the time limit. Indicates the starting point. This is the deadline for load dispatch operation.

[0031] In the aforementioned microgrid energy trading method based on multi-agent deep reinforcement learning, the intermediate market pricing strategy calculates the buy and sell prices in the local community market based on the average of all seller quotes, including...

[0032] When local power generation is exactly equal to local demand, both the market purchase price and the market selling price are equal to the market midpoint.

[0033] When local power generation is less than total demand, the local purchase price is higher than the median price, while the local selling price is equal to the median price.

[0034] When local power generation exceeds total demand, the local selling price is lower than the median price, while the buying price remains at the median price.

[0035] No. The transaction revenue obtained by a microgrid in the community market is calculated as follows:

[0036] .

[0037] The above-mentioned microgrid energy trading method based on multi-agent deep reinforcement learning,

[0038] FDS controller private state and ET controller private state The definition is as follows:

[0039]

[0040]

[0041] in, Represents intelligent agents At any moment The completion status of flexible loads reflects which flexible loads have been completed and which loads are still pending scheduling. and These represent the buy and sell prices of the power grid, respectively. Represents intelligent agents Net energy, This indicates the demand for inflexible loads;

[0042] The action definition of the FDS controller is as follows:

[0043]

[0044] Among them, binary variables , , , , Indicates whether the flexible loads EV, HVAC, EF, SA, and DW are currently scheduled for execution;

[0045] The actions of the ET controller include energy quotes and quantities participating in P2P market transactions;

[0046]

[0047] in, Energy pricing for microgrids, Indicates energy trading volume; when This means microgrids He will purchase electricity as a consumer, at this time ;when This means microgrids They will sell electricity as sellers at this time. .

[0048] In the aforementioned microgrid energy trading method based on multi-agent deep reinforcement learning, the microgrid... Transaction revenue in the P2P community market can be represented as follows:

[0049]

[0050] in, , .

[0051] In the aforementioned microgrid energy trading method based on multi-agent deep reinforcement learning, the goal during controller training is to maximize the controller's cumulative expected reward.

[0052]

[0053] in, Indicates the discount factor. For intelligent agents The set of all strategies, In addition to intelligent agents The best strategy for all other intelligent agents;

[0054] The specific training steps include:

[0055] Initialize the neural network parameters of the FDS controller and ET controller, and the experience playback pool information used to store experience data;

[0056] The FDS controller acquires the private state and outputs the action, then passes the generated action to the ET controller.

[0057] The ET controller acquires the private state and outputs an action. After executing the action and calculating the current reward, it acquires the next state.

[0058] After storing the current experience record in the experience follow-up pool, repeat the reward calculation step until the number of samples in the follow-up pool meets the set threshold.

[0059] Randomly select several experience samples from the experience revisit pool, update the controller neural network parameters, and repeat the reward calculation and controller neural network parameter update steps until the neural network converges.

[0060] A microgrid energy trading system based on multi-agent deep reinforcement learning, including

[0061] The first module is configured to construct an FDS controller for flexible load demand scheduling within a microgrid and an ET controller for energy trading between the microgrid and other microgrids, wherein the output data of the FDS controller is the input data of the ET controller.

[0062] The second module is configured to train the controller using empirical data to obtain a trained FDS controller and an ET controller.

[0063] The third module is configured to use a trained controller to control the microgrid to complete P2P energy transactions based on an intermediate market pricing strategy.

[0064] Therefore, the present invention has the following advantages: 1. For the P2P energy trading framework between multiple microgrids, a new hierarchical microgrid controller intelligent agent structure is designed, which uses two sub-controllers to simulate the coupled interaction between flexible demand scheduling inside the microgrid and autonomous bidding for external energy, and then describes the transaction sequence decision problem as a partially observable Markov decision process.

[0065] 2. In order to enable microgrids to independently bid and report quantities, an improved intermediate market rate (IMMR) pricing strategy is proposed. Combined with a penalty mechanism, it ensures that microgrids with better cost-effectiveness can prioritize trading their local renewable energy sources, thus positively incentivizing microgrids to actively participate in the P2P energy trading market.

[0066] 3. To reduce computational complexity, a hierarchical multi-agent DQN algorithm is proposed, which decomposes the optimal trading strategy learning workload into two sub-task sequences and divides the larger neural network into two smaller hierarchical neural networks with the same reward, thereby improving the action exploration efficiency in sequence learning tasks while maintaining operational flexibility.

[0067] 4. A prototype system was developed to verify the feasibility of the framework. Compared with existing methods, the method proposed in this invention has lower computational complexity, faster convergence, and can bring higher economic benefits to microgrids participating in transactions. Attached Figure Description

[0068] Appendix Figure 1 This is a schematic diagram of the IMMR pricing method under different values ​​of the current remaining energy in the entire market.

[0069] Appendix Figure 2 This is a schematic diagram of the neural network training framework for a microgrid energy trading method and system based on multi-agent deep reinforcement learning;

[0070] Appendix Figure 3 This is a schematic diagram of the controller training algorithm.

[0071] Appendix Figure 4 This is a flowchart illustrating a microgrid energy trading method based on multi-agent deep reinforcement learning. Detailed Implementation

[0072] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0073] Example:

[0074] In this embodiment, a scalable multi-microgrid interconnection model is first defined, wherein... This represents a collection of microgrids, each containing different combinations of distributed resources. Each microgrid can achieve energy supply and demand balance in a multi-microgrid area by scheduling its internal flexible loads to directly trade surplus energy with other microgrids. Each day is divided into... Equal time intervals are established, and then the following steps are performed.

[0075] I. Construct an FDS controller for flexible load demand scheduling within the microgrid, and an ET controller for energy trading between the microgrid and other microgrids. The output data of the FDS controller is the input data of the ET controller.

[0076] 1. First, the mathematical models of flexible loads and energy storage systems within the microgrid are defined. 1.1 Flexible loads include high-flexibility loads, medium-flexibility loads, and low-flexibility loads.

[0077] 1) Highly flexible load

[0078] Such loads can be scheduled on a minute-by-minute basis, and their operation can be interrupted or postponed to another scheduling window. For example, Heating, Ventilation, and Air Conditioning (HVAC), as a highly flexible load, requires electrical energy to maintain indoor temperature within a certain range, thereby ensuring user thermal comfort. Its dynamic mathematical model is as follows:

[0079] (1)

[0080] (2)

[0081] (3)

[0082] in, Indicates indoor temperature. and These represent the minimum and maximum acceptable temperatures, respectively. For air heating efficiency. For thermal resistance, For heat capacity, For HVAC power demand, This represents the maximum demand.

[0083] 2) Medium flexibility load

[0084] Such loads can be scheduled on an hourly basis, and their power consumption rate and operating time can be changed or interrupted. For example, electric vehicles (EVs), as a medium-flexibility load, can function as both electricity users and suppliers in distributed microgrid scenarios. Their dynamic charge-discharge model is represented as follows:

[0085] (4)

[0086] in, and These represent the charging and discharging efficiencies, respectively. and These represent the charging and discharging energy of the EV, respectively. Meanwhile, the technical constraints of EVs are related to the battery's capacity constraints and maximum charging and discharging constraints, as detailed below:

[0087] (5)

[0088] (6)

[0089] (7)

[0090] Among them, binary variables Indicates the charging status of the EV. This is a schedulable flag for EVs. The maximum depth of discharge, At maximum charging state, For the maximum capacity of EV batteries, and These represent the maximum charging and discharging energy, respectively.

[0091] 3) Low flexibility load

[0092] Such loads can be scheduled on a daily basis and must be executed according to a strict operating plan. That is, only the starting point of its operating window can be modified. For example, low-flexibility loads such as industrial electric furnaces (EF), commercial dishwashers (DW), and household appliances (SA) cannot have their operating cycles modified or interrupted once started. Each cycle includes an independent, fixed operating time. The relevant mathematical model is expressed as follows:

[0093] (8)

[0094] in, , and These represent the electrical energy required by EF, DW, and SA, respectively. This refers to the load operation cycle.

[0095] The primary task of this system in the first phase is to schedule internal flexible loads while meeting the non-flexible load requirements of the microgrid, enabling the microgrid to obtain higher returns when participating in the P2P energy trading in the second phase. Definition For flexible load sets, Indicates the first The first microgrid One flexible load. Among them This indicates the electrical energy required to meet the load. The time interval for deferring loads without penalty. Each flexible load should be within the allowed scheduling window. Completed within the time limit. Indicates the starting point. This is the deadline for load dispatch operation.

[0096] 1.2 The operating constraints of the energy storage system are as follows:

[0097] 1) Operational constraints

[0098] Energy storage systems (ESS) can smooth out intermittent solar power generation by charging during off-peak hours and discharging during peak hours. Their charging and discharging operations must meet corresponding power and capacity constraints to ensure the safe and stable operation of the storage batteries. The mathematical model is defined as follows:

[0099] (9)

[0100] in, This indicates the charging and discharging efficiency of the ESS. and These represent the charging and discharging power, respectively.

[0101] 2) Operating costs

[0102] The lifespan of batteries in an energy storage system typically decreases with repeated charge and discharge cycles. The operating cost of an ESS over a charge-discharge cycle can be expressed as follows:

[0103] (10)

[0104] in, The linear approximate slope representing the battery life cycle. The total charge and discharge volume of the battery. Fixed purchase cost for ESS batteries.

[0105] 2. Next, define the FDS controller and ET controller.

[0106] In this system, each microgrid is modeled as a hierarchical agent with two controllers: an internal Flexible Demand Scheduling (FDS) controller and an external Energy Trading (ET) controller. The community market represents the environment in which the agents interact. The system operates based on the market environment status. Each agent will receive a private observation state, and then, based on the observed private state... Deployment strategy This allows the selection of an action to execute and the transition to the next state. At time... Each agent executes actions simultaneously based on its own private observation state. and return instant rewards and the state at the next moment Each agent's goal is to maximize its cumulative expected reward by learning the optimal action policy:

[0107] (19)

[0108] in, Indicates the discount factor. For intelligent agents The set of all strategies, In addition to intelligent agents The best strategy for all other intelligent agents besides [the one mentioned].

[0109] 1) Private state of the FDS controller: based on the microgrid Microgrid types Its FDS controller private state It can be defined as follows:

[0110] (20)

[0111] in, Represents intelligent agents At any moment The completion status of flexible loads reflects which flexible loads have been completed and which loads are still pending scheduling. and These represent the buy and sell prices of the power grid, respectively. Represents intelligent agents Net energy, This indicates the demand for inflexible loads.

[0112] 2) FDS Controller Actions: The actions of the FDS controller mainly include scheduling decisions for flexible loads (EV, HVAC, EF, SA, DW). Depending on the type of microgrid, the FDS controller actions are defined as follows:

[0113] (twenty one)

[0114] Among them, binary variables , , , , Indicates whether the flexible loads EV, HVAC, EF, SA, and DW are currently scheduled for execution.

[0115] 3) ET controller private state: ET controller private state The definition is as follows:

[0116] (twenty two)

[0117] Among them, the ET controller will control the actions of the FDS controller. As one of its state parameters, the two controllers learn a cooperative strategy to obtain the optimal scheduling and trading strategy to maximize mutual rewards.

[0118] 4) ET controller actions: based on private state The actions of the ET controller mainly include energy quotes and quantities participating in P2P market transactions.

[0119] (twenty three)

[0120] in, Energy pricing for microgrids, Indicates energy trading volume. When... This means microgrids He will purchase electricity as a consumer, at this time ;when This means microgrids They will sell electricity as sellers at this time. .

[0121] 5) Reward Function: While meeting local needs, this system designs a reward function that maximizes transaction profits. According to formula (17), the microgrid... Transaction revenue in the P2P community market can be represented as follows:

[0122] (twenty four)

[0123] in, , .

[0124] 2. Train the controller using empirical data to obtain the trained FDS controller and ET controller;

[0125] The agent is trained using a combination of hierarchical deep reinforcement learning and DQN techniques, primarily consisting of two parts: the first part trains the FDS controller neural network to learn the optimal flexible load scheduling strategy; the second part trains the ET controller neural network, learning the optimal trading strategy based on the FDS controller's action output and current state information. Furthermore, to improve the learning speed and efficiency of the neural network, priority experience replay and importance sampling mechanisms are used. The hierarchical deep neural network structure is as follows: Figure 2 As shown, the training steps are as described in Algorithm 1.

[0126]

[0127]

[0128] Each microgrid agent comprises two sub-controllers: an FDS controller and an ET controller. They divide the optimal policy learning load into two sub-task sequences. The FDS controller's actions determine the currently scheduled flexible load and then provide this information to the ET controller, which, based on the FDS's actions and the current energy state, decides on the price and quantity of energy transactions. The goal of the two sub-controllers is to maximize their shared reward through cooperation.

[0129] The first step is the training process of the FDS controller neural network.

[0130] 1. FDS Main Network Will As input, and output The probability of each action at any given moment.

[0131] 2. The FDS controller starts from having -Select an action from the action space of the greedy strategy. And send to the ET target network ,in -The exploration process of greedy returns a probability of Random actions, while the exploitation process is based on 1- Choose one with probability that makes Q value The action that maximizes the outcome.

[0132] Next, the neural network of the ET controller is trained.

[0133] 3. The ET controller will receive actions from the FDS controller. As one of its state parameters, the action This indicates the selected flexible load to be executed.

[0134] 4. Based on the observed status of the ET controller Its neural network Output by quotation and transaction amount Composition of actions .

[0135] 5. In each time period Microgrids calculate their rewards And observe the next state. Then transfer records Stored in the experience replay pool ,in , , .

[0136] Finally, priority experience replay and importance sampling mechanisms are used to improve training efficiency.

[0137] 6. After a fixed training cycle, sample a small batch of data from the replay pool to train the next training iteration.

[0138] Training networks. In addition, this system uses two independent FDS target networks. and ET target network Evaluate the expected reward and minimize the loss for each state-action pair. Loss function It is expressed as follows:

[0139] (25)

[0140] in, For state The action to be performed For the target network weights, Main network weights.

[0141] 7. To more effectively utilize empirical data for learning, a priority-based experience replay mechanism is adopted.

[0142] Important records in the experience pool are frequently replayed. The priority of each record is organized using a SumTree structure. When the loss value of a record increases, it is more likely to be sampled to update the network. The sampling probability can be expressed as:

[0143] (26)

[0144] in, It is a record priority, This is the size of the experience replay pool. (Index) Determine how much priority to use. Corresponding to unified sampling. Record. Priority can be achieved through To calculate, where It is a record The timing difference error, Use a small positive constant to ensure that all records can be...

[0145] They are sampled with a certain probability.

[0146] 8. To account for changes in distribution, network updates are weighted using importance sampling weights, which can be defined as:

[0147] (27)

[0148] in, Control the number of importance samples, when Importance sampling is not performed when Perform full and important sampling at that time. from The linear change is 1.

[0149] Third, using a trained controller, the microgrid is controlled to complete P2P energy transactions based on an intermediate market pricing strategy.

[0150] The intermediate market pricing strategy is defined as follows:

[0151] To promote and incentivize energy trading among microgrids, a peer-to-peer (P2P) energy trading market was established. This market achieves supply and demand balance within interconnected microgrid areas through centralized limited information exchange and distributed P2P energy trading. Building upon this, an improved market median price (IMMR) pricing method is further proposed, with its specific functions and principles explained below:

[0152] Assumption and These respectively represent the community market in Net generation and net load at any given moment, and the current surplus energy in the entire market. As net load of microgrid The function can be defined as follows:

[0153] (11)

[0154] Different types of microgrids have different net load calculations due to the different types of loads within them. This system designs three different types of microgrids, and their specific net loads can be calculated as follows:

[0155] (12)

[0156] in, Indicates the first Inflexible loads of a microgrid Indicates the power of the ESS. This refers to the power output of an EV. and These represent the charging and discharging power of the ESS, respectively, and both must satisfy the operational relationship constraint of formula (9).

[0157] The IMMR pricing method is based on all seller quotes. Calculate the average value of the purchase price in the local community market. and selling price The price calculation process requires adjustments based on the imbalance between total market demand and electricity generation. The IMMR pricing method can be interpreted in three ways, as follows: Figure 1 As shown.

[0158] (1) When local power generation is exactly equal to local demand, i.e. Then the market buying price and selling price All equal to the market median price .

[0159] (13)

[0160] (2) When local power generation is less than total demand, i.e. The energy gap must then be covered by the main grid at a higher grid purchase price. Provided. Therefore, the additional cost incurred. The cost needs to be shared proportionally based on the net demand of each microgrid. At this point, the local purchase price is higher than the median price, while the local selling price remains equal to the median price. .

[0161] (14)

[0162] (15)

[0163] (3) When local power generation exceeds total demand, i.e. Then the remaining energy in the market can only be sold at low prices. Sold to the power grid. Therefore, the difference in income... This cost should be borne by all microgrids according to their net generation ratio. At this point, the local selling price is lower than the median price, while the buying price remains set at the median price.

[0164] (16)

[0165] (17)

[0166] In summary, the first The transaction revenue obtained by a microgrid in the community market can be calculated as follows:

[0167] (18)

[0168] in, .

[0169] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A microgrid energy trading method based on multi-agent deep reinforcement learning, characterized in that, An FDS controller for flexible load demand scheduling within a microgrid and an ET controller for energy trading between the microgrid and other microgrids are constructed, wherein the output data of the FDS controller is the input data of the ET controller. The FDS controller and ET controller are trained using empirical data. Using a trained controller, a microgrid is controlled to complete P2P energy trading based on an improved intermediate market pricing strategy; Each FDS controller and each ET controller can obtain the current market environment state to generate their respective private state at the current moment; and each FDS controller obtains a flexible load scheduling action strategy based on the obtained private state and inputs the action strategy into the ET controller. Each ET controller obtains an energy trading action strategy based on the private state it acquires and the action strategy input by the FDS controller, selects an action to execute, and returns an instant reward and the next moment's state. The private state of the FDS controller includes: agent At any moment The completion status of flexible loads reflects which flexible loads have been completed and which are still pending dispatch; the grid's buy and sell prices; and the intelligent agent. The net energy and inflexible load requirements; The actions of the FDS controller include: scheduling decisions for different flexible loads; The private state of the ET controller includes: the buy and sell prices of the power grid; and the intelligent agent. Net energy; the demand of inflexible loads and the operation of the FDS controller; The actions of the ET controller include: energy quotes and energy quantities participating in P2P market transactions.

2. The microgrid energy trading method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, Flexible loads include Highly flexible loads are represented by the following model: For a moderately flexible load, the model is represented as follows: Low-flexibility loads are represented by the following model: in, Indicates indoor temperature. and These represent the minimum and maximum acceptable temperatures, respectively. For air heating efficiency. For thermal resistance, For heat capacity, For HVAC power demand, To the maximum demand; and These represent the charging and discharging efficiencies, respectively. and These represent the charging and discharging energy of the EV, respectively. , and These represent the electrical energy required by EF, DW, and SA, respectively. This refers to the load operation cycle.

3. The microgrid energy trading method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In medium-flexibility loads, the technical constraints of EVs are related to battery capacity constraints and maximum charge and discharge constraints, as shown below: binary variables Indicates the charging status of the EV. This is a schedulable flag for EVs. The maximum depth of discharge, At maximum charging state, For the maximum capacity of EV batteries, and These represent the maximum charging and discharging energy, respectively.

4. The microgrid energy trading method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, definition For flexible load sets, Indicates the first The first microgrid One flexible load; among which This indicates the electrical energy required to meet the load. The time interval for deferring loads without penalty; each flexible load should be within the allowed scheduling window. Completed within the time limit. Indicates the starting point. This is the deadline for load dispatch operation.

5. The microgrid energy trading method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, The improved intermediate market pricing strategy calculates the buy and sell prices in the local community market based on the average of all seller quotes, including... When local power generation is exactly equal to local demand, both the market purchase price and the market selling price are equal to the market midpoint. When local power generation is less than total demand, the local purchase price is higher than the median price, while the local selling price is equal to the median price. When local power generation exceeds total demand, the local selling price is lower than the median price, while the buying price remains at the median price. No. The transaction revenue obtained by a microgrid in the community market is calculated as follows: 。 6. The microgrid energy trading method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, FDS controller private state and ET controller private state The definition is as follows: in, Represents intelligent agents At any moment The completion status of flexible loads reflects which flexible loads have been completed and which loads are still pending scheduling. and These represent the buy and sell prices of the power grid, respectively. Represents intelligent agents Net energy, This indicates the demand for inflexible loads; The action definition of the FDS controller is as follows: Among them, binary variables , , , , Indicates whether the flexible loads EV, HVAC, EF, SA, and DW are currently scheduled for execution; The actions of the ET controller include energy quotes and quantities participating in P2P market transactions; in, Energy pricing for microgrids, Indicates energy trading volume; when This means microgrids He will purchase electricity as a consumer, at this time ;when This means microgrids They will sell electricity as sellers at this time. .

7. The microgrid energy trading method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, microgrids Transaction revenue in the P2P community market can be represented as follows: in, , .

8. The microgrid energy trading method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, When training the controller, the goal is to maximize the controller's cumulative expected reward: in, Indicates the discount factor. For intelligent agents The set of all strategies, In addition to intelligent agents The best strategy for all other intelligent agents; The specific training steps include: Initialize the neural network parameters of the FDS controller and ET controller, and the experience playback pool information used to store experience data; The FDS controller acquires the private state and outputs the action, then passes the generated action to the ET controller. The ET controller acquires the private state and outputs an action. After executing the action and calculating the current reward, it acquires the next state. After storing the current experience record in the experience follow-up pool, repeat the reward calculation step until the number of samples in the follow-up pool meets the set threshold. Randomly select several experience samples from the experience revisit pool, update the controller neural network parameters, and repeat the reward calculation and controller neural network parameter update steps until the neural network converges.

9. A microgrid energy trading system based on multi-agent deep reinforcement learning, characterized in that, include The first module is configured to construct an FDS controller for flexible load demand scheduling within a microgrid and an ET controller for energy trading between the microgrid and other microgrids, wherein the output data of the FDS controller is the input data of the ET controller. The second module is configured to train the controller using empirical data to obtain a trained FDS controller and an ET controller. The third module is configured to use a trained controller to control the microgrid to complete P2P energy transactions based on an improved intermediate market pricing strategy. Each FDS controller and each ET controller can obtain the current market environment state to generate their respective private state at the current moment; and each FDS controller obtains a flexible load scheduling action strategy based on the obtained private state and inputs the action strategy into the ET controller. Each ET controller obtains an energy trading action strategy based on the private state it acquires and the action strategy input by the FDS controller, selects an action to execute, and returns an instant reward and the next moment's state. The private state of the FDS controller includes: agent At any moment The completion status of flexible loads reflects which flexible loads have been completed and which are still pending dispatch; the grid's buy and sell prices; and the intelligent agent. The net energy and inflexible load requirements; The actions of the FDS controller include: scheduling decisions for different flexible loads; The private state of the ET controller includes: the buy and sell prices of the power grid; and the intelligent agent. Net energy; the demand of inflexible loads and the operation of the FDS controller; The actions of the ET controller include: energy quotes and energy quantities participating in P2P market transactions.

Images