Microgrid group scheduling optimization method and device based on DDQN algorithm and action mask

Abstract

The application discloses a micro-grid group scheduling optimization method and device based on a DDQN algorithm and an action mask, and the method comprises the following steps: regarding each micro-grid as an intelligent agent to construct a micro-grid group multi-agent model; the micro-grid group multi-agent model takes maximizing total operation income of the micro-grid group as a target, and considers power balance constraints, output constraints, ESS constraints and interactive power constraints; Markov modeling is performed according to the micro-grid group multi-agent model to obtain a Markov decision process; the micro-grid group is scheduled and optimized according to the Markov decision process in combination with a first algorithm to obtain a micro-grid group joint scheduling scheme; the first algorithm is based on the DDQN algorithm, and illegal actions are shielded through the action mask at each time step of the DDQN algorithm.

Description

Technical Field

[0001] This invention relates to the field of microgrid group scheduling technology, and in particular to a microgrid group scheduling optimization method and apparatus based on the DDQN algorithm and action mask. Background Technology

[0002] Compared to traditional energy sources, distributed generation (DG) offers advantages such as economy, flexibility, and environmental friendliness. However, due to its small capacity, geographically dispersed nature, and intermittent output, dispatching agencies often find it difficult to directly and effectively regulate it. To address the technical challenges of DG grid integration, microgrid clusters (MGs) have emerged as a new power system optimization model. MGs offer an effective way to absorb and manage DG. Composed of multiple geographically adjacent microgrids, MGs are small-scale power systems integrating DG, energy storage systems (ESS), loads, and their control equipment, achieving integrated operation of DG and loads.

[0003] Existing research has yielded systematic theoretical results in modeling energy management systems (MGs) and has further explored MG management. To achieve more efficient, flexible, and low-cost MG management, thereby supporting the grid connection and consumption of large-scale renewable energy, numerous studies have addressed the MG optimal scheduling problem. Existing technology one proposes a two-layer optimal scheduling strategy for MGs based on an improved particle swarm optimization algorithm; the first layer optimizes the MGs themselves, achieving maximum economic benefits and minimum operational risks; the second layer optimizes the MGs themselves, minimizing the interaction power and fluctuations between the MGs and the distribution network. Existing technology two establishes a hierarchical energy management architecture for isolated MGs based on a multi-agent system, proposing an optimal scheduling method that considers the interaction willingness of MGs and the risks of insufficient flexibility. Existing technology three proposes an optimal scheduling method for MGs based on a second-order Newton algorithm and a consensus algorithm. This method does not rely on a centralized controller, but collaborates through point-to-point communication, achieving fully distributed optimal scheduling and improving the flexibility and scalability of MGs. Existing technologies four focus on researching multi-agent collaborative control strategies for microgrid groups, proposing a microgrid group sub-Bruker game-based optimization scheduling method with shared energy storage, and combining Nash negotiation theory to coordinate benefit distribution, effectively improving the economic efficiency of multi-agent collaboration in microgrid groups. However, although existing technologies one through four have studied MGs strategy optimization, they mostly use classical analytical optimization algorithms or traditional intelligent algorithms, which rely heavily on detailed mathematical modeling and have weak generalization ability. With the advancement of artificial intelligence technology, reinforcement learning has also been widely applied to power system scheduling optimization in recent years. Reinforcement learning is based on data-driven and environment interaction, does not rely on precise mathematical models, has strong adaptability, and is more suitable for nonlinear systems that are difficult to model. However, due to the high safety requirements of power systems, reinforcement learning itself has a certain degree of uncontrollability and cannot ensure that the system always adheres to safety constraints when performing scheduling tasks; at the same time, existing reinforcement learning-based scheduling optimization methods still have considerable room for improvement in optimization efficiency and stability when dealing with high-dimensional complex environments.

[0004] Therefore, a new technical solution is urgently needed to address the technical problem of how to solve the MGs scheduling strategy safely and efficiently based on reinforcement learning. Summary of the Invention

[0005] This invention provides a method and apparatus for optimizing microgrid group scheduling based on the DDQN algorithm and action mask, which solves the technical problem of how to solve the MGs scheduling strategy safely and efficiently based on reinforcement learning.

[0006] To achieve the above objectives, this invention provides a microgrid group scheduling optimization method based on the DDQN algorithm and action masking, comprising:

[0007] Each microgrid is treated as an intelligent agent to construct a multi-agent model of the microgrid group; the multi-agent model of the microgrid group aims to maximize the total operating benefits of the microgrid group, and considers power balance constraints, output constraints, ESS constraints and interactive power constraints.

[0008] Markov modeling is performed based on the multi-agent model of the microgrid group to obtain the Markov decision process; the scheduling optimization of the microgrid group is performed based on the Markov decision process and the first algorithm to obtain the joint scheduling scheme of the microgrid group; the first algorithm is based on the DDQN algorithm, and illegal actions are masked by action masking at each time step of the DDQN algorithm.

[0009] Preferably, the microgrid group is optimized for scheduling based on the Markov decision process combined with the first algorithm, resulting in a joint scheduling scheme for the microgrid group, including:

[0010] In each time step, the action mask is first calculated based on the current state and action space of the Markov decision process, and the action space under the current state is masked according to the action mask. Then, the Markov decision process is optimized according to the DDQN algorithm. After completing the optimization iteration of all time steps, the joint scheduling scheme of the microgrid group is obtained. The joint scheduling scheme of the microgrid group includes the DG output command and ESS charging and discharging power command of each microgrid in the microgrid group.

[0011] Preferably, the process of calculating an action mask based on the current state and action space in the Markov decision process, and then masking the action space in the current state based on the action mask, includes:

[0012] The agent in a Markov decision process obtains the current time step from the environment. Current state And based on the current state Calculate the current state using the action space. The following are legal action sequences With action mask :

[0013] ;

[0014] ;

[0015] Based on action mask Current state Next action of Assign values:

[0016] ;

[0017] in, For state Constraints of the multi-agent model for microgrid clusters; action mask It is an array with the same specifications as the action space, and each element corresponds one-to-one with an action in the action space; Current state Next action corresponding value, For the first in the action space One action;

[0018] Action mask Import the training and target networks of the DDQN algorithm.

[0019] Preferably, optimizing the Markov decision process using the DDQN algorithm includes:

[0020] F1, based on the action mask and current state Combination A greedy strategy selects an action from the action space. ; Perform actions Environmental output reward and the next state ; the quadruple Store to the experience replay buffer;

[0021] F2. Randomly sample a set of experience samples from the experience playback buffer. ;

[0022] F3. Calculate the state through the target network. All actions below value And select the largest value According to the reward and Achieve the goal value :

[0023] ;

[0024] in, This is a discount factor used to balance immediate rewards and future rewards; These are the parameters of the target network;

[0025] If the state For the state containing the maximum time step, then ;

[0026] F4. Calculate the state by training the network. Next action Estimate value , Parameters for training the network;

[0027] F5, According to the target value and estimates value Calculate the loss function :

[0028] ;

[0029] in, For expectations;

[0030] F6. Based on the loss function Combine gradient descent algorithm to train network parameters Perform backpropagation updates; based on the preset update frequency. Regularly update the parameters of the training network Parameters synchronized to the target network ;

[0031] F7, if the current time step If the maximum time step is reached, the iterative optimization ends; otherwise, proceed to the next time step.

[0032] Preferably, the microgrid cluster multi-agent model, with the objective of maximizing the total operating benefits of the microgrid cluster, includes:

[0033] The total operating revenue of a microgrid cluster includes revenue from electricity trading. Carbon emission costs Scheduling costs and MG operation and maintenance costs Scheduling costs Including ESS energy storage costs MG inter-energy exchange cost Costs of curtailing wind and solar power The objective function then includes:

[0034] ;

[0035] in, and These are the sets of MGs in MGs and the sets of schedulable DGs in their corresponding MGs. Indicates the first MG; This refers to the set of time nodes in the joint scheduling strategy of MGs; , , and These are the weighting coefficients for different benefits and costs; For the ESS charge / discharge power of each MG, This represents the discharge power, and vice versa for the charging power; The DG power generation capacity of each MG, including wind power output. Photovoltaic power output and micro gas turbine output .

[0036] Preferably, the total operating benefits of a microgrid cluster specifically include:

[0037] Electricity trading revenue Determined through time-of-use pricing and interactive power:

[0038] ;

[0039] in, , for The time-of-use electricity price for MG's purchase and sale of electricity with the power grid during the same time period; For the interaction power between MG and the grid, when A time interval indicates that MG purchases electricity from the grid, while a time interval indicates that MG sells electricity to the grid.

[0040] Carbon emission costs Differentiate based on the source of electricity used or produced by MG:

[0041] ;

[0042] in, The unit cost of carbon emissions corresponding to MG's use of electricity from the main grid; The unit cost of carbon emissions when producing or using electricity from DG; For DG in Actual power during the time period;

[0043] MG scheduling cost Energy storage cost of ESS MG inter-energy exchange cost Costs of curtailing wind and solar power express:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] in, and These are the ESS discharge and charging power, respectively. The current power level of the ESS; The ESS motion loss coefficient; The self-dissipation coefficient of ESS; The energy dissipation cost coefficient for ESS (Energy Saving Service); The grid access service fee paid for power exchange between MGs using the power grid; Inter-MG power; The cost coefficient for wind and solar curtailment in MG; For DG in Maximum power during the time period;

[0049] MG's operation and maintenance costs The cost of wind power generation for each DG in MG Cost of photovoltaic power generation and the cost of micro gas turbine power generation composition:

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] in, This is the cost coefficient for wind power generation; This refers to the energy consumption coefficient of the wind turbine unit. This represents the maximum power output of the wind turbine. This represents the cost coefficient for photovoltaic power generation. , and This represents the power generation consumption coefficient of the micro gas turbine.

[0055] Preferably, the consideration of power balance constraints, output constraints, ESS constraints, and interactive power constraints includes:

[0056] ;

[0057] in, For DG in Actual power during the time period and These are the maximum and minimum outputs of the DG. The ESS charging and discharging power in MG is the difference between the discharging power and the charging power. The interaction power between the MG and external systems includes the interaction power with the main network and the interaction power with other MGs. A positive value indicates the power transmitted from the MG to the outside, while a negative value indicates the power transmitted from the outside to the MG. For MG load; and For ESS The charging and discharging power during a given period; and This represents the maximum charging and discharging power of the ESS. For the first microgrid exist The state of charge (ESS) during a given time period; and These represent the minimum and maximum states of charge of the ESS. and for The minimum and maximum values.

[0058] Preferred options also include:

[0059] At each time step, the energy storage dynamics of the ESS are modeled, including:

[0060] ;

[0061] ;

[0062] in, For the first microgrid The maximum energy storage capacity of the ESS in the system; The charge / discharge efficiency coefficient of the ESS;

[0063] exist and In the middle, the minimum value is determined by The minimum output power is defined as the difference between the minimum output power and the maximum load power, while the maximum output power is defined as the opposite:

[0064] ;

[0065] ;

[0066] Preferably, Markov modeling is performed based on the microgrid group multi-agent model to obtain the Markov decision process, which includes:

[0067] Markov modeling is performed based on all constraints of the microgrid group multi-agent model to obtain the environment of the Markov decision process; the state space, action space, and reward function of the Markov decision process are obtained based on the microgrid group multi-agent model.

[0068] Time step in state space The actions include:

[0069] ;

[0070] in, For MGs, the set of states of charge of the energy storage system; For the load demand set of MGs; The distributed power output set for MGs; For real-time electricity prices; For time steps;

[0071] The actions at time step t in the action space include:

[0072] ;

[0073] in, The command sequence for ESS charging and discharging power; This is the DG output command sequence;

[0074] The reward function consists of positive rewards. With negative rewards Composition, including:

[0075] ;

[0076] ;

[0077] ;

[0078] in, This is the transaction reward coefficient; the electricity transaction reward is based on the electricity transaction revenue. Multiply get; To constrain the reward coefficient; The constraint indicator function is a 0-1 variable. This indicates that the agent is in a state of strictly enforcing constraints in the current state, and a reward is given. , This indicates that the agent is in a state where the constraints are not strictly enforced; The energy storage reward factor is calculated by multiplying the stored electrical energy within each time step of the ESS by [a certain factor]. Provide energy storage incentives; As a cost penalty coefficient, the carbon emission cost, scheduling cost, and operation and maintenance cost of MGs during operation are multiplied by [a certain factor]. Reflected in negative rewards middle; To constrain the penalty coefficient; The unbalanced power generated during the agent's decision-making process, multiplied by Punishment shall be imposed.

[0079] The present invention also provides a microgrid group scheduling optimization device based on DDQN algorithm and action mask, which is used in the method of the present invention. The device includes a first module, a second module and a third module.

[0080] The first module is used to construct a multi-agent model of the microgrid group, treating each microgrid as an agent. The multi-agent model of the microgrid group aims to maximize the total operating benefits of the microgrid group and considers power balance constraints, output constraints, ESS constraints, and interactive power constraints. The second module is used to perform Markov modeling based on the multi-agent model of the microgrid group to obtain the Markov decision process. The third module is used to perform scheduling optimization of the microgrid group based on the Markov decision process and the first algorithm to obtain the joint scheduling scheme of the microgrid group. The first algorithm is based on the DDQN algorithm and uses an action mask to block illegal actions at each time step of the DDQN algorithm.

[0081] The present invention has the following beneficial effects:

[0082] This invention presents a microgrid group scheduling optimization method based on the DDQN algorithm and action masking. It constructs a multi-agent joint operation mode for microgrids (MGs), which, compared to independent operation of each MG, effectively improves the utilization rate of distributed generation (DG) while balancing the economic efficiency and low-carbon nature of MG operation. The first algorithm of this invention is better suited to handling complex problems than traditional reinforcement learning, improving upon the inability of traditional reinforcement learning to strictly enforce model constraints during training. Simultaneously, it compresses the solution space, resulting in better convergence stability and faster training speed, making it more suitable for application requirements in MG scenarios. This invention's method alleviates the difficulty in finding optimal solutions due to the curse of dimensionality in complex environments and improves the safety of applying reinforcement learning algorithms to MG scenarios. It solves the problem of energy coordination and optimization scheduling in MGs. This invention's method enables safe and efficient solution of MG scheduling strategies based on reinforcement learning.

[0083] The microgrid group scheduling optimization device based on the DDQN algorithm and action mask of the present invention, used in the method of the present invention, has the same beneficial effects as the method of the present invention.

[0084] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0085] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0086] Figure 1 This is a schematic diagram of the method flow of a preferred embodiment of the present invention;

[0087] Figure 2 This is a schematic diagram of the Markov decision process optimization according to a preferred embodiment of the present invention;

[0088] Figure 3 This is a schematic diagram of the simulated MGs model topology of a preferred embodiment of the present invention;

[0089] Figure 4 This is a schematic diagram of some MGs power generation and consumption data according to a preferred embodiment of the present invention;

[0090] Figure 5 This is a schematic diagram of the energy scheduling result of MG1 in Scheme 1 of the preferred embodiment of the present invention;

[0091] Figure 6 This is a schematic diagram of the energy scheduling result of MG2 in Scheme 1 of the preferred embodiment of the present invention;

[0092] Figure 7 This is a schematic diagram of the energy scheduling results of MG3 in Scheme 1 of the preferred embodiment of the present invention;

[0093] Figure 8 This is a schematic diagram of the energy scheduling results of MG4 in Scheme 1 of the preferred embodiment of the present invention;

[0094] Figure 9 This is a schematic diagram of the energy scheduling strategy of MG1 in Scheme 2 of the preferred embodiment of the present invention;

[0095] Figure 10 This is a schematic diagram of the energy scheduling strategy of MG2 in Scheme 2 of the preferred embodiment of the present invention;

[0096] Figure 11 This is a schematic diagram of the energy scheduling strategy of MG3 in Scheme 2 of the preferred embodiment of the present invention;

[0097] Figure 12 This is a schematic diagram of the energy scheduling strategy of MG4 in Scheme 2 of the preferred embodiment of the present invention;

[0098] Figure 13 This is a schematic diagram of the overall energy scheduling strategy of MGs in Scheme 2 of the preferred embodiment of the present invention;

[0099] Figure 14 This is a comparative schematic diagram of the algorithm iteration process of a preferred embodiment of the present invention. Detailed Implementation

[0100] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0101] See Figure 1In a preferred embodiment of the present invention, a microgrid group scheduling optimization method based on the DDQN algorithm and action mask is provided, comprising:

[0102] Y1. Construct a multi-agent model for the microgrid cluster, treating each microgrid as an agent. The multi-agent model aims to maximize the total operating benefits of the microgrid cluster, and considers power balance constraints, output constraints, ESS constraints, and interactive power constraints.

[0103] In a preferred embodiment of the present invention, the microgrid cluster multi-agent model aims to maximize the total operational benefits of the microgrid cluster, including:

[0104] The total operating revenue of a microgrid cluster includes revenue from electricity trading. Carbon emission costs Scheduling costs and MG operation and maintenance costs Scheduling costs Including ESS energy storage costs MG inter-energy exchange cost Costs of curtailing wind and solar power The objective function then includes:

[0105] ;

[0106] in, and These are the sets of MGs in MGs and the sets of schedulable DGs in their corresponding MGs. Indicates the first MG; This refers to the set of time nodes in the joint scheduling strategy of MGs; , , and These are the weighting coefficients for different benefits and costs; For the ESS charge / discharge power of each MG, This represents the discharge power, and vice versa for the charging power; The DG power generation capacity of each MG, including wind power output. Photovoltaic power output and micro gas turbine output .

[0107] In a preferred embodiment of the present invention, the total operating benefits of the microgrid group specifically include:

[0108] Electricity trading revenue Determined through time-of-use pricing and interactive power:

[0109] ;

[0110] in, , for The time-of-use electricity price for MG's purchase and sale of electricity with the power grid during the same time period; For the interaction power between MG and the grid, when A time interval indicates that MG purchases electricity from the grid, while a time interval indicates that MG sells electricity to the grid.

[0111] Carbon emission costs Differentiate and express based on the source of electrical energy used or produced by MG:

[0112] ;

[0113] in, The unit cost of carbon emissions corresponding to MG's use of electricity from the main grid; The unit cost of carbon emissions when producing or using electricity from DG (different types of DG have different unit costs of carbon emissions). For DG in Actual power during the time period;

[0114] MG scheduling cost Energy storage cost of ESS MG inter-energy exchange cost Costs of curtailing wind and solar power express:

[0115] ;

[0116] ;

[0117] ;

[0118] ;

[0119] in, and These are the ESS discharge and charging power, respectively. The current power level of the ESS; The ESS motion loss coefficient; The self-dissipation coefficient of ESS; The energy dissipation cost coefficient for ESS (Energy Saving Service); The grid access service fee paid for power exchange between MGs using the power grid; Inter-MG power; The cost coefficient for wind and solar curtailment in MG; For DG in Maximum power during the time period;

[0120] MG's operation and maintenance costs The cost of wind power generation for each DG in MG Cost of photovoltaic power generation and the cost of micro gas turbine power generation composition:

[0121] ;

[0122] ;

[0123] ;

[0124] ;

[0125] in, This is the cost coefficient for wind power generation; This refers to the energy consumption coefficient of the wind turbine unit. This represents the maximum power output of the wind turbine. This represents the cost coefficient for photovoltaic power generation. , and This represents the power generation consumption coefficient of the micro gas turbine.

[0126] In a preferred embodiment of the present invention, consideration of power balance constraints, output constraints, ESS constraints, and interactive power constraints includes:

[0127] ;

[0128] in, For DG in Actual power during the time period and These are the maximum and minimum outputs of the DG. The ESS charging and discharging power in MG is the difference between the discharging power and the charging power. The interaction power between the MG and external systems includes the interaction power with the main network and the interaction power with other MGs. A positive value indicates the power transmitted from the MG to the outside, while a negative value indicates the power transmitted from the outside to the MG. For MG load; and For ESS The charging and discharging power during a given period; and This represents the maximum charging and discharging power of the ESS. For the first microgrid exist The state of charge (ESS) during a given time period; and These represent the minimum and maximum states of charge of the ESS. and for The minimum and maximum values.

[0129] At each time step, the energy storage dynamics of the ESS are modeled, including:

[0130] ;

[0131] ;

[0132] in, For the first microgrid The maximum energy storage capacity of the ESS in the system; The charge / discharge efficiency coefficient of the ESS;

[0133] exist and In the middle, the minimum value is determined by The minimum output power is defined as the difference between the minimum output power and the maximum load power, while the maximum output power is defined as the opposite:

[0134] ;

[0135] ;

[0136] Y2. Based on the microgrid group multi-agent model, Markov modeling is performed to obtain the Markov decision process. Y2 specifically includes:

[0137] Markov modeling is performed based on all constraints of the microgrid group multi-agent model to obtain the environment of the Markov decision process; the state space, action space, and reward function of the Markov decision process are obtained based on the microgrid group multi-agent model.

[0138] Time step in state space The actions include:

[0139] ;

[0140] in, For MGs, the set of states of charge of the energy storage system; For the load demand set of MGs; The distributed power output set for MGs; For real-time electricity prices; For time steps;

[0141] Time step in action space The actions include:

[0142] ;

[0143] in, The command sequence for ESS charging and discharging power; This is the DG output command sequence;

[0144] The reward function consists of positive rewards. With negative rewards Composition, including:

[0145] ;

[0146] ;

[0147] ;

[0148] in, This is the transaction reward coefficient; the electricity transaction reward is based on the electricity transaction revenue. Multiply get; To constrain the reward coefficient; The constraint indicator function is a 0-1 variable. This indicates that the agent is in a state of strictly enforcing constraints in the current state, and a reward is given. , This indicates that the agent is in a state where the constraints are not strictly enforced; The energy storage reward factor is calculated by multiplying the stored electrical energy within each time step of the ESS by [a certain factor]. Provide energy storage incentives; As a cost penalty coefficient, the carbon emission cost, scheduling cost, and operation and maintenance cost of MGs during operation are multiplied by [a certain factor]. Reflected in negative rewards middle; To constrain the penalty coefficient; The unbalanced power generated during the agent's decision-making process, multiplied by Punishment shall be imposed.

[0149] Y3. Based on the Markov decision process and the first algorithm, the scheduling optimization of the microgrid group is performed to obtain the joint scheduling scheme of the microgrid group. The first algorithm is based on the DDQN algorithm, and illegal actions are masked through action masking at each time step of the DDQN algorithm.

[0150] In a preferred embodiment of the present invention, the microgrid group is optimized for scheduling based on a Markov decision process combined with a first algorithm, resulting in a joint scheduling scheme for the microgrid group, including:

[0151] In each time step, the action mask is first calculated based on the current state and action space of the Markov decision process, and the action space under the current state is masked according to the action mask. Then, the Markov decision process is optimized according to the DDQN algorithm. After completing the optimization iteration of all time steps, the joint scheduling scheme of the microgrid group is obtained. The joint scheduling scheme of the microgrid group includes the DG output command and ESS charging and discharging power command of each microgrid in the microgrid group.

[0152] In a preferred embodiment of the present invention, calculating an action mask based on the current state and action space in the Markov decision process, and masking the action space in the current state based on the action mask includes:

[0153] The agent in a Markov decision process obtains the current time step from the environment. Current state And based on the current state Calculate the current state using the action space. The following are legal action sequences With action mask :

[0154] ;

[0155] ;

[0156] Based on action mask Current state Next action of Assign values:

[0157] ;

[0158] in, For state s t Constraints of the multi-agent model for microgrid clusters; action mask It is an array with the same specifications as the action space, and each element corresponds one-to-one with an action in the action space; Current state Next action corresponding value, This represents the i-th action in the action space; when the address mask value corresponding to the action is 0, the action is illegal, and its corresponding Q value is assigned a value of 0. This ensures that the agent does not choose illegal actions during the exploration process. When the address mask value corresponding to an action is 1, it means that the action is legal and its corresponding value is not changed. value.

[0159] Action mask Import the training and target networks of the DDQN algorithm.

[0160] Assuming in state Unexplored violations account for a portion of the total unexplored action space. ( And the agent has already discovered the action with the optimal Q value, because The greedy strategy still works even without setting an action mask. The probability of selecting an illegal action is reduced to 0 after setting an action mask, thus avoiding the risk of the agent selecting illegal actions and greatly improving the security of reinforcement learning in microgrid systems.

[0161] ;

[0162] in, For state The set of violations was not explored.

[0163] In a preferred embodiment of the present invention, the action mask can block out illegal actions through a mask, allowing the agent to strictly execute the constraints set in the model, while compressing the action space and effectively improving the efficiency of exploration and optimization. By introducing prior knowledge and restricting the action exploration area, the first algorithm can ensure that the agent strictly follows the model constraints when making each decision, thereby avoiding dangerous and non-compliant actions.

[0164] See Figure 2 In a preferred embodiment of the present invention, optimizing the Markov decision process according to the DDQN algorithm includes:

[0165] F1, based on the action mask and current state Combination A greedy strategy selects an action from the action space. ; Perform actions Environmental output reward and the next state ; the quadruple Store to the experience replay buffer;

[0166] F2. Randomly sample a set of experience samples from the experience playback buffer. ;

[0167] F3. Calculate the state through the target network. All actions below value And select the largest value According to the reward and Achieve the goal value :

[0168] ;

[0169] in, This is a discount factor used to balance immediate rewards and future rewards; These are the parameters of the target network;

[0170] If the state For the state containing the maximum time step, then ;

[0171] F4. Calculate the state by training the network. Next action Estimate value , Parameters for training the network;

[0172] F5, According to the target value and estimates value Calculate the loss function :

[0173] ;

[0174] in, For expectations;

[0175] F6. Based on the loss function Combine gradient descent algorithm to train network parameters Perform backpropagation updates; based on the preset update frequency. Regularly update the parameters of the training network Parameters synchronized to the target network ;

[0176] F7, if the current time step If the maximum time step is reached, the iterative optimization ends; otherwise, proceed to the next time step.

[0177] In a preferred embodiment of the present invention, the first algorithm significantly reduces the complexity of the action space, alleviates the dimensionality explosion problem, and improves the security of reinforcement learning applied to power systems.

[0178] This invention presents a microgrid group scheduling optimization method based on the DDQN algorithm and action masking. It constructs a multi-agent joint operation mode for microgrids (MGs), which, compared to independent operation of each MG, effectively improves the utilization rate of distributed generation (DG) while balancing the economic efficiency and low-carbon nature of MG operation. The first algorithm of this invention is better suited to handling complex problems than traditional reinforcement learning, improving upon the inability of traditional reinforcement learning to strictly enforce model constraints during training. Simultaneously, it compresses the solution space, resulting in better convergence stability and faster training speed, making it more suitable for application requirements in MG scenarios. This invention's method alleviates the difficulty in finding optimal solutions due to the curse of dimensionality in complex environments and improves the safety of applying reinforcement learning algorithms to MG scenarios. It solves the problem of energy coordination and optimization scheduling in MGs. This invention's method enables safe and efficient solution of MG scheduling strategies based on reinforcement learning.

[0179] The present invention also provides a microgrid group scheduling optimization device based on DDQN algorithm and action mask, which is used in the method of the present invention. The device includes a first module, a second module and a third module.

[0180] The first module is used to construct a multi-agent model of the microgrid group, treating each microgrid as an agent. The multi-agent model of the microgrid group aims to maximize the total operating benefits of the microgrid group and considers power balance constraints, output constraints, ESS constraints, and interactive power constraints. The second module is used to perform Markov modeling based on the multi-agent model of the microgrid group to obtain the Markov decision process. The third module is used to perform scheduling optimization of the microgrid group based on the Markov decision process and the first algorithm to obtain the joint scheduling scheme of the microgrid group. The first algorithm is based on the DDQN algorithm and uses an action mask to block illegal actions at each time step of the DDQN algorithm.

[0181] The microgrid group scheduling optimization device based on the DDQN algorithm and action mask of the present invention, used in the method of the present invention, has the same beneficial effects as the method of the present invention.

[0182] Verification section:

[0183] See Figure 3This is the topology diagram of the MGs model simulated in this embodiment, consisting of four MGs. MG1 operates on a self-consumption, surplus power-to-grid mode, where the electricity generated by the DG is prioritized for use by the MG's internal loads. When the generated power exceeds local consumption capacity, the excess power is automatically fed into the grid. MG2 and MG3 operate on a fully self-consumption mode, where the electricity generated by the DG is only used by the MG's internal loads and is not fed into the grid. When the electricity generated by the DG is insufficient to supply its own loads, the grid supplements the shortfall. MG4 operates on a full grid connection mode, where all the electricity generated by the DG is directly fed into the grid, and users meet their electricity needs by purchasing electricity. Simultaneously, the MGs as a whole adhere to the principle of local power consumption. Load represents the load, PV represents photovoltaic power, WG represents wind power, and MGT represents a micro gas turbine.

[0184] The aforementioned microgrids (MGs) are equipped with distributed generators (DGs) that produce different amounts of energy depending on weather conditions, and each MG is configured with an energy storage system (ESS) to meet the peak power demand of the load. MG1 and MG3 are microgrids containing DGs and loads, with the DGs primarily generating photovoltaic power (output period 06:00-18:00), MG2 being a load-type microgrid, and MG4 being a power-type microgrid primarily generating wind power (output exhibits a high nighttime and low daytime characteristic). The relevant parameters of the DGs are shown in Table 1, and the parameters of the ESSs are shown in Table 2. This study investigates the online economic dispatch problem of MGs based on sequence data using deep reinforcement learning. Real production data of photovoltaic and wind turbines and load electricity consumption in a certain province and city were used as the training set. The real-time dispatch interval was set to 1 hour. Some of the MGs' power generation and consumption data are shown in Table 2. Figure 4 As shown in Table 3, the transaction between the power grid and the MG uses time-of-use pricing. MGs need to pay a grid transfer fee of 0.05 yuan / kWh to transfer electricity through the power grid. The example uses electricity market data from a certain province and city, and the time-of-use pricing is shown in Table 3.

[0185] Table 1. Parameters of each DG

[0186] ;

[0187] Table 2 ESS Parameters

[0188] ;

[0189] Table 3 Time-of-use electricity prices

[0190] ;

[0191] This invention runs on Python 3.12, with the environment deployed on Gymnasium. A DNN was written using PyTorch, and the hardware configuration consisted of an Intel(R) Core(TM) i5-13400F processor, an RTX4070 graphics card, and 32GB of RAM. The hyperparameter settings of the first algorithm are shown in Table 4.

[0192] Table 4 Algorithm Hyperparameter Settings

[0193] ;

[0194] To explore the behavioral patterns of the interaction between mining services (MGs) and the electricity market, and to study the advantages of joint operation of MGs over individual operation of MGs, two operation schemes were designed and analyzed:

[0195] Option 1: Each MG runs independently.

[0196] Option 2: Each MG is combined into MGs for joint operation.

[0197] When the model operates under Scheme 1, each MG runs independently without exchanging power with others, and the optimization objective for each MG is to improve its own operational benefits; the energy scheduling results for Scheme 1 can be found in [link to Scheme 1]. Figures 5 to 8 The results are for MG1 to MG4, respectively.

[0198] Under Operation Scheme 1 (independent operation of each MG): From 0:00 to 7:00, because the DG output of MG1, MG2, and MG3 is less than the load demand, each MG needs to purchase electricity from the main grid. Although MG4 has higher wind power output during this period, because each MG operates in independent mode, MG4 chooses its own optimal economic strategy based on the time-of-use price and does not cooperate with other MGs. At the same time, to cope with the power shortage during high-price periods, the energy storage (ESS) of each MG stores electricity during low-price periods. From 7:00 to 14:00, the photovoltaic output of MG1 is insufficient to support the full load demand and needs to rely on ESS discharge and external power purchase. The photovoltaic output of MG3 is sufficient, and with the help of the ESS, it can meet the needs of MG3 and store surplus electricity. Afterwards, each ESS chooses to continue storing electricity and selectively discharges during the second-highest price period to reduce electricity costs. From 14:00 to 18:00, the photovoltaic output of MG1 and MG3 gradually decreases to zero. During this period, each ESS stores electricity to cope with the upcoming period of high electricity prices. From 18:00 to 23:00, the price of electricity purchased by MG from external sources is relatively high. During the period of high electricity prices, each ESS releases electricity to supply MG. However, due to the limitation of energy storage capacity, the stored electricity is still insufficient to cover MG's electricity demand during the period of high prices. A large amount of electricity still needs to be purchased from the grid, which is less economical.

[0199] When the model operates under Scheme 2, the various MGs form MGs that operate jointly, and the MGs can exchange power through the distribution network; the energy dispatch results of Scheme 2 are shown below, where... Figures 9 to 12 The energy scheduling strategies for each MG in Scheme 2 correspond to MG1 to MG4 respectively; Figure 13 The overall energy scheduling strategy for MGs in Scheme 2.

[0200] Compared to Operation Scheme 1, under Operation Scheme 2 (where each MG forms an MGs system operating jointly): From 0:00 to 7:00, MG4's high wind power output can be directly supplied to MGs experiencing power shortages, and excess energy can be stored in the Energy Storage System (ESS), reducing the overall electricity purchase cost of the MGs system. From 7:00 to 14:00, MG4's wind power output is supplied to MG1, MG2, and MG3 to meet the load demand during this period, while the ESS maintains an energy dispatch strategy similar to that of Scheme 1. From 14:00 to 18:00, each MG exchanges power based on overall economic efficiency. For example, MG2 stores the wind power generated by MG4 through the ESS from 16:00 to 17:00, which can increase revenue in the next period compared to Scheme 1 where wind power is directly fed into the grid. During the period from 18:00 to 23:00, in Option 1, MG4 aims to quickly profit by discharging the ESS early during periods of high electricity prices to sell electricity to the grid. In Option 2, to balance the overall load demand within the microgrid group, MG4 chooses to delay discharging the ESS until other MGs have exhausted their energy storage capacity, thereby reducing the overall electricity cost of MGs from a system perspective.

[0201] The carbon emissions, scheduling costs, and operational benefits of each operating scheme are compared and analyzed, as shown in Table 5.

[0202] Table 5. Comprehensive Comparison of Different Schemes

[0203] ;

[0204] In Operation Scheme 1, each MG relies solely on its own DG, which is insufficient to meet its energy needs and fails to achieve complementarity between different resources. MGs purchase a large amount of electricity from the grid, resulting in high carbon emissions and low overall operating benefits. In Operation Scheme 2, each MG forms MGs that share power generation resources. Through joint dispatch of MGs, the overall energy needs of MGs are basically met, achieving complementarity between different resources. MGs purchase less electricity from the grid, resulting in low carbon emissions and high overall operating benefits.

[0205] Quantitative comparison shows that, compared with Scheme 1, although the interaction of electrical energy between MGs incurs additional dispatch costs, Scheme 2 reduces carbon emissions by 60.66% and increases overall operational benefits by 6.65%. In summary, the above analysis demonstrates that Scheme 2 has significant advantages over Scheme 1 in several key indicators, and the joint operation of MGs by various MGs is a more efficient solution.

[0206] To further verify the superiority of the first algorithm of this invention over traditional reinforcement learning algorithms, the Q-learning algorithm and the D3QN algorithm were selected for comparison and solution. The algorithm performance and computational results were compared and analyzed. Detailed evaluations were conducted on six indicators: training time, memory usage, number of iterations, carbon emissions, operational benefits, and number of violations. The results of multiple training iterations were averaged, and a comparison chart of the algorithm iteration process was drawn. (See attached chart.) Figure 14 , Figure 14 Mask-DQN in the figure is the first algorithm of this invention. The shaded part in the figure represents the 95% confidence interval, and the solid line represents the average value of the training results. The results are shown in Table 6.

[0207] Table 6 Comparison of different algorithms

[0208] ;

[0209] This comparative simulation training was set up with 1600 iterations. The training curve analysis shows that the reward curve exhibits obvious fluctuations in the initial stage. This is attributed to the need for the agent to continuously try and fail in order to avoid getting trapped in local optima. As the training progresses, the reward curve gradually converges to the steady-state range.

[0210] The training results above show that, due to the large action space, the Q-learning algorithm suffers from difficulties in table lookup, extremely long computation time, and high memory consumption. Furthermore, the training curve oscillates and fails to converge, resulting in a large number of illegal actions. The D3QN algorithm, by introducing a two-layer adversarial DNN, improves the iterative computation speed and shortens the computation time compared to the Q-learning algorithm. However, it fails to mask invalid actions, resulting in wasted computational resources and a low starting point for training. Ultimately, it converges to a local optimum, and the resulting strategy still contains a small number of illegal actions. The first algorithm of this invention, through an action masking mechanism, can reduce the action space exploration area by 37.14%-83.85% under different states, reducing invalid exploration and achieving a high starting point in the early stages of training. It boasts the fastest computation speed, good convergence of the training curve, the lowest oscillation amplitude, and no illegal actions, demonstrating a significant improvement in overall performance compared to the two algorithms mentioned above.

[0211] In summary, the first algorithm of this invention outperforms traditional reinforcement learning algorithms in terms of computational efficiency and convergence, improving the operational benefits of MGs and reducing carbon emissions, while also enhancing the safety and availability of the scheduling strategy. In comparison, the first algorithm of this invention has shorter training time and stronger optimization decision-making capabilities, making it more suitable for solving energy scheduling strategies for MGs.

[0212] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A microgrid group scheduling optimization method based on the DDQN algorithm and action masking, characterized in that, include: Each microgrid is treated as an intelligent agent to construct a multi-agent model of the microgrid group; the multi-agent model of the microgrid group aims to maximize the total operating benefits of the microgrid group and considers power balance constraints, output constraints, ESS constraints and interactive power constraints. Markov modeling is performed based on the microgrid group multi-agent model to obtain the Markov decision process; Based on the Markov decision process and the first algorithm, the microgrid group is optimized for scheduling to obtain a joint scheduling scheme for the microgrid group. The first algorithm is based on the DDQN algorithm, and at each time step of the DDQN algorithm, it uses an action mask to block illegal actions, including: In each time step, an action mask is first calculated based on the current state and action space of the Markov decision process, and the action space under the current state is masked according to the action mask. Then, the Markov decision process is optimized according to the DDQN algorithm. After completing the optimization iteration of all time steps, the joint scheduling scheme of the microgrid group is obtained. The joint scheduling scheme of the microgrid group includes the DG output command and ESS charging and discharging power command of each microgrid in the microgrid group. Calculating an action mask based on the current state and action space in the Markov decision process, and masking actions in the action space under the current state based on the action mask, includes: The agent in the Markov decision process obtains the current time step from the environment. Current state And based on the current state Calculate the current state using the action space. The following are legal action sequences With action mask : ； ； According to the action mask Current state Next action Assign a value to the Q value: ； in, For state Constraints of the multi-agent model for microgrid clusters; action mask It is an array with the same specifications as the action space, and each element corresponds one-to-one with an action in the action space; Current state Next action The corresponding Q value, For the first in the action space One action; The action mask Import the training network and target network of the DDQN algorithm.

2. The microgrid group scheduling optimization method based on DDQN algorithm and action mask according to claim 1, characterized in that, Optimizing the Markov decision process using the DDQN algorithm includes: F1, based on the action mask and current state Combination A greedy strategy selects an action from the action space. ; Perform actions Environmental output reward and the next state ; the quadruple Store to the experience replay buffer; F2. Randomly sample a set of experience samples from the experience playback buffer. ; F3. Calculate the state through the target network. All actions below value And select the largest value According to the reward and Achieve the goal value : ； in, This is a discount factor used to balance immediate rewards and future rewards; These are the parameters of the target network; If the state For the state containing the maximum time step, then ; F4. Calculate the state by training the network. Next action Estimate value , Parameters for training the network; F5, According to the target value and estimates value Calculate the loss function : ； in, For expectations; F6. Based on the loss function Combine gradient descent algorithm to train network parameters Perform backpropagation updates; based on the preset update frequency. Regularly update the parameters of the training network Parameters synchronized to the target network ; F7, if the current time step If the maximum time step is reached, the iterative optimization ends; otherwise, proceed to the next time step.

3. The microgrid group scheduling optimization method based on DDQN algorithm and action mask according to claim 2, characterized in that, The microgrid cluster multi-agent model aims to maximize the total operational benefits of the microgrid cluster, including: The total operating revenue of the microgrid cluster includes revenue from electricity trading. Carbon emission costs Scheduling costs and MG operation and maintenance costs The scheduling cost Including ESS energy storage costs MG inter-energy exchange cost Costs of curtailing wind and solar power The objective function then includes: ； in, and Let j represent the set of MGs in MGs and the set of schedulable DGs in their corresponding MGs, respectively, where j represents the j-th MG; This refers to the set of time nodes in the joint scheduling strategy of MGs; , , and These are the weighting coefficients for different benefits and costs; For the ESS charge / discharge power of each MG, This represents the discharge power, and vice versa for the charging power; The DG power generation capacity of each MG, including wind power output. Photovoltaic power output and micro gas turbine output .

4. The microgrid group scheduling optimization method based on DDQN algorithm and action mask according to claim 3, characterized in that, The total operational benefits of the microgrid group specifically include: Electricity trading revenue Determined through time-of-use pricing and interactive power: ； in, , for The time-of-use electricity price for MG's purchase and sale of electricity with the power grid during the same time period; For the interaction power between MG and the grid, when A time interval indicates that MG purchases electricity from the grid, while a time interval indicates that MG sells electricity to the grid. Carbon emission costs Differentiate based on the source of electricity used or produced by MG: ； in, The unit cost of carbon emissions corresponding to MG's use of electricity from the main grid; The unit cost of carbon emissions when producing or using electricity from DG; For DG in Actual power during the time period; MG scheduling cost Energy storage cost of ESS MG inter-energy exchange cost Costs of curtailing wind and solar power express: ； ； ； ； in, and These are the ESS discharge and charging power, respectively. The current power level of the ESS; The ESS motion loss coefficient; The self-dissipation coefficient of ESS; The energy dissipation cost coefficient for ESS (Energy Saving Service); The grid access service fee paid for power exchange between MGs using the power grid; Inter-MG power; The cost coefficient for wind and solar curtailment in MG; For DG in Maximum power during the time period; MG's operation and maintenance costs The cost of wind power generation for each DG in MG Cost of photovoltaic power generation and the cost of micro gas turbine power generation composition: ； ； ； ； in, This is the cost coefficient for wind power generation; This refers to the energy consumption coefficient of the wind turbine unit. This represents the maximum power output of the wind turbine. This represents the cost coefficient for photovoltaic power generation. , and This represents the power generation consumption coefficient of the micro gas turbine.

5. The microgrid group scheduling optimization method based on DDQN algorithm and action mask according to claim 4, characterized in that, The consideration of power balance constraints, output constraints, ESS constraints, and interactive power constraints includes: ； in, For DG in Actual power during the time period and These are the maximum and minimum outputs of the DG. The ESS charging and discharging power in MG is the difference between the discharging power and the charging power. The interaction power between the MG and external systems includes the interaction power with the main network and the interaction power with other MGs. A positive value indicates the power transmitted from the MG to the outside, while a negative value indicates the power transmitted from the outside to the MG. For MG load; and For ESS The charging and discharging power during a given period; and This represents the maximum charging and discharging power of the ESS. For the j-th microgrid exist The state of charge (ESS) of a given time period; and These represent the minimum and maximum states of charge of the ESS. and for The minimum and maximum values.

6. The microgrid group scheduling optimization method based on DDQN algorithm and action mask according to claim 5, characterized in that, Also includes: At each time step, the energy storage dynamics of the ESS are modeled, including: ； ； in, For the j-th microgrid The maximum energy storage capacity of the ESS in the system; The charge / discharge efficiency coefficient of the ESS; exist and In the middle, the minimum value is determined by The minimum output power is defined as the difference between the minimum output power and the maximum load power, while the maximum output power is defined as the opposite: ； 。 7. The microgrid group scheduling optimization method based on DDQN algorithm and action mask according to claim 6, characterized in that, Based on the microgrid group multi-agent model, Markov modeling is performed to obtain the Markov decision process, which includes: Markov modeling is performed based on all constraints of the microgrid group multi-agent model to obtain the environment of the Markov decision process; the state space, action space, and reward function of the Markov decision process are obtained based on the microgrid group multi-agent model. The actions at time step t in the state space include: ； in, Let be the set of states of charge of the energy storage system of MGs; For the set of load requirements of MGs; The distributed power output set for MGs; For real-time electricity prices; For time steps; The actions at time step t in the action space include: ； in, The command sequence for ESS charging and discharging power; This is the DG output command sequence; The reward function consists of positive rewards. With negative rewards Composition, including: ； ； ； in, This is the transaction reward coefficient; the electricity transaction reward is based on the electricity transaction revenue. Multiply get; To constrain the reward coefficient; The constraint indicator function is a 0-1 variable. This indicates that the agent is in a state of strictly enforcing constraints in the current state, and a reward is given. , This indicates that the agent is in a state where the constraints are not strictly enforced; The energy storage reward factor is calculated by multiplying the stored electrical energy within each time step of the ESS by [a certain factor]. Provide energy storage incentives; As a cost penalty coefficient, the carbon emission cost, scheduling cost, and operation and maintenance cost of MGs during operation are multiplied by [a certain factor]. Reflected in negative rewards middle; To constrain the penalty coefficient; The unbalanced power generated during the agent's decision-making process, multiplied by Punishment shall be imposed.

8. A microgrid group scheduling optimization device based on the DDQN algorithm and action mask, used in the method described in any one of claims 1 to 7, characterized in that, The device includes a first module, a second module, and a third module; The first module is used to construct a multi-agent model of the microgrid group, treating each microgrid as an agent. The multi-agent model of the microgrid group aims to maximize the total operating benefits of the microgrid group and considers power balance constraints, output constraints, ESS constraints, and interactive power constraints. The second module is used to perform Markov modeling based on the multi-agent model of the microgrid group to obtain a Markov decision process. The third module is used to perform scheduling optimization of the microgrid group based on the Markov decision process and the first algorithm to obtain a joint scheduling scheme for the microgrid group. The first algorithm is based on the DDQN algorithm and uses an action mask to block illegal actions at each time step of the DDQN algorithm.

Images