Hybrid micro-grid system planning capacity configuration method and system based on double-layer optimization

Abstract

According to the micro-grid system planning capacity configuration method and system based on double-layer optimization, a deep Q network reinforcement learning method is adopted, and a capacity configuration scheme of each decision period is determined according to a state space and an action space; forming a capacity configuration scheme of the long-term planning period by the capacity configuration schemes of all decision periods in the long-term planning period; determining a capacity configuration scheme of each short-term planning period in the current decision period according to the target model by adopting a mathematical planning method under the constraint condition of the target model and the constraint of the upper limit of the configuration capacity of each short-term planning period in the current decision period; using the updated decision period reward function to update the economic index of the long-term planning period capacity configuration scheme; and when the updated economic index is smaller than the economic index before updating, feeding back the updated reward function of which the decision cycle is implemented so as to update the capacity configuration scheme of the long-term planning cycle, and solving the problem of planning capacity configuration of the micro-grid system under the uncertainty of multiple time scales.

Description

Technical Field

[0001] The present invention belongs to the technical field of microgrid system planning, and specifically, relates to a microgrid system planning capacity configuration method and system based on a double-layer optimization model, which is used for double-layer optimization of multi-period capacity investment planning of a hybrid microgrid (HM). Background Art

[0002] Given that the share of renewable energy in the global energy mix, led by wind power, is expected to rise dramatically in the coming decades, a systematic approach is needed to plan long-term investment strategies.

[0003] In the prior art, microgrid system planning cannot be performed over a long period, and there is dynamic uncertainty in the power trading market. Rapid temporal fluctuations in renewable energy further complicate the problem, which may lead to grid instability and unmet demand. The prior art combines energy storage devices and water electrolysis to produce hydrogen with energy management systems, but doing so further increases the complexity of the planning problem. The long-term capacity planning of hybrid microgrid systems is formulated as a multi-period stochastic decision problem that takes into account uncertainties occurring at multiple time scales. Long-term capacity decisions are inherently linked to energy dispatch and storage decisions occurring at short-term time scales and need to be satisfied simultaneously. However, as the size of the hybrid microgrid increases, the calculation becomes cumbersome. The two-stage stochastic programming (2SSP) model used in the prior art addresses the short-term uncertainty of weather and demand changes, so the two-stage stochastic programming model is short-term because it does not take into account the multi-period nature of capacity investment decisions and the impact of long-term uncertainty. Dynamic Movement Primitives (DMP) are also used in the prior art. Although the multi-period nature of capacity investment and changing economic factors are taken into account, it does not take into account the uncertainty of these factors. Therefore, when the assumed trajectory of economic factors is far from the actual trajectory, the results of the DMP method may deviate significantly from the actual situation. Summary of the invention

[0004] In order to address the deficiencies in the prior art, the present invention provides a hybrid microgrid system planning capacity configuration method and system based on double-layer optimization, which solves the problem of microgrid system planning capacity configuration under multi-time scale uncertainty, while meeting the requirements of stable energy management and maximizing the economy.

[0005] The present invention adopts the following technical solution.

[0006] The present invention proposes a capacity configuration method for microgrid system planning based on double-layer optimization, wherein the microgrid system planning includes a long-term planning cycle and a short-term planning cycle, wherein the facilities in the microgrid system include: wind turbines, energy storage equipment and water electrolyzers; including:

[0007] Set the length of the decision cycle; a long-term planning cycle includes multiple decision cycles, and a decision cycle includes multiple short-term planning cycles;

[0008] The state space of the decision cycle is constructed with the allowable operating capacity of each facility in the decision cycle, and the action space of the decision cycle is constructed with the allowable expansion of the allowable operating capacity of each facility in the decision cycle; the capacity allocation scheme of each decision cycle is determined according to the state space and action space using the deep Q network reinforcement learning method; the capacity allocation scheme of all decision cycles in the long-term planning cycle is used to form the capacity allocation scheme of the long-term planning cycle;

[0009] The total allowable operating capacity of each facility in the current decision cycle is constructed by the sum of the state space of the previous decision cycle and the action space of the current decision cycle; the total allowable operating capacity of each facility in the current decision cycle is taken as the upper limit of the capacity of each short-term planning cycle in the current decision cycle, and the maximum annual operating profit in the current decision cycle is taken as the target model; the mathematical programming method is used to determine the capacity configuration plan of each short-term planning cycle in the current decision cycle according to the target model under the constraints of the target model and the upper limit of the configuration capacity of each short-term planning cycle in the current decision cycle;

[0010] After the capacity allocation plan of each short-term planning cycle is implemented, the actual value of power production in the decision cycle is composed of the actual value of power production in each short-term planning cycle within a decision cycle; the reward function of the decision cycle is updated according to the actual value of power production in the decision cycle; the economic index of the capacity allocation plan of the long-term planning cycle is updated by using the updated reward function of the decision cycle; when the updated economic index is less than the economic index before the update, the updated reward function of the decision cycle in which the capacity allocation plan has been implemented is fed back to update the capacity allocation plan of the long-term planning cycle.

[0011] Preferably, the length of the decision cycle is set to N years, and the long-term planning cycle includes W decision cycles, then the length of the long-term planning cycle is N×W years; one decision cycle includes m short-term planning cycles, and the length of the short-term planning cycle is W / m years.

[0012] Preferably, the state space of the decision cycle is constructed with the allowable operating capacity of each facility in the decision cycle, satisfying the following relationship:

[0013] S P-1 =[s T,P-1 s B,P-1 sW,P-1 ] T (1)

[0014] In the formula, S P-1 is the state space of the P-1th decision cycle, s T,P-1 、s B,P-1 、s W,P-1 They are respectively the allowable operating capacity of the wind turbine, the allowable operating capacity of the energy storage device, and the allowable operating capacity of the water electrolyzer in the P-1th decision cycle.

[0015] Preferably, the action space of the decision cycle is constructed by the allowable expansion amount of the allowable operating capacity of each facility in the decision cycle, satisfying the following relationship:

[0016] A P =[ΔX T,P ΔX B,P ΔX W,P ] T (2)

[0017] In the formula, A P is the action space of the Pth decision cycle, ΔX T,P , ΔX B,P , ΔX W,P They are respectively the allowable expansion amount of the allowable operating capacity of the wind turbine in the Pth decision cycle, the allowable expansion amount of the allowable operating capacity of the energy storage device, and the allowable expansion amount of the allowable operating capacity of the water electrolyzer.

[0018] Preferably, a deep Q network reinforcement learning method is used to determine the capacity configuration scheme of each decision cycle according to the state space and action space, including:

[0019] 1) According to the state space S of the P-1th decision cycle P-1 and action space A P-1 Set the initial Q value Q of the facility in state s and action a (s,a) , where s∈S P-1 , a∈A P-1 ;

[0020] 2) The state space S in the P-1th decision cycle P-1 Next, select and execute the action space A of the Pth decision cycle P , to obtain the state space S of the Pth decision cycle P and reward function;

[0021] The reward function of the Pth decision cycle satisfies the following relationship:

[0022]

[0023] In the formula, is the reward function of the facility in state s and action a in the Pth decision cycle, C total,P is the total installation cost of each facility in the Pth decision cycle, R op,P,y is the total operating income of each facility in the Pth decision cycle in the yth year, r is the discount rate, and N is the number of years in a decision cycle;

[0024] 3) Update the Q value according to the deep Q network reinforcement learning rule to satisfy the following relationship:

[0025]

[0026] In the formula, Q (s,a),P is the Q value in the Pth decision cycle, γ is the discount factor, satisfying The discount factor is used to discount the value of future rewards to the current value; P (s′|s,a) To perform action a∈A P Then from the current state s∈S P-1 Transfer to the updated state s′∈S P The probability of max a′ Q (s′,a′) To make Q (s′,a′) The action a′∈A corresponding to the maximum value P ;

[0027] 4) In the updated state s′, repeat steps 2) and 3) to iteratively update the Q value until a predetermined number of iterations is reached;

[0028] 5) In the Pth decision cycle, the action space corresponding to the maximum value of Q value in the state space is used as the capacity configured in the decision cycle, satisfying the following relationship:

[0029] π(s)=max a Q (s,a) (6)

[0030] Where π(s) is the capacity configured in the decision cycle.

[0031] Preferably, the total allowed operating capacity of each facility in the current decision cycle is constructed by the sum of the state space of the previous decision cycle and the action space of the current decision cycle, satisfying the following relationship:

[0032]

[0033] In the formula, TX T,P TX B,P TX W,PThey are respectively the total allowable operating capacity of wind turbines, the total allowable operating capacity of energy storage devices, and the total allowable operating capacity of water electrolyzers in the Pth decision cycle.

[0034] Preferably, the total allowed operating capacity of each facility in the current decision cycle is used as the upper limit of the capacity of each short-term planning cycle in the current decision cycle, and the maximum annual operating profit in the current decision cycle is used as the target model.

[0035] Satisfies the following relationship:

[0036] max∏ P,y =Rev P,y -C op,P,y -C main,P,y (8)

[0037] In the formula, Π P,y is the annual operating profit based on electricity price and operating cost in the yth year in the Pth decision cycle, Rev P,y is the total annual income in the yth year of the Pth decision cycle, C op,P,y is the annual total operating cost in the yth year of the Pth decision cycle, C main,P,y is the annual total maintenance cost in the yth year in the Pth decision cycle;

[0038] Among them, the total annual revenue includes electricity sales and hydrogen sales, which satisfies the following relationship:

[0039]

[0040] Where EPrice is the market price of electricity, ESupply P,y,t is the amount of electricity supplied to the market in period t in year y during the Pth decision cycle, H Price is the market price of hydrogen, and H Pr od P,y,t is the hydrogen supplied to the market in time period t in year y during the Pth decision cycle, where T is the total number of time periods;

[0041] The annual total operating cost satisfies the following relationship:

[0042]

[0043] In the formula, C penalty is the penalty cost for not meeting electricity demand, D P,y,t is the electricity demand in time period t in year y during the Pth decision cycle, E sup ply,P,y,t is the amount of electricity supplied in time period t in year y during the Pth decision cycle, C carbon is the carbon tax cost per unit of electricity, E purchase,P,y,t is the amount of electricity purchased from the main grid in time period t in the yth year of the Pth decision cycle, and T is the total number of time periods;

[0044] The total annual maintenance cost satisfies the following relationship:

[0045] C main,P,y =(TX j,P ×IC j,P ×Main C ofef j ) (11)

[0046] In the formula, TX j,P is the total allowed operating capacity of facilities of type j in the Pth decision cycle, IC j,P is the unit investment cost of facility type j in the Pth decision cycle, MaintCoef j is the maintenance cost coefficient of facility type j.

[0047] Preferably, the constraints of the target model include:

[0048] 1) Constraints on the total operating capacity of each facility

[0049] The total permissible operating capacity of the wind turbine generator satisfies the following relationship:

[0050] TX T,P =ΔX T,P +TX T,P-1 (12)

[0051] In the formula, TX T,P is the total allowed operating capacity of wind turbines in the Pth decision cycle, ΔX T,P is the allowable expansion of the wind turbine operating capacity in the Pth decision cycle, TX T,P-1 is the total allowed operating capacity of wind turbines in the P-1th decision cycle;

[0052] The total allowable operating capacity of the energy storage device and the water electrolyzer satisfies the following relationship:

[0053] TX B,P +TX W,P =TX B,P-1 +TX W,P-1 +ΔX B,P +ΔX W,P (13)

[0054] Where ΔX B,P is the allowable expansion of the operating capacity of the energy storage device in the Pth decision cycle, ΔX W,P is the allowable expansion of the operating capacity of the water electrolyzer in the Pth decision cycle, TX B,P-1 TX W,P-1 are the total allowed operating capacities of the energy storage device and the water electrolyzer in the P-1th decision cycle, respectively;

[0055] 2) The constraints of energy balance satisfy the following relationship:

[0056]

[0057] In the formula, GenPower P,t is the amount of electricity generated by the wind turbine in time period t during the Pth decision cycle, Eff grid ImportPower is the AC-to-AC conversion efficiency of the power grid. P,t is the amount of electricity purchased from the main grid during time period t in the Pth decision cycle, Discharge P,t Demand is the discharge amount of the energy storage device in time period t in the Pth decision cycle. P,t is the power demand in time period t in the Pth decision cycle, H 2 Prod new,P,t is the amount of hydrogen produced by the newly installed water electrolyzer in the Pth decision cycle during time period t, H 2 Prod existing,P,t is the amount of hydrogen produced by the previously installed water electrolyzer in time period t during the Pth decision cycle, Curtail P,t is the amount of power cut in the Pth decision cycle in time period t due to grid instability or unmet demand;

[0058] 3) The energy storage system constraints satisfy the following relationship:

[0059]

[0060] InitialSorage P -FinalStorage P =0 (16)

[0061] In the formula, ESS P,y,t+1 is the energy storage system status at the end of period t+1 in the yth year of the Pth decision cycle, ESS P,y,t is the state of the energy storage system at the beginning of time period t in the yth year of the Pth decision cycle, σ B is the self-discharge rate of the energy storage device, η AC / DC is the efficiency of the rectifier, η B is the discharge efficiency of the energy storage device, Charge P,y,t is the charging power in time period t in year y in the Pth decision cycle, Discharging P,y,t is the discharge power in time period t in year y in the Pth decision cycle, InitialStorage P and FinalStorage Pare the energy storage device power at the beginning and end of the Pth decision cycle, respectively, and Δt is the time increment;

[0062] 4) Supply and demand constraints satisfy the following relationship:

[0063] sup P,y,t ≤D P,y,t (17)

[0064] In the formula, sup P,y,t is the power supply in time period t in year y during the Pth decision cycle, D P,y,t is the electricity demand in time period t in year y during the Pth decision cycle.

[0065] Preferably, the capacity allocation scheme for the decision cycle includes a target value for power production of each facility within the decision cycle;

[0066] Capacity allocation plans for long-term planning periods, including power production targets for microgrid systems;

[0067] Capacity allocation plan for the short-term planning cycle, including power production target values ​​and dispatch instructions for each facility.

[0068] Preferably, the economic performance index of the capacity allocation scheme in the long-term planning period is updated by using the reward function of the updated decision period according to the following relationship:

[0069]

[0070] Where maxN PV is the economic index of the capacity configuration plan in the long-term planning period. To find the expected value of the function, P finish is the decision cycle number of the capacity allocation plan that has been implemented, P plan is the decision cycle number in which the capacity allocation plan has not yet been implemented. The updated reward function for the decision cycle where the capacity allocation scheme has been implemented, Reward function for decision cycles where the capacity allocation solution has not yet been implemented.

[0071] The present invention also proposes a microgrid system planning capacity configuration system based on double-layer optimization, wherein the microgrid system planning includes a long-term planning cycle and a short-term planning cycle, wherein the facilities in the microgrid system include: wind turbines, energy storage equipment and water electrolyzers; including:

[0072] The cycle setting module is used to set the length of the decision cycle; a long-term planning cycle includes multiple decision cycles, and a decision cycle includes multiple short-term planning cycles;

[0073] The long-term planning scheme configuration module is used to construct the state space of the decision cycle with the allowable operating capacity of each facility within the decision cycle, and to construct the action space of the decision cycle with the allowable expansion of the allowable operating capacity of each facility within the decision cycle; the capacity configuration scheme of each decision cycle is determined according to the state space and the action space using the deep Q network reinforcement learning method; the capacity configuration scheme of the long-term planning cycle is formed by the capacity configuration schemes of all decision cycles within the long-term planning cycle;

[0074] The short-term planning scheme configuration module is used to construct the total allowable operating capacity of each facility in the current decision cycle based on the sum of the state space of the previous decision cycle and the action space of the current decision cycle; the total allowable operating capacity of each facility in the current decision cycle is used as the upper limit of the capacity of each short-term planning cycle in the current decision cycle, and the maximum annual operating profit in the current decision cycle is used as the target model; the capacity configuration scheme of each short-term planning cycle in the current decision cycle is determined according to the target model using a mathematical programming method under the constraints of the target model and the upper limit of the configuration capacity of each short-term planning cycle in the current decision cycle;

[0075] The long-term planning scheme updating module is used for, after the capacity configuration scheme of each short-term planning cycle is implemented, the actual value of power production of the decision cycle is formed by the actual value of power production of each short-term planning cycle within a decision cycle; the reward function of the decision cycle is updated according to the actual value of power production of the decision cycle; the economic index of the capacity configuration scheme of the long-term planning cycle is updated by using the updated reward function of the decision cycle; when the updated economic index is less than the economic index before the update, the updated reward function of the decision cycle in which the capacity configuration scheme has been implemented is fed back to update the capacity configuration scheme of the long-term planning cycle.

[0076] Assume that the length of the decision cycle is N years, and the long-term planning cycle includes W decision cycles, then the length of the long-term planning cycle is N×W years; a decision cycle includes m short-term planning cycles, and the length of the short-term planning cycle is W / m years.

[0077] The state space of the decision cycle is constructed with the allowable operating capacity of each facility in the decision cycle, satisfying the following relationship:

[0078] S P-1 =[s T,P-1 s B,P-1 s W,P-1 ] T (1)

[0079] In the formula, S P-1 is the state space of the P-1th decision cycle, s T,P-1 、s B,P-1 、s W,P-1They are respectively the allowable operating capacity of the wind turbine, the allowable operating capacity of the energy storage device, and the allowable operating capacity of the water electrolyzer in the P-1th decision cycle.

[0080] The action space of the decision cycle is constructed by the allowable expansion of the allowable operating capacity of each facility in the decision cycle, satisfying the following relationship:

[0081] A P =[ΔX T,P ΔX B,P Δ XW,P ] T (2)

[0082] In the formula, A P is the action space of the Pth decision cycle, ΔX T,P , ΔX B,P , ΔX W,P They are respectively the allowable expansion amount of the allowable operating capacity of the wind turbine in the Pth decision cycle, the allowable expansion amount of the allowable operating capacity of the energy storage device, and the allowable expansion amount of the allowable operating capacity of the water electrolyzer.

[0083] The deep Q network reinforcement learning method is used to determine the capacity configuration scheme for each decision cycle based on the state space and action space, including:

[0084] 1) According to the state space S of the P-1th decision cycle P-1 and action space A P-1 Set the initial Q value Q of the facility in state s and action a (s,a) , where s∈S P-1 , a∈A P-1 ;

[0085] 2) The state space S in the P-1th decision cycle P-1 Next, select and execute the action space A of the Pth decision cycle P , to obtain the state space S of the Pth decision cycle P and reward function;

[0086] The reward function of the Pth decision cycle satisfies the following relationship:

[0087]

[0088] In the formula, is the reward function of the facility in state s and action a in the Pth decision cycle, C total,P is the total installation cost of each facility in the Pth decision cycle, R op,P,y is the total operating income of each facility in the Pth decision cycle in the yth year, r is the discount rate, and N is the number of years in a decision cycle;

[0089] 3) Update the Q value according to the deep Q network reinforcement learning rule to satisfy the following relationship:

[0090]

[0091] In the formula, Q (s,a),P is the Q value in the Pth decision cycle, γ is the discount factor, satisfying The discount factor is used to discount the value of future rewards to the current value; P (s′|s,a) To perform action a∈A P Then from the current state s∈S P-1 Transfer to the updated state s′∈S P The probability of max a′ Q (s′,a′) To make Q (s′,a′) The action a′∈A corresponding to the maximum value P ;

[0092] 4) In the updated state s′, repeat steps 2) and 3) to iteratively update the Q value until a predetermined number of iterations is reached;

[0093] 5) In the Pth decision cycle, the action space corresponding to the maximum value of Q value in the state space is used as the capacity configured in the decision cycle, satisfying the following relationship:

[0094] π(s)=max a Q (s,a) (6)

[0095] Where π(s) is the capacity configured in the decision cycle.

[0096] The sum of the state space of the previous decision cycle and the action space of the current decision cycle is used to construct the total allowable operating capacity of each facility in the current decision cycle, satisfying the following relationship:

[0097]

[0098] In the formula, TX T,P TX B,P TX W,P They are respectively the total allowable operating capacity of wind turbines, the total allowable operating capacity of energy storage devices, and the total allowable operating capacity of water electrolyzers in the Pth decision cycle.

[0099] The total allowed operating capacity of each facility in the current decision cycle is used as the upper limit of the capacity of each short-term planning cycle in the current decision cycle, and the maximum annual operating profit in the current decision cycle is used as the target model.

[0100] Satisfies the following relationship:

[0101] max∏ P,y =Re v P,y -C op,P,y -C main,P,y (8)

[0102] In the formula, Π P,y is the annual operating profit based on electricity price and operating cost in the yth year in the Pth decision cycle, Rev P,y is the total annual income in the yth year of the Pth decision cycle, C op,P,y is the annual total operating cost in the yth year of the Pth decision cycle, C main,P,y is the annual total maintenance cost in the yth year in the Pth decision cycle;

[0103] Among them, the total annual revenue includes electricity sales and hydrogen sales, which satisfies the following relationship:

[0104]

[0105] Where EPrice is the market price of electricity, ESupply P,y,t is the amount of electricity supplied to the market in period t in year y during the Pth decision cycle, H Price is the market price of hydrogen, and HProd P,y,t is the hydrogen supplied to the market in time period t in year y during the Pth decision cycle, where T is the total number of time periods;

[0106] The annual total operating cost satisfies the following relationship:

[0107]

[0108] In the formula, C penalty is the penalty cost for not meeting electricity demand, D P,y,t is the electricity demand in the tth year in the Pth decision cycle and in the time period t, E sup ply,P,y,t is the amount of electricity supplied in time period t in year y during the Pth decision cycle, C carbon is the carbon tax cost per unit of electricity, F purchase,P,y,t is the amount of electricity purchased from the main grid in time period t in the yth year of the Pth decision cycle, and T is the total number of time periods;

[0109] The total annual maintenance cost satisfies the following relationship:

[0110] C main,P,y =(TX j,P ×IC j,P ×Maint Coef j ) (11)

[0111] In the formula, TX j,Pis the total allowed operating capacity of facilities of type j in the Pth decision cycle, IC j,P is the unit investment cost of facility type j in the Pth decision cycle, MaintCoef j is the maintenance cost coefficient of facility type j.

[0112] The constraints of the target model include:

[0113] 1) Constraints on the total operating capacity of each facility

[0114] The total permissible operating capacity of the wind turbine generator satisfies the following relationship:

[0115] TX T,P =ΔX T,P +TX T,P-1 (12)

[0116] In the formula, TX T,P is the total allowed operating capacity of wind turbines in the Pth decision cycle, ΔX T,P is the allowable expansion of the wind turbine operating capacity in the Pth decision cycle, TX T,P-1 is the total allowed operating capacity of wind turbines in the P-1th decision cycle;

[0117] The total allowable operating capacity of the energy storage device and the water electrolyzer satisfies the following relationship:

[0118] TX B,P +TX W,P =TX B,P-1 +TX W,P-1 +ΔX B,P +ΔX W,P (13)

[0119] Where ΔX B,P is the allowable expansion of the operating capacity of the energy storage device in the Pth decision cycle, ΔX W,P is the allowable expansion of the operating capacity of the water electrolyzer in the Pth decision cycle, TX B,P-1 TX W,P-1 are the total allowed operating capacities of the energy storage device and the water electrolyzer in the P-1th decision cycle, respectively;

[0120] 2) The constraints of energy balance satisfy the following relationship:

[0121]

[0122] In the formula, GenPower P,t is the amount of electricity generated by the wind turbine in time period t during the Pth decision cycle, Eff gridImportPower is the AC-to-AC conversion efficiency of the power grid. P,t is the amount of electricity purchased from the main grid during time period t in the Pth decision cycle, Discharging P,t Demand is the discharge amount of the energy storage device in time period t in the Pth decision cycle. P,t is the power demand in time period t in the Pth decision cycle, H 2 Prod new,P,t is the amount of hydrogen produced by the newly installed water electrolyzer in the Pth decision cycle during time period t, H 2 Prod existing,P,t is the amount of hydrogen produced by the previously installed water electrolyzer in time period t during the Pth decision cycle, Curtail P,t is the amount of power cut in the Pth decision cycle in time period t due to grid instability or unmet demand;

[0123] 3) The energy storage system constraints satisfy the following relationship:

[0124]

[0125] IiitilStorage P -FinalStorage P =0 (16)

[0126] In the formula, ESS P,y,t+1 is the energy storage system status at the end of period t+1 in the yth year of the Pth decision cycle, ESS P,y,t is the state of the energy storage system at the beginning of time period t in the yth year of the Pth decision cycle, σ B is the self-discharge rate of the energy storage device, η AC / DC is the efficiency of the rectifier, η B is the discharge efficiency of the energy storage device, Charge P,y,t is the charging power in time period t in year y in the Pth decision cycle, Discharge P,y,t is the discharge power in time period t in year y in the Pth decision cycle, InitialStorage P and FinalStorage P are the energy storage device power at the beginning and end of the Pth decision cycle, respectively, and Δt is the time increment;

[0127] 4) Supply and demand constraints satisfy the following relationship:

[0128] sup P,y,t ≤D P,y,t (17)

[0129] In the formula, supP,y,t is the power supply in time period t in year y during the Pth decision cycle, D P,y,t is the electricity demand in time period t in year y during the Pth decision cycle.

[0130] The capacity allocation plan for the decision cycle, including the target value of power production for each facility during the decision cycle;

[0131] Capacity allocation plans for long-term planning periods, including power production targets for microgrid systems;

[0132] Capacity allocation plan for the short-term planning cycle, including power production target values ​​and dispatch instructions for each facility.

[0133] Using the reward function of the updated decision cycle, the economic indicators of the capacity allocation scheme in the long-term planning cycle are updated as follows:

[0134]

[0135] Where max N PV is the economic index of the capacity allocation plan in the long-term planning period. To find the expected value of the function, P finish is the decision cycle number of the capacity allocation plan that has been implemented, P plan is the decision cycle number in which the capacity allocation plan has not yet been implemented. The updated reward function for the decision cycle where the capacity allocation scheme has been implemented, Reward function for decision cycles where the capacity allocation solution has not yet been implemented.

[0136] A terminal comprises a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method.

[0137] A computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0138] The beneficial effects of the present invention are that, compared with the prior art, at least the coordination relationship between the long-term capacity configuration and the short-term capacity configuration proposed by the present invention is realized through a two-layer optimization framework, which combines the reinforcement learning (RL) method to make an upper-level decision on the long-term capacity configuration and the mathematical programming (MP) method to make a lower-level decision on the short-term capacity configuration. The upper-level decision sets capacity constraints for the lower-level decision, and the lower-level decision is optimized in each period based on the short-term uncertainty achieved. The coordination between the long-term capacity configuration and the short-term capacity configuration is ensured through this coupling relationship.

[0139] The method proposed in the present invention realizes multi-period coupled optimization of hybrid microgrid system capacity planning, and can also be applied to other similar multi-time scale optimization problems with random uncertainty. BRIEF DESCRIPTION OF THE DRAWINGS

[0140] Figure 1 It is a flow chart of a microgrid system planning capacity configuration method based on double-layer optimization proposed by the present invention. DETAILED DESCRIPTION

[0141] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only embodiments of a part of the present invention, rather than all embodiments. Based on the spirit of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention.

[0142] The present invention proposes a capacity configuration method for microgrid system planning based on double-layer optimization. The microgrid system planning includes a long-term planning cycle and a short-term planning cycle. The facilities in the microgrid system include: wind turbines, energy storage equipment and water electrolyzers; Figure 1 As shown, the method includes:

[0143] Step 1, set the length of the decision cycle; a long-term planning cycle includes multiple decision cycles, and a decision cycle includes multiple short-term planning cycles.

[0144] The length of the decision cycle is set to N years. In the embodiment, the length of a decision cycle is 5 years. The length of the decision cycle is adjusted according to the system planning scale requirements. The long-term planning cycle includes W decision cycles. In the embodiment, W=3, then the length of the long-term planning cycle is N×W=15 years. A decision cycle includes m short-term planning cycles. In the embodiment, m=5, then the length of the short-term planning cycle is W / m=1 year.

[0145] Step 2: Construct the state space of the decision cycle with the allowable operating capacity of each facility within the decision cycle, and construct the action space of the decision cycle with the allowable expansion of the allowable operating capacity of each facility within the decision cycle; use the deep Q network reinforcement learning method to determine the capacity configuration plan of each decision cycle according to the state space and action space; and construct the capacity configuration plan of the long-term planning cycle with the capacity configuration plans of all decision cycles within the long-term planning cycle.

[0146] The capacity allocation plan for the decision cycle includes but is not limited to the power production target value of each facility.

[0147] Capacity allocation plans for long-term planning periods, including but not limited to power production targets for microgrid systems.

[0148] Specifically, step 2 includes:

[0149] Step 2.1, construct the state space of the decision cycle with the allowed operating capacity of each facility in the decision cycle;

[0150] The state space represents the physical state and information state of the hybrid microgrid system. In any decision cycle, the following relationship is satisfied:

[0151] S P-1 =[s T,P-1 s B,P-1 s W,P-1 ] T (1)

[0152] In the formula, S P-1 is the state space of the P-1th decision cycle, s T,P-1 、s B,P-1 、s W,P-1 They are respectively the allowable operating capacity of the wind turbine, the allowable operating capacity of the energy storage device, and the allowable operating capacity of the water electrolyzer in the P-1th decision cycle.

[0153] The allowable operating capacity refers to the total capacity that can be used to generate or store energy during the decision cycle, taking into account the remaining life of installed facilities and the capacity of newly installed facilities. The allowable operating capacity determines the range of resources that can be dispatched during the decision cycle.

[0154] Step 2.2, constructing the action space of the decision cycle with the allowable expansion amount of the allowable operating capacity of each facility in the decision cycle;

[0155] In any decision cycle, the action space of the decision cycle is constructed with the allowable expansion of the operating capacity of each facility, satisfying the following relationship:

[0156] A P =[ΔX T,P ΔX B,P ΔX W,P ] T (2)

[0157] In the formula, A P is the action space of the Pth decision cycle, ΔX T,P , ΔX B,P , ΔX W,P They are respectively the allowable expansion amount of the allowable operating capacity of the wind turbine in the Pth decision cycle, the allowable expansion amount of the allowable operating capacity of the energy storage device, and the allowable expansion amount of the allowable operating capacity of the water electrolyzer.

[0158] The allowable expansion of operating capacity refers to the ability to expand existing energy facilities within the decision cycle based on current energy demand, technology development trends, economic factors and environmental uncertainties. The allowable expansion is determined by considering uncertainties and decision-making processes at multiple time scales within the long-term planning framework.

[0159] Step 2.3, using the deep Q network reinforcement learning method, the capacity configuration scheme for each decision cycle is determined based on the state space and action space, including:

[0160] 1) According to the state space S of the P-1th decision cycle P-1 and action space A P-1 Set the initial Q value Q of the facility in state s and action a (s,a) , where s∈S P-1 , a∈A P-1 ;

[0161] 2) The state space S in the P-1th decision cycle P-1 Next, select and execute the action space A of the Pth decision cycle P , to obtain the state space S of the Pth decision cycle P and reward function;

[0162] The reward function is used to evaluate the long-term benefits of taking a specific action under a given state. It can consider multiple objectives such as cost, benefit, risk, etc. at the same time, realize multi-objective optimization coupling, and introduce the reward function as a feedback value to dynamically adjust the Q value to find the optimal strategy;

[0163] The reward function of the Pth decision cycle satisfies the following relationship:

[0164]

[0165] In the formula, is the reward function of the facility in state s and action a in the Pth decision cycle, C total,P is the total installation cost of each facility in the Pth decision cycle, R op,P,y is the total operating income of each facility in the Pth decision cycle in the yth year, r is the discount rate, and N is the number of years in a decision cycle.

[0166] Among them, the total installation cost of the facility in the Pth decision cycle is C total,P Satisfies the following relationship:

[0167] C total,P =∑ j [C unit,j,P ×Q j,P ] (4)

[0168] In the formula, C unit,j,P is the installation cost of a facility of type j in the Pth decision cycle, Q j,P is the number of installations of type j in the Pth decision cycle, where j=T represents a wind turbine, j=B represents an energy storage device, and j=W represents a water electrolyzer.

[0169] 3) Update the Q value according to the deep Q network reinforcement learning rule to satisfy the following relationship:

[0170]

[0171] In the formula, Q (s,a),P is the Q value in the Pth decision cycle, γ is the discount factor, satisfying The discount factor is used to discount the value of future rewards to the current value; P (s′|s,a) To perform action a∈A P Then from the current state s∈S P-1 Transfer to the updated state s′∈S P The probability of max a′ Q (s′,a′) To make Q (s′,a′) The action a′∈A corresponding to the maximum value P .

[0172] The reward function is introduced as a feedback value in the Q-value update process to determine the optimal Q-value by coupling the configuration capacity with economic indicators.

[0173] 4) In the updated state s′, steps 2) and 3) are repeated to iteratively update the Q value until a predetermined number of iterations is reached.

[0174] 5) In the Pth decision cycle, the action space corresponding to the maximum value of Q value in the state space is used as the capacity configured in the decision cycle, satisfying the following relationship:

[0175] π(s)=max a Q (s,a) (6)

[0176] Where π(s) is the capacity configured in the decision cycle.

[0177] Step 2.4, the capacity allocation plan of the long-term planning period is formed by the capacity allocation plans of all decision cycles within the long-term planning period.

[0178] Step 3, construct the total allowable operating capacity of each facility in the current decision cycle by the sum of the state space of the previous decision cycle and the action space of the current decision cycle; take the total allowable operating capacity of each facility in the current decision cycle as the upper limit of the capacity of each short-term planning cycle in the current decision cycle, and take the maximum annual operating profit in the current decision cycle as the target model; use mathematical programming methods to determine the capacity configuration plan of each short-term planning cycle in the current decision cycle according to the target model under the constraints of the target model and the upper limit of the configuration capacity of each short-term planning cycle in the current decision cycle.

[0179] Capacity allocation plans for short-term planning cycles, including but not limited to power production targets and dispatch instructions for each facility.

[0180] The present invention realizes an upper-layer optimization of the capacity of the long-term planning cycle configuration in step 2, and the capacity of the long-term planning cycle configuration obtained after the upper-layer optimization becomes the feasible space for the lower-layer optimization.

[0181] Specifically, step 3 includes:

[0182] Step 3.1: The sum of the state space of the P-1th decision cycle and the action space of the Pth decision cycle is taken as the total allowable operating capacity of each facility in the Pth decision cycle, satisfying the following relationship:

[0183]

[0184] In the formula, TX T,P TX B,P TX W,P They are respectively the total allowable operating capacity of wind turbines, the total allowable operating capacity of energy storage devices, and the total allowable operating capacity of water electrolyzers in the Pth decision cycle.

[0185] Step 3.2, the total allowed operating capacity of each facility in the Pth decision cycle is used as the upper limit of the configuration capacity of each short-term planning period in the Pth decision cycle, and the maximum annual operating profit in the Pth decision cycle is used as the target model, satisfying the following relationship:

[0186] max∏ P,y =Rev P,y -C op,P,y -C main,P,y (8)

[0187] In the formula, Π P,y is the annual operating profit based on electricity price and operating cost in the yth year in the Pth decision cycle, Rev P,y is the total annual income in the yth year of the Pth decision cycle, C op,P,y is the annual total operating cost in the yth year of the Pth decision cycle, C main,P,yis the total annual maintenance cost in the yth year in the Pth decision cycle.

[0188] Among them, the total annual revenue includes electricity sales and hydrogen sales, which satisfies the following relationship:

[0189]

[0190] Where EPrice is the market price of electricity, ESupply P,y,t is the amount of electricity supplied to the market in period t in year y during the Pth decision cycle, H Price is the market price of hydrogen, and H Pr od P,y,t is the hydrogen supplied to the market in time period t in year y during the Pth decision cycle, where T is the total number of time periods;

[0191] The annual total operating cost satisfies the following relationship:

[0192]

[0193] In the formula, C penalty is the penalty cost for not meeting electricity demand, D P,y,t is the electricity demand in time period t in year y during the Pth decision cycle, E sup ply,P,y,t is the amount of electricity supplied in time period t in year y during the Pth decision cycle, C carbon is the carbon tax cost per unit of electricity, E purchase,P,y,t is the amount of electricity purchased from the main grid in time period t in the yth year of the Pth decision cycle, and T is the total number of time periods;

[0194] The total annual maintenance cost satisfies the following relationship:

[0195] C main,P,y =(TX j,P ×IC j,P ×MaintCoef j ) (11)

[0196] In the formula, TX j,P is the total allowed operating capacity of facilities of type j in the Pth decision cycle, IC j,P is the unit investment cost of facility type j in the Pth decision cycle, MaintCoef j is the maintenance cost coefficient of facility type j.

[0197] Step 3.3, establish the constraints of the target model;

[0198] Based on real-time data and forecast information, in order to ensure the balance between power supply and power demand, a set of constraints are required to complete the optimization problem of the target model. The constraints of the target model include:

[0199] 1) Constraints on the total operating capacity of each facility

[0200] The total permissible operating capacity of the wind turbine generator satisfies the following relationship:

[0201] TX T,P =ΔX T,P +TX T,P-1 (12)

[0202] In the formula, TX T,P is the total allowed operating capacity of wind turbines in the Pth decision cycle, ΔX T,P is the allowable expansion of the wind turbine operating capacity in the Pth decision cycle, TX T,P-1 is the total allowed operating capacity of wind turbines in the P-1th decision cycle.

[0203] The total allowable operating capacity of the energy storage device and the water electrolyzer satisfies the following relationship:

[0204] TX B,P +TX W,P =TX B,P-1 +TX W,P-1 +ΔX B,P +ΔX W,P (13)

[0205] Where ΔX B,P is the allowable expansion of the operating capacity of the energy storage device in the Pth decision cycle, ΔX W,P is the allowable expansion of the operating capacity of the water electrolyzer in the Pth decision cycle, TX B,P-1 TX W,P-1 are the total allowed operating capacities of the energy storage equipment and the water electrolyzer in the P-1th decision cycle, respectively.

[0206] 2) The constraints of energy balance satisfy the following relationship:

[0207]

[0208] In the formula, GenPower P,t is the amount of electricity generated by the wind turbine in time period t during the Pth decision cycle, Eff grid ImportPower is the AC-to-AC conversion efficiency of the power grid. P,t is the amount of electricity purchased from the main grid during time period t in the Pth decision cycle, Discharging P,t Demand is the discharge amount of the energy storage device in time period t in the Pth decision cycle. P,t is the power demand in time period t in the Pth decision cycle, H2 Prod new,P,t is the amount of hydrogen produced by the newly installed water electrolyzer in the Pth decision cycle during time period t, H 2 Prod existing,P,t is the amount of hydrogen produced by the previously installed water electrolyzer in time period t during the Pth decision cycle, Curtail P,t is the amount of power cut in the Pth decision cycle during time period t due to grid instability or unmet demand.

[0209] Energy balance constraints ensure the balance between power supply and demand and are a key component in the microgrid energy management system (EMS).

[0210] 3) The energy storage system constraints satisfy the following relationship:

[0211]

[0212] InitialStorage P -FinalStorage P =0 (16)

[0213] In the formula, ESS P,y,t+1 is the energy storage system status at the end of period t+1 in the yth year of the Pth decision cycle, ESS P,y,t is the state of the energy storage system at the beginning of time period t in the yth year of the Pth decision cycle, σ B is the self-discharge rate of the energy storage device, η AC / DC is the efficiency of the rectifier, η B is the discharge efficiency of the energy storage device, Charge P,y,t is the charging power in time period t in year y in the Pth decision cycle, Discharg ep ,y,t is the discharge power in time period t in year y in the Pth decision cycle, InitialStorage P and FinalStorage P They are the energy storage device power at the beginning and end of the Pth decision cycle respectively. Δt is the time increment, which represents the time step in the battery charging and discharging calculation, usually in hours.

[0214] 4) In order to prevent oversupply of a given demand, supply and demand constraints are set to satisfy the following relationship:

[0215] sup P,y,t ≤D P,y,t (17)

[0216] In the formula, sup P,y,tis the power supply in time period t in year y during the Pth decision cycle, D P,y,t is the electricity demand in time period t in year y during the Pth decision cycle.

[0217] Step 3.4, using mathematical programming methods, under the constraints of the target model and the upper limit of the capacity configuration of each short-term planning period in the current decision cycle, the configuration capacity of each short-term planning period in the current decision cycle is determined according to the target model.

[0218] Long-term capacity decisions are integrated with short-term energy scheduling and storage decisions to solve these problems at the same time, rather than separately. This integrated approach can more effectively optimize the performance of the overall system. Specifically, after the upper-level decision determines the long-term capacity configuration plan, the lower-level decision uses a mathematical programming (MP) method to formulate a short-term capacity configuration plan. In the embodiment, a short-term goal of maximizing the annual operating profit is first formulated, followed by the calculation of the total annual revenue, total operating costs and total maintenance costs, and finally the annual operating profit is calculated in combination with the above revenue and costs, and the mathematical programming problem is solved to determine the optimal short-term configuration plan. The short-term planning cycle configures capacity, including but not limited to hourly scheduling decisions, such as wind turbine power generation and wind curtailment, energy storage equipment charging and discharging, and whether the water electrolyzer starts electrolyzing water to produce hydrogen.

[0219] The current total allowable operating capacity is determined based on the long-term capacity allocation plan, which is derived through reinforcement learning and takes into account environmental uncertainties such as investment costs and the scale of electricity demand. The long-term capacity allocation plan involves the capacity configuration of each facility (such as wind turbines, energy storage equipment, and water electrolyzers), which is based on a comprehensive consideration of factors such as forecasts of future electricity growth demand and the remaining life of existing facilities. At the beginning of each decision cycle, the current total allowable operating capacity determined according to the long-term capacity allocation plan is the total capacity that can be used for power generation or storage during that cycle. The current total allowable operating capacity refers to the total capacity that can be used for power generation or storage during the short-term operating cycle, taking into account the remaining life of installed facilities and the capacity of newly installed facilities.

[0220] Initializing the short-term operation plan includes determining the current total allowed operating capacity, setting the initial state of the short-term operation cycle, formulating and implementing energy management strategies based on real-time data and forecast information during the short-term operation cycle, and then starting to implement the short-term operation plan. During the cycle, performance is continuously monitored to ensure the effective implementation of the strategy and make adjustments as needed. The execution results of short-term operations will be fed back to long-term planning to guide future long-term capacity configuration decisions, forming a dynamic optimization and adjustment process.

[0221] The present invention has a different logical order from the existing power grid planning ideas. Traditional power grid planning usually adopts a top-down method, first directly determining the long-term planning capacity, and then decomposing it into short-term planning capacity, so as to realize that the short-term planning capacity is constrained by the long-term planning capacity. The present invention adopts a two-layer optimization framework, in which the capacity configured for each decision cycle determined by the upper decision stage (i.e., the long-term planning cycle) using the reinforcement learning method based on the deep Q network is expressed as the optimal state space and the optimal action space in each decision cycle, and the sum of the capacity configured for each decision cycle constitutes the capacity configured for the long-term planning cycle, and then the capacity configured for the long-term planning cycle is used as the capacity constraint of the lower decision stage (i.e., the short-term planning cycle). Therefore, the optimal state space and the optimal action space in each decision cycle jointly determine the feasible space of the short-term planning cycle. The feasible space provides the boundaries of the physical state and the information state for the short-term planning cycle. In the feasible space, a mathematical programming method is used to optimize the capacity for the uncertainty factors in each short-term planning cycle, so as to determine the capacity configured for the short-term planning cycle.

[0222] Step 4, after the capacity allocation plan of each short-term planning cycle is implemented, the actual value of power production in the decision cycle is formed by the actual value of power production in each short-term planning cycle within a decision cycle; the reward function of the decision cycle is updated according to the actual value of power production in the decision cycle; the economic index of the capacity allocation plan of the long-term planning cycle is updated by using the updated reward function of the decision cycle; when the updated economic index is less than the economic index before the update, the updated reward function of the decision cycle in which the capacity allocation plan has been implemented is fed back to step 2 to update the capacity allocation plan of the long-term planning cycle.

[0223] Specifically, step 4 includes:

[0224] Step 4.1: After the capacity allocation scheme of each short-term planning period is implemented, the actual value of power production in the decision period is formed by the actual value of power production in each short-term planning period within a decision period.

[0225] Step 4.2, update the reward function of the decision cycle according to the actual value of the power production in the decision cycle.

[0226] Specifically, the total operating income R of each facility in the decision cycle is calculated based on the actual value of power production in the Pth decision cycle: op,P,y , thereby updating the reward function of the Pth decision cycle to satisfy the following relationship:

[0227]

[0228] In the formula, is the reward function of the facility in state s and action a in the Pth decision cycle after update, C total,Pis the total installation cost of each facility in the Pth decision cycle, is the updated total operating income of each facility in the Pth decision cycle in the yth year, r is the discount rate, and N is the number of years in a decision cycle.

[0229] Step 4.3, using the reward function of the updated decision cycle, update the economic index of the capacity allocation plan in the long-term planning cycle with the following relationship:

[0230]

[0231] Where max N PV is the economic index of the capacity allocation plan in the long-term planning period. To find the expected value of the function, P finish is the decision cycle number of the capacity allocation plan that has been implemented, P plan is the decision cycle number in which the capacity allocation plan has not yet been implemented. The updated reward function for the decision cycle where the capacity allocation scheme has been implemented, Reward function for decision cycles where the capacity allocation solution has not yet been implemented.

[0232] The economic index represents the maximization of the net present value of the capacity configuration scheme in the long-term planning cycle, and is the average value of the weighted sum of the reward functions of each decision cycle; the reward function of the decision cycle updated according to the actual value of power output in the decision cycle not only represents the influence of uncertainty and randomness of renewable energy output in the microgrid system, but also reflects the technical and economic impacts brought about by the scheduling decision. The reward function determined by the power output target value of the decision cycle is an important factor in Q value optimization and a key indicator of the capacity configuration scheme in the decision cycle. The weighted sum average of the two is a combination of the actual execution effect and the expected effect of the plan. Therefore, the economic index is updated in time after the implementation of each decision cycle, and the updated reward function of the decision cycle in which the capacity configuration scheme has been implemented is used as the feedback value. The deep Q network reinforcement learning method is used again to update the capacity configuration scheme of each decision cycle according to the state space and action space, thereby updating the capacity configuration scheme of the long-term planning cycle and achieving the coordination and consistency between the capacity configuration scheme of the long-term planning cycle and the capacity configuration scheme of the short-term planning cycle.

[0233] In addition, economic indicators are constructed using the average value of the weighted sum of reward functions, making full use of the characteristic that the reward function is a comprehensive function of multiple objectives such as cost, benefit and risk. The realization of economic indicators can comprehensively consider different operating scenarios, which is conducive to improving the applicability of capacity configuration plans in long-term planning cycles.

[0234] In step 4.4, when the updated economic index is less than the economic index before the update, the updated reward function of the decision cycle in which the capacity allocation plan has been implemented is fed back to step 2 to update the capacity allocation plan for the long-term planning cycle.

[0235] In the present invention, the reward function is used as a coupling variable between the long-term capacity configuration decision implemented by the upper-level optimization and the short-term operation decision implemented by the lower-level optimization, allowing the feedback of short-term operations to guide and adjust the long-term capacity configuration decision. This coupling ensures the coordination between long-term and short-term decisions. Secondly, the reward function is used in reinforcement learning to evaluate the long-term benefits of taking specific actions in a given state, which includes multiple objectives such as cost, benefit and risk. This method takes into account the continuous development trends of technical economy and other uncertainties, and provides a more dynamic and flexible decision-making process. Thirdly, the design of the reward function allows the model to make effective investment and operation decisions in the face of long-term and short-term uncertainties, such as technological development, price changes, demand growth, etc. This treatment of uncertainty is rare in the prior art. Finally, by updating the reward function, a more consistent and predictable transition is achieved over multiple periods of different scales.

[0236] The present invention also proposes a microgrid system planning capacity configuration system based on double-layer optimization, wherein the microgrid system planning includes a long-term planning cycle and a short-term planning cycle, wherein the facilities in the microgrid system include: wind turbines, energy storage equipment and water electrolyzers; including:

[0237] The cycle setting module is used to set the length of the decision cycle; a long-term planning cycle includes multiple decision cycles, and a decision cycle includes multiple short-term planning cycles;

[0238] The long-term planning scheme configuration module is used to construct the state space of the decision cycle with the allowable operating capacity of each facility within the decision cycle, and to construct the action space of the decision cycle with the allowable expansion of the allowable operating capacity of each facility within the decision cycle; the capacity configuration scheme of each decision cycle is determined according to the state space and the action space using the deep Q network reinforcement learning method; the capacity configuration scheme of the long-term planning cycle is formed by the capacity configuration schemes of all decision cycles within the long-term planning cycle;

[0239] The short-term planning scheme configuration module is used to construct the total allowable operating capacity of each facility in the current decision cycle based on the sum of the state space of the previous decision cycle and the action space of the current decision cycle; the total allowable operating capacity of each facility in the current decision cycle is used as the upper limit of the capacity of each short-term planning cycle in the current decision cycle, and the maximum annual operating profit in the current decision cycle is used as the target model; the capacity configuration scheme of each short-term planning cycle in the current decision cycle is determined according to the target model using a mathematical programming method under the constraints of the target model and the upper limit of the configuration capacity of each short-term planning cycle in the current decision cycle;

[0240] The long-term planning scheme updating module is used for, after the capacity configuration scheme of each short-term planning cycle is implemented, the actual value of power production of the decision cycle is formed by the actual value of power production of each short-term planning cycle within a decision cycle; the reward function of the decision cycle is updated according to the actual value of power production of the decision cycle; the economic index of the capacity configuration scheme of the long-term planning cycle is updated by using the updated reward function of the decision cycle; when the updated economic index is less than the economic index before the update, the updated reward function of the decision cycle in which the capacity configuration scheme has been implemented is fed back to update the capacity configuration scheme of the long-term planning cycle.

[0241] Assume that the length of the decision cycle is N years, and the long-term planning cycle includes W decision cycles, then the length of the long-term planning cycle is N×W years; a decision cycle includes m short-term planning cycles, and the length of the short-term planning cycle is W / m years.

[0242] The state space of the decision cycle is constructed with the allowable operating capacity of each facility in the decision cycle, satisfying the following relationship:

[0243] S P-1 =[s T,P-1 s B,P-1 s W,P-1 ] T (1)

[0244] In the formula, S P-1 is the state space of the P-1th decision cycle, s T,P-1 、s B,P-1 、s W,P-1 They are respectively the allowable operating capacity of the wind turbine, the allowable operating capacity of the energy storage device, and the allowable operating capacity of the water electrolyzer in the P-1th decision cycle.

[0245] The action space of the decision cycle is constructed by the allowable expansion of the allowable operating capacity of each facility in the decision cycle, satisfying the following relationship:

[0246] A P =[ΔX T,P ΔX B,P ΔXW,P ] T (2)

[0247] In the formula, A P is the action space of the Pth decision cycle, ΔX T,P , ΔX B,P , ΔX W,P They are respectively the allowable expansion amount of the allowable operating capacity of the wind turbine in the Pth decision cycle, the allowable expansion amount of the allowable operating capacity of the energy storage device, and the allowable expansion amount of the allowable operating capacity of the water electrolyzer.

[0248] The deep Q network reinforcement learning method is used to determine the capacity configuration scheme for each decision cycle based on the state space and action space, including:

[0249] 1) According to the state space S of the P-1th decision cycle P-1 and action space A P-1 Set the initial Q value Q of the facility in state s and action a (s,a) , where s∈S P-1 , a∈A P-1 ;

[0250] 2) The state space S in the P-1th decision cycle P-1 Next, select and execute the action space A of the Pth decision cycle P , to obtain the state space S of the Pth decision cycle P and reward function;

[0251] The reward function of the Pth decision cycle satisfies the following relationship:

[0252]

[0253] In the formula, is the reward function of the facility in state s and action a in the Pth decision cycle, C total,P is the total installation cost of each facility in the Pth decision cycle, R op,P,y is the total operating income of each facility in the Pth decision cycle in the yth year, r is the discount rate, and N is the number of years in a decision cycle;

[0254] 3) Update the Q value according to the deep Q network reinforcement learning rule to satisfy the following relationship:

[0255]

[0256] In the formula, Q (s,a),P is the Q value in the Pth decision cycle, γ is the discount factor, satisfying The discount factor is used to discount the value of future rewards to the current value; P (s′|s,a)To perform action a∈A P Then from the current state s∈S P-1 Transfer to the updated state s′∈S P The probability of max a′ Q (s′,a′) To make Q (s′,a′) The action a′∈A corresponding to the maximum value P ;

[0257] 4) In the updated state s′, repeat steps 2) and 3) to iteratively update the Q value until a predetermined number of iterations is reached;

[0258] 5) In the Pth decision cycle, the action space corresponding to the maximum value of Q value in the state space is used as the capacity configured in the decision cycle, satisfying the following relationship:

[0259] π(s)=max a Q (s,a) (6)

[0260] Where π(s) is the capacity configured in the decision cycle.

[0261] The sum of the state space of the previous decision cycle and the action space of the current decision cycle is used to construct the total allowable operating capacity of each facility in the current decision cycle, satisfying the following relationship:

[0262]

[0263] In the formula, TX T,P TX B,P TX W,P They are respectively the total allowable operating capacity of wind turbines, the total allowable operating capacity of energy storage devices, and the total allowable operating capacity of water electrolyzers in the Pth decision cycle.

[0264] The total allowed operating capacity of each facility in the current decision cycle is used as the upper limit of the capacity of each short-term planning cycle in the current decision cycle, and the maximum annual operating profit in the current decision cycle is used as the target model.

[0265] Satisfies the following relationship:

[0266] max∏ P,y =Re v P,y -C op,P,y -C main,P,y (8)

[0267] In the formula, Π P,y is the annual operating profit based on electricity price and operating cost in the yth year in the Pth decision cycle, Rev P,y is the total annual income in the yth year of the Pth decision cycle, Cop,P,y is the annual total operating cost in the yth year of the Pth decision cycle, C main,P,y is the annual total maintenance cost in the yth year in the Pth decision cycle;

[0268] Among them, the total annual revenue includes electricity sales and hydrogen sales, which satisfies the following relationship:

[0269]

[0270] Where EPrice is the market price of electricity, ESupply P,y,t is the amount of electricity supplied to the market in period t in year y during the Pth decision cycle, H Price is the market price of hydrogen, and H Prod P,y,t is the hydrogen supplied to the market in time period t in year y during the Pth decision cycle, where T is the total number of time periods;

[0271] The annual total operating cost satisfies the following relationship:

[0272]

[0273] In the formula, C penalty is the penalty cost for not meeting electricity demand, D P,y,t is the electricity demand in time period t in year y during the Pth decision cycle, E sup ply,P,y,t is the amount of electricity supplied in time period t in year y during the Pth decision cycle, C carbon is the carbon tax cost per unit of electricity, E purchase,P,y,t is the amount of electricity purchased from the main grid in time period t in the yth year of the Pth decision cycle, and T is the total number of time periods;

[0274] The total annual maintenance cost satisfies the following relationship:

[0275] C main,P,y =(TX j,P ×IC j,P ×Maint Coef j ) (11)

[0276] In the formula, TX j,P is the total allowed operating capacity of facilities of type j in the Pth decision cycle, IC j,P is the unit investment cost of facility type j in the Pth decision cycle, MaintCoef j is the maintenance cost coefficient of facility type j.

[0277] The constraints of the target model include:

[0278] 1) Constraints on the total operating capacity of each facility

[0279] The total permissible operating capacity of the wind turbine generator satisfies the following relationship:

[0280] TX T,P =ΔX T,P +TX T,P-1 (12)

[0281] In the formula, TX T,P is the total allowed operating capacity of wind turbines in the Pth decision cycle, ΔX T,P is the allowable expansion of the wind turbine operating capacity in the Pth decision cycle, TX T,P-1 is the total allowed operating capacity of wind turbines in the P-1th decision cycle;

[0282] The total allowable operating capacity of the energy storage device and the water electrolyzer satisfies the following relationship:

[0283] TX B,P +TX W,P =TX B,P-1 +TX W,P-1 +ΔX B,P +ΔX W,P (13)

[0284] Where ΔX B,P is the allowable expansion of the operating capacity of the energy storage device in the Pth decision cycle, ΔX W,P is the allowable expansion of the operating capacity of the water electrolyzer in the Pth decision cycle, TX B,P-1 TX W,P-1 are the total allowed operating capacities of the energy storage device and the water electrolyzer in the P-1th decision cycle, respectively;

[0285] 2) The constraints of energy balance satisfy the following relationship:

[0286]

[0287] In the formula, GenPower P,t is the amount of electricity generated by the wind turbine in time period t during the Pth decision cycle, Eff grid ImportPower is the AC-to-AC conversion efficiency of the power grid. P,t is the amount of electricity purchased from the main grid during time period t in the Pth decision cycle, Discharging P,t Demand is the discharge amount of the energy storage device in time period t in the Pth decision cycle. P,t is the power demand in time period t in the Pth decision cycle, H 2 Prod new,P,t is the amount of hydrogen produced by the newly installed water electrolyzer in the Pth decision cycle during time period t, H2 Prod existiong,P,t is the amount of hydrogen produced by the previously installed water electrolyzer in time period t during the Pth decision cycle, Curtail P,t is the amount of power cut in the Pth decision cycle in time period t due to grid instability or unmet demand;

[0288] 3) The energy storage system constraints satisfy the following relationship:

[0289]

[0290]

[0291] InitialStorage P -FinalStrorage P =0 (16)

[0292] In the formula, ESS P,y,t+1 is the energy storage system status at the end of period t+1 in the yth year of the Pth decision cycle, ESS P,y,t is the state of the energy storage system at the beginning of time period t in the yth year of the Pth decision cycle, σ B is the self-discharge rate of the energy storage device, η AC / DC is the efficiency of the rectifier, η B is the discharge efficiency of the energy storage device, Charge P,y,t is the charging power in time period t in year y in the Pth decision cycle, Discharging P,y,t is the discharge power in time period t in year y in the Pth decision cycle, InitialStorage P and FinalStorage P are the energy storage device power at the beginning and end of the Pth decision cycle, respectively, and Δt is the time increment;

[0293] 4) Supply and demand constraints satisfy the following relationship:

[0294] sup P,y,t ≤D P,y,t (17)

[0295] In the formula, sup P,y,t is the power supply in time period t in year y during the Pth decision cycle, D P,y,t is the electricity demand in time period t in year y during the Pth decision cycle.

[0296] The capacity allocation plan for the decision cycle, including the target value of power production for each facility during the decision cycle;

[0297] Capacity allocation plans for long-term planning periods, including power production targets for microgrid systems;

[0298] Capacity allocation plan for the short-term planning cycle, including power production target values ​​and dispatch instructions for each facility.

[0299] Using the reward function of the updated decision cycle, the economic indicators of the capacity allocation scheme in the long-term planning cycle are updated as follows:

[0300]

[0301] Where max N PV is the economic index of the capacity allocation plan in the long-term planning period. To find the expected value of the function, P finish is the decision cycle number of the capacity allocation plan that has been implemented, P plan is the decision cycle number in which the capacity allocation plan has not yet been implemented. The updated reward function for the decision cycle where the capacity allocation scheme has been implemented, Reward function for decision cycles where the capacity allocation solution has not yet been implemented.

[0302] The technical characteristics of coupling the lower-level decision to the upper-level decision are reflected in that the lower-level decision formulates and executes energy management strategies based on real-time data and forecast information in each period to ensure the balance between power supply and demand. The feedback of these short-term operations is used to guide and adjust the long-term capacity configuration decisions of the upper layer, forming a dynamic optimization process. This coupling ensures the coordination between long-term and short-term decisions, so that the long-term investment plan not only meets the actual needs of short-term operations, but also can adapt to environmental changes and uncertainties, and realize more flexible and adaptable microgrid system planning. The present invention has a different logical order from the existing power grid planning ideas. Traditional power grid planning usually adopts a top-down approach, first determining long-term goals and then decomposing them into short-term actions. The method proposed in the present invention adopts a two-layer optimization, allowing long-term decisions to take into account the feedback of short-term operations, and adapt to environmental changes through reinforcement learning, so as to achieve more flexible and adaptable planning.

[0303] Fossil fuel-based energy has been installed in the hybrid microgrid in the embodiment. On a shorter time scale, the energy management system (EMS) makes operational decisions to manage hourly fluctuations in wind power and electricity demand. The EMS provides electricity to meet electricity demand, uses batteries as energy storage devices, and produces green hydrogen through water electrolyzers. The term wind curtailment refers to the deliberate idling and excessive power outages of wind turbines to avoid overload. In the case of overload or underload, the EMS can implement wind curtailment or purchase electricity with carbon emission penalties from the main grid at any time. Although the uncertainty and short-term fluctuations in demand and wind speed will play a disruptive role, the controller (EMS) attempts to maximize profits through green hydrogen sales after meeting electricity demand. For the hybrid microgrid system in the embodiment, the two-level optimization method proposed in the present invention is used to solve uncertainties at multiple time scales. At a higher level, strategic plans are made to maximize the long-term profit margin of the aggregated state, system description, and uncertainty after the current planning period. At a lower level, short-term operational decisions are made to ensure the execution of the plan while considering various constraints. Capacity investment decisions are determined by reinforcement learning (RL) and are influenced by Markov processes representing environmental uncertainties, such as investment costs and demand size. High-level decisions then set capacity constraints for more frequent low-level operational decisions in the corresponding period, which are formulated as mathematical programs (MPs) based on realized fast timescale uncertainties (such as wind speed and demand profiles). The net present value (NPV) is calculated through low-level simulations of various scenarios. Capacity investments in the hybrid microgrid system in the embodiment are made based on current realization information and future probability distributions of the random and time-varying environment and the remaining life of previously installed facilities. Power demand growth and learning rate are considered as random parameters, while technological development is a time-varying parameter. The state consists of physical state variables, which contain information about previously installed capacity and the remaining life of facilities. Other state variables contain current realization information of random environmental factors, such as power demand, size, and investment cost of each facility in the year of investment decision. Markov Decision Process Formulation for High-Level Decisions This study presents long-term planning of a hybrid microgrid (HM) involving capacity investment decisions for each facility with P-year-long decision intervals that can also serve as the basis for future scenarios due to the Markov nature of uncertainties. The proposed approach produces capacity deployment plans that enable more robust and cost-effective energy system planning decisions and more consistent and predictable transitions compared to the 2SSP approach. This improvement comes from the approach's ability to consider evolving trends in techno-economic and other uncertainties in the decision-making process. The proposed approach also outperforms the DMP approach in enabling more robust and cost-effective energy system planning decisions. The strategic configuration prioritizes increased investments in energy storage and investments in water electrolyzers to mitigate long-term uncertainties.The approach proposed in this invention also emphasizes the importance of reflecting the multi-period nature of capacity investment decisions, taking into account not only long-term changes but also the uncertainty of these changes in relevant economic factors and using optimization models with sufficiently fine time resolution.

[0304] In addition, the method proposed in the present invention adopts a two-layer framework, which provides constraints for short-term operational decisions through long-term capacity configuration, ensuring the consistency between long-term goals and short-term actions. Through RL, the two-layer optimization framework can dynamically adapt to environmental changes and update decision-making strategies in real time, while 2SSP and DMP usually only make decisions once in each stage and lack dynamic adaptability. The introduction of RL enables the two-layer optimization framework to have the ability to learn from experience and to discover and utilize potential patterns and laws. When the external environment or policies change, the two-layer optimization framework can adjust strategies more quickly and adapt to new situations, while traditional 2SSP and DMP may need to be re-planned, which will lead to cumbersome computational problems.

[0305] In the embodiments, the system proposed in the present invention is applied to various occasions, including: hybrid microgrid (HM), which uses multiple energy sources and supervises energy infrastructure in a specific area through decentralized management; green hydrogen production, which uses renewable energy to produce hydrogen through a water electrolysis process; multi-time scale decision-making, which considers long-term capacity decisions and energy scheduling and storage decisions on short-term time scales; energy storage system (ESS) and water electrolysis hydrogen production technology to balance supply and demand and improve system reliability.

[0306] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0307] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0308] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0309] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0310] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A capacity configuration method for microgrid system planning based on double-layer optimization, wherein the microgrid system planning includes a long-term planning cycle and a short-term planning cycle, wherein: The facilities in the microgrid system include: wind turbines, energy storage equipment and water electrolyzers; the characteristics include: Set the length of the decision cycle; a long-term planning cycle includes multiple decision cycles, and a decision cycle includes multiple short-term planning cycles; The state space of the decision cycle is constructed with the allowable operating capacity of each facility in the decision cycle, and the action space of the decision cycle is constructed with the allowable expansion of the allowable operating capacity of each facility in the decision cycle; the capacity allocation scheme of each decision cycle is determined according to the state space and action space using the deep Q network reinforcement learning method; the capacity allocation scheme of all decision cycles in the long-term planning cycle is used to form the capacity allocation scheme of the long-term planning cycle; The total allowable operating capacity of each facility in the current decision cycle is constructed by the sum of the state space of the previous decision cycle and the action space of the current decision cycle; the total allowable operating capacity of each facility in the current decision cycle is taken as the upper limit of the capacity of each short-term planning cycle in the current decision cycle, and the maximum annual operating profit in the current decision cycle is taken as the target model; the mathematical programming method is used to determine the capacity configuration plan of each short-term planning cycle in the current decision cycle according to the target model under the constraints of the target model and the upper limit of the configuration capacity of each short-term planning cycle in the current decision cycle; After the capacity allocation plan of each short-term planning cycle is implemented, the actual value of power production in the decision cycle is composed of the actual value of power production in each short-term planning cycle within a decision cycle; the reward function of the decision cycle is updated according to the actual value of power production in the decision cycle; the economic index of the capacity allocation plan of the long-term planning cycle is updated by using the updated reward function of the decision cycle; when the updated economic index is less than the economic index before the update, the updated reward function of the decision cycle in which the capacity allocation plan has been implemented is fed back to update the capacity allocation plan of the long-term planning cycle.

2. The microgrid system planning capacity configuration method based on double-layer optimization according to claim 1 is characterized in that: Assume that the length of the decision cycle is N years, and the long-term planning cycle includes W decision cycles, then the length of the long-term planning cycle is N×W years; a decision cycle includes m short-term planning cycles, and the length of the short-term planning cycle is W / m years.

3. The microgrid system planning capacity configuration method based on double-layer optimization according to claim 1 is characterized in that: The state space of the decision cycle is constructed with the allowable operating capacity of each facility in the decision cycle, satisfying the following relationship: S P-1 =[s T，P-1 s B，P-1 s W，P-1 ] T (1) In the formula, S P-1 is the state space of the P-1th decision cycle, s T，P-1 、s B，P-1 、s W，P-1 They are respectively the allowable operating capacity of the wind turbine, the allowable operating capacity of the energy storage device, and the allowable operating capacity of the water electrolyzer in the P-1th decision cycle.

4. The microgrid system planning capacity configuration method based on double-layer optimization according to claim 3 is characterized in that: The action space of the decision cycle is constructed by the allowable expansion of the allowable operating capacity of each facility in the decision cycle, satisfying the following relationship: A P =[ΔX T，P ΔX B，P ΔX W，P ] T (2) In the formula, A P is the action space of the Pth decision cycle, ΔX T，P , ΔX B，P , ΔX W,P They are respectively the allowable expansion amount of the allowable operating capacity of the wind turbine in the Pth decision cycle, the allowable expansion amount of the allowable operating capacity of the energy storage device, and the allowable expansion amount of the allowable operating capacity of the water electrolyzer.

5. The method for planning capacity configuration of a microgrid system based on double-layer optimization according to claim 4 is characterized in that: The deep Q network reinforcement learning method is used to determine the capacity configuration scheme for each decision cycle based on the state space and action space, including: 1) According to the state space S of the P-1th decision cycle P-1 and action space A P-1 Set the initial Q value Q of the facility in state s and action a (s，a) , where s∈S P-1 , a∈A P-1 ; 2) The state space S in the P-1th decision cycle P-1 Next, select and execute the action space A of the Pth decision cycle P , to obtain the state space S of the Pth decision cycle P and reward function; The reward function of the Pth decision cycle satisfies the following relationship: In the formula, is the reward function of the facility in state s and action a in the Pth decision cycle, C total，P is the total installation cost of each facility in the Pth decision cycle, R op,P，y is the total operating income of each facility in the Pth decision cycle in the yth year, r is the discount rate, and N is the number of years in a decision cycle; 3) Update the Q value according to the deep Q network reinforcement learning rules to satisfy the following relationship: In the formula, Q (s，a)，P is the Q value in the Pth decision cycle, γ is the discount factor, satisfying The discount factor is used to discount the value of future rewards to the current value; P (s′|s,a) To perform action a∈A P Then from the current state s∈S P-1 Transfer to the updated state s′∈S P The probability of max a′ Q (s′，a′) To make Q (s′，a′) The action a′∈A corresponding to the maximum value P ; 4) In the updated state s′, repeat steps 2) and 3) to iteratively update the Q value until a predetermined number of iterations is reached; 5) In the Pth decision cycle, the action space corresponding to the maximum value of Q value in the state space is used as the capacity configured in the decision cycle, satisfying the following relationship: π(s)=max a Q (s，a) (6) Where π(s) is the capacity configured in the decision cycle.

6. The method for planning capacity configuration of a microgrid system based on double-layer optimization according to claim 5 is characterized in that: The total allowable operating capacity of each facility in the current decision cycle is constructed by the sum of the state space of the previous decision cycle and the action space of the current decision cycle, satisfying the following relationship: In the formula, TX T，P TX B，P TX W，P They are respectively the total allowable operating capacity of wind turbines, the total allowable operating capacity of energy storage devices, and the total allowable operating capacity of water electrolyzers in the Pth decision cycle.

7. The method for planning capacity configuration of a microgrid system based on double-layer optimization according to claim 6 is characterized in that: The total allowed operating capacity of each facility in the current decision cycle is used as the upper limit of the capacity of each short-term planning cycle in the current decision cycle, and the maximum annual operating profit in the current decision cycle is used as the target model. Satisfies the following relationship: max∏ P，y =Re v P，y -C oP，P,y -C main，P，y (8) In the formula, Π P，y is the annual operating profit based on electricity price and operating cost in the yth year in the Pth decision cycle, Rev P，y is the total annual income in the yth year of the Pth decision cycle, C oP，P，y is the annual total operating cost in the yth year of the Pth decision cycle, C main，P，y is the annual total maintenance cost in the yth year in the Pth decision cycle; Among them, the total annual revenue includes electricity sales and hydrogen sales, which satisfies the following relationship: Where EPrice is the market price of electricity, ESupply P,y，t is the amount of electricity supplied to the market in period t in year y during the Pth decision cycle, HPrice is the market price of hydrogen, and HProd P,y，t is the hydrogen supplied to the market in time period t in year y during the Pth decision cycle, where T is the total number of time periods; The total annual operating cost satisfies the following relationship: In the formula, C penalty is the penalty cost for not meeting electricity demand, D P,y，t is the electricity demand in time period t in year y during the Pth decision cycle, E supply,P,y，t is the amount of electricity supplied in time period t in year y during the Pth decision cycle, C carbon is the carbon tax cost per unit of electricity, E purchase，P,y，t is the amount of electricity purchased from the main grid in time period t in the yth year of the Pth decision cycle, and T is the total number of time periods; The total annual maintenance cost satisfies the following relationship: C main，P，y =(TX j，P ×IC j，P ×MaintCoef j ) (11) In the formula, TX j，P is the total allowed operating capacity of facilities of type j in the Pth decision cycle, IC j，P is the unit investment cost of facility type j in the Pth decision cycle, MaintCoef j is the maintenance cost coefficient of facility type j.

8. The method for planning capacity configuration of a microgrid system based on double-layer optimization according to claim 7 is characterized in that: The constraints of the target model include: 1) Constraints on the total operating capacity of each facility The total permissible operating capacity of the wind turbine generator satisfies the following relationship: TX T,P =ΔX T，P +TX T，P-1 (12) In the formula, TX T，P is the total allowed operating capacity of wind turbines in the Pth decision cycle, ΔX T，P is the allowable expansion of the wind turbine operating capacity in the Pth decision cycle, TX T,P-1 is the total allowed operating capacity of wind turbines in the P-1th decision cycle; The total allowable operating capacity of the energy storage device and the water electrolyzer satisfies the following relationship: TX B，P +TX W,P =TX B，P-1 +TX W，P-1 +ΔX B，P +ΔX W，P (13) Where ΔX B，P is the allowable expansion of the operating capacity of the energy storage device in the Pth decision cycle, ΔX W，P is the allowable expansion of the operating capacity of the water electrolyzer in the Pth decision cycle, TX B，P-1 TX W，P-1 are the total allowed operating capacities of the energy storage device and the water electrolyzer in the P-1th decision cycle, respectively; 2) The constraints of energy balance satisfy the following relationship: GenPower P，t ×Eff grid +ImportPower P,t +Discharge P,t =Demand P，t +H2Prod new,P，t +H2Prod existing，P，t +Curtail P，t (14) In the formula, GenPower P，t is the amount of electricity generated by the wind turbine in time period t during the Pth decision cycle, Eff grid ImportPower is the AC-to-AC conversion efficiency of the power grid. P，t is the amount of electricity purchased from the main grid during time period t in the Pth decision cycle, Discharge P，t Demand is the discharge amount of the energy storage device in time period t in the Pth decision cycle. P，t H2Prod is the power demand in time period t in the Pth decision cycle, new，P，t H2Prod is the amount of hydrogen produced by the newly installed water electrolyzer in the Pth decision cycle during time period t, existing，P，t is the amount of hydrogen produced by the previously installed water electrolyzer in time period t during the Pth decision cycle, Curtail P，t is the amount of power cut in the Pth decision cycle in time period t due to grid instability or unmet demand; 3) The energy storage system constraints satisfy the following relationship: InitialStorage P -FinalStorage P =0 (16) In the formula, ESS P，y，t+1 is the energy storage system status at the end of period t+1 in the yth year of the Pth decision cycle, ESS P，y，t is the state of the energy storage system at the beginning of time period t in the yth year of the Pth decision cycle, σ B is the self-discharge rate of the energy storage device, η AC / DC is the efficiency of the rectifier, η B is the discharge efficiency of the energy storage device, Charge P,y，t is the charging power in time period t in year y in the Pth decision cycle, Discharge P,y，t is the discharge power in time period t in year y in the Pth decision cycle, InitialStorage P and FinalStorage P are the energy storage device power at the beginning and end of the Pth decision cycle, respectively, and Δt is the time increment; 4) Supply and demand constraints satisfy the following relationship: sup P,y,t ≤D P,y，t (17) In the formula, sup P,y，t is the power supply in time period t in year y during the Pth decision cycle, D P,y，t is the electricity demand in time period t in year y during the Pth decision cycle.

9. The microgrid system planning capacity configuration method based on double-layer optimization according to claim 8 is characterized in that: The capacity allocation plan for the decision cycle, including the target value of power production for each facility during the decision cycle; Capacity allocation plans for long-term planning periods, including power production targets for microgrid systems; Capacity allocation plan for the short-term planning cycle, including power production target values ​​and dispatch instructions for each facility.

10. The method for planning capacity configuration of a microgrid system based on double-layer optimization according to claim 9, characterized in that: Using the reward function of the updated decision cycle, the economic indicators of the capacity allocation scheme in the long-term planning cycle are updated as follows: Where maxNPV is the economic index of the capacity allocation plan in the long-term planning period. To find the expected value of the function, P finish is the decision cycle number of the capacity allocation plan that has been implemented, P plan is the decision cycle number in which the capacity allocation plan has not yet been implemented. The updated reward function for the decision cycle where the capacity allocation scheme has been implemented, Reward function for decision cycles where the capacity allocation solution has not yet been implemented.

11. A microgrid system planning capacity configuration system based on double-layer optimization, the microgrid system planning includes a long-term planning cycle and a short-term planning cycle, wherein: The facilities in the microgrid system include: wind turbines, energy storage equipment and water electrolyzers; the characteristics include: The cycle setting module is used to set the length of the decision cycle; a long-term planning cycle includes multiple decision cycles, and a decision cycle includes multiple short-term planning cycles; The long-term planning scheme configuration module is used to construct the state space of the decision cycle with the allowable operating capacity of each facility within the decision cycle, and to construct the action space of the decision cycle with the allowable expansion of the allowable operating capacity of each facility within the decision cycle; the capacity configuration scheme of each decision cycle is determined according to the state space and the action space using the deep Q network reinforcement learning method; the capacity configuration scheme of the long-term planning cycle is formed by the capacity configuration schemes of all decision cycles within the long-term planning cycle; The short-term planning scheme configuration module is used to construct the total allowable operating capacity of each facility in the current decision cycle based on the sum of the state space of the previous decision cycle and the action space of the current decision cycle; the total allowable operating capacity of each facility in the current decision cycle is used as the upper limit of the capacity of each short-term planning cycle in the current decision cycle, and the maximum annual operating profit in the current decision cycle is used as the target model; the capacity configuration scheme of each short-term planning cycle in the current decision cycle is determined according to the target model using a mathematical programming method under the constraints of the target model and the upper limit of the configuration capacity of each short-term planning cycle in the current decision cycle; The long-term planning scheme updating module is used for, after the capacity configuration scheme of each short-term planning cycle is implemented, the actual value of power production of the decision cycle is formed by the actual value of power production of each short-term planning cycle within a decision cycle; the reward function of the decision cycle is updated according to the actual value of power production of the decision cycle; the economic index of the capacity configuration scheme of the long-term planning cycle is updated by using the updated reward function of the decision cycle; when the updated economic index is less than the economic index before the update, the updated reward function of the decision cycle in which the capacity configuration scheme has been implemented is fed back to update the capacity configuration scheme of the long-term planning cycle.

12. The microgrid system planning capacity configuration system based on double-layer optimization according to claim 11, characterized in that: Assume that the length of the decision cycle is N years, and the long-term planning cycle includes W decision cycles, then the length of the long-term planning cycle is N×W years; a decision cycle includes m short-term planning cycles, and the length of the short-term planning cycle is W / m years.

13. The microgrid system planning capacity configuration system based on double-layer optimization according to claim 11, characterized in that: The state space of the decision cycle is constructed with the allowable operating capacity of each facility in the decision cycle, satisfying the following relationship: S P-1 =[s T，P-1 s B，P-1 s W，P-1 ] T (1) In the formula, S P-1 is the state space of the P-1th decision cycle, s T，P-1 、s B，P-1 、s W，P-1 They are respectively the allowable operating capacity of the wind turbine, the allowable operating capacity of the energy storage device, and the allowable operating capacity of the water electrolyzer in the P-1th decision cycle.

14. The microgrid system planning capacity configuration system based on double-layer optimization according to claim 13, characterized in that: The action space of the decision cycle is constructed by the allowable expansion of the allowable operating capacity of each facility in the decision cycle, satisfying the following relationship: A P =[ΔX T，P ΔX B，P ΔX W，P ] T (2) In the formula, A P is the action space of the Pth decision cycle, ΔX T，P , ΔX B，P , ΔX W,P They are respectively the allowable expansion amount of the allowable operating capacity of the wind turbine in the Pth decision cycle, the allowable expansion amount of the allowable operating capacity of the energy storage device, and the allowable expansion amount of the allowable operating capacity of the water electrolyzer.

15. The microgrid system planning capacity configuration system based on double-layer optimization according to claim 14, characterized in that: The deep Q network reinforcement learning method is used to determine the capacity configuration scheme for each decision cycle based on the state space and action space, including: 1) According to the state space S of the P-1th decision cycle P-1 and action space A P-1 Set the initial Q value Q of the facility in state s and action a (s，a) , where s∈S P-1 , a∈A P-1 ; 2) The state space S in the P-1th decision cycle P-1 Next, select and execute the action space A of the Pth decision cycle P , to obtain the state space S of the Pth decision cycle P and reward function; The reward function of the Pth decision cycle satisfies the following relationship: In the formula, is the reward function of the facility in state s and action a in the Pth decision cycle, C total,P is the total installation cost of each facility in the Pth decision cycle, R oP,P，y is the total operating income of each facility in the Pth decision cycle in the yth year, r is the discount rate, and N is the number of years in a decision cycle; 3) Update the Q value according to the deep Q network reinforcement learning rules to satisfy the following relationship: In the formula, Q (s，a)，P is the Q value in the Pth decision cycle, γ is the discount factor, satisfying The discount factor is used to discount the value of future rewards to the current value; P (s′|s，a) To perform action a∈A P Then from the current state s∈S P-1 Transfer to the updated state s′∈S P The probability of max a′ Q (s′，a′) To make Q (s′，a′) The action a′∈A corresponding to the maximum value P ; 4) In the updated state s′, repeat steps 2) and 3) to iteratively update the Q value until a predetermined number of iterations is reached; 5) In the Pth decision cycle, the action space corresponding to the maximum value of Q value in the state space is used as the capacity configured in the decision cycle, satisfying the following relationship: π(s)=max a Q (s，a) (6) Where π(s) is the capacity configured in the decision cycle.

16. The microgrid system planning capacity configuration system based on double-layer optimization according to claim 15, characterized in that: The total allowable operating capacity of each facility in the current decision cycle is constructed by the sum of the state space of the previous decision cycle and the action space of the current decision cycle, satisfying the following relationship: In the formula, TX T，P TX B，P TX W，P They are respectively the total allowable operating capacity of wind turbines, the total allowable operating capacity of energy storage devices, and the total allowable operating capacity of water electrolyzers in the Pth decision cycle.

17. The microgrid system planning capacity configuration system based on double-layer optimization according to claim 16, characterized in that: The total allowed operating capacity of each facility in the current decision cycle is used as the upper limit of the capacity of each short-term planning cycle in the current decision cycle, and the maximum annual operating profit in the current decision cycle is used as the target model. Satisfies the following relationship: max∏ P，y =Rev P，y -C oP，P,y -C main，P，y (8) In the formula, Π P，y is the annual operating profit based on electricity price and operating cost in the yth year in the Pth decision cycle, Rev P，y is the total annual income in the yth year of the Pth decision cycle, C oP，P，y is the annual total operating cost in the yth year of the Pth decision cycle, C main，P，y is the annual total maintenance cost in the yth year in the Pth decision cycle; Among them, the total annual revenue includes electricity sales and hydrogen sales, which satisfies the following relationship: Where EPrice is the market price of electricity, ESupplyP,y,t is the amount of electricity supplied to the market in period t in year y during the Pth decision cycle, HPrice is the market price of hydrogen, and HProd P,y，t is the hydrogen supplied to the market in time period t in year y during the Pth decision cycle, where T is the total number of time periods; The total annual operating cost satisfies the following relationship: In the formula, C penalty is the penalty cost for not meeting electricity demand, D P,y，t is the electricity demand in time period t in year y during the Pth decision cycle, E supply,P,y，t is the amount of electricity supplied in time period t in year y during the Pth decision cycle, C carbon is the carbon tax cost per unit of electricity, E purchase，P,y，t is the amount of electricity purchased from the main grid in time period t in the yth year of the Pth decision cycle, and T is the total number of time periods; The total annual maintenance cost satisfies the following relationship: C main，P，y =(TX j，P ×IC j，P ×MaintCoef j ) (11) In the formula, TX j，P is the total allowed operating capacity of facilities of type j in the Pth decision cycle, IC j，P is the unit investment cost of facility type j in the Pth decision cycle, MaintCoef j is the maintenance cost coefficient of facility type j.

18. The microgrid system planning capacity configuration system based on double-layer optimization according to claim 17, characterized in that: The constraints of the target model include: 1) Constraints on the total operating capacity of each facility The total permissible operating capacity of the wind turbine generator satisfies the following relationship: TX T,P =ΔX T，P +TX T，P-1 (12) In the formula, TX T，P is the total allowed operating capacity of wind turbines in the Pth decision cycle, ΔX T，P is the allowable expansion of the wind turbine operating capacity in the Pth decision cycle, TX T，P-1 is the total allowed operating capacity of wind turbines in the P-1th decision cycle; The total allowable operating capacity of the energy storage device and the water electrolyzer satisfies the following relationship: TX B，P +TX W,P =TX B，P-1 +TX W，P-1 +ΔX B，P +ΔX W，P (13) Where ΔX B，P is the allowable expansion of the operating capacity of the energy storage device in the Pth decision cycle, ΔX W，P is the allowable expansion of the operating capacity of the water electrolyzer in the Pth decision cycle, TX B，P-1 TX W，P-1 are the total allowed operating capacities of the energy storage device and the water electrolyzer in the P-1th decision cycle, respectively; 2) The constraints of energy balance satisfy the following relationship: GenPower P，t ×Eff grid +ImportPower P,t +Discharge P,t =Demand P，t +H2Prod new,P,t +H2Prod existing,P,t +Curtail P，t (14) In the formula, GenPower P，t is the amount of electricity generated by the wind turbine in time period t during the Pth decision cycle, Eff grid ImportPower is the AC-to-AC conversion efficiency of the power grid. P，t is the amount of electricity purchased from the main grid during time period t in the Pth decision cycle, Discharge P，t Demand is the discharge amount of the energy storage device in time period t in the Pth decision cycle. P，t H2Prod is the power demand in time period t in the Pth decision cycle, new，P，t H2Prod is the amount of hydrogen produced by the newly installed water electrolyzer in the Pth decision cycle during time period t, existing，P，t is the amount of hydrogen produced by the previously installed water electrolyzer in time period t during the Pth decision cycle, Curtail P，t is the amount of power cut in the Pth decision cycle in time period t due to grid instability or unmet demand; 3) The energy storage system constraints satisfy the following relationship: InitialStorage P -FinalStorage P =0 (16) In the formula, ESS P，y，t+1 is the energy storage system status at the end of period t+1 in the yth year of the Pth decision cycle, ESS P，y，t is the state of the energy storage system at the beginning of time period t in the yth year of the Pth decision cycle, σ B is the self-discharge rate of the energy storage device, η AC / DC is the efficiency of the rectifier, η B is the discharge efficiency of the energy storage device, Charge P,y，t is the charging power in time period t in year y in the Pth decision cycle, Discharge P,y，t is the discharge power in time period t in year y in the Pth decision cycle, InitialStorage P and FinalStorage P are the energy storage device power at the beginning and end of the Pth decision cycle, respectively, and Δt is the time increment; 4) Supply and demand constraints satisfy the following relationship: sup P,y,t ≤D P,y，t (17) In the formula, sup P,y，t is the power supply in time period t in year y during the Pth decision cycle, D P,y，t is the electricity demand in time period t in year y during the Pth decision cycle.

19. The microgrid system planning capacity configuration system based on double-layer optimization according to claim 18, characterized in that: The capacity allocation plan for the decision cycle, including the target value of power production for each facility during the decision cycle; Capacity allocation plans for long-term planning periods, including power production targets for microgrid systems; Capacity allocation plan for the short-term planning cycle, including power production target values ​​and dispatch instructions for each facility.

20. The microgrid system planning capacity configuration system based on double-layer optimization according to claim 19, characterized in that: Using the reward function of the updated decision cycle, the economic indicators of the capacity allocation scheme in the long-term planning cycle are updated as follows: Where maxNPV is the economic index of the capacity allocation plan in the long-term planning period. To find the expected value of the function, P finish is the decision cycle number of the capacity allocation plan that has been implemented, P plan is the decision cycle number in which the capacity allocation plan has not yet been implemented. The updated reward function for the decision cycle where the capacity allocation scheme has been implemented, Reward function for decision cycles where the capacity allocation solution has not yet been implemented.

21. A terminal, comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method according to any one of claims 1-10.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 10 are implemented.

Images