Micro-grid joint optimization scheduling method and system integrating heterogeneous flexible resources

Abstract

The invention discloses a microgrid joint optimization scheduling method and system integrating heterogeneous flexible resources. The method comprises the following steps: establishing a photovoltaic load model; establishing a day-ahead energy storage renting model; based on the photovoltaic load model and the day-ahead energy storage renting and recruitment model, establishing a producer and consumer optimization model; establishing an intraday virtual energy storage model; establishing an intra-day shared energy storage model; establishing a micro-grid operator optimization model based on the producer and consumer optimization model, the intra-day virtual energy storage model and the intra-day shared energy storage model; the micro-grid operator optimization model is converted into an established Markov decision process; and solving a Markov decision process, and updating and iterating scheduling parameters to realize optimal scheduling of the micro-grid. According to the method, the multi-type flexible resource adjustment potential can be stimulated, and the flexible load dynamic adjustment capability is improved.

Description

Technical Field

[0001] The present invention belongs to the field of power system automation, relates to microgrid optimal scheduling technology, and specifically relates to a method and system for joint optimal scheduling of a microgrid integrating heterogeneous flexibility resources. Background Technique

[0002] With the accelerated promotion of global energy transformation and the dual-carbon goal, the power system is facing a series of challenges. Due to limitations such as unidirectional energy flow, long-distance power transmission losses, and insufficient flexibility of traditional centralized power grids, it is difficult to adapt to the large-scale access of distributed energy sources (such as photovoltaic and wind power) and the new requirements of the coordinated interaction of "source-load-storage" on the user side. As a small power system integrating distributed generation, energy storage devices, power electronic equipment, and loads, the microgrid has become an important technical path for building a new power system and promoting low-carbon energy development due to its advantages such as flexible networking, seamless off-grid / on-grid switching, and local energy consumption.

[0003] In traditional microgrids, end-users of electricity are usually regarded as passive consumers, that is, the microgrid operator can directly determine the operation strategy according to the total energy demand of the users, and its operation goal is usually to minimize the cost of purchasing energy from the public power grid. With the rapid development of distributed energy, traditional energy consumers are gradually evolving into prosumers. Prosumers have both the roles of energy production and consumption. By deploying a large number of distributed photovoltaic devices, they can not only achieve self-energy supply but also transmit electricity to the power grid when there is an excess. Therefore, the microgrid operator can not only purchase electric energy from the external power grid but also purchase the redundant distributed energy of prosumers. This operation mode promotes the accelerated transformation of the global energy structure towards low-carbon and distributed directions. However, the load demand and photovoltaic output are easily affected by environmental factors, and their power is uncertain. At the same time, the electricity price of the external power grid fluctuates randomly with the changes in the power market, bringing challenges to the economic and efficient operation of the microgrid.

[0004] Currently, with the accelerated deployment of demand-side flexibility resources (such as distributed energy storage, electric vehicle clusters, and intelligent temperature control devices), flexible adjustable resources are gradually becoming an important means to effectively cope with the volatility of the external power grid electricity price and renewable energy output. However, the large number and diverse operating characteristics of flexibility resources in the microgrid increase the complexity of microgrid optimal scheduling. In addition, the uncertainty of prosumer behavior will affect the participation of flexibility resources in microgrid regulation, bringing challenges to the optimal operation of the microgrid.

[0005] Existing research mainly focuses on optimal scheduling based on distributed energy storage or flexible loads, and the current research on the optimal scheduling of flexible loads is mainly achieved through demand response technology. In addition, in terms of the solution of optimization algorithms, existing research mainly focuses on deterministic methods based on mathematical programming and search methods based on heuristic rules. Therefore, the existing technology has the following specific disadvantages:

[0006] 1. Most studies focus on the independent optimization of a single type of flexibility resource (such as distributed energy storage or flexible load), lacking a systematic framework for the dynamic coupling mechanism between different types of resources, resulting in the underutilization of the collaborative potential of multiple resources.

[0007] 2. Currently, the research on the optimal scheduling of flexible loads mainly relies on demand response technology. However, this regulation strategy is mostly limited to fixed time periods (such as only allowing load curtailment at the current moment or shifting some loads to adjacent periods), lacking the dynamic adjustment ability across time periods and multiple cycles.

[0008] 3. Due to the influence of various factors such as economic incentives and user habits on users, and the large differences in shared energy storage capacity and demand response resources among different users, it is difficult for existing research to select an appropriate optimization model to accurately describe the response behavior of each user, posing challenges to the solution of microgrid resource optimization. Summary of the Invention

[0009] Object of the Invention: To solve the problem that existing optimization methods are difficult to handle the uncertainties and differences in the adjustable capabilities of heterogeneous flexibility resources, a microgrid joint optimal scheduling method and system integrating heterogeneous flexibility resources are provided, which can stimulate the adjustment potential of multiple types of flexibility resources and enhance the dynamic adjustment ability of flexible loads.

[0010] Technical Solution: To achieve the above object, the present invention provides a microgrid joint optimal scheduling method integrating heterogeneous flexibility resources, including the following steps:

[0011] S1: Establish a photovoltaic load model;

[0012] S2: Establish a day-ahead energy storage leasing model;

[0013] S3: Based on the photovoltaic load model and the day-ahead energy storage leasing model, establish a prosumer optimization model;

[0014] S4: Establish an intra-day virtual energy storage model;

[0015] S5: Establish an intra-day shared energy storage model;

[0016] S6: Based on the prosumer optimization model, the intra-day virtual energy storage model, and the intra-day shared energy storage model, establish a microgrid operator optimization model;

[0017] S7: Convert the microgrid operator optimization model into a well-established Markov decision process;

[0018] S8: Solve the Markov decision process in step S7, update and iterate the scheduling parameters, and achieve the optimal scheduling of the microgrid.

[0019] Furthermore, the establishment of the photovoltaic load model in step S1 includes:

[0020] The mathematical expressions of photovoltaic power and load demand are as follows:

[0021]

[0022]

[0023] Where and are the power of the PV panel and the load power of prosumer i within time t, respectively; and are the diffusion coefficients of the PV panel power and the load power, respectively, and are the Brownian motion processes used to describe the randomness of the PV panel and the load, respectively.

[0024] Furthermore, the establishment of the day-ahead energy storage leasing model in step S2 includes:

[0025] The day-ahead energy storage leasing model includes the revenue and cost of the prosumer leasing the energy storage capacity;

[0026] The mathematical expression of the revenue of the prosumer leasing the energy storage capacity is as follows:

[0027]

[0028] Where is the energy storage capacity leased by prosumer i at time t, is the highest leasing price, is the trend of the leasing price curve, is the integration variable;

[0029] The mathematical expression of the cost of the prosumer leasing the energy storage capacity is as follows:

[0030]

[0031] Where and and are the quadratic term, linear term, and constant term coefficients of the leasing energy storage cost of the i-th prosumer, respectively.

[0032] Furthermore, the establishment of the prosumer optimization model in step S3 includes:

[0033] The goal of the prosumer leasing the idle energy storage capacity is to maximize its own net profit. The optimization model mathematical expression of the prosumer leasing the energy storage capacity is as follows:

[0034]

[0035] Among them, is the optimized prosumer leased energy storage capacity, which is the maximum energy storage capacity leased by the prosumer.

[0036] Furthermore, the establishment of the intraday virtual energy storage model in step S4 includes:

[0037] The dispatchable load of the prosumer includes shiftable load and curtailable load; for thermostatically controlled load, its power consumption mathematical expression is as follows:

[0038]

[0039] Among them, is the power consumption of thermostatically controlled load i, is the baseline power consumption of thermostatically controlled load i, is the regulation power of thermostatically controlled load i participating in the response; the thermodynamic kinetic equation of the thermostatically controlled load is as follows:

[0040]

[0041] Among them, is the internal heat capacity of building i, and are the equivalent heat gain and thermal resistance of building i respectively, is the indoor temperature of building i; is the coefficient of thermostatically controlled load i; is the set temperature of thermostatically controlled load i; is the diffusion coefficient of the flexible load; is the Wiener process of random disturbance caused by prosumer behavior or environmental impact;

[0042] Define the auxiliary variables as and and reconstruct the thermodynamic kinetic equation into a virtual energy storage model, and its mathematical expression is as follows:

[0043]

[0044] Among them, represents the remaining capacity of virtual energy storage unit i; represents the energy loss rate of virtual energy storage unit i; and represent the minimum and maximum remaining capacities respectively; and represent the minimum and maximum power constraints respectively; the initial remaining capacity is defined as ; the range of the remaining capacity after termination is , ;

[0045] The virtual energy storage model includes the compensation cost of the microgrid operator, and its mathematical expression is as follows:

[0046]

[0047] Wherein, and are the compensation coefficients of the first constant term and the second constant term respectively, and N is the number of thermostatic control loads.

[0048] Furthermore, the establishment of the intra-day shared energy storage model in step S5 includes:

[0049] The microgrid operator stores or releases electricity into the shared energy storage during the day-ahead stage through the leased energy storage capacity, and the mathematical expression of the shared energy storage model is as follows:

[0050]

[0051] Wherein, is the derivative of, is the remaining energy storage capacity of the prosumer leased by the microgrid operator, represents the energy storage power, and are the minimum and maximum power of the energy storage, represents the energy dissipation rate of the energy storage, represents the efficiency of the energy storage charge and discharge power, and its mathematical expression is as follows:

[0052]

[0053] Wherein, and are the charging and discharging power efficiencies respectively.

[0054] Furthermore, the establishment of the microgrid operator optimization model in step S6 includes:

[0055] The mathematical expression of the cost / income obtained from the power exchange between the operator and the external power grid area is as follows:

[0056]

[0057] Wherein, is the real-time electricity price of the external power grid, is the interaction power between the operator and the external power grid;

[0058] The mathematical expression of the cost / income obtained by the operator through purchasing or selling electricity to the prosumer is as follows:

[0059]

[0060] Among them, is the price at which the operator sells energy to prosumers, is the interactive power between the operator and the prosumer, is a coefficient; the I function is an indicator function;

[0061] The microgrid satisfies the power balance constraint during operation in each time slot:

[0062]

[0063] Among them, is the load demand of prosumer i;

[0064] The mathematical expression of the optimization model of the microgrid operator is as follows:

[0065]

[0066] Among them, the objective function consists of four parts: the revenue / cost generated by the power interaction between the operator and the prosumer the revenue / cost generated by the power interaction between the operator and the external power grid the cost of leasing the energy storage capacity and the compensation cost for motivating the prosumer .

[0067] Furthermore, the Markov decision process in step S7 consists of four parts: state space, action space, state transition probability, and reward function, which are specifically as follows:

[0068] State space: The system state is divided into the day-ahead stage state and the intra-day stage state ; however, the ES capacity state in the day-ahead stage cannot be obtained in advance, and its corresponding state is unobservable; in the intra-day stage, the state is observable, and its mathematical expression is as follows:

[0069]

[0070] Among them, the state information in the intra-day stage includes the load demand of the prosumer, distributed photovoltaic, time-of-use electricity price, real-time electricity price, shared energy storage capacity, and virtual energy storage capacity;

[0071] Action space: The operator's actions are divided into day-ahead stage actions and intra-day stage actions ; During the day-ahead stage, the operator obtains the shared energy storage capacity by formulating a leasing strategy. During the intraday stage, the operator obtains revenue by managing the shared energy storage and virtual energy storage; the action space is defined as:

[0072]

[0073] Among them, and The values of are used to determine the day-ahead leasing strategy, are the power of the shared energy storage and virtual energy storage respectively scheduled by the operator intraday;

[0074] Transition probability function: The transition probability is a dynamic variable, which is gradually estimated by the agent through interaction with the environment;

[0075] Reward function: The reward function of the system is divided into day-ahead reward and intraday reward , and the reward function is defined as:

[0076]

[0077] Among them, is the reward coefficient;

[0078] According to the above Markov decision process formula, the discounted cumulative reward within the time slot t is defined as:

[0079]

[0080] Among them, represents the reward in the day-ahead or intraday time slot , is the discount factor; The goal of the Markov decision process is to maximize the expected cumulative discounted return:

[0081]

[0082] Among them, is described as a mapping function that maps the state to the action , is the mathematical expectation; and represent the day-ahead or intraday state and action respectively.

[0083] Furthermore, in step S8, an asynchronous semi-coupled optimization method based on double-delayed deep deterministic policy gradient is used to solve the Markov decision process, specifically as follows:

[0084] Whether it is the training process in the intraday stage or the day-ahead stage, it is executed through the actor-critic process of double-delayed deep deterministic policy gradient. Among them, the actor network learns the policy π and generates actions in a given state, and the critic network evaluates the learning quality of the actor network under the policy π, that is, the objective of the Markov decision process, maximizing the expected discounted return;

[0085] The critic network completes policy evaluation by constructing an action value function. The action value function quantifies the expected return of taking a specific action in a given state; the action value function is defined as:

[0086]

[0087] where, is the current network parameter, and the value of Q is iteratively updated by using the Bellman equation:

[0088]

[0089] where, is the target Q value, is the target network parameter, and are the next state and action respectively; is the reward for executing action a in state s; is the expected value;

[0090] The Q value is iteratively updated by the double Q learning method:

[0091]

[0092] The parameters of the critic network are updated by minimizing the loss function ; the actor network learns to optimize the policy π by maximizing the expected discounted return, and updates the parameters of the actor network by using the policy gradient ; the policy gradient is defined as follows:

[0093] where, is the gradient of action a; is the gradient of the network parameter ; is the network parameter the policy of the next state S;

[0094] Based on the above update iteration of the optimization method parameters, the optimal scheduling problem of microgrid flexibility resources is solved.

[0095] The present invention also provides a microgrid joint optimal scheduling system integrating heterogeneous flexible resources, including:

[0096] A model establishment module for establishing a photovoltaic load model, a day-ahead energy storage leasing model, a prosumer optimization model, an intra-day virtual energy storage model, an intra-day shared energy storage model, and a microgrid operator optimization model;

[0097] A decision-making process establishment module for establishing a Markov decision-making process and transforming the microgrid operator optimization model into the established Markov decision-making process;

[0098] A solution module for solving the Markov decision-making process.

[0099] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0100] 1. An innovative day-ahead and intra-day two-stage microgrid demand-side resource scheduling strategy is proposed. This strategy uses a curriculum reinforcement learning method for optimal regulation, fully utilizing the schedulable potential of flexible resources and improving the economic efficiency of microgrid operators.

[0101] 2. A cross-time virtual energy storage model is developed to describe the dynamic demand response process of prosumer flexible loads, overcoming the limitations of the static management of existing demand response models, and further improving the economic benefits of microgrid operators.

[0102] 3. A dual-delay deep deterministic policy gradient-based asynchronous semi-coupled optimization method is proposed, solving the problem of difficulty in establishing an accurate model due to the uncertainty and difference in the regulation ability of demand-side resources, and realizing the full utilization of flexible resources and the maximization of the benefits of microgrid operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 It is a flow schematic diagram of the method of the present invention;

[0104] Figure 2 It is a comparison chart of reward values. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0105] The present invention will be further clarified below with reference to the drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, various equivalent forms of modification of the present invention by those skilled in the art all fall within the scope defined by the appended claims of this application.

[0106] Embodiment 1:

[0107] As Figure 1 shown, this embodiment provides a microgrid joint optimal scheduling method integrating heterogeneous flexible resources, including the following steps:

[0108] S1: Establish a photovoltaic load model;

[0109] The microgrid system includes N prosumers with different characteristics. Prosumers can reduce their net load by using the equipped rooftop photovoltaic power generation. The mathematical expressions of photovoltaic power and load demand are as follows:

[0110]

[0111]

[0112] Among them, and are the power of the PV panel and the load power of prosumer i at time t, respectively. and are the diffusion coefficients of the PV panel power and the load power, respectively. and are the Brownian motion processes used to describe the randomness of the PV panel and the load, respectively.

[0113] S2: Establish a day-ahead energy storage leasing model;

[0114] During the day-ahead stage, prosumers can obtain a certain amount of income by leasing their idle energy storage capacity to the microgrid operator.

[0115] The day-ahead energy storage leasing model includes the income and cost of prosumers leasing energy storage capacity.

[0116] The mathematical expression of the income of prosumers leasing energy storage capacity is as follows:

[0117]

[0118] Among them, is the energy storage capacity leased by prosumer i at time t, is the highest leasing price, is the trend of the leasing price curve, is the integration variable;

[0119] The mathematical expression of the cost of prosumers leasing energy storage capacity is as follows:

[0120]

[0121] Among them, and and are the quadratic term, linear term, and constant term coefficients of the leasing energy storage cost of the i-th prosumer, respectively.

[0122] S3: Based on the photovoltaic load model and the day-ahead energy storage leasing model, establish a prosumer optimization model;

[0123] The goal of prosumers leasing idle energy storage capacity is to maximize their own net profit. The mathematical expression of the optimization model for prosumers leasing energy storage capacity is as follows:

[0124]

[0125] where, is the optimized energy storage capacity leased by the prosumer, is the maximum energy storage capacity leased by the prosumer.

[0126] S4: Establish an intraday virtual energy storage model;

[0127] During the intraday stage, when the real-time electricity price is higher than the peak-valley electricity price, the prosumer can reduce the cost of the microgrid operator by adjusting the dispatchable load. The dispatchable load of the prosumer includes shiftable load and curtailable load;

[0128] For thermostatic control loads, the mathematical expression of their power consumption is as follows:

[0129]

[0130] where, is the power consumption of thermostatic control load i, is the baseline power consumption of thermostatic control load i, is the regulation power of thermostatic control load i participating in the response; The thermodynamics kinetic equation of the thermostatic control load is as follows:

[0131]

[0132] where, is the internal heat capacity of building i, and are the equivalent heat gain and thermal resistance of building i respectively, is the indoor temperature of building i; is the coefficient of thermostatic control load i; is the set temperature of thermostatic control load i; is the diffusion coefficient of the flexible load; is the Wiener process of random interference caused by prosumer behavior or environmental impact;

[0133] Define the auxiliary variables as and and reconstruct the thermodynamics kinetic equation into a virtual energy storage model, and its mathematical expression is as follows:

[0134]

[0135] where, represents the remaining capacity of virtual energy storage unit i; Denote the energy loss rate of the virtual energy storage unit \(i\); and denote the minimum and maximum remaining capacities respectively; and denote the minimum and maximum power constraints respectively; The initial remaining capacity is defined as ; The range of the remaining capacity after termination is , ; The conversion process of the demand response model for other flexible loads follows a similar procedure and will not be elaborated here.

[0136] Since the participation of prosumers' virtual energy storage in the microgrid operator's dispatching will lead to a decrease in power comfort, the microgrid operator needs to provide economic compensation to encourage the active participation of prosumers. The compensation cost of the microgrid operator is included in the virtual energy storage model, and its mathematical expression is as follows:

[0137]

[0138] Among them, and are the compensation coefficients of the first constant term and the second constant term respectively, and \(N\) is the number of thermostatic control loads.

[0139] S5: Establish an intraday shared energy storage model;

[0140] The microgrid operator stores or releases electricity into the shared energy storage in the day-ahead stage through the leased energy storage capacity. The mathematical expression of the shared energy storage model is as follows:

[0141]

[0142] Among them, is 's derivative, is the remaining energy storage capacity leased by the microgrid operator from the prosumer, represents the energy storage power, and are the minimum and maximum powers of the energy storage respectively, represents the energy dissipation rate of the energy storage; represents the efficiency of the energy storage charging and discharging power, and its mathematical expression is as follows:

[0143]

[0144] Among them, and are the charging and discharging power efficiencies respectively.

[0145] S6: Based on the prosumer optimization model, the intraday virtual energy storage model, and the intraday shared energy storage model, establish the microgrid operator optimization model;

[0146] The microgrid operator purchases electricity from the external power grid and resells it to prosumers. When the operator has an excess of electricity, it can also sell electricity to the external power grid. The mathematical expression for the cost / revenue obtained from the electricity exchange between the operator and the external power grid area is as follows:

[0147]

[0148] where, is the real-time electricity price of the external power grid, is the interactive power between the operator and the external power grid. A negative value indicates the electricity sold by the operator to the external power grid, while a positive value indicates the electricity purchased by the operator from the external power grid;

[0149] Similarly, when the photovoltaic power generation of the prosumer exceeds the load demand, the operator can purchase electricity from the prosumer; the mathematical expression for the cost / revenue obtained by the operator through purchasing or selling electricity to the prosumer is as follows:

[0150]

[0151] where, is the price at which the operator sells energy to the prosumer, is the interactive power between the operator and the prosumer, is a coefficient whose value range is (0, 1); the I function is an indicator function;

[0152] To ensure the security of system operation, the microgrid needs to satisfy the power balance constraint during operation in each time slot:

[0153]

[0154] where, is the load demand of prosumer i;

[0155] The mathematical expression of the optimization model of the microgrid operator is as follows:

[0156]

[0157] where the objective function consists of four parts: the revenue / cost generated by the electricity interaction between the operator and the prosumer 、the revenue / cost generated by the electricity interaction between the operator and the external power grid 、the cost of leasing energy storage capacity and the compensation cost for motivating the prosumer .

[0158] S7: Convert the optimization model of the microgrid operator into the established Markov decision process;

[0159] The Markov decision process consists of four parts: state space, action space, state transition probability, and reward function, which are specifically as follows:

[0160] State space: The system state is divided into the day-ahead stage state and the intra-day stage state ; however, the ES capacity state in the day-ahead stage cannot be obtained in advance, and its corresponding state is unobservable; in the intra-day stage, the state is observable, and its mathematical expression is as follows:

[0161]

[0162] Among them, the state information in the intra-day stage includes the load demand of prosumers, distributed photovoltaic power, time-of-use electricity price, real-time electricity price, shared energy storage capacity, and virtual energy storage capacity;

[0163] Action space: The actions of the operator are divided into day-ahead stage actions and intra-day stage actions ; in the day-ahead stage, the operator obtains the shared energy storage capacity by formulating a leasing strategy. In the intra-day stage, the operator manages the shared energy storage and virtual energy storage to obtain income; the action space is defined as:

[0164]

[0165] Among them, and The values of are used to determine the day-ahead leasing strategy, are the powers of the shared energy storage and virtual energy storage scheduled by the operator intra-day, respectively;

[0166] Transition probability function: The transition probability is usually not a static parameter known in advance, but a dynamic variable. The agent estimates it step by step through interactions with the environment;

[0167] Reward function: The reward function of the system is divided into day-ahead reward and intra-day reward , in the day-ahead stage, the goal of the operator is to minimize the leasing cost of the energy storage capacity while meeting the real-time capacity demand of the energy storage; in the intra-day stage, the goal of the operator is to maximize its income; in summary, the reward function is defined as:

[0168]

[0169] Among them, is the reward coefficient;

[0170] According to the above Markov decision process formula, the discounted cumulative reward within time slot t is defined as:

[0171]

[0172] Among them, represents the reward before or within the time slot days, is the discount factor; the goal of the Markov decision process is to maximize the expected cumulative discounted reward:

[0173]

[0174] Among them, is described as a mapping function that maps the state to the action , is the mathematical expectation; and represent the state and action before or within the day, respectively.

[0175] S8: Solve the Markov decision process in step S7, update and iterate the scheduling parameters, and achieve the optimal scheduling of the microgrid;

[0176] In this embodiment, an asynchronous semi-coupled optimization method based on double-delayed deep deterministic policy gradient is used to solve the Markov decision process, specifically as follows:

[0177] For the training process in both the within-day stage and the day-ahead stage, it is executed through the actor-critic process of double-delayed deep deterministic policy gradient. Among them, the actor network learns the policy π and generates actions under a given state, and the critic network evaluates the learning quality of the actor network under the policy π, that is, the goal of the Markov decision process, the maximized expected discounted reward;

[0178] The critic network completes policy evaluation by constructing an action-value function, and the action-value function quantifies the expected return of taking a specific action under a given state; the action-value function is defined as:

[0179]

[0180] Among them, are the current network parameters, and the value of Q is iteratively updated by using the Bellman equation:

[0181]

[0182] Among them, is the target Q value, are the target network parameters, and are the next state and action, respectively; is the reward for executing action a in state s; is the expected value;

[0183] To suppress the overestimation bias of the Q-value and steadily train the learning strategy, the Q-value is iteratively updated through the double Q-learning method:

[0184]

[0185] This method utilizes two independent critic networks and , and when calculating, the minimum value of the two is taken to reduce the Q-value estimation bias and improve stability.

[0186] The loss function of the critic network is usually based on the time difference error, and its goal is to make the expected Q-value as close as possible to the true expected return. The parameters of the critic network are updated by minimizing the loss function ; the actor network learns to optimize the policy π by maximizing the expected discounted return, and updates the parameters of the actor network by using the policy gradient ; the policy gradient is defined as follows:

[0187]

[0188] where, is the gradient of action a; is the gradient of the network parameter ; is the network parameter under the policy of state S;

[0189] Based on the update iteration of the above optimization method parameters, the optimal scheduling problem of microgrid flexibility resources is solved.

[0190] Embodiment 2:

[0191] Based on the method of Embodiment 1, this embodiment provides a microgrid joint optimal scheduling system integrating heterogeneous flexibility resources, including:

[0192] A model establishment module for establishing a photovoltaic load model, a day-ahead energy storage leasing model, a prosumer optimization model, an intra-day virtual energy storage model, an intra-day shared energy storage model, and a microgrid operator optimization model;

[0193] A decision process establishment module for establishing a Markov decision process and transforming the microgrid operator optimization model into the established Markov decision process;

[0194] A solution module for solving the Markov decision process.

[0195] Embodiment 3:

[0196] To verify the effectiveness of the method of the present invention, this embodiment conducts a comparison through simulation experiments, which is specifically as follows:

[0197] In this embodiment, during 10,000 training iterations, the reward values obtained by the method proposed in the present invention are compared with three existing methods, which are TD3, DDPG, and PSO respectively, and the reward value comparison graph as Figure 2 shown is obtained.

[0198] As Figure 2 shown, due to being in the exploration stage, the reward value of the method proposed in the present invention is relatively low in the initial stage, but as time goes by, the reward value gradually stabilizes. Specifically, the method proposed in the present invention reaches stability after about 4,100 iterations, while the TD3 method requires about 5,200 iterations to reach stability. This shows that the method of the present invention combined with curriculum learning significantly speeds up the training process and reaches stability 19.23% faster in terms of the number of iterations than TD3. In addition, the method proposed in the present invention and TD3 both finally reach a reward value of about 58, while the reward values of DDPG and PSO are lower than 40. Therefore, the method proposed in the present invention not only has an advantage in the convergence speed, but also significantly outperforms DDPG and PSO in terms of performance.

Claims

1. A microgrid joint optimal scheduling method integrating heterogeneous flexible resources, characterized in that, It includes the following steps: S1: Establish a photovoltaic load model; S2: Establish a day-ahead energy storage leasing model; S3: Based on the photovoltaic load model and the day-ahead energy storage leasing model, establish a prosumer optimization model; S4: Establish an intra-day virtual energy storage model; S5: Establish an intra-day shared energy storage model; S6: Based on the prosumer optimization model, the intra-day virtual energy storage model and the intra-day shared energy storage model, establish a microgrid operator optimization model; S7: Convert the microgrid operator optimization model into a well-established Markov decision process; S8: Solve the Markov decision process in step S7, update and iterate the scheduling parameters, and realize the optimal scheduling of the microgrid.

2. The microgrid joint optimal scheduling method for integrating heterogeneous flexible resources according to claim 1, characterized in that The establishment of the photovoltaic load model in step S1 includes: The mathematical expressions of photovoltaic power and load demand are as follows: ； ； Among them, and are the power of the PV panel and the load power of prosumer i within time t, respectively; and are the diffusion coefficients of the PV panel power and the load power, respectively, and are the Brownian motion processes used to describe the randomness of the PV panel and the load, respectively.

3. A microgrid joint optimal scheduling method for integrating heterogeneous flexible resources according to claim 2, characterized in that The establishment of the day-ahead energy storage leasing model in step S2 includes: The day-ahead energy storage leasing model includes the revenue and cost of the prosumer leasing the energy storage capacity; The mathematical expression of the revenue of the prosumer leasing the energy storage capacity is as follows: ； Among them, is the energy storage capacity leased by prosumer i at time t, is the highest leasing price, is the trend of the leasing price curve, is the integral variable; The mathematical expression of the cost of the prosumer leasing the energy storage capacity is as follows: ； Among them, , and are the quadratic term, linear term, and constant term coefficients of the rental energy storage cost of the i-th prosumer, respectively.

4. A microgrid joint optimal scheduling method for integrating heterogeneous flexible resources according to claim 3, characterized in that, The establishment of the prosumer optimization model in step S3 includes: The goal of the prosumer leasing the idle energy storage capacity is to maximize its own net profit. The mathematical expression of the optimization model of the prosumer leasing the energy storage capacity is as follows: ； Among them, is the optimized prosumer leased energy storage capacity, which is the maximum energy storage capacity leased by the prosumer.

5. The microgrid joint optimal scheduling method for integrating heterogeneous flexible resources according to claim 4, characterized in that The establishment of the intra-day virtual energy storage model in step S4 includes: The dispatchable load of the prosumer includes shiftable load and curtailable load; for the thermostatic control load, its power consumption mathematical expression is as follows: ； wherein, is the power consumption of the thermostatic control load i, is the baseline power consumption of the thermostatic control load i, is the regulation power of the thermostatic control load i participating in the response; the thermodynamic kinetic equation of the thermostatic control load is as follows: ； Among them, is the internal heat capacity of building i, and are the equivalent heat gain and thermal resistance of building i respectively, is the indoor temperature of building i; is the coefficient of the constant temperature control load i; is the set temperature of the constant temperature control load i; is the diffusion coefficient of the flexible load; is the Wiener process of random disturbance caused by prosumer behavior or environmental impact; Define the auxiliary variables as and , and reconstruct the thermodynamics kinetic equation into a virtual energy storage model, whose mathematical expression is as follows: ； Among them, represents the remaining capacity of the virtual energy storage unit i; represents the energy loss rate of the virtual energy storage unit i; and represent the minimum and maximum remaining capacities respectively; and represent the minimum and maximum power constraints respectively; The initial remaining capacity is defined as ; The remaining capacity range after termination is , ; The virtual energy storage model includes the compensation cost of the microgrid operator, and its mathematical expression is as follows: ； wherein, and are the compensation coefficients of the first constant term and the second constant term respectively, and N is the number of constant temperature control loads.

6. The microgrid joint optimal scheduling method for integrating heterogeneous flexible resources according to claim 5, characterized in that The establishment of the intra-day shared energy storage model in step S5 includes: The microgrid operator stores or releases electricity into the shared energy storage in the day-ahead stage through the leased energy storage capacity. The mathematical expression of the shared energy storage model is as follows: ； Among them, is the derivative of, is the remaining energy storage capacity leased by the microgrid operator from the prosumer, represents the energy storage power, and are the minimum and maximum power of the energy storage respectively, represents the energy dissipation rate of the energy storage, represents the efficiency of the energy storage charging and discharging power, and its mathematical expression is as follows: ； Among them, and are the efficiencies of the charging and discharging powers respectively.

7. A microgrid joint optimal scheduling method for integrating heterogeneous flexible resources according to claim 6, characterized in that, The establishment of the microgrid operator optimization model in step S6 includes: The mathematical expression of the cost / revenue obtained from the power exchange between the operator and the external grid area is as follows: ； Among them, is the real-time electricity price of the external power grid, is the interactive power between the operator and the external power grid; The mathematical expression of the cost / revenue obtained by the operator from buying or selling electricity to the prosumer is as follows: ； wherein, is the price at which the operator sells energy to prosumers, is the interaction power between the operator and the prosumers, is a coefficient; the I function is an indicator function; The microgrid satisfies the power balance constraint during operation in each time slot: ； Among them, is the load demand of prosumer i; The mathematical expression of the microgrid operator optimization model is as follows: ； Among them, the objective function consists of four parts: the revenue / cost generated by the electricity interaction between the operator and the prosumer , the revenue / cost generated by the electricity interaction between the operator and the external power grid , the cost of leasing energy storage capacity , and the compensation cost for incentivizing the prosumer .

8. A microgrid joint optimal scheduling method for integrating heterogeneous flexible resources according to claim 7, characterized in that, The Markov decision process in step S7 consists of four parts: state space, action space, state transition probability, and reward function, which are specifically as follows: State space: The system state is divided into the day-ahead stage state and the intraday stage state ; However, the ES capacity status at the previous stage cannot be obtained in advance, and its corresponding status is unobservable; during the intraday stage, the status is observable, and its mathematical expression is as follows: ； Among them, the state information in the intra-day stage includes the load demand of the prosumer, distributed photovoltaics, time-of-use electricity price, real-time electricity price, shared energy storage capacity, and virtual energy storage capacity; Action Space: The actions of the operator are divided into day-ahead stage actions and intra-day stage actions ; In the day-ahead stage, the operator obtains shared energy storage capacity by formulating leasing strategies. In the intra-day stage, the operator obtains revenue by managing shared energy storage and virtual energy storage; The action space is defined as: ； ； Among them, and The values are used to determine the day-ahead leasing strategy, which are the shared energy storage and virtual energy storage powers for the operator's intra-day scheduling respectively; Transition probability function: The transition probability is a dynamic variable, and the agent gradually estimates it through interaction with the environment; Reward function: The system's reward function is divided into day-ahead reward and intra-day reward , and the reward function is defined as: ； ； Among them, is the reward coefficient; According to the Markov decision process formula, the discounted cumulative reward in time slot t is defined as: ； Among them, indicates the reward on or within the time slot before the date, is the discount factor; the goal of the Markov decision process is to maximize the expected cumulative discounted reward: ； Among them, is described as a mapping function that maps the state to the action , is the mathematical expectation; and represent the state and action of day-ahead or intra-day, respectively.

9. The joint optimal scheduling method for a microgrid integrating heterogeneous flexible resources according to claim 8, characterized in that In step S8, an asynchronous semi-coupled optimization method based on the twin-delayed deep deterministic policy gradient is used to solve the Markov decision process, which is specifically as follows: Whether it is the training process in the intraday stage or the day-ahead stage, it is executed through the actor-critic process of double-delay deep deterministic policy gradient, where the actor network learns the policy π and generates actions in a given state, and the critic network evaluates the learning quality of the actor network under the policy π, that is, the objective of the Markov decision process, maximizing the expected discounted return; The critic network completes policy evaluation by constructing an action value function, which quantifies the expected return of taking a specific action in a given state; the action value function is defined as: ； Among them, are the current network parameters, and the value of Q is iteratively updated by using the Bellman equation: ； Among them, is the target Q value, is the target network parameter, and are the next state and action respectively; is the reward for executing action a in state s; is the expected value; The Q value is iteratively updated through the double Q-learning method: ； The parameters of the critic network are updated by minimizing the loss function ; the actor network learns to optimize the policy π by maximizing the expected discounted return, and updates the parameters of the actor network by using the policy gradient ; the policy gradient is defined as follows: ； Among them, is the gradient of action a; is the gradient of network parameter ; is the policy of the next state S of network parameter ; Based on the above update iteration of the optimization method parameters, the optimal scheduling problem of microgrid flexibility resources is solved.

10. A microgrid joint optimal scheduling system integrating heterogeneous flexible resources, characterized in that, Including: A model establishment module for establishing a photovoltaic load model, a day-ahead energy storage leasing model, a prosumer optimization model, an intraday virtual energy storage model, an intraday shared energy storage model, and a microgrid operator optimization model; A decision process establishment module for establishing a Markov decision process and transforming the microgrid operator optimization model into the established Markov decision process; A solution module for solving the Markov decision process.

Images