User side energy storage system operation method and system considering extremely cold dark scene

Abstract

The invention discloses a user-side energy storage system operation method and system considering an extremely cold dark scene, and relates to the technical field of energy storage configuration, and the method comprises the steps: in a wind and light output scene set in extreme weather, according to the influence of the discharge depth on the cycle life of user-side energy storage, calculating the cycle life of the user-side energy storage; establishing a quintuple MDP reinforcement learning model composed of a state set, an action set, a state transition probability, a reward set and a reward discount factor; and based on the quintuple MDP reinforcement learning model, minimizing the operation cost of the user side energy storage by using an MILP algorithm, and carrying out optimization control on the system operation of the user side energy storage. According to the method, a new thought and a technical path are provided for the stability and the reliability of the optimal operation strategy of the energy storage system of the battery under the consideration of the generation comprehensiveness of the extremely cold dark scene.

Description

Technical Field

[0001] The present invention relates to the technical field of energy storage configuration, and in particular to a method and system for operating a user-side energy storage system in an extremely cold and lightless scenario. Background Art

[0002] With the construction of a new power system based on renewable energy, enhancing the ability to absorb renewable energy has become a challenge, and the development of distributed photovoltaic storage systems on the user side is an important measure. However, under extreme weather conditions, the output characteristics of photovoltaics show significant volatility and uncertainty, which greatly affects the supply and demand balance on the user side and directly affects the production and life of power users. The regulation capability of the user-side energy storage system can help realize the flexible scheduling of electric energy, improve the absorption capacity of photovoltaics, and realize price arbitrage through "low storage and high release". The sources of energy storage income on the user side also include demand management, demand response and power market services, which effectively save electricity costs for users. The technical field to which the present invention belongs involves the generation of extreme weather scenarios and the optimization of energy storage operation strategies and investment returns of user-side photovoltaic storage systems, especially in the energy storage MDP solution algorithm.

[0003] At present, the research on the generation technology of renewable energy output scenarios under extreme weather conditions mainly analyzes historical meteorological and renewable energy output data, and uses deep learning and statistical methods to build extreme scenario generation models, which can provide a basis for high-risk scenario evaluation for the power system. Under the premise of considering the operation constraints of energy storage, minimizing the operating cost of the energy storage system is a strategic goal, and the research focus is mainly on the user-side energy storage economic model. As an important means to improve the flexibility of the power system, energy storage can effectively alleviate the risks brought by the volatility of renewable energy by optimizing configuration and operation strategies. At the same time, user-side energy storage has a low initial investment and high flexibility, and users can choose the appropriate energy storage capacity according to their needs. In addition, user-measured energy storage can further improve the quality of electricity, alleviate power fluctuations, and stabilize voltage and frequency.

[0004] However, the existing technology still faces many shortcomings. First, there is a certain gap in the comprehensiveness and accuracy of extreme scenario generation technology, which is difficult to fully reflect the extreme risks in the actual system; second, the traditional energy storage operation optimization problem is modeled as a mixed integer programming or dynamic programming problem. These methods require the established mathematical model to be very accurate. However, as the complexity of the system increases and when uncertain factors are considered, the scalability and convergence of such methods are difficult to meet real-time requirements; third, deep reinforcement learning (DRL) has been widely used as an important algorithm for regulating user-side energy storage to absorb photovoltaic power generation and meet user electricity demand. However, during the application process, it is difficult for DRL agents to strictly enforce operating constraints, resulting in unreliable actions, threatening the safety of energy storage system operation.

[0005] Therefore, how to accurately model the economic model of user-side energy storage while taking into account the comprehensiveness of extreme scenario generation, making the intelligent agent strictly implement energy storage operation constraints and the algorithm model can be accurately solved, is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In order to solve the above problems, the purpose of the present invention is to provide a reinforcement learning optimization operation technology for user-side energy storage systems considering extremely cold and lightless scenarios, aiming to take user-side energy storage as the research object. First, under extreme weather conditions, with the goal of minimizing the energy storage operation cost, the degradation process of energy storage during charging and discharging is fully considered, and a real-time operation optimization model considering battery degradation cost and extreme photovoltaic conditions is established. The MDP decision-making process is introduced and an energy storage operation MDP model containing power balance constraints is constructed; second, a MILP-TD3 algorithm is proposed to solve the optimal operation strategy, so that the algorithm can strictly execute the operation constraints in the action space and ensure the reliability of the desired strategy.

[0007] In order to achieve the above technical objectives, the present application provides a method for operating a user-side energy storage system in an extremely cold and dark scenario, comprising the following steps:

[0008] Under the scenario set of wind and solar power output in extreme weather conditions, according to the impact of discharge depth on the cycle life of user-side energy storage, a five-tuple MDP reinforcement learning model consisting of a state set, an action set, a state transition probability, a reward set, and a reward discount factor was built;

[0009] Based on the five-tuple MDP reinforcement learning model, the MILP algorithm is used to minimize the operating cost of the user-side energy storage and optimize the system operation of the user-side energy storage.

[0010] Preferably, in the process of obtaining the set of wind and solar power output scenarios under extreme weather conditions, the historical wind and solar power output and meteorological data are preprocessed, including interpolation, dimensionality reduction, standardization and smoothing;

[0011] Based on the pre-processed historical wind and solar output and meteorological data, a PV low output threshold is defined to determine whether a certain day is a PV low output day;

[0012] Based on the low-output days of photovoltaic power generation, the conditions of extremely cold and dark days are defined, the extremely cold and dark days in the historical data are identified, and the 24-hour photovoltaic output curves within the extremely cold and dark days are included in the extremely cold and dark scene set;

[0013] For the curves in the extremely cold and lightless scene set, k-means clustering is used to obtain the set of wind and solar power output scenes under extremely cold and lightless extreme weather conditions.

[0014] Preferably, in the process of defining the photovoltaic low output threshold, the photovoltaic low output threshold is defined by using cubic spline interpolation according to the frequency distribution of the photovoltaic daily average output, and by obtaining the empirical probability distribution and cumulative probability distribution of the photovoltaic daily average output.

[0015] Preferably, in the process of analyzing the influence of the discharge depth on the cycle life of the user-side energy storage, the influence of the discharge depth on the cycle life of the user-side energy storage is analyzed according to the energy storage system operation model of the user-side energy storage, wherein:

[0016] Considering the degradation cost of the energy storage battery and taking minimizing the user-side operating cost as the goal, an objective function is constructed, and the energy storage system operation model is constructed by defining constraints, including power balance constraints, charging and discharging power constraints, and SOC and its continuity constraints.

[0017] Preferably, in the process of obtaining the influence of discharge depth on the cycle life of user-side energy storage, based on the energy storage system operation model, the lithium battery cycle life index is defined according to the lithium battery cycle life and the single-cycle battery degradation cost, and based on the actual capacity and discharge depth, the influence of discharge depth on the cycle life of user-side energy storage is obtained.

[0018] Preferably, in the process of constructing the five-tuple MDP reinforcement learning model, the user load power, photovoltaic power generation power, energy storage SOC and time-of-use electricity price are used as the state set;

[0019] The charging and discharging power of energy storage is used as the action set;

[0020] The uncertainty of the energy storage system environment is simulated through the state transfer function to obtain the state transfer probability;

[0021] Through the reward function, the immediate return obtained from charging and discharging operations of energy storage in a certain period of time is obtained;

[0022] The power balance constraint is used as a constraint condition of the five-tuple MDP reinforcement learning model, and the violation of the power balance constraint is added as a penalty term to the reward function as a reward discount factor.

[0023] Preferably, in the process of minimizing the operating cost of user-side energy storage by using the MILP algorithm, the MILP-TD3 algorithm is used to minimize the operating cost of user-side energy storage, wherein the MILP-TD3 algorithm includes two processes, training and online execution. The training process is used to estimate the neural network parameters of the approximate action value function, and the online execution process is used to select actions to be executed based on the obtained action value function, so as to minimize the operating cost of user-side energy storage.

[0024] The present invention also discloses a user-side energy storage system operation system considering an extremely cold and lightless scene, which is used for the user-side energy storage system operation method considering an extremely cold and lightless scene mentioned above, comprising:

[0025] Database module, used to store system data;

[0026] Database logic control module, used to control data interaction and algorithm logic implementation between the database module and other system modules;

[0027] Data collection and processing module, used to collect historical wind and solar power output and meteorological data, and generate a set of wind and solar power output scenarios under extreme weather conditions;

[0028] The model building module is used to build a five-tuple MDP reinforcement learning model consisting of a state set, an action set, a state transition probability, a reward set, and a reward discount factor based on the impact of discharge depth on the cycle life of user-side energy storage under a set of wind and solar power output scenarios under extreme weather conditions;

[0029] The optimization control module is used to minimize the operating cost of the user-side energy storage based on the five-tuple MDP reinforcement learning model and utilize the MILP algorithm to optimize the system operation of the user-side energy storage.

[0030] The present invention discloses the following technical effects:

[0031] The present invention generates typical extreme weather scenarios by combining meteorological data and user-side operation information, which can accurately simulate the impact of extreme weather on the user side, making the generated typical scenarios more representative;

[0032] The present invention configures an energy storage system on the user side, allowing users to store electricity purchased from renewable energy (such as solar energy, wind energy) or the power grid and use it when demand is high, thereby improving energy self-sufficiency and reducing dependence on the external power grid. In areas where peak and valley electricity prices are implemented, energy storage can effectively reduce the electricity consumption burden during peak hours and reduce electricity bills.

[0033] The present invention takes into account the battery degradation cost in the real-time operation optimization of energy storage: the battery of the energy storage system will degrade during the charging and discharging process, and this degradation will cause the battery capacity and efficiency to decrease, thereby affecting its service life and operating cost. By considering the degradation process, the residual value and operating cost of the energy storage system can be predicted more accurately, thereby optimizing the operation strategy of energy storage, reducing excessive load or frequent charging and discharging caused by degradation, and reducing the total operating cost of the system.

[0034] The present invention uses the MDP reinforcement learning model. The five-tuple MDP model provides a concise and standardized framework that can clearly describe the key elements in reinforcement learning. Through this standardized representation, any reinforcement learning problem can be converted into an MDP problem, thereby simplifying the understanding and analysis of the problem. In terms of structured expression, elements such as state space, action space, reward, and transition probability are clear at a glance, so that the complexity of the problem can be decomposed into these basic components, helping to analyze the problem more quickly.

[0035] The present invention combines the mixed integer linear programming (MILP) and TD3 (Twin Delayed Deep Deterministic Policy Gradient) reinforcement learning algorithms through the MILP-TD3 algorithm, and can solve problems with mixed discrete and continuous decision spaces. In particular, in the field of optimization decision and control, MILP-TD3 can add structured optimization ideas to the traditional reinforcement learning algorithm, so that when faced with complex decision-making problems, multi-objective optimization can be performed efficiently, and constraints and nonlinear problems that are difficult to solve by some classical reinforcement learning algorithms can be handled. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is the extreme weather scene generation described in the present invention;

[0038] Figure 2 This is the MILP-PD3 algorithm training process described in the present invention;

[0039] Figure 3 It is the online execution process of the MILP-TD3 algorithm described in the present invention;

[0040] Figure 4 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0041] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0042] The present invention provides a method for operating a user-side energy storage system in an extremely cold and lightless scenario, comprising the following steps:

[0043] Under the scenario set of wind and solar power output in extreme weather conditions, according to the impact of discharge depth on the cycle life of user-side energy storage, a five-tuple MDP reinforcement learning model consisting of a state set, an action set, a state transition probability, a reward set, and a reward discount factor was built;

[0044] Based on the five-tuple MDP reinforcement learning model, the MILP algorithm is used to minimize the operating cost of the user-side energy storage and optimize the system operation of the user-side energy storage.

[0045] Further preferably, the user-side energy storage system operation method provided by the present invention in the extremely cold and lightless scene considers, in the process of obtaining the wind and solar power output scene set under extreme weather, pre-processes the historical wind and solar power output and meteorological data, including interpolation, dimensionality reduction, standardization and smoothing;

[0046] Based on the pre-processed historical wind and solar output and meteorological data, a PV low output threshold is defined to determine whether a certain day is a PV low output day;

[0047] Based on the low-output days of photovoltaic power generation, the conditions of extremely cold and dark days are defined, the extremely cold and dark days in the historical data are identified, and the 24-hour photovoltaic output curves within the extremely cold and dark days are included in the extremely cold and dark scene set;

[0048] For the curves in the extremely cold and lightless scene set, k-means clustering is used to obtain the set of wind and solar power output scenes under extremely cold and lightless extreme weather conditions.

[0049] Further preferably, the method for operating the user-side energy storage system in the extremely cold and light-free scenario provided by the present invention, in the process of defining the photovoltaic low output threshold, adopts cubic spline interpolation according to the frequency distribution of the average daily photovoltaic output, and defines the photovoltaic low output threshold by obtaining the empirical probability distribution and cumulative probability distribution of the average daily photovoltaic output.

[0050] Further preferably, the method for operating the user-side energy storage system in the extremely cold and lightless scenario provided by the present invention analyzes the influence of the discharge depth on the cycle life of the user-side energy storage according to the energy storage system operation model of the user-side energy storage in the process of analyzing the influence of the discharge depth on the cycle life of the user-side energy storage, wherein:

[0051] Considering the degradation cost of the energy storage battery and taking minimizing the user-side operating cost as the goal, an objective function is constructed, and the energy storage system operation model is constructed by defining constraints, including power balance constraints, charging and discharging power constraints, and SOC and its continuity constraints.

[0052] Further preferably, the method for operating the user-side energy storage system in the extremely cold and lightless scenario provided by the present invention, in the process of obtaining the influence of the discharge depth on the cycle life of the user-side energy storage, is based on the energy storage system operation model, defines the lithium battery cycle life index according to the lithium battery cycle life and the single-cycle battery degradation cost, and obtains the influence of the discharge depth on the cycle life of the user-side energy storage according to the actual capacity and discharge depth.

[0053] Further preferably, the user-side energy storage system operation method provided by the present invention in the extremely cold and lightless scenario takes the user load power, photovoltaic power generation power, energy storage SOC and time-of-use electricity price as the state set in the process of constructing the five-tuple MDP reinforcement learning model;

[0054] The charging and discharging power of energy storage is used as the action set;

[0055] The uncertainty of the energy storage system environment is simulated through the state transfer function to obtain the state transfer probability;

[0056] Through the reward function, the immediate return obtained from charging and discharging operations of energy storage in a certain period of time is obtained;

[0057] The power balance constraint is used as a constraint condition of the five-tuple MDP reinforcement learning model, and the violation of the power balance constraint is added as a penalty term to the reward function as a reward discount factor.

[0058] Further preferably, the user-side energy storage system operation method provided by the present invention considering the extremely cold and lightless scenario, in the process of minimizing the operating cost of the user-side energy storage by using the MILP algorithm, uses the MILP-TD3 algorithm to minimize the operating cost of the user-side energy storage, wherein the MILP-TD3 algorithm includes two processes, training and online execution, the training process is used to estimate the neural network parameters of the approximate action value function, and the online execution process is used to select actions to execute based on the obtained action value function, so as to minimize the operating cost of the user-side energy storage.

[0059] The present invention also discloses a user-side energy storage system operation system considering an extremely cold and lightless scene, which is used for the user-side energy storage system operation method considering an extremely cold and lightless scene mentioned above, comprising:

[0060] Database module, used to store system data;

[0061] Database logic control module, used to control data interaction and algorithm logic implementation between the database module and other system modules;

[0062] Data collection and processing module, used to collect historical wind and solar power output and meteorological data, and generate a set of wind and solar power output scenarios under extreme weather conditions;

[0063] The model building module is used to build a five-tuple MDP reinforcement learning model consisting of a state set, an action set, a state transition probability, a reward set, and a reward discount factor based on the impact of discharge depth on the cycle life of user-side energy storage under a set of wind and solar power output scenarios under extreme weather conditions;

[0064] The optimization control module is used to minimize the operating cost of the user-side energy storage based on the five-tuple MDP reinforcement learning model and utilize the MILP algorithm to optimize the system operation of the user-side energy storage.

[0065] Example: Figure 1-4 As shown, the present invention provides a reinforcement learning optimization operation strategy for a user-side energy storage system in an extremely cold and lightless scenario, including the following steps:

[0066] Step 1: Based on historical wind and solar power output and meteorological data, establish a set of wind and solar power output scenarios under extreme weather conditions.

[0067] Step 2 takes user-side energy storage as the research object and establishes a user-side energy storage optimization system operation model.

[0068] Step 3: The cycle life of the energy storage battery gradually decays during operation. Considering the impact of discharge depth on cycle life, a battery degradation cost model is established.

[0069] Step 4 models a five-tuple MDP reinforcement learning model including five elements: state set, action set, state transition probability, reward set and reward discount factor.

[0070] Step 5 uses the MILP algorithm to minimize the operating cost of the user-side energy storage. The training process of the MILP algorithm estimates the neural network parameters that approximate the action value function, and the online execution process selects actions to execute based on the obtained action value function.

[0071] In step 1, based on historical photovoltaic output and meteorological data, a set of photovoltaic output scenarios for extreme cold and no light weather is generated, which includes the following steps:

[0072] Step 1.1 Preprocess the historical wind and solar power output and meteorological data, including interpolation, dimensionality reduction, normalization and smoothing to ensure data quality.

[0073] Step 1.2 defines the PV low output threshold, which is used to determine whether a certain day is a PV low output day, as follows:

[0074] According to the 8760h photovoltaic output curve of historical years, the average daily photovoltaic output is calculated respectively;

[0075] Taking the daily photovoltaic output as a sample, the frequency distribution of the average daily photovoltaic output in historical years (P v );

[0076] According to the frequency distribution, cubic spline interpolation is used, and the negative values ​​that may be included after interpolation are set to zero, and the empirical probability distribution PDF (P v ), and calculate the cumulative probability distribution CDF (P v );

[0077] Select CDF(P v ) is equal to 0.2, take the corresponding P v value, as the photovoltaic low output threshold, denoted as P v,low ;

[0078] Step 1.3 defines the conditions for extremely cold and dark days and identifies extremely cold and dark days in historical data, as follows:

[0079] The date that meets the following conditions in the historical data is considered to be an extremely cold and dark day, and the 24-hour photovoltaic output curve of the day is included in the extremely cold and dark scene set ψ v,low :

[0080] Low PV output: The average PV output on that day is lower than the PV low output threshold P. v,low ;

[0081] Low temperature: The lowest temperature of the day is not higher than 0℃;

[0082] Low light: The weather is cloudy, rainy or snowy on that day;

[0083] Step 1.4 Set ψ for the extremely cold and lightless scene v,low The curves in the figure are clustered by k-means to obtain a typical wind and solar output scene set under extreme cold and lightless weather conditions.

[0084] In step 2, the photovoltaic power generation system provides power to users, and the energy storage system can charge and store excess power, and release power to users when the photovoltaic output is insufficient. At the same time, the photovoltaic storage system can trade with the power grid to meet user needs and absorb photovoltaic power. The establishment of the energy storage system operation model includes the following steps:

[0085] Step 2.1 The user-side energy storage system mainly saves electricity costs by arbitrage between peak and valley prices. The goal of the optimization problem is to minimize the user-side operating costs, taking into account the degradation cost of the energy storage battery. The mathematical expression of the objective function is as follows:

[0086]

[0087] Where t is the time period, COST grid (t) is the cost of electricity purchased from the power grid by the user; COST ess (t) is the degradation cost of the energy storage battery; T0 is the number of time periods in a cycle, with one day as a cycle, and there are 24 time periods in total; P grid (t) is the power purchased by the user from the power grid; ρ(t) is the time-of-use electricity price.

[0088] Step 2.2 defines the constraints of the user energy storage system model, including power balance constraints, charge and discharge power constraints, and SOC and continuity constraints. The mathematical expressions are as follows:

[0089]

[0090] Where P load (t) is the user load power; P pv (t) is the power of photovoltaic power generation; P ess (t) is the charging and discharging power of the energy storage system, P ess (t)>0 indicates charging state, P ess (t)<0 indicates the discharge state, P ess (t) = 0 means neither charging nor discharging. ess,max is the rated power of the energy storage system. SOC(t) is the state of charge of the energy storage system; E ess,max is the rated capacity of the energy storage system; η ch , η dis are the charging and discharging efficiency of the energy storage system, respectively.

[0091] In step 3, the degradation mechanism of energy storage batteries is complex, mainly composed of calendar life and cycle life. Calendar life refers to the attenuation of battery capacity over time, which is usually related to the chemical properties and environmental factors of the battery; while cycle life refers to the maximum number of complete charge and discharge cycles allowed for the battery, which is mainly affected by the charge and discharge actions of the battery. The energy storage operation optimization problem only considers the impact of cycle life. The cycle life gradually decays with the operation of the system. Considering the impact of the depth of discharge (DOD) on the cycle life, a battery degradation cost model is established to optimize the energy storage operation strategy. The following steps are included:

[0092] Step 3.1 Define the lithium battery cycle life indicators, which mainly include the lithium battery cycle life and the single-cycle battery degradation cost, where the lithium battery life T life Related to DOD; when the energy storage completes a single full cycle, its degradation cost is the ratio of the energy storage investment cost to the cycle life under the corresponding DOD. DOD =1-SOC, the single-cycle battery degradation cost can be defined as a function of SOC. Its mathematical expression is as follows:

[0093]

[0094] In the formula, δ0, δ1, δ2 are fitting coefficients; COST ess COST is the energy storage degradation cost; e The investment cost of energy storage.

[0095] Step 3.2 takes into account the actual capacity and DOD, so the battery degradation model is given the following two assumptions: 1) the degradation process is time-linear throughout the battery life; 2) the degradation cost of each charge and discharge cycle event at the same DOD level is the same at different SOC levels. Therefore, the degradation cost model of the battery can directly represent its actual capacity and life. Considering a discharge event starting at time t, the average power in a time interval Δt is P B (t), DOD (α DOD ) can be expressed as:

[0096]

[0097] Average degradation cost per unit of energy:

[0098]

[0099] Where: C B is the battery replacement cost, α DOD is the discharge depth of the battery within Δt.

[0100] When DOD is d BWhen the degradation cost of each charging and discharging event is considered to be the same, the degradation cost of the battery corresponding to the discharge event can be obtained by multiplying the average degradation cost by the energy output of the battery:

[0101]

[0102] After the cycling event, the actual capacity of the battery is proportionally reduced within the time t+Δt as follows:

[0103]

[0104] Where, T life (α DOD ) is the DOD in the time Δt, which is α DOD Battery life

[0105] In step 4, the five-tuple MDP reinforcement learning problem is modeled, which includes the following steps:

[0106] Step 4.1 Define the strategy to maximize the cumulative reward, given state s t Select an action α t is the strategy π(a t |s t ), the agent takes corresponding actions in the current state to form an MDP chain μ = (s0, α0, s1, α1) to describe the process of interaction between the agent and the environment in reinforcement learning. The goal is to find a strategy that maximizes the cumulative reward. Its mathematical expression is as follows:

[0107]

[0108] In the formula, E μ～π represents the expectation, T is the length of μ, The value range is

[0109] Step 4.2 defines the state set. In any period t, the state space consists of user load power, photovoltaic power generation power, energy storage SOC and time-of-use electricity price. The constraints are as follows:

[0110] s t = {P load,t ,P pv,t ,SOC t ,ρ t},s t ∈S

[0111] Where P load,t Indicates user load power, P pv,t Indicates photovoltaic load power, SOC t represents the state of charge of the energy storage system, ρ t Indicates time-of-use electricity price;

[0112] Step 4.3 defines the action set, and defines the charging and discharging power of the energy storage as the action space. The constraints are as follows:

[0113] a t = {P ess,t},a t ∈A

[0114] Where P ess,t Indicates the charging and discharging power of the energy storage system;

[0115] Step 4.4 defines the state transition probability. The state transition function simulates the uncertainty of the energy storage system environment. The agent can learn this uncertainty from historical data and interaction with the environment. t Execute action a t Transition to next state s t+1 The transition probability function is defined as follows:

[0116] p(S t+1 |S t ,A t )=pr{S t+1 =s t+1 |S t =s t ,A t =a t};

[0117] Step 4.5 defines the reward function. In the energy storage operation problem, the reward function represents the immediate return obtained from charging and discharging operations in a certain period of time. It helps guide the agent to learn effective strategies and choose actions that reduce costs. In order to minimize the operating cost, this goal is opposite to the reward mechanism of reinforcement learning. The function is as follows:

[0118]

[0119] In the formula, COST grid,t Indicates the user side electricity purchase cost, COST ess,t represents the energy storage degradation cost;

[0120] Step 4.6 defines the MDP model including power balance constraints. The energy storage system operation optimization problem is expressed as an MDP, which requires that the actions defined by the agent must strictly implement the action constraints to ensure the reliable operation of the system. During the operation of the energy storage on the user side, power balance must be maintained. The constraint is established as an equality constraint. In this way, the agent is searching for the optimal strategy in a reliable set containing constraints. C ={π:R C (π)=0}, the function is as follows:

[0121]

[0122] In the formula, R C (π) is the penalty function for violating constraint C.

[0123] Step 4.7 In order to strictly enforce the power balance constraint, the violation of the power balance constraint is added as a penalty term to the reward function. The function is as follows

[0124]

[0125] Where P grid,t Indicates the power purchased by the user from the grid, ΔP t is the unbalanced power in period t, β1 and β2 are weight coefficients, both are positive numbers and are two hyperparameters.

[0126] In step 5, the MILP-TD3 algorithm includes two processes: training and online execution. The training process estimates the neural network parameters of the approximate action value function, and the online execution process selects actions to be executed based on the obtained action value function to minimize the operating cost of the user-side energy storage. The following steps are included:

[0127] Step 5.1 During the training process, MILP-TD3 combines the ideas of reinforcement learning and MILP (mixed integer linear programming) to improve the performance of the agent in complex environments. The training process is divided into the following steps:

[0128] (1) Value-based reinforcement learning algorithm:

[0129] In status t Execute action α according to strategy π t The cumulative reward G t for:

[0130]

[0131] Action-Value Function To estimate G t Expectations:

[0132] Qπ(S t ,A t )=Eπ[G t S t =s t ,A t =a t ];

[0133] The optimal action-value function is obtained by solving the Bellman equation With recursion:

[0134]

[0135] Solving the above equation using the time difference method yields:

[0136]

[0137] In the formula, for is an approximate function of , τ is the learning rate, and its value range is 0≤τ≤1. Through iterative update, when Converge to After the optimal value, the state s is obtained t The best action a under t ~π * (s t ), the optimal strategy is expressed as:

[0138]

[0139] Optimal Action-Value Function It can be approximated by a recursive neural network (RNN) with a parameter of θ, that is, The training process can be regarded as a problem of optimizing the parameter θ. In the continuous state and action space, the exhaustive method is unreliable. ω To solve this problem, the AC (actor-critic, AC) algorithm was born.

[0140] (2) AC algorithm training process:

[0141] The AC algorithm consists of an actor network π ω and critic network Q θ , first randomly initialize π ω and Q θ The parameters of the two networks; then according to π ω In state t s Downsample and perform an action a t , by interacting with the environment, the environment is transferred to the next state t s+1 , and receive a reward t As feedback for the execution of actions; after continuous iteration, the formed experience (S t ,Q t ,Y t ,S t+1 ) is stored in the experience replay pool; finally, a small batch of samples is selected to update the network parameters θ and ω. The TD3 algorithm is a typical AC algorithm that uses a dual critic network to estimate the action value function and takes the smaller value of the two as the estimated value to reduce the overestimation bias.

[0142] (3) TD3 algorithm network parameter update:

[0143] The actor network parameter ω update expression is:

[0144]

[0145] In the formula, B represents, express;

[0146] The update expression of actor network parameter θ is:

[0147]

[0148] Step 5.2 After training converges, obtain the fixed parameter Q θ The network controls the actions of the energy storage system, which is the online execution process. In this process, the actions obtained based on the action value function defined by RNN cannot strictly enforce the action constraints. For this reason, RNN is converted into MILP formula. The online execution process is divided into the following steps:

[0149] (1) MILP formula of RNN:

[0150] Assume Q θ The RNN structure of (s,a) consists of K+1 layers of neural networks, numbered from 0 to K+1. The 0th layer is the input of the RNN, and the Kth layer is the output of the RNN. Each layer k∈{0,1,...,K} has n k neurons, and define the jth neuron in the kth layer as U(j,k). Define x k is the output vector of the kth layer, is the jth j∈{0,1,...,n k}The output vector of the neuron, where is the jth input of the RNN. For the kth (k ≥ 1) layer, the output vector expression of U(j, k) is defined as:

[0151] x k =σ(w k-1 x k-1 +b k-1 )

[0152] Where σ(·) is the activation function, which is the Tanh function here; w k-1 is the given weight matrix; b k-1 is the bias vector, then the parameters of the RNN are θ = {w, b}.

[0153] Use a binary variable for each neuron U(j,k) Activation, the MILP formula of RNN can be expressed as:

[0154]

[0155] Constraints:

[0156]

[0157] In the formula, is a constant parameter. The above formula constrains the output of the activation function and the upper and lower limits of the variables x and s. When k = 0, the upper and lower limits are the same as the input limits of the RNN; when k ≥ 1, the upper and lower limits can be defined based on the parameter θ.

[0158] (2) Online execution process including constraints:

[0159] In any state t , the optimal action a t Q can be formed by solving the formula θ The MILP model of (s,Q) is solved. During the execution process, the decision variable is the action a t , adding the charging and discharging power constraint and energy storage SOC constraint, the running optimization model is updated as follows:

[0160]

[0161] The present invention proposes a user-side energy storage system optimization operation technology that takes into account the extreme cold and no light scenario. The first innovation is that the extreme cold and no light scenario and battery degradation cost are considered in the real-time operation optimization of energy storage: this new extreme weather scenario generation method can accurately reflect the impact of extreme weather on the user side and provide reliable data support for user-side energy storage scheduling; the defined battery degradation cost formula only needs to track the charge state of the battery under the discharge state, which improves the optimization efficiency; the second innovation is that compared with the standard TD3 algorithm, which cannot meet the power balance constraint in unknown data sets, the proposed MILP-TD3 algorithm converts the RNN model into a MILP formula, which can further consider all action space constraints, and the charging and discharging strategy defined by it can keep the unbalanced power always at 0, ensuring the reliability of the obtained operation strategy in actual operation. These innovations provide new ideas and technical paths for the stability and reliability of the optimal operation strategy of the battery energy storage system under the comprehensive generation of the extreme cold and no light scenario.

[0162] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0164] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. Considering the operation method of the user-side energy storage system in the extremely cold and dark scene, it is characterized by: The following steps are involved: Under the scenario set of wind and solar power output under extreme weather conditions, according to the impact of discharge depth on the cycle life of user-side energy storage, a five-tuple MDP reinforcement learning model consisting of state set, action set, state transition probability, reward set and reward discount factor is built; Based on the five-tuple MDP reinforcement learning model, the MILP algorithm is used to minimize the operating cost of the user-side energy storage, and the system operation of the user-side energy storage is optimized and controlled.

2. The method for operating a user-side energy storage system in an extremely cold and dark environment according to claim 1, characterized in that: In the process of obtaining the set of wind and solar power output scenarios under extreme weather conditions, the historical wind and solar power output and meteorological data are preprocessed, including interpolation, dimensionality reduction, standardization and smoothing. Based on the pre-processed historical wind and solar output and meteorological data, a PV low output threshold is defined to determine whether a certain day is a PV low output day; Based on the photovoltaic low output day, the conditions of the extremely cold and dark day are defined, the extremely cold and dark days in the historical data are identified, and the 24-hour photovoltaic output curve within the extremely cold and dark days is included in the extremely cold and dark scene set; For the curves in the extremely cold and lightless scene set, k-means clustering is used to obtain the set of wind and solar power output scenes under extremely cold and lightless extreme weather conditions.

3. The method for operating a user-side energy storage system in an extremely cold and dark environment according to claim 2, characterized in that: In the process of defining the PV low output threshold, according to the frequency distribution of the average daily PV output, cubic spline interpolation is used to define the PV low output threshold by obtaining the empirical probability distribution and cumulative probability distribution of the average daily PV output.

4. The method for operating a user-side energy storage system in an extremely cold and dark environment according to claim 3, characterized in that: In the process of analyzing the influence of the discharge depth on the cycle life of the user-side energy storage, the influence of the discharge depth on the cycle life of the user-side energy storage is analyzed according to the energy storage system operation model of the user-side energy storage, wherein: Considering the degradation cost of the energy storage battery and taking minimizing the user-side operating cost as the goal, an objective function is constructed, and the energy storage system operation model is constructed by defining constraints, including power balance constraints, charge and discharge power constraints, and SOC and its continuity constraints.

5. The method for operating the user-side energy storage system in the extremely cold and dark scene according to claim 4 is characterized in that: In the process of obtaining the influence of discharge depth on the cycle life of user-side energy storage, based on the energy storage system operation model, the lithium battery cycle life index is defined according to the lithium battery cycle life and the single-cycle battery degradation cost, and based on the actual capacity and discharge depth, the influence of discharge depth on the cycle life of user-side energy storage is obtained.

6. The method for operating the user-side energy storage system in the extremely cold and dark scene according to claim 5 is characterized by: In the process of constructing the five-tuple MDP reinforcement learning model, the user load power, photovoltaic power generation power, energy storage SOC and time-of-use electricity price are taken as the state set; The charging and discharging power of energy storage is used as the action set; The uncertainty of the energy storage system environment is simulated through the state transfer function to obtain the state transfer probability; Through the reward function, the immediate return obtained from charging and discharging operations of energy storage in a certain period of time is obtained; The power balance constraint is used as a constraint condition of the five-tuple MDP reinforcement learning model, and the violation of the power balance constraint condition is added as a penalty term into the reward function as a reward discount factor.

7. The method for operating a user-side energy storage system in an extremely cold and dark environment according to claim 6, characterized in that: In the process of minimizing the operating cost of the user-side energy storage by using the MILP algorithm, the MILP-TD3 algorithm is used to minimize the operating cost of the user-side energy storage, wherein the MILP-TD3 algorithm includes two processes: training and online execution. The training process is used to estimate the neural network parameters of the approximate action value function, and the online execution process is used to select action execution based on the obtained action value function to minimize the operating cost of the user-side energy storage.

8. A user-side energy storage system operation system considering extreme cold and no light scene, used to implement the user-side energy storage system operation method considering extreme cold and no light scene as described in any one of claims 1-7, characterized in that: include: Database module, used to store system data; A database logic control module is used to control the data interaction and algorithm logic implementation between the database module and other system modules; Data collection and processing module, used to collect historical wind and solar power output and meteorological data, and generate a set of wind and solar power output scenarios under extreme weather conditions; The model building module is used to build a five-tuple MDP reinforcement learning model consisting of a state set, an action set, a state transition probability, a reward set, and a reward discount factor based on the impact of discharge depth on the cycle life of user-side energy storage under a set of wind and solar power output scenarios under extreme weather conditions; The optimization control module is used to minimize the operation cost of the user-side energy storage based on the five-tuple MDP reinforcement learning model and to optimize the system operation of the user-side energy storage by using the MILP algorithm.

Images