User side energy storage power station operation decision-making method based on hierarchical reinforcement learning

Abstract

The invention discloses a user side energy storage power station operation decision-making method based on hierarchical reinforcement learning. The method comprises the following steps: constructing a prediction model and a hierarchical reinforcement learning framework; performing day-ahead electricity price and load prediction by using the prediction model, designing a Markov decision process, defining an upper reinforcement learning model, designing a mapping method, and obtaining day-ahead declaration amount; performing real-time electricity price and load prediction by using the prediction model, designing a Markov decision process, defining a middle-level reinforcement learning model, designing a mapping method, and obtaining a charging and discharging plan, an output upper limit and an SOC target; combining the frequency, the day-ahead declaration amount, the output upper limit and the SOC target, and utilizing a lower-layer optimization model to decide charging and discharging output; the real electricity price and the load data are utilized to calculate and update operation incomes and rewards; dividing the electricity price data and the load data into a prediction training set, a reinforcement learning training set and a test set, training a prediction model, training a layered reinforcement learning framework, and testing and verifying an operation effect by using the test set.

Description

Technical Field

[0001] This invention relates to the field of power energy storage technology, and more specifically, to a user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning. Background Technology

[0002] In the construction of modern power systems, user-side energy storage power stations play an indispensable role. Renewable energy sources have low inertia and insufficient stability, which can easily affect the stability of power system operation. The charging and discharging process of energy storage power stations can smooth their output and improve power system stability. Simultaneously, user-side energy storage power stations generate considerable revenue through basic operating modes such as peak-valley arbitrage, profiting from the price differences in time-of-use electricity prices, including industrial and commercial electricity prices, in the retail market. Furthermore, user-side energy storage power stations can broaden their operating modes and increase revenue through participation in the spot market, frequency regulation ancillary services, and other means.

[0003] Currently, user-side energy storage power stations face numerous challenges in improving profitability: simply reducing operating costs is no longer sufficient to further increase revenue. Existing methods primarily employ mixed-integer linear programming to construct multi-market participation models and rely on model predictive control for operational decisions, but this approach is significantly affected by prediction accuracy. Furthermore, the difficulty in accurately predicting future electricity prices and load information limits its practical application effectiveness.

[0004] Reinforcement learning technology has been applied to energy storage optimization research. Through continuous interaction with the environment, it trains to derive sound decision-making strategies, offering advantages in handling uncertainty and reducing computational complexity. However, existing research lacks in-depth exploration of user-side energy storage power stations' participation in multi-timescale spot markets, peak-valley arbitrage, and frequency regulation ancillary services. Traditional reinforcement learning methods struggle to achieve multi-timescale decision-making and cannot formulate reasonable operational strategies to improve the profitability of user-side energy storage power stations. Summary of the Invention

[0005] The purpose of this invention is to overcome the defects and shortcomings of the prior art and provide a user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning. By constructing an accurate prediction model and a hierarchical reinforcement learning framework, the invention enables energy storage to achieve coordinated and optimized operation in participating in the spot market, peak-valley arbitrage, and frequency regulation ancillary services.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning includes the following steps:

[0008] S1: Construct a prediction model and a hierarchical reinforcement learning framework. The prediction model includes an electricity price prediction model and a load prediction model. The hierarchical reinforcement learning framework includes an upper-level reinforcement learning model, a middle-level reinforcement learning model, and a lower-level optimization model.

[0009] S2: Use the forecasting model to forecast day-ahead electricity prices and loads, design a Markov decision process, define an upper-level reinforcement learning model, design a mapping method, and obtain the day-ahead reporting volume;

[0010] S3: Use the prediction model to predict real-time electricity prices and loads, design a Markov decision process, define a mid-level reinforcement learning model, design a mapping method, obtain a charging and discharging plan, and output the upper limit of output power and the SOC target.

[0011] S4: Combining historical frequency statistics, day-ahead reporting volume, output ceiling and SOC target, the lower-level optimization model is used to decide the charging and discharging output participating in the operation of the power system, which is used for the actual operation of user-side energy storage power stations.

[0012] S5: Using real electricity price and load data, calculate the revenue from participating in the operation of the power system, calculate the rewards for the upper-layer reinforcement learning model and the middle-layer reinforcement learning model, and update the upper-layer reinforcement learning model and the middle-layer reinforcement learning model.

[0013] S6: Divide the electricity price data and load data into a prediction training set, a reinforcement learning training set, and a test set. Use the prediction training set to train the prediction model, load the trained prediction model, use the reinforcement learning training set to train the hierarchical reinforcement learning framework, and use the test set to test and verify the running effect.

[0014] Furthermore, in step S2, a Markov decision process is designed, defining the state space, action space, and reward function of the upper-level reinforcement learning model, as well as their normalization methods, specifically:

[0015] State: The state space s consists of day-ahead electricity prices and load forecasts. upper , is represented as:

[0016] s upper ={λ pda ,λ ci ,P load_da}

[0017] In the formula, λ pda Let λ be the vector of day-ahead electricity price forecasts; ci P represents the industrial and commercial electricity price vector; load_da This is a vector of load forecast values ​​used for day-ahead market reporting;

[0018] Action: Action space a upper It contains 24 actions with values ​​in the range [0,1], represented as follows:

[0019] a upper ={a u (0),a u (1),…a u(T-1)|a u (t)∈[0,1]}

[0020] In the formula, a u (t) represents the action of the agent in time period t; a u (0) represents the action in the 0th time period of the agent's decision-making; a u (1) is the action in the first time period of the agent's decision-making; a u (T-1) represents the action taken by the agent in the (T-1)th time period; T is the number of time periods in a day;

[0021] Reward: The reward value R of the upper-level reinforcement learning model is obtained by normalizing the real spot market and peak-valley arbitrage profits. upper , is represented as:

[0022]

[0023] In the formula, Z e Z represents the daily arbitrage profit obtained in the spot market and peak-valley arbitrage; d For demand difference electricity charges; Z actual_max This represents the theoretically highest arbitrage profit for the day.

[0024] Furthermore, a mapping method is designed to obtain the daily reporting volume, specifically:

[0025] Design a mapping method to map actions to derive day-ahead market declarations, including:

[0026] The safe charging and discharging range is expressed as follows:

[0027] P socmax (t)=(1-S OCda (t))E / η c

[0028] P socmin (t)=-S OCda (t)Eη d

[0029] P abmin (t)=P' gmin α-P load_da (t)

[0030] In the formula, P socmax (t) and P socmin (t) represents the upper and lower limits of charge / discharge output that satisfy the SOC safety constraints during time period t; P abmin (t) is the lower limit of charge / discharge output to meet the backflow prevention safety constraint; S OCda (t) represents the SOC value at time t during the day-ahead market declaration decision-making process; E represents the rated capacity of the energy storage battery; η c and ηd These represent charging efficiency and discharging efficiency, respectively; P' gmin P represents the minimum operating power of the power grid; α is the coefficient for the minimum power; load_da (t) represents the load forecast value for the t-th time period used in the day-ahead market submission;

[0031] By integrating the charge and discharge constraints, we obtain the upper limit of safe charge and discharge output P. uupper (t) and the lower limit of safe charge and discharge output P ulower (t), represented as:

[0032] P uupper (t)=min(P socmax (t),P max )

[0033] P ulower (t)=max(P socmin (t),-P max ,P abmin (t))

[0034] In the formula, P max This represents the maximum output power of the energy storage system.

[0035] Day-ahead charge / discharge plan: Day-ahead charge / discharge plan P for time period t da (t) is derived from the mapping between the current action and the upper and lower limits of safe charging and discharging output, and is expressed as:

[0036] P da (t)=P ulower (t)+a u (t)(P uupper (t)-P ulower (t))

[0037] Day-ahead market submissions: Day-ahead market submissions P for period t dec (t) is represented as:

[0038] P dec (t)=P load_da (t)+P da (t)

[0039] The 24-hour charge and discharge plan is calculated on a rolling basis to obtain the day-ahead market declaration volume vector P. dec , is represented as:

[0040] P dec ={P dec (0),P dec (1),…,P dec (T-1)}

[0041] In the formula, P dec(0) represents the market order volume before the 0th time period; P dec (1) Market order volume for the first period; P dec (T-1) represents the market declaration volume for the T-1 period prior to the date.

[0042] Furthermore, in step S3, a Markov decision process is designed, defining the state space, action space, and reward function of the mid-level reinforcement learning model, as well as their normalization methods, specifically as follows:

[0043] State: The state space s consists of the day-ahead reported volume, actual day-ahead electricity price, industrial and commercial electricity price, forecast real-time electricity price, forecast load, previous period's SOC, and the current arbitrage period. middle , is represented as:

[0044]

[0045] In the formula, λ da Let λ be the vector of day-ahead electricity prices; prt S is a vector of real-time electricity price forecasts; OC (t-1) represents the SOC value at the end of the previous time period; t represents the current time period; P load (0) represents the actual load in time period 0; P load (t) represents the actual load in time period t; P load_rt (t+1) represents the predicted load for the (t+1)th time period; P load_rt (T-1) represents the predicted load for the T-1 time period;

[0046] Action: Action space a middle It contains two actions with values ​​[0,1], represented as:

[0047] a middle ={a m0 ,a m1 |a m0 ,a m1 ∈[0,1]}

[0048] In the formula, a m0 This is the 0th action in the agent's decision-making process; a m1 The first action for the agent to make decisions;

[0049] Reward: The reward value R of the mid-level reinforcement learning model is obtained by normalizing the current real-time market and peak-valley arbitrage profits. middle (t), represented as:

[0050]

[0051] In the formula, F ci (t) represents the industrial and commercial revenue in the current time period t; F da(t) represents the electricity cost t days prior to the current time period; F rt (t) represents the real-time electricity cost for the current time period t; F dev (t) represents the deviation assessment cost for the current time period t.

[0052] Furthermore, a mapping method is designed to obtain the charge / discharge plan, outputting the upper limit of power output and the SOC target, specifically as follows:

[0053] The design uses a mapping method to map actions to derive charging and discharging plans for the spot market and peak-valley arbitrage, and outputs the upper limit of output and SOC target for each time period, including:

[0054] Deviation assessment constraints are expressed as follows:

[0055]

[0056] In the formula, P devmax (t) and P devmin (t) represents the upper and lower limits of charge / discharge output to avoid deviation assessment costs, respectively; σ is the allowable arbitrage ratio in the spot market;

[0057] The charging / discharging output range constraint is expressed as:

[0058] P mupper (t)=min[P socmax (t),P max ,P devmax (t)]

[0059] P mlower (t)=max[P socmin (t),-P max ,P abmin (t),P devmin (t)]

[0060] In the formula, P mupper (t) and P mlower (t) represents the upper limit of the real-time charge / discharge output range and the lower limit of the real-time charge / discharge output range, respectively;

[0061] Charging and discharging power P of mid-level reinforcement learning decision making rt (t) by a m0 The mapping yields the following representation:

[0062] P rt (t)=P mlower (t)+a m0 (P mupper (t)-P mlower (t))

[0063] Charging and discharging power is further subdivided into charging power P c_rt(t) and discharge power P d_rt (t), represented as:

[0064]

[0065] The target SOC value S for the current time period is obtained through the charging and discharging process. OCtar (t), represented as:

[0066]

[0067] The upper limit of charging and discharging output P of the lower-level optimization model in the current period lupper (t) by a m1 The mapping yields the following representation:

[0068]

[0069] In the formula, P llower (t) represents the lower bound of the charging and discharging output mapping of the lower-level optimization model in the current time period.

[0070] Furthermore, in step S4, the charging and discharging output participating in the operation of the power system is determined using the lower-level optimization model, including:

[0071] Objective function: The objective function is established to maximize the profit in the current time period, and is expressed as:

[0072] maxM sl (t)+M fl (t)

[0073] In the formula, M sl (t) and M fl (t) represents the spot market and peak-valley arbitrage profits, respectively, and their expressions are as follows:

[0074] M sl (t)=F ci (t)-F da (t)-F rt_pre (t)-F dev_pre (t)

[0075] M fl (t)=F f (t)

[0076] In the formula, F rt_pre (t) and F dev_pre (t) represents the real-time market electricity cost and deviation assessment cost calculated based on the predicted real-time electricity price, respectively; F f (t) represents the compensation revenue for primary frequency modulation ancillary services.

[0077] Furthermore, the charging and discharging outputs that participate in power system operation through lower-level optimization model decisions also include:

[0078] Energy storage operation constraints are expressed as:

[0079] P min ≤P c,t (i)≤P max

[0080] P min ≤P d,t (i)≤P max

[0081] P c,t (i)P d,t (i)=0

[0082] In the formula, P c,t (i) and P d,t (i) represent the charging power and discharging power of the i-th 15-minute period in the current time period, respectively, P min This represents the minimum energy storage output.

[0083] The relationship between the SOC value change in the i-th 15-minute period of the current time is as follows:

[0084]

[0085] In the formula, E fr,t (i) represents the change in power output caused by frequency modulation during the i-th 15-minute period in the current time period; S OC,t (i) represents the SOC value of the i-th 15-minute period in the current time period; the SOC value satisfies the following constraints:

[0086]

[0087] The corresponding incoming power P of the power grid g,t (i) is:

[0088] P g,t (i)=P load (t)+P c,t (i)-P d,t (i)-P fm,t (i)

[0089] In the formula, P fm,t (i) represents the average frequency modulation output of the i-th 15-minute period in the current time period; P load (t) represents the actual load value for the current time period; P g,t (i) Satisfies:

[0090] P' gmin α≤P g,t (i)≤P gd

[0091] In the formula, P'gmin P represents the minimum operating power of the power grid; α is the coefficient for the minimum power; gd This represents the maximum power output of the incoming power grid line.

[0092] Constraints on Spot Market and Peak-Valley Arbitrage Profits: The expressions for the components of spot market and peak-valley arbitrage profits are as follows:

[0093] F da (t)=λ da (t)P dec (t)

[0094] F ci (t)=λ ci (t)P load (t)

[0095]

[0096] In the formula, λ da (t), λ ci (t), λ prt (t) represents the actual day-ahead electricity price, the actual industrial and commercial electricity price, and the predicted real-time electricity price for the current period, respectively;

[0097] Primary frequency modulation (FM) auxiliary service constraint: Assume there are N frequency adjustment periods within a given time period where the frequency deviates from the FM dead zone. Establish the FM function and take the average frequency deviation Δf within the frequency adjustment period. ave (n i If ) is used as the independent variable of the frequency modulation function, then the nth digit in the current time period i The primary frequency modulation power for each frequency adjustment period is:

[0098]

[0099] In the formula, P f,t (s) represents the frequency modulation power at the s-th second of the current time period, satisfying s∈[0,3599]; P fb,t (i) represents the upper limit of frequency modulation output during the i-th frequency adjustment period in the current time period; K f,t (i) represents the frequency modulation coefficient for the i-th frequency adjustment period in the current time period; f1 and f2 are the boundaries of the frequency modulation interval, satisfying 0 <f1<f2;

[0100] The primary frequency modulation power is further subdivided into frequency modulation charging power P. fc,t (s) and frequency-modulated discharge power P fd,t (s), denoted as:

[0101]

[0102]

[0103] Therefore, the change in energy storage frequency regulation power E is calculated. fr,t (i) is represented as:

[0104]

[0105] To control the demand difference in electricity costs caused by the charging and discharging power of energy storage, the energy storage output must not exceed the upper limit of the charging and discharging output determined by the mid-level reinforcement learning decision, expressed as:

[0106] P lupper (t)≥P c,t (i)-P d,t (i)+P fm,t (i)

[0107]

[0108] Theoretical frequency modulation action integral power p' fr (t) is:

[0109]

[0110] In the formula, S is the number of seconds in a day;

[0111] Then calculate the compensation revenue F for primary frequency regulation ancillary services in the current period. f (t), represented as:

[0112]

[0113] In the formula, μ is the ratio of the actual frequency modulation action integral energy to the theoretical frequency modulation action integral energy, and R is the compensation standard for the excess integral energy of primary frequency modulation.

[0114] The total revenue Z for one frequency modulation ancillary service on that day f for:

[0115]

[0116] Furthermore, in step S5, using real electricity prices and load data, the benefits of participating in the operation of the power system are calculated, including:

[0117] Real spot market and peak-valley arbitrage profits: Using the arbitrage charging and discharging power and frequency modulation charging and discharging power recorded by the lower-level optimization model, the real spot market and peak-valley arbitrage profits for each time period are calculated and expressed as follows:

[0118] F da (t)=λ da (t)P dec (t)

[0119] F ci (t)=λ ci (t)P load(t)

[0120]

[0121] In the formula, λ rt (t) represents the actual real-time market electricity price for the t-th time period;

[0122] The total profit Z from the spot market and peak-valley arbitrage on that day et for:

[0123]

[0124] The total revenue Z after the revenue sharing is calculated based on the revenue sharing agreement in the energy performance contracting agreement. e , is represented as:

[0125]

[0126] In the formula, k s The profit-sharing ratio stipulated in the contract;

[0127] Based on the recorded maximum power of the incoming power grid and the actual load data, calculate the daily demand difference electricity charge Z. d , is represented as:

[0128]

[0129] In the formula, P loadmax Where β is the maximum load power, and T is the demand electricity price. d This represents the number of days in the current month.

[0130] Furthermore, calculating the benefits of participating in the operation of the power system using real electricity prices and load data also includes:

[0131] Theoretical Peak Spot Market and Peak-Valley Arbitrage Profits: Calculate the theoretical peak spot market and peak-valley arbitrage profits for the day using disclosed real electricity prices and recorded real load data.

[0132] The objective function is expressed as:

[0133] maxZ actual_max (P c ,P d ,P g ,P dec ,λ da ,λ rt ,λ ci )

[0134] The constraints are expressed as follows:

[0135] Z actual_max =Z d +Z e

[0136] P min ≤P c (t)≤P max

[0137] P min ≤P d (t)≤P max

[0138] P c (t)P d (t)=0

[0139]

[0140] P g (t)=P load (t)+P c (t)-P d (t)

[0141] P' gmin α≤P g (t)≤P gd

[0142] Z d ≤(P loadmax -P gd )β / T d

[0143] Z d ≤0

[0144] F da (t)=λ da (t)P dec (t)

[0145] F rt (t)=λ rt (t)[P g (t)-P dec (t)]

[0146]

[0147] F ci (t)=λ ci (t)P load (t)

[0148]

[0149] In the formula, P c and P d For charging power vector and discharging power vector; P g λ is the power vector of the incoming power grid. rt This represents the real-time electricity price vector.

[0150] Furthermore, the rewards for the upper-layer and middle-layer reinforcement learning models are calculated as follows:

[0151] Using the calculated Z actual_max Z e Z d F ci (t), F da F rt (t) and F dev (t) calculates the rewards for the upper-layer reinforcement learning model and the middle-layer reinforcement learning model, and passes them to the upper-layer reinforcement learning model and the middle-layer reinforcement learning model for updating, adjusting the corresponding neural network parameters.

[0152] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0153] 1. The hierarchical reinforcement learning strategy proposed in this invention can integrate the spot market, peak-valley arbitrage, and primary frequency regulation ancillary service market to make operational decisions, thereby deriving reasonable day-ahead market reporting and real-time charging and discharging strategies and obtaining considerable operational benefits.

[0154] 2. The hierarchical reinforcement learning strategy proposed in this invention can effectively address the impact of prediction errors on actual operating performance, maintain a stable level of returns in a complex and volatile spot market environment, provide reliable strategy support for user-side energy storage power stations to participate in the spot market and peak-valley arbitrage as well as primary frequency regulation ancillary services, enhance the adaptability of energy storage in market fluctuations, and make returns more robust.

[0155] 3. The hierarchical reinforcement learning strategy proposed in this invention can complete the required decisions in a short time and quickly provide operational strategies for user-side energy storage to participate in the spot market, peak-valley arbitrage, and primary frequency regulation ancillary services. Attached Figure Description

[0156] Figure 1 This is a framework diagram of a user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning.

[0157] Figure 2 This is a framework diagram of a user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning.

[0158] Figure 3 The training process reward curve is shown in the example.

[0159] Figure 4 The test process reward and benefit curves are shown in the example.

[0160] Figure 5 The typical test process for this example includes the pre-declaration and charge / discharge plan. Detailed Implementation

[0161] The following description, in conjunction with the accompanying drawings and specific embodiments, further illustrates the user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning of the present invention.

[0162] Please see Figure 1 and Figure 2 This invention discloses a user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning, comprising the following steps:

[0163] S1: Construct a prediction model and a hierarchical reinforcement learning framework. The prediction model includes an electricity price prediction model and a load prediction model. The hierarchical reinforcement learning framework includes an upper-level reinforcement learning model, a middle-level reinforcement learning model, and a lower-level optimization model.

[0164] S2: Use the forecasting model to forecast day-ahead electricity prices and loads, design a Markov decision process, define an upper-level reinforcement learning model, design a mapping method, and obtain the day-ahead reporting volume;

[0165] S3: Use the prediction model to predict real-time electricity prices and loads, design a Markov decision process, define a mid-level reinforcement learning model, design a mapping method, obtain a charging and discharging plan, and output the upper limit of power output and the SOC (State of Charge) target.

[0166] S4: Combining historical frequency statistics, day-ahead reporting volume, output ceiling and SOC target, the lower-level optimization model is used to decide the charging and discharging output participating in the operation of the power system, which is used for the actual operation of user-side energy storage power stations.

[0167] S5: Using real electricity price and load data, calculate the revenue from participating in the operation of the power system, calculate the rewards for the upper-layer reinforcement learning model and the middle-layer reinforcement learning model, and update the upper-layer reinforcement learning model and the middle-layer reinforcement learning model.

[0168] S6: Divide the electricity price data and load data into a prediction training set, a reinforcement learning training set, and a test set. Use the prediction training set to train the prediction model, load the trained prediction model, use the reinforcement learning training set to train the hierarchical reinforcement learning framework, and use the test set to test and verify the running effect.

[0169] Specifically, a day-ahead electricity price forecasting model, a real-time electricity price forecasting model, and a load forecasting model are constructed using a long short-term memory neural network, while a hierarchical reinforcement learning framework is constructed using a deep deterministic policy gradient algorithm.

[0170] A day-ahead electricity price forecasting model and a real-time electricity price forecasting model were constructed using a long short-term memory neural network. Relevant parameters were set, and the day-ahead electricity price and real-time electricity price of 24 time periods each day over the past 7 days were used as inputs. The predicted day-ahead electricity price and real-time electricity price of 24 time periods were output as the electricity price forecast value for the operating day.

[0171] A load forecasting model is constructed using a long short-term memory neural network, and relevant parameters are set. Assuming that the market reporting point is 10:00 AM and there are 38 time periods before the last time period of the operating day, load data from 24 points in the past 7 days is used as input to predict the load forecast values ​​for the next 38 time periods. When reporting the load on the day before, the last 24 values ​​are used as the load forecast values ​​for the operating day. During each time period of the operating day, the latest load data is used for rolling forecasting, and the first 24 values ​​are taken as the future load forecast values.

[0172] The hierarchical reinforcement learning framework consists of three parts: an upper-layer reinforcement learning model, a middle-layer reinforcement learning model, and a lower-layer optimization model. The upper-layer and middle-layer reinforcement learning models are constructed using the deep deterministic policy gradient algorithm. The buffer, Actor network, Actor target network, Critic network, and Critic target network are constructed and relevant parameters are set. A delayed reward method is also designed.

[0173] The upper-level reinforcement learning model participates in day-ahead market reporting decisions, outputting the day-ahead reporting volume and passing it to the middle-level reinforcement learning model. During real-time operation, the middle-level reinforcement learning model makes decisions based on the real-time market and peak-valley arbitrage, outputting the current period's SOC target value and maximum output ceiling value, and passing them to the lower-level optimization model. Upon receiving instructions from the middle-level reinforcement learning model, the lower-level optimization model immediately makes decisions, deriving energy storage charging and discharging strategies for participating in the spot market, peak-valley arbitrage, and frequency regulation ancillary services, guiding actual charging and discharging actions.

[0174] The upper-level reinforcement learning model is used for day-ahead market submissions for energy storage. It uses a prediction model to forecast day-ahead electricity prices and loads, and designs a Markov decision process. It defines the state space, action space, and reward function of the upper-level reinforcement learning model and their normalization methods. At the same time, it designs a mapping method to map the actions to obtain the day-ahead market submission volume.

[0175] The Markov decision process is as follows:

[0176] State: The state space s consists of day-ahead electricity prices and load forecasts. upper :

[0177] s upper ={λ pda ,λ ci ,P load_da}

[0178] In the formula, λ pda Let λ be the vector of day-ahead electricity price forecasts. ci Let P be the industrial and commercial electricity price vector. load_da This is a vector of load forecast values ​​used for day-ahead market reporting.

[0179] Action: Action space a upper It contains 24 actions with values ​​in the range [0, 1]:

[0180] a upper ={a u (0),a u (1),…a u (T-1)|a u (t)∈[0,1]}

[0181] In the formula, a u (t) represents the action of the agent in time period t, where a u (0) represents the action in the 0th time period of the agent's decision-making; a u (1) is the action in the first time period of the agent's decision-making; a u (T-1) represents the action taken by the agent in the T-1th time period; T is the number of time periods in a day.

[0182] Reward: The reward value R of the upper-level reinforcement learning model is obtained by normalizing the real spot market and peak-valley arbitrage profits. upper :

[0183]

[0184] In the formula, Z e Z is the daily arbitrage profit obtained in the spot market and peak-valley arbitrage. d For demand difference electricity charges, Z actual_max This represents the theoretically highest arbitrage profit for the day.

[0185] The mapping yields the current day's reported volume, specifically:

[0186] Safe charging and discharging range:

[0187] P socmax (t)=(1-S OCda (t))E / η c

[0188] P socmin (t)=-S OCda (t)Eη d

[0189] P abmin (t)=P' gmin α-P load_da (t)

[0190] In the formula, P socmax (t) and P socmin (t) represents the upper and lower limits of the charging and discharging output that meet the SOC safety constraints in time period t, respectively. abmin (t) To meet the lower limit of charge and discharge output for anti-reverse current safety constraints, SOCda (t) represents the SOC value at time t during the day-ahead market declaration decision-making process, E is the rated capacity of the energy storage battery, and η c and η d P' represents the charging and discharging efficiency, respectively. gmin P represents the minimum operating power of the power grid, where α is the coefficient for the minimum power. load_da (t) represents the load forecast value for the t-th time period used in the day-ahead market declaration.

[0191] By integrating the charge and discharge constraints, we obtain the upper limit of safe charge and discharge output P. uupper (t) and lower limit P ulower (t):

[0192] P uupper (t)=min(P socmax (t),P max )

[0193] P ulower (t)=max(P socmin (t),-P max ,P abmin (t))

[0194] In the formula, P max This represents the maximum output of the energy storage.

[0195] Day-ahead charge / discharge plan: Day-ahead charge / discharge plan P for time period t da (t) is derived from the mapping between the current action and the upper and lower limits of safe charging and discharging output:

[0196] P da (t)=P ulower (t)+a u (t)(P uupper (t)-P ulower (t))

[0197] Day-ahead market submissions: Day-ahead market submissions P for period t dec (t) is:

[0198] P dec (t)=P load_da (t)+P da (t)

[0199] The 24-hour charge and discharge plan is calculated on a rolling basis to obtain the day-ahead market declaration volume vector P. dec :

[0200] P dec ={P dec (0),P dec (1),...,P dec (T-1)}

[0201] In the formula, P dec (0) represents the market order volume before the 0th time period; P dec (1) Market order volume for the first period; P dec (T-1) represents the market order volume for the day before the T-1 time period.

[0202] The mid-level reinforcement learning model targets the real-time market and peak-valley arbitrage of energy storage participation in operation. It uses a prediction model to forecast real-time electricity prices and loads, and designs a Markov decision process. It defines the state space, action space, and reward function of the mid-level reinforcement learning model and their normalization methods. At the same time, it designs a mapping method to map the actions to derive charging and discharging plans for the spot market and peak-valley arbitrage, and outputs the upper limit of output and SOC target for each time period to guide the optimization of the lower-level model.

[0203] During the operating day, the model makes decisions for the real-time spot market according to the arbitrage cycle. User-side energy storage power stations monitor load power in real time and make new load forecasts based on the latest monitored load data for each time period, while the predicted electricity price remains unchanged. The mid-level reinforcement learning model determines the SOC target value and the charging / discharging output ceiling for the lower-level optimization model on a cycle-by-cycle basis. The SOC target value provides an overall scheduling framework for the charging / discharging power of the user-side energy storage power station, while the output ceiling controls maximum electricity demand.

[0204] The Markov decision process is as follows:

[0205] State: The state space s consists of the day-ahead reported volume, actual day-ahead electricity price, industrial and commercial electricity price, forecast real-time electricity price, forecast load, previous period's SOC, and the current arbitrage period. middle :

[0206]

[0207] In the formula, λ da Let λ be the vector of day-ahead electricity prices; prt S is a vector of real-time electricity price forecasts; OC (t-1) represents the SOC value at the end of the previous time period; t represents the current time period; P load (0) represents the actual load in time period 0; P load (t) represents the actual load in time period t; P load_rt (t+1) represents the predicted load for the (t+1)th time period; P load_rt (T-1) represents the predicted load for the T-1 time period.

[0208] Action: Action space a middle It includes two actions with values ​​in the range [0, 1]:

[0209] a middle ={am0 ,a m1 |a m0 ,a m1 ∈[0,1]}

[0210] In the formula, a m0 This is the 0th action in the agent's decision-making process; a m1 This is the first action for the agent to make decisions.

[0211] Reward: The reward value R of the mid-level reinforcement learning model is obtained by normalizing the current real-time market and peak-valley arbitrage profits. middle (t):

[0212]

[0213] In the formula, F ci (t) represents the current period's industrial and commercial revenue, F da F represents the day-ahead electricity cost for the current period. rt (t) represents the real-time electricity cost for the current period, F dev (t) represents the deviation assessment cost for the current time period.

[0214] The mapping yields the target SOC value and the upper limit of charge / discharge output, specifically:

[0215] Deviation assessment constraints:

[0216]

[0217] In the formula, P devmax (t) and P devmin (t) represents the upper and lower limits of the charging and discharging output to avoid deviation assessment costs, respectively, and σ is the allowable arbitrage ratio in the spot market.

[0218] Charge / discharge output range constraints:

[0219] P mupper (t)=min[P socmax (t),P max ,P devmax (t)]

[0220] P mlower (t)=max[P socmin (t),-P max ,P abmin (t),P devmin (t)]

[0221] In the formula, P mupper (t) and P mlower (t) represents the upper and lower limits of the real-time charging and discharging output range, respectively.

[0222] Charging and discharging power P of mid-level reinforcement learning decision making rt (t) by a m0 The mapping yields:

[0223] P rt (t)=P mlower (t)+a m0 (P mupper (t)-P mlower (t))

[0224] Charging and discharging power is further subdivided into charging power P c_rt (t) and discharge power P d_rt (t):

[0225]

[0226] The target SOC value S for the current time period is obtained through the charging and discharging process. OCtar (t):

[0227]

[0228] Meanwhile, the upper limit P of the charging and discharging output of the lower-level optimization model during the current period lupper (t) by a m1 The mapping yields:

[0229]

[0230] In the formula, P llower (t) represents the lower bound of the charging and discharging output mapping of the lower-level optimization model in the current time period.

[0231] The lower-level optimization model targets energy storage participation in the spot market, peak-valley arbitrage, and primary frequency regulation ancillary services. It combines historical frequency statistics and the output ceiling and SOC target output from the middle-level reinforcement learning to make decisions on the real-time frequency regulation charging and discharging output plan for participating in the spot market and peak-valley arbitrage and primary frequency regulation ancillary services during the current period, which is then used for actual operation.

[0232] Objective function: The objective function is established to maximize the profit in the current time period.

[0233] maxM sl (t)+M fl (t)

[0234] In the formula, M sl (t) and M fl (t) represents the spot market and peak-valley arbitrage profits, respectively, and their expressions are as follows:

[0235] M sl (t)=F ci (t)-F da(t)-F rt_pre (t)-F dev_pre (t)

[0236] M fl (t)=F f (t)

[0237] In the formula, F rt_pre (t) and F dev_pre (t) represents the real-time market electricity cost and deviation assessment cost calculated based on the predicted real-time electricity price, respectively. f (t) represents the compensation revenue for primary frequency modulation ancillary services.

[0238] Constraints include:

[0239] Energy storage operation constraints are expressed as:

[0240] P min ≤P c,t (i)≤P max

[0241] P min ≤P d,t (i)≤P max

[0242] P c,t (i)P d,t (i)=0

[0243] In the formula, P c,t (i) and P d,t (i) represent the charging and discharging power of the i-th 15-minute period in the current time period, respectively, P min This represents the minimum energy output from energy storage.

[0244] The relationship between the SOC value change in the i-th 15-minute period of the current time is as follows:

[0245]

[0246] In the formula, E fr,t (i) represents the change in power output caused by frequency modulation during the i-th 15-minute period in the current time period, S OC,t (i) represents the SOC value for the i-th 15-minute period of the current time period. The SOC value satisfies the following constraints:

[0247]

[0248] The corresponding incoming power P of the power grid g,t (i) is:

[0249] P g,t (i)=P load (t)+P c,t(i)-P d,t (i)-P fm,t (i)

[0250] In the formula, P fm,t (i) represents the average frequency modulation output of the i-th 15-minute period in the current time period, P load (t) represents the actual load value for the current time period. P g,t (i) Satisfies:

[0251] P' gmin α≤P g,t (i)≤P gd

[0252] In the formula, P' gmin P represents the minimum operating power of the power grid, α is the coefficient for the minimum power, and P gd This represents the maximum power output of the incoming power grid line.

[0253] Constraints on Spot Market and Peak-Valley Arbitrage Profits: The expressions for the components of spot market and peak-valley arbitrage profits are as follows:

[0254] F da (t)=λ da (t)P dec (t)

[0255] F ci (t)=λ ci (t)P load (t)

[0256]

[0257]

[0258] In the formula, λ da (t), λ ci (t), λ prt (t) represents the actual day-ahead electricity price, the actual industrial and commercial electricity price, and the predicted real-time electricity price for the current period, respectively.

[0259] Primary frequency modulation (FM) auxiliary service constraint: Assume there are N frequency adjustment periods within a given time period where the frequency deviates from the FM dead zone. Establish the FM function and take the average frequency deviation Δf within the frequency adjustment period. ave (n i If ) is used as the independent variable of the frequency modulation function, then the nth digit in the current time period i The primary frequency modulation power for each frequency adjustment period is:

[0260]

[0261] In the formula, P f,t(s) is the primary frequency regulation power at the s-th second in the current period, where s ∈ [0, 3599], P fb,t (i) is the upper limit of the frequency regulation output in the i-th frequency regulation period in the current period, K f,t (i) is the frequency regulation coefficient in the i-th frequency regulation period in the current period. f1 and f2 are the boundaries of the frequency regulation range, satisfying 0 < f1 < f2. The primary frequency regulation power is further divided into the frequency regulation charging power P fc,t (s) and the frequency regulation discharging power P fd,t (s):

[0262]

[0263] Thus, the change in the energy storage frequency regulation power E fr,t (i) is calculated:

[0264]

[0265] To control the demand difference electricity cost caused by the energy storage charge-discharge power, the energy storage output should not exceed the upper limit of the charge-discharge output determined by the middle-layer reinforcement learning:

[0266] P lupper (t) ≥ P c,t (i) - P d,t (i) + P fm,t (i)

[0267]

[0268] The theoretical integral electricity of the frequency regulation action p’ fr (t) is:

[0269]

[0270] In the formula, S is the number of seconds in a day.

[0271] Furthermore, the compensation revenue for the primary frequency regulation ancillary service in the current period is calculated:

[0272]

[0273] In the formula, μ is the ratio of the actual integral electricity of the frequency regulation action to the theoretical integral electricity of the frequency regulation action, and R is the compensation standard for the excess integral electricity of the primary frequency regulation.

[0274] The total compensation revenue Z for the primary frequency regulation ancillary service on the current day f is:

[0275]

[0276] The system uses the state space input data of the mid-level reinforcement learning model to solve the problem, obtains and records the four 15-minute arbitrage charging and discharging powers and frequency regulation charging and discharging powers for the current time period, and records the maximum power of the grid incoming line and the actual load data, which are used to calculate arbitrage profits and rewards.

[0277] After the real-time electricity price is disclosed, the arbitrage profits of the spot market and peak-valley arbitrage are calculated using the real spot electricity price and load data, and the rewards of the upper-layer reinforcement learning model and the middle-layer reinforcement learning model are calculated and updated in a one-time delay.

[0278] Real spot market and peak-valley arbitrage profits:

[0279] Using the arbitrage charging and discharging power and frequency modulation charging and discharging power recorded by the lower-level optimization model, calculate the real spot market and peak-valley arbitrage profits for each time period:

[0280] F da (t)=λ da (t)P dec (t)

[0281] F ci (t)=λ ci (t)P load (t)

[0282]

[0283] In the formula, λ rt (t) represents the real-time market electricity price for the t-th time period.

[0284] The total profit Z from the spot market and peak-valley arbitrage on that day et for:

[0285]

[0286] The total revenue Z after the revenue sharing is calculated based on the revenue sharing agreement in the energy performance contracting agreement. e :

[0287]

[0288] In the formula, k s The profit-sharing ratio stipulated in the contract.

[0289] In addition, based on the recorded maximum power of the incoming power grid and the actual load data, the daily demand difference electricity charge Z is calculated. d :

[0290]

[0291] In the formula, P loadmax Where β is the maximum load power, and T is the demand electricity price. dThis represents the number of days in the current month.

[0292] Theoretical maximum spot market and peak-valley arbitrage profit:

[0293] Using disclosed real electricity prices and recorded real load data, calculate the theoretical peak spot market and peak-valley arbitrage profits for the day:

[0294] Objective function:

[0295] maxZ actual_max (P c ,P d ,P g ,P dec ,λ da ,λ rt ,λ ci )

[0296] Constraints:

[0297] Z actual_max =Z d +Z e

[0298] P min ≤P c (t)≤P max

[0299] P min ≤P d (t)≤P max

[0300] P c (t)P d (t)=0

[0301]

[0302] P g (t)=P load (t)+P c (t)-P d (t)

[0303] P' gmin α≤P g (t)≤P gd

[0304] Z d ≤(P loadmax -P gd )β / T d

[0305] Z d ≤0

[0306] F da (t)=λ da(t)P dec (t)

[0307] F rt (t)=λ rt (t)[P g (t)-P dec (t)]

[0308]

[0309] F ci (t)=λ ci (t)P load (t)

[0310]

[0311] In the formula, P c and P d For the charging and discharging power vector, P g Let λ be the incoming power vector of the power grid. rt This represents the real-time electricity price vector.

[0312] Using the calculated Z actual_max Z e Z d F ci (t), F da F rt (t) and F dev (t) calculates the rewards of the upper-layer reinforcement learning model and the middle-layer reinforcement learning model, and passes them to the upper-layer reinforcement learning model and the middle-layer reinforcement learning model for updating, and adjusts the corresponding neural network parameters.

[0313] Initialize relevant environmental parameters, and divide the day-ahead electricity price, real-time electricity price, and load data into three parts: prediction training set, reinforcement learning training set, and test set. First, train the prediction model using the prediction training set and save the neural network parameters. After training, load the prediction model, train the hierarchical reinforcement learning framework using the reinforcement learning training set, and save the neural network parameters. After training, test the model using the test set to verify the hierarchical reinforcement learning operation effect of the user-side energy storage power station.

[0314] The following example uses a typical user-side energy storage power station project. This power station has a battery capacity of 12192 kWh, a charge / discharge efficiency of 0.93, and a maximum charge / discharge power of 5670 kW. The local industrial and commercial electricity price follows a peak-valley time-of-use pricing policy, differentiating between peak and valley periods and their corresponding prices, and includes a primary frequency regulation ancillary service mechanism. User load, day-ahead electricity price, and real-time electricity price are based on two years of real data from a power sales company. Electricity price levels, peak-valley period divisions, and primary frequency regulation compensation standards adopt relevant standards for the region where the energy storage power station is located to ensure the strategy implementation has practical reference value.

[0315] Utilizing the smart meters and related communication links equipped in the energy storage power station, data such as the power station's state of charge (SOC), charging and discharging power, load power at the grid connection point, and grid frequency are collected and recorded in real time. The collected data undergoes preprocessing to remove outliers.

[0316] A reinforcement learning model and its neural network are constructed using a deep deterministic policy gradient algorithm. The learning rates for the Actor and Actor-target networks are set to 5 × 10⁻⁶, and the learning rates for the Critic and Critic-target networks are set to 1 × 10⁻³. The discount factor is 0.99, the soft update parameter is 5 × 10⁻³, the experience replay buffer size is 204800, the batch size is 20480, the noise value is 0.35, and the number of training epochs is 2000. The lower-level optimization model and the theoretically highest spot market and peak-valley arbitrage profit optimization model are constructed using the MILP algorithm. After construction, the hierarchical reinforcement learning implementation steps of this invention are followed.

[0317] Analyzing the reward curve during the hierarchical reinforcement learning training process, as shown below. Figure 3 As shown, the reward curve and payout curve during the test are as follows: Figure 4 As shown, and the typical test date declaration and charge / discharge plan are as follows: Figure 5 As shown, the verification and optimization strategy is reasonable for day-ahead reporting and real-time charging and discharging operation, and whether it meets the goal of maximizing the comprehensive benefits of the spot market, peak-valley arbitrage, and frequency regulation ancillary services.

[0318] Test analysis shows that the average return across 51 rounds was 25,223.17 yuan, with the highest return occurring in round 26 at 35,504.22 yuan, and the lowest return in round 24 at 10,874.51 yuan. The average reward value was 0.8643, meaning the test return was 86.43% of the theoretically highest return from the spot market and peak-valley arbitrage. This indicates that even with prediction errors, hierarchical reinforcement learning can still maintain high returns, demonstrating strong robustness and improving profitability. Furthermore, the average time per round was 8.294 seconds, indicating fast decision-making speed.

[0319] The results show that the user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning in this invention can participate in peak-valley arbitrage, while also participating in the spot market and frequency regulation ancillary services. It utilizes the predicted electricity price and load value with prediction errors to make day-ahead declaration decisions and real-time operation charging and discharging decisions, thereby obtaining stable and considerable returns.

[0320] The above description is a detailed description of the preferred embodiments of the present invention. However, the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modifications made under the technical spirit disclosed in the present invention should fall within the patent scope covered by the present invention.

Claims

1. A user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning, characterized in that, Includes the following steps: S1: Construct a prediction model and a hierarchical reinforcement learning framework. The prediction model includes an electricity price prediction model and a load prediction model. The hierarchical reinforcement learning framework includes an upper-level reinforcement learning model, a middle-level reinforcement learning model, and a lower-level optimization model. S2: Use the forecasting model to forecast day-ahead electricity prices and loads, design a Markov decision process, define an upper-level reinforcement learning model, design a mapping method, and obtain the day-ahead reporting volume; S3: Use the prediction model to predict real-time electricity prices and loads, design a Markov decision process, define a mid-level reinforcement learning model, design a mapping method, obtain a charging and discharging plan, and output the upper limit of output power and the SOC target. S4: Combining historical frequency statistics, day-ahead reporting volume, output ceiling and SOC target, the lower-level optimization model is used to decide the charging and discharging output participating in the operation of the power system, which is used for the actual operation of user-side energy storage power stations. S5: Using real electricity price and load data, calculate the revenue from participating in the operation of the power system, calculate the rewards for the upper-layer reinforcement learning model and the middle-layer reinforcement learning model, and update the upper-layer reinforcement learning model and the middle-layer reinforcement learning model. S6: Divide the electricity price data and load data into a prediction training set, a reinforcement learning training set, and a test set. Use the prediction training set to train the prediction model, load the trained prediction model, use the reinforcement learning training set to train the hierarchical reinforcement learning framework, and use the test set to test and verify the running effect.

2. The user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning according to claim 1, characterized in that, In step S2, a Markov decision process is designed, defining the state space, action space, and reward function of the upper-level reinforcement learning model, as well as their normalization methods, specifically: State: The state space s consists of day-ahead electricity prices and load forecasts. upper , is represented as: s upper ={λ pda ,l ci ,P load_da } In the formula, λ pda Let λ be the vector of day-ahead electricity price forecasts; ci P represents the industrial and commercial electricity price vector; load_da This is a vector of load forecast values ​​used for day-ahead market reporting; Action: Action space a upper It contains 24 actions with values ​​in the range [0,1], represented as follows: a upper ={a u (0),a u (1),…a u (T-1)|a u (t)∈[0,1]} In the formula, a u (t) represents the action of the agent in time period t; a u (0) represents the action in the 0th time period of the agent's decision-making; a u (1) The action in the first time period for the agent's decision-making; a u (T-1) represents the action taken by the agent in the (T-1)th time period; T is the number of time periods in a day; Reward: The reward value R of the upper-level reinforcement learning model is obtained by normalizing the profits from the real spot market and peak-valley arbitrage. upper , is represented as: In the formula, Z e Z represents the daily arbitrage profit obtained in the spot market and peak-valley arbitrage; d For demand difference electricity charges; Z actual_max This represents the theoretically highest arbitrage profit for the day.

3. The user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning according to claim 2, characterized in that, The mapping method is designed to obtain the current day's reported volume, specifically as follows: Design a mapping method to map actions to derive day-ahead market declarations, including: The safe charging and discharging range is expressed as follows: P socmax (t)=(1-S OCda (t))E / η c P socmin (t)=-S OCda (t)Eη d P abmin (t)=P' gmin α-P load_da (t) In the formula, P socmax (t) and P socmin (t) represents the upper and lower limits of charge / discharge output that satisfy the SOC safety constraints during time period t; P abmin (t) is the lower limit of charge / discharge output to meet the backflow prevention safety constraint; S OCda (t) represents the SOC value at time t during the day-ahead market declaration decision-making process; E represents the rated capacity of the energy storage battery; η c and η d These represent charging efficiency and discharging efficiency, respectively; P' gmin P represents the minimum operating power of the power grid; α is the coefficient for the minimum power; load_da (t) represents the load forecast value for the t-th time period used in the day-ahead market submission; By integrating the charge and discharge constraints, we obtain the upper limit of safe charge and discharge output P. uupper (t) and the lower limit of safe charge and discharge output P ulower (t), represented as: P uupper (t)=min(P socmax (t),P max ) P ulower (t)=max(P socmin (t),-P max ,P abmin (t)) In the formula, P max This represents the maximum output power of the energy storage system. Day-ahead charge / discharge plan: Day-ahead charge / discharge plan P for time period t da (t) is derived from the mapping between the current action and the upper and lower limits of safe charging and discharging output, and is expressed as: P da (t)=P ulower (t)+a u (t)(P uupper (t)-P ulower (t)) Day-ahead market submissions: Day-ahead market submissions P for period t dec (t) is represented as: P dec (t)=P load_da (t)+P da (t) The 24-hour charge and discharge plan is calculated on a rolling basis to obtain the day-ahead market declaration volume vector P. dec , is represented as: P dec ={P dec (0),P dec (1),...,P dec (T-1)} In the formula, P dec (0) represents the market order volume before the 0th time period; P dec (1) Market order volume for the first period; P dec (T-1) represents the market declaration volume for the T-1 period prior to the date.

4. The user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning according to claim 3, characterized in that, In step S3, a Markov decision process is designed, defining the state space, action space, and reward function of the mid-level reinforcement learning model, along with their normalization methods, specifically as follows: State: The state space s consists of the day-ahead reported volume, actual day-ahead electricity price, industrial and commercial electricity price, forecast real-time electricity price, forecast load, previous period's SOC, and the current arbitrage period. middle , is represented as: In the formula, λ da Let λ be the vector of day-ahead electricity prices; prt S is a vector of real-time electricity price forecasts; OC (t-1) represents the SOC value at the end of the previous time period; t represents the current time period; P load (0) represents the actual load in time period 0; P load (t) represents the actual load in time period t; P load_rt (t+1) represents the predicted load for the (t+1)th time period; P load_rt (T-1) represents the predicted load for the T-1 time period; Action: Action space a middle It contains two actions with values ​​[0,1], represented as: a middle ={a m0 ,a m1 |a m0 ,a m1 ∈[0,1]} In the formula, a m0 This is the 0th action in the agent's decision-making process; a m1 The first action for the agent to make decisions; Reward: The reward value R of the mid-level reinforcement learning model is obtained by normalizing the current real-time market and peak-valley arbitrage profits. middle (t), represented as: In the formula, F ci (t) represents the industrial and commercial revenue in the current time period t; F da (t) represents the electricity cost t days prior to the current time period; F rt (t) represents the real-time electricity cost for the current time period t; F dev (t) represents the deviation assessment cost for the current time period t.

5. The user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning according to claim 4, characterized in that, The design mapping method yields the charge / discharge plan, outputting the upper limit of power output and the SOC target, specifically: The design uses a mapping method to map actions to derive charging and discharging plans for the spot market and peak-valley arbitrage, and outputs the upper limit of output and SOC target for each time period, including: Deviation assessment constraints are expressed as follows: In the formula, P devmax (t) and P devmin (t) represents the upper and lower limits of charge / discharge output to avoid deviation assessment costs, respectively; σ is the allowable arbitrage ratio in the spot market; The charging / discharging output range constraint is expressed as: P mupper (t)=min[P socmax (t),P max ,P devmax (t)] P mlower (t)=max[P socmin (t),-P max ,P abmin (t),P devmin (t)] In the formula, P mupper (t) and P mlower (t) represents the upper limit of the real-time charge / discharge output range and the lower limit of the real-time charge / discharge output range, respectively; Charging and discharging power P of mid-level reinforcement learning decision making rt (t) by a m0 The mapping yields the following representation: P rt (t)=P mlower (t)+a m0 (P mupper (t)-P mlower (t)) Charging and discharging power is further subdivided into charging power P c_rt (t) and discharge power P d_rt (t), represented as: The target SOC value S for the current time period is obtained through the charging and discharging process. OCtar (t), represented as: The upper limit of charging and discharging output P of the lower-level optimization model in the current period lupper (t) by a m1 The mapping yields the following representation: In the formula, P llower (t) represents the lower bound of the charging and discharging output mapping of the lower-level optimization model in the current time period.

6. The user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning according to claim 5, characterized in that, In step S4, the charging and discharging outputs participating in the operation of the power system are determined using the lower-level optimization model, including: Objective function: The objective function is established to maximize the profit in the current time period, and is expressed as: maxM sl (t)+M fl (t) In the formula, M sl (t) and M fl (t) represents the spot market and peak-valley arbitrage profits, respectively, and their expressions are as follows: M sl (t)=F ci (t)-F da (t)-F rt_pre (t)-F dev_pre (t) M fl (t)=F f (t) In the formula, F rt_pre (t) and F dev_pre (t) represents the real-time market electricity cost and deviation assessment cost calculated based on the predicted real-time electricity price, respectively; F f (t) represents the compensation revenue for primary frequency modulation ancillary services.

7. The user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning according to claim 6, characterized in that, The use of lower-level optimization models to determine the charging and discharging output participating in power system operation also includes: Energy storage operation constraints are expressed as: P min ≤P c,t (i)≤P max P min ≤P d,t (i)≤P max P c,t (i)P d,t (i)=0 In the formula, P c,t (i) and P d,t (i) represent the charging power and discharging power of the i-th 15-minute period in the current time period, respectively, P min This represents the minimum energy storage output. The relationship between the SOC value change in the i-th 15-minute period of the current time is as follows: In the formula, E fr,t (i) represents the change in power output caused by frequency modulation during the i-th 15-minute period in the current time period; S OC,t (i) represents the SOC value of the i-th 15-minute period in the current time period; the SOC value satisfies the following constraints: The corresponding incoming power P of the power grid g,t (i) is: P g,t (i)=P load (t)+P c,t (i)-P d,t (i)-P fm,t (i) In the formula, P fm,t (i) represents the average frequency modulation output of the i-th 15-minute period in the current time period; P load (t) represents the actual load value for the current time period; P g,t (i) Satisfies: P' gmin α≤P g,t (i)≤P gd In the formula, P' gmin P represents the minimum operating power of the power grid; α is the coefficient for the minimum power; gd This represents the maximum power output of the incoming power grid line. Constraints on Spot Market and Peak-Valley Arbitrage Profits: The expressions for the components of spot market and peak-valley arbitrage profits are as follows: F da (t)=λ da (t)P dec (t) F ci (t)=λ ci (t)P load (t) In the formula, λ da (t), λ ci (t), λ prt (t) represents the actual day-ahead electricity price, the actual industrial and commercial electricity price, and the predicted real-time electricity price for the current period, respectively; Primary frequency modulation (FM) auxiliary service constraint: Assume there are N frequency adjustment periods within a given time period where the frequency deviates from the FM dead zone. Establish the FM function and take the average frequency deviation Δf within the frequency adjustment period. ave (n i If ) is used as the independent variable of the frequency modulation function, then the nth digit in the current time period i The primary frequency modulation power for each frequency adjustment period is: In the formula, P f,t (s) represents the frequency modulation power at the s-th second of the current time period, satisfying s∈[0,3599]; P fb,t (i) represents the upper limit of frequency modulation output during the i-th frequency adjustment period in the current time period; K f,t (i) represents the frequency modulation coefficient for the i-th frequency adjustment period in the current time period; f1 and f2 are the boundaries of the frequency modulation interval, satisfying 0 <f1<f2; The primary frequency modulation power is further subdivided into frequency modulation charging power P. fc,t (s) and frequency-modulated discharge power P fd,t (s), denoted as: Therefore, the change in energy storage frequency regulation power E is calculated. fr,t (i) is represented as: To control the demand difference in electricity costs caused by the charging and discharging power of energy storage, the energy storage output must not exceed the upper limit of the charging and discharging output determined by the mid-level reinforcement learning decision, expressed as: P lupper (t)≥P c,t (i)-P d,t (i)+P fm,t (i) Theoretical frequency modulation action integral power p' fr (t) is: In the formula, S is the number of seconds in a day; Then calculate the compensation revenue F for primary frequency regulation ancillary services in the current period. f (t), represented as: In the formula, μ is the ratio of the actual frequency modulation action integral energy to the theoretical frequency modulation action integral energy, and R is the compensation standard for the excess integral energy of primary frequency modulation. The total revenue Z for one frequency modulation ancillary service on that day f for:

8. The user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning according to claim 7, characterized in that, In step S5, using real electricity prices and load data, the benefits of participating in the operation of the power system are calculated, including: Real spot market and peak-valley arbitrage profits: Using the arbitrage charging and discharging power and frequency modulation charging and discharging power recorded by the lower-level optimization model, the real spot market and peak-valley arbitrage profits for each time period are calculated and expressed as follows: F da (t)=λ da (t)P dec (t) F ci (t)=λ ci (t)P load (t) In the formula, λ rt (t) represents the actual real-time market electricity price for the t-th time period; The total profit Z from the spot market and peak-valley arbitrage on that day et for: The total revenue Z after the revenue sharing is calculated based on the revenue sharing agreement in the energy performance contracting agreement. e , is represented as: In the formula, k s The profit-sharing ratio stipulated in the contract; Based on the recorded maximum power of the incoming power grid and the actual load data, calculate the daily demand difference electricity charge Z. d , is represented as: In the formula, P loadmax Where β is the maximum load power, and T is the demand electricity price. d This represents the number of days in the current month.

9. The user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning according to claim 8, characterized in that, Calculating the benefits of participating in power system operation using real electricity price and load data also includes: Theoretical Peak Spot Market and Peak-Valley Arbitrage Profits: Calculate the theoretical peak spot market and peak-valley arbitrage profits for the day using disclosed real electricity prices and recorded real load data. The objective function is expressed as: maxZ actual_max (P c ,P d ,P g ,P dec ,l da ,l rt ,l ci ) The constraints are expressed as follows: WITH actual_max =Z d +Z e P min ≤P c (t)≤P max P min ≤P d (t)≤P max P c (t)P d (t)=0 P g (t)=P load (t)+P c (t)-P d (t) P' gmin α≤P g (t)≤P gd Z d ≤(P loadmax -P gd )β / T d Z d ≤0 F da (t)=λ da (t)P dec (t) F rt (t)=λ rt (t)[P g (t)-P dec (t)] F ci (t)=λ ci (t)P load (t) In the formula, P c and P d For charging power vector and discharging power vector; P g λ is the incoming power vector of the power grid; rt This represents the real-time electricity price vector.

10. The user-side energy storage power station operation decision-making method based on hierarchical reinforcement learning according to claim 9, characterized in that, The rewards for the upper-layer and middle-layer reinforcement learning models are calculated as follows: Using the calculated Z actual_max Z e Z d F ci (t), F da F rt (t) and F dev (t) calculates the rewards for the upper-layer reinforcement learning model and the middle-layer reinforcement learning model, and passes them to the upper-layer reinforcement learning model and the middle-layer reinforcement learning model for updating, adjusting the corresponding neural network parameters.

Images