Energy storage day-week coordination self-scheduling decision-making method and equipment considering long time scale

Abstract

The invention discloses an energy storage day-week coordination self-scheduling decision-making method and device considering a long time scale, and the method comprises the steps: building an energy storage day-ahead self-scheduling model with the maximum difference between day-ahead charging and discharging income and life loss conversion cost as a target; establishing an energy storage cross-day energy state coordination model by taking the maximum day-ahead self-scheduling income in a self-scheduling period as a target; combining the energy storage day-ahead self-scheduling model and the energy storage cross-day energy state coordination model to obtain an energy storage day-week coordination self-scheduling model; constructing a D3QN-SAC network model of two-stage deep reinforcement learning in combination with the D3QN and the SAC, and training the D3QN-SAC network model in combination with the energy storage day-week coordination self-scheduling model; and inputting power market state data to be decided into the trained D3QN-SAC network model, and outputting the data as a day-week coordination self-scheduling strategy. According to the invention, the problem of two-stage discrete-continuous decision making is solved, and energy storage day-week coordination self-scheduling decision making under a long time scale is realized.

Description

Technical Field

[0001] The present invention relates to energy storage scheduling decision technology, and in particular to a method and device for energy storage daily-weekly coordinated self-scheduling decision-making considering a long time scale. Background Art

[0002] With the large-scale access of new energy to the power system, the intermittent and uncertain output of new energy units may cause the market electricity price to remain high or low for a period of time. Under the premise of uncertainty in the output of new energy, it is very necessary to rationally plan long-term energy storage to meet load demand. Therefore, it is very important to coordinate energy storage on a short-term scale with a long-term scale.

[0003] Due to the complex operating constraints of energy storage, most of the existing research is based on analytical methods or intelligent algorithms, and the time scale is concentrated within one day. The self-dispatching mode does not require clearing of the electricity market, and the declared electricity volume is the winning electricity volume. Therefore, in scenarios where the penetration rate of energy storage installed capacity is not high, the self-dispatching mode is a common method in the current mainstream electricity wholesale markets in Europe and the United States. From the perspective of the solution algorithm, the intelligent agent in reinforcement learning gradually learns how to improve the strategy through repeated interactions with the market clearing process, which is more in line with the needs of actual scenarios. However, there is a problem of training difficulties when reinforcement learning is used to solve problems with multiple decision variables, and it is even more difficult to solve two-stage continuous decision-making problems. Summary of the invention

[0004] In view of the problems existing in the prior art, the purpose of the present invention is to provide a method and device for energy storage daily-weekly coordinated self-scheduling decision-making considering long time scales, which can solve the problem of training difficulties.

[0005] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0006] A method for daily-weekly coordinated self-dispatching decision-making of energy storage considering a long time scale includes the following steps:

[0007] (1) Taking 1 day as the decision period and maximizing the difference between the day-ahead charging and discharging income and the life loss discounted cost as the goal, a day-ahead self-dispatching model for energy storage is established, wherein the day-ahead self-dispatching model for energy storage is used to make decisions on the day-ahead energy state change trend of energy storage according to the day-ahead electricity price trend;

[0008] (2) Taking one week as the self-dispatching cycle and maximizing the self-dispatching benefit on the day before within the self-dispatching cycle as the goal, a cross-day energy state coordination model for energy storage is established, wherein the cross-day energy state coordination model for energy storage is used to make decisions on the change trend of the energy state of energy storage on a single day according to the future electricity price trend;

[0009] (3) combining the energy storage day-ahead self-scheduling model and the energy storage cross-day energy state coordination model to obtain an energy storage day-weekly coordinated self-scheduling model;

[0010] (4) Combine D3QN and SAC to build a two-stage deep reinforcement learning D3QN-SAC network model, and combine it with the energy storage daily-weekly coordinated self-dispatching model, and use the historical data of the interaction between the intelligent agent and the market clearing module to train the D3QN-SAC network model;

[0011] (5) The power market status data to be decided is input into the trained D3QN-SAC network model, and the output is the daily-weekly coordinated self-dispatching strategy.

[0012] Furthermore, the energy storage day-ahead self-dispatching model specifically includes:

[0013]

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024] In the formula, profit k,d,t , C k,d,t are the charging and discharging benefits and life loss discount costs of energy storage k at time period t on day d; T is the total period of energy storage self-dispatching, which is 24 hours in day-ahead self-dispatching, and λ k,d,t is the node electricity price of the node where the energy storage k is located in the t period on the d day; and are the discharge and charging power of energy storage k at time t on day d, c p is the life loss conversion coefficient of energy storage; SOC t is the state of charge of the energy storage at time period t; is the energy state of energy storage k in period t; and are the maximum and minimum energy states of energy storage respectively; The maximum power for discharge and charge, They respectively indicate whether the energy storage k is in discharge mode or charging mode at the t period on the dth day. Specifically, they are 0-1 variables. If it is in discharge mode, then represents the energy state of energy storage k at the time period t+1 and t on the dth day, η d and η c are the charging and discharging efficiencies of energy storage respectively; the length of a single period of Δt is 1h in the day-ahead self-dispatching. and are the maximum and minimum energy states of energy storage respectively; is the energy state of energy storage k at the end of the 24th period, is the energy state of energy storage k at the beginning of the first period, that is, the initial energy state of the day-ahead self-dispatching.

[0025] Furthermore, the energy storage cross-day energy state coordination model is specifically:

[0026]

[0027]

[0028] Where D is the self-scheduling period; profit k,d The day-ahead self-dispatching revenue of energy storage k on day d obtained by solving the day-ahead self-dispatching model of energy storage; is the daily energy state change trend of energy storage k on day d, that is, the difference between the final energy and the initial energy on day d; is the initial energy state of the energy storage on day d; is the final energy state of the energy storage on day d; is the final energy state of energy storage k at the end of one week of self-dispatching; is the initial energy state of energy storage k at the beginning of one week of self-scheduling.

[0029] Furthermore, the D3QN-SAC network model specifically includes a D3QN algorithm network in the first stage and a SAC network in the second stage. The D3QN algorithm network is a deep Q network composed of Current Q Network-Target Q Network, and the SAC network is an Actor-Critic network.

[0030] Furthermore, the training of the D3QN-SAC network model specifically includes:

[0031] Constructing a first experience replay library, the first experience replay library is used to store historical data of the interaction between the intelligent agent and the market clearing module, each piece of historical data includes current power market state data, decision action, action reward and next power market state data;

[0032] Taking the energy storage day-ahead self-dispatching model in the energy storage day-weekly coordinated self-dispatching model as the optimization target, the historical power market status data is input into the D3QN algorithm network for training, and the decision action, i.e., the charging and discharging power, is selected during the training process. Finally, the discrete decision action is converted into a target vector that represents the energy storage energy state change trend;

[0033] Constructing a second experience replay library, the second experience replay library is used to store historical data of the interaction between the agent and the market clearing module, each piece of historical data includes current power market state data, decision action, action reward, next power market state data and target vector;

[0034] Taking the energy storage cross-day energy state coordination model in the energy storage day-week coordinated self-scheduling model as the optimization target, the target vector is input into the SAC network for training, and decision actions are selected during the training process, and the optimal decision action is finally output.

[0035] Furthermore, when training the D3QN algorithm network, the Current Q Network trains the network parameters of the Current Q Network based on the historical data extracted from the experience replay library, and the loss function is as follows:

[0036] L(θ)=(r i +γQ'(s i+1 ,a i+1 |θ')-Q(s i ,a i |θ)) 2

[0037] Where: L(θ) is the loss function, θ is the network parameter of Current Q Network, r i is the revenue; γ is the discount value; Q(s i ,a i |θ) is the output of Current Q Network, Q'(s i+1 ,a i+1 |θ') is the Target Q Network output, s i+1 ,a i+1 are the electricity market status and decision-making actions in the i+1th period, s i ,a i are the electricity market status and decision-making actions in the i-th period respectively;

[0038] Target Q Network updates the network parameters of Target Q Network through soft update according to the training degree of Current Q Network. The specific update method is as follows:

[0039] θ'=πθ+(1-π)θ'

[0040] Where, π is the soft update rate, θ' is the network parameter of Target Q Network;

[0041] Furthermore, when training the D3QN algorithm network, in the early stage of training, the agent randomly selects decision actions and explores the feasible decision action space. When the number of training times increases to a certain value, a threshold is set. The threshold decreases with the increase in the number of training times. In each subsequent training process, the agent generates a random number. If the random number is greater than the threshold, the greedy method is used to select the decision action, otherwise the decision action is randomly selected.

[0042] Furthermore, when the SAC network is trained, continuous decision actions are selected. The specific selection method is as follows: at the beginning, random decision actions are selected, but when the number of training times increases to a preset threshold, greedy actions are selected. When selecting decision actions, the action space is enlarged, and actions beyond the boundary are regarded as boundary actions, so as to improve the efficiency of the SAC algorithm in exploring boundary values. The specific form is as follows:

[0043]

[0044]

[0045] In the formula, A d,t is the decision action of the SAC network at the t-th day, i.e., the charging and discharging power action; a d,t The original action given to the Actor network, is the maximum discharge and charge power, p ess,k,t is the power of energy storage k after adopting the improved action method.

[0046] Furthermore, when the SAC network is trained, the reward function is specifically:

[0047] R d,t =profit k,d,t -C k,d,t -C W,d,t -C E,d,t

[0048]

[0049]

[0050] In the formula, R d,t is the reward function for the t-th period on the d-th day after combining the HER mechanism; C k,d,t are the lifetime loss discount costs of energy storage k at the dth day and the tth period, C E,d,t The energy state of a single period after the charge and discharge action of period t is executed Beyond the limit and lower limit The equivalent penalty term of the constraint; if the line is crossed, the charge and discharge power at this moment is set to 0, and a penalty C0 is given; C W,d,t K is the equivalent penalty term for whether the daily energy state change trend constraint or the initial and final energy state constraint is completed; D is the coefficient of the energy state equivalent penalty term; G' d is the actual energy state change target of energy storage on day d; G is the overall charge and discharge target set of energy storage in a single day; G' d =G d Indicates that the overall charge and discharge of energy level on day d meets the target requirements, is the final energy state of the energy storage on day d, The target end energy state for energy storage on day d is, It is the target energy termination state on the dth day when the energy storage has not reached the target termination energy state but is within the target energy allowable range.

[0051] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above method.

[0052] Compared with the prior art, the present invention has the following beneficial effects: the present invention considers discretizing the first stage and establishing a self-scheduling model for daily-weekly coordinated energy storage. First, the two-stage self-scheduling model of energy storage is modeled, and then the two-stage coordinated optimization algorithm combining the dueling double deep Q network (Dueling Double Deep Q Network, D3QN) and the soft actor-critic (Soft Actor-Critic, SAC) is used to solve the discrete and continuous mixed decision quantities in the model. In order to solve the problem of sparse rewards caused by the superposition of penalty terms in the last decision quantity in long sequence decisions, the present invention introduces the Hindsight Experience Replay (HER) method. The decision-making strategy of energy storage in daily-weekly coordinated self-scheduling is determined by combining the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1It is a flow chart of a method for energy storage daily-weekly coordinated self-dispatching decision-making considering a long time scale provided by an embodiment of the present invention;

[0054] Figure 2 This is a diagram showing the changes in electricity prices within a week;

[0055] Figure 3 This is a schematic diagram of the charging and discharging strategy after daily-weekly coordinated self-dispatching;

[0056] Figure 4 It is the daily charge and discharge amount and energy state change trend of the energy storage entity after daily-weekly coordinated self-dispatching;

[0057] Figure 5 This is the result of the self-dispatching operation of the energy storage system, which maintains a 100MWh energy state throughout the day.

[0058] Figure 6 It is the change trend of the equivalent daily charge and discharge capacity and energy state of energy storage after efficiency conversion in 7 days;

[0059] Figure 7 The benchmark electricity price is 25.18 yuan / MWh (the average electricity price on the 7th day), and the cumulative revenue increment after equivalent processing of the daily energy state changes and the daily revenue of energy storage under different self-dispatching modes are calculated. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0061] Embodiment 1

[0062] The embodiment of the present invention provides a method for energy storage day-week coordinated self-dispatching decision-making considering a long time scale, such as Figure 1 As shown, the following steps are included:

[0063] (1) Taking 1 day as the decision period and maximizing the difference between the day-ahead charging and discharging benefits and the life loss discounted cost as the goal, a day-ahead self-dispatching model for energy storage is established.

[0064] The energy storage day-ahead self-dispatching model is used to make decisions on the energy state change trend of the energy storage day-ahead according to the day-ahead electricity price trend. The energy storage day-ahead self-dispatching model specifically includes:

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076] In the formula, profit k,d,t , C k,d,t are the charging and discharging benefits and life loss discount costs of energy storage k at time period t on day d; T is the total period of energy storage self-dispatching, which is 24 hours in day-ahead self-dispatching, and λ k,d,t is the node electricity price of the node where the energy storage k is located in the t period on the d day; and are the discharge (generation) and charging (power consumption) power of energy storage k at time t on day d, c p is the life loss conversion coefficient of energy storage; SOC t is the state of charge of the energy storage at time period t; is the energy state of energy storage k in period t; and are the maximum and minimum energy states of energy storage respectively; The maximum power for discharge and charge, They respectively indicate whether the energy storage k is in discharge mode or charging mode at the t period on the dth day. Specifically, they are 0-1 variables. If it is in discharge mode, then represents the energy state of energy storage k at the time period t+1 and t on the dth day, η d and η c are the charging and discharging efficiencies of energy storage respectively; the length of a single period of Δt is 1h in the day-ahead self-dispatching. and are the maximum and minimum energy states of energy storage respectively; is the energy state of energy storage k at the end of the 24th period, is the energy state of energy storage k at the beginning of the first period, that is, the initial energy state of the day-ahead self-dispatching.

[0077] (2) Taking one week as the self-dispatching cycle and maximizing the day-ahead self-dispatching benefit within the self-dispatching cycle as the goal, a cross-day energy state coordination model for energy storage is established.

[0078] The energy storage cross-day energy state coordination model is used to make decisions on the energy storage daily energy state change trend according to the future electricity price trend. This helps energy storage achieve daily-weekly energy state coordination. The decision variable is the daily energy state change trend, the decision period is 1 day, and the self-dispatching period is 7 days. The specific model is as follows:

[0079]

[0080]

[0081] Where D is the self-scheduling period; profit k,d The day-ahead self-dispatching revenue of energy storage k on day d obtained by solving the day-ahead self-dispatching model of energy storage; is the daily energy state change trend of energy storage k on the dth day, that is, the difference between the final energy and the initial energy on the dth day. It serves as the link between the first-stage model (cross-day energy state coordination) and the second-stage model (day-ahead self-scheduling) and is an important decision-making quantity in the daily-weekly coordinated self-scheduling model. is the initial energy state of the energy storage on day d; is the final energy state of the energy storage on day d; is the final energy state of energy storage k at the end of one week of self-dispatching; is the initial energy state of energy storage k at the beginning of one week of self-scheduling.

[0082] (3) The energy storage day-ahead self-scheduling model and the energy storage cross-day energy state coordination model are combined to obtain an energy storage day-weekly coordinated self-scheduling model.

[0083] (4) Combining D3QN and SAC, a two-stage deep reinforcement learning D3QN-SAC network model is constructed, and combined with the energy storage daily-weekly coordinated self-scheduling model, the D3QN-SAC network model is trained using historical data.

[0084] Among them, the D3QN-SAC network model specifically includes a D3QN algorithm network in the first stage and a SAC network in the second stage. The D3QN algorithm network is a deep Q network composed of Current Q Network-Target Q Network, and the SAC network is an Actor-Critic network.

[0085] Among them, the method for training the D3QN-SAC network model specifically includes the following steps:

[0086] (4.1) constructing a first experience replay library, which is used to store historical data of the interaction between the agent and the market clearing module, each piece of historical data includes current power market status data, decision actions, action rewards and next power market status data;

[0087] (4.2) Taking the energy storage day-ahead self-dispatching model in the energy storage day-weekly coordinated self-dispatching model as the optimization target, the historical power market status data is input into the D3QN algorithm network for training. During the training process, the decision action is selected, i.e., the charging and discharging power. Finally, the discrete decision action is converted into a target vector representing the energy storage energy state change trend; the details are as follows:

[0088] A d =G d

[0089] In the formula, A d is the decision action of the dth day output by the D3QN algorithm; G d is the target vector of the energy state change trend on the dth day of energy storage, a one-hot vector with a dimension of 10.

[0090] Taking the energy storage with a maximum energy state of 200MWh as an example, three energy states of low (20MWh), medium (100MWh), and high (180MWh) are set, and the specific meanings of the 10 target vectors formed are shown.

[0091]

[0092]

[0093] (4.3) constructing a second experience replay library, wherein the second experience replay library is used to store historical data of the interaction between the agent and the market clearing module, each piece of historical data includes current power market state data, decision action, action reward, next power market state data and target vector;

[0094] (4.4) Taking the cross-day energy state coordination model of energy storage in the energy storage day-week coordinated self-scheduling model as the optimization target, the target vector is input into the SAC network for training, and decision actions are selected during the training process, and finally the optimal decision action is output.

[0095] Among them, when training the D3QN algorithm network, the Current Q Network trains the network parameters of the Current Q Network based on the historical data extracted from the experience replay library, and the loss function is as follows:

[0096] L(θ)=(r i +γQ'(s i+1 ,a i+1|θ')-Q(s i ,a i |θ)) 2

[0097] Where: L(θ) is the loss function, θ is the network parameter of Current Q Network, r i is the revenue; γ is the discount value; Q(s i ,a i |θ) is the output of Current Q Network, Q'(s i+1 ,a i+1 |θ') is the Target Q Network output, s i+1 ,a i+1 are the electricity market status and decision-making actions in the i+1th period, s i ,a i are the electricity market status and decision actions in the i-th period respectively; the training goal of Current Q Network is to minimize the loss function L(θ), and use this as the goal to perform gradient descent training on the parameter θ.

[0098] Target Q Network updates the network parameters of Target Q Network through soft update (SoftUpdate) according to the training degree of Current Q Network. The specific update method is as follows:

[0099] θ'=πθ+(1-π)θ'

[0100] Where, π is the soft update rate, θ' is the network parameter of Target Q Network;

[0101] When training the D3QN algorithm network, in the early stage of training, the agent randomly selects decision actions and explores the feasible decision action space. When the number of training times increases to a certain value, a threshold is set. The threshold decreases with the increase in the number of training times. In each subsequent training process, the agent generates a random number. If the random number is greater than the threshold, the greedy method is used to select the decision action, otherwise the decision action is randomly selected.

[0102] When the SAC network is trained, continuous decision actions are selected. The specific selection method is as follows: at the beginning, random decision actions are selected, but when the number of training times increases to a preset threshold, greedy actions are selected. When selecting decision actions, the action space is enlarged, and actions beyond the boundary are regarded as boundary actions, so as to improve the efficiency of the SAC algorithm in exploring boundary values. The specific form is as follows:

[0103]

[0104]

[0105] In the formula, A d,t is the decision action of the SAC network at the t-th day, i.e., the charging and discharging power action; a d,t The original action given to the Actor network, is the maximum discharge and charge power, p ess,k,t is the power of energy storage k after adopting the improved action method.

[0106] When the SAC network is trained, the reward function is specifically:

[0107] R d,t =profit k,d,t -C k,d,t -C W,d,t -C E,d,t

[0108]

[0109]

[0110] In the formula, R d,t is the reward function for the t-th period on the d-th day after combining the HER mechanism; C k,d,t are the lifetime loss discount costs of energy storage k at the dth day and the tth period, C E,d,t The energy state of a single period after the charge and discharge action of period t is executed Beyond the limit and lower limit The equivalent penalty term of the constraint; if the line is crossed, the charge and discharge power at this moment is set to 0, and a penalty C0 is given; C W,d,t K is the equivalent penalty term for whether the daily energy state change trend constraint or the initial and final energy state constraint is completed; D is the coefficient of the energy state equivalent penalty term; G' d is the actual energy state change target of energy storage on day d; G is the overall charge and discharge target set of energy storage in a single day; G' d =G d Indicates that the overall charge and discharge of energy level on day d meets the target requirements, is the final energy state of the energy storage on day d, The target end energy state for energy storage on day d is, It is the target energy termination state on the dth day when the energy storage has not reached the target termination energy state but is within the target energy allowable range.

[0111] The SAC network loss function is the same as the SAC network loss function in the prior art and will not be described in detail.

[0112] (5) The power market status data to be decided is input into the trained D3QN-SAC network model, and the output is the daily-weekly coordinated self-dispatching strategy.

[0113] Embodiment 2

[0114] The embodiment of the present invention provides a computer device, which provides services for implementing the method of the first embodiment of the present invention. The device may include: a memory storing a computer executable program; a processor coupled to the memory; the processor calls the computer executable program stored in the memory to execute the steps in the method described in the first embodiment.

[0115] The memory may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the memory may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as "hard drives"). A program / utility having a set (at least one) of program modules may be stored in, for example, the memory, such program modules including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The computer executable program of the program module typically performs the functions and / or methods in the embodiments described in the present invention.

[0116] The processor executes various functional applications and data processing by running the program stored in the memory, such as implementing the method provided in the first embodiment of the present invention.

[0117] The code of the computer executable program can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages.

[0118] The present invention is simulated and verified as follows: Figure 2 It shows the change of electricity price within a week, and Table 1 shows the technical parameters of energy storage.

[0119] Table 1 Energy storage technical parameters

[0120]

[0121] Figure 3The charging and discharging strategy after daily-weekly coordinated self-dispatching is demonstrated. It can be seen from the figure that after extending the day-ahead self-dispatching strategy to the daily-weekly coordinated strategy, the energy storage still follows the low-charging and high-discharging strategy. Figure 4 The figure further shows the daily charge and discharge volume and energy state change trend of the energy storage entity after the daily-weekly coordinated self-dispatching. As can be seen from the figure, the terminal energy state of the energy storage on different days changes, which indicates that the energy storage begins to adjust the daily energy state change trend to cope with the overall increase or decrease in electricity prices on different days. On the first day, the market electricity price was high, and the energy storage was mainly discharged, and the terminal energy state changed from 100MWh to 20MWh. On the second day, the market electricity price continued to be high, and the daily charge and discharge volume of the energy storage was basically flat, keeping the terminal energy state at a low level of 20MWh. On the third day, the market electricity price dropped slightly, and the energy storage increased the terminal energy state from 20MWh to 100MWh, and on the fourth day, the energy state was reduced to 20MWh. The overall charge and discharge on the third and fourth days achieved the cross-day transfer of about 80MWh of energy. On the sixth day, the electricity price dropped, and the energy storage increased the energy state to 100MWh again, and remained at this level until the end of the daily-weekly coordinated self-dispatching.

[0122] In order to highlight the advantages of the algorithm of the present invention, the present invention studies the self-scheduling strategy with fixed initial and final energy states and no daily-weekly coordination. The energy storage is set to meet the constraint of 100MWh for each day of the 7-day period, and the simulation is performed only based on the second-stage HER-SAC algorithm.

[0123] Figure 5 The results of the self-dispatching operation of the energy storage system are shown, with the energy state of the energy storage system always maintaining 100MWh at the beginning and end of the day. In order to analyze the energy storage system's daily-weekly coordinated self-dispatching strategy, Figure 6 The figure shows the change trend of the equivalent charge and discharge amount and energy state of energy storage in a single day after efficiency conversion in 7 days. It can be seen from the figure that the terminal energy state of energy storage on different days has basically not changed, and because the energy storage needs to meet the energy state of 100MWh at the beginning and end of a single day, the equivalent charge and discharge amount of each day is basically equal. After 7 days of self-dispatching, the final weekly self-dispatching income of energy storage stabilized at 7,304 yuan.

[0124] Figure 7The cumulative revenue increment after equivalent processing of the single-day energy state change with 25.18 yuan / MWh as the benchmark electricity price (the average electricity price on the 7th day) and the single-day revenue of energy storage under different self-dispatching modes are shown. After the daily-weekly coordinated self-dispatching, the energy storage was mainly discharged on the first day, and the energy state dropped to a low level. Assuming that the subsequent energy storage can be charged at the benchmark electricity price (25.18 yuan / MWh) to restore the energy state to 100MWh, compared with the self-dispatching mode with a fixed energy state, the equivalent cumulative revenue on the first day after considering the daily-weekly coordination increased by 37.7%, about 467 yuan. On the second day, although the single-day self-dispatching revenue of energy storage was positive, the equivalent cumulative revenue increment was slightly reduced (-38 yuan). This is because the discharge on the first day reduced the energy state of energy storage. Compared with the operation mode in which the energy state at the beginning and end was fixed at 100MWh, the lower energy state (20MWh) affected the self-dispatching revenue of energy storage. On the third day, energy storage charging increased the energy state to 100MWh. Although the daily self-dispatching income on the third day was negative, the equivalent cumulative income increment still increased by 93 yuan. This is because the electricity price during charging on the third day was lower than the benchmark electricity price. After conversion, the behavior of charging at this low price is reflected as an increase in the equivalent cumulative income increment. In the following days, energy storage continued to adjust the trend of daily energy state changes, and the equivalent cumulative income maintained a gradual increase overall. After 7 days of self-dispatching operation, compared with the income of fixed initial and final energy states, the equivalent cumulative income increment of energy storage after considering daily-weekly coordination remained at 833 yuan.

[0125] In general, through the daily-weekly coordinated self-dispatching method, the energy storage adjusts the initial and final energy states on the 1st, 3rd, 4th, and 6th days to achieve cross-day energy transfer. After 7 days of daily-weekly coordinated self-dispatching, the energy storage self-dispatching income finally stabilized at 8,137 yuan, compared with 7,304 yuan under the fixed energy state, the energy storage income increased by about 11.4%. Therefore, from the perspective of maximizing the multi-day self-dispatching income of energy storage, it is very necessary to conduct daily-weekly coordinated self-dispatching by adjusting the initial and final energy states of a single day.

[0126] It should be understood that the above embodiments and descriptions only describe the principles, main features and advantages of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, and these changes and improvements all fall within the scope of protection of the present invention.

Claims

1. A decision-making method for daily-weekly coordinated self-dispatching of energy storage considering long time scales. It is characterized in that The steps include: (1) Taking 1 day as the decision period and maximizing the difference between the day-ahead charging and discharging income and the life loss discounted cost as the goal, a day-ahead self-dispatching model for energy storage is established, wherein the day-ahead self-dispatching model for energy storage is used to make decisions on the day-ahead energy state change trend of energy storage according to the day-ahead electricity price trend; (2) Taking one week as the self-dispatching cycle and maximizing the self-dispatching benefit on the day before within the self-dispatching cycle as the goal, a cross-day energy state coordination model for energy storage is established, wherein the cross-day energy state coordination model for energy storage is used to make decisions on the change trend of the energy state of energy storage on a single day according to the future electricity price trend; (3) combining the energy storage day-ahead self-scheduling model and the energy storage cross-day energy state coordination model to obtain an energy storage day-weekly coordinated self-scheduling model; (4) Combining D3QN and SAC, a two-stage deep reinforcement learning D3QN-SAC network model is constructed, and combined with the energy storage daily-weekly coordinated self-scheduling model, the D3QN-SAC network model is trained using historical data; (5) The power market status data to be decided is input into the trained D3QN-SAC network model, and the output is the daily-weekly coordinated self-dispatching strategy.

2. The energy storage daily-weekly coordinated self-dispatching decision-making method considering long time scale according to claim 1, It is characterized in that The energy storage day-ahead self-dispatching model specifically includes: In the formula, profit k,d,t , C k,d,t are the charging and discharging benefits and life loss discount costs of energy storage k at time period t on day d; T is the total period of energy storage self-dispatching, which is 24 hours in day-ahead self-dispatching, and λ k,d,t is the node electricity price of the node where the energy storage k is located in the t period on the d day; and are the discharge and charging power of energy storage k at time t on day d, c p is the life loss conversion coefficient of energy storage; SOC t is the state of charge of the energy storage at time period t; is the energy state of energy storage k in period t; and are the maximum and minimum energy states of energy storage respectively; The maximum power for discharge and charge, They respectively indicate whether the energy storage k is in discharge mode or charging mode at the t period on the dth day. Specifically, they are 0-1 variables. If it is in discharge mode, then represents the energy state of energy storage k at the time period t+1 and t on the dth day, η d and η c are the charging and discharging efficiencies of energy storage respectively; the length of a single period of Δt is 1h in the day-ahead self-dispatching. and are the maximum and minimum energy states of energy storage respectively; is the energy state of energy storage k at the end of the 24th period, is the energy state of energy storage k at the beginning of the first period, that is, the initial energy state of the day-ahead self-dispatching.

3. The energy storage daily-weekly coordinated self-dispatching decision-making method considering long time scale according to claim 1, It is characterized in that The energy storage cross-day energy state coordination model is specifically: Where D is the self-scheduling period; profit k,d The day-ahead self-dispatching revenue of energy storage k on day d obtained by solving the day-ahead self-dispatching model of energy storage; is the daily energy state change trend of energy storage k on day d, that is, the difference between the final energy and the initial energy on day d; is the initial energy state of the energy storage on day d; is the final energy state of the energy storage on day d; is the final energy state of energy storage k at the end of one week of self-dispatching; is the initial energy state of energy storage k at the beginning of one week of self-scheduling.

4. The energy storage daily-weekly coordinated self-dispatching decision-making method considering long time scale according to claim 1, It is characterized in that The D3QN-SAC network model specifically includes a D3QN algorithm network in the first stage and a SAC network in the second stage. The D3QN algorithm network is a deep Q network composed of Current Q Network-Target Q Network, and the SAC network is an Actor-Critic network.

5. The energy storage daily-weekly coordinated self-dispatching decision-making method considering long time scale according to claim 4, It is characterized in that The training of the D3QN-SAC network model specifically includes: Constructing a first experience replay library, the first experience replay library is used to store historical data of the interaction between the intelligent agent and the market clearing module, each piece of historical data includes current power market state data, decision action, action reward and next power market state data; Taking the energy storage day-ahead self-dispatching model in the energy storage day-weekly coordinated self-dispatching model as the optimization target, the historical power market status data is input into the D3QN algorithm network for training, and the decision action, i.e., the charging and discharging power, is selected during the training process. Finally, the discrete decision action is converted into a target vector that represents the energy storage energy state change trend; Constructing a second experience replay library, the second experience replay library is used to store historical data of the interaction between the agent and the market clearing module, each piece of historical data includes current power market state data, decision action, action reward, next power market state data and target vector; Taking the energy storage cross-day energy state coordination model in the energy storage day-week coordinated self-scheduling model as the optimization target, the target vector is input into the SAC network for training, and decision actions are selected during the training process, and the optimal decision action is finally output.

6. The energy storage daily-weekly coordinated self-dispatching decision-making method considering long time scale according to claim 5, It is characterized in that When training the D3QN algorithm network, the Current Q Network trains the network parameters of the Current Q Network based on the historical data extracted from the experience replay library. The loss function is as follows: L(θ)=(ri+γQ'(si +1 oh, oh +1 |θ')-Q(si,ai|θ)) 2 Where: L(θ) is the loss function, θ is the network parameter of Current Q Network, r i is the revenue; γ is the discount value; Q(s i ,a i |θ) is the output of Current Q Network, Q'(s i+1 ,a i+1 |θ') is the Target Q Network output, s i+1 ,a i+1 are the electricity market status and decision-making actions in the i+1th period, s i ,a i are the electricity market status and decision-making actions in the i-th period respectively; Target Q Network updates the network parameters of Target QNetwork through soft update according to the training degree of Current Q Network. The specific update method is as follows: θ'=πθ+(1-π)θ' Where π is the soft update rate and θ' is the network parameter of Target Q Network.

7. The energy storage daily-weekly coordinated self-dispatching decision-making method considering long time scale according to claim 5, It is characterized in that When training the D3QN algorithm network, in the early stage of training, the agent randomly selects decision actions and explores the feasible decision action space. When the number of training times increases to a certain value, a threshold is set. The threshold decreases with the increase in the number of training times. In each subsequent training process, the agent generates a random number. If the random number is greater than the threshold, the greedy method is used to select the decision action, otherwise the decision action is randomly selected.

8. The energy storage daily-weekly coordinated self-dispatching decision-making method considering long time scale according to claim 5, It is characterized in that When the SAC network is trained, continuous decision actions are selected. The specific selection method is as follows: at the beginning, random decision actions are selected, but when the number of training times increases to a preset threshold, greedy actions are selected. When selecting decision actions, the action space is enlarged, and actions beyond the boundary are regarded as boundary actions, so as to improve the efficiency of the SAC algorithm in exploring boundary values. The specific form is as follows: In the formula, A d,t is the decision action of the SAC network at the t-th day, i.e., the charging and discharging power action; a d,t The original action given to the Actor network, is the maximum discharge and charge power, p ess,k,t is the power of energy storage k after adopting the improved action method.

9. The energy storage daily-weekly coordinated self-dispatching decision-making method considering long time scale according to claim 5, It is characterized in that When the SAC network is trained, the reward function is specifically: R d,t =profit k,d,t -C k,d,t -C W,d,t -C E,d,t In the formula, R d,t is the reward function for the t-th period on the d-th day after combining the HER mechanism; C k,d,t are the lifetime loss discount costs of energy storage k at the dth day and the tth period, C E,d,t The energy state of a single period after the charge and discharge action of period t is executed Beyond the limit and lower limit The equivalent penalty term of the constraint; if the line is crossed, the charge and discharge power at this moment is set to 0, and a penalty C is given 0 ; C W,d,t K is the equivalent penalty term for whether the daily energy state change trend constraint or the initial and final energy state constraint is completed; D is the coefficient of the energy state equivalent penalty term; G' d is the actual energy state change target of energy storage on day d; G is the overall charge and discharge target set of energy storage in a single day; G' d =G d Indicates that the overall charge and discharge of energy level on day d meets the target requirements, is the final energy state of the energy storage on day d, The target end energy state for energy storage on day d is, It is the target energy termination state on the dth day when the energy storage has not reached the target termination energy state but is within the target energy allowable range.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, Features: The processor executes the computer program to implement the method according to any one of claims 1 to 9.

Images