A real-time control method and system for wind power energy storage station

Abstract

The present invention provides a method and system for real-time control of a wind power energy storage station, the method comprising: constructing an online operation model of the wind power energy storage station; optimizing the online operation model based on the Lyapunov optimization method to obtain a basic online scheduling strategy; scheduling the wind power energy storage station using the basic online scheduling strategy to obtain basic operation data; constructing an advanced scheduling model of the wind power energy storage station based on the DDPG reinforcement learning algorithm; training the advanced scheduling model according to the basic operation data until the advanced scheduling model meets the preset evaluation criteria, obtaining an advanced online scheduling strategy; and controlling the wind power energy storage station using the advanced online scheduling strategy. The real-time control method provided by the present invention supports the safe and stable operation of the wind power energy storage station in the initial stage of its establishment through the basic online scheduling strategy, and supports the economical and efficient operation of the wind power energy storage station in the later stage through the advanced online scheduling strategy. Its implementation does not need to rely on historical data, and the wind power energy storage station can be controlled online in real time.

Description

Technical Field

[0001] The present invention relates to the field of electric power technology, and in particular to a real-time control method and system for a wind power energy storage station. Background Art

[0002] With the introduction of the "dual carbon" goals and the increasingly severe energy problems, the development of renewable energy has become a key path for the current power system reform. Generally speaking, coastal areas have abundant offshore wind power resources. For these areas, developing offshore wind power is an important means to achieve clean and green power generation. In actual operation, in order to reduce the impact of wind power volatility and randomness, wind farms are usually equipped with a certain capacity of energy storage and operated as a wind-storage combined power station. However, newly built wind-storage combined power stations usually lack historical operating data, making it difficult to train a prediction model in a short period of time to predict uncertain quantities such as wind power output. Therefore, it is impossible to consider future information and can only make scheduling decisions based on the current state of the system. In this context, how to design a suitable real-time control algorithm to support the online operation of wind-storage combined power stations has become a key issue.

[0003] Existing methods for real-time wind power and energy storage control typically employ algorithms such as approximate dynamic programming and Q-learning for online scheduling. However, these methods rely heavily on historical data. When historical data is scarce or absent, decisions and samples must be obtained through random sampling within the decision space. This can pose risks to the safe and stable operation of the system, and the algorithm often exhibits poor convergence. Furthermore, Q-learning typically discretizes the decision space, limiting the possible values ​​of the decision variables and consequently limiting its effectiveness. Summary of the Invention

[0004] The present invention aims to provide a real-time control method and system for a wind power energy storage station to solve the above technical problems, without relying on historical data, and can achieve online real-time control of the wind power energy storage station.

[0005] In order to solve the above technical problems, the present invention provides a real-time control method for a wind power energy storage station, comprising the following steps:

[0006] Build an online operation model of a wind power energy storage station;

[0007] Optimize the online operation model based on the Lyapunov optimization method to obtain the basic online scheduling strategy; use the basic online scheduling strategy to dispatch the wind power energy storage station and obtain basic operation data;

[0008] Build an advanced scheduling model for wind power storage stations based on the DDPG reinforcement learning algorithm;

[0009] The advanced scheduling model is trained based on the basic operation data until it meets the preset evaluation criteria and the advanced online scheduling strategy is obtained.

[0010] An advanced online dispatching strategy is used to regulate the wind power storage station.

[0011] In the absence of historical data and future forecast information, the above solution uses the Lyapunov optimization method to optimize the online operation model, obtain a basic online scheduling strategy, schedule the wind power energy storage station, and collect basic operation data. After the amount of basic operation data has accumulated to a certain level, the advanced scheduling model constructed based on the DDPG reinforcement learning algorithm is trained to implement advanced control of the wind power energy storage station. Among them, the basic online scheduling strategy is used to support the safe and stable operation of the wind power energy storage station in the initial stage of establishment, and the advanced online scheduling strategy can support the economical and efficient operation of the wind power energy storage station in the later stage. The setting of the evaluation standard provides a reference for the online operation mode of the wind power energy storage station. The implementation of the above solution does not need to rely on historical data and can perform online real-time control of the wind power energy storage station.

[0012] Furthermore, constructing an online operation model for a wind energy storage station is the foundation for subsequently developing an online scheduling strategy. "Online" refers to the fact that the operation of the wind energy storage station relies solely on past and current information, rather than on forecasts of future uncertainties. Given the high randomness of uncertainties such as wind power in practical applications, accurate predictions are difficult in the absence of historical data. Therefore, constructing an online operation model for a wind energy storage station is of practical significance for newly built wind energy storage stations. This online operation model includes constraints on wind power output, energy storage capacity, and transmission line capacity.

[0013] Furthermore, the online operation model is optimized based on the Lyapunov optimization method to obtain the basic online scheduling strategy, specifically:

[0014] Based on the Lyapunov optimization method, the online running model optimization at time t is modeled as a Lyapunov optimization problem:

[0015] minτQ t (P t ws -P t sg )+V[-π t τ(P t wg +η d P t sg )]

[0016] sT t ws ≥0, Pt wg ≥0, P t sg ≥0

[0017]

[0018] P t wg +η d P t sg ≤T u

[0019] P t ws ≤P u , P t sg ≤P u

[0020] Where: τ represents the time interval between each decision; Q t It is a virtual queue built based on energy storage capacity, with Q t =E t -η, E t represents the energy storage capacity at time t, η is a constant; P t ws is the wind power used for energy storage charging; P t wg is the wind power directly transmitted to the receiving grid; V is a constant; π t represents the real-time electricity price; η d is the discharge efficiency of energy storage; P t sg The power transmitted from the energy storage station to the receiving grid; η c is the charging efficiency of energy storage; P t w represents the maximum wind power output at time t; T u is the transmission line capacity; P u is the maximum charge and discharge power of energy storage;

[0021] Solve the Lyapunov optimization problem and obtain basic online scheduling strategies.

[0022] In the above solution, when a newly built wind energy storage station is first put into operation, the lack of historical operating data makes it difficult to train an online scheduling strategy. At this time, a basic scheduling strategy can be established based on the Lyapunov optimization method to support the safe and stable operation of the wind energy storage station in its initial stage of construction. As an online optimization method, the Lyapunov optimization method makes decisions based solely on the current state of the wind energy storage station. It does not rely on future forecast information or past historical operating data, and therefore can serve as a basic online scheduling strategy for wind energy storage stations.

[0023] Furthermore, the basic online scheduling strategy is used to schedule the wind power energy storage station and obtain basic operating data specifically as follows:

[0024] A basic online dispatching strategy is used to dispatch the wind power energy storage station, obtaining the state and decision variables at each moment, and obtaining a series of state-action data;

[0025] Calculate the corresponding reward based on the state-action data and establish a data quadruple as the basic running data.

[0026] Furthermore, the advanced scheduling model of the wind power energy storage station is constructed based on the DDPG reinforcement learning algorithm, specifically:

[0027] An advanced scheduling model for wind power energy storage stations is constructed based on the DDPG reinforcement learning algorithm. It includes an actor network, a critic network, an actor target network, and a critic target network.

[0028] The Actor network is used to generate a decision-making strategy, that is, after inputting a state, it will output the corresponding action and generate the Actor target network update parameters;

[0029] The critic network is used to evaluate the decision-making strategy, that is, after inputting the state and action, it will output the corresponding score and generate the critic target network update parameters;

[0030] The Actor target network is used to generate a simulated real decision strategy based on the updated parameters and input status of the Actor target network;

[0031] The critic network is used to update parameters and input status of the critic target network, simulate the real decision strategy for evaluation, and generate a simulated real score;

[0032] In the advanced scheduling model of the wind power energy storage station, the status includes the energy storage capacity, maximum wind power output and real-time electricity price; the actions include the power of energy storage charging, the power directly transmitted to the receiving grid, and the power transmitted from the energy storage station to the receiving grid; the score is the long-term benefit brought by the energy storage charging and discharging actions; and the simulated real score is the real long-term benefit brought by the simulated energy storage charging and discharging actions.

[0033] Furthermore, the advanced scheduling model is trained according to the basic operation data until the advanced scheduling model meets the preset evaluation criteria, and the advanced online scheduling strategy is obtained, specifically:

[0034] The basic operation data is a data quadruple, including state, action, reward and state at the next moment;

[0035] Based on basic operation data, the back propagation algorithm is used to train the Actor network and Critic network, continuously update the network parameters, and obtain the current scheduling strategy;

[0036] The training goal of the Actor network is to maximize the long-term benefits of energy storage charging and discharging actions; the goal of the Critic network is to minimize the error between the long-term benefits of energy storage and the simulated real long-term benefits of energy storage.

[0037] A judgment is made based on the profit of the wind power energy storage station under the current scheduling strategy and the profit under the basic online scheduling strategy. If the preset evaluation criteria are met, the current scheduling strategy is used as the advanced online scheduling strategy. Otherwise, the Actor network and Critic network continue to be trained.

[0038] The above solution provides a real-time control method for wind energy storage stations. The basic online scheduling strategy it proposes can ensure the safe operation of the wind energy storage station. Based on the basic operating data obtained from the basic online scheduling strategy, an advanced scheduling model for the wind energy storage station constructed by the DDPG reinforcement learning algorithm is trained. This can further optimize the online operating efficiency of the wind energy storage station and provide a reference for the online operation of the wind energy storage station.

[0039] The present invention also provides a real-time control system for a wind power energy storage station, comprising an online operation module, a Lyapunov optimization module, a basic scheduling module, a DDPG algorithm module, a training and evaluation module, and an advanced scheduling module; wherein:

[0040] The online operation module is used to construct an online operation model of the wind power energy storage station;

[0041] The Lyapunov optimization module is used to optimize the online operation model based on the Lyapunov optimization method to obtain the basic online scheduling strategy;

[0042] The basic scheduling module is used to schedule the wind power energy storage station using a basic online scheduling strategy and obtain basic operating data;

[0043] The DDPG algorithm module is used to build an advanced scheduling model for wind power storage stations based on the DDPG reinforcement learning algorithm;

[0044] The training and evaluation module is used to train the advanced scheduling model according to the basic operation data until the advanced scheduling model meets the preset evaluation criteria and obtains the advanced online scheduling strategy;

[0045] The advanced scheduling module is used to regulate the wind power energy storage station using an advanced online scheduling strategy.

[0046] Furthermore, the Lyapunov optimization module is used to optimize the online operation model based on the Lyapunov optimization method to obtain the basic online scheduling strategy, specifically:

[0047] Based on the Lyapunov optimization method, the online running model optimization at time t is modeled as a Lyapunov optimization problem:

[0048] minτQ t (P t ws -P t sg )+V[-π t τ(P t wg +η d P t sg )]

[0049] sT t ws ≥0, P t wg ≥0, P t sg ≥0

[0050]

[0051] P t wg +η d P t sg ≤T u

[0052] P t ws ≤P u , P t sg ≤P u

[0053] Where: τ represents the time interval between each decision; Q t It is a virtual queue built based on energy storage capacity, with Q t =E t -η, E t represents the energy storage capacity at time t, η is a constant; P t ws is the wind power used for energy storage charging; P t wg is the wind power directly transmitted to the receiving grid; V is a constant; π t represents the real-time electricity price; η d is the discharge efficiency of energy storage; P t sgThe power transmitted from the energy storage station to the receiving grid; η c is the charging efficiency of energy storage; P t w represents the maximum wind power output at time t; T u is the transmission line capacity; P u is the maximum charge and discharge power of energy storage;

[0054] Solve the Lyapunov optimization problem and obtain basic online scheduling strategies.

[0055] Furthermore, the basic scheduling module is used to schedule the wind power energy storage station using a basic online scheduling strategy and obtain basic operating data, specifically:

[0056] A basic online dispatching strategy is used to dispatch the wind power energy storage station, obtaining the state and decision variables at each moment, and obtaining a series of state-action data;

[0057] Calculate the corresponding reward based on the state-action data and establish a data quadruple as the basic running data.

[0058] Furthermore, the DDPG algorithm module is used to construct an advanced scheduling model for wind power energy storage stations based on the DDPG reinforcement learning algorithm, specifically:

[0059] An advanced scheduling model for wind power energy storage stations is constructed based on the DDPG reinforcement learning algorithm, including an actor network, a critic network, an actor target network, and a critic target network.

[0060] The Actor network is used to generate a decision-making strategy, that is, after inputting a state, it will output the corresponding action and generate the Actor target network update parameters;

[0061] The critic network is used to evaluate the decision-making strategy, that is, after inputting the state and action, it will output the corresponding score and generate the critic target network update parameters;

[0062] The Actor target network is used to generate a simulated real decision strategy based on the updated parameters and input status of the Actor target network;

[0063] The critic network is used to update parameters and input status of the critic target network, simulate the real decision strategy for evaluation, and generate a simulated real score;

[0064] In the advanced scheduling model of the wind power energy storage station, the status includes the energy storage capacity, maximum output and real-time electricity price; the actions include the power of energy storage charging, the power directly transmitted to the receiving grid, and the power transmitted from the energy storage station to the receiving grid; the score is the long-term benefit brought by the energy storage charging and discharging actions; and the simulated real score is the real long-term benefit brought by the simulated energy storage charging and discharging actions.

[0065] Furthermore, the training and evaluation module is used to train the advanced scheduling model based on the basic operation data until the advanced scheduling model meets the preset evaluation criteria and obtain the advanced online scheduling strategy, specifically:

[0066] The basic operation data is a data quadruple, including state, action, reward and state at the next moment;

[0067] Based on basic operation data, the back propagation algorithm is used to train the Actor network and Critic network, continuously update the network parameters, and obtain the current scheduling strategy;

[0068] The training goal of the Actor network is to maximize the energy storage capacity; the goal of the Critic network is to minimize the error between the energy storage capacity and the simulated actual energy storage capacity.

[0069] A judgment is made based on the profit of the wind power energy storage station under the current scheduling strategy and the profit under the basic online scheduling strategy. If the preset evaluation criteria are met, the current scheduling strategy is used as the advanced online scheduling strategy. Otherwise, the Actor network and Critic network continue to be trained.

[0070] The present invention also provides a real-time control method and device for a wind power energy storage station, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the real-time control method for a wind power energy storage station is implemented.

[0071] The present invention also provides a storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the real-time control method for a wind power energy storage station. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A schematic flow chart of a real-time control method for a wind power energy storage station provided by one embodiment of the present invention;

[0073] Figure 2 A schematic diagram of the structure of a wind power energy storage station provided by one embodiment of the present invention;

[0074] Figure 3A schematic diagram of the advanced scheduling model structure of a wind power energy storage station provided by one embodiment of the present invention;

[0075] Figure 4 A schematic diagram of a basic online scheduling strategy assisted advanced scheduling model training provided by an embodiment of the present invention;

[0076] Figure 5 A schematic diagram of a process for determining whether an advanced scheduling model meets evaluation criteria according to an embodiment of the present invention;

[0077] Figure 6 A schematic diagram of the structure of a real-time control system for a wind power energy storage station provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0078] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0079] See Figure 1 A real-time control method for a wind power energy storage station comprises the following steps:

[0080] S1: Build an online operation model of the wind power energy storage station;

[0081] S2: Optimize the online operation model based on the Lyapunov optimization method to obtain the basic online scheduling strategy;

[0082] S3: Use the basic online dispatching strategy to dispatch the wind power energy storage station and obtain basic operating data;

[0083] S4: Building an advanced dispatching model for wind power storage stations based on the DDPG reinforcement learning algorithm;

[0084] S5: Train the advanced scheduling model based on the basic operation data until the advanced scheduling model meets the preset evaluation criteria and obtains the advanced online scheduling strategy;

[0085] S6: Use advanced online dispatching strategies to control wind power storage stations.

[0086] In the absence of historical data and future forecast information, this embodiment uses the Lyapunov optimization method to optimize the online operation model to obtain a basic online scheduling strategy, schedule the wind power energy storage station and collect basic operation data; after the amount of basic operation data accumulates to a certain level, the advanced scheduling model constructed based on the DDPG reinforcement learning algorithm is trained to achieve advanced control of the wind power energy storage station. Among them, the basic online scheduling strategy is used to support the safe and stable operation of the wind power energy storage station in the initial stage of establishment, and the advanced online scheduling strategy can support the economical and efficient operation of the wind power energy storage station in the later stage. The setting of the evaluation standard provides a reference for the online operation mode of the wind power energy storage station. The implementation of the above solution does not need to rely on historical data, and the wind power energy storage station can be regulated online in real time.

[0087] Furthermore, the construction of an online operation model for a wind energy storage station is the basis for subsequently developing an online scheduling strategy. Online operation refers to the fact that the operation of a wind energy storage station relies solely on past and current information, rather than on forecasts of future uncertainties. Given the high randomness of uncertainties such as wind power in practical applications, accurate forecasts are difficult to make without historical data. Therefore, constructing an online operation model for a wind energy storage station is of practical significance for newly constructed wind energy storage stations. See [Note: The original text appears to be corrupted and should likely be omitted.] Figure 2 ,The online operation model of the wind power and energy storage station is constructed including wind power output constraints, energy storage capacity constraints, and transmission line capacity constraints. Specifically:

[0088] Assume that the maximum output of the wind farm at time t is P t w , where the power for charging the energy storage is P t ws , the power directly transmitted to the receiving grid is P t wg If there is still unusable wind energy, it will be discarded. The power of energy storage transmitted to the receiving grid is P t sg . P at each moment t ws 、P t wg 、P t sg are decision variables, which should satisfy the following constraints.

[0089] First, each decision variable is non-negative:

[0090]

[0091] The actual output of a wind farm at any time cannot exceed the maximum output, so there is

[0092]

[0093] where ηc is the charging efficiency of energy storage. The power flowing to the receiving grid cannot exceed the capacity of the transmission line, so:

[0094]

[0095] where η d is the discharge efficiency of energy storage, T u is the transmission line capacity;

[0096] The operating constraints of energy storage are as follows:

[0097]

[0098]

[0099]

[0100]

[0101] Among them, P u is the maximum charge and discharge power of energy storage, E l , E u The first two constraints are the power constraints of the energy storage, the third is the state equation of the energy storage capacity, and the fourth is the energy storage capacity constraint.

[0102] As for the objective function, the operation goal of the wind power energy storage station is to maximize the operating benefits within a period of time. Let π t represents the real-time electricity price purchased by the receiving grid from the wind power station at time t, T is the operation period, and the objective function is:

[0103]

[0104] The online running model is obtained as follows:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112] The problem targeted by the online operation model is a linear programming problem, which is easy to solve. However, the optimization problem of this model requires all parameters from time 0 to T to be known in advance, so that global optimization can be performed in advance based on the above model. However, in actual operation, the maximum wind power output P t w 、Real-time electricity price π t Such information is usually difficult to predict, and only its upper and lower bounds are known. π t ∈[π min ,π max ], so these parameters need to be modeled as random quantities. Accordingly, the objective function should be written as:

[0113]

[0114] in Indicates expected value.

[0115] In actual operation, since future random quantity information is difficult to predict, online decisions can only be made based on current information to maximize the above expected benefits. In the problem involved in this embodiment, the decision variable at time t is P t ws 、P t wg 、P t sg , and the known information is the current energy storage capacity E t , maximum wind power output P t w 、Real-time electricity price π t Therefore, the decision quantity and state quantity at time t can be recorded as

[0116] a t =(P t ws ,P t wg ,P t sg )

[0117] s t =(E t ,P t w ,π t )

[0118] where a t represents the decision amount at time t, s t Represents the state quantity at time t. The online scheduling strategy is to find the mapping from the state quantity to the decision quantity. t =φ t (s t), so that the expected benefit of the entire operation cycle can be optimized as much as possible under the premise of satisfying various constraints.

[0119] It should be noted that the above online operation model can be used for online decision-making of wind power energy storage stations. The specific application process is as follows:

[0120] Step 1: Observe the energy storage capacity E at the current time t t , maximum wind power output P t w 、Real-time electricity price π t etc. status information;

[0121] Step 2: Use the current status information (E t ,P t w ,π t ) as input, run some online scheduling strategy algorithm, and output the current decision (P t ws ,P t wg ,P t sg ), the single-step benefit π brought by executing this decision t τ(P t wg +η d P t sg ) and the corresponding energy storage capacity E after the decision is executed t+1 .

[0122] Step 3: Enter time t+1 and return to step 1.

[0123] The key to the above application process lies in the online scheduling strategy in step 2. The difficulty lies in that in the online operation mode, only the single-step benefit brought by the current decision can be calculated. However, since the current decision will affect the future system state, that is, the energy storage capacity E at the next moment, t+1 This in turn affects future decisions, so that the optimal single-step benefit does not necessarily mean the optimal overall benefit for the entire operation cycle. In this case, how to optimize the benefit of the entire operation cycle as much as possible is the key technical issue facing the design of online scheduling strategies.

[0124] Furthermore, the online operation model is optimized based on the Lyapunov optimization method to obtain the basic online scheduling strategy, specifically:

[0125] Based on the Lyapunov optimization method, the online running model optimization at time t is modeled as a Lyapunov optimization problem:

[0126] minτQ t (P tws -P t sg )+V[-π t τ(P t wg +η d P t sg )]

[0127] sT t ws ≥0, P t wg ≥0, P t sg ≥0

[0128]

[0129] P t wg +η d P t sg ≤T u

[0130] P t ws ≤P u , P t sg ≤P u

[0131] Where: τ represents the time interval between each decision; Q t It is a virtual queue built based on energy storage capacity, with Q t =E t -η, E t represents the energy storage capacity at time t, η is a constant; P t ws is the wind power used for energy storage charging; P t wg is the wind power directly transmitted to the receiving grid; V is a constant; π t represents the real-time electricity price; η d is the discharge efficiency of energy storage; P t sg The power transmitted from the energy storage station to the receiving grid; η c is the charging efficiency of energy storage; P t w represents the maximum wind power output at time t; T u is the transmission line capacity; P u is the maximum charge and discharge power of energy storage;

[0132] Solve the Lyapunov optimization problem and obtain basic online scheduling strategies.

[0133] In the above scheme, when the newly built wind power energy storage station is just put into operation, it is difficult to train the online scheduling strategy due to the lack of historical operating data. At this time, a basic scheduling strategy can be established based on the Lyapunov optimization method to support the safe and stable operation of the wind power energy storage station in the initial stage of establishment. As an online optimization method, the Lyapunov optimization method makes decisions only based on the current state of the wind power energy storage station. It does not rely on future prediction information nor requires past historical operating data. Therefore, it can be used as the basic online scheduling strategy for wind power energy storage stations. Although the Lyapunov optimization method cannot guarantee global optimality, it can ensure the safe operation of the system, that is, the operating strategy at any time can fully meet the various operating constraints of the system.

[0134] It should be noted that the Lyapunov optimization method is widely used in fields such as communications and networks. Its core idea is to establish a virtual queue and ensure its stability while optimizing the objective function, that is, the virtual queue length is as close to a constant as possible. Based on this idea, the stability of the energy storage capacity in the operation of wind power energy storage stations can be modeled as follows:

[0135]

[0136] That is, over a long period of time, the net charge of the energy storage is expected to approach zero, thereby ensuring the stability of the energy storage capacity, and neither significantly charging beyond the upper limit of the capacity nor significantly discharging beyond the lower limit of the capacity.

[0137] Based on the above ideas, the online operation model at time t can be optimized as a Lyapunov optimization problem. It should be noted that the objective function of the Lyapunov optimization problem is the energy storage queue drift τ(P t ws -P t sg ) and the original single-stage objective function π t τ(P t wg +η d P t sg ) is weighted to take into account both the stability of energy storage and the benefits of wind power storage stations, that is, to ensure system security and economy as much as possible under limited information. Parameter Q t =E t -η is a virtual queue built based on energy storage capacity, where η is a constant; parameter V is also a constant. Regarding the values ​​of η and V, it can be proved that when η=E l +τP u +Vπ max ,and When all the constraints in the original operation optimization problem can be satisfied, especially the amount of energy storage can be maintained at [E l ,E u ] interval. Therefore, the online scheduling strategy can obtain a better result while ensuring the operation safety of the system.

[0138] It should be further explained that the Lyapunov optimization problem described above does not contain any temporal coupling terms and only requires making decisions based on the state at the current time t. Therefore, it is an online decision-making paradigm. Furthermore, the problem is a linear programming problem and is easy to solve.

[0139] Furthermore, the basic online scheduling strategy is used to schedule the wind power energy storage station and obtain basic operating data specifically as follows:

[0140] A basic online dispatching strategy is used to dispatch the wind power energy storage station, obtaining the state and decision variables at each moment, and obtaining a series of state-action data;

[0141] Calculate the corresponding reward based on the state-action data and establish a data quadruple as the basic running data.

[0142] In this embodiment, after a period of operation, the basic online scheduling strategy derived from the Lyapunov optimization problem accumulates historical operating data, allowing for training and optimization of the scheduling strategy to further improve the operational profitability of the wind energy storage station. DDPG, as an efficient reinforcement learning algorithm, can be applied to scenarios where the control variable is a continuous variable and is suitable for the online operation of wind energy storage stations.

[0143] The DDPG algorithm, initialized using the Lyapunov optimization problem, fully leverages the advantages of both Lyapunov optimization and the DDPG algorithm. Traditional reinforcement learning algorithms typically randomly select actions to obtain samples during the initial training phase. This approach is highly random and requires a large number of samples to achieve optimal results. Furthermore, the safety of random actions is difficult to guarantee and may violate certain constraints. In contrast, the DDPG algorithm, initialized using the Lyapunov optimization problem, allows the agent to quickly learn the Lyapunov optimization scheduling pattern early in training, ensuring the system's safe online operation. Compared to the random selection model, it offers significant advantages in terms of convergence and safety. Furthermore, as a reinforcement learning algorithm, the DDPG algorithm can fully exploit historical data when using a large sample size. Based on a learning model of exploration and exploitation, it can spontaneously optimize scheduling strategies, improve the efficiency of wind-storage power plants, and achieve superior results compared to Lyapunov optimization in later stages.

[0144] Specifically, as a reinforcement learning algorithm, the DDPG algorithm’s corresponding training data is usually in the form of a quadruple (s t ,a t ,r t ,s t+1 ), where s t ,a t ,r t Represent the current state, action, and reward value respectively, and s t+1 Action a is executed t The state at the next moment. In this problem, the state s t =(E t ,P t w ,π t ), and the decision quantity a t full =(P t ws ,P t wg ,P t sg ). Considering a t full The dimension is high, and each action component can be represented by the net energy storage charging action P t c It is derived that the decision quantity can be reduced to obtain action a t =P t c ∈[-P u ,P u ], and the decision quantity P t ws ,P t wg ,P t sg About P t c Function:

[0145]

[0146] P t wg =P t w -P t ws

[0147]

[0148] Therefore, once action a t Once determined, each decision-making quantity in the wind power energy storage station can be directly calculated.

[0149] The reward function r can bet It is defined as:

[0150] r t =π t rel (η d P t sg +P t wg )τ

[0151] That is, the income obtained by the power station discharging to the grid. t rel =π t -π av is the relative electricity price at time t, π av Represents the average electricity price. If the absolute electricity price is used as the calculation method for benefits, then due to the short-sighted nature of reinforcement learning, it is more inclined to use energy storage to discharge as much as possible to obtain short-term benefits. In contrast, using relative electricity prices to calculate rewards reflects the impact of real-time electricity price fluctuations on long-term benefits, allowing energy storage to discharge when electricity prices are high and charge when electricity prices are low. It is also worth mentioning that since the application scenario is a newly built power station, its historical information may be less, so the average electricity price is not a known quantity. In this case, an estimated value of the average electricity price can be obtained based on the sliding average method. Instead of π av , that is, update at each moment Where α is the weight of the sliding average.

[0152] Therefore, the state, action, and reward functions in the DDPG algorithm are as follows:

[0153] s t =(E t ,P t w ,π t )

[0154] a t =P t c ∈[-P u ,P u ]

[0155] r t =π t rel (η d P t sg +P t wg )τ

[0156] Thus, after each action is executed, a quaternion (s t ,a t ,rt ,s t+1 ) and save it as basic operating data.

[0157] Furthermore, the advanced scheduling model of the wind power energy storage station is constructed based on the DDPG reinforcement learning algorithm, specifically:

[0158] An advanced scheduling model for wind power energy storage stations is constructed based on the DDPG reinforcement learning algorithm, including an actor network, a critic network, an actor target network, and a critic target network.

[0159] The Actor network is used to generate a decision-making strategy, that is, after inputting a state, it will output the corresponding action and generate the Actor target network update parameters;

[0160] The critic network is used to evaluate the decision-making strategy, that is, after inputting the state and action, it will output the corresponding score and generate the critic target network update parameters;

[0161] The Actor target network is used to generate a simulated real decision strategy based on the updated parameters and input status of the Actor target network;

[0162] The critic network is used to update parameters and input status of the critic target network, simulate the real decision strategy for evaluation, and generate a simulated real score;

[0163] In the advanced scheduling model of the wind power energy storage station, the status includes the energy storage capacity, maximum output and real-time electricity price; the actions include the power of energy storage charging, the power directly transmitted to the receiving grid, and the power transmitted from the energy storage station to the receiving grid; the score is the long-term benefit brought by the energy storage charging and discharging actions; and the simulated real score is the real long-term benefit brought by the simulated energy storage charging and discharging actions.

[0164] It should be noted that the long-term benefits brought by the energy storage charging and discharging actions are the "single-step benefits brought by charging and discharging the electric vehicle at the moment" plus the "future benefits that the energy storage electricity after the charging and discharging actions can bring (i.e. the economic value of the energy storage electricity).

[0165] In this embodiment, the advanced scheduling model of the wind power energy storage station can be found in Figure 3 , which is mainly composed of two neural networks, the Actor network and the Critic network, where the parameters of the Actor network are denoted as θ μ , the parameters of the Critic network are recorded as θ Q The Actor network gives a strategy a t =μ(s t |θ μ ), that is, the input state s tThe Actor network will then output the corresponding action a t ; The Critic network evaluates the strategy, that is, the input state s t and action a t The Critic network will then output the corresponding score Q(s t ,a t ). Both networks use the data in the sample pool (s t ,a t ,r t ,s t+1 ), training is performed based on the back-propagation algorithm, and network parameters are continuously updated. The objective function of the Actor network is to maximize the Q value, that is:

[0166]

[0167] The goal of the Critic network is to minimize its predicted Q value Q pred (s,μ(s|θ μ )|θ Q ) and the true Q value Q actual The error between them is:

[0168]

[0169] Among them, the calculation method of the predicted Q value is:

[0170] Q pred (s t ,a t |θ Q )=r t +γQ(s t+1 ,a t+1 |θ Q )

[0171] Where γ∈(0,1] is the discount factor, which needs to be given in advance; the state s t+1 is in state s t Next, perform action a t The action a generated in the training phase is obtained. t (and a t+1 ) is often the action μ(s|θ) output by the Actor network μ ) based on the Gaussian random noise N(0,σ 2 ) is obtained as shown in the following formula. Its purpose is to add a certain amount of randomness to the action in order to try to explore a better strategy.

[0172] a t =μ(s t |θ μ )+N(0,σ 2 )

[0173] In addition to the real-time updated Actor network and Critic network, the DDPG algorithm also requires a set of slowly updated Actor target network and Critic target network, such as Figure 3 As shown, its role is to simulate the real Q value Q actual ,Right now

[0174]

[0175] in are the parameters of the Actor target network and the Critic target network, which are updated by soft update, namely

[0176]

[0177]

[0178] The hyperparameter α∈(0,1) is the soft update weight. Usually, the value of α is small, so the update of the Actor target network and the Critic target network is slow to ensure training stability.

[0179] Therefore, the training of the advanced scheduling model is essentially to continuously train the Actor network and the Critic network using basic operation data, based on the objective function L μ / L Q And back propagation algorithm to update network parameters θ in real time μ / θ Q ; The Actor target network and the Critic target network use soft updates to adjust the parameters Perform slower updates.

[0180] Furthermore, the advanced scheduling model is trained according to the basic operation data until the advanced scheduling model meets the preset evaluation criteria, and the advanced online scheduling strategy is obtained, specifically:

[0181] The basic operation data is a data quadruple, including state, action, reward and state at the next moment;

[0182] Based on basic operation data, the back propagation algorithm is used to train the Actor network and Critic network, continuously update the network parameters, and obtain the current scheduling strategy;

[0183] The training goal of the Actor network is to maximize the energy storage capacity; the goal of the Critic network is to minimize the error between the energy storage capacity and the simulated actual energy storage capacity.

[0184] A judgment is made based on the profit of the wind power energy storage station under the current scheduling strategy and the profit under the basic online scheduling strategy. If the preset evaluation criteria are met, the current scheduling strategy is used as the advanced online scheduling strategy. Otherwise, the Actor network and Critic network continue to be trained.

[0185] This embodiment specifically describes the training process for the advanced scheduling model. Specifically, in the early stage of power plant operation, the basic scheduling strategy is executed and the state and decision quantities at each moment are recorded. Naturally, a series of state-action data (s t ,a t ), t=0,1,2,..., and based on these data, the corresponding reward value r can also be calculated t , thus continuously building (s t ,a t ,r t ,s t+1 ) quadruple and add it to the sample pool for DDPG model training. When the DDPG model is trained sufficiently, the DDPG model can be used as an advanced scheduling strategy to directly perform online scheduling based on the output actions of the Actor network, replacing the basic scheduling strategy to support the online operation of the power station, such as Figure 4 As shown in Figure 2, the DDPG model, as an advanced strategy, not only learns the decision-making patterns of the basic scheduling strategy but also performs spontaneous exploration and optimization based on historical data, bringing more benefits than the basic scheduling strategy.

[0186] It should be noted that this embodiment addresses another key issue: when to switch between the basic scheduling strategy and the advanced scheduling strategy, that is, when can the DDPG model be considered to be fully trained and its performance has exceeded that of the Lyapunov optimization model? To address this issue, this embodiment uses preset evaluation criteria for comparison and judgment, specifically:

[0187] Set the calculation interval. In this embodiment, each operation day is used as a calculation interval. That is, at the end of each operation day, the performance of DDPG is evaluated based on the following evaluation algorithm. If the DDPG algorithm model meets the evaluation criteria for M consecutive operation days, it is considered that the performance of DDPG is sufficient to support online operation. At this time, the online operation strategy can be switched from the basic scheduling strategy to the advanced scheduling strategy, such as Figure 5 As shown in Figure 2. M is a hyperparameter, and its value must be specified in advance.

[0188] Step 1: Given the values ​​of hyperparameters K and β%;

[0189] Step 2: Taking the operation day as the cycle, randomly select K complete operation days of historical data from the basic operation data, including wind power output and electricity price data at each moment.

[0190] Step 3: Based on the selected historical data, run simulations for each complete operating day using both the basic scheduling strategy and the current DDPG strategy, and calculate the daily returns obtained under the two strategies.

[0191] Step 4: Compare the daily returns of the two strategies. If at least β% of the complete running days in the simulation meet the requirement that the daily return of the DDPG strategy is higher than the daily return of the basic scheduling strategy, then the evaluation passes, that is, the DDPG strategy is considered to be superior to the basic strategy in this evaluation; otherwise, the evaluation fails.

[0192] The evaluation criteria preset in this embodiment constitute an online operation algorithm for a wind farm energy station based on basic and advanced scheduling strategies, and describe the criteria for switching from a basic strategy to an advanced strategy. This online operation algorithm can provide effective guidance for the real-time control of wind farm energy stations.

[0193] This embodiment provides a real-time control method for a wind energy storage station. The proposed basic online scheduling strategy can ensure the safe operation of the wind energy storage station. The advanced scheduling model for the wind energy storage station, constructed using the DDPG reinforcement learning algorithm, is trained based on basic operating data obtained from the basic online scheduling strategy. This can further optimize the online operating efficiency of the wind energy storage station and provide a reference for the online operation of the wind energy storage station.

[0194] See Figure 6 This embodiment provides a real-time control system for a wind power energy storage station, which is used to implement a real-time control method for a wind power energy storage station. The system specifically includes an online operation module, a Lyapunov optimization module, a basic scheduling module, a DDPG algorithm module, a training and evaluation module, and an advanced scheduling module.

[0195] The online operation module is used to construct an online operation model of the wind power energy storage station;

[0196] The Lyapunov optimization module is used to optimize the online operation model based on the Lyapunov optimization method to obtain the basic online scheduling strategy;

[0197] The basic scheduling module is used to schedule the wind power energy storage station using a basic online scheduling strategy and obtain basic operating data;

[0198] The DDPG algorithm module is used to build an advanced scheduling model for wind power storage stations based on the DDPG reinforcement learning algorithm;

[0199] The training and evaluation module is used to train the advanced scheduling model according to the basic operation data until the advanced scheduling model meets the preset evaluation criteria and obtains the advanced online scheduling strategy;

[0200] The advanced scheduling module is used to regulate the wind power energy storage station using an advanced online scheduling strategy.

[0201] Furthermore, the Lyapunov optimization module is used to optimize the online operation model based on the Lyapunov optimization method to obtain the basic online scheduling strategy, specifically:

[0202] Based on the Lyapunov optimization method, the online running model optimization at time t is modeled as a Lyapunov optimization problem:

[0203] minτQ t (P t ws -P t sg )+V[-π t τ(Pt wg +η d P t sg )]

[0204] sT t ws ≥0, P t wg ≥0, P t sg ≥0

[0205]

[0206] P t wg +η d P t sg ≤T u

[0207] P t ws ≤P u , P t sg ≤P u

[0208] Where: τ represents the time interval between each decision; Q t It is a virtual queue built based on energy storage capacity, with Q t =E t -η, E t represents the energy storage capacity at time t, η is a constant; P t ws is the wind power used for energy storage charging; P t wg is the wind power directly transmitted to the receiving grid; V is a constant; π t represents the real-time electricity price; η dis the discharge efficiency of energy storage; P t sg The power transmitted from the energy storage station to the receiving grid; η c is the charging efficiency of energy storage; P t w represents the maximum wind power output at time t; T u is the transmission line capacity; P u is the maximum charge and discharge power of energy storage;

[0209] Solve the Lyapunov optimization problem and obtain basic online scheduling strategies.

[0210] Furthermore, the basic scheduling module is used to schedule the wind power energy storage station using a basic online scheduling strategy and obtain basic operating data, specifically:

[0211] A basic online dispatching strategy is used to dispatch the wind power energy storage station, obtaining the state and decision variables at each moment, and obtaining a series of state-action data;

[0212] Calculate the corresponding reward based on the state-action data and establish a data quadruple as the basic running data.

[0213] Furthermore, the DDPG algorithm module is used to construct an advanced scheduling model for wind power energy storage stations based on the DDPG reinforcement learning algorithm, specifically:

[0214] An advanced scheduling model for wind power energy storage stations is constructed based on the DDPG reinforcement learning algorithm. It includes an actor network, a critic network, an actor target network, and a critic target network.

[0215] The Actor network is used to generate a decision-making strategy, that is, after inputting a state, it will output the corresponding action and generate the Actor target network update parameters;

[0216] The critic network is used to evaluate the decision-making strategy, that is, after inputting the state and action, it will output the corresponding score and generate the critic target network update parameters;

[0217] The Actor target network is used to generate a simulated real decision strategy based on the updated parameters and input status of the Actor target network;

[0218] The critic network is used to update parameters and input status of the critic target network, simulate the real decision strategy for evaluation, and generate a simulated real score;

[0219] In the advanced scheduling model of the wind power energy storage station, the status includes the energy storage capacity, maximum output and real-time electricity price; the actions include the power of energy storage charging, the power directly transmitted to the receiving grid, and the power transmitted from the energy storage station to the receiving grid; the score is the long-term benefit brought by the energy storage charging and discharging actions; and the simulated real score is the real long-term benefit brought by the simulated energy storage charging and discharging actions.

[0220] Furthermore, the training and evaluation module is used to train the advanced scheduling model based on the basic operation data until the advanced scheduling model meets the preset evaluation criteria and obtain the advanced online scheduling strategy, specifically:

[0221] The basic operation data is a data quadruple, including state, action, reward and state at the next moment;

[0222] Based on basic operation data, the back propagation algorithm is used to train the Actor network and Critic network, continuously update the network parameters, and obtain the current scheduling strategy;

[0223] The training goal of the Actor network is to maximize the energy storage capacity; the goal of the Critic network is to minimize the error between the energy storage capacity and the simulated actual energy storage capacity.

[0224] A judgment is made based on the profit of the wind power energy storage station under the current scheduling strategy and the profit under the basic online scheduling strategy. If the preset evaluation criteria are met, the current scheduling strategy is used as the advanced online scheduling strategy. Otherwise, the Actor network and Critic network continue to be trained.

[0225] This embodiment provides a method and device for real-time control of a wind power energy storage station, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for real-time control of a wind power energy storage station is implemented.

[0226] This embodiment provides a storage medium, which includes a stored computer program. When the computer program is executed, the device where the storage medium is located is controlled to execute the method for real-time control of a wind power energy storage station.

[0227] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A real-time control method for a wind power energy storage station, characterized in that: The following steps are involved: Build an online operation model of a wind power energy storage station; Optimize the online operation model based on the Lyapunov optimization method to obtain the basic online scheduling strategy; Use basic online dispatching strategies to dispatch wind power storage stations and obtain basic operating data; An advanced scheduling model for wind power energy storage stations is constructed based on the DDPG reinforcement learning algorithm, including an actor network, a critic network, an actor target network, and a critic target network. The Actor network is used to generate a decision-making strategy, that is, after inputting a state, it will output the corresponding action and generate the Actor target network update parameters; The critic network is used to evaluate the decision-making strategy, that is, after inputting the state and action, it will output the corresponding score and generate the critic target network update parameters; The Actor target network is used to generate a simulated real decision strategy based on the updated parameters and input status of the Actor target network; The critic network is used to update parameters and input status of the critic target network, simulate the real decision strategy for evaluation, and generate a simulated real score; In the advanced scheduling model of the wind power energy storage station, the state includes the energy storage capacity, maximum output, and real-time electricity price; the action includes the power of energy storage charging, the power directly transmitted to the receiving grid, and the power transmitted from the energy storage station to the receiving grid; the score is the long-term benefit brought by the energy storage charging and discharging action; the simulated real score is the real long-term benefit brought by the simulated energy storage charging and discharging action; The advanced scheduling model is trained based on the basic operation data until it meets the preset evaluation criteria and the advanced online scheduling strategy is obtained. Specifically: The basic operation data is a data quadruple, including state, action, reward and state at the next moment; Based on basic operation data, the back propagation algorithm is used to train the Actor network and Critic network, continuously update the network parameters, and obtain the current scheduling strategy; The training goal of the Actor network is to maximize the long-term benefits of energy storage charging and discharging actions; the goal of the Critic network is to minimize the error between the long-term benefits of energy storage and the simulated real long-term benefits of energy storage. The wind power storage station's revenue under the current scheduling strategy is judged against the revenue under the basic online scheduling strategy. If the preset evaluation criteria are met, the current scheduling strategy is used as the advanced online scheduling strategy. Otherwise, the Actor network and Critic network continue to be trained. An advanced online dispatching strategy is used to regulate the wind power storage station.

2. A real-time control method for a wind power energy storage station according to claim 1, characterized in that: The online operation model is optimized based on the Lyapunov optimization method to obtain the basic online scheduling strategy, specifically: Based on the Lyapunov optimization method, the online running model optimization at time t is modeled as a Lyapunov optimization problem: Where: τ represents the time interval between each decision; Q t It is a virtual queue built based on energy storage capacity, with Q t =E t -η, E t represents the energy storage capacity at time t, η is a constant; P t ws is the wind power used for energy storage charging; P t wg is the wind power directly transmitted to the receiving grid; V is a constant; π t represents the real-time electricity price; η d is the discharge efficiency of energy storage; P t sg The power transmitted from the energy storage station to the receiving grid; η c is the charging efficiency of energy storage; P t w represents the maximum wind power output at time t; T u is the transmission line capacity; P u is the maximum charge and discharge power of energy storage; Solve the Lyapunov optimization problem and obtain basic online scheduling strategies.

3. A real-time control method for a wind power energy storage station according to claim 1, characterized in that: The basic online scheduling strategy is used to schedule the wind power energy storage station and obtain basic operating data as follows: A basic online dispatching strategy is used to dispatch the wind power energy storage station, obtaining the state and decision variables at each moment, and obtaining a series of state-action data; Calculate the corresponding reward based on the state-action data and establish a data quadruple as the basic running data.

4. A real-time control system for a wind power energy storage station, characterized in that: It includes online operation module, Lyapunov optimization module, basic scheduling module, DDPG algorithm module, training evaluation module and advanced scheduling module; among them: The online operation module is used to construct an online operation model of the wind power energy storage station; The Lyapunov optimization module is used to optimize the online operation model based on the Lyapunov optimization method to obtain the basic online scheduling strategy; The basic scheduling module is used to schedule the wind power energy storage station using a basic online scheduling strategy and obtain basic operating data; The DDPG algorithm module is used to build an advanced scheduling model for wind power energy storage stations based on the DDPG reinforcement learning algorithm, including an actor network, a critic network, an actor target network, and a critic target network, where: The Actor network is used to generate a decision-making strategy, that is, after inputting a state, it will output the corresponding action and generate the Actor target network update parameters; The critic network is used to evaluate the decision-making strategy, that is, after inputting the state and action, it will output the corresponding score and generate the critic target network update parameters; The Actor target network is used to generate a simulated real decision strategy based on the updated parameters and input status of the Actor target network; The critic network is used to update parameters and input status of the critic target network, simulate the real decision strategy for evaluation, and generate a simulated real score; In the advanced scheduling model of the wind power energy storage station, the state includes the energy storage capacity, maximum output, and real-time electricity price; the action includes the power of energy storage charging, the power directly transmitted to the receiving grid, and the power transmitted from the energy storage station to the receiving grid; the score is the long-term benefit brought by the energy storage charging and discharging action; the simulated real score is the real long-term benefit brought by the simulated energy storage charging and discharging action; The training and evaluation module is used to train the advanced scheduling model based on the basic operation data until the advanced scheduling model meets the preset evaluation criteria and obtains the advanced online scheduling strategy; specifically: The basic operation data is a data quadruple, including state, action, reward and state at the next moment; Based on basic operation data, the back propagation algorithm is used to train the Actor network and Critic network, continuously update the network parameters, and obtain the current scheduling strategy; The training goal of the Actor network is to maximize the long-term benefits of energy storage charging and discharging actions; the goal of the Critic network is to minimize the error between the long-term benefits of energy storage and the simulated real long-term benefits of energy storage. The wind power storage station's revenue under the current scheduling strategy is judged against the revenue under the basic online scheduling strategy. If the preset evaluation criteria are met, the current scheduling strategy is used as the advanced online scheduling strategy. Otherwise, the Actor network and Critic network continue to be trained. The advanced scheduling module is used to regulate the wind power energy storage station using an advanced online scheduling strategy.

5. A real-time control system for a wind power energy storage station according to claim 4, characterized in that: The Lyapunov optimization module is used to optimize the online operation model based on the Lyapunov optimization method to obtain the basic online scheduling strategy, specifically: Based on the Lyapunov optimization method, the online running model optimization at time t is modeled as a Lyapunov optimization problem: Where: τ represents the time interval between each decision; Q t It is a virtual queue built based on energy storage capacity, with Q t =E t -η, E t represents the energy storage capacity at time t, η is a constant; P t ws is the wind power used for energy storage charging; P t wg is the wind power directly transmitted to the receiving grid; V is a constant; π t represents the real-time electricity price; η d is the discharge efficiency of energy storage; P t sg The power transmitted from the energy storage station to the receiving grid; η c is the charging efficiency of energy storage; P t w represents the maximum wind power output at time t; T u is the transmission line capacity; P u is the maximum charge and discharge power of energy storage; Solve the Lyapunov optimization problem and obtain basic online scheduling strategies.

6. A real-time control system for a wind power energy storage station according to claim 4, characterized in that: The basic scheduling module is used to schedule the wind power energy storage station using a basic online scheduling strategy and obtain basic operating data, specifically: A basic online dispatching strategy is used to dispatch the wind power energy storage station, obtaining the state and decision variables at each moment, and obtaining a series of state-action data; Calculate the corresponding reward based on the state-action data and establish a data quadruple as the basic running data.

Images