Decision-making devices and methods for offshore wind power to participate in the spot electricity market with energy storage configuration

Abstract

The application discloses a device and method for decision-making of off-shore wind power configuration energy storage participating in spot power market, comprising a data acquisition module, a virtual environment module, an algorithm model module and a test update module, the data acquisition module is used to collect the basic operation information of off-shore wind power configuration water storage energy storage system and the dispatching information provided by the power grid; the virtual environment module is used to construct the operation environment of off-shore wind power configuration energy storage system according to the short-term power dispatching demand of the power grid department and form a virtual environment; the algorithm model module is used to make mathematical modeling of off-shore wind power configuration energy storage system according to the basic operation information, and construct a deep double Q network model based on the system model for training intelligent agents to make decisions and participate in short-term spot power market transactions; the test update module is used to evaluate the training effect of the current deep double Q network model and update the parameters until the output of the decision-making model meets the expected set evaluation standard. The model and method improve the operation efficiency and income of the system.

Description

Technical Field

[0001] This invention relates to the field of electricity market technology, and in particular to a decision-making device and method for offshore wind power to participate in the spot electricity market with energy storage. Background Technology

[0002] Offshore wind power is a renewable energy technology that utilizes offshore wind energy resources. It boasts advantages such as abundant wind resources, stable wind speeds, small land footprint, and minimal environmental impact. However, offshore wind power faces challenges including fluctuating power output, difficulties in grid connection, and high operation and maintenance costs. Especially in the context of electricity market liberalization, the economic viability and competitiveness of offshore wind power have been challenged. Hydroelectric energy storage is an energy storage technology that utilizes hydroelectric generators for charging and discharging. It offers advantages such as large storage capacity, long lifespan, and high efficiency. Hydroelectric energy storage can complement and coordinate with wind power by smoothing wind power fluctuations, providing frequency regulation and peak shaving services, and participating in electricity market transactions. However, traditional hydroelectric energy storage power stations face challenges such as site selection limitations and low water resource utilization efficiency.

[0003] Offshore wind power with hydroelectric energy storage involves constructing hydroelectric energy storage power stations in the sea area near offshore wind farms, utilizing seabed topography and artificial structures to create an integrated system of offshore wind power and hydroelectric energy storage. Seawater is inexhaustible, and using it as an energy storage medium can significantly reduce the construction and operation costs of hydroelectric energy storage power stations. However, existing offshore wind power hydroelectric energy storage systems often rely on thermal power units as auxiliary power suppliers for stable power supply, failing to consider the volatility of power system demand after electricity market liberalization, and the application scenarios for offshore wind power hydroelectric energy storage systems to participate in electricity market transactions. Hydroelectric energy storage can perform charging and discharging operations according to the power fluctuations of offshore wind power and changes in market prices, thereby smoothing the output power of offshore wind power and improving the power quality and market competitiveness of offshore wind power. Therefore, to improve the market participation capability of offshore wind power with energy storage systems, reduce their impact on the grid, and increase their profitability, appropriate dispatch and trading strategies are needed. Summary of the Invention

[0004] To address the challenges of establishing models for offshore wind-powered hydroelectric energy storage systems and the low participation and profitability of these systems in the electricity market in existing technologies, this invention provides a decision-making device and method for offshore wind power-based energy storage systems participating in the spot electricity market. This enables intelligent decision-making and trading for offshore wind power-based hydroelectric energy storage systems, improving system operating efficiency and profitability, reducing operating costs and risks, and enhancing system adaptability and robustness. The specific technical solution is as follows:

[0005] The first aspect of this invention provides a device for configuring energy storage in offshore wind power to participate in spot electricity market decision-making, comprising:

[0006] The data acquisition module is used to collect basic operating information of the offshore wind power configured with water storage energy storage system and dispatch information provided by the power grid. The basic operating information includes the energy storage capacity of the offshore wind power configured with water storage energy storage system and the capacity balance of water storage energy storage and its upper and lower physical constraints.

[0007] The virtual environment module is used to construct and form a virtual environment for the operation of offshore wind power energy storage systems based on the short-term power dispatch needs of the power grid department.

[0008] The algorithm model module performs mathematical modeling of the offshore wind power energy storage system based on the aforementioned basic operational information, and constructs a deep double-Q network model based on the system model to train the agent to make decisions and participate in short-term spot electricity market transactions; and,

[0009] The test update module is used to evaluate the training effect of the current deep double-Q network model and update the parameters until the model output meets the expected evaluation criteria.

[0010] Preferably, the energy storage capacity The formula for determining it is as follows:

[0011]

[0012] in, represent The energy storage capacity of the offshore wind power configuration with hydroelectric energy storage system is specified at the time. represent Time of the first The capacity of the aforementioned water storage power station, Representing the upper reservoir Power station Water storage capacity at all times This represents the maximum water storage capacity of the upper reservoir.

[0013] Preferably, the formulas for determining the capacity balance and upper and lower physical constraints of the water storage energy are as follows:

[0014] ,

[0015]

[0016] in, and Representing the first Power station The water storage levels of the upper and lower reservoirs at all times. This represents the self-loss caused by reservoir leakage or evaporation. and Representing the first The power station Maximum pumping volume and maximum discharge volume within a given time period and These represent charging and discharging commands for the energy storage power station, respectively.

[0017] ,

[0018]

[0019] in, and These represent the maximum water storage capacity of the upper and lower reservoirs, respectively. and These represent the minimum water storage capacity of the upper and lower reservoirs, respectively.

[0020] Preferably, , .

[0021] Preferably, the system energy exchange determination formula for the operating environment is as follows:

[0022] ,

[0023]

[0024] in, and Representing the first Each power station Maximum pumping volume and maximum discharge volume within a given time period and Representing the first The maximum power of the pumps and turbines in the energy storage power station Represents the pumping coefficient. Represents the power generation coefficient;

[0025] ,

[0026]

[0027] in, , and These represent the efficiencies of the pumps, motors, and pipelines in the energy storage power station, respectively. , and These represent the efficiency of the turbine, generator, and pipeline, respectively.

[0028] Preferably, the formula for determining the system penalty of the operating environment is as follows:

[0029] ,

[0030] ,

[0031] ,

[0032] ,

[0033] ,

[0034] in, The representative system is The total penalty at any moment This represents the penalty coefficient for exceeding the limit. and These represent the expected maximum and minimum power of the power grid system, respectively. Represents the deviation penalty coefficient. Representing the power grid sector The system's power generation is expected to be constant. Representing the power grid sector The difference between the actual power generation and the expected power generation of the system is allowed at all times. Represents the maintenance penalty coefficient. Representing the One fan in The workload requires constant maintenance by the repair team. On behalf of the maintenance team The time has reached the 10th One fan.

[0035] A second aspect of this invention provides a method for decision-making regarding offshore wind power with energy storage participating in the spot electricity market. The method utilizes the aforementioned offshore wind power with energy storage decision-making device, and its decision-making steps are as follows:

[0036] S1: Mathematical modeling of offshore wind power energy storage system is performed based on the basic operation information of offshore wind power station and water storage energy station. The basic operation information includes the total power of offshore wind power station, the planned workload of maintenance team, the reservoir volume of water storage energy station, the power and efficiency of pumps and motors, turbines and generators of energy storage energy station, and pipeline losses.

[0037] S2: Construct an operating environment for offshore wind power configuration energy storage system and form a virtual environment based on the short-term power dispatch needs of the power grid department. The power dispatch needs include the expected system power generation, the maximum allowable deviation, and the maximum and minimum allowable system power. The operating environment needs to consider the load limiting of the system's active devices and the load limiting of the power grid, including the pumps and motors, turbines and generators of the energy storage power station, and the maximum and minimum expected system power of the power grid.

[0038] S3: Based on the virtual environment of the system, a deep double-Q network model is used to train an agent to make decisions and participate in short-term spot electricity market transactions. The deep double-Q network model includes the agent, the virtual environment, the state space, the action space, and the reward function.

[0039] S4: Update the deep reinforcement learning model based on the training results and evaluation metrics, and use the tested strategies to determine the trading strategies for offshore wind power with energy storage systems participating in the short-term electricity spot market.

[0040] Preferably, step S3 includes the following steps:

[0041] S31: Establish a deep dual-Q neural network consisting of a Q Network, a Target Q Network, and a Replay buffer, and initialize the network according to the system model. The Q Network and the Target Q Network have completely identical structures. The Replay buffer unit is used to record four pieces of information: the agent's current state, action, reward, and state at the next moment.

[0042] S32: The agent in Moments based on Q Network from action space Select an action, submit the current action to the virtual environment, and calculate the benefit based on the feedback from the virtual environment. And record the state of the agent during this action. ,action ,award and the state at the next moment As a vector group Stored in the Replay buffer, where , and Representing the intelligent agent in The state space, action space, and reward at any given moment. Representing the intelligent agent in The state space at time step 1, where the reward function is... Representing the intelligent agent in Perform actions at all times Become a state Instant rewards;

[0043] S33: When the data stored in the Replay buffer reaches a set value, a batch of sample data is randomly extracted from it for Q Network parameter training, and the parameters are updated by the optimizer according to the principle of minimizing the loss function. The randomly extracted batch of sample data is obtained by a Gaussian distribution sampler, and the optimizer is a stochastic gradient descent optimizer.

[0044] Preferably, the state space is determined by the following set of vectors:

[0045]

[0046] in, The representative system is The state space at any given moment, represent Time of the first The capacity of a single hydroelectric energy storage power station, This represents the system in The total power transmitted to the power grid at all times. Representing the One fan in The workload requires constant maintenance by the repair team;

[0047] The action space is determined by the following set of vectors:

[0048]

[0049] in, The representative system is The space of action at any moment and These represent charging and discharging commands for the energy storage power station, respectively. The grid connection coefficient represents the grid connection factor of offshore wind power plants. and Representing the maintenance team The time has reached the 10th One wind turbine and heading to the first On the way to the wind turbine;

[0050] The formula for determining the reward function is as follows:

[0051]

[0052] in, Representing the intelligent agent in Momentary rewards The representative system is The electricity price at any given time, This represents the system in The total power transmitted to the power grid at all times. and Representing the first The maximum power of the pumps and turbines in the energy storage power station Representing the Energy storage aging coefficient of an energy storage power station Representing the One fan in The workload requires constant maintenance by the repair team. This represents the labor remuneration coefficient for wind power station maintenance personnel. The representative system is The total penalty at any moment Represents other operation and maintenance costs, This represents the safety factor of the energy storage system.

[0053] Preferably, the following steps are included after step S33:

[0054] S41: Periodically update the parameters of the Q Network to the Target Q Network;

[0055] S42: Determine whether the network output meets the expected evaluation criteria; if yes, end the training; if not, continue to update and iterate the deep reinforcement learning network from step S32.

[0056] And / or, the formula for determining the loss function in the evaluation criteria is as follows:

[0057]

[0058] in, For loss function, Representing the intelligent agent in Momentary rewards Represents the discount factor. Representing Target Q Network Moment State Next, use a greedy strategy to select actions. The corresponding Q value, The parameters representing the Q Network, The parameters represent the Target Q Network.

[0059] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0060] This model and method fully utilize the characteristics and advantages of offshore wind power and hydroelectric energy storage to construct an integrated system for offshore wind power with hydroelectric energy storage. Considering the uncertainties of wind power, grid demand and market prices, a deep double-Q network learning method is adopted to realize intelligent decision-making and trading of offshore wind power with hydroelectric energy storage, thereby improving the system's operating efficiency and revenue, reducing the system's operating costs and risks, and improving the system's adaptability and robustness. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the offshore wind power energy storage configuration device module for participating in spot electricity market decision-making according to the present invention;

[0062] Figure 2 This is a schematic diagram of the data acquisition module in the method for configuring energy storage in offshore wind power to participate in spot electricity market decision-making in this invention;

[0063] Figure 3 This is a schematic diagram of the test and update module in the method for configuring energy storage in offshore wind power to participate in the spot electricity market decision-making process of the present invention;

[0064] Figure 4 This is a schematic diagram of the overall process of the method for offshore wind power with energy storage to participate in the spot electricity market decision-making process according to the present invention;

[0065] Figure 5 This is a schematic diagram of the algorithm model module in the decision-making method for offshore wind power with energy storage participating in the spot electricity market of the present invention;

[0066] Figure 6 This is a schematic diagram of the test and update module process in the offshore wind power configuration energy storage participation in the spot electricity market decision-making method of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0069] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0070] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0071] To resolve the above issues, please refer to Figure 1As shown, the first aspect of this invention provides a decision-making device for offshore wind power-configured energy storage participating in the spot electricity market, including a data acquisition module, a virtual environment module, an algorithm model module, and a test and update module. The data acquisition module is used to collect basic operating information of the offshore wind power-configured hydroelectric energy storage system and dispatch information provided by the power grid. The basic operating information includes the energy storage capacity of the offshore wind power-configured hydroelectric energy storage system and the capacity balance of hydroelectric energy storage, as well as its upper and lower physical constraints. The virtual environment module is used to construct the operating environment of the offshore wind power-configured energy storage system and form a virtual environment according to the short-term power dispatch needs of the power grid. The algorithm model module performs mathematical modeling of the offshore wind power-configured energy storage system based on the basic operating information, and constructs a deep double-Q network model based on the system model to train the agent to make decisions and participate in short-term spot electricity market transactions. The test and update module is used to evaluate the current training effect of the deep double-Q network model and update the parameters until the output of the decision model meets the expected evaluation criteria.

[0072] The data acquisition module needs to collect information such as grid dispatch information, real-time capacity of hydroelectric power stations, real-time power transmitted to the grid by the system, and the amount of maintenance work required for wind turbines at each moment. This information helps the virtual environment module, algorithm model module, and test update module optimize the trading strategy of offshore wind power with hydroelectric power systems participating in the spot electricity market. The algorithm model module provides decision-making for the agent in the virtual environment and learns and improves the strategy through the test update module to obtain the optimal strategy for offshore wind power with hydroelectric power systems participating in the spot electricity market.

[0073] In optional embodiments, such as Figure 2 As shown, the energy storage capacity The formula for determining it is as follows:

[0074]

[0075] in, represent The energy storage capacity of the offshore wind power configuration with hydroelectric energy storage system is specified at the time. represent Time of the first The capacity of the aforementioned water storage power station, Representing the upper reservoir Power station Water storage capacity at all times This represents the maximum water storage capacity of the upper reservoir.

[0076] In optional embodiments, such as Figure 2 As shown, the formulas for determining the capacity balance and upper and lower physical constraints of the water storage energy are as follows:

[0077] ,

[0078]

[0079] in, and Representing the first Power station The water storage levels of the upper and lower reservoirs at all times. This represents the self-loss caused by reservoir leakage or evaporation. and Representing the first The power station Maximum pumping volume and maximum discharge volume within a given time period and These represent charging and discharging commands for the energy storage power station, respectively.

[0080] ,

[0081]

[0082] in, and These represent the maximum water storage capacity of the upper and lower reservoirs, respectively. and These represent the minimum water storage capacity of the upper and lower reservoirs, respectively.

[0083] In an optional embodiment, , .

[0084] In optional embodiments, such as Figure 2 As shown, the system energy exchange determination formula for the operating environment is as follows:

[0085] ,

[0086]

[0087] in, and Representing the first Each power station Maximum pumping volume and maximum discharge volume within a given time period and Representing the first The maximum power of the pumps and turbines in the energy storage power station Represents the pumping coefficient. The power generation coefficient is represented by ρ, which represents the density of seawater (1.03 × 10³ kg / m³), g represents the gravitational acceleration (9.81 m / s²), and h refers to the height from the water surface of the upper or lower reservoir to the inlet and outlet of its pipeline.

[0088] ,

[0089]

[0090] in, , and These represent the efficiency (loss) of the pumps, motors, and pipelines in the energy storage power station, respectively. , and These represent the efficiency (losses) of the turbine, generator, and pipeline, respectively.

[0091] In optional embodiments, such as Figure 2 As shown, the formula for determining the system penalty of the operating environment is as follows:

[0092] ,

[0093] ,

[0094] ,

[0095] ,

[0096] ,

[0097] in, The representative system is The total penalty at any moment This represents the penalty coefficient for exceeding the limit. and These represent the expected maximum and minimum power of the power grid system, respectively. Represents the deviation penalty coefficient. Representing the power grid sector The system's power generation is expected to be constant. Representing the power grid sector The difference between the actual power generation and the expected power generation of the system is allowed at all times. Represents the maintenance penalty coefficient. Representing the One fan in The workload requires constant maintenance by the repair team. On behalf of the maintenance team The time has reached the 10th One fan.

[0098] The second aspect of this invention provides a method for configuring energy storage in offshore wind power to participate in spot electricity market decision-making, such as... Figures 4 to 6As shown, the decision-making steps of the offshore wind power configuration energy storage participation in the spot electricity market decision-making device are as follows:

[0099] S1: Mathematical modeling of offshore wind power energy storage system is performed based on the basic operation information of offshore wind power station and water storage energy station. The basic operation information includes the total power of offshore wind power station, the planned workload of maintenance team, the reservoir volume of water storage energy station, the power and efficiency of pumps and motors, turbines and generators of energy storage energy station, and pipeline losses.

[0100] S2: Construct an operating environment for offshore wind power configuration energy storage system and form a virtual environment based on the short-term power dispatch needs of the power grid department. The power dispatch needs include the expected system power generation, the maximum allowable deviation, and the maximum and minimum allowable system power. The operating environment needs to consider the load limiting of the system's active devices and the load limiting of the power grid, including the pumps and motors, turbines and generators of the energy storage power station, and the maximum and minimum expected system power of the power grid.

[0101] S3: Based on the virtual environment of the system, a deep double-Q network model is used to train an agent to make decisions and participate in short-term spot electricity market transactions. The deep double-Q network model includes the agent, the virtual environment, the state space, the action space, and the reward function.

[0102] S4: Update the deep reinforcement learning model based on the training results and evaluation metrics, and use the tested strategies to determine the trading strategies for offshore wind power with energy storage systems participating in the short-term electricity spot market.

[0103] Preferably, such as Figure 4 and Figure 5 As shown, step S3 includes the following steps:

[0104] S31: Establish a deep dual-Q neural network consisting of a Q Network, a Target Q Network, and a Replay buffer, and initialize the network according to the system model. The Q Network and the Target Q Network have completely identical structures. The Replay buffer unit is used to record four pieces of information: the agent's current state, action, reward, and state at the next moment.

[0105] S32: The agent in Moments based on Q Network from action space Select an action, submit the current action to the virtual environment, and calculate the benefit based on the feedback from the virtual environment. And record the state of the agent during this action. ,action ,award and the state in the next moment As a vector group Stored in the Replay buffer, where , and Representing the intelligent agent in The state space, action space, and reward at any given moment. Representing the intelligent agent in The state space at time step 1, where the reward function is... Representing the intelligent agent in Perform actions at all times Become a state The immediate reward; wherein, in this embodiment, the decision-making model for offshore wind power configuration with energy storage to participate in the spot electricity market is a Markov decision process (MDP). The establishment of this Markov decision process is based on a vector group of four vectors, the vector group being specifically... ,

[0106] S33: When the data stored in the Replay buffer reaches a set value, a batch of sample data is randomly extracted from it for Q Network parameter training, and the parameters are updated by the optimizer according to the principle of minimizing the loss function. The randomly extracted batch of sample data is obtained by a Gaussian distribution sampler, and the optimizer is a stochastic gradient descent optimizer.

[0107] In an optional embodiment, the state is considered continuous over the entire time interval, while the actions are discrete, conforming to the properties of the main power components within the system, wherein the state space is determined by the following set of vectors:

[0108]

[0109] in, The representative system is The state space at any given moment, represent Time of the first The capacity of the aforementioned water storage power station, This represents the system in The total power transmitted to the power grid at all times. Representing the One fan in The workload requires constant maintenance by the repair team;

[0110] The action space is determined by the following set of vectors:

[0111]

[0112] in, The representative system is The space of action at any moment and These represent charging and discharging commands for the energy storage power station, respectively. The grid connection coefficient represents the grid connection factor of offshore wind power plants. and Representing the maintenance team The time has reached the 10th One wind turbine and heading to the first On the way to the wind turbine;

[0113] The formula for determining the reward function is as follows:

[0114]

[0115] in, Representing the intelligent agent in Momentary rewards The representative system is The electricity price at any given time, This represents the system in The total power transmitted to the power grid at all times. and Representing the first The maximum power of the pumps and turbines in the energy storage power station Representing the Energy storage aging coefficient of an energy storage power station Representing the One fan in The workload requires constant maintenance by the repair team. This represents the labor remuneration coefficient for wind power station maintenance personnel. The representative system is The total penalty at any moment Represents other operation and maintenance costs, This represents the safety factor of the energy storage system.

[0116] In optional embodiments, such as Figure 3 , Figures 4 to 6 As shown, the following steps are included after step S33:

[0117] S41: Periodically update the parameters of the Q Network to the Target Q Network;

[0118] S42: Determine whether the network output meets the expected evaluation criteria; if yes, end the training; if not, continue to update and iterate the deep reinforcement learning network from step S32.

[0119] And / or, the formula for determining the loss function in the evaluation criteria is as follows:

[0120]

[0121] in, For loss function, Representing the intelligent agent in Momentary rewards Represents the discount factor. Representing Target Q Network Moment State Next, use a greedy strategy to select actions. The corresponding Q value, The parameters representing the Q Network, The parameters represent the Target Q Network.

[0122] In summary, this model and method fully utilize the characteristics and advantages of offshore wind power and hydroelectric energy storage to construct an integrated system for configuring offshore wind power with hydroelectric energy storage. Considering the uncertainties of wind power, grid demand, and market prices, a deep double-Q network learning method is adopted to realize intelligent decision-making and trading of the offshore wind power-hydroelectric energy storage system, thereby improving the system's operating efficiency and revenue, reducing operating costs and risks, and enhancing the system's adaptability and robustness.

[0123] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the patent application of the present invention. All equivalent changes, substitutions or modifications made within the technical spirit and principles indicated by the present invention should be included within the scope of patent protection covered by the present invention.

Claims

1. An offshore wind farm configuration energy storage participation in spot power market decision device, characterized in that, Comprise: The data collection module is configured to collect basic operation information of the offshore wind power configuration water storage energy storage system and dispatching information provided by the power grid, wherein the basic operation information includes energy storage capacity of the offshore wind power configuration water storage energy storage system and the capacity balance of the water storage energy storage and its upper and lower limit physical constraints; a virtual environment module for constructing an offshore wind power configuration energy storage system operating environment according to the short-term power dispatching demand of the power grid department and forming a virtual environment; an algorithm model module for mathematical modeling of the offshore wind power configuration energy storage system according to the basic operating information, and constructing a deep double Q network model based on the system model for training an intelligent agent to make decisions and participate in short-term spot power market transactions, which includes the following steps: S31: a deep double Q neural network composed of Q Network, Target Q Network and Replay buffer is established, and network initialization is performed according to the system model, the Q Network and the Target Q Network have the same structure, and the Replay buffer unit is used to record the current state, action, reward and next time state of the intelligent agent; S32: The intelligent agent selects an action from the action space based on the Q Network at the time t, submits the current action to the virtual environment, and calculates the return according to the virtual environment feedback , and represent the state space, action space, and reward of the intelligent agent at the time t, respectively the immediate reward of the intelligent agent when performing the action a at the time t to become the state s​​​​​​​​​​​​​​ S33: when the data stored in the Replay buffer reaches a set value, a batch of sample data is randomly extracted therefrom for QNetwork parameter training and parameter updating by an optimizer according to the loss function minimization principle, the batch of sample data is obtained by a Gaussian distribution sampler, and the optimizer is a stochastic gradient descent optimizer; The state space is determined by the following vector group: wherein, a state space of the system at a time instant, a capacity of the water storage energy storage plant at a time instant, a total power transmitted to the grid by the system at a time instant, a workload of the wind turbine to be maintained by the maintenance team at a time instant, and a workload of the wind turbine to be maintained by the maintenance team at a time instant. The action space is determined by the following vector group: wherein, represents the system in the action space at time and represent the charging and discharging instructions of the energy storage power station, respectively, represents the grid connection coefficient of the offshore wind power station, and represent that the maintenance team has arrived at the th wind turbine and is on the way to the th wind turbine at time , respectively; The determination formula of the reward function is as follows: wherein, represents the reward of the agent at time, represents the electricity selling price of the system at time, represents the total power transmitted to the grid by the system at time, and respectively represent the maximum power of the pump and the turbine of the th energy storage station, represents the energy storage aging coefficient of the th energy storage station, represents the workload of the th wind turbine that needs to be maintained by the maintenance team at time, represents the labor remuneration coefficient of the wind power station maintenance personnel, represents the total penalty of the system at time, represents other operation and maintenance costs, represents the safe operation coefficient of the energy storage system; and, a test update module for evaluating the training effect of the deep double Q network model and updating parameters until the output of the decision model meets the expected set evaluation standard.

2. The offshore wind farm configuration energy storage participation in real-time electricity market decision apparatus according to claim 1, characterized in that, The energy storage capacity The determination formula is as follows: wherein, represents the energy storage capacity of the offshore wind farm at the time instant, represents the capacity of the i-th water storage energy station at the time instant, represents the capacity of the i-th power station of the upper reservoir at the time instant, represents the water storage amount at the time instant, represents the maximum water storage amount of the upper reservoir.

3. The offshore wind farm configuration energy storage participation in real-time electricity market decision apparatus according to claim 2, characterized in that, The capacity balance of the water storage energy storage and the determination formula of the upper and lower limit physical constraints are as follows: ， wherein, and respectively represent the storage capacity of the upper reservoir and the lower reservoir at the time, represent the storage capacity of the upper reservoir and the lower reservoir at the time, and respectively represent the maximum pumping capacity and the maximum releasing capacity of the time, time, and respectively represent the charging and discharging instructions of the energy storage power station; ， wherein, and respectively represent the maximum storage capacity of the upper reservoir and the lower reservoir, and respectively represent the minimum storage capacity of the upper reservoir and the lower reservoir.

4. The offshore wind power configuration energy storage participating in spot power market decision device according to claim 3, characterized in that, 。 5. The offshore wind farm configuration energy storage participation in real-time electricity market decision apparatus according to claim 1, characterized in that, The system energy exchange determination formula of the operating environment is as follows: ， wherein, and respectively represent the maximum pumping and maximum releasing amount of the nth pumped storage power station at the time instant t, respectively represent the maximum power of the pump and turbine of the nth pumped storage power station, respectively represent the maximum power of the pump and turbine of the nth pumped storage power station, represents the pumping coefficient, represents the generating coefficient, ρ represents the density of seawater, taken as 1.03×103 kg / m3, g represents the gravitational acceleration, taken as 9.81 m / s2, and h refers to the height from the water surface of the upper reservoir or the water surface of the lower reservoir to the water inlet and outlet of the pipeline;​​​ ， wherein, , and represent the efficiencies of the energy storage plant pump, motor and piping, respectively, , and represent the efficiencies of the turbine, generator and piping, respectively.

6. The offshore wind farm configuration energy storage participation in real-time electricity market decision apparatus according to claim 1, characterized in that, The system penalty determination formula of the operating environment is as follows: ， ， ， ， ， in, The representative system is The total penalty at any moment This represents the penalty coefficient for exceeding the limit. and These represent the expected maximum and minimum power of the power grid system, respectively. Represents the deviation penalty coefficient. Representing the power grid sector The system's power generation is expected to be constant. Representing the power grid sector The difference between the actual power generation and the expected power generation of the system is allowed at all times. Represents the maintenance penalty coefficient. Representing the One fan in The workload requires constant maintenance by the repair team. On behalf of the maintenance team The time has reached the 10th One fan, The representative system is The total power transmitted to the power grid at any given time.

7. The offshore wind farm configuration energy storage participation in real-time electricity market decision apparatus according to claim 1, characterized in that, After the step S33, the following steps are included: S41: update the parameters of the Q Network network to the Target Q Network at intervals; S42: determine whether the network output meets the expected evaluation standard; if yes, end the training; if not, continue to update the iterative deep reinforcement learning network from step S32; And / or, the determination formula of the loss function in the evaluation standard is as follows: wherein, is a loss function, represents the reward of the agent at time t, represents a discount factor, represents the parameters of the Target Q Network at time t, corresponding to the action chosen by the greedy policy, represents the parameters of the Q Network, represents the parameters of the Target Q Network.

8. A method for offshore wind power configuration energy storage participation in spot power market decision, characterized in that, The offshore wind power configuration energy storage participating in spot power market decision device of any one of claims 1 to 7, The decision steps are as follows: S1: mathematical modeling of the offshore wind power configuration energy storage system according to the basic operating information of the offshore wind power station and the water storage energy storage power station, the basic operating information including the total power of the offshore wind power station, the planned workload of the maintenance team, the reservoir volume of the water storage energy storage power station, the power and efficiency of the pump and motor, turbine and generator of the energy storage power station, and the pipeline loss; S2: Constructing an offshore wind power configuration energy storage system operation environment and forming a virtual environment according to the short-term power dispatching demand of the power grid department, the power dispatching demand including the power grid department expected system power generation, the system maximum deviation allowed, the system maximum minimum power allowed, the operation environment needing to consider the system active device load limit and the power grid load limit, including energy storage power station pumps and motors, turbines and generators, and the power grid expected system maximum minimum power; S3: Using a deep double Q network model based on the system model to train an agent to make decisions and participate in short-term spot power market transactions according to the system virtual environment, the deep double Q network model including the agent, the virtual environment, the state space, the action space and the reward function; S4: Updating the deep reinforcement learning model based on the training results and evaluation indicators, and using the tested strategy to determine the transaction strategy of the offshore wind power configuration energy storage system participating in the short-term power spot market.

Images