Decision-making model and decision-making method for offshore wind power configuration energy storage participating in spot electricity market

Abstract

The invention discloses an offshore wind power configuration energy storage participation spot electricity market decision-making model and decision-making method, and the model comprises a data collection module, a virtual environment module, an algorithm model module, and a test updating module. The data acquisition module is used for collecting basic operation information of the offshore wind power configuration water storage and energy storage system and scheduling information provided by a power grid; the virtual environment module is used for constructing an operation environment of the offshore wind power configuration energy storage system according to short-term power dispatching requirements of a power grid department and forming a virtual environment; the algorithm model module performs mathematical modeling on the offshore wind power configuration energy storage system according to the basic operation information, and constructs a deep double-Q network model based on a system model for training an intelligent agent to make a decision and participate in short-term spot electricity market transaction; and the test updating module is used for evaluating the training effect of the current deep double-Q network model and performing parameter updating until the decision model output accords with an expected set evaluation standard. The model and the method improve the operation efficiency and benefits of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of power market, and in particular to a decision model and a decision method for offshore wind power configuration energy storage to participate in the spot power market. Background Art

[0002] Offshore wind power is a renewable energy technology that uses offshore wind energy resources to generate electricity. It has the advantages of abundant wind energy resources, stable wind speed, small land occupation, and small environmental impact. However, offshore wind power faces problems such as wind power fluctuations, difficulty in grid access, and high operation and maintenance costs. Especially in the context of electricity marketization, the economy and competitiveness of offshore wind power have been challenged. Water storage is a storage technology that uses hydroelectric generators for charging and discharging. It has the advantages of large energy storage capacity, long life, and high efficiency. Water storage can achieve complementarity and coordination with wind power by smoothing wind power fluctuations, providing frequency and peak regulation services, and participating in electricity market transactions. However, traditional water storage power stations face problems such as limited site selection and low water resource utilization efficiency.

[0003] Offshore wind power is equipped with water storage, that is, in the sea area near the offshore wind farm, using the seabed topography and artificial structures, to build a water storage power station and construct an integrated system of offshore wind power and water storage power station. Seawater is inexhaustible. Using seawater as an energy storage medium can greatly reduce the construction and operation and maintenance costs of water storage power stations. However, the existing offshore wind power equipped with water storage energy storage systems are mostly equipped with thermal power units as auxiliary as a stable power supplier, without considering the volatility of power system demand after the marketization of electricity, and the application scenarios of offshore wind power equipped with water storage energy storage systems participating in electricity market transactions. Water storage energy storage can be charged and discharged according to the power fluctuations of offshore wind power and changes in market prices, thereby smoothing the output power of offshore wind power and improving the power quality and market competitiveness of offshore wind power. Therefore, in order to improve the market participation ability of offshore wind power equipped with energy storage systems, reduce its impact on the power grid, and increase its benefits, it is necessary to adopt appropriate scheduling strategies and trading strategies. Summary of the invention

[0004] In view of the difficulty in establishing a model of offshore wind power configuration water storage energy storage system in the prior art and the low ability of the system to participate in electricity market transactions and its profitability, the present invention provides a decision model and decision method for offshore wind power configuration energy storage to participate in the spot electricity market, which can realize intelligent decision-making and trading of offshore wind power configuration water storage energy storage system, improve the operating efficiency and profitability of the system, reduce the operating cost and risk of the system, and improve the adaptability and robustness of the system. The specific technical scheme is as follows:

[0005] The first aspect of the present invention provides a decision-making model for offshore wind power configuration energy storage to participate in the spot power market, including:

[0006] A data acquisition module, used to collect basic operation information of the offshore wind power configuration water storage energy storage system and dispatch information provided by the power grid, wherein the basic operation information includes the energy storage capacity of the offshore wind power configuration water storage energy storage system and the capacity balance of the water storage energy storage and its upper and lower physical constraints;

[0007] The virtual environment module is used to build an offshore wind power configuration energy storage system operating environment and form a virtual environment according to the short-term power dispatching needs of the power grid department;

[0008] An algorithm model module, which mathematically models the offshore wind power configuration energy storage system according to the basic operation information, and constructs a deep double-Q network model based on the system model for training the intelligent agent to make decisions and participate in short-term spot power market transactions; and

[0009] The test update module is used to evaluate the training effect of the current deep dual-Q network model and update the parameters until the model output meets the expected set evaluation criteria.

[0010] Preferably, the energy storage capacity SoC t The formula for determining is as follows:

[0011]

[0012] Among them, SoC t Represents the energy storage capacity of the offshore wind power configuration water storage system at time t, SoC k,t represents the capacity of the kth water storage power station at time t, represents the water storage capacity of the kth power station in the upper reservoir at time t, Represents the maximum water storage capacity of the upper reservoir.

[0013] Preferably, the capacity balance of the water storage and the upper and lower physical constraints are determined by the following formula:

[0014]

[0015]

[0016] in, and They represent the water storage capacity of the upper reservoir and the lower reservoir of the kth power station at time t, τ represents the self-loss caused by reservoir leakage or evaporation, and They represent the maximum pumping and maximum discharge of the kth power station at time t, and Respectively represent the charging and discharging instructions of the energy storage power station;

[0017]

[0018]

[0019] in, and Represent the maximum water storage capacity of the upper reservoir and the lower reservoir respectively, and Represent the minimum water storage capacity of the upper reservoir and the lower reservoir respectively.

[0020] Preferably,

[0021] Preferably, the system energy exchange determination formula of the operating environment is as follows:

[0022]

[0023]

[0024] in, and They represent the maximum pumping and maximum discharge of the kth power station at time t, and Represent the maximum power of the pump and turbine of the kth energy storage power station, c p represents the pumping coefficient, c g represents the power generation coefficient;

[0025] η pump =η p ×η m ×η pipe ,

[0026] η gen =η TR ×η GR ×η pipe

[0027] Among them, η p , η m and η pipe Respectively represent the efficiency of the pump, motor and pipeline of the energy storage power station, η TR , η GR and η pi pe represents the efficiency of the turbine, generator and pipeline respectively.

[0028] Preferably, the formula for determining the system penalty of the operating environment is as follows:

[0029] P 1,t =υmax[0,(p total,t -p max )],

[0030] P 2,t =υmax[0,(p min -ptotal,t )],

[0031]

[0032]

[0033] P t =P 1,t +P 2,t +P 3,t +P 4,t ,

[0034] Among them, P t represents the total penalty of the system at time t, υ represents the over-limit penalty coefficient, and p max and p min Represent the maximum power and minimum power of the expected power grid system respectively, ω represents the deviation penalty coefficient, represents the power generation capacity of the system expected by the power grid department at time t, p allow,t represents the difference between the actual power generation and the expected power allowed by the power grid department at time t, ψ represents the maintenance penalty coefficient, M r,t represents the workload of the maintenance team for the rth wind turbine at time t, It means that the maintenance team has arrived at the rth wind turbine at time t.

[0035] The second aspect of the present invention provides a method for making decisions on offshore wind power configuration energy storage to participate in the spot power market, including the offshore wind power configuration energy storage to participate in the spot power market decision model, and the decision-making steps are as follows:

[0036] S1: mathematical modeling of the offshore wind power configuration energy storage system based on the basic operation information of the offshore wind power station and the water storage energy storage power station, wherein the basic operation information includes the total power of the offshore wind power station, the planned workload of the maintenance team, the reservoir volume of the water storage energy storage power station, the power and efficiency of the pump and motor, turbine and generator of the energy storage power station, and the pipeline loss;

[0037] S2: Construct an offshore wind power configuration energy storage system operating environment and form a virtual environment based on the short-term power dispatching requirements of the power grid department. The power dispatching requirements include the power grid department's expected system power generation, the maximum allowable system deviation, and the maximum and minimum allowable system power. The operating environment needs to consider the load limit of the system's active devices and the load limit of the power grid, including the energy storage power station pumps and motors, turbines and generators, and the maximum and minimum power of the power grid's expected system.

[0038] S3: According to the system virtual environment, a deep double Q network model is used to train an intelligent agent based on a system model to make decisions and participate in short-term spot power market transactions, wherein the deep double Q network model includes the intelligent agent, a virtual environment, a state space, an action space, and a reward function;

[0039] S4: The deep reinforcement learning model is updated based on the training results and evaluation indicators, and the tested strategy is used to determine the trading strategy for the offshore wind power configuration energy storage system to participate in the short-term electricity spot market.

[0040] Preferably, step S3 comprises the following steps:

[0041] S31: Establish a deep dual Q neural network consisting of Q Network, Target Q Network and Replay buffer, and initialize the network according to the system model. The Q Network and Target Q Network have the same structure. The Replay buffer unit is used to record four pieces of information: the current state, action, reward and next state of the agent.

[0042] S32: The agent moves from action space a based on Q Network at time t t Select an action in the virtual environment, submit the current action to the virtual environment, and calculate the benefit R based on the virtual environment feedback. t , and the state S of the agent at this action t 、Action a t , Reward R t and the state S at the next moment t+1 As a vector group [S t ,a t ,R t ,S t+1 ], stored in the Replay buffer, where S t 、a t and R t Represent the state space, action space and reward of the agent at time t, S t+1 Represents the state space of the agent at time t+1, where the reward function R t Represents the agent in S t Execute action a at any time t Change to state S t+1 Instant rewards;

[0043] S33: When the data stored in the Replay buffer reaches the set value, batch sample data is randomly extracted therefrom for QNetwork parameter training and the parameters are updated using the optimizer according to the principle of minimizing the loss function. The randomly extracted batch sample data is obtained using a Gaussian distribution sampler, and the optimizer is a stochastic gradient descent optimizer.

[0044] Preferably, the state space is determined by the following vector group:

[0045] S t =[SoC k,t ,p load,t ,M r,t ]

[0046] Among them, S t Represents the state space of the system at time t, SoC k,t represents the capacity of the kth water storage power station at time t, p total,t represents the total power transmitted to the grid by the system at time t, M r,t represents the workload of the maintenance team for the rth wind turbine at time t;

[0047] The action space is determined by the following set of vectors:

[0048]

[0049] Among them, a t represents the action space of the system at time t,

[0050] and Represent the charging and discharging instructions of the energy storage power station, α t Represents the grid connection coefficient of the offshore wind power station, and They represent that the maintenance team has arrived at the r-th wind turbine and is on the way to the r-th wind turbine at time t respectively;

[0051] The reward function is determined as follows:

[0052]

[0053] Among them, R t represents the reward of the agent at time t, C t represents the electricity price of the system at time t, p total,t represents the total power transmitted to the grid by the system at time t, and Represent the maximum power of the pump and turbine of the kth energy storage power station, ξ k Represents the energy storage aging coefficient of the kth energy storage power station, M r,t represents the workload of the maintenance team for the rth wind turbine at time t, C r Represents the labor remuneration coefficient of wind power station maintenance personnel, P t represents the total penalty of the system at time t, C op Represents other operation and maintenance costs, Represents the safe operation factor of the energy storage system.

[0054] Preferably, the following steps are included after step S33:

[0055] S41: Synchronize and update the parameters of the Q Network to the Target Q Network at regular intervals;

[0056] S42: Determine whether the network output meets the expected evaluation criteria; if so, end the training; if not, continue to update the iterative deep reinforcement learning network from step S32;

[0057] And / or, the loss function in the evaluation criteria is determined by the following formula:

[0058]

[0059] Among them, L(θ c ) is the loss function, R t represents the reward of the agent at time t, γ represents the discount factor, Represents the state s of Target Q Network at time t+1 t+1 Next, use the greedy strategy to select the Q value corresponding to action a, θ c represents the parameters of Q Network, θ c ′ represents the parameters of Target Q Network.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] This model and method make full use of the characteristics and advantages of offshore wind power and hydroelectric energy storage to construct an integrated system of offshore wind power and hydroelectric energy storage. Taking into account the uncertainty of wind power, grid demand and market price, a deep double-Q network learning method is used to realize intelligent decision-making and trading of offshore wind power and hydroelectric energy storage systems, improve the operating efficiency and benefits of the system, reduce the operating costs and risks of the system, and improve the adaptability and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the decision-making model module of offshore wind power configuration energy storage participating in the spot power market of the present invention;

[0063] Figure 2 It is a schematic diagram of the principle of the data acquisition module in the decision-making method of offshore wind power configuration energy storage participating in the spot power market of the present invention;

[0064] Figure 3 It is a schematic diagram of the principle of the test update module in the decision-making method of offshore wind power configuration energy storage participating in the spot power market of the present invention;

[0065] Figure 4It is a schematic diagram of the overall process of the decision-making method for offshore wind power configuration energy storage to participate in the spot power market of the present invention;

[0066] Figure 5 It is a flow chart of the algorithm model module in the decision-making method for offshore wind power configuration energy storage to participate in the spot power market of the present invention;

[0067] Figure 6 It is a flow chart of the test update module in the decision-making method of offshore wind power configuration energy storage participating in the spot power market of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0070] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0071] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0072] To solve the above problems, see Figure 1As shown, the first aspect of the present invention provides a decision-making model for offshore wind power configuration energy storage to participate in the spot power market, including a data acquisition module, a virtual environment module, an algorithm model module and a test update module, the data acquisition module is used to collect basic operating information of the offshore wind power configuration water storage energy storage system and the scheduling information provided by the power grid, the basic operating information includes the energy storage capacity of the offshore wind power configuration water storage energy storage system and the capacity balance of the water storage and its upper and lower physical constraints; the virtual environment module is used to construct an operating environment for the offshore wind power configuration energy storage system and form a virtual environment according to the short-term power scheduling needs of the power grid department; the algorithm model module mathematically models the offshore wind power configuration energy storage system according to the basic operating information, and constructs a deep dual-Q network model based on the system model for training intelligent agents to make decisions and participate in short-term spot power market transactions; the test update module is used to evaluate the current training effect of the deep dual-Q network model and update parameters until the decision model output meets the expected set evaluation criteria.

[0073] Among them, the data acquisition module needs to collect information such as grid dispatching information, the real-time capacity of the water storage power station, the real-time power transmitted to the grid by the system, and the workload of the wind turbine maintenance at each moment, so as to help the virtual environment module, the algorithm model module and the test update module optimize the trading strategy of the offshore wind power configuration water storage system to participate in the spot power market; the algorithm model module provides decision-making for the intelligent agent in the virtual environment, and learns and improves the strategy through the test update module to obtain the optimal strategy for the offshore wind power configuration water storage system to participate in the spot power market transaction.

[0074] In an optional embodiment, if Figure 2 As shown, the energy storage capacity SoC t The formula for determining is as follows:

[0075]

[0076] Among them, SoC t Represents the energy storage capacity of the offshore wind power configuration water storage system at time t, SoC k,t represents the capacity of the kth water storage power station at time t, represents the water storage capacity of the kth power station in the upper reservoir at time t, Represents the maximum water storage capacity of the upper reservoir.

[0077] In an optional embodiment, if Figure 2 As shown, the formula for determining the capacity balance of the water storage energy and the upper and lower physical constraints is as follows:

[0078]

[0079]

[0080] in, and They represent the water storage capacity of the upper reservoir and the lower reservoir of the kth power station at time t, τ represents the self-loss caused by reservoir leakage or evaporation, and They represent the maximum pumping and maximum discharge of the kth power station at time t, and Respectively represent the charging and discharging instructions of the energy storage power station;

[0081]

[0082]

[0083] in, and Represent the maximum water storage capacity of the upper reservoir and the lower reservoir respectively, and Represent the minimum water storage capacity of the upper reservoir and the lower reservoir respectively.

[0084] In an alternative embodiment,

[0085] In an optional embodiment, if Figure 2 As shown, the system energy exchange determination formula of the operating environment is as follows:

[0086]

[0087]

[0088] in, and They represent the maximum pumping and maximum discharge of the kth power station at time t, and Represent the maximum power of the pump and turbine of the kth energy storage power station, c p represents the pumping coefficient, c g represents the power generation coefficient;

[0089] η pump =η p ×η m ×η pipe ,

[0090] η gen =η TR ×η GR ×η pipe

[0091] Among them, η p , η m and η pipeRespectively represent the efficiency (loss) of the energy storage power station pump, motor and pipeline, η TR , η GR and η pipe Represent the efficiency (loss) of the turbine, generator and pipeline respectively.

[0092] In an optional embodiment, if Figure 2 As shown, the formula for determining the system penalty of the operating environment is as follows:

[0093] P 1,t =υmax[0,(p total,t -p max )],

[0094] P 2,t =υmax[0,(p min -p total,t )],

[0095]

[0096]

[0097] P t =P 1,t +P 2,t +P 3,t +P 4,t ,

[0098] Among them, P t represents the total penalty of the system at time t, υ represents the over-limit penalty coefficient, and p max and p min Represent the maximum power and minimum power of the expected power grid system respectively, ω represents the deviation penalty coefficient, represents the power generation capacity of the system expected by the power grid department at time t, p allow,t represents the difference between the actual power generation and the expected power allowed by the power grid department at time t, ψ represents the maintenance penalty coefficient, M r,t represents the workload of the maintenance team for the rth wind turbine at time t, It means that the maintenance team has arrived at the rth wind turbine at time t.

[0099] The second aspect of the present invention provides a decision-making method for offshore wind power configuration energy storage to participate in the spot power market, such as Figures 4 to 6 As shown, including the decision-making model of offshore wind power configuration energy storage participating in the spot power market, the decision-making steps are as follows:

[0100] S1: mathematical modeling of the offshore wind power configuration energy storage system based on the basic operation information of the offshore wind power station and the water storage energy storage power station, wherein the basic operation information includes the total power of the offshore wind power station, the planned workload of the maintenance team, the reservoir volume of the water storage energy storage power station, the power and efficiency of the pump and motor, turbine and generator of the energy storage power station, and the pipeline loss;

[0101] S2: Construct an offshore wind power configuration energy storage system operating environment and form a virtual environment based on the short-term power dispatching requirements of the power grid department. The power dispatching requirements include the power grid department's expected system power generation, the maximum allowable system deviation, and the maximum and minimum allowable system power. The operating environment needs to consider the load limit of the system's active devices and the load limit of the power grid, including the energy storage power station pumps and motors, turbines and generators, and the maximum and minimum power of the power grid's expected system.

[0102] S3: According to the system virtual environment, a deep double Q network model is used to train an intelligent agent based on a system model to make decisions and participate in short-term spot power market transactions, wherein the deep double Q network model includes the intelligent agent, a virtual environment, a state space, an action space, and a reward function;

[0103] S4: The deep reinforcement learning model is updated based on the training results and evaluation indicators, and the tested strategy is used to determine the trading strategy for the offshore wind power configuration energy storage system to participate in the short-term electricity spot market.

[0104] Preferably, if Figure 4 and Figure 5 As shown, step S3 includes the following steps:

[0105] S31: Establish a deep dual Q neural network consisting of Q Network, Target Q Network and Replay buffer, and initialize the network according to the system model. The Q Network and Target Q Network have the same structure. The Replay buffer unit is used to record four pieces of information: the current state, action, reward and next state of the agent.

[0106] S32: The agent moves from action space a based on Q Network at time t t Select an action in the virtual environment, submit the current action to the virtual environment, and calculate the benefit R based on the virtual environment feedback. t , and the state S of the agent at this action t 、Action a t , Reward R t and the state S at the next moment t+1 As a vector group [S t ,a t ,R t ,St+1 ], stored in the Replay buffer, where S t 、a t and R t Represent the state space, action space and reward of the agent at time t, S t+1 Represents the state space of the agent at time t+1, where the reward function R t Represents the agent in S t Execute action a at any time t Change to state S t+1 Instant reward; In this embodiment, the decision model for configuring offshore wind power storage to participate in the spot power market is a Markov decision process (MDP), and the establishment of the Markov decision process is based on a vector group of four vectors, and the vector group is specifically [S t ,a t ,R t ,S t+1 ],

[0107] S33: When the data stored in the Replay buffer reaches the set value, batch sample data is randomly extracted therefrom for QNetwork parameter training and the parameters are updated using the optimizer according to the principle of minimizing the loss function. The randomly extracted batch sample data is obtained using a Gaussian distribution sampler, and the optimizer is a stochastic gradient descent optimizer.

[0108] In an alternative embodiment, the state is considered to be continuous over the entire time interval, while the action is discrete, making it consistent with the nature of the main power components within the system, where the state space is determined by the following set of vectors:

[0109] S t =[SoC k,t ,p load,t ,M r,t ]

[0110] Among them, S t Represents the state space of the system at time t, SoC k,t represents the capacity of the kth water storage power station at time t, p total,t represents the total power transmitted to the grid by the system at time t, M r,t represents the workload of the maintenance team for the rth wind turbine at time t;

[0111] The action space is determined by the following set of vectors:

[0112]

[0113] Among them, a trepresents the action space of the system at time t, and Represent the charging and discharging instructions of the energy storage power station, α t Represents the grid connection coefficient of the offshore wind power station, and They represent that the maintenance team has arrived at the r-th wind turbine and is on the way to the r-th wind turbine at time t respectively;

[0114] The reward function is determined as follows:

[0115]

[0116] Among them, R t represents the reward of the agent at time t, C t represents the electricity price of the system at time t, p total,t represents the total power transmitted to the grid by the system at time t, and Represent the maximum power of the pump and turbine of the kth energy storage power station, ξ k Represents the energy storage aging coefficient of the kth energy storage power station, M r,t represents the workload of the maintenance team for the rth wind turbine at time t, C r Represents the labor remuneration coefficient of wind power station maintenance personnel, P t represents the total penalty of the system at time t, C op Represents other operation and maintenance costs, Represents the safe operation factor of the energy storage system.

[0117] In an optional embodiment, if Figure 3 , Figures 4 to 6 As shown, after step S33, the following steps are included:

[0118] S41: Synchronize and update the parameters of the Q Network to the Target Q Network at regular intervals;

[0119] S42: Determine whether the network output meets the expected evaluation criteria; if so, end the training; if not, continue to update the iterative deep reinforcement learning network from step S32;

[0120] And / or, the loss function in the evaluation criteria is determined by the following formula:

[0121]

[0122] Among them, L(θ c ) is the loss function, R t represents the reward of the agent at time t, γ represents the discount factor, Represents the state s of Target Q Network at time t+1 t+1 Next, use the greedy strategy to select the Q value corresponding to action a, θ c represents the parameters of Q Network, θ c ′ represents the parameters of Target Q Network.

[0123] In summary, this model and method make full use of the characteristics and advantages of offshore wind power and hydroelectric energy storage to construct an integrated system of offshore wind power and hydroelectric energy storage. Considering the uncertainty of wind power, grid demand and market price, a deep double Q network learning method is adopted to realize intelligent decision-making and trading of offshore wind power and hydroelectric energy storage system, improve the operating efficiency and benefits of the system, reduce the operating costs and risks of the system, and improve the adaptability and robustness of the system.

[0124] The above description is only a preferred feasible embodiment of the present invention and is not intended to limit the scope of the patent application of the present invention. All equivalent changes, equivalent substitutions or modified changes performed within the technical spirit and principles suggested by the present invention should be included in the scope of patent protection covered by the present invention.

Claims

1. A decision-making model for offshore wind power configuration energy storage to participate in the spot power market, characterized in that: include: Data acquisition module, used to collect basic operation information of offshore wind power configuration water storage energy storage system and dispatch information provided by the power grid, the basic operation information includes the energy storage capacity SoC of the offshore wind power configuration water storage energy storage system t and the capacity balance of hydroelectric energy storage and its upper and lower physical constraints; The virtual environment module is used to build an offshore wind power configuration energy storage system operating environment and form a virtual environment according to the short-term power dispatching needs of the power grid department; An algorithm model module, which mathematically models the offshore wind power configuration energy storage system according to the basic operation information, and constructs a deep double-Q network model based on the system model for training the intelligent agent to make decisions and participate in short-term spot power market transactions; and The test update module is used to evaluate the training effect of the current deep dual-Q network model and update the parameters until the decision model output meets the expected set evaluation criteria.

2. The decision-making model for offshore wind power configuration energy storage to participate in the spot power market according to claim 1 is characterized in that: The energy storage capacity SoC t The formula for determining is as follows: Among them, SoC t Represents the energy storage capacity of the offshore wind power configuration water storage system at time t, SoC k,t represents the capacity of the kth water storage power station at time t, represents the water storage capacity of the kth power station in the upper reservoir at time t, Represents the maximum water storage capacity of the upper reservoir.

3. The decision-making model for offshore wind power configuration energy storage to participate in the spot power market according to claim 2 is characterized in that: The capacity balance of the water storage and the upper and lower physical constraints are determined by the following formula: in, and They represent the water storage capacity of the upper reservoir and the lower reservoir of the kth power station at time t, τ represents the self-loss caused by reservoir leakage or evaporation, and They represent the maximum pumping and maximum discharge of the kth power station at time t, and Respectively represent the charging and discharging instructions of the energy storage power station; in, and Represent the maximum water storage capacity of the upper reservoir and the lower reservoir respectively, and Represent the minimum water storage capacity of the upper reservoir and the lower reservoir respectively.

4. The decision-making model for offshore wind power configuration energy storage to participate in the spot power market according to claim 3 is characterized in that:

5. The decision-making model for offshore wind power configuration energy storage to participate in the spot power market according to claim 1 is characterized in that: The system energy exchange determination formula for the operating environment is as follows: in, and They represent the maximum pumping and maximum discharge of the kth power station at time t, and Represent the maximum power of the pump and turbine of the kth energy storage power station, c p represents the pumping coefficient, c g represents the power generation coefficient; or pump =the p ×h m ×h pipe , or gen =the TR ×h GR ×h pipe Among them, η p , η m and η pi pe represents the efficiency of the pump, motor and pipeline of the energy storage power station, η TR , η GR and η pi pe represents the efficiency of the turbine, generator and pipeline respectively.

6. The decision-making model for offshore wind power configuration energy storage to participate in the spot power market according to claim 1 is characterized in that: The formula for determining the system penalty of the operating environment is as follows: P 1,t =υmax[0,(p total,t -p max )], P 2,t =υmax[0,(p min -p total,t )], P t =P 1,t +P 2,t +P 3,t +P 4,t , Among them, P t represents the total penalty of the system at time t, υ represents the over-limit penalty coefficient, and p max and p min Represent the maximum power and minimum power of the expected power grid system respectively, ω represents the deviation penalty coefficient, represents the power generation capacity of the system expected by the power grid department at time t, p all ow,t represents the difference between the actual power generation and the expected power allowed by the power grid department at time t, ψ represents the maintenance penalty coefficient, M r,t represents the workload of the maintenance team for the rth wind turbine at time t, It means that the maintenance team has arrived at the rth wind turbine at time t.

7. A decision-making method for offshore wind power configuration energy storage to participate in the spot power market, characterized in that: The decision-making model for offshore wind power configuration energy storage to participate in the spot power market as described in any one of claims 1 to 6 has the following decision-making steps: S1: mathematical modeling of the offshore wind power configuration energy storage system based on the basic operation information of the offshore wind power station and the water storage energy storage power station, wherein the basic operation information includes the total power of the offshore wind power station, the planned workload of the maintenance team, the reservoir volume of the water storage energy storage power station, the power and efficiency of the pump and motor, turbine and generator of the energy storage power station, and the pipeline loss; S2: Construct an offshore wind power configuration energy storage system operating environment and form a virtual environment based on the short-term power dispatching requirements of the power grid department. The power dispatching requirements include the power grid department's expected system power generation, the maximum allowable system deviation, and the maximum and minimum allowable system power. The operating environment needs to consider the load limit of the system's active devices and the load limit of the power grid, including the energy storage power station pumps and motors, turbines and generators, and the maximum and minimum power of the power grid's expected system. S3: According to the system virtual environment, a deep double Q network model is used to train an intelligent agent based on a system model to make decisions and participate in short-term spot power market transactions, wherein the deep double Q network model includes the intelligent agent, a virtual environment, a state space, an action space, and a reward function; S4: The deep reinforcement learning model is updated based on the training results and evaluation indicators, and the tested strategy is used to determine the trading strategy for the offshore wind power configuration energy storage system to participate in the short-term electricity spot market.

8. The decision-making method for offshore wind power configuration energy storage to participate in the spot power market according to claim 7 is characterized in that: The step S3 comprises the following steps: S31: Establish a deep dual Q neural network consisting of Q Network, Target Q Network and Replay buffer, and initialize the network according to the system model. The Q Network and Target Q Network have the same structure. The Replay buffer unit is used to record four pieces of information: the current state, action, reward and next state of the agent. S32: The agent moves from action space a based on Q Network at time t t Select an action in the virtual environment, submit the current action to the virtual environment, and calculate the benefit R based on the virtual environment feedback. t , and the state S of the agent at this action t 、Action a t , Reward R t and the state S at the next moment t+1 As a vector group [S t ,a t ,R t ,S t+1 ], stored in the Replay buffer, where S t 、a t and R t Represent the state space, action space and reward of the agent at time t, S t+1 Represents the state space of the agent at time t+1, where the reward function R t Represents the agent in S t Execute action a at any time t Change to state S t+1 Instant rewards; S33: When the data stored in the Replay buffer reaches the set value, batch sample data is randomly extracted therefrom for QNetwork parameter training and the parameters are updated using the optimizer according to the principle of minimizing the loss function. The randomly extracted batch sample data is obtained using a Gaussian distribution sampler, and the optimizer is a stochastic gradient descent optimizer.

9. The decision-making method for offshore wind power configuration energy storage to participate in the spot power market according to claim 8 is characterized in that: The state space is determined by the following set of vectors: S t =[SoC k,t ,p load,t ,M r,t ] Among them, S t Represents the state space of the system at time t, SoC k,t represents the capacity of the kth water storage power station at time t, p total,t represents the total power transmitted to the grid by the system at time t, M r,t represents the workload of the maintenance team for the rth wind turbine at time t; The action space is determined by the following set of vectors: Among them, a t represents the action space of the system at time t, and Represent the charging and discharging instructions of the energy storage power station, α t Represents the grid connection coefficient of the offshore wind power station, and They represent that the maintenance team has arrived at the r-th wind turbine and is on the way to the r-th wind turbine at time t respectively; The reward function is determined as follows: Among them, R t represents the reward of the agent at time t, C t represents the electricity price of the system at time t, p total,t represents the total power transmitted to the grid by the system at time t, and Represent the maximum power of the pump and turbine of the kth energy storage power station, ξ k Represents the energy storage aging coefficient of the kth energy storage power station, M r,t represents the workload of the maintenance team for the rth wind turbine at time t, C r Represents the labor remuneration coefficient of wind power station maintenance personnel, P t represents the total penalty of the system at time t, C op Represents other operation and maintenance costs, Represents the safe operation factor of the energy storage system.

10. The decision-making method for offshore wind power configuration energy storage to participate in the spot power market according to claim 8, characterized in that: After step S33, the following steps are included: S41: Synchronize and update the parameters of the Q Network to the Target Q Network at regular intervals; S42: Determine whether the network output meets the expected evaluation criteria; if so, end the training; if not, continue to update the iterative deep reinforcement learning network from step S32; And / or, the loss function in the evaluation criteria is determined by the following formula: Among them, L(θ c ) is the loss function, R t represents the reward of the agent at time t, γ represents the discount factor, Represents the state s of Target Q Network at time t+1 t+1 Next, use the greedy strategy to select the Q value corresponding to action a, θ c represents the parameters of Q Network, θ c ′ represents the parameters of Target Q Network.

Images