A Market-Based Production Control Method for Energy Storage Power Stations

Abstract

This invention discloses a production control method for energy storage power stations based on a market environment, belonging to the field of energy storage power station production control technology. The method includes: acquiring operational data of the target power grid; predicting and outputting the optimal energy storage power configuration scheme based on a pre-trained TD3 agent; training the TD3 agent using a reinforcement learning algorithm; the reward function of the reinforcement learning algorithm aims to maximize the product of the energy configuration amount and the corresponding electricity price configuration value; the penalty term of the reward function includes the deviation between the current energy storage charging power and the historical optimal charging power, the deviation between the energy storage discharging power and the historical optimal discharging power, and the energy storage state of charge constraint violation rate. This invention solves the problems of low efficiency and poor returns in the production control of existing energy storage power stations.

Description

Technical Field

[0001] This invention relates to a production control method for energy storage power stations based on market conditions, belonging to the field of energy storage power station production control technology. Background Technology

[0002] Current energy storage power station technology has formed a diversified technological system, mainly including mechanical energy storage, electrochemical energy storage, thermal energy storage, and electromagnetic energy storage. Among them, lithium-ion batteries have become the mainstream application due to their high energy density and long cycle life, and are widely used in scenarios such as grid peak shaving, frequency regulation, and renewable energy consumption. Energy storage power stations effectively alleviate the problem of mismatch between power supply and demand in time and space by storing and releasing electrical energy, and improve grid stability and renewable energy utilization. Existing systems integrate BMS battery management systems, PCS energy storage converters, and intelligent dispatching algorithms to realize charge and discharge control, energy conversion, and grid interaction. In terms of policy and market mechanisms, many regions have promoted revenue models such as peak-valley price arbitrage, ancillary service compensation, and capacity pricing to promote the commercialization of energy storage.

[0003] However, existing technologies still face bottlenecks such as energy density, lifespan, and cost control, which restrict their large-scale application and result in core problems such as low production control efficiency and poor returns. Summary of the Invention

[0004] The purpose of this invention is to provide a market-based energy storage power station production control method. By utilizing reinforcement learning algorithms and dynamically embedded price-power experience pools, the optimal energy storage configuration scheme is obtained to control the production of the energy storage power station under the premise of meeting safe operation constraints, thereby solving the problems of low production control efficiency and poor returns of existing energy storage power stations.

[0005] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution.

[0006] This invention provides a production control method for energy storage power stations based on market conditions, comprising:

[0007] Obtain operational data of the target power grid, including charging power / discharging power and time-of-use electricity prices for the corresponding time periods;

[0008] Based on the operational data, the optimal energy storage configuration scheme is output by predicting the optimal energy storage configuration scheme according to the pre-trained TD3 (Twin Delayed Deep Deterministic Policy Gradient) agent.

[0009] Utilize the optimal energy storage configuration scheme to control the production of energy storage power stations;

[0010] The TD3 agent is trained using a reinforcement learning algorithm. The reward function of the reinforcement learning algorithm aims to maximize the product of the energy allocation amount and the corresponding electricity price allocation value. The penalty term of the reward function includes the deviation between the current energy storage charging power and the historical best energy storage charging power, the deviation between the energy storage discharging power and the historical best energy storage discharging power, and the energy storage state of charge constraint violation rate.

[0011] Furthermore, the training method for the TD3 agent includes:

[0012] Collect historical operating data of the target power grid, including historical charging power / historical discharging power and historical time-of-use electricity prices for the corresponding time periods;

[0013] Using a Transformer model with embedded mean square error loss function and time series mean square error loss function, based on the supply and demand relationship between historical charging power and historical time-of-use electricity price for the corresponding period / the supply and demand relationship between historical discharging power and historical time-of-use electricity price for the corresponding period, a mapping set between electricity spot market price and charging power / electricity spot market price and discharging power is constructed, and an electricity price-power experience pool is formed.

[0014] The electricity price-power experience pool is dynamically embedded at the input end of the reinforcement learning environment parameters, and the electricity price corresponding to the historical best energy storage charging power / the electricity price corresponding to the historical best energy storage discharging power is called as the initial configuration value of the electricity price.

[0015] The MILP-TD3 (Mixed integer linear program-TwinDelayed Deep Deterministic Policy Gradient) algorithm is improved by introducing energy storage state-of-charge constraints. The initial electricity price configuration value is input into the TD3 agent, and the energy storage energy configuration scheme is defined as a Markov decision problem. The optimal energy storage energy configuration scheme is generated by iterative training based on the improved MILP-TD3 algorithm.

[0016] The energy storage configuration scheme includes the amount of energy configured and the corresponding electricity price.

[0017] Furthermore, the mean squared error loss function is expressed as:

[0018] ;

[0019] In the formula, This represents the loss value of the mean squared error loss function. Represents a time series set. Indicates time Predicted energy storage charging power / energy storage discharging power , , This represents the supply and demand relationship between the historical optimal charging power of energy storage and the historical time-of-use electricity price for the corresponding period / the supply and demand relationship between the historical optimal discharging power of energy storage and the historical time-of-use electricity price for the corresponding period. Indicates time The corresponding time-of-use electricity price, Indicates time The corresponding historical time-of-use electricity price, Indicates time The corresponding historical charging power / historical discharging power Represents the policy function;

[0020] The time-series mean squared error loss function is expressed as follows:

[0021] ;

[0022] In the formula, This represents the loss value of the time-series mean squared error loss function. Indicates time The corresponding historical best charging power / historical best discharging power of energy storage. Indicates time The weighting coefficients corresponding to the deviation between the current energy storage charging power and the historical best charging power / the deviation between the energy storage discharging power and the historical best discharging power. , This represents the sine function.

[0023] Furthermore, the Markov decision problem is expressed as:

[0024] ;

[0025] In the formula, Represents the set of state spaces. Represents the action space set, Represents the state transition probability. Represents the set of reward values. This represents the reward discount factor.

[0026] Furthermore, in the state space set, the state space includes energy storage power configuration schemes, predicted energy storage power configuration schemes, time-of-use pricing, and energy storage state of charge. The state values ​​in the state space are represented as follows:

[0027] ;

[0028] In the formula, Indicates time The corresponding state value, Indicates time The corresponding energy storage state of charge value.

[0029] Furthermore, in the set of action spaces, the action space includes the maximum energy storage charging power / discharging power and the energy storage state of charge, and the action space is represented as:

[0030] ;

[0031] ;

[0032] ;

[0033] In the formula, This indicates the maximum charging / discharging power of the energy storage. Indicates time The corresponding energy storage state of charge value.

[0034] Furthermore, the state transition probability is expressed as:

[0035] ;

[0036] In the formula, Represents the state transition probability. Indicates time The corresponding state of the TD3 agent, Indicates time The corresponding state of the TD3 agent, Indicates time The corresponding actions taken by the TD3 agent Indicates time The corresponding state value, Indicates time The corresponding state value, Represents a probability function. Indicates time The corresponding action value of the TD3 agent.

[0037] Furthermore, the set of reward values ​​is the set of output values ​​of the reward function, which is expressed as:

[0038]

[0039] In the formula, Represents the reward function, This represents the penalty coefficient for the deviation between the energy storage charging power and the historical best charging power / the deviation between the energy storage discharging power and the historical best discharging power. The penalty coefficient representing the violation rate of the energy storage state of charge constraint. It represents the product of the electrical energy allocation amount and the corresponding electricity price allocation value.

[0040] Furthermore, the improved MILP-TD3 algorithm includes an actor network and a critic network. During iterative training, the parameters of the actor network and the critic network are first randomly initialized, and then the parameters are adjusted according to the time of the actor network. Sample the corresponding state and execute the time. The corresponding actions interact with the environment and are transferred to the moment. For the corresponding state, a penalty term is added to the reward function using a DNN formula in the form of MILP to obtain the time step. Corresponding rewards to form a sample The parameters of the actor network and the critic network are updated by selecting the sample corresponding to the optimal energy storage power configuration scheme from the electricity price-power experience pool.

[0041] The parameter update expression for the actor network is expressed as:

[0042] ;

[0043] In the formula, The parameters representing the actor network, This indicates calculating the gradient matrix of the parameters of the actor network. Indicates time The corresponding critic network, Represents actor network In state The energy storage configuration scheme adopted below, Represents a set of states. This represents a function to find the minimum value;

[0044] The parameter update expression for the critic network is expressed as:

[0045] ;

[0046] In the formula, Represents the parameters of the critic network Find the minimum value. Indicates time The corresponding network Q value, Represents actor network At time Corresponding state The energy storage configuration scheme adopted below, Indicates the first One neuron;

[0047] The output vector expression is as follows:

[0048] ;

[0049] In the formula, Indicates the first The output vector of a layer of neurons This indicates that the ReLU function is activated. Indicates the preset first The weight coefficient matrix of each neuron layer Indicates the first The bias vector of each neuron layer;

[0050] The MILP-form DNN formula is expressed as follows:

[0051] ;

[0052] In the formula, This represents the total number of layers in the DNN network. Indicates the first The total number of neurons in each neuron layer Indicates the first The first neuron layer The output vector of each neuron The constraint constants, Indicates the first The first neuron layer Input vector of each neuron The constraint constants;

[0053] The constraints of the MILP-form DNN formula are expressed as follows:

[0054] ;

[0055] In the formula, Indicates the first The total number of neurons in each neuron layer Indicates the preset first The first neuron layer The first neuron and the first The weight coefficient matrix of each neuron. Indicates the first The first neuron layer The output vector of each neuron Indicates the first The first neuron layer The bias vector of each neuron Indicates the first The first neuron layer Integer variable values ​​for each neuron;

[0056] The first The first neuron layer The output vector of each neuron Constraints and the first The first neuron layer Input vector of each neuron The constraints are expressed as follows:

[0057] ;

[0058] ;

[0059] In the formula, Indicates the first The first neuron layer The upper limit of the output vector of a neuron. Indicates the first The first neuron layer The lower bound of the input vector of each neuron. Indicates the first The first neuron layer The upper limit of the output vector of a neuron. Indicates the first The first neuron layer The lower bound of the input vector of each neuron. This represents the upper limit of the output vector. This represents the lower bound of the output vector. This represents the lower bound of the input vector. This indicates the upper bound of the input vector;

[0060] Furthermore, the total reward value is expressed as:

[0061] ;

[0062] In the formula, This indicates taking the maximum value. Represents the total reward value. This represents the learning rate of the reinforcement learning algorithm. This represents the training rounds after convergence. Indicates time The corresponding reward discount factor, Indicates time The corresponding reward value.

[0063] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0064] This invention directly obtains the optimal energy storage configuration scheme by using a TD3 agent to maximize the objective of multiplying the energy configuration amount and the electricity price configuration value in the reward function. At the same time, through the dual constraints of the charging / discharging power deviation and the violation rate of the state of charge constraint by the penalty term, the energy storage power station can strictly adhere to the battery safety boundary while operating efficiently. This reduces the energy loss caused by power deviation and the accelerated battery aging caused by exceeding the state of charge limit, and solves the problems of low production control efficiency and poor returns of existing energy storage power stations. Attached Figure Description

[0065] Figure 1 This is a schematic flowchart of a production control method for an energy storage power station based on a market environment, provided by an embodiment of the present invention.

[0066] Figure 2 This is a schematic diagram of the reward value curve provided in an embodiment of the present invention;

[0067] Figure 3 This is a comparative schematic diagram of energy storage configuration schemes provided in the embodiments of the present invention. Detailed Implementation

[0068] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0069] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0070] Example 1

[0071] like Figure 1 As shown in the figure, this embodiment introduces a production control method for energy storage power stations based on market environment, including:

[0072] Obtain operational data of the target power grid, including charging / discharging power and time-of-use electricity prices for the corresponding time periods.

[0073] This embodiment constructs an accurate profile of the power grid's operating status by collecting dynamic electricity price and charging / discharging power data in real time. This provides a highly timely and multi-dimensional input basis for the energy storage configuration scheme of energy storage power stations, ensuring that the TD3 intelligent agent can respond to real-time market fluctuations and improve the accuracy and scenario adaptability of the energy storage configuration scheme.

[0074] Based on the operational data, the optimal energy storage configuration scheme is output by predicting the optimal energy storage configuration scheme according to the pre-trained TD3 agent. The energy storage configuration scheme includes the charging power / discharging power setting value of the energy storage power station and the corresponding market price data.

[0075] This embodiment utilizes the continuous decision-making capability of the TD3 reinforcement learning algorithm to achieve rapid convergence of energy storage configuration schemes under complex electricity price and load uncertainty scenarios. At the same time, it ensures decision compliance through a nested mechanism of safety operation constraints, effectively breaking through the scalability bottleneck of traditional mixed integer programming methods under real-time requirements, and achieving the maximization of total revenue and comprehensive optimization of grid operation.

[0076] In this embodiment, the TD3 agent is trained using a reinforcement learning algorithm, and the reward function of the reinforcement learning algorithm aims to maximize the product of the electrical energy allocation amount and the corresponding electricity price allocation value.

[0077] This embodiment guides the TD3 agent to focus on improving economic benefits during training by designing a reward function that optimizes the product of the energy allocation amount and the corresponding electricity price allocation value. This ensures that the energy storage energy allocation scheme is highly coordinated with the grid market operation goals and enhances the economic benefit orientation of decision-making.

[0078] In this embodiment, the penalty term of the reward function includes the deviation between the current energy storage charging power and the historical best energy storage charging power / the deviation between the energy storage discharging power and the historical best energy storage discharging power, as well as the energy storage state of charge constraint violation rate.

[0079] This embodiment uses a dual-constraint design with a power deviation penalty term and a charge state constraint violation rate penalty term to guide the agent to actively learn from historical optimal power experience, reduce decision fluctuations, and strictly constrain energy storage to operate within a safe charge range, effectively balancing economic benefits and operational safety, and improving the robustness and operability of decision-making.

[0080] Example 2

[0081] Based on the same inventive concept as Embodiment 1, this embodiment introduces the implementation steps of a market-based energy storage power station production control method, including:

[0082] Step 1: Obtain the operating data of the target power grid, including charging power / discharging power and time-of-use electricity price for the corresponding time period.

[0083] Step 2: Based on the operational data, make predictions according to the pre-trained TD3 agent and output the optimal energy storage configuration scheme.

[0084] In this embodiment, the training method for the TD3 agent includes:

[0085] Step 2.1: Collect historical operating data of the target power grid, including historical charging power / historical discharging power and historical time-of-use electricity prices for the corresponding time periods.

[0086] Step 2.2: Using the Transformer model with embedded mean square error loss function and time series mean squared error loss function, construct a mapping set between electricity spot market price and charging power / electricity spot market price and discharging power based on the supply and demand relationship between historical charging power and historical time-of-use electricity price for the corresponding period / the supply and demand relationship between historical discharging power and historical time-of-use electricity price for the corresponding period, and form an electricity price-power experience pool.

[0087] In this embodiment, the mean squared error loss function is expressed as:

[0088] ;

[0089] In the formula, This represents the loss value of the mean squared error loss function. Represents a time series set. Indicates time Predicted energy storage charging power / energy storage discharging power , , This represents the supply and demand relationship between the historical optimal charging power of energy storage and the historical time-of-use electricity price for the corresponding period / the supply and demand relationship between the historical optimal discharging power of energy storage and the historical time-of-use electricity price for the corresponding period. Indicates time The corresponding time-of-use electricity price, Indicates time The corresponding historical time-of-use electricity price, Indicates time The corresponding historical charging power / historical discharging power Represents the strategy function.

[0090] In this embodiment, the time-series mean squared error loss function is expressed as:

[0091] ;

[0092] In the formula, This represents the loss value of the time-series mean squared error loss function. Indicates time The corresponding historical best charging power / historical best discharging power of energy storage. Indicates time The weighting coefficients corresponding to the deviation between the current energy storage charging power and the historical best charging power / the deviation between the energy storage discharging power and the historical best discharging power. , This represents the sine function.

[0093] Step 2.3: Dynamically embed the electricity price-power experience pool at the input end of the reinforcement learning environment parameters, and call the electricity price corresponding to the historical best energy storage charging power / the electricity price corresponding to the historical best energy storage discharging power as the initial configuration value of the electricity price.

[0094] Step 2.4: Improve the MILP-TD3 algorithm by introducing energy storage state of charge constraints. Input the initial electricity price configuration value into the TD3 agent, define the energy storage energy configuration scheme as a Markov decision problem, and perform iterative training based on the improved MILP-TD3 algorithm to generate the optimal energy storage energy configuration scheme.

[0095] The energy storage configuration scheme includes the amount of energy configured and the corresponding electricity price.

[0096] In this embodiment, the Markov decision problem is represented as:

[0097] ;

[0098] In the formula, Represents the set of state spaces. Represents the action space set, Represents the state transition probability. Represents the set of reward values. This represents the reward discount factor.

[0099] In the state space set, the state space includes energy storage power configuration schemes, predicted energy storage power configuration schemes, time-of-use pricing, and energy storage state of charge. The state values ​​in the state space are represented as follows:

[0100] ;

[0101] In the formula, Indicates time The corresponding state value, Indicates time The corresponding energy storage state of charge value.

[0102] In the set of action spaces, the action space includes the maximum energy storage charging power / discharging power and the energy storage state of charge, and the action space is represented as:

[0103] ;

[0104] ;

[0105] ;

[0106] In the formula, This indicates the maximum charging / discharging power of the energy storage. Indicates time The corresponding energy storage state of charge value.

[0107] In this embodiment, the state transition probability is expressed as:

[0108] ;

[0109] In the formula, Represents the state transition probability. Indicates time The corresponding state of the TD3 agent, Indicates time The corresponding state of the TD3 agent, Indicates time The corresponding actions taken by the TD3 agent Indicates time The corresponding state value, Indicates time The corresponding state value, Represents a probability function. Indicates time The corresponding action value of the TD3 agent.

[0110] In this embodiment, the reward value set is the output value set of the reward function, and the reward function is expressed as:

[0111] ;

[0112] In the formula, Represents the reward function, This represents the penalty coefficient for the deviation between the energy storage charging power and the historical best charging power / the deviation between the energy storage discharging power and the historical best discharging power. The penalty coefficient representing the violation rate of the energy storage state of charge constraint. It represents the product of the electrical energy allocation amount and the corresponding electricity price allocation value.

[0113] In this embodiment, the improved MILP-TD3 algorithm includes an actor network and a critic network. During iterative training, the parameters of the actor network and the critic network are first randomly initialized, and then the parameters of the actor network at time step [missing information]. Sample the corresponding state and execute the time. The corresponding actions interact with the environment and are transferred to the moment. For the corresponding state, a penalty term is added to the reward function using a DNN formula in the form of MILP to obtain the time step. Corresponding rewards to form a sample The parameters of the actor network and the critic network are updated by selecting the sample corresponding to the optimal energy storage power configuration scheme from the electricity price-power experience pool.

[0114] The parameter update expression for the actor network is expressed as:

[0115] ;

[0116] In the formula, The parameters representing the actor network, This indicates calculating the gradient matrix of the parameters of the actor network. Indicates time The corresponding critic network, Represents actor network In state The energy storage configuration scheme adopted below, Represents a set of states. This represents a function to find the minimum value;

[0117] The parameter update expression for the critic network is expressed as:

[0118] ;

[0119] In the formula, Represents the parameters of the critic network Find the minimum value. Indicates time The corresponding network Q value, Represents an actor network At time Corresponding state The energy storage configuration scheme adopted below, Indicates the first One neuron;

[0120] The output vector expression is as follows:

[0121] ;

[0122] In the formula, Indicates the first The output vector of a layer of neurons This indicates that the ReLU function is activated. Indicates the preset first The weight coefficient matrix of each neuron layer Indicates the first The bias vector of each neuron layer;

[0123] The MILP-form DNN formula is expressed as follows:

[0124] ;

[0125] In the formula, This represents the total number of layers in the DNN network. Indicates the first The total number of neurons in each neuron layer Indicates the first The first neuron layer The output vector of each neuron The constraint constants, Indicates the first The first neuron layer Input vector of each neuron The constraint constants;

[0126] The constraints of the MILP-form DNN formula are expressed as follows:

[0127] ;

[0128] In the formula, Indicates the first The total number of neurons in each neuron layer Indicates the preset first The first neuron layer The first neuron and the first The weight coefficient matrix of each neuron. Indicates the first The first neuron layer The output vector of each neuron Indicates the first The first neuron layer The bias vector of each neuron Indicates the first The first neuron layer Integer variable values ​​for each neuron;

[0129] The first The first neuron layer The output vector of each neuron Constraints and the first The first neuron layer Input vector of each neuron The constraints are expressed as follows:

[0130] ;

[0131] ;

[0132] In the formula, Indicates the first The first neuron layer The upper limit of the output vector of a neuron. Indicates the first The first neuron layer The lower bound of the input vector of each neuron. Indicates the first The first neuron layer The upper limit of the output vector of a neuron. Indicates the first The first neuron layer The lower bound of the input vector of each neuron. This represents the upper limit of the output vector. This represents the lower bound of the output vector. This represents the lower bound of the input vector. This represents the upper limit of the input vector.

[0133] In this embodiment, the total reward value is represented as:

[0134] ;

[0135] In the formula, This indicates taking the maximum value. Represents the total reward value. This represents the learning rate of the reinforcement learning algorithm. This represents the training rounds after convergence. Indicates time The corresponding reward discount factor, Indicates time The corresponding reward value.

[0136] The TD3 agent is trained using a reinforcement learning algorithm. The reward function of the reinforcement learning algorithm aims to maximize the product of the energy allocation amount and the corresponding electricity price allocation value. The penalty term of the reward function includes the deviation between the current energy storage charging power and the historical best energy storage charging power, the deviation between the energy storage discharging power and the historical best energy storage discharging power, and the energy storage state of charge constraint violation rate.

[0137] Step 3: Case Analysis:

[0138] The reinforcement learning parameters used in this embodiment are shown in Table 1.

[0139] Table 1 Reinforcement Learning Parameters

[0140] State space dimension 4 Discount factor 0.99 Action space dimension 2 noise 0.3 Learning rate 3e-4 <![CDATA[c1]]> 0.05 Experience pool capacity 10000 <![CDATA[c2]]> 0.1

[0141] Figure 2 This is a schematic diagram of the reward value curve in this embodiment. Figure 3This is a comparative diagram of the energy storage configuration schemes provided in this embodiment. It shows the comparison curves between the optimal energy storage configuration scheme obtained by the prior art using only reinforcement learning and the optimal energy storage configuration scheme obtained by the embodiment of this invention. It illustrates that compared with the prior art, the present invention fully incorporates historical experience, has the characteristics of good convergence and faster calculation speed, and solves the problem that the prior art cannot solve complex energy storage configuration schemes due to insufficient historical experience mining. At the same time, the safety of energy storage operation is ensured by the penalty term for violation of the state of charge constraint. Finally, it maximizes the product of the energy configuration amount and the corresponding electricity price configuration value and improves the comprehensive optimization capability of the power grid operation.

[0142] Example 3

[0143] Based on the same inventive concept as other embodiments, this embodiment describes a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the methods of Embodiment 1 or 2 described above.

[0144] Example 4

[0145] Based on the same inventive concept as other embodiments, this embodiment introduces a computer program product, including computer instructions that, when executed by a processor, implement the steps of the methods described in Embodiment 1 or 2 above.

[0146] In summary, this invention, through the TD3 agent, directly obtains the optimal energy storage configuration scheme by maximizing the product of the energy configuration amount and the electricity price configuration value in the reward function. Simultaneously, by applying a penalty term to the dual constraints of charging / discharging power deviation and state-of-charge constraint violation rate, the energy storage power station can operate efficiently while strictly adhering to battery safety boundaries. This reduces energy loss caused by power deviation and accelerated battery aging caused by exceeding the state-of-charge limit, thus solving the problems of low production control efficiency and poor returns in existing energy storage power stations.

[0147] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A production control method for an energy storage power station based on a market environment, characterized in that, include: Obtain operational data of the target power grid, including charging or discharging power and time-of-use electricity prices for the corresponding time periods; Based on the operational data, the optimal energy storage configuration scheme is output by predicting the optimal energy storage configuration based on the pre-trained TD3 agent. Utilize the optimal energy storage configuration scheme to control the production of energy storage power stations; The TD3 agent is trained using a reinforcement learning algorithm. The reward function of the reinforcement learning algorithm aims to maximize the product of the energy configuration amount and the corresponding electricity price configuration value. The penalty term of the reward function includes the deviation between the current energy storage charging power and the energy storage historical best charging power or the energy storage discharging power and the energy storage historical best discharging power, as well as the energy storage state of charge constraint violation rate. The training method for the TD3 agent includes: Collect historical operating data of the target power grid, including historical charging power or historical discharging power and historical time-of-use electricity prices for the corresponding time periods; Using a Transformer model with embedded mean square error loss function and time series mean square error loss function, based on the supply and demand relationship between historical charging power and historical time-of-use electricity price for the corresponding period, or the supply and demand relationship between historical discharging power and historical time-of-use electricity price for the corresponding period, a mapping set between electricity spot market price and charging power or electricity spot market price and discharging power is constructed, and an electricity price-power experience pool is formed. The electricity price-power experience pool is dynamically embedded at the input end of the reinforcement learning environment parameters, and the electricity price corresponding to the historical best energy storage charging power or the electricity price corresponding to the historical best energy storage discharging power is called as the initial configuration value of the electricity price. The MILP-TD3 algorithm is improved by introducing energy storage state of charge constraints. The initial electricity price configuration value is input into the TD3 agent, and the energy storage energy configuration scheme is defined as a Markov decision problem. The optimal energy storage energy configuration scheme is generated by iterative training based on the improved MILP-TD3 algorithm. The energy storage configuration scheme includes the amount of energy configured and the corresponding electricity price.

2. The energy storage power station production control method based on market environment according to claim 1, characterized in that, The mean squared error loss function is expressed as follows: ； In the formula, This represents the loss value of the mean squared error loss function. Represents a time series set. Indicates time Predicted energy storage charging power or energy storage discharging power, , , This indicates the supply and demand relationship between the historical optimal charging power of energy storage and the historical time-of-use electricity price for the corresponding period, or the supply and demand relationship between the historical optimal discharging power of energy storage and the historical time-of-use electricity price for the corresponding period. Indicates time The corresponding time-of-use electricity price, Indicates time The corresponding historical time-of-use electricity price, Indicates time The corresponding historical charging power or historical discharging power, Represents the decision function; The time-series mean squared error loss function is expressed as follows: ； In the formula, This represents the loss value of the time-series mean squared error loss function. Indicates time The corresponding historical best charging power or historical best discharging power of energy storage, Indicates time The weighting coefficients corresponding to the deviation between the current energy storage charging power and the historical best charging power, or the deviation between the energy storage discharging power and the historical best discharging power. , This represents the sine function.

3. The energy storage power station production control method based on market environment according to claim 2, characterized in that, The Markov decision problem is represented as follows: ； In the formula, Represents the set of state spaces. Represents the action space set, Represents the state transition probability. Represents the set of reward values. This represents the reward discount factor.

4. The energy storage power station production control method based on market environment according to claim 3, characterized in that, In the state space set, the state space includes energy storage power configuration schemes, predicted energy storage power configuration schemes, time-of-use pricing, and energy storage state of charge. The state values ​​in the state space are represented as follows: ； In the formula, Indicates time The corresponding state value, Indicates time The corresponding energy storage state of charge value.

5. The energy storage power station production control method based on market environment according to claim 4, characterized in that, In the set of action spaces, the action space includes the maximum energy storage charging power or discharging power and the energy storage state of charge, and the action space is represented as: ； ； ； In the formula, Indicates the maximum charging or discharging power of the energy storage. Indicates time The corresponding energy storage state of charge value.

6. The energy storage power station production control method based on market environment according to claim 5, characterized in that, The state transition probability is expressed as: ； In the formula, Represents the state transition probability. Indicates time The corresponding state of the TD3 agent, Indicates time The corresponding state of the TD3 agent, Indicates time The corresponding actions taken by the TD3 agent Indicates time The corresponding state value, Indicates time The corresponding state value, Represents a probability function. Indicates time The corresponding action value of the TD3 agent.

7. The energy storage power station production control method based on market environment according to claim 6, characterized in that, The set of reward values ​​is the set of output values ​​of the reward function, which is expressed as: ； In the formula, Represents the reward function, This represents the penalty coefficient for the deviation between the energy storage charging power and the historical best charging power, or the deviation between the energy storage discharging power and the historical best discharging power. The penalty coefficient representing the violation rate of the energy storage state of charge constraint. It represents the product of the electrical energy allocation amount and the corresponding electricity price allocation value.

8. The energy storage power station production control method based on market environment according to claim 7, characterized in that, The improved MILP-TD3 algorithm includes an actor network and a critic network. During iterative training, the parameters of the actor network and the critic network are first randomly initialized, and then the parameters of the actor network are adjusted according to the time step. Sample the corresponding state and execute the time. The corresponding actions interact with the environment and are transferred to the moment. For the corresponding state, a penalty term is added to the reward function using a DNN formula in the form of MILP to obtain the time step. Corresponding rewards to form a sample The parameters of the actor network and the critic network are updated by selecting the sample corresponding to the optimal energy storage power configuration scheme from the electricity price-power experience pool. The parameter update expression for the actor network is expressed as: ； In the formula, The parameters representing the actor network, This indicates calculating the gradient matrix of the parameters of the actor network. Indicates time The corresponding critic network, Represents actor network In state The energy storage configuration scheme adopted below Represents a set of states. This represents a function to find the minimum value; The parameter update expression for the critic network is expressed as: ； In the formula, Represents the parameters of the critic network Find the minimum value. Indicates time The corresponding network Q value, Represents actor network At time Corresponding state The energy storage configuration scheme adopted below Indicates the first One neuron; The expression for the output vector is: ； In the formula, Indicates the first The output vector of a layer of neurons This indicates that the ReLU function is activated. Indicates the preset first The weight coefficient matrix of each neuron layer Indicates the first The bias vector of each neuron layer; The MILP-form DNN formula is expressed as follows: ； In the formula, This represents the total number of layers in the DNN network. Indicates the first The total number of neurons in each neuron layer Indicates the first The first neuron layer The output vector of each neuron The constraint constants, Indicates the first The first neuron layer Input vector of each neuron The constraint constants; The constraints of the MILP-form DNN formula are expressed as follows: ； In the formula, Indicates the first The total number of neurons in each neuron layer Indicates the preset first The first neuron layer The first neuron and the first The weight coefficient matrix of each neuron. Indicates the first The first neuron layer The output vector of each neuron Indicates the first The first neuron layer The bias vector of each neuron Indicates the first The first neuron layer Integer variable values ​​for each neuron; The first The first neuron layer The output vector of each neuron Constraints and the first The first neuron layer Input vector of each neuron The constraints are expressed as follows: ； ； In the formula, Indicates the first The first neuron layer The output vector of each neuron The lower limit value, Indicates the first The first neuron layer The output vector of each neuron The upper limit, Indicates the first The first neuron layer Input vector of each neuron The lower limit value, Indicates the first The first neuron layer Input vector of each neuron The upper limit, This represents the lower bound of the output vector. This represents the lower bound of the input vector. This represents the upper limit of the output vector. This represents the upper limit of the input vector.

9. The energy storage power station production control method based on market environment according to claim 8, characterized in that, The total reward value is expressed as: ； In the formula, This indicates taking the maximum value. This represents the total reward value. This represents the learning rate of the reinforcement learning algorithm. This represents the training rounds after convergence. Indicates time The corresponding reward discount factor, Indicates time The corresponding reward value.

Images