A comprehensive energy storage system optimization method based on constraint-aware reinforcement learning

Abstract

This invention relates to an optimization method for integrated energy storage systems based on constraint-aware reinforcement learning, comprising: constructing a Q-value network, wherein the Q-value network includes several neural units, and the activation function of the neural units is the ReLU function; taking the output power of the gas turbine unit and the output power of the energy storage unit as actions, the output power of the wind power system, the load power, and the state of charge of the energy storage system as states, and establishing a reward function with the operating cost of the gas turbine unit, the cost of purchasing and selling electricity, and the power imbalance as penalty terms, and training the Q-value network to obtain the optimal parameters of the Q-value network; fixing the Q-value network parameters, converting the Q-value network into a mixed integer programming model, wherein the mixed integer programming model has microgrid constraints; inputting the current state into the mixed integer programming model, solving the mixed integer programming model through a gurobi solver, and obtaining the optimal action, i.e., the energy dispatch scheme.

Description

Technical Field

[0001] This invention relates to an optimization method for integrated energy storage systems based on constraint-aware reinforcement learning, belonging to the field of energy system optimization. Background Technology

[0002] The development of integrated energy systems supports the integration of diversified energy resources, including solar, wind, and bioenergy, and promotes the efficient use of energy and innovation in energy storage technologies. Currently, mathematical programming or reinforcement learning algorithms are commonly used to solve integrated energy optimization problems. Mathematical programming involves establishing a mathematical model of the research object and then using tools such as Cplex to find the optimal solution, thus achieving optimization. This method has high computational efficiency, but it struggles to obtain globally optimal solutions for many uncertainties in energy systems, such as weather and price fluctuations. Reinforcement learning algorithms effectively address this problem; they can interactively learn the dynamics of the energy system, providing high-quality solutions. However, these solutions may not meet the operational constraints of the energy system, including power balance constraints and generator uplink / downlink constraints. Therefore, a more computationally efficient energy dispatch optimization method that satisfies the operational constraints of the energy system is needed.

[0003] The patent with publication number CN117710146A, entitled "Optimization Operation Method and Device for Multi-Microgrid System Based on Hierarchical Constraint Reinforcement Learning", discloses the following steps: decomposing the optimization task of the integrated energy multi-microgrid system into an upper-layer agent and a lower-layer microgrid model, constructing the agent state space and agent action space of the upper-layer agent, constructing the constraint conditions of the lower-layer microgrid model, and solving the hierarchical reinforcement learning optimization scheduling model based on the constraint reinforcement learning algorithm to obtain the optimization operation strategy of the integrated energy multi-microgrid system. Summary of the Invention

[0004] To overcome the problems existing in the prior art, this invention designs an optimization method for integrated energy storage systems based on constraint-aware reinforcement learning.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] Technical Solution 1

[0007] An optimization method for integrated energy storage systems based on constraint-aware reinforcement learning includes the following steps:

[0008] Construct a Q-value network, which includes several neural units, and the activation function of each neural unit is the ReLU function;

[0009] The output power of the gas turbine unit and the output power of the energy storage unit are taken as actions, the output power of the wind power system, the load power, and the state of charge of the energy storage system are taken as states, and the operating cost of the gas turbine unit, the cost of purchasing and selling electricity, and the power imbalance are taken as penalty terms to establish a reward function. The Q-value network is trained to obtain the optimal parameters of the Q-value network.

[0010] With fixed Q-value network parameters, the Q-value network is converted into a mixed-integer programming model, which includes microgrid constraints.

[0011] Input the current state into the mixed integer programming model, solve the mixed integer programming model using the gurobi solver, and obtain the optimal action, i.e., the energy scheduling scheme.

[0012] Furthermore, the mixed-integer programming model is expressed by the following formula:

[0013]

[0014]

[0015] In the formula, x i Indicates the i-th action; y j z represents the j-th state; ij It is the reward value corresponding to taking action i in state j; Q(·) represents the Q-value function.

[0016] Furthermore, the training of the Q-value network includes the following steps:

[0017] Initialize the Q-value network, the target Q-value network, the parameters in the execution policy function, initialize the response buffer, and input environment parameters;

[0018] Based on the execution policy function and noise sampling actions; interact with the environment, calculate rewards, generate the next state, and store the new actions, rewards, and states in the response buffer; repeat this process until the training time exceeds the threshold;

[0019] Randomly sample from the response buffer to compute the loss function; backpropagate the loss through the gradient of the neural network to update the Q-value network parameters; update the parameters in the execution policy function; update the target Q-value network parameters; if the Q-value network parameters converge, the training is complete; otherwise, repeat the steps.

[0020] Furthermore, the reward function is expressed by the formula:

[0021]

[0022] In the formula, σ1 and σ2 are the penalty factors for cost minimization and power imbalance, respectively; Let i be the operating cost of the i-th gas turbine unit. The cost of purchasing electricity from the main grid; ΔP is the active power imbalance.

[0023] Furthermore, the microgrid constraints include power balance constraints and uphill and downhill constraints of the gas turbine unit.

[0024] Furthermore, the power balance constraint is characterized by the following formula:

[0025]

[0026] In the formula, These are the number of wind power generation systems, the number of energy storage systems, and the number of load groups, respectively. Output / input power to the main grid, x G x V x N x B x L These are binary variables corresponding to the gas turbine unit, wind power generation system unit, energy storage system unit, and load unit, respectively. Binary constraints are used to select whether to connect.

[0027] Furthermore, the uphill and downhill constraints of the gas turbine unit are expressed by the following formulas:

[0028]

[0029]

[0030] In the formula, RU i RD i These represent the pressure boosting and pressure reduction capabilities of the gas turbine unit, x t-1 x t x t+1 These are binary variables corresponding to different moments during the start-up and shutdown process of the gas turbine unit, and the start-up and shutdown status of the unit is controlled by binary constraints.

[0031] Technical Solution Two

[0032] An integrated energy storage system optimization device based on constraint-aware reinforcement learning includes a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor. The processor executes the program to implement the steps described in technical solution one.

[0033] Compared with the prior art, the present invention has the following features and beneficial effects:

[0034] This invention utilizes deep reinforcement learning algorithms to train Q-value networks, which can effectively solve the uncertainty problem in integrated energy systems and the problem of the rapid increase in computational complexity as the system scale increases. At the same time, by using binary constraints to model the Q-value network function as a mixed integer programming model, combined with the constraints of various parts in the microgrid, all operational constraints in the system can be strictly enforced, ensuring the feasibility of the scheduling scheme in real-time operation. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating the principle and flow of the constraint-aware reinforcement learning algorithm according to an embodiment of the present invention.

[0036] Figure 2 This is a detailed flowchart illustrating the constraint-aware reinforcement learning algorithm according to an embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram of the energy and environment according to an embodiment of the present invention.

[0038] Figure 4 This is a prediction diagram of various loads and fan output in an embodiment of the present invention.

[0039] Figure 5 This is the optimal scheduling scheme in this embodiment of the invention. Detailed Implementation

[0040] The present invention will now be described in more detail with reference to the embodiments.

[0041] Example 1

[0042] like Figure 1 as well as Figure 2 As shown, this embodiment provides a comprehensive energy storage system optimization method based on constraint-aware reinforcement learning, including the following steps:

[0043] S1. Construct a mathematical model for optimal energy storage scheduling:

[0044] Construct an objective function that minimizes the day-ahead operating cost of an integrated energy storage system:

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051] In the formula, These are, respectively, the cost of electricity purchase, the cost of gas purchase, the cost of heat purchase, the cost of wind and solar curtailment penalties, and the operating cost of the i-th unit of the gas turbine system; The penalty cost per unit of power purchased for electricity, gas, heat, wind power curtailment, and solar power curtailment per unit of time; T is the dispatch cycle; P is the penalty cost per unit of power purchased for electricity, gas, heat, wind power curtailment, and solar power curtailment per unit of time; t E P t Gas , For purchasing electricity, gas, and heat from the main grid; P t w P t v The amount of wind and solar power curtailed per unit time; This represents the power output of the i-th unit in the gas turbine system. Unit operating cost per generating unit; U t,e S is the start-up and shutdown variable for the operating unit. e denoted as , where is the start-up and shutdown cost of the unit; n represents the number of gas turbine units in the system.

[0052] S2. Construct microgrid operation constraints, including:

[0053] Power balance constraints:

[0054]

[0055] In the formula, For wind power generation system output power, power output / input to main grid, energy storage system output power, and average load power; G, V, B, and L represent gas turbine system, wind power generation system, energy storage system, and load system, respectively; x i,G x i,V x i,N x i,B x i,L These are binary variables corresponding to gas turbine units, wind turbine units, energy storage units, and load units, respectively. Binary constraints are used to select whether to connect to the microgrid.

[0056] Uphill and downhill constraints for gas turbine units:

[0057]

[0058]

[0059] In the formula, RU i RD i These represent the pressure boosting and pressure reduction capabilities of the gas turbine unit, x t-1 x t x t+1These are binary variables corresponding to different times (t-1, t, t+1) during the start-up and shutdown process of the gas turbine unit, and the start-up and shutdown status of the unit is controlled by binary constraints.

[0060] S3, Offline training of DQN network:

[0061] S31. Construct a deep neural network with K hidden layers, called a DQN network. Each hidden layer has several neurons and uses the ReLU activation function. The functional form of the Q-value network is the Q-value function Q. θ The functional form of the target Q-value network is the target Q-value function Q. θ target.

[0062] The Q-value function is defined as follows:

[0063] Q θ (s t ,a t )=(1-α)·Q θ (s t ,a t )+α·(r t +γ·max a' Q θ (s t ',a t '))

[0064] In the formula, θ is the function Q θ The parameter in; r t γ is the reward function; γ is the discount factor; s t For state variables; a t For action variables. max a' Q θ (s t ',a t ') indicates that in the next state s', the action with the largest Q value is selected from all possible actions.

[0065] The state space, actions, and reward function are defined as follows:

[0066]

[0067] In the formula, Let t be the output power of the wind power generation system. Let t be the load system power. The power output of the gas turbine system at time t-1; SOC t Let t be the state of charge of the energy storage system.

[0068]

[0069] In the formula, Let be the output power of the i-th gas turbine unit at time t. Let be the output power of the i-th energy storage unit at time t.

[0070]

[0071] In the formula, σ1 and σ2 are the penalty factors for cost minimization and power imbalance, respectively; Let i be the operating cost of the i-th gas turbine unit. The cost of purchasing electricity from the main grid; ΔP is the active power imbalance.

[0072] S32, Q-initialization function Q θ Target Q-value function Q θ target, execution strategy function π ω The parameters in the input environment include the initialization of the response buffer R and its size L.

[0073] S33. Based on the execution strategy function and noise sampling action a T ~π ω (s t )+ε,ε~N(0,σ); interact with the environment, calculate the reward rt, and generate a new state s. t+1 , will (s t ,a t ,r t ,s t+1 Store in the response buffer; repeat step S33 until the training duration exceeds the threshold.

[0074] S34. Randomly select a small batch (s) from R. t ,a t ,r t ,s t+1 The sample is used to calculate the loss function, which is defined as the mean squared error based on the Bellman variance, and the calculation formula is as follows:

[0075]

[0076] In the formula, θ is the function Q θ The parameter in the equation; |B| is the number of samples.

[0077] S35. Update the Q-value network parameters by backpropagating the loss through the gradient of the neural network.

[0078] S36. Update the parameter ω in the execution strategy function, using the following formula:

[0079]

[0080] S37. Update the target Q-value network parameters θ target The updated formula is shown below:

[0081] θ target ←τθ+(1-τ)θ target

[0082] In the formula, τ is the soft update coefficient.

[0083] S38. If the Q-value function Q θ If the parameters converge, proceed to the next step; otherwise, repeat steps S33-S37.

[0084] S4. Convert the Q-value network into a mixed-integer programming model:

[0085] The network parameters have fixed Q-values; the binary constraints limit the variables to only one of two possible values, typically 0 or 1. This invention utilizes binary variables to represent decisions in a DQN network. For example: the Q-value of each action in a DQN network is determined by parameters. To represent these decisions in a mixed-integer programming model, we first define a binary variable x. i,j Let x represent the state and j represent the action. If action j is chosen to be executed in state i, then x i,j =1 otherwise 0, to determine whether to connect various energy sources in the microgrid. Secondly, the objective function and constraints are defined using Q-values ​​and binary variables, where each power of the microgrid can be represented as a corresponding Q-value parameter. This constraint ensures that only one action is selected in each state. The DQN network function is modeled as an efficient mixed-integer programming model based on the binary constraints, ensuring that the Q-value function can accurately predict the Q-value for a given state and action, expressed by the formula:

[0086]

[0087]

[0088] Where, x i Let y represent the i-th action variable, i.e., the output power of the gas turbine unit or the output power of the energy storage system. j Let z represent the j-th state variable, i.e., the output power, load power, or energy storage charging state of the wind power generation system. ij It is the reward value corresponding to taking action i under state j. The reward consists of the sum of the gas turbine unit operating cost, the cost of purchasing and selling electricity, and the power imbalance.

[0089] The mixed-integer programming model also includes the microgrid operation constraints.

[0090] S5. Input the current environment, use the gurobi solver to solve the mixed integer programming model to obtain the optimal scheduling scheme and then execute the energy scheduling.

[0091] Example 2

[0092] like Figure 3 , Figure 4 as well as Figure 5 As shown, Python was used for simulation analysis. The original dataset was the annual demand for electricity, heat, gas and wind power generation in a certain region, and the data was sampled in hourly units. Figure 3 The energy environment used in this embodiment mainly consists of wind turbine generators, energy storage, and combined heat and power units. Figure 4 The graphs showing the load and turbine output forecasts are displayed. The original dataset is further divided into two datasets: a training dataset and a test dataset. The training dataset contains the first three weeks of each month, while the test dataset contains the remaining data. This allows the deep reinforcement learning algorithm to learn any seasonal and weekly behavior in the wind power generation and load data. Figure 5 This embodiment presents the optimal scheduling scheme obtained using the method proposed in this invention. The integrated energy storage system primarily utilizes wind power generation. During the period from 09:00 to 20:00, wind power output is high, allowing the system to absorb wind power without purchasing electricity from the upstream grid. Furthermore, any excess power can be converted into electrical energy through boilers, and a portion of this energy is stored in energy storage devices. During the nighttime periods from 01:00 to 05:00 and from 22:00 to 24:00, when electricity prices are low, the system can purchase electricity from the upstream grid for use in electrolysis cells to store the energy in hydrogen fuel cells. During peak electricity prices, the stored energy is used by gas turbine units to supply heat to the system. This allows the integrated energy system to fully utilize the energy storage devices during operation, effectively shaving off peak loads and reducing fluctuations.

[0093] It should be noted that the aforementioned integrated energy storage system optimization device based on constraint-aware reinforcement learning is also used to achieve the above-mentioned... Figure 1 The method steps corresponding to each embodiment of the integrated energy storage system optimization method based on constraint-aware reinforcement learning shown are not repeated here.

[0094] It should be noted that the functional units / modules in the various embodiments of the present invention can be integrated into one processing unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated into one unit / module. The integrated unit / module described above can be implemented in hardware or in the form of software functional units / modules.

[0095] From the above description of the embodiments, those skilled in the art will clearly understand that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code, or any suitable combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), processor, controller, microcontroller, microprocessor, other electronic units designed to implement the functions described herein, or combinations thereof. For software implementation, some or all of the processes of the embodiments can be implemented by a computer program instructing the associated hardware. During implementation, the program can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a computer. Computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having the form of instructions or data structures and accessible to a computer.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should be able to analyze that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A method for optimizing an integrated energy storage system based on constraint-aware reinforcement learning, characterized in that, Includes the following steps: Construct a Q-value network, which includes several neural units, and the activation function of each neural unit is the ReLU function; The output power of the gas turbine unit and the output power of the energy storage unit are taken as actions, the output power of the wind power system, the load power, and the state of charge of the energy storage system are taken as states, and the operating cost of the gas turbine unit, the cost of purchasing and selling electricity, and the power imbalance are taken as penalty terms to establish a reward function. The Q-value network is trained to obtain the optimal parameters of the Q-value network. With fixed Q-value network parameters, the Q-value network is converted into a mixed-integer programming model, which includes microgrid constraints. Input the current state into the mixed integer programming model, solve the mixed integer programming model using the gurobi solver, and obtain the optimal action, i.e., the energy scheduling scheme; The mixed-integer programming model is expressed by the following formula: ； ； In the formula, This represents the i-th action; This represents the j-th state; It is the reward value corresponding to taking action i in state j; Q(·) represents the Q-value function; The reward function is expressed by the formula: ； In the formula, , These are the penalty factors for cost minimization and power imbalance, respectively. Let i be the operating cost of the i-th gas turbine unit. Cost of purchasing electricity from the main grid; This refers to the imbalance of active power. The microgrid constraints include power balance constraints and uphill and downhill constraints of the gas turbine units.

2. The method for optimizing a comprehensive energy storage system based on constraint-aware reinforcement learning according to claim 1, characterized in that, The training of the Q-value network includes the following steps: Initialize the Q-value network, the target Q-value network, the parameters in the execution policy function, initialize the response buffer, and input environment parameters; Based on the execution policy function and noise sampling actions; interact with the environment, calculate rewards, generate the next state, and store the new actions, rewards, and states in the response buffer; repeat this process until the training time exceeds the threshold; Randomly sample from the response buffer to compute the loss function; backpropagate the loss through the gradients of the neural network to update the Q-value network parameters; update the parameters in the execution policy function; Update the target Q-value network parameters; if the Q-value network parameters converge, the training is complete; otherwise, repeat the step.

3. The method for optimizing a comprehensive energy storage system based on constraint-aware reinforcement learning according to claim 1, characterized in that, The power balance constraint is expressed by the following formula: ； In the formula, , , These are the number of wind power generation systems, the number of energy storage systems, and the number of load groups, respectively. Output / input power to the main grid, , , , These are binary variables corresponding to the gas turbine unit, wind power generation system unit, energy storage system unit, and load unit, respectively. Binary constraints are used to select whether to connect.

4. The method for optimizing a comprehensive energy storage system based on constraint-aware reinforcement learning according to claim 1, characterized in that, The uphill and downhill constraints of the gas turbine unit are expressed by the following formulas: ； ； In the formula, RU i , RD i These refer to the pressure boosting and pressure reduction capabilities of the gas turbine unit. , , These are binary variables corresponding to different moments during the start-up and shutdown process of the gas turbine unit, and the start-up and shutdown status of the unit is controlled by binary constraints.

5. A comprehensive energy storage system optimization device based on constraint-aware reinforcement learning, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the program, it performs the following steps: A Q-value network is pre-set, which includes several neural units, and the activation function of the neural units is the ReLU function. The output power of the gas turbine unit and the output power of the energy storage unit are taken as actions, the output power of the wind power system, the load power, and the state of charge of the energy storage system are taken as states, and the operating cost of the gas turbine unit, the cost of purchasing and selling electricity, and the power imbalance are taken as penalty terms to establish a reward function. The optimal parameters of the Q-value network are obtained by training the Q-value network. With fixed Q-value network parameters, the Q-value network is converted into a mixed-integer programming model, which includes microgrid constraints. Input the current state into the mixed integer programming model, solve the mixed integer programming model using the gurobi solver, and obtain the optimal action, i.e., the energy scheduling scheme; The mixed-integer programming model is expressed by the following formula: ； ； In the formula, This represents the i-th action; This represents the j-th state; It is the reward value corresponding to taking action i in state j; Q(·) represents the Q-value function; The reward function is expressed by the formula: ； In the formula, , These are the penalty factors for cost minimization and power imbalance, respectively. Let i be the operating cost of the i-th gas turbine unit. Cost of purchasing electricity from the main grid; This refers to the imbalance of active power. The microgrid constraints include power balance constraints and uphill and downhill constraints of the gas turbine units.

6. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the integrated energy storage system optimization method based on constraint-aware reinforcement learning as described in any one of claims 1 to 4.

Images