Distributed energy storage system optimization scheduling method based on LSTM and DQN algorithms

Abstract

The invention discloses a distributed energy storage system optimal scheduling method based on LSTM and DQN algorithms, and belongs to the technical field of energy storage system scheduling. The method comprises the following steps: preprocessing time sequence data of photovoltaic power generation and load demands, and dividing the data into a training set and a test set; training by using an LSTM network, minimizing a prediction error through hyper-parameter optimization, selecting and executing a scheduling action by using a DQN algorithm according to a current state, transferring to a next state, and obtaining rewards at the same time; the state, the action, the reward and the new state are stored in an experience playback pool, and the neural network weight of the DQN is updated through regularly sampled experience, so that the scheduling strategy of the model is optimized; and judging whether a termination state is reached or not and whether the Q value is converged or not until the model is converged and outputting an optimal scheduling strategy. The method can respond to requirements in real time, can significantly enhance the flexibility, response speed and reliability of the system, reduces the power transmission loss, and improves the overall energy utilization rate.

Description

Technical Field

[0001] The present invention relates to the technical field of energy storage system scheduling, and in particular to a distributed energy storage system optimization scheduling method based on LSTM and DQN algorithms. Background Art

[0002] As global climate change intensifies, countries are actively promoting the transformation of energy structures and promoting the development and utilization of renewable energy (such as wind power and solar energy). However, the intermittent nature and uncertainty of renewable energy pose huge challenges to the stable operation of the power system. In order to address this problem, distributed energy storage systems (DESS) are widely used to achieve energy storage, regulation and release, and improve the safety and stability of the power system. In this context, traditional energy storage scheduling methods face many problems:

[0003] 1. Single energy storage unit: Existing methods often only design one energy storage unit to balance power supply and power demand for multiple different buildings. This not only poses the risk of single point failure, but also leads to higher transmission losses, reducing the overall energy utilization efficiency.

[0004] 2. Lack of consideration of spatiotemporal characteristics: Although existing technologies apply deep reinforcement learning to optimize energy storage scheduling, most of the work does not fully consider the spatiotemporal characteristics in energy storage scheduling, ignoring the differences in resources in different geographical locations and the dynamic changes in the time dimension, thus affecting the effectiveness of the overall scheduling strategy.

[0005] 3. Insufficient ability to cope with uncertainty and dynamic environments: Traditional linear programming and dynamic programming methods are difficult to achieve real-time optimization in complex, dynamic and highly uncertain environments. In addition, relying solely on traditional time series forecasting methods or simple reinforcement learning algorithms often results in scheduling solutions that cannot adapt to rapidly changing electricity demand and the volatility of renewable energy.

[0006] Therefore, an optimal scheduling method for distributed energy storage systems based on LSTM and DQN algorithms is needed to improve the overall efficiency of the system and reduce energy loss. Summary of the invention

[0007] The purpose of the present invention is to propose a distributed energy storage system optimization scheduling method based on LSTM and DQN algorithms, comprising the following steps:

[0008] Preprocess the time series data of photovoltaic power generation and load demand and divide the data into training set and test set;

[0009] Use LSTM network for training, minimize prediction error through hyperparameter optimization, select and execute scheduling actions based on the current state using DQN algorithm, transfer to the next state, and obtain rewards at the same time;

[0010] Store states, actions, rewards, and new states in an experience replay pool, and update the DQN neural network weights through regularly sampled experience to optimize the model's scheduling strategy;

[0011] Determine whether the termination state is reached and whether the Q value converges until the model converges and outputs the optimal scheduling strategy.

[0012] Target Q value target k The calculation method is as follows:

[0013]

[0014] in, θ′ represents the output and parameters of the target network, r k represents the reward obtained by the agent for taking an action, γ represents the discount factor, and A′ represents the state S of the target network at the next moment according to the environment. k+1 The action with the largest Q value estimate is chosen.

[0015] The minimization loss function is:

[0016]

[0017] Among them, target k is the target Q value, θ is the network parameter, represents the expectation, that is, the entire loss function is the mean square error loss, S k represents the state of the environment at time k, A k Represents the agent based on the environment state S k The action performed at time k.

[0018] A distributed energy storage system optimization scheduling device based on LSTM and DQN algorithms, comprising:

[0019] The preprocessing module is used to preprocess the time series data of photovoltaic power generation and load demand and divide the data into training sets and test sets;

[0020] The training module is used to train the LSTM network, minimize the prediction error through hyperparameter optimization, select and execute the scheduling action based on the current state using the DQN algorithm, transfer to the next state, and obtain rewards at the same time;

[0021] The optimization module is used to store states, actions, rewards, and new states in the experience replay pool and update the DQN neural network weights through regularly sampled experience to optimize the model's scheduling strategy;

[0022] The judgment module is used to judge whether the termination state has been reached and whether the Q value has converged, until the model converges and outputs the optimal scheduling strategy.

[0023] Target Q value target in the optimization module k The calculation method is as follows:

[0024]

[0025] in, θ′ represents the output and parameters of the target network, r k represents the reward obtained by the agent for taking an action, γ represents the discount factor, and A′ represents the state S of the target network at the next moment according to the environment. k+1 The action with the largest Q value estimate is chosen.

[0026] The loss function minimized in the optimization module is:

[0027]

[0028] Among them, target k is the target Q value, θ is the network parameter, represents the expectation, that is, the entire loss function is the mean square error loss, S k represents the state of the environment at time k, A k Represents the agent based on the environment state S k The action performed at time k.

[0029] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of a distributed energy storage system optimization scheduling method based on LSTM and DQN algorithms is implemented.

[0030] A storage medium stores a computer program, which, when executed by a processor, implements various steps in a distributed energy storage system optimization scheduling method based on LSTM and DQN algorithms.

[0031] The beneficial effects of the present invention are:

[0032] 1. The present invention can respond to demand instantly by configuring distributed energy storage units for different buildings, which can significantly enhance the flexibility, response speed and reliability of the system, while reducing transmission losses and improving overall energy utilization.

[0033] 2. The present invention fully considers the temporal and spatial characteristics and incorporates the temporal and spatial change patterns into the scheduling model, which not only further improves the accuracy and applicability of the energy storage scheduling strategy, but also enables efficient allocation of energy in time and space, reduces the risk of local supply and demand imbalance, and improves the overall energy utilization and economic benefits of the system.

[0034] 3. The present invention proposes a distributed energy storage scheduling solution based on LSTM-DQN: LSTM predicts photovoltaic power generation and power load, and the prediction results are input into DQN as part of the system state to achieve the optimal charging and discharging decision. By combining LSTM and DQN methods, it is possible to adaptively cope with the complex and dynamic environment in energy storage scheduling, achieve real-time optimization, and enable the system to understand future power generation and load requirements in advance, and formulate more reasonable charging and discharging strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a system framework diagram of the distributed energy storage system optimization scheduling method based on LSTM-DQN;

[0036] Figure 2 Flowchart of the distributed energy storage system optimization scheduling method based on LSTM-DQN. DETAILED DESCRIPTION

[0037] The present invention proposes a distributed energy storage system optimization scheduling method based on LSTM and DQN algorithms, and the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0038] Figure 1 This is the system framework diagram of the distributed energy storage system optimization scheduling method based on LSTM-DQN, which mainly includes photovoltaic power plants, cogeneration units and lithium battery energy storage units, and takes into account that different buildings have corresponding distributed energy storage units.

[0039] Figure 2This is a flowchart of the distributed energy storage system optimization scheduling method based on LSTM-DQN. The LSTM-DQN algorithm first preprocesses the time series data of photovoltaic power generation and load demand, then divides the data into training sets and test sets, and uses the LSTM network for training to predict future power and demand. During the training process, the prediction error is minimized by hyperparameter optimization to improve the accuracy of the model. Then, the DQN algorithm is used to select scheduling actions (such as charging or discharging) according to the current state (such as battery charging state, electricity price, load demand, etc.), and the action is executed to transfer to the next state and obtain rewards at the same time. These states, actions, rewards, and new states will be stored in the experience replay pool, and the neural network weights of DQN will be updated through regularly sampled experience to optimize the scheduling strategy of the model. During the training process, the algorithm continuously checks whether the termination state is reached and whether the Q value converges. When the model converges and obtains the optimal scheduling strategy, the system outputs the strategy for actual operation, thereby achieving efficient energy storage management and optimal economic benefits in a complex dynamic environment.

[0040] The details and functions of each module are as follows:

[0041] 1. LSTM module: Long short-term memory network (LSTM) is an improved version of recurrent neural network (RNN), which is specially designed to process time series data. LSTM effectively solves the gradient vanishing and explosion problems that are prone to occur in standard RNN in long-term dependent tasks by introducing memory units and three gating mechanisms (input gate, forget gate and output gate). The LSTM module is used in this algorithm to predict the power generation of renewable energy (such as photovoltaic power generation) and power load demand. Since power generation and load demand are affected by time and external environmental factors (such as weather, season, etc.), LSTM can capture the time correlation and nonlinear characteristics in these data and provide more accurate future prediction values. With these prediction results, the system can better prepare for subsequent energy storage scheduling.

[0042] Specifically, LSTM contains the following key components. First, the forget gate f t The decision needs to be made from the previous memory state C t-1 What information is forgotten in the input gate i t Determine which parts of the current input need to be written to the current memory state. Candidate memory state Indicates new candidate memory information. Current memory state C t Update according to the results of the forget gate and the input gate. Finally, the output gate o t Controls how much information in the current memory state needs to be output as the hidden state h t The hidden state h tIt will be used for information transfer in the next time step. This gating mechanism enables LSTM to effectively solve the gradient vanishing problem and capture long-range dependencies in the sequence.

[0043]

[0044] 2.DQN module: Deep Q network (DQN) is an algorithm widely used in deep reinforcement learning to solve complex decision-making problems. The core function of the DQN module is to optimize the scheduling of the energy storage system based on the results of LSTM prediction. This module takes the state of the energy storage system (such as battery charging status, electricity price fluctuations, load demand, etc.) as input, and outputs Q values ​​through a neural network to evaluate the long-term rewards of each possible action (such as charging, discharging, and maintaining). DQN continuously learns the optimal scheduling strategy through continuous interaction with the environment. Through the experience replay mechanism, DQN learns from historical experience and updates the network weights to maximize the long-term cumulative rewards of the system. The important advantage of the DQN module is that it does not require precise modeling of the energy storage system, can adapt to complex environmental changes, and provide flexible and efficient scheduling strategies.

[0045] (1) State (S)

[0046] The state represents all relevant information of the energy storage system at a certain moment. In this embodiment, the state space includes the battery state of charge (SOC), photovoltaic power generation output information, electricity price fluctuations, natural gas price information, and power load and heat load. Through these states, the agent can perceive the current status of the power system and make scheduling decisions accordingly. The definition and construction of the state space is crucial to the adaptability and optimization effect of the algorithm, and determines the accuracy and feasibility of the scheduling strategy.

[0047] (2) Action (A)

[0048] Actions represent the operations chosen by the agent in each state. In the DQN model, the action set contains the charging and discharging decisions of the energy storage battery. Specifically, each user's energy storage battery can choose to charge, discharge, or remain unchanged. The action space consists of these charging and discharging operations, and the operation of energy storage is optimized by maximizing the cumulative reward. In this framework, different users can choose different charging and discharging powers in different states, thereby achieving optimal management of the battery.

[0049] (3) Reward (R)

[0050] The reward function measures the quality of each action, that is, the feedback of the system after performing a certain action. The higher the reward value, the more beneficial the action is to the system. The reward function is designed to guide the agent to choose actions that can reduce the operating cost of the system and improve efficiency. Specifically, the reward value is based on the operating cost of the power system (including gas purchase cost, electricity purchase cost, battery charging and discharging cost) and the system's responsiveness to load demand. If the battery capacity exceeds the reasonable range or the power demand and supply are unbalanced, the agent will be punished. For example, when the system load demand is greater than the power supply, additional electricity needs to be purchased from the grid, which increases the penalty.

[0051] (4) Dynamic Scheduling

[0052] First, an evaluation network is initialized to fit the Q-value function Q(s,a), where s represents the state, a represents the action, and θ represents the network parameters. Then, the experience replay pool is initialized to store the experience samples (S t ,A t ,R t ,S t+1 ). These samples include the current state S t , Action A t , the reward R t , and the next state S reached after executing the action t+1 At each time step t, according to the current state S t , using the ∈-greedy strategy to select an action A t Specifically, with probability 1-∈, select the action with the largest Q value, that is, the action with the highest expected reward. With probability ∈, randomly select an action to explore a new potential optimal strategy. Then execute the selected action A t (Charge and discharge), get reward R t , and observe the next state S t+1 The current experience is stored in the experience replay pool. Then, the experience replay is used for updating: a small batch of samples (S k ,A k ,R k ,S k+1 ), used for training. Target Q value target k The calculation method is as follows:

[0053]

[0054] in, θ′ represents the output and parameters of the target network, r krepresents the reward obtained by the agent for taking an action, γ represents the discount factor, and A′ represents the state S of the target network at the next moment according to the environment. k+1 The chosen Q-value estimates the largest action. Next, the parameters θ of the evaluation network are updated by minimizing the loss function:

[0055]

[0056] Among them, target k is the target Q value, θ is the network parameter, represents the expectation, that is, the entire loss function is the mean square error loss, S k represents the state of the environment at time k, A k Represents the agent based on the environment state S k The action performed at time k.

[0057] Every fixed number of steps, the parameters θ of the evaluation network are copied to the parameters θ′ of the target network:

[0058] θ′←θ (4)

[0059] The algorithm is executed repeatedly until the specified stopping condition is reached, such as reaching a certain convergence level or reaching a specified number of training iterations, and finally outputs the optimal scheduling strategy.

[0060] This embodiment also includes an electronic device and a storage medium. An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the distributed energy storage system optimization scheduling method based on the LSTM and DQN algorithms is implemented. A storage medium stores a computer program on it. When the computer program is executed by the processor, each step in the distributed energy storage system optimization scheduling method based on the LSTM and DQN algorithms is implemented.

[0061] The present invention can respond to demand instantly by configuring distributed energy storage units for different buildings, which can significantly enhance the flexibility, response speed and reliability of the system, while reducing transmission losses and improving overall energy utilization. Taking full account of the spatiotemporal characteristics and incorporating the temporal and spatial change patterns into the scheduling model, it not only further improves the accuracy and applicability of the energy storage scheduling strategy, but also enables efficient allocation of energy in time and space, reduces the risk of local supply and demand imbalance, and improves the overall energy utilization and economic benefits of the system. By combining the LSTM and DQN methods, it is possible to adaptively respond to the complex and dynamic environment in energy storage scheduling, achieve real-time optimization, and enable the system to understand future power generation and load requirements in advance, and formulate more reasonable charging and discharging strategies.

[0062] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0063] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0066] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0067] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A distributed energy storage system optimization scheduling method based on LSTM and DQN algorithms, characterized in that: The following steps are involved: Preprocess the time series data of photovoltaic power generation and load demand and divide the data into training set and test set; Use LSTM network for training, minimize prediction error through hyperparameter optimization, select and execute scheduling actions based on the current state using DQN algorithm, transfer to the next state, and obtain rewards at the same time; Store states, actions, rewards, and new states in an experience replay pool, and update the DQN neural network weights through regularly sampled experience to optimize the model's scheduling strategy; Determine whether the termination state is reached and whether the Q value converges until the model converges and outputs the optimal scheduling strategy.

2. The distributed energy storage system optimization scheduling method based on LSTM and DQN algorithm according to claim 1 is characterized in that: Target Q value target k The calculation method is as follows: in, θ′ represents the output and parameters of the target network, r k represents the reward obtained by the agent for taking an action, γ represents the discount factor, and A′ represents the state S of the target network at the next moment according to the environment. k+1 The action with the largest Q value estimate is chosen.

3. The distributed energy storage system optimization scheduling method based on LSTM and DQN algorithm according to claim 1 or 2 is characterized in that: The minimization loss function is: Among them, target k is the target Q value, θ is the network parameter, represents the expectation, that is, the entire loss function is the mean square error loss, S k represents the state of the environment at time k, A k Represents the agent based on the environment state S k The action performed at time k.

4. A distributed energy storage system optimization scheduling device based on LSTM and DQN algorithms, characterized in that: include: The preprocessing module is used to preprocess the time series data of photovoltaic power generation and load demand and divide the data into training sets and test sets; The training module is used to train the LSTM network, minimize the prediction error through hyperparameter optimization, select and execute the scheduling action based on the current state using the DQN algorithm, transfer to the next state, and obtain rewards at the same time; The optimization module is used to store states, actions, rewards, and new states in the experience replay pool and update the DQN neural network weights through regularly sampled experience to optimize the model's scheduling strategy; The judgment module is used to judge whether the termination state has been reached and whether the Q value has converged, until the model converges and outputs the optimal scheduling strategy.

5. The distributed energy storage system optimization scheduling device based on LSTM and DQN algorithms according to claim 4 is characterized in that: The target Q value target in the optimization module k The calculation method is as follows: in, θ′ represents the output and parameters of the target network, r k represents the reward obtained by the agent for taking an action, γ represents the discount factor, and A′ represents the state S of the target network at the next moment according to the environment. k+1 The action with the largest Q value estimate is chosen.

6. The distributed energy storage system optimization scheduling device based on LSTM and DQN algorithm according to claim 4 or 5, characterized in that: The loss function minimized in the optimization module is: Among them, target k is the target Q value, θ is the network parameter, represents the expectation, that is, the entire loss function is the mean square error loss, S k represents the state of the environment at time k, A k Represents the agent based on the environment state S k The action performed at time k.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, each step of the distributed energy storage system optimization scheduling method based on LSTM and DQN algorithms as described in any one of claims 1 to 3 is implemented.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, each step of the distributed energy storage system optimization scheduling method based on LSTM and DQN algorithms as claimed in any one of claims 1 to 3 is implemented.

Images