Energy storage grid-connected scheduling decision-making method and system based on deep reinforcement learning

Abstract

The invention discloses an energy storage grid-connected scheduling decision-making method and system based on deep reinforcement learning, and relates to the technical field of power systems, and the method comprises the steps: constructing a state space in a deep reinforcement learning calculation model, and determining a return function which is finally given to an intelligent agent by an environment; constructing a network of agents in the deep reinforcement learning calculation model; and constructing a deep neural network to provide action constraints for the intelligent agent. According to the method, features can be automatically extracted from original data by using deep reinforcement learning, manual feature design is not needed, a charge and discharge scheduling task can be automatically executed, and human intervention and operation errors are reduced; randomness and uncertainty in the charging and discharging process of the energy storage battery can be processed, dynamic changes of an energy storage system and a power grid environment can be adapted by using a deep reinforcement learning model, and a charging and discharging strategy is adjusted in real time to cope with changes of power requirements; and continuous learning can be carried out in an operation process, and a charging and discharging strategy is continuously improved so as to adapt to new characteristics of a power grid and an energy storage system changing along with time.

Description

Technical Field

[0001] The present invention relates to the technical field of power systems, and in particular to a method and system for energy storage grid-connected scheduling decision-making based on deep reinforcement learning. Background Art

[0002] With the transformation of the global energy structure and the in-depth implementation of the sustainable development strategy, the large-scale development and utilization of new energy has become a hot topic in the energy field [1]. However, new energy sources such as wind energy and solar energy have significant intermittency and instability, which poses new challenges to the stable operation and efficient scheduling of power systems. In order to solve this problem, energy storage technology, especially battery energy storage system (BESS), plays an increasingly important role [2]. Battery energy storage system is regarded as one of the key technologies to improve the grid's ability to accept new energy and optimize the operation of power systems due to its fast response speed, strong regulation ability and flexible layout. In practical applications, battery energy storage system can not only balance the grid load and improve the stability and reliability of grid operation through reasonable charging and discharging scheduling, but also significantly improve the utilization rate of new energy and reduce the phenomenon of wind and solar power abandonment [3]. However, the charging and discharging process of battery energy storage system is complex, involving multiple fields such as electrochemistry, thermodynamics, and materials science. The charging and discharging strategy of the battery directly affects its cycle life, safety and economy [4]. Traditional charging and discharging control methods often ignore the health status and life loss of the battery, resulting in unsatisfactory actual operation efficiency and economic benefits of the energy storage system. Summary of the invention

[0003] In view of the above-mentioned problems, the present invention is proposed.

[0004] Therefore, the technical problem solved by the present invention is: how to accurately reflect the complex underground environment and cable operation status, reduce errors, process large-scale data and make real-time dynamic predictions.

[0005] To solve the above technical problems, the present invention provides the following technical solutions: a method for energy storage grid-connected scheduling decision-making based on deep reinforcement learning, comprising: constructing a state space in a deep reinforcement learning calculation model to determine the reward function that the environment ultimately gives to the intelligent agent; constructing a network of the intelligent agent in the deep reinforcement learning calculation model; and constructing a deep neural network to provide action constraints for the intelligent agent.

[0006] As a preferred solution of the energy storage grid-connected scheduling decision method based on deep reinforcement learning described in the present invention, the state space should include the following variables: real-time electricity price price(t), energy storage battery charging and discharging power P(t), energy storage battery real-time capacity state SOC(t), the state space is represented as a set S of elements, namely:

[0007] S=[price(t),P(t),SOC(t)]

[0008] Among them, price(t) is the electricity price of the energy storage system connected to the grid at time t, P(t) is the charging and discharging power of the energy storage battery at time t, and SOC(t) is the capacity state of the energy storage battery at time t.

[0009] As a preferred solution of the energy storage grid-connected scheduling decision method based on deep reinforcement learning described in the present invention, the reward function includes a reward function and a penalty function, and the reward function is expressed as:

[0010] Cost(t)=price(t)*P(t)*Ts

[0011] Reward(t)=-Cost(t)

[0012] The penalty function is expressed as: punish(t)=-|SOC(t)-0.5|*100

[0013] The report function is obtained by combining the reward function and the penalty function, which is expressed as:

[0014] R(t)=reward(t)+punish(t)

[0015] Among them, Cost(t) is the revenue of the energy storage system at time t, a positive value represents its sale of electricity to the grid, and a negative value represents its purchase of electricity from the grid; price(t) is the electricity price when the energy storage system is connected to the grid at time t, P(t) is the power value of the energy storage battery at time t, a positive value represents discharge, and a negative value represents charging; Ts is the sampling step; Reward(t) is the reward value, SOC(t) is the capacity state of the energy storage battery at time t, and punish(t) is the penalty value.

[0016] As a preferred solution of the energy storage grid-connected scheduling decision method based on deep reinforcement learning described in the present invention, the network in the intelligent agent includes an Actor network and a Critic network. The Actor network approximates the real strategy through a parameterized strategy function π(a|s; θ). The strategy function maps the state s to the probability distribution of the action a, which is expressed as:

[0017] π(a|s;θ)≈π(a|s;W,b)

[0018] Among them, W and b are the weight and bias parameters of the network;

[0019] The policy optimization of the Actor network is achieved by maximizing the expected return J(θ), where the policy gradient Used to update network parameters:

[0020]

[0021] Among them, Q π (s,a) is the expected return of the state-action pair, ρπ is the state distribution under strategy π, and the parameter update of the tor network adopts the gradient descent method. By calculating the gradient of the loss function and updating the weights, the strategy is gradually optimized:

[0022]

[0023] Among them, A t =Q π (s t ,a t )-V π (s t ) is the advantage function, α is the learning rate.

[0024] As a preferred solution of the energy storage grid-connected scheduling decision method based on deep reinforcement learning described in the present invention, wherein: the Critic network approximates the real value function through a parameterized value function Q(s,a;W,b), and the value function maps the state and action to a numerical value, which represents the expected return under the state-action pair, and the state-action value function is expressed as: Q(s,a;W,b)≈Q(s,a);

[0025] At each time step, the Critic network receives the current state and action and generates a corresponding value estimate, which is used to calculate the temporal difference TD error, expressed as:

[0026] δ t =r t +γQ(s t+1 ,a t+1 )-Q(s t ,a t )

[0027] Among them, r t is in state s t The reward obtained after taking the action, γ is the discount factor, Q(s t+1 ,a t+1 ) is the value estimate of the next state, Q(s t ,a t ) is the value estimate of the current state;

[0028] The loss function of the Critic network uses the mean square error loss function to calculate the difference between the predicted value and the target value. The loss function is used for back propagation to update the parameters of the Critic network. The loss function of its state-action value function is:

[0029]

[0030] The parameter update of the Critic network adopts the gradient descent method, which gradually optimizes the value estimation by calculating the gradient of the loss function and updating the weights: Here, α is the learning rate.

[0031] As a preferred solution of the energy storage grid-connected scheduling decision method based on deep reinforcement learning described in the present invention, the construction of the deep neural network includes designing a network architecture, initializing network parameters, selecting an activation function, defining a loss function, forward propagation and back propagation, parameter updating and regularization strategies;

[0032] The network architecture includes determining the number of layers of the network, the number of nodes in each layer, and the activation function;

[0033] The network parameters are initialized using Xavier initialization, where the weights are initialized as: ij ~N(0,σ 2 ), the bias is initialized to: b i ←0

[0034] The activation function is selected by applying the activation function to each neuron in the hidden layer after calculating the weighted sum;

[0035] The loss function is defined as mean square error loss:

[0036]

[0037] The forward propagation is the forward propagation of data in the network, and each layer calculates its output until the final output layer. The output of each layer is: [l] =W [l] a [l-1] +b [l] , where a [l-1] is the output of the previous layer, expressed as: a [l] =f [l] (z [l] ), where f [l] is a nonlinear activation function;

[0038] The back propagation uses the chain rule to calculate the gradient of the loss function with respect to the network parameters, which is used to update the network weights and biases. The gradient calculation formula is:

[0039]

[0040] Among them, σ ′[l] is the derivative of the activation function;

[0041] The parameter update includes updating the DNN network parameters using gradient descent: Where η is the learning rate;

[0042] The regularization strategy involves adding an L2 regularization term to the loss function

[0043]

[0044] where λ is the regularization coefficient, ||·|| F is the Frobenius norm.

[0045] As a preferred solution of the energy storage grid-connected scheduling decision method based on deep reinforcement learning described in the present invention, wherein: after the neural network is constructed, a training process of multiple iterations is performed, each iteration includes forward propagation, loss calculation, back propagation and parameter update, and after each traversal, the model performance is evaluated using a validation set, and hyperparameters are adjusted as needed;

[0046] Verify the generalization ability of the model on an independent test set, deploy the trained and verified model to the constructed overall model, make real-time predictions on the battery capacity, and after the model is deployed, collect feedback data, evaluate the performance of the model in actual applications, and perform iterative optimization as needed.

[0047] In the second aspect, another object of the present invention is to provide a remote data transmission prediction system for cable thermal degradation gas, including: a state space and reward function construction module, an intelligent network construction and training module, and a model deployment and real-time optimization module; the state space and reward function construction module is used to construct and determine the state space of the energy storage system, combined with the reward and penalty functions, and dynamically evaluate the benefits of the system behavior through the revenue and battery power adjustment of the energy storage system; the intelligent network construction and training module is used to construct the Actor-Critic network architecture in the intelligent agent, use a deep neural network training model, and continuously optimize the network parameters through forward propagation, back propagation and gradient descent methods; the model deployment and real-time optimization module is used to deploy the trained intelligent agent model to the actual energy storage system, perform real-time prediction and scheduling decisions on the battery capacity status, collect actual feedback data, evaluate the performance of the model in the application, and perform hyperparameter adjustment and model iterative optimization in combination with real-time data.

[0048] In a third aspect, a computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the energy storage grid-connected scheduling decision method based on deep reinforcement learning as described above are implemented.

[0049] In a fourth aspect, a computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the energy storage grid-connected scheduling decision method based on deep reinforcement learning as described above.

[0050] Beneficial effects of the present invention: 1. The present invention uses deep reinforcement learning to automatically extract features from raw data without manually designing features, and can automatically perform charging and discharging scheduling tasks, reducing human intervention and operational errors. In addition, the model can design strategies to maximize long-term cumulative rewards, which is consistent with the long-term operating benefits of energy storage batteries (such as battery life and economic benefits).

[0051] 2. The present invention is particularly suitable for complex and uncertain environments, and can handle the randomness and uncertainty in the charging and discharging process of energy storage batteries. The use of deep reinforcement learning models can adapt to the dynamic changes of energy storage systems and power grid environments, and adjust charging and discharging strategies in real time to cope with changes in power demand.

[0052] 3. The present invention can continuously learn during operation and continuously improve the charging and discharging strategies to adapt to the new characteristics of the power grid and energy storage system that change over time. It improves learning efficiency by balancing exploration (finding new strategies) and utilization (using known optimal strategies), which helps to quickly find effective charging and discharging strategies for energy storage batteries.

[0053] 4. With the improvement of computing power and optimization of algorithms, the present invention can achieve real-time or near real-time decision-making to meet the needs of rapid response of energy storage systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0055] Figure 1 An overall flow chart of a method for energy storage grid-connected scheduling decision-making based on deep reinforcement learning provided by one embodiment of the present invention;

[0056] Figure 2 A deep reinforcement learning energy storage grid-connected scheduling decision model diagram of an energy storage grid-connected scheduling decision method based on deep reinforcement learning provided by one embodiment of the present invention;

[0057] Figure 3 A deep reinforcement learning framework diagram of a deep reinforcement learning-based energy storage grid-connected scheduling decision method provided by one embodiment of the present invention;

[0058] Figure 4 A critic network topology diagram of a deep reinforcement learning-based energy storage grid-connected scheduling decision method provided by an embodiment of the present invention;

[0059] Figure 5An Actor network topology diagram of a deep reinforcement learning-based energy storage grid-connected scheduling decision method provided by an embodiment of the present invention;

[0060] Figure 6 A 24h real-time electricity price trend chart of a storage grid-connected scheduling decision method based on deep reinforcement learning provided by an embodiment of the present invention;

[0061] Figure 7 A deep reinforcement learning model training process diagram of a deep reinforcement learning-based energy storage grid-connected scheduling decision method provided by one embodiment of the present invention;

[0062] Figure 8 A model decision storage state of charge (SOC) trend diagram of a storage grid-connected scheduling decision method based on deep reinforcement learning provided by an embodiment of the present invention;

[0063] Fig. 9 A model decision energy storage power P trend chart of a deep reinforcement learning-based energy storage grid-connected scheduling decision method provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0065] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0066] Example 1

[0067] Reference Figure 1-Figure 5 , which is an embodiment of the present invention, provides a method for energy storage grid-connected scheduling decision-making based on deep reinforcement learning, including:

[0068] S1: Construct the state space in the deep reinforcement learning computational model and determine the reward function that the environment ultimately gives to the agent;

[0069] Furthermore, in deep reinforcement learning algorithms, the environment is a dynamic system that provides feedback (such as rewards or penalties) based on the agent's behavior and allows the agent to learn optimal strategies through interaction with the environment. The environment not only defines the actions that the agent can take and the states that are observed, but also determines how these actions affect the transformation of environmental states and the distribution of rewards. By continuously exploring and utilizing the environment, the agent is able to gradually optimize its decision-making process to achieve the goal of maximizing the cumulative reward in a given task. In short, the environment is the basis for the agent to learn and adapt during reinforcement learning. The environment construction process of this algorithm is divided into the following steps:

[0070] 101: Determine the state quantity used in the invention and construct the corresponding state space

[0071] During the operation of the system, the variables in the state space correspond one-to-one to the actual state variables of the research system. In order to fully capture the system state, the state space should include the following variables: real-time electricity price price(t), energy storage battery charge and discharge power P(t), and energy storage battery real-time capacity state SOC(t). Therefore, the state space can be represented as a set S of elements, that is,

[0072] S=[price(t),P(t),SOC(t)]

[0073] Where price(t) is the electricity price of the energy storage system connected to the grid at time t, P(t) is the charge and discharge power of the energy storage battery at time t, and SOC(t) is the capacity state of the energy storage battery at time t.

[0074] 102: Set the corresponding reward value for the agent in the environment

[0075] After the agent selects a set of actions, the environment will give corresponding rewards based on the effects of these actions. This is the core mechanism of reinforcement learning, which maximizes long-term returns through continuous trial and error, feedback and evaluation.

[0076] Cost(t)=price(t)*P(t)*Ts

[0077] Reward(t)=-Cost(t)

[0078] Where: Cost(t) is the revenue of the energy storage system at time t, a positive value represents the sale of electricity to the grid, and a negative value represents the purchase of electricity from the grid; price(t) is the electricity price when the energy storage system is connected to the grid at time t, P(t) is the power value of the energy storage battery at time t, a positive value represents discharge, and a negative value represents charging; Ts is the sampling step, which is 1 hour; Reward(t) is the reward value.

[0079] 103: Set the corresponding penalty value for the agent in the environment

[0080] When the agent chooses an inappropriate action, the environment will punish it. This is mainly for the agent to stop midway during the decision-making process, that is,

[0081] punish(t)=-|SOC(t)-0.5|*100

[0082] Where: SOC(t) is the capacity state of the energy storage battery at time t, and punish(t) is the penalty value.

[0083] 104: Determine the reward function that the environment ultimately gives to the agent

[0084] The reward function is the key to evaluating the quality of the agent's actions, and is mainly composed of a reward function and a penalty function. Based on 102 and 103, the reward function R(t) can be determined as:

[0085] R(t)=reward(t)+punish(t).

[0086] S2: Constructing a network of agents in a deep reinforcement learning computational model;

[0087] Furthermore, in the deep reinforcement learning algorithm, the agent is the main body of the learning process. It observes the state, takes actions, and adjusts its own strategy according to the feedback (rewards or penalties) of the environment through interaction with the environment. The goal of the agent is to learn an optimal strategy so that the maximum cumulative reward can be obtained in the long-term interaction with the environment. Through continuous trial and error and learning, the agent can gradually understand how to choose the best action in different states to achieve its goals. In short, the agent plays the role of decision maker and learner in reinforcement learning, and its core task is to optimize its behavior through learning to adapt to the environment and achieve its goals. The present invention selects the DDPG agent, the core of which is the Actor and Critic network, and the specific construction process is divided into the following steps:

[0088] 201: Building the Actor Network in DDPG Agent

[0089] In reinforcement learning, the main role of the Actor network (also called the policy network) is to generate actions. It decides which action to take based on the current state of the environment. The output of the Actor network is usually a probability distribution of actions or directly a continuous action value. The Actor network interacts with the environment, generates actions and receives feedback from the environment, thereby continuously optimizing the strategy to maximize the cumulative reward.

[0090] Actor network usually consists of input layer, multiple hidden layers and output layer. The input layer receives the state information of the environment, and the hidden layer performs feature extraction and nonlinear transformation. The hidden layer contains two fully connected layers and two ReLU activation layers. Each fully connected layer contains 500 neurons. The output layer generates the action space. The output here is the charge and discharge power of the energy storage battery. The network topology is shown in the figure below. Figure 4 As shown.

[0091] Define the policy function:

[0092] The Actor Network approximates the true policy through a parameterized policy function π(a|s;θ) that maps a state s to a probability distribution over actions a.

[0093] π(a|s;θ)≈π(a|s;W,b)

[0094] Among them, W and b are the weight and bias parameters of the network.

[0095] Define the action space of the DDPG agent:

[0096] In this energy storage system, after observing the current state, the agent will select appropriate actions in the action space according to its strategy. These actions are mainly power setting values ​​for the energy storage battery, including the charging power and discharging power of the energy storage battery. Therefore, the action space A can be defined as

[0097] A=[P c (t),P f (t)]

[0098] Where: P c (t) is the charging power of the energy storage system at time t, P f (t) is the discharge power of the energy storage system at time t.

[0099] Actor network strategy optimization and parameter update:

[0100] Based on the previous steps, the Actor network will optimize its own strategy in continuous learning and update internal parameters to output better action values.

[0101] The optimization of the strategy is achieved by maximizing the expected return J(θ). This involves the policy gradient method, where the gradient ▽ θ J(θ) is used to update the network parameters.

[0102] ▽ θ J(θ)=E s～ρπ,a～πθ [▽ θ logπ θ (s,a)Q π (s,a)]

[0103] Among them, Q π (s,a) is the expected return of a state-action pair, and ρπ is the state distribution under policy π.

[0104] Here, the parameters of the Tor network are updated using the gradient descent method. By calculating the gradient of the loss function and updating the weights, the strategy is gradually optimized.

[0105] θ t+1 =θ t +α▽ θ [-logπ(a t ∣s t θ t )A t ]

[0106] Among them, A t =Q π (s t ,a t )-V π (s t ) is the advantage function, α is the learning rate, here it is 0.0001.

[0107] 202: Building the Critic Network in the DDPG Agent

[0108] In reinforcement learning, the Critic network is a neural network used to evaluate the quality of a strategy. Its core function is to estimate the expected return of a given state or state-action pair, thereby providing feedback to the Actor network to help it optimize its strategy. The output of the Critic network is usually a value function, which can be a state-action value function Q(s,a)

[0109] The structure of the Critic network usually includes an input layer, multiple hidden layers, and an output layer. The input layer receives state information, and the hidden layer performs feature extraction and nonlinear transformation. The hidden layer here contains two fully connected layers, two ReLU activation layers, and a tanh activation layer. Each fully connected layer contains 100 neurons. The tanh activation layer is used to limit the output to the range of [-1,1]. The output layer generates a value estimate of the state or state-action pair. The network topology is shown in the figure below. Figure 5 As shown.

[0110] The Critic network approximates the true value function through a parameterized value function Q(s,a;W,b). These functions map states and actions to a numerical value representing the expected reward for that state-action pair.

[0111] State-action value function: Q(s,a;W,b)≈Q(s,a)

[0112] Calculate the time difference error:

[0113] At each time step, the Critic network receives the current state and action and generates a corresponding value estimate. This value estimate is used to calculate the temporal difference (TD) error, which in turn guides the update of the strategy. The TD error is

[0114] δ t =r t +γQ(s t+1 ,a t+1 )-Q(s t ,a t )

[0115] Among them, r t is in state s t The reward obtained after taking the action, γ is the discount factor, Q(s t+1 ,a t+1 ) is the value estimate of the next state, Q(s t ,a t ) is an estimate of the value of the current state.

[0116] Define the loss function:

[0117] The loss function of the Critic network usually uses the mean square error loss function to calculate the difference between the predicted value and the target value. This loss function is used for back propagation to update the parameters of the Critic network. The loss function of its state-action value function is

[0118]

[0119] Update internal parameters:

[0120] The parameter update of the critic network usually adopts the gradient descent method. By calculating the gradient of the loss function and updating the weights, the value estimation is gradually optimized.

[0121]

[0122] Among them, α is the learning rate, which is 0.001 here.

[0123] S3: Build a deep neural network to provide action constraints for the agent.

[0124] Furthermore, in deep reinforcement learning, deep neural networks can be used not only to approximate strategies and value functions, but also to constrain the actions of intelligent agents to ensure that they meet specific requirements or restrictions. In the process of interacting with the environment, deep neural networks can help intelligent agents understand and predict the consequences of violating constraints and adjust strategies accordingly. Deep neural networks are used as prediction models to predict the impact of different actions on the environment, including whether constraints are violated. Through feedback obtained from the environment, the intelligent agent can learn which actions violate constraints and adjust its strategy accordingly. The present invention uses a DNN network to provide action constraints for the intelligent agent, and the specific construction process is divided into the following steps:

[0125] 301: Designing DNN network architecture

[0126] The construction of DNN begins with the design of the network architecture, including determining the number of layers in the network, the number of nodes in each layer, and the activation function. The input layer mainly receives the original state and observation data, where the input is SOCk-1 (the capacity of the battery at the previous moment) and Pk (the charge and discharge power at the current moment); DNN consists of 3 hidden layers, each containing 500 neurons. The output of each layer serves as the input of the next layer. The design of the hidden layer includes selecting the appropriate number and size, and consists of multiple fully connected layers, each of which uses an activation function to introduce nonlinearity. The output layer is the observation data, the corrected agent action of the output layer, and the ReLU function is used to output the action probability distribution. The output here is SOCk (the battery capacity at the current moment).

[0127] 302: Initialize network parameters

[0128] The weights and biases of DNN need to be initialized. Common initialization methods include random initialization, Xavier initialization or He initialization. The present invention adopts Xavier initialization to avoid the problem of gradient disappearance or explosion.

[0129] The weights are initialized as:

[0130] w ij ~N(0,σ 2 )

[0131] The bias is initialized as:

[0132] b i ←0

[0133] 303 Select activation function

[0134] Based on step 301, each neuron in the hidden layer applies an activation function after calculating the weighted sum. Common activation functions include ReLU, tanh or sigmoid functions. Here, the ReLU function is selected to introduce nonlinearity, allowing the network to learn complex patterns.

[0135] 304: Define loss function

[0136] In deep learning, the loss function is a key indicator to measure the difference between the model prediction and the actual results. It quantifies the prediction error of the model and guides the model to adjust the network parameters through the back-propagation algorithm to minimize the prediction error, thereby improving the accuracy and generalization ability of the model. The choice of loss function directly affects the learning objectives and training process of the model, and is a core component for achieving deep learning tasks.

[0137] The present invention adopts mean square error loss

[0138]

[0139] 305: Forward Propagation

[0140] Data propagates forward through the network, with each layer computing its output until the final output layer.

[0141] The output of each layer is

[0142] z [l] =W [l] a [l-1] +b [l]

[0143] where a [l-1] is the output of the previous layer.

[0144] a [l] =f [l] (z [l] )

[0145] where f [l] is a non-linear activation function.

[0146] 306: Back Propagation

[0147] The chain rule is used to calculate the gradient of the loss function with respect to the network parameters, which is used to update the network weights and biases. The gradient calculation formula is:

[0148]

[0149] Among them, σ ′[l] is the derivative of the activation function

[0150] 307: Parameter update

[0151] The present invention uses gradient descent to update DNN network parameters

[0152]

[0153] Where η is the learning rate

[0154] 308: Regularization Strategy

[0155] To prevent overfitting, the present invention adds an L2 regularization term to the loss function

[0156]

[0157] where λ is the regularization coefficient, ||·|| F is the Frobenius norm

[0158] 309: Training model and fine-tuning

[0159] Based on steps 301-308, the DNN network model has been basically constructed. Next, the DNN model needs to be trained and some parameters in the model need to be adjusted according to the training results to enable the model to achieve higher prediction accuracy.

[0160] The training process is repeated for multiple iterations, each of which includes forward propagation, loss calculation, backpropagation, and parameter update. After each epoch, the model performance is evaluated using the validation set and hyperparameters are adjusted as needed. Hyperparameters such as learning rate, batch size, network depth and width, regularization coefficient, etc. need to be adjusted based on the validation set performance.

[0161] Verify the generalization ability of the model on an independent test set to ensure that the model performs well on unseen data, deploy the trained and verified model into the overall model of the present invention, and predict the capacity of the battery in real time. After the model is deployed, collect feedback data, evaluate the performance of the model in actual applications, and perform iterative optimization as needed.

[0162] Example 2

[0163] An embodiment of the present invention provides a deep reinforcement learning-based energy storage grid-connected scheduling decision system, including: a state space and reward function construction module, an agent network construction and training module, and a model deployment and real-time optimization module; the state space and reward function construction module is used to construct and determine the state space of the energy storage system, and dynamically evaluate the benefits of the system behavior through the revenue and battery power adjustment of the energy storage system in combination with the reward and penalty functions; the agent network construction and training module is used to construct the Actor-Critic network architecture in the agent, use the deep neural network training model, and continuously optimize the network parameters through forward propagation, back propagation and gradient descent methods; the model deployment and real-time optimization module is used to deploy the trained agent model to the actual energy storage system, perform real-time prediction and scheduling decisions on the battery capacity status, collect actual feedback data, evaluate the performance of the model in the application, and perform hyperparameter adjustment and model iterative optimization in combination with real-time data.

[0164] Example 3

[0165] An embodiment of the present invention is different from the first two embodiments in that:

[0166] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0167] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0168] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0169] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0170] Example 4

[0171] See also Figure 6-Figure 9 One embodiment of the present invention provides a method for energy storage grid-connected scheduling decision-making based on deep reinforcement learning. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0172] In deep reinforcement learning, the role of model training is to continuously optimize the agent's behavior strategy by simulating interactions with the environment, so that it learns how to choose the best action in a specific state to obtain the maximum cumulative reward. This process involves the agent's understanding and prediction of environmental dynamics, the approximation of strategies and value functions, and balancing risks and rewards by exploring the environment. During the training process, the agent uses the powerful feature extraction capabilities of deep neural networks to process high-dimensional perceptual data, and adjusts network parameters with the help of optimization algorithms such as gradient descent to reduce prediction errors and enhance the accuracy and robustness of its decisions. In addition, training also promotes the generalization ability of the agent in unknown situations, ensuring that the best decision can be made effectively in practical applications. The model training and application of the present invention can be divided into the following steps:

[0173] Model training:

[0174] The present invention uses 24h real-time electricity prices as model training samples, such as Figure 6 As shown in the figure, 1h is the step length. The electricity prices of the power grid at different times of the day are also different. The model is trained with this sample, and the training results are as follows Figure 7 As shown in the figure, the DDPG deep reinforcement learning model was trained 1000 times and stopped training after reaching the maximum number of training times. At the beginning of the training, its reward showed a rapid upward trend, and then fluctuated and rose as a whole. In the final stage, the reward value was basically horizontal, which means that the agent was basically intelligent after these 1000 training times. It can be seen that in this curve, the largest reward values ​​appear in the middle. Among them, when the number of training times reached 954, its reward reached the maximum value of 55.3233. The DDPG agent under this training value is used as an intelligent agent for practical application.

[0175] Model Application:

[0176] Based on step 401, the grid electricity price is also taken as a sample, and the trained deep reinforcement learning model is used to implement the decision on the battery charging and discharging power. The real-time decision results are shown in Table 2, and the battery capacity status corresponding to different times is shown in Table 3. The model decision storage energy charge state SOC trend is as follows Figure 8 As shown, the trend of the model decision energy storage power P is as follows Fig. 9 .

[0177] From the figure, we can see that the model can charge when the electricity price is low, that is, buy electricity from the grid, and discharge when the electricity price is high, that is, sell electricity to the outside, and each step is within the constraint range, and the battery capacity has never exceeded the limit. When targeting the grid electricity price in other regions at different times, we only need to change the sample in the environment.

[0178] The electricity prices at different times are shown in Table 1. The charging and discharging power of the battery is determined by the action of the intelligent agent, and the capacity state of the energy storage battery is determined by its charging and discharging power.

[0179] Table 1 Real-time electricity prices

[0180] time Electricity price (unit: 10,000 yuan / MWh) 0：00-8：00 0.03647 8：00-14：00 0.06805 14：00-17：00 0.10911 17：00-19：00 0.06805 19：00-21：00 0.10911 21：00-24：00 0.06805

[0181] Table 2 Agent action values ​​at different time periods

[0182]

[0183] Table 3 Energy storage battery capacity status at different times

[0184]

[0185]

Claims

1. A method for energy storage grid-connected scheduling decision-making based on deep reinforcement learning, characterized in that: include: Construct the state space in the deep reinforcement learning computational model and determine the reward function that the environment ultimately gives to the agent; Constructing networks of agents in deep reinforcement learning computational models; Construct a deep neural network to provide action constraints for the agent.

2. The energy storage grid-connected scheduling decision method based on deep reinforcement learning according to claim 1, characterized in that: The state space should include the following variables: The real-time electricity price price(t), the energy storage battery charging and discharging power P(t), and the real-time capacity state SOC(t) of the energy storage battery. The state space is represented as a set of elements S, namely: S=[price(t),P(t),SOC(t)] Among them, price(t) is the electricity price of the energy storage system connected to the grid at time t, P(t) is the charging and discharging power of the energy storage battery at time t, and SOC(t) is the capacity state of the energy storage battery at time t.

3. The energy storage grid-connected scheduling decision method based on deep reinforcement learning as claimed in claim 2, characterized in that: The reward function includes a reward function and a penalty function, and the reward function is expressed as: Cost(t)=price(t)*P(t)*Ts Reward(t)=-Cost(t) The penalty function is expressed as: punish(t)=-|SOC(t)-0.5|*100 The report function is obtained by combining the reward function and the penalty function, which is expressed as: R(t)=reward(t)+punish(t) Among them, Cost(t) is the revenue of the energy storage system at time t, a positive value represents its sale of electricity to the grid, and a negative value represents its purchase of electricity from the grid; price(t) is the electricity price when the energy storage system is connected to the grid at time t, P(t) is the power value of the energy storage battery at time t, a positive value represents discharge, and a negative value represents charging; Ts is the sampling step; Reward(t) is the reward value, SOC(t) is the capacity state of the energy storage battery at time t, and punish(t) is the penalty value.

4. The energy storage grid-connected scheduling decision method based on deep reinforcement learning as claimed in claim 3, characterized in that: The network in the agent includes an Actor network and a Critic network. The Actor network is parameterized by a strategy function π(a|s; θ) to approximate the true strategy. The strategy function maps the state s to the probability distribution of the action a, expressed as: π(a|s;θ)≈π(a|s;W,b) Among them, W and b are the weight and bias parameters of the network; The policy optimization of the Actor network is achieved by maximizing the expected return J(θ), where the policy gradient ▽ θ J(θ) is used to update the network parameters: ▽ θ J(θ)=E s～ρπ,a～πθ [▽ θ logπ θ (s,a)Q π (s,a)] Among them, Q π (s,a) is the expected return of the state-action pair, ρπ is the state distribution under strategy π, and the parameter update of the tor network adopts the gradient descent method. By calculating the gradient of the loss function and updating the weights, the strategy is gradually optimized: i t+1 =θ t +a▽ θ [-logπ(a t ∣s t ;θ t )A t ] Among them, A t =Q π (s t ,a t )-V π (s t ) is the advantage function, α is the learning rate.

5. The energy storage grid-connected scheduling decision method based on deep reinforcement learning as claimed in claim 4, characterized in that: The Critic network approximates the true value function through a parameterized value function Q(s,a;W,b). The value function maps the state and action to a numerical value, which represents the expected return under the state-action pair. The state-action value function is expressed as: Q(s,a;W,b)≈Q(s,a); At each time step, the Critic network receives the current state and action and generates a corresponding value estimate, which is used to calculate the temporal difference TD error, expressed as: δ t =r t +γQ(s t+1 ,a t+1 )-Q(s t ,a t ) Among them, r t is in state s t The reward obtained after taking the action, γ is the discount factor, Q(s t+1 ,a t+1 ) is the value estimate of the next state, Q(s t ,a t ) is the value estimate of the current state; The loss function of the Critic network uses the mean square error loss function to calculate the difference between the predicted value and the target value. The loss function is used for back propagation to update the parameters of the Critic network. The loss function of its state-action value function is: The parameter update of the Critic network adopts the gradient descent method, which gradually optimizes the value estimation by calculating the gradient of the loss function and updating the weights: W t+1 =W t -α▽ W L critic , where α is the learning rate.

6. The energy storage grid-connected scheduling decision method based on deep reinforcement learning as claimed in claim 5, characterized in that: The construction of the deep neural network includes designing a network architecture, initializing network parameters, selecting an activation function, defining a loss function, forward propagation and back propagation, parameter updating and regularization strategies; The network architecture includes determining the number of layers of the network, the number of nodes in each layer, and the activation function; The network parameters are initialized using Xavier initialization, where the weights are initialized as: ij ~N(0,σ 2 ), the bias is initialized to: b i ←0 The activation function is selected by applying the activation function to each neuron in the hidden layer after calculating the weighted sum; The loss function is defined as mean square error loss: The forward propagation is the forward propagation of data in the network, and each layer calculates its output until the final output layer. The output of each layer is: [l] =W [l] a [l-1] +b [l] , where a [l-1] is the output of the previous layer, expressed as: a [l] =f [l] (z [l] ), where f [l] is a nonlinear activation function; The back propagation uses the chain rule to calculate the gradient of the loss function with respect to the network parameters, which is used to update the network weights and biases. The gradient calculation formula is: Among them, σ ′[l] is the derivative of the activation function; The parameter update includes updating the DNN network parameters using gradient descent: Where η is the learning rate; The regularization strategy involves adding an L2 regularization term to the loss function where λ is the regularization coefficient, ||·|| F is the Frobenius norm.

7. The energy storage grid-connected scheduling decision method based on deep reinforcement learning according to claim 6, characterized in that: After the neural network is built, it is trained in multiple iterations. Each iteration includes forward propagation, loss calculation, back propagation, and parameter update. After each iteration, the model performance is evaluated using the validation set, and hyperparameters are adjusted as needed. Verify the generalization ability of the model on an independent test set, deploy the trained and verified model to the constructed overall model, make real-time predictions on the battery capacity, and after the model is deployed, collect feedback data, evaluate the performance of the model in actual applications, and perform iterative optimization as needed.

8. A system using the energy storage grid-connected scheduling decision method based on deep reinforcement learning as described in any one of claims 1 to 7, characterized in that: include: State space and reward function construction module, agent network construction and training module, model deployment and real-time optimization module; The state space and reward function building module is used to build and determine the state space of the energy storage system, combine the reward and penalty functions, and dynamically evaluate the benefits of the system behavior through the revenue and battery power adjustment of the energy storage system; The agent network construction and training module is used to construct the Actor-Critic network architecture in the agent, use the deep neural network training model, and continuously optimize the network parameters through forward propagation, back propagation and gradient descent methods; The model deployment and real-time optimization module is used to deploy the trained intelligent agent model to the actual energy storage system, make real-time predictions and scheduling decisions on the battery capacity status, collect actual feedback data, evaluate the performance of the model in the application, and perform hyperparameter adjustment and model iterative optimization based on real-time data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the energy storage grid-connected scheduling decision method based on deep reinforcement learning described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the energy storage grid-connected scheduling decision method based on deep reinforcement learning described in any one of claims 1 to 7 are implemented.

Images