Dynamic charging and discharging strategy coordination optimization method for prolonging life of energy storage system

Abstract

The application relates to the technical field of energy storage systems, in particular to a dynamic charging and discharging strategy collaborative optimization method for prolonging the service life of an energy storage system, which comprises the following steps: inputting historical electricity price time sequence data based on a preset long short-term memory network model to generate an electricity price prediction sequence; constructing a state vector, wherein the state vector comprises the internal state of the energy storage system at the current time t and the electricity price prediction sequence; inputting the state vector into a pre-trained deep reinforcement learning intelligent agent, wherein the deep reinforcement learning intelligent agent outputs a selected specific action from a predefined discrete charging and discharging action set based on the input state vector; converting the output specific action into a charging and discharging power control instruction and delivering the charging and discharging power control instruction to an energy storage converter for execution. Through deep reinforcement learning, the application combines accurate electricity price prediction and dynamic battery health modeling to realize the collaborative optimization of the charging and discharging strategy of the energy storage system in a complex market environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of energy storage systems, in particular to a dynamic charging and discharging strategy collaborative optimization method for prolonging the service life of an energy storage system. BACKGROUND

[0002] As the core asset of an energy storage system, the cycle life and health state of a battery are limited, and frequent or improper charging and discharging will accelerate its capacity attenuation and performance degradation. Therefore, how to develop an intelligent charging and discharging strategy that can maximize economic benefits while taking into account battery health and prolonging its service life has become a core scientific problem and key technical bottleneck that needs to be solved in the current energy storage technology field. Existing charging and discharging strategies for energy storage systems have significant limitations. On the one hand, a large number of rule-based control strategies based on fixed thresholds have rigid logic and cannot adapt to rapidly changing electricity market price signals, often missing arbitrage opportunities or leading to uneconomic charging and discharging behavior, with low profitability and operating efficiency. On the other hand, traditional mathematical programming optimization methods, while able to seek theoretical optimal solutions, not only require high accuracy of the model and complex calculation processes, making it difficult to meet the demand for real-time control, but also often use overly simplified models to quantify the complex and nonlinear electrochemical process of long-term battery health loss, which cannot accurately reflect the true degradation cost of the battery under different states of charge (SOC) and health (SOH), resulting in less-than-ideal actual effects on prolonging battery life. SUMMARY

[0003] The present application relates to the technical field of energy storage systems, in particular to a dynamic charging and discharging strategy collaborative optimization method for prolonging the service life of an energy storage system.

[0004] The present application is implemented by the following technical solutions:

[0005] The dynamic charging and discharging strategy collaborative optimization method for prolonging the service life of an energy storage system includes the following steps:

[0006] Based on a pre-set long short-term memory network model, input historical electricity price time series data to generate an electricity price prediction sequence;

[0007] A state vector is constructed, including the internal state of the energy storage system at the current time t and the electricity price prediction sequence;

[0008] inputting the state vector into a pre-trained deep reinforcement learning agent, the deep reinforcement learning agent outputting a selected specific action from a pre-defined discrete charging and discharging action set based on the input state vector;

[0009] converting the output specific action into a charging and discharging power control instruction and delivering the charging and discharging power control instruction to the energy storage converter for execution.

[0010] Optionally, the pre-set long short-term memory network model is constructed in the following process:

[0011] constructing a neural network structure including an input layer, an LSTM hidden layer and a fully connected output layer;

[0012] inputting N time step historical electricity price time series data into the input layer;

[0013] processing the N historical electricity price data points in sequence through the LSTM hidden layer to update the internal state at each time step and extract feature information encoding the time dependency of the historical electricity price data;

[0014] the fully connected output layer generating an electricity price prediction sequence including M time step electricity price prediction values based on the extracted feature information.

[0015] Optionally, the state vector is constructed in the following process:

[0016] obtaining the current state of charge SOC(t) and the state of health SOH(t) of the energy storage system as internal state data;

[0017] obtaining the electricity price prediction sequence including M electricity price prediction values from the current time t+1 to t+M;

[0018] sequentially concatenating the internal state data and the electricity price prediction sequence in a pre-set order to form the state vector S(t).

[0019] Optionally, the state vector S(t) is calculated in the following formula:

[0020]

[0021] wherein S(t) represents the complete state vector at time t; SOC(t) represents the battery state of charge at time t, which is a scalar between 0 and 1; SOH(t) represents the battery state of health at time t, which is a scalar between 0 and 1; represents the predicted grid electricity price at time t; and M is the total number of time steps predicted by the long short-term memory network model.

[0022] ​Optionally, the deep reinforcement learning agent, whose specific carrier is a deep Q network, has a construction process as follows:

[0023] A multi-layer perceptron network structure is constructed, which includes an input layer, at least two fully connected hidden layers, and an output layer.

[0024] The number of neurons in the input layer is set to match the dimension of the state vector S(t).

[0025] The number of neurons in the output layer is set to be exactly the same as the total number of actions in the predefined discrete charging and discharging action set.

[0026] Optionally, the deep reinforcement learning agent has a specific training process as follows:

[0027] The evaluation network and the target network are initialized, and an experience replay pool is prepared for storing interaction data.

[0028] In the battery simulation environment, each interaction in the battery simulation environment is evaluated using the defined immediate reward function R(t), and a data tuple containing the state, action, reward, and next state is generated.

[0029] The data tuple is stored in an experience replay pool.

[0030] Data is randomly extracted from the experience replay pool for calculating target Q values and current Q values.

[0031] The weights of the deep Q network are iteratively updated based on the loss between the target Q values and the current Q values, until the maximum number of iterations is reached, completing the training of the deep reinforcement learning agent. The action value function Q of the deep reinforcement learning agent can accurately predict the expected value of the long-term cumulative return defined in claim 7 for each candidate action based on the input state.

[0032] Optionally, the defined immediate reward function R(t) has a solving step as follows:

[0033] Based on the state vector S(t), the predicted electricity prices for the next N time steps are extracted.

[0034] Each of the N predicted electricity prices is multiplied by the corresponding charging and discharging power value of the currently selected specific action to obtain N single-step expected revenue values.

[0035] The N single-step expected revenue values are summed to obtain the future N-step expected economic benefit item.

[0036] The current state of charge (SOC)(t) and the state of health (SOH)(t) are taken as inputs, and the state-related health cost coefficient is output through the preset health cost coefficient function k.

[0037] Combine the health cost coefficient with the specific action currently selected. Multiply the corresponding preset action stress factor f, and determine the product result as the state-dependent health cost item;

[0038] Multiply the expected economic benefits of the next N steps by the first preset weight α to obtain the first calculation result;

[0039] Multiply the state-dependent health cost item by the second preset weight β to obtain the second calculation result;

[0040] Subtracting the first calculation result from the second calculation result, the final difference is represented by the instantaneous reward function R(t).

[0041] Optionally, the preset health cost coefficient function k is further limited to having the following output characteristics:

[0042] The region between the first preset SOC threshold and the second preset SOC threshold is defined as the normal SOC working range; when the input state of charge SOC(t) is outside the normal SOC working range, the output value of the health cost coefficient function k is higher than the output value when it is inside the normal SOC working range.

[0043] When the input health status SOH(t) is lower than the preset health level threshold, the output value of the health cost coefficient function k increases non-linearly as SOH(t) decreases.

[0044] Optionally, the specific action selected from the predefined set of discrete charging and discharging actions is as follows:

[0045] The current state vector S(t) is input into the trained deep reinforcement learning agent to perform actions for each pre-defined candidate action. To obtain the predicted value of the corresponding action. ;

[0046] From all the obtained action values, select the candidate action corresponding to the maximum value. And define it as a specific output action. .

[0047] Optionally, the specific action of the output is converted into a charging / discharging power control command, which specifically includes:

[0048] Based on the specific output action determine the corresponding nominal charge-discharge rate;

[0049] obtain a preset nominal energy capacity of the energy storage system;

[0050] multiply the nominal charge-discharge rate and the nominal energy capacity to determine a product as a target charge-discharge power instruction;

[0051] issue the target charge-discharge power instruction to a storage converter of the energy storage system as a power control setting value of the storage converter.

[0052] The technical scheme of the present application has at least the following advantages and beneficial effects:

[0053] On the one hand, the present application uses a long short-term memory network to deeply mine the time sequence dependency in historical electricity price data, realizes accurate long-term prediction of future electricity prices, and provides forward-looking market insights for the decision-making of intelligent agents. On the other hand, the present application creatively constructs a composite immediate-time reward function, which not only takes into account long-term economic benefits, but also introduces a health cost term related to the dynamic state of the battery state of charge and the state of health. It can accurately quantify the marginal impact of each charge-discharge decision on the battery life, and the present application can perform end-to-end learning and decision-making in this complex, multi-dimensional state space through a deep reinforcement learning agent. At each decision-making moment, it can prospectively find an optimal balance point that takes into account the maximization of short-term benefits and the minimization of long-term life costs, thereby significantly improving the overall economic benefits of the energy storage system throughout its life cycle. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A flowchart of the dynamic charge-discharge strategy collaborative optimization method for prolonging the life of the energy storage system provided by the present application.

[0055] Figure 2 A core module principle diagram of the dynamic charge-discharge strategy collaborative optimization method for prolonging the life of the energy storage system provided by the present application. DETAILED DESCRIPTION

[0056] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations.

[0057] Referring to Figure 1 as shown, Figure 1 A flowchart of the dynamic charge-discharge strategy collaborative optimization method for prolonging the life of the energy storage system provided by the present application.

[0058] In an embodiment, a dynamic charging and discharging strategy optimization method for prolonging the service life of an energy storage system includes the following steps:

[0059] Based on a preset long short-term memory network model, historical electricity price time series data are input to generate an electricity price prediction sequence;

[0060] A state vector is constructed, which includes the internal state of the energy storage system at the current time t and the electricity price prediction sequence;

[0061] The state vector is input into a pre-trained deep reinforcement learning agent, which outputs a selected specific action from a predefined set of discrete charging and discharging actions based on the input state vector;

[0062] The output specific action is converted into a charging and discharging power control instruction and is issued to the energy storage converter for execution.

[0063] As shown in Figure 2 , the method of the embodiment is deployed on a computing device, such as an industrial computer, a server, or an embedded controller, which communicates with the battery management system (BMS) and the energy storage converter (PCS) of the energy storage system. The core of the entire method is a deep reinforcement learning agent, which acts as a wise brain that makes the optimal charging and discharging action at each decision-making time t by perceiving the environmental state. Specifically, the embodiment can be divided into the following core modules: an electricity price prediction module, the core of which is a pre-trained long short-term memory network (LSTM) model. It is responsible for receiving historical electricity price data and predicting the electricity price trend in the future based on these data, providing a key basis for the economic decision-making of the agent. A state construction module is responsible for collecting real-time battery internal states (such as SOC and SOH) from the BMS of the energy storage system at each decision-making time t, and combining the future electricity price sequence output by the electricity price prediction module to construct a comprehensive state vector S(t). This vector is the only entrance for the agent to perceive the environment. A decision-making agent module, which is the heart of the embodiment, has a pre-trained deep Q network (DQN) as its carrier. It receives the state vector S(t) generated by the state construction module and outputs an optimal discrete charging and discharging action under the current state. A control execution module is responsible for converting the abstract action output by the agent into specific and executable engineering instructions, i.e., target charging and discharging power instructions, and issuing them to the PCS of the energy storage system, which accurately executes the power instructions.

[0064] Specifically, the preset long short-term memory network model has a construction process as follows:

[0065] a neural network structure including an input layer, an LSTM hidden layer, and a fully connected output layer is constructed;

[0066] N time step historical electricity price time series data are taken as input of the input layer;

[0067] the N historical electricity price data points are sequentially processed by the LSTM hidden layer, and the internal state is updated at each time step to extract feature information encoding time dependence in the historical electricity price data;

[0068] the fully connected output layer generates a time series of electricity price prediction values including M time steps based on the extracted feature information, and the time series is represented as an electricity price prediction sequence.

[0069] In implementation, the first step of the embodiment is to generate an accurate prediction of future electricity prices, which is a prerequisite for achieving proactive economic optimization. In the embodiment, the electricity price prediction module uses a long short-term memory (LSTM) model. The reason for choosing LSTM is that the electricity market price has significant time series characteristics, that is, the current price is highly correlated with the price in the past period. As a special recurrent neural network (RNN), the LSTM has an ingenious gating structure (forget gate, input gate, and output gate) inside, which can effectively learn and remember the long-term dependence in time series data, and thus performs better than traditional statistical models or ordinary feedforward neural networks in prediction tasks. The specific construction process of the LSTM model is as follows: step S201: constructing a network structure. The model mainly consists of three parts: an input layer, one or more LSTM hidden layers, and a fully connected output layer. Step S202: defining input. The input layer receives N time step historical electricity price time series data, represented as . Wherein, is the actual electricity price at the i-th time step before the current time t. The choice of time step length can be determined according to the actual application scenario, for example, if the electricity price data points are 15 minutes, N can be set to 96, representing the input of 24 hours of historical data. Step S203: feature extraction. After data input, it flows through the LSTM hidden layer. In this layer, the LSTM unit processes the input of 96 historical electricity price data points through its internal cell state (CellState) and gating mechanism. At each time step, the LSTM unit updates its internal state, thereby gradually extracting high-level feature information that encodes the time dependence of historical electricity price data. For example, it can learn to recognize the typical double-peak double-valley pattern in a day, or the electricity price difference pattern between weekdays and weekends. Step S204: generate a prediction sequence. The feature information extracted by the LSTM hidden layer is finally passed to the fully connected output layer. The output layer is a standard multilayer perceptron structure, with the number of neurons set to M. It decodes the feature information encoded by the LSTM layer and finally generates a time sequence containing M time steps of electricity price prediction values, represented as an electricity price prediction sequence . Similarly, if the time step is 15 minutes, M can be set to 24, representing the prediction of electricity prices for the next 6 hours. The LSTM model of the present embodiment is trained offline in a supervised learning manner. The training data set can be obtained from a large amount of historical electricity price data obtained by the power grid operator or the power trading center. During training, the historical data is divided into a large number of (input, label) data pairs, where the input is a sequence of N steps, and the label is the real electricity price sequence of the subsequent M steps. Using optimizers such as Adam, the objective is to minimize the mean square error (MSE) between the predicted value and the true value, and the network weights are iteratively updated through the backpropagation algorithm until the model converges.

[0070] In the present embodiment, the state vector is constructed, specifically:

[0071] The current state of charge SOC(t) and the health state SOH(t) of the energy storage system are obtained as internal state data;

[0072] The electricity price prediction sequence including M electricity price prediction values from the current time t+1 to t+M is obtained;

[0073] The internal state data and the electricity price prediction sequence are sequentially spliced in a predetermined order to form the state vector S(t).

[0074] The calculation formula of the state vector S(t) is:

[0075]

[0076] wherein S(t) represents a complete state vector at time t; SOC(t) represents a battery state of charge at time t, which is a scalar between 0 and 1; SOH(t) represents a battery state of health at time t, which is a scalar between 0 and 1; represents a predicted electricity price at time t; and M is a total number of time steps predicted by the long short-term memory network model.

[0077] Specifically, after obtaining the future electricity price prediction, the embodiment constructs a state vector S(t) for the deep reinforcement learning agent, which comprehensively describes the current decision environment of the agent. The state vector is the only source of information for the agent to make decisions, and the design of the state vector directly affects the final control effect. The construction of the state vector S(t) is as follows: step S301: obtaining internal state data. At each decision time t, the system communicates with the battery management system (BMS) of the energy storage system to obtain two key internal state data of the battery in real time: the current state of charge SOC(t), which represents the percentage of the remaining battery capacity. The value is normalized to 0 to 1. SOC is a basic constraint for determining whether the battery can be charged or discharged at present. The current state of health SOH(t) represents the degree of performance retention of the battery compared to the new battery out of the factory, which is usually measured by capacity retention rate. The value is also normalized to 0 to 1. SOH is a key indicator for evaluating the long-term life loss of the battery. Step S302: obtaining the electricity price prediction sequence. Directly call the output of the electricity price prediction module, i.e. a sequence containing M electricity price prediction values from the next time t+1 to t+M . Step S303: concatenating to form a state vector. The internal state data obtained above and the electricity price prediction sequence are sequentially concatenated in a predetermined fixed order to form the final state vector S(t). It can be understood that if M=24, S(t) is a 26-dimensional vector. The first dimension is SOC, the second dimension is SOH, and the third to twenty-sixth dimensions are the predicted electricity prices for the next 6 hours (every 15 minutes). The embodiment applies this design to make the state vector S(t) contain the physical state (SOC, SOH) of the energy storage system itself and the economic environment (future electricity price) of the external market, providing complete information for the agent to make decisions that take both into account.

[0078] In the specific application of the embodiment, the deep reinforcement learning agent, which is specifically a deep Q network, is constructed as follows:

[0079] constructing a multi-layer perceptron network structure including an input layer, at least two fully connected hidden layers, and an output layer;

[0080] ​The number of neurons of the input layer is set to match the dimension of the state vector S(t);

[0081] The number of neurons of the output layer is set to be exactly the same as the total number of actions in the predefined set of discrete charging and discharging actions.

[0082] In this embodiment, the specific carrier of the deep reinforcement learning agent is a deep Q network (DQN). DQN is the product of deep learning combined with Q-Learning, which is particularly suitable for solving decision-making problems with high-dimensional state space and discrete action space, and perfectly fits the application scenario of the present application. The network structure of DQN is a standard multi-layer perceptron (MLP), and its construction process is as follows: step S501: constructing network structure. The network includes an input layer, at least two fully connected hidden layers, and an output layer. Step S502: setting the input layer. The number of neurons of the input layer must match the dimension of the state vector S(t). In our example, the dimension is 2+M, i.e. 26 neurons. Step S503: setting the hidden layer. In order to extract complex nonlinear features in the state vector, at least two fully connected hidden layers are set. It can be understood that the first hidden layer can have 256 neurons, and the second hidden layer can have 128 neurons. Each neuron is followed by a nonlinear activation function, such as a rectified linear unit (ReLU), to enhance the expression ability of the network. Step S504: setting the output layer. The number of neurons of the output layer must be exactly the same as the total number of actions in the predefined set of discrete charging and discharging actions, and the predefined set of discrete charging and discharging actions A should be defined first. In order to make the output of the agent easy to process, this embodiment discretizes the continuous charging and discharging power. As an example, a set of 5 actions is defined: wherein each action corresponds to a specific nominal charging and discharging rate, for example: -1.0C (discharging at 1 times the maximum power); -0.5C (discharging at 0.5 times the rate); 0C (not charging and discharging, idle); +0.5C (charging at 0.5 times the rate); +1.0C (charging at 1 times the maximum power). Therefore, the output layer should have 5 neurons. When a state S(t) is input, the 5 neurons will output a Q value respectively, representing the expected value of the long-term cumulative reward expected to be obtained by executing to the five actions in the current state S(t), i.e. the output is .

[0083] Specifically, the specific training process of the deep reinforcement learning agent is:

[0084] Initialize the evaluation network and the target network, and prepare an experience replay pool for storing interaction data;

[0085] In the battery simulation environment, each interaction in the battery simulation environment is evaluated using the defined instantaneous reward function R(t), and a data tuple containing state, action, reward, and next state is generated.

[0086] The data tuples are stored in an experience replay pool;

[0087] Data is randomly extracted from the experience replay pool to calculate the target Q value and the current Q value;

[0088] Based on the loss between the target Q-value and the current Q-value, the weights of the deep Q-network are iteratively updated until the maximum number of iterations is reached, thus completing the training of the deep reinforcement learning agent. The action value function Q of the deep reinforcement learning agent can accurately predict the expected value of the long-term cumulative reward as defined in claim 7, which can be obtained by performing each candidate action based on any input state.

[0089] In this embodiment, the training process of the deep reinforcement learning agent is an iterative process following the classic DQN algorithm framework, conducted in a battery simulation environment. Step S501: Initialization. Construct two DQN networks with identical structures: evaluate the network. and target network The evaluation network is used to generate current Q-value predictions and perform gradient updates during training, while the target network is used to generate the target Q-value. Its weights are periodically copied from the evaluation network to increase training stability. The experience replay pool D is initialized. This is a first-in, first-out queue used to store data generated by the agent's interaction with the environment, breaking the temporal correlation between data. Step S502: Environment Interaction and Data Acquisition. Training is performed in a loop. At each time step t: the agent observes the current state S(t) of the simulation environment. The agent selects an action using an ε-greedy policy. That is, with a probability of 1-ε, the action that the current evaluation network considers to have the largest Q-value is selected, and with a probability of ε, an action is randomly selected. The simulation environment executes the action. Based on the internal battery and electricity price models, the system calculates the next state S(t+1) after the action and the immediate reward R(t) obtained from this interaction. The data tuples generated from this interaction are then processed. Store the data in the experience replay pool D. Step S503: Network learning and updating. When the amount of data in the experience replay pool reaches a certain scale, network learning begins: a small batch of data tuples is randomly selected from the experience replay pool D. For each data tuple, two calculations are performed: calculate the target Q value: Here, γ is a discount factor (e.g., 0.99), representing the importance placed on future rewards, and the target network is used to predict the maximum future value that the next state can bring. Calculate the current Q-value: Used to evaluate the network performance under S(t). The current estimate of the value. Calculate the loss. Based on the difference between the target Q-value y and the current Q-value, calculate the loss function: The loss function is calculated using the backpropagation algorithm to evaluate the network. The gradients of the weights in each layer are calculated, and an optimizer such as Adam is used to update the gradients along the direction of gradient descent. The weights are assigned. The network is evaluated at regular intervals. All weights are completely copied to the target network. Step S504: Repeat steps S502 and S503 until the preset maximum number of iterations or total training steps are reached. After training, the resulting evaluation network... This refers to a deep reinforcement learning agent that has completed training. Its internal action value function Q is capable of accurately predicting, based on any input state, the expected long-term cumulative reward value determined by a defined immediate reward function, which will bring about the execution of each candidate action.

[0090] The steps to solve the defined instantaneous reward function R(t) are as follows:

[0091] Based on the state vector S(t), extract the predicted electricity price corresponding to each of the next N time steps;

[0092] For each of the N predicted electricity prices, compare it with the currently selected specific action. Multiply the corresponding charging and discharging power values ​​to obtain N single-step expected revenue values;

[0093] The expected returns of N individual steps are summed, and the sum is determined as the expected economic return item for the next N steps.

[0094] The current state of charge (SOC)(t) and the state of health (SOH)(t) are taken as inputs, and the state-related health cost coefficient is output through the preset health cost coefficient function k.

[0095] Combine the health cost coefficient with the specific action currently selected. The corresponding preset action stress factor f is multiplied, and the product result is determined as the state-dependent health cost item.

[0096] Multiply the expected economic benefits of the next N steps by the first preset weight α to obtain the first calculation result;

[0097] The state-dependent health cost term is multiplied by a second preset weight β to obtain a second calculation result;

[0098] The first calculation result is subtracted from the second calculation result, and the final difference is represented as the instant reward value R(t).

[0099] In this embodiment, the instant reward function R(t) is a guide for the agent training, which defines what is good behavior and what is bad behavior. One of the core innovations of this embodiment is to design a composite reward function that can optimize the economic benefits and battery life together. The solving steps are as follows:

[0100] Step S701: Calculate the future N-step expected economic benefit term, which aims to quantify the short-term economic value of the action . First, from the current state vector S(t), extract the predicted electricity price corresponding to each of the future N time steps. Here, N can be equal to M, or less than M, representing the time window of the short-term economic return that the agent is concerned about. Then, convert the current selected specific action to the corresponding charging and discharging power value , where charging is negative and discharging is positive. Multiply each of the N predicted electricity prices with the power value to obtain N single-step expected benefit values. Finally, accumulate and sum the N single-step expected benefit values, and determine the result as the future N-step expected economic benefit term.

[0101] Step S702: Calculate the state-dependent health cost term, which aims to quantify the damage cost of the action to the battery life. It is obtained by multiplying two parts: the first part: the health cost coefficient function k. This is a preset function , its output value is closely related to the current state of the battery, and reflects the degree of vulnerability of the battery under different states. Regarding the characteristics of SOC, two SOC threshold values are set, for example, the first preset SOC threshold is 20%, and the second preset SOC threshold is 90%. The region between the two thresholds (20%-90%) is defined as the normal SOC working interval. When the input SOC(t) is outside this interval, i.e. below 20% or above 90% over-discharge / over-charge region, the output value of function k will be significantly higher than its value within the normal interval. Form a U-shaped curve, means that when the battery power is too low or too high, the health cost of charging and discharging will increase sharply. Regarding the characteristics of SOH, a preset health level threshold is set, for example, 95%. When the input SOH(t) is lower than this threshold, the output value of function k will show a nonlinear growth with the decrease of SOH(t), which means that for an old battery that has already decayed, the health cost of actions that cause equal damage to it will be much higher than a new battery. In the implementation of the present embodiment, function k can be set as a piecewise function, or as a two-dimensional lookup table based on a large amount of experimental data. The second part is the action stress factor f, which is specifically the preset value related to the specific action . It represents the size of the physical stress on the battery caused by different charging and discharging rates. Generally, the larger the charging and discharging rate, the greater the damage to the battery. Therefore, it can be preset: . For example, , , . Multiply the two parts together, the health cost term . This product term can dynamically and accurately quantify the health damage caused by performing a specific action under the current battery state.

[0102] Step S703: Calculate the final immediate reward value R(t), the present embodiment introduces two preset weight coefficients, the first preset weight a (economic benefit weight) and the second preset weight β (health cost weight). These two weights are hyperparameters, allowing the operator of the energy storage system to adjust according to its own operation strategy (aggressive pursuit of benefits, or conservative pursuit of longevity). Multiply the economic benefit term of step S701 by a to get the first calculation result. Multiply the health cost term of step S702 by β to get the second calculation result. The final immediate reward value R(t) is obtained by subtracting the second calculation result from the first calculation result. By taking the health cost as a negative reward term, the agent must learn to actively avoid actions that will cause high health costs and damage the battery life while pursuing economic benefits in order to maximize the long-term cumulative reward during the training process.

[0103] ​More specifically, the specific action selected from the predefined set of discrete charging and discharging actions is as follows:

[0104] The current state vector S(t) is input into the trained deep reinforcement learning agent to perform actions for each pre-defined candidate action. To obtain the predicted value of the corresponding action. ;

[0105] From all the obtained action values, select the candidate action corresponding to the maximum value. And determine it as the specific output action. .

[0106] The specific steps of converting the output action into a charging / discharging power control command are as follows:

[0107] Based on the specific output action Determine the corresponding nominal charge / discharge rate;

[0108] Obtain the preset nominal energy capacity of the energy storage system;

[0109] The nominal charge / discharge rate is multiplied by the nominal energy capacity, and the result of the product is determined as the target charge / discharge power command.

[0110] The target charging and discharging power command is sent to the energy storage converter of the energy storage system as the power control setting value of the energy storage converter.

[0111] In this embodiment, after the agent is trained, it is deployed to an actual energy storage system for online real-time reasoning and control. The workflow is as follows: Step S601: Obtain and construct state S(t). The system periodically executes a decision. At decision time t, all functions of the aforementioned state construction module are first executed to obtain real-time SOC(t) and SOH(t), and the LSTM model is called to predict future electricity prices, ultimately concatenating the current state vector S(t). Step S602: Agent decision-making. The current state vector S(t) is used as input and fed into the trained deep Q-network (i.e., the decision agent module). The network performs a forward propagation calculation, and the output layer will perform calculations for each preset candidate action. (Right now arrive ( ), giving a predicted action value The system selects the candidate action with the highest Q-value from all the obtained action values. And determine it as the current optimal specific output action. This process is called a greedy policy. Step S603: Action conversion and instruction issuing, the action output by the intelligent agent is an abstract symbol and needs to be converted into a physical instruction that can be understood by the PCS. This process is completed by the control execution module. According to , a predefined action-multiple mapping table is consulted to determine a corresponding nominal charge-discharge multiple. For example, if is , the table lookup gives a nominal charge-discharge multiple of +0.5C. The nominal energy capacity preset for the energy storage system is obtained from the system configuration. For example, a 1MWh energy storage system has a nominal energy capacity of 1000kWh. The nominal charge-discharge multiple is multiplied by the nominal energy capacity to obtain the target charge-discharge power instruction. In the above example, the target charge-discharge power instruction = 0.5*1000kWh = 500kW (charging). The calculated target charge-discharge power instruction (500kW) is issued to the energy storage converter PCS of the energy storage system through the communication interface. After receiving the instruction, the PCS will adjust the switching state of its internal power electronic devices such as IGBT to accurately control the battery on the DC side to charge at a power of 500kW. By cyclically executing the steps of S601 to S603, this embodiment realizes full-automatic, intelligent closed-loop control of the energy storage system with the dual goals of prolonging the service life and improving economic efficiency.

[0112] The above is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for dynamic charging and discharging strategy collaborative optimization oriented to prolonging the service life of an energy storage system, characterized in that, The method comprises the following steps: Based on the preset long short-term memory network model, input the historical electricity price time series data to generate an electricity price prediction sequence; Construct a state vector, which includes the internal state of the energy storage system at the current time t and the electricity price prediction sequence; Input the state vector into a pre-trained deep reinforcement learning agent, which outputs a selected specific action based on a predefined set of discrete charging and discharging actions according to the input state vector; Convert the output specific action into a charging and discharging power control instruction and issue it to the energy storage converter for execution; The specific training process of the deep reinforcement learning agent is as follows: Initialize the evaluation network and the target network, and prepare an experience replay pool for storing interaction data; In the battery simulation environment, evaluate each interaction in the battery simulation environment using the defined immediate reward function R(t), and generate a data tuple containing state, action, reward, and next state; Store the data tuple in an experience replay pool; Randomly extract data from the experience replay pool for calculating target Q values and current Q values; Iteratively update the weights of the deep reinforcement learning agent based on the loss between the target Q values and the current Q values until the maximum number of iterations is reached, and the training of the deep reinforcement learning agent is completed; The solving steps of the defined immediate reward function R(t) are as follows: Based on the state vector S(t), extract the predicted electricity price corresponding to each of the next N time steps; For each of the N predicted electricity prices, compare it with the currently selected specific action. Multiply the corresponding charging and discharging power values ​​to obtain N single-step expected revenue values; Sum the N single-step expected revenue values to obtain the future N-step expected economic benefit item; Input the state of charge SOC(t) and the state of health SOH(t) at the current time as inputs, and output the state-dependent health cost coefficient through the preset health cost coefficient function k; multiplying the health cost coefficient by a current selected specific action corresponding to a preset action stress factor f, to obtain a first product result, and determining the first product result as the state-dependent health cost term; Multiply the future N-step expected economic benefit item by a first preset weight alpha to obtain a first calculation result; Multiply the state-dependent health cost item by a second preset weight beta to obtain a second calculation result; Subtract the first calculation result from the second calculation result, and the final difference represents the immediate reward function R(t).

2. The method of claim 1, wherein, The construction process of the preset long short-term memory network model is as follows: Construct a neural network structure including an input layer, an LSTM hidden layer, and a fully connected output layer; Input the historical electricity price time series data of N time steps as the input of the input layer; Through the LSTM hidden layer, sequence the N historical electricity price data points, update their internal states at each time step, and extract the feature information that encodes the time dependence of the historical electricity price data; The fully connected output layer generates an electricity price prediction sequence including a time series of M electricity price prediction values based on the extracted feature information.

3. The method of claim 2, wherein, The construction of the state vector is as follows: Obtain the current state of charge SOC(t) and the state of health SOH(t) of the energy storage system as internal state data; Obtain the electricity price prediction sequence including M electricity price prediction values from the current time t+1 to t+M; The internal state data is sequentially spliced with the electricity price prediction sequence in a preset order to form the state vector S(t).

4. The method of claim 3, wherein, The state vector S(t) has a calculation formula as follows: wherein S(t) represents a complete state vector at time t; SOC(t) represents a battery state of charge at time t, which is a scalar between 0 and 1; SOH(t) represents a battery state of health at time t, which is a scalar between 0 and 1; represents a predicted grid electricity price at time t; and M is a total number of time steps predicted by the long short-term memory network model. represents a predicted grid electricity price at time t; and M is a total number of time steps predicted by the long short-term memory network model.

5. The method of claim 4, wherein, The deep reinforcement learning agent has a specific carrier of a deep Q network, and a construction process of the deep reinforcement learning agent is as follows: A multi-layer perceptron network structure is constructed, which includes an input layer, at least two fully connected hidden layers, and an output layer. The number of neurons of the input layer is set to match the dimension of the state vector S(t). The number of neurons of the output layer is set to be completely identical to the total number of actions in the predefined discrete charging and discharging action set.

6. The method of claim 5, wherein, The preset health cost coefficient function k is further limited to have the following output characteristics: An area between the first preset SOC threshold and the second preset SOC threshold is defined as a regular SOC working interval; when the input state of charge SOC(t) is located outside the regular SOC working interval, the output value of the health cost coefficient function k is higher than that when the input state of charge SOC(t) is located inside the regular SOC working interval; When the input state of health SOH(t) is lower than a preset health level threshold, the output value of the health cost coefficient function k non-linearly increases with the decrease of SOH(t).

7. The method of claim 6, wherein, The specific action selected from the predefined discrete charging and discharging action set is specifically as follows: inputting the current state vector S(t) into the deep reinforcement learning agent that has completed training to obtain a predicted corresponding action value for each preset candidate action ;​ Among all the action values obtained, the candidate action corresponding to the maximum value is selected and determined as the specific output action .

8. The method of claim 7, wherein, The specific action is converted into a charging and discharging power control instruction, which is specifically as follows: in accordance with the specific output action , determine a corresponding nominal charge-discharge rate; A preset nominal energy capacity of the energy storage system is acquired; A nominal charging and discharging rate is multiplied by the nominal energy capacity to obtain a second product result, and the second product result is determined as a target charging and discharging power instruction; The target charging and discharging power instruction is sent to a storage converter of the energy storage system as a power control setting value of the storage converter.

Images