Active power distribution network energy storage optimization configuration method based on DeepAR model and reinforcement learning

Abstract

The invention relates to an active power distribution network energy storage optimization configuration method based on a DeepAR model and reinforcement learning. According to the method, historical load and power generation data are collected, future load is predicted by using a DeepAR model after preprocessing, a cost model is established by combining established charging and discharging characteristic models of a lithium battery and a super capacitor and considering energy power purchase cost and energy storage loss, and various constraint conditions are determined. And then an optimal energy storage charging and discharging strategy is determined through iterative training by using a state representation network, an action value network and a target network in the deep Q network, so that energy scheduling is optimized in a long time, and the cost is minimized. According to the method, load prediction, energy storage characteristics, power cost and constraint conditions are fully considered, the global optimization strategy is determined by applying the deep reinforcement learning algorithm, and the economic benefit of the power distribution network energy storage system can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of energy storage of new energy power systems, and specifically relates to an active distribution network energy storage optimization configuration method based on a DeepAR model and reinforcement learning. Background Art

[0002] With the increase in energy consumption and the widespread application of renewable energy, active distribution networks, as an important part of modern power systems, have the characteristics of high flexibility and strong reliability, and are receiving more and more attention. In active distribution networks, energy storage technology, as one of the key means of energy management and balance, is widely used in energy storage and dispatching control. However, the existing active distribution network energy storage optimization configuration technology still has some limitations. The current active distribution network energy storage optimization configuration technology usually adopts rule-based or experience-based strategies. Although it can achieve a certain degree of energy balance and cost optimization, it lacks comprehensiveness and flexibility. These methods often find it difficult to fully consider complex power system changes, such as load fluctuations, uncertainty in renewable energy, etc. In addition, traditional rule-based methods are usually unable to adapt to complex system topologies and power quality requirements, which limits the effective application of energy storage equipment.

[0003] In recent years, with the rapid development of artificial intelligence, big data analysis and optimization algorithms, the optimal configuration of energy storage in active distribution networks is moving towards a smarter and more adaptive direction. More and more studies are combining advanced prediction models, optimization algorithms and energy storage technologies to achieve more accurate and efficient energy scheduling strategies. The application of technologies such as machine learning and reinforcement learning enables the system to learn and optimize from historical data to adapt to different load and energy fluctuations. In addition, considering factors such as multi-energy complementarity, market participation and grid stability, distributed energy management systems and intelligent energy management systems have gradually received attention, bringing more possibilities for optimal configuration of energy storage.

[0004] The method for optimizing the configuration of energy storage in active distribution networks based on DeepAR model and reinforcement learning involved in this patent has great potential. Based on dynamic programming algorithms and advanced prediction models, this method can minimize energy costs over long time scales and take into account the charging and discharging characteristics, efficiency, and various constraints of energy storage equipment. By comprehensively considering factors such as energy purchase costs, performance of energy storage equipment, participation in the electricity market, and grid stability, this method can formulate optimal energy scheduling strategies under different scenarios. Therefore, this patent will inject new impetus into the technological development in the field of optimizing the configuration of energy storage in active distribution networks and provide important support for the sustainability and intelligence of power systems. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and to provide an active distribution network energy storage optimization configuration method based on the DeepAR model and reinforcement learning. By comprehensively considering factors such as energy purchase cost, performance of energy storage equipment, electricity market participation and grid stability, this method can formulate the optimal energy scheduling strategy under different scenarios.

[0006] The technical solution adopted by the present invention is: an active distribution network energy storage optimization configuration method based on DeepAR model and reinforcement learning, comprising the following steps:

[0007] S1: Data collection and preprocessing

[0008] Collect historical power load data, renewable energy generation data, and energy storage system data. Preprocess this data, including data cleaning, denoising, and normalization, to prepare data sets for prediction and optimization.

[0009] S2: Load forecasting model construction

[0010] Use the selected prediction model (such as DeepAR) to build a load prediction model. This model will predict the load situation in the future based on historical load data and provide basic data for subsequent energy optimization.

[0011] S3: Modeling Energy Storage Characteristics

[0012] Energy storage characteristic models of different energy storage devices are established, taking into account their charge and discharge characteristics, efficiency, capacity, and charge and discharge rate limits. These models will be used to determine the charge and discharge behavior of energy storage devices.

[0013] S4: Energy cost model establishment

[0014] An energy cost model is established, which comprehensively considers the electricity purchase cost and the charging and discharging efficiency loss of energy storage equipment. This model will be used as the objective function in the optimization to minimize the energy cost.

[0015] S5: Constraint Definition

[0016] Clearly define various constraints, including energy storage capacity limitations, charging and discharging rate limitations, equipment life limitations, grid stability constraints, etc. These constraints will ensure the feasibility and rationality of the optimization plan.

[0017] S6: Dynamic Programming Energy Optimization

[0018] Based on the reinforcement learning algorithm, the optimal energy dispatch strategy is formulated with load forecast, energy storage characteristic model, energy cost model and constraints as input. The algorithm will optimize over a long time scale to minimize energy cost.

[0019] Specifically, the load forecasting model in S2 is as follows:

[0020] The historical data (historical power load data, renewable energy generation data, energy storage system data) and external feature data (corresponding meteorological data) are appropriately represented and transformed so that they can be input into the deep factor decomposition model (DeepAR). The historical data is represented as a time series:

[0021] X=x 1 , x 2 , …, x T

[0022] where x t Represents the data of the tth time step. The external feature data is represented as a matrix:

[0023] F=[f 1 , f 2 , …, f T ],

[0024] where f t represents the external feature vector at the t-th time step.

[0025] The DeepAR model is a probabilistic generative model based on a recurrent neural network (RNN) architecture. It takes time series data and external feature data as input to generate a predicted distribution for future time steps. The main components of the model are as follows:

[0026] Embedding layer: maps the external feature data F to the embedding space to capture the relationship between features. Each external feature vector f t is mapped to an embedding vector:

[0027] e t =Embedding(f t )

[0028] Recurrent Neural Network: The model uses a recurrent neural network to capture the historical dependencies of time series data. At each time step t, the RNN receives input e t and the historical hidden state h t-1 , and then calculate the hidden state of the current time step:

[0029] h t =RNN(e t ,h t-1 ).

[0030] Distributed parameter network: The model uses the hidden state h t To generate the traffic distribution parameters for future time steps. Specifically, a fully connected layer is used to map the hidden state to the mean μ t and standard deviation σt , which is used to define the Gaussian distribution.

[0031] For each time step t, the actual observed value x of the flow rate t To determine the distribution parameters (μ t and σ t ). The process can be expressed as:

[0032] x t ~N(μ t , σ t )

[0033] During training, the model adjusts its parameters via maximum likelihood estimation so that the generated distribution approximates the distribution of actual observations.

[0034] Specifically, the energy storage characteristic model in S3 includes a simplified lithium-ion battery model and a supercapacitor model. The present invention uses parameters such as battery voltage, charge / discharge efficiency, and battery capacity to perform modeling. Among them, the lithium-ion battery model includes a battery voltage model and a charge / discharge efficiency model:

[0035] Battery voltage model: There is a relationship between the voltage of a lithium-ion battery and its state of charge (SOC), which can generally be approximated using a linear model:

[0036] V battery =V nom -R int I-OCV(SOC)

[0037] Where: V battery is the battery terminal voltage; V nom is the rated voltage of the battery; R int is the internal resistance of the battery; I is the battery current; OCV(SOC) is the open circuit voltage, which is related to the state of charge (SOC).

[0038] Charge / discharge efficiency model: Lithium-ion batteries have a certain amount of energy loss during the charge and discharge process, which can be modeled using efficiency parameters:

[0039] η charge =η discharge =η

[0040] Where η is the charge and discharge efficiency;

[0041] The supercapacitor model includes a capacitor voltage model and a charge / discharge efficiency model. The charge / discharge efficiency model is similar to that of a lithium-ion battery, and the relationship between the voltage of a supercapacitor and its state of charge can be expressed by a linear model:

[0042]

[0043] Among them, V capacitor is the capacitor voltage; V init is the initial voltage; Q is the capacitor charge; C cap is the capacitor capacity; R ESR is the equivalent series resistance; I is the capacitor current.

[0044] Specifically, the energy cost model described in S4 is to consider the energy purchase cost and the energy loss during the charging and discharging process of the energy storage device in the optimal configuration of the energy storage device. The following is a simplified energy cost model proposed by the present invention, which takes into account the energy purchase cost and the charging and discharging efficiency loss of the energy storage device.

[0045] Energy purchase cost model:

[0046] Assuming that the price of electricity on the grid changes over time, P(t) represents the electricity price at time t (in terms of unit energy cost), and E(t) represents the energy demand at time t (in terms of unit energy amount). Then the energy purchase cost at time t can be expressed as:

[0047] C purchase (t) = P(t)·E(t)

[0048] Energy storage equipment charging and discharging efficiency loss model:

[0049] Assuming that the energy storage device will have a certain amount of energy loss during the charging and discharging process, the charging efficiency η can be used charge and discharge efficiency η discharge At time t, the device is switched from E start (t) Unit energy starts charging and then discharges E end (t) unit energy, then the energy loss is:

[0050]

[0051] Considering the energy purchase cost and the efficiency loss of energy storage equipment charging and discharging, the total energy cost at time t can be expressed as:

[0052] C total (t)=P(t)·E(t)+E loss (t)

[0053] This model can be used to optimize the charging and discharging strategy of energy storage equipment to minimize the total energy cost under different electricity prices and load demands. It should be noted that other factors may need to be considered in actual applications, such as electricity market participation, maintenance costs of energy storage equipment, etc. In addition, the model of electricity prices may also be more complex, taking into account market factors, seasonal changes, etc.

[0054] Specifically, in the step S5 constraint definition, various constraints need to be considered to ensure the feasibility, rationality and stability of the configuration solution. The following are some constraints that may need to be considered:

[0055] Energy storage capacity limit: The charge and discharge capacity of the energy storage device cannot exceed its maximum capacity, that is:

[0056] 0≤E(t)≤E max

[0057] Among them, E(t) is the charge and discharge amount at time t, E max It is the maximum capacity of the energy storage device.

[0058] Charge and discharge rate limitation: The charge and discharge rate of energy storage devices are limited to prevent too fast or too slow charge and discharge processes, which may cause battery performance degradation or grid instability. The charge rate and discharge rate can be expressed as R charge (t) and R discharge (t), then the constraint can be expressed as:

[0059] 0≤R charge (t)≤R charge，max

[0060] 0≤R discharge (t)≤R discharge，max

[0061] Among them, R charge，max and R discharge，max are the maximum rates for charging and discharging, respectively.

[0062] Equipment life limit: The charge and discharge cycles of energy storage equipment will affect its life. Some energy storage technologies have a limited life, so the number of charge and discharge cycles needs to be limited to extend the service life of the equipment.

[0063] Grid stability constraints: Ensure that the charging and discharging process of energy storage equipment does not affect the stability of the grid, including restrictions on voltage, frequency, etc.

[0064] Specifically, in step S6, the dynamic programming energy optimization refers to using the deep Q network in reinforcement learning to optimize the configuration of the energy storage device. The following steps are required to build and train the deep Q network:

[0065] S6.1: Establish a state representation network. It is necessary to define the state of the energy storage device, including the current state of the energy storage device, the power system load, the new energy generation data, and the model prediction data. The state representation network is a neural network used to map state information to vector representation. Assume that the state is s t, the state representation network maps it to a feature vector x t :

[0066] x t =SRN(s t )

[0067] The goal of this step is to transform the raw state information into a meaningful, low-dimensional vector representation.

[0068] S6.2: Build an action value network. The action value network is a function used to estimate the value of taking an action in a given state. In this example, the action value is the charge and discharge control command for each energy storage device. This network takes the feature vector of the state as input and outputs the value of each possible action. Assuming that the action space is a discrete N action, the action value network will output a vector Q(x t , a), means in state x t The value of taking action a is estimated as follows:

[0069] Q target (x t+1 , a′; θ - )=max a' Q(x t+1 , a′; θ - )

[0070] S6.3: Establish the target network. In order to enhance the stability of training, the target network is introduced to calculate the target Q value. The calculation of the target Q value will be divided into two parts: the current reward (comprehensive energy purchase cost and energy storage equipment charging and discharging efficiency loss C total (t)) and the future reward (the maximum Q value of the next state predicted by the target network). The target network is a network with the same structure as the action-value network, but its parameters are updated less frequently.

[0071] S6.4: Training the network. During training, the goal of DQN is to minimize the following loss function to gradually optimize the parameters of the action value network:

[0072]

[0073] Use gradient descent to update the parameters of the action-value network to minimize the loss function, where α is the learning rate:

[0074]

[0075] Update the parameters of the target network to maintain its stability:

[0076] θ - ←θ

[0077] Among them, r is the immediate reward, which means that in state xt The reward obtained after taking action a; γ is a discount factor used to balance the importance of current rewards and future rewards. target (x t+1 , a'; θ - ) is the maximum Q value of the next state predicted by the target network; Q(x t , a; θ) is the action value network prediction at state x t The Q value of taking action a; θ is the parameter of the action value network, θ - are the parameters of the target network;

[0078] S6.5: Repeat iteration. Repeat step S6.4 to continuously iterate and update network parameters and optimize configuration strategies. Through these steps, the trained deep Q network can learn the best charging and discharging strategy in continuous interaction, so that energy storage devices can decide when to charge and discharge and how to distribute energy according to the current state and predicted load conditions, so as to achieve efficient management of the power system.

[0079] Beneficial effects of the present invention: The present invention proposes an active distribution network energy storage optimization configuration method based on the DeepAR model and reinforcement learning, aiming to achieve high efficiency and reliability of the power distribution system. By adopting a predictive model and deep reinforcement learning, the present method enables the energy storage device to intelligently decide when to charge and discharge and how to distribute energy based on real-time load conditions and forecast information. Compared with traditional methods, the present invention performs energy scheduling on a long time scale, minimizes electricity costs, and optimizes the utilization efficiency of energy storage equipment to the greatest extent. In addition, the method of the present invention can also fully consider factors such as the charging and discharging characteristics, efficiency, and capacity of the energy storage equipment, and reduce energy losses during charging and discharging to improve the overall efficiency of the power system. By accurately modeling energy storage characteristics, energy costs, and various constraints, the present method can provide comprehensive decision-making support for power system managers, enabling energy storage devices to learn optimal strategies in continuous interaction, further improving the stability and reliability of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 A flowchart of the steps of the active distribution network energy storage optimization configuration method according to the present invention;

[0081] Figure 2 The step S6 of the present invention is a dynamic programming energy optimization flow chart;

[0082] Figure 3 A schematic diagram of a microgrid topology in an embodiment of the present invention;

[0083] Figure 4 This is a curve diagram of the change of loss function value-iteration number in an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention, and are specifically described in combination with the embodiments below.

[0085] like Figure 1 and 2 As shown, the present invention comprises the following steps:

[0086] S1: Data collection and preprocessing: In this embodiment, the collected data includes historical power load data, wind power and photovoltaic power generation data, and energy storage device group data. The collected data is preprocessed, including outlier detection and elimination, smoothing filtering, standardization, etc., to obtain a cleaned data set for modeling and prediction.

[0087] S2: Load forecasting model construction: Use the deep autoregressive model DeepAR to build a load forecasting model. Set the recurrent neural network as the encoder to encode the historical load sequence into a state vector; set the fully connected network as the decoder to output the predicted distribution parameters based on the state vector. Use the maximum likelihood estimation to train the model so that the predicted distribution is close to the actual load distribution.

[0088] Among them, historical data can be expressed as a time series as follows:

[0089] X=x 1 , x 2 , …, x T

[0090] where x t Represents the data of the tth time step. The external feature data is represented as a matrix:

[0091] F=[f 1 , f 2 , …, f T ],

[0092] where f t represents the external feature vector at the t-th time step.

[0093] The main components of constructing a deep factorization model are as follows:

[0094] Embedding layer: maps the external feature data F to the embedding space to capture the relationship between features. Each external feature vector f t is mapped to an embedding vector:

[0095] e t =Embedding(f t )

[0096] Recurrent Neural Network: The model uses a recurrent neural network to capture the historical dependencies of time series data. At each time step t, the RNN receives input e t and the historical hidden state h t-1 , and then calculate the hidden state of the current time step:

[0097] h t =RNN(e t ,h t-1 ).

[0098] Distributed parameter network: The model uses the hidden state h t To generate the traffic distribution parameters for future time steps. Specifically, a fully connected layer is used to map the hidden state to the mean μ t and standard deviation σ t , which is used to define a Gaussian distribution.

[0099] For each time step t, the actual observed value x of the flow rate t To determine the distribution parameter (μ) of the normal distribution N t and σ t ). The process can be expressed as:

[0100] x t ~N(μ t , σ t )

[0101] During training, the model adjusts its parameters via maximum likelihood estimation so that the generated distribution approximates the distribution of actual observations.

[0102] S3: Energy storage characteristics modeling; use voltage model and efficiency model to model lithium battery. The voltage model takes into account the internal resistance voltage drop; the efficiency model takes into account the charging and discharging efficiency. For supercapacitors, establish voltage model and efficiency model. Parameters are determined according to the equipment data sheet.

[0103] Lithium battery voltage model: There is a relationship between the battery voltage and its state of charge SOC, which can generally be approximated using a linear model:

[0104] V battery =V nom -R int I-OCV(SOC)

[0105] Among them, V battery is the battery terminal voltage; V nom is the rated voltage of the battery; R int is the internal resistance of the battery; I is the battery current; OCV(SOC) is the open circuit voltage, which is related to the state of charge (SOC).

[0106] Lithium battery charging / discharging efficiency model: The battery will have a certain amount of energy loss during the charging and discharging process, which can be modeled using efficiency parameters:

[0107] η charge =η discharge =η

[0108] Where η is the charge and discharge efficiency;

[0109] The supercapacitor model includes a capacitor voltage model and a charge / discharge efficiency model, where the charge / discharge efficiency model is the same as that of a lithium-ion battery. The relationship between the voltage of a supercapacitor and its state of charge can be expressed by a linear model:

[0110]

[0111] Among them, V capacitor is the capacitor voltage; V init is the initial voltage; Q is the capacitor charge; C cap is the capacitor capacity; R ESR is the equivalent series resistance; I is the capacitor current.

[0112] S4: Energy cost model is established; energy cost takes into account electricity purchase cost and energy storage loss. The electricity purchase cost is calculated according to the electricity price during the period; the energy storage loss is calculated according to the charging and discharging efficiency. Combining the two items, the total energy cost model is obtained.

[0113] P(t) is the electricity price at time t, and E(t) is the electricity demand at time t. The energy purchase cost at time t is:

[0114] C purchase (t) = P(t)·E(t)

[0115] Energy storage devices will lose energy during charging and discharging, which is expressed as the charging efficiency η. charge and discharge efficiency η discharge Modeling. At time t, the device is start (t) Unit energy starts charging and then discharges E end (t) Unit energy, energy loss is:

[0116]

[0117] Considering the energy purchase cost and the efficiency loss of energy storage equipment charging and discharging, the total energy cost at time t can be expressed as:

[0118] C total (t)=P(t)·E(t)+E loss (t)

[0119] S5: Determine the constraints: determine the energy storage capacity constraints, charge and discharge rate constraints, cycle life constraints, and grid stability constraints. Capacity constraints limit the charge and discharge amount; rate constraints limit the charge and discharge rate; cycle life constraints limit the number of charge and discharge cycles; grid constraints ensure grid stability.

[0120] Energy storage capacity limit: The charge and discharge capacity of the energy storage device cannot exceed its maximum capacity, that is:

[0121] 0≤E(t)≤E max

[0122] Among them, E(t) is the charge and discharge amount at time t, E max It is the maximum capacity of the energy storage device.

[0123] The charging rate and discharging rate are denoted by R charge (t) and R discharge (t), then the constraint can be expressed as:

[0124] 0≤R charge (t)≤R charge，max

[0125] 0≤R discharge (t)≤R discharge，max

[0126] Among them, R charge，max and R discharge，max are the maximum rates for charging and discharging, respectively.

[0127] S6: Deep Q network optimization; establish state representation network, action value network and target network. The network input is the system state and the output is the value of each action. With the goal of minimizing cost, iteratively train the network to obtain the optimal charging and discharging strategy.

[0128] S6.1: Establish a state representation network; define the state of the energy storage device, including the current state of the energy storage device, power system load, new energy generation data, and model prediction data. Assume that the state is s t , the state representation network maps it to a feature vector x t :

[0129] x t =SRN(s t )

[0130] S6.2: Establish an action value network; the action value is the charge and discharge control command of each energy storage device. This network takes the characteristic vector of the state as input and outputs the value of each possible action. Assuming that the action space is a discrete N action, the action value network will output a vector Q(x t , a), means in state x tThe value of taking action a is estimated as follows:

[0131] Q target (x t+1 , a′; θ - )=max a' Q(x t+1 , a′; θ - )

[0132] S6.3: Establish the target network; the calculation of the target Q value will be divided into two parts: the current reward (comprehensive energy purchase cost and energy storage equipment charging and discharging efficiency loss C total (t)) and the future reward (the maximum Q-value of the next state predicted by the target network).

[0133] S6.4: Training the network. During training, the goal of DQN is to minimize the following loss function to gradually optimize the parameters of the action value network:

[0134]

[0135] Use gradient descent to update the parameters of the action-value network to minimize the loss function, where α is the learning rate:

[0136]

[0137] Update the parameters of the target network to maintain its stability:

[0138] θ - ←θ

[0139] Among them, r is the immediate reward, which means that in state x t The reward obtained after taking action a; γ is a discount factor used to balance the importance of current rewards and future rewards. target (x t+1 , a'; θ - ) is the maximum Q value of the next state predicted by the target network; Q(x t , a; θ) is the action value network prediction at state x t The Q value of taking action a; θ is the parameter of the action value network, θ - are the parameters of the target network;

[0140] S6.5: Repeat the iteration. Repeat step S6.4 to continuously iterate and update the network parameters and optimize the configuration strategy.

[0141] The effectiveness of the present invention is verified as follows:

[0142] The topology of a regional microgrid is as follows: Figure 3As shown in the figure, the power load data, wind power generation data and photovoltaic power generation data of a certain area from January to June 2022 are collected. The data time step is 15 minutes, with a total of 35,232 records. The data contains missing values ​​and outliers.

[0143] The data is preprocessed as follows:

[0144] (1) Outlier detection: Use the 3σ criterion to detect outliers and eliminate them.

[0145] (2) Missing value filling: Use linear interpolation to fill missing values.

[0146] (3) Smoothing: Use a three-point moving average to smooth the data.

[0147] (4) Standardization: The load data, wind power data, and photovoltaic data are standardized in turn and mapped to the [0, 1] interval.

[0148] After the above processing, a cleaned data set that can be used for model training is obtained, totaling 35,200 data items.

[0149] Set the recurrent neural network as encoder, the input layer dimension is 50, the recurrent layer is RNN, the number of nodes is 100, and the output encoding state vector dimension = 64. Set the decoder to a fully connected network, the input layer is the encoder output, the number of nodes in the middle layer is 128, the number of nodes in the output layer is 1, and the activation function is ReLu. Model compilation, the optimizer uses Adam, and the loss function is MSE. Divide the training set and the validation set in a ratio of 7:3, and set the number of training iterations to 100. Model training, train the model parameters on the training set. Model verification, verify the model effect on the validation set, MSE = 0.0042. The trained DeepAR prediction model is obtained, which can be used to predict future loads.

[0150] Energy storage characteristics modeling: There are two types of energy storage devices configured for microgrids: lithium batteries and supercapacitors.

[0151] Its parameters are as follows:

[0152] Lithium battery: charging efficiency η charge =0.9; discharge efficiency η discharge =0.95; rated voltage V nom =25V; internal resistance R int =0.05Ω;

[0153] Supercapacitor: charge and discharge efficiency η = 0.98; rated voltage V init =16V; Capacitance C cap =200F; internal resistance R ESR =0.001Ω;

[0154] According to the above parameters, the voltage model and efficiency model of lithium battery, as well as the voltage model and efficiency model of supercapacitor are established. The electricity price is set for 24 periods of one day, with peak and valley prices of 0.5 yuan / kWh and 0.2 yuan / kWh respectively. The electricity consumption of the microgrid is counted in one day, a total of 320 kWh. The lithium battery is charged from 50 kWh to 150 kWh, while the supercapacitor is discharged from 80 kWh to 120 kWh. The electricity purchase cost is: 320*electricity price = 128 yuan. Considering the energy storage efficiency loss, the lithium battery loss is 60 kWh and the supercapacitor loss is 22 kWh. The total energy cost = 128+60+22 = 210 yuan. According to the above parameter values, the energy purchase cost model and energy storage loss model are established, and the total energy cost is calculated.

[0155] Identify the following constraints:

[0156] Lithium battery capacity limit: 0≤charge and discharge capacity≤200Ah; supercapacitor capacity limit: 0≤charge and discharge capacity≤20F; lithium battery charging rate limit: 0≤charge rate≤0.5C; lithium battery discharge rate limit: 0≤discharge rate≤1C; supercapacitor charging and discharging rate limit: 0≤charge and discharge rate≤20A; charge and discharge cycle limit: lithium battery≤30,000 times;

[0157] Deep Q Network Optimization:

[0158] (1) Define the system state, input the state representation network, and obtain the state feature vector dim=20.

[0159] (2) Set the action space to 10 charging and discharging control commands, establish an action value network, and output the action value vector dim=10.

[0160] (3) Establish a target network with the same structure as the action value network.

[0161] (4) Randomly initialize network parameters and set learning rate and discount factor.

[0162] (5) Repeat the training iteratively, evaluate every 100 iterations, and train 5000 times in total.

[0163] (6) The optimal strategy network model is obtained, and the control strategy can be output in real time according to the system status.

[0164] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations within the meaning and scope of the elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An active distribution network energy storage optimization configuration method based on DeepAR model and reinforcement learning, characterized in that: The following steps are involved: S1: Data collection and preprocessing, collecting historical power load data, renewable energy generation data, and energy storage system data, and preprocessing the data; S2: Load forecasting model construction, using DeepAR model to predict future load based on historical load data; S3: Energy storage characteristics modeling, establishing lithium battery model and supercapacitor model, considering parameters such as charge and discharge characteristics and efficiency; S4: Energy cost model establishment, considering electricity purchase cost and energy storage loss to establish a cost model; S5: Constraint definition, defining energy storage capacity constraints, charge and discharge rate constraints, cycle life constraints, etc.; S6: Dynamic programming energy optimization, using a deep Q-network to determine the optimal charging and discharging strategy to minimize energy cost.

2. The method for optimizing energy storage configuration of active distribution network based on DeepAR model and reinforcement learning according to claim 1, characterized in that: The load forecasting model described in step S2 is a deep autoregressive model DeepAR, which is specifically composed of a recurrent neural network and a fully connected network; specifically, the main components of the model are as follows: Embedding layer: maps the external feature data F to the embedding space to capture the relationship between features; each external feature vector f t is mapped to an embedding vector: e t =Embedding(f t ) Recurrent Neural Network: The model uses a recurrent neural network to capture the historical dependencies of time series data; at each time step t, the RNN receives input e t and the historical hidden state h t-1 , and then calculate the hidden state of the current time step: h t =RNN(ie t , h t-1 ); Distributed parameter network: The model uses the hidden state h t To generate the traffic distribution parameters of the future time step; specifically, a fully connected layer maps the hidden state to the mean μ t and standard deviation σ t , used to define the Gaussian distribution; For each time step t, the actual observed value x of the flow rate t To determine the distribution parameters (μ t and σ t ); the process can be expressed as: x t ~N(μ t ,s t )。 3. The method for optimizing energy storage configuration of active distribution network based on DeepAR model and reinforcement learning according to claim 1, characterized in that: The lithium battery model described in step S3 includes a voltage model and an efficiency model; the supercapacitor model includes a capacitor voltage model and an efficiency model; specifically, the lithium-ion battery model includes a battery voltage model and a charge / discharge efficiency model: Battery voltage model: There is a relationship between the voltage of a lithium-ion battery and its state of charge (SOC), which can generally be approximated using a linear model: V battery =V nom -R int ·I-OCV(SOC) Where: V battery is the battery terminal voltage; V nom is the rated voltage of the battery; R int is the internal resistance of the battery; I is the battery current; OCV(SOC) is the open circuit voltage, which is related to the state of charge (SOC); Charge / discharge efficiency model: Lithium-ion batteries have a certain amount of energy loss during the charge and discharge process, which can be modeled using efficiency parameters: or charge =the discharge =the Where η is the charge and discharge efficiency; The supercapacitor model includes a capacitor voltage model and a charge / discharge efficiency model. The charge / discharge efficiency model is similar to that of a lithium-ion battery, and the relationship between the voltage of a supercapacitor and its state of charge can be expressed by a linear model: Among them, V capacitor is the capacitor voltage; V init is the initial voltage; Q is the capacitor charge; C cap is the capacitor capacity; R ESR is the equivalent series resistance; I is the capacitor current.

4. The method for optimizing energy storage configuration of active distribution network based on DeepAR model and reinforcement learning according to claim 1, characterized in that: The energy cost model described in step S4 calculates the total energy cost by taking into account the electricity price during the period and the charge and discharge loss; specifically, a simplified energy cost model is provided, which takes into account the energy purchase cost and the charge and discharge efficiency loss of the energy storage device; Energy purchase cost model: Assuming that the price of electricity on the grid changes over time, let P(t) represent the electricity price at time t (in terms of unit energy cost), and E(t) represent the energy demand at time t (in terms of unit energy amount); then the energy purchase cost at time t can be expressed as: C purchase (t)=P(t)·E(t) Energy storage equipment charging and discharging efficiency loss model: Assuming that the energy storage device will have a certain amount of energy loss during the charging and discharging process, the charging efficiency η can be used charge and discharge efficiency η discharge To model; at time t, the device changes from E start (t) Unit energy starts charging and then discharges E end (t) unit energy, then the energy loss is: Considering the energy purchase cost and the efficiency loss of energy storage equipment charging and discharging, the total energy cost at time t can be expressed as: C total (t)=P(t)·E(t)+E loss (t) This model can be used to optimize the charging and discharging strategies of energy storage devices to minimize the total energy cost under different electricity prices and load demands.

5. The method for optimizing energy storage configuration of active distribution network based on DeepAR model and reinforcement learning according to claim 1, characterized in that: The constraints described in step S5 include energy storage capacity limitation, charge and discharge rate limitation, and cycle life limitation; specifically, energy storage capacity limitation: the charge and discharge amount of the energy storage device cannot exceed its maximum capacity, that is: 0≤E(t)≤E max Among them, E(t) is the charge and discharge amount at time t, E max is the maximum capacity of the energy storage device; Charge and discharge rate limitation: The charge and discharge rate of energy storage devices are limited to prevent too fast or too slow charge and discharge processes, which may cause battery performance degradation or grid instability; the charge rate and discharge rate can be expressed as R and R respectively. charge (t) and R discharge (t), then the constraint can be expressed as: 0≤R charge (t)≤R charge，max 0≤R discharge (t)≤R discharge，max Among them, R charge，max and R discharge，max are the maximum rates of charge and discharge, respectively; Equipment life limit: The charge and discharge cycle of the energy storage device will affect its life; the life of some energy storage technologies is limited, so the number of charge and discharge cycles needs to be limited to extend the service life of the equipment; grid stability constraints: ensure that the charge and discharge process of the energy storage device does not affect the stability of the grid, including restrictions on voltage, frequency, etc.

6. The method for optimizing energy storage configuration of active distribution network based on DeepAR model and reinforcement learning according to claim 1, characterized in that: The deep Q network described in step S6 includes a state representation network, an action value network and a target network, and the optimal charging and discharging strategy is determined by training the network; specifically, in the step S6, the dynamic programming energy optimization refers to using the deep Q network in reinforcement learning to optimize the configuration of the energy storage device, and the following steps are required to build and train the deep Q network: S6.1: Establish a state representation network; the state of the energy storage device needs to be defined, including the current state of the energy storage device, the power system load, the new energy generation data, and the model prediction data; the state representation network is a neural network used to map the state information to a vector representation; assuming the state is s t , the state representation network maps it to a feature vector x t : x t =SRN(s t ) S6.2: Establish an action value network; the action value network is used to estimate the value function of taking a certain action under a given state; the action value is the charge and discharge control command of each energy storage device; this network takes the characteristic vector of the state as input and outputs the value of each possible action; the action space is a discrete N action, and the action value network will output a vector Q(x t , a), means in state x t The value of taking action a is estimated as follows: Q target (x t+1 ,a′;θ - )=max a' Q(x t+1 ,a′;θ - ) S6.3: Establish the target network; In order to enhance the stability of training, the target network is introduced to calculate the target Q value; The calculation of the target Q value will be divided into two parts: the current reward (comprehensive energy purchase cost and energy storage equipment charging and discharging efficiency loss C total (t)) and future reward (the maximum Q-value of the next state predicted by the target network); S6.4: Training the network; During the training process, the goal of DQN is to minimize the following loss function to gradually optimize the parameters of the action value network: Use gradient descent to update the parameters of the action-value network to minimize the loss function, where α is the learning rate: Update the parameters of the target network to maintain its stability: i - ←θ Among them, r is the immediate reward, which means that in state x t The reward obtained after taking action a; γ is a discount factor used to balance the importance of current rewards and future rewards; Q target (x t+1 , a'; θ - ) is the maximum Q value of the next state predicted by the target network; Q(x t , a; θ) is the action value network prediction at state x t The Q value of taking action a; θ is the parameter of the action value network, θ - are the parameters of the target network; S6.5: Repeat the iteration; repeat step S6.4 to continuously iterate and update the network parameters and optimize the configuration strategy.

Images