Distributed energy storage scheduling method based on reinforcement learning algorithm

Abstract

The application discloses a distributed energy storage scheduling method based on a reinforcement learning algorithm, relates to the technical field of distributed energy storage, and comprises the following steps: an environment model of a battery swap station is constructed, and the scheduling management of the battery swap station is abstracted as a Markov decision process; wherein the environment model comprehensively considers battery charging and discharging characteristics, battery degradation and user charging and battery swap demands; an improved DDPG algorithm network architecture is introduced to strengthen the optimal scheduling strategy output by the Markov decision process; an experience replay pool is set, DDPG algorithm network parameters are updated, and the final optimal scheduling strategy is optimized and obtained. The application abstracts the scheduling management of the battery swap station as a Markov decision process, introduces the improved DDPG algorithm network architecture, and sets the experience replay pool, so that a small amount of abnormal sample experience is avoided from being ignored, the final optimal scheduling strategy can be suitable for a complex dynamic power grid environment, the uncertainty problem of the distributed energy storage power grid can be solved, and actual scheduling demands can be met.

Description

Technical Field

[0001] This invention relates to the field of distributed energy storage technology, and more specifically, to a distributed energy storage scheduling method based on reinforcement learning algorithms. Background Technology

[0002] Electric vehicles, with their low-carbon, environmentally friendly, flexible, and controllable characteristics, are widely regarded as an important way to improve the competitiveness of the automotive industry, transform into a low-carbon economy, and ensure energy security. However, with the large-scale production and application of electric vehicles, a large number of them are connected to the grid for charging during peak grid load periods, further increasing the peak load on the grid and leading to a series of problems such as insufficient grid capacity and compromised power quality.

[0003] Currently, electric vehicle (EV) charging modes mainly include three types: slow charging, fast charging, and battery swapping. Among these, the separability of the EV and battery in the battery swapping mode allows battery charging to be independent of the EV's downtime, facilitating centralized management of battery charging and discharging and avoiding the adverse effects of large-scale random charging of EVs on grid operation. Treating the battery as a distributed micro-energy storage unit of the grid and orderly regulating its charging and discharging can achieve peak shaving and valley filling, promote the consumption of new energy sources, and regulate voltage and frequency, thereby realizing a two-way friendly interaction between EVs and the grid. Furthermore, compared to battery charging stations, battery swapping stations can provide EV operators with greater economic benefits for the same service capacity.

[0004] However, due to the increasing demand for real-time scheduling of dynamic loads and distributed energy storage systems, accurate modeling has become a key challenge. Traditional modeling methods are ill-equipped to handle the complexity and dynamism of the power grid environment, and cannot simultaneously ensure power supply stability and economic efficiency. Therefore, it is urgent to improve the accuracy of modeling to compensate for the shortcomings of traditional modeling and scheduling methods and better address the uncertainties of distributed energy storage power grids. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a distributed energy storage scheduling method based on reinforcement learning algorithms, which solves the problem that traditional modeling and scheduling methods cannot cope with uncertainties such as rising grid load peaks and insufficient system capacity.

[0006] To achieve the above objectives, embodiments of the present invention provide a distributed energy storage scheduling method based on reinforcement learning algorithms, comprising:

[0007] An environmental model for battery swapping stations is constructed, and the scheduling and management of battery swapping stations is abstracted into a Markov decision process; wherein, the environmental model comprehensively considers battery charging and discharging characteristics, battery degradation, and user charging and swapping needs;

[0008] An improved DDPG algorithm network architecture is introduced to reinforce the optimal scheduling strategy based on Markov decision outputs.

[0009] Set up an experience replay pool, update the DDPG algorithm network parameters, optimize and obtain the final optimized scheduling strategy.

[0010] In a preferred embodiment, the environmental model comprehensively considers battery charge and discharge characteristics, battery degradation, and user charging and swapping needs, specifically:

[0011] The battery charge and discharge characteristics include: the state of charge of the electric vehicle battery, the charge and discharge constraints of the battery, and the charge and discharge power constraints of the battery.

[0012] The battery degradation includes battery capacity degradation; and

[0013] The user's charging and swapping needs include battery inventory.

[0014] In a preferred embodiment, the Markov decision includes: a state space, an action space, state transition probabilities, a reward function, and a discount factor; wherein, the state space includes the real-time electricity price, the number of fully charged batteries, the number of batteries requiring charging, and the charge status of each charging pile battery in the previous time period; the action space includes the charging and discharging power of each charging pile; and the reward function is the negative of the total operating cost of the battery swapping station.

[0015] In a preferred embodiment, the total operating cost of the battery swapping station includes:

[0016] Battery degradation costs; and

[0017] Electricity cost is calculated by comprehensively considering the real-time electricity price of the grid, violations of charge limits and charging and discharging characteristics, and the inability to meet user needs.

[0018] In a preferred embodiment, the improved DDPG algorithm network architecture includes:

[0019] The main network, which has a deep neural network architecture; and

[0020] The target network has the same deep neural network architecture as the main network;

[0021] The main network includes an action network and an evaluation network; the target network includes a target action network and a target evaluation network.

[0022] In a preferred embodiment, setting up the experience replay pool specifically involves:

[0023] Set up a first experience replay pool to store experiences that have been penalized.

[0024] Set up a second experience replay pool to store normal experiences; and

[0025] A comprehensive sampling ratio coefficient is set to balance the sampling ratio in the first experience playback pool and the second experience playback pool.

[0026] The comprehensive sampling ratio is the sampling mixing ratio of the first experience playback pool and the second experience playback pool.

[0027] In a preferred embodiment, both the first experience replay pool and the second experience replay pool are configured with a sub-partition structure based on state clustering and a priority queue for storing sample priorities.

[0028] In a preferred embodiment, updating the DDPG algorithm network parameters and optimizing and obtaining the final optimized scheduling strategy includes:

[0029] Calculate the combined sampling ratio of the first experience replay pool and the second experience replay pool, and calculate the final sampling probability of the sample in combination with the dynamic storage sample priority;

[0030] Based on the final sampling probability, samples are sampled from the first empirical replay pool and the second empirical replay pool, and the importance weight of the samples is calculated.

[0031] Update the action network and the evaluation network, synchronously soft update the target action network and the target evaluation network, and dynamically update the sample priority;

[0032] The optimal scheduling strategy output by the Markov decision is iteratively optimized until the strategy converges, thus obtaining the final optimized scheduling strategy.

[0033] In a preferred embodiment, the iterative optimization process is as follows:

[0034] Adding Gaussian noise to the output of the action network encourages the exploration of the final optimized scheduling strategy;

[0035] Repeat the training and evaluate the optimal scheduling strategy obtained in the current training cycle to determine whether it has converged or reached the preset threshold.

[0036] When the convergence condition or preset threshold is reached, the current agent is output as the final optimized scheduling strategy; if the convergence condition or preset threshold is not reached, new experience is collected and trained on the basis of the original first experience replay pool and second experience replay pool.

[0037] In a preferred embodiment, updating the DDPG algorithm network parameters includes:

[0038] The evaluation network is trained by minimizing the time difference or loss function, and the parameters of the evaluation network are updated based on the learning rate of the evaluation network.

[0039] Calculate the value gradient to update the action network, and update the action network parameters based on the learning rate of the action network;

[0040] Based on the updated evaluation network parameters and the updated action network parameters, the target action network parameters and the target evaluation network parameters are updated using a soft update method.

[0041] The beneficial effects of this invention are:

[0042] (1) By considering the charging and discharging characteristics of the battery in the battery swapping station, battery degradation and user charging and swapping needs, an environmental model of the battery swapping station is constructed so that the environmental model can accurately characterize and describe the dynamic characteristics of the distributed energy storage grid system and reduce the impact of random charging behavior on the grid.

[0043] (2) By abstracting the scheduling and management of the battery swapping station into a Markov decision process, an improved DDPG algorithm network architecture is introduced, and an experience replay pool is set up to avoid ignoring a small number of abnormal sample experiences. This makes the final optimized scheduling strategy applicable to complex and dynamic power grid environments, and can cope with uncertainties such as load peak rise and power shortage in distributed energy storage power grid systems, thus meeting actual scheduling needs. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the distributed energy storage scheduling method based on reinforcement learning algorithm of the present invention.

[0045] Figure 2 This is a schematic diagram of the architecture of the distributed energy storage scheduling method based on reinforcement learning algorithm of the present invention;

[0046] Figure 3 This is a schematic diagram of the framework of the battery swapping station environment model;

[0047] Figure 4 A schematic diagram of the Markov decision-making framework;

[0048] Figure 5 A schematic diagram outlining the total operating cost of a battery swapping station;

[0049] Figure 6 A schematic diagram of the improved DDPG algorithm network architecture. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0051] Please see Figure 1 and Figure 2 A distributed energy storage scheduling method based on reinforcement learning algorithm includes: constructing a battery swapping station environment model and abstracting the battery swapping station scheduling and management into a Markov decision process; wherein, the environment model comprehensively considers battery charging and discharging characteristics, battery degradation and user charging and swapping needs; introducing an improved DDPG algorithm network architecture, and reinforcing learning the optimal scheduling strategy output by the Markov decision; setting up an experience replay pool, updating the DDPG algorithm network parameters, optimizing and obtaining the final optimized scheduling strategy.

[0052] By considering the charging and discharging characteristics of batteries at battery swapping stations, battery degradation, and user charging and swapping needs, an environmental model for battery swapping stations is constructed. This model comprehensively considers the actual discharge situation of batteries under centralized management at swapping stations, reducing the impact of random charging behavior of traditional charging stations on the power grid. Simultaneously, this environmental model accurately depicts the actual operating scenarios of battery swapping stations and precisely describes the dynamic characteristics of distributed energy storage grid systems. Furthermore, the scheduling and management of battery swapping stations is abstracted as a Markov decision process, and an improved DDPG algorithm network architecture is introduced, enabling the distributed energy storage scheduling problem to be solved within a reinforcement learning framework. Moreover, by setting up an experience replay pool and updating the DDPG algorithm network parameters based on the data in the experience replay pool, the effect of ignoring a small number of abnormal sample experiences can be avoided, improving the effectiveness of experience learning and achieving adaptive learning and optimization. This lays the foundation for obtaining intelligent scheduling strategies for distributed energy storage grid systems, making the final optimized scheduling strategy applicable to complex and dynamic grid environments, capable of addressing uncertainties such as load peak increases and power shortages in distributed energy storage grid systems, and meeting actual scheduling needs.

[0053] Please see Figure 3 To accurately depict the actual operation and scheduling management of battery swapping stations, this embodiment further explains the construction of the environmental model. The environmental model comprehensively considers battery charging and discharging characteristics, battery degradation, and user charging and swapping needs. Specifically, the battery charging and discharging characteristics include: the state of charge of the electric vehicle battery, battery charging and discharging charge constraints, and battery charging and discharging power constraints; battery degradation includes battery capacity degradation; and user charging and swapping needs include battery inventory. Based on this, this embodiment provides a preferred method for constructing a battery swapping station environmental model as follows:

[0054] S1, Constructing the battery swapping station environment model:

[0055] S11, During battery charging and discharging, the state of charge (SOC) of the electric vehicle battery is updated. The updated formula is as follows:

[0056]

[0057] In the formula: Indicates discrete operation time; Indicates the index of a charging dock; Indicates the state of charge of the battery; This is the temperature correction factor. Indicates time The internal charge / discharge control is positive when charging and negative when discharging to the grid. Indicates the maximum battery capacity; The decision variable is set to 1 when the battery is swapped, and 0 otherwise. This indicates the state of charge of the new battery received in the exchange. This refers to the battery's charging and discharging efficiency.

[0058] Among them, battery charge and discharge efficiency The calculation formula is as follows:

[0059]

[0060] In the formula: Improve battery charging efficiency; This refers to the battery discharge efficiency.

[0061] Exchange decision variables within one operation time period The calculation formula is as follows:

[0062]

[0063] In the formula: It is the minimum battery swapping threshold; This is the floor symbol.

[0064] S12, To prevent overcharging or over-discharging of the battery, the following charge constraints are established during the charging and discharging process:

[0065]

[0066] In the formula, This indicates the state of charge of the existing battery in the electric vehicle.

[0067] S13, Considering that the battery swapping station meets the user's battery swapping needs, assuming at time t, we have: One battery needs to be swapped. If a battery is fully charged and ready for a battery swap, the battery inventory at the battery swapping station is updated using the following formula:

[0068]

[0069] In the formula: express The number of batteries in the inventory that can be swapped at any given time; express Real-time user demand for battery swapping; express The number of batteries newly replaced and put into storage at all times; The battery swapping loss rate indicates that some batteries become temporarily unusable during the warehouse due to reasons such as battery swapping by the robotic arm or omissions in recording.

[0070] To ensure customer satisfaction and to ensure that the number of swappable batteries does not exceed the maximum inventory capacity, the following constraints must be followed:

[0071]

[0072]

[0073] In the formula: This indicates the maximum inventory capacity.

[0074] S14, To make the charging and discharging behavior more closely resemble the real environment, the following charging and discharging power constraints are established:

[0075]

[0076] In the formula: This indicates the battery's charge and discharge power. Determine if the battery state of charge exceeds the fast charging threshold. If the threshold is exceeded, the value is set to 1; otherwise, it is set to 0. Indicates the maximum allowable charge and discharge power. This indicates the charging / discharging state; it is 1 if charging / discharging is in progress, and 0 otherwise.

[0077] To ensure the safe operation of the distribution network, the following constraints are established:

[0078]

[0079] In the formula: The threshold for the power exchange between the battery swapping station and the power grid.

[0080] S15, Battery Capacity Degradation: Considering that each charge and discharge cycle leads to a decrease in battery health and capacity, a battery degradation model is established, with the following formula:

[0081]

[0082] In the formula: It is the daily degradation rate. This is the total number of daily charge / discharge cycles of the battery. This is the degradation coefficient.

[0083] Please see Figure 4 Considering the dynamic characteristics of distributed energy storage grid systems, in order to obtain a scheduling strategy that minimizes the total operating cost of battery swapping stations, this embodiment abstracts the operation and management of battery swapping stations into a Markov decision process. The Markov decision process includes: state space, action space, state transition probability, reward function, and discount factor. The state space includes the real-time electricity price, the number of fully charged batteries, the number of batteries with charging needs, and the charge status of each charging pile battery in the previous time period. The action space includes the charging and discharging power of each charging pile. The reward function is the negative value of the total operating cost of the battery swapping station.

[0084] The total operating cost of the battery swapping station includes: battery degradation cost; and electricity cost, which is calculated by comprehensively considering the real-time electricity price, violations of charge limits and charge / discharge characteristics, and the inability to meet user demand. Specifically:

[0085] S2 transforms the scheduling and management of battery swapping stations into a Markov decision process:

[0086] Transform the scheduling and management of battery swapping stations into a system with... A Markov decision process with a discrete time step and unknown state transition probabilities, wherein the Markov decision process is called an MDP. The MDP consists of a quintuple. Where S is the state space, A is the action space, P is the dynamic transition probability, and R is the reward function. It is a discount factor.

[0087] S21, State space S:

[0088] The state space S should contain factors that have a significant impact on the scheduling strategy. The state space S includes the real-time electricity price, the number of fully charged batteries, the number of batteries with charging needs, and the charge status of the batteries in each charging station in the previous time period. The state space S is represented as follows:

[0089]

[0090] In the formula: This represents the number of fully charged batteries in inventory. The number of batteries that require charging. For the real-time electricity price of the power grid, This shows the battery charge status in each charging station.

[0091] S22, Action Space A:

[0092] The action space A contains the charging and discharging power of each charging station. Since power is a continuous variable, a continuous action range is used to establish the action space, as expressed by the following formula:

[0093]

[0094]

[0095] In the formula: This indicates the charging and discharging power of the charging pile. This refers to the rated charging power allowed by the charging station.

[0096] S23, State transition probability P:

[0097] Due to the randomness of electric vehicle arrivals, the state transition probability is unknown. Therefore, the SOC changes of charging piles and the number of fully charged batteries in the inventory can be represented by the state of charge update of electric vehicle batteries in S11 and the battery inventory update in S13. Meanwhile, battery swapping demand and real-time electricity prices are obtained through interaction with the environment.

[0098] S24, Reward function R:

[0099] The current state is fed into a Markov decision process, which outputs an action to interact with the environment. The environment provides an immediate reward and transitions to the next state. An immediate reward is generated after the action interacts with the environment to reflect the merit of the action, i.e., the ability to achieve certain goals.

[0100] Please see Figure 5 In the scheduling model of this distributed energy storage scheduling method, the main objective is to minimize the total operating cost of the battery swapping station, which mainly includes electricity costs and battery degradation costs. Therefore, the reward of this scheduling model is directly defined as the negative value of the total cost, and the reward formula is as follows:

[0101]

[0102] In the formula: For the real-time electricity price of the power grid, This represents the sum of the charging and discharging power of all batteries within that time step. For the penalty function, Costs associated with battery degradation.

[0103] S241, the battery degradation cost is:

[0104]

[0105] In the formula: It is the daily battery degradation rate. Indicates the maximum battery capacity. This represents the cost coefficient per unit capacity decay.

[0106] S242, the penalty function formula is:

[0107]

[0108] In the formula: This indicates a violation of SOC restrictions. This indicates that the user's needs cannot be met. This indicates a violation of charge / discharge characteristics. Among them:

[0109] S2421, the following formula violates the SOC restriction:

[0110]

[0111] In the formula: As a SOC violation penalty factor, The rated charging power allowed by the charging station. The rated discharge power allowed by the charging pile This indicates the charging / discharging state; it is 1 if charging / discharging is in progress, and 0 otherwise.

[0112] S2422, the formula that cannot meet the user's requirements is as follows:

[0113]

[0114] In the formula: This is a penalty factor for failing to meet user needs.

[0115] S2423 violates the following charge / discharge characteristic formula:

[0116]

[0117] In the formula: Penalty factor for violating battery charge and discharge characteristics, Determine if the battery state of charge exceeds the fast charging threshold. If the threshold is exceeded, it is set to 1; otherwise, it is set to 0.

[0118] Markov decision processes (MDFs) involve selecting actions under different states. The transition probability of the selected action is uncertain, but the selected action will lead to a new state and generate immediate benefits. During the random state transitions, the ultimate goal is to find an action strategy that maximizes long-term benefits—that is, to find a set of action rules that maximizes total revenue over a period of time. Therefore, Markov decision processes can ensure that the final optimized scheduling strategy minimizes the total operating cost of the battery swapping station, demonstrating good economic efficiency.

[0119] Please see Figure 6 After converting the scheduling and management of battery swapping stations into a Markov decision process, this embodiment introduces an improved DDPG algorithm network architecture to achieve distributed energy storage intelligent scheduling of battery swapping stations. The improved DDPG algorithm network architecture includes: a main network with a deep neural network architecture; and a target network with the same deep neural network architecture as the main network. The main network includes an action network and an evaluation network; the target network includes a target action network and a target evaluation network. Specifically:

[0120] S3 introduces and applies an improved DDPG algorithm network architecture and sets up an experience replay pool. It then uses reinforcement learning to optimize the scheduling strategy, updates the network parameters, and iteratively optimizes to obtain the final optimized scheduling strategy.

[0121] S31 introduces an improved DDPG algorithm network architecture:

[0122] The core network structure of the Deep Deterministic Policy Gradient (DDPG) algorithm is a dual-network design consisting of a "main network + target network" to address the instability problem in reinforcement learning training within a continuous action space.

[0123] Main networks: Action network and evaluation network, both of which have deep neural network architectures. The action network receives input states. Output a continuous action , For action network parameters; evaluate the network input state. and actions Output the value of the action. , To evaluate network parameters.

[0124] Target Network: The target action network and target evaluation network have the same DNN architecture as the main network. The target action network has the same number of DNN layers and neurons as the main action network to ensure that the dimension and scope of the output action are consistent. The target evaluation network has the same DNN structure as the main evaluation network to ensure that the logic of value assessment is consistent.

[0125] The target action network does not update in real time with the main action network, but instead slowly replicates the parameters of the main action network. This provides a stable target action for the target evaluation network, preventing drastic fluctuations in the target value of the evaluation network due to sudden changes in the parameters of the main action network. The target evaluation network also uses soft updates, slowly replicating the parameters of the main evaluation network. The target value is calculated and used as the training label for the main evaluation network. Because the parameters of the target evaluation network update slowly, the target value is more stable, thus effectively suppressing gradient explosion or oscillation during training. Therefore, by introducing an improved DDPG algorithm network architecture for reinforcement learning, the stability of learning can be greatly improved.

[0126] S32, Set up the experience replay pool:

[0127] Please see Figure 1 and Figure 2 To address the issue of sparse rewards, this embodiment also incorporates an experience replay pool within the improved DDPG algorithm network architecture. This experience replay pool is a composite adaptive experience replay pool.

[0128] S321, Experience replay pool initialization:

[0129] The setting of the experience replay pool specifically involves: setting a first experience replay pool. A second experience playback pool is set up to store experiences containing penalty signals. The system includes a storage area for normal signal experience and a comprehensive sampling ratio coefficient to balance the sampling ratios in the first and second experience playback pools. The comprehensive sampling ratio is the mixing ratio of the samples in the first and second experience playback pools.

[0130] Both the first and second experience replay pools are configured with a sub-partition structure based on state clustering and a priority queue for storing sample priorities. Specifically, an online clustering algorithm is used to cluster the states... Perform real-time clustering to divide it into K clusters, denoted as K. The first experience replay pool and the second experience replay pool maintain K sub-partitions accordingly, i.e. , Maintain a priority for each stored experience sample i. .

[0131] By setting up two experience replay pools to store experience, the first pool stores penalized experience, and the second pool stores normal experience. This ensures that a small number of anomalous samples, i.e., penalized experience, are not overwhelmed by the massive amount of normal experience during sampling. By setting a comprehensive sampling ratio coefficient to balance the sampling ratio in the first and second experience replay pools, the trained scheduling model can be guaranteed to handle anomalous samples better.

[0132] S322, Experience Storage and Processing:

[0133] At time t, the agent interacts with the environment to generate experience data. Process and store as follows.

[0134] S3221, based on the reward signal Judgment, if If it contains a penalty item, then... Store in the first experience replay pool; otherwise, store in the second experience replay pool.

[0135] S3222 employs an online clustering algorithm to analyze the state. Perform real-time clustering and assign the data to the nearest cluster. In the middle, and store it in the corresponding subpartition. or .

[0136] S3223, Calculate the initial priority of experience Priority is determined by the time difference error, and its calculation formula is as follows:

[0137]

[0138] In the formula, The initial time difference error can be obtained through the initial network evaluation. It is a very small positive number used to ensure that the priority is not zero.

[0139] S3224, Update the corresponding subpartition and Number of samples .

[0140] S33, update dual network parameters:

[0141] When updating the DDPG algorithm network parameters, the action network and the evaluation network are updated based on the optimal scheduling strategy data stored in the first and second experience replay pools, i.e., based on the samples and their corresponding importance weights. Simultaneously, the target action network and the target evaluation network are softly updated. Specifically, a batch of sample data is sampled from the first and / or second experience replay pools to update the network parameters.

[0142] The process of updating the DDPG algorithm network parameters and optimizing and obtaining the final optimized scheduling strategy includes: calculating the combined sampling ratio of the first experience replay pool and the second experience replay pool, and calculating the final sampling probability of the samples in combination with the dynamic storage sample priority; based on the final sampling probability, sampling from the first experience replay pool and the second experience replay pool and calculating the importance weight of the samples; updating the action network and the evaluation network, synchronously soft updating the target action network and the target evaluation network, and dynamically updating the sample priority; iteratively optimizing the optimal scheduling strategy output by the Markov decision until the strategy converges to obtain the final optimized scheduling strategy.

[0143] S331, Adaptive Hybrid Sampling:

[0144] During the training of the scheduling model, sampling from the first and second experience replay pools is specifically based on the current training cycle. Dynamically calculate the proportion sampled from the first empirical playback pool. This ratio decreases adaptively as training progresses, and the decrease formula is:

[0145]

[0146] in, and These represent the upper and lower limits of the sampling ratio, respectively; k is the attenuation coefficient.

[0147] Next, determine the sampling ratio for each cluster. For the k-th cluster, its sampling ratio within the first empirical replay pool or the second empirical replay pool is... for:

[0148]

[0149] in, To balance hyperparameters, used to control the degree of influence of sample size on sampling ratio.

[0150] Finally, the final sampling probability of sample i is calculated by combining the dynamic priority. The sampling probability of sample i is determined by its empirical replay pool, its cluster, and its own priority, and is calculated as follows:

[0151]

[0152] in, For indicator functions; It is a priority index used to adjust the degree of influence of priorities.

[0153] S332, Importance Sampling and Weight Calculation:

[0154] Based on the final sampling probability of sample i Sample data from a batch of samples in the first and / or second experience replay pools. Calculate the importance sampling weight for each sample i. To correct for biases introduced by priority sampling:

[0155]

[0156] In the formula, This represents the base sampling probability of sample i within its sub-partition, without considering the final mixing ratio. This is the importance sampling weight index, whose value changes during training from its initial value. The value is gradually increased to 1 to offset the bias introduced by priority sampling. In the early stages of training, because the estimation of the target Q-value is inherently inaccurate and the time difference (TD) error based on the priority is unreliable, complete bias correction is unnecessary; exploratory learning relies on priority. As training progresses, the estimation of the target Q-value becomes increasingly accurate, thus requiring complete bias correction. Gradually increasing to 1 strengthens the learning process, making it more stable and allowing it to converge to a better strategy.

[0157] S333, Update evaluation network parameters:

[0158] The method for updating the DDPG algorithm network parameters includes: training the evaluation network by minimizing the time difference or loss function, and updating the evaluation network parameters based on the learning rate of the evaluation network; calculating the value gradient to update the action network, and updating the action network parameters based on the learning rate of the action network; and updating the target action network parameters and the target evaluation network parameters respectively using a soft update method based on the updated evaluation network parameters and the updated action network parameters.

[0159] The training and evaluation network is achieved by minimizing the error of the time difference (TD). When calculating the loss function, the loss of each sample is weighted using importance weights, and then minimized using the loss function. The formula is as follows:

[0160]

[0161] In the formula: It is the amount of experience used in training. For time The target Q value at that location.

[0162] Among them, time The formula for calculating the target Q value at the given location is as follows:

[0163]

[0164] In the formula: For instant rewards, A discount factor to reflect the significance between current rewards and future rewards. For the target action network parameters, To evaluate the network parameters for the target.

[0165] The formula for updating the evaluation network parameters is as follows:

[0166]

[0167] In the formula: To evaluate the learning rate of the network.

[0168] S334, Update ActionNet parameters:

[0169] The gradient of the value function Q is calculated using the backpropagation algorithm. The calculated gradient determines which action changes lead to higher Q values. Specifically, the action network is updated using the gradient of the evaluation network and the policy gradient ascent is used to update the parameters according to the following formula:

[0170]

[0171] In the formula: This reflects which direction changing the action will result in a higher reward value. This demonstrates how adjusting the motion network parameters can produce better motion output.

[0172] The formula for updating the action network parameters is as follows:

[0173]

[0174] In the formula: is the learning rate of the action network.

[0175] S335, Update the target evaluation network parameters and target action network parameters:

[0176] The parameters of the target action network and the target evaluation network are updated using a soft update method, with the following update rules:

[0177]

[0178]

[0179] In the formula: It is a small positive number that is much smaller than 1.

[0180] S336, Experience Priority Update: New Time Difference Error Calculated After Network Update Re-update the priority of the sampled samples. This allows for dynamic adjustment of priorities.

[0181] S34, Explore the final optimized scheduling strategy:

[0182] When optimizing and obtaining the final optimized scheduling strategy, the optimal scheduling strategy output by the Markov decision process is iteratively optimized until the strategy converges, thus obtaining the final optimized scheduling strategy. The iterative optimization process involves: adding Gaussian noise to the output of the action network to encourage exploration of the final optimized scheduling strategy; repeating training and evaluating the optimal scheduling strategy obtained in the current training cycle to determine if it has converged or reached a preset threshold; when the convergence condition or the preset threshold is reached, the current agent is output as the final optimized scheduling strategy; if the convergence condition or the preset threshold is not reached, new experience is collected and training continues based on the existing first and second experience replay pools. Specifically, based on the experience data in the two set experience replay pools, the parameters of the main evaluation network are updated through gradient descent, the parameters of the main action network are updated through gradient ascent, and the parameters of the target action network and the target evaluation network are simultaneously softly updated, iteratively optimizing the optimal scheduling strategy output by the Markov decision process until the strategy converges, thus obtaining the final optimized scheduling strategy.

[0183] To encourage exploration, Gaussian noise is added to the output of the action network, and the exploration rule formula is as follows:

[0184]

[0185] In the formula: This is the noise attenuation rate, initialized to 0 during training, and then gradually increased by a small positive number as training progresses. It is Gaussian noise.

[0186] By adding Gaussian noise to the output of the action network, the scheduling model can be prevented from adhering to a non-optimal scheduling strategy during reinforcement learning and optimization, and the noise gradually decays as training progresses.

[0187] S35 Iterative optimization to obtain the final optimized scheduling strategy

[0188] Repeat steps S32-S34, evaluating the current policy after each training cycle. Determine the performance of the current policy, such as whether the average cumulative reward has converged or reached a preset threshold.

[0189] If the requirements are met, training is terminated, and the current agent is output as the final optimized scheduling strategy.

[0190] If the requirements are not met, new experience will continue to be collected and trained based on the existing first and second experience replay pools, that is, the iteration optimization will continue.

[0191] In summary, by comprehensively considering factors such as battery charging and discharging characteristics, battery degradation, and user demand, the environmental model of the battery swapping station is constructed. This ensures that the model fully accounts for the dynamic load and the real-time scheduling requirements of the distributed energy storage system, improving modeling accuracy and the accurate characterization of the dynamic characteristics of the distributed energy storage grid. This lays a reliable foundation for subsequent scheduling strategy acquisition. By abstracting the scheduling management of the battery swapping station into a Markov decision process, the distributed energy storage scheduling problem can be learned and optimized within a reinforcement learning framework. This enables intelligent scheduling of distributed energy storage at the battery swapping station, balancing electricity costs, equipment lifespan, and user demand in a complex and dynamic grid environment. This represents a breakthrough optimization compared to traditional scheduling methods. By setting up an experience replay pool and introducing an improved DDPG algorithm network architecture, the situation where a small number of abnormal samples are ignored can be effectively avoided. Furthermore, by setting a comprehensive sampling ratio coefficient, the trained scheduling model can be guaranteed to handle abnormal samples better, enabling effective scheduling and optimization even under uncertainty. By configuring a state clustering-based sub-partition structure and a priority queue for storing sample priorities in both the first and second experience replay pools, state clustering partitioning ensures that the agent can learn evenly from all representative system states during the learning process, avoiding overfitting to certain common states and ignoring rare states, thereby improving the generalization ability and robustness of the policy in the face of various complex conditions. Through adaptive hybrid sampling, the training process initially focuses on rapidly learning constraint boundaries and safety criteria from penalized experiences, while gradually increasing the proportion of normal experiences in the later stages to optimize long-term returns. This dynamic adjustment mechanism aligns better with human learning patterns, resulting in a smoother and more stable training process. Furthermore, dynamic priority sampling automatically focuses on samples with poor current learning performance, effectively avoiding resource waste under uniform sampling and accelerating convergence. Therefore, by setting the experience replay pool as a composite adaptive experience replay pool, it is easy to implement and debug within existing deep reinforcement learning frameworks, providing a powerful tool for solving practical industrial control problems with high-dimensional states, complex constraints, and sparse rewards, such as energy storage scheduling. In addition, the distributed energy storage scheduling method based on reinforcement learning algorithms can adaptively learn the optimal scheduling strategy output by a Markov decision process, thus exhibiting strong robustness to uncertainties in distributed energy storage grid systems, and its final optimized scheduling scheme is easy to implement.

[0192] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0193] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0194] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0195] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0196] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0197] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A distributed energy storage scheduling method based on reinforcement learning algorithm, characterized in that, include: An environmental model for battery swapping stations is constructed, and the scheduling and management of battery swapping stations is abstracted into a Markov decision process; wherein, the environmental model comprehensively considers battery charging and discharging characteristics, battery degradation, and user charging and swapping needs; An improved DDPG algorithm network architecture is introduced to reinforce the optimal scheduling strategy based on Markov decision outputs. Set up an experience replay pool, update the DDPG algorithm network parameters, optimize and obtain the final optimized scheduling strategy; specifically, setting up the experience replay pool involves: Set up a first experience replay pool to store experiences that have been penalized. Set up a second experience replay pool to store normal experiences; and A comprehensive sampling ratio coefficient is set to balance the sampling ratio in the first experience replay pool and the second experience replay pool; the comprehensive sampling ratio is the sampling mixing ratio of the first experience replay pool and the second experience replay pool; both the first experience replay pool and the second experience replay pool are configured with a sub-partition structure based on state clustering and a priority queue for storing sample priorities.

2. The distributed energy storage scheduling method based on reinforcement learning algorithm according to claim 1, characterized in that, The environmental model comprehensively considers battery charge and discharge characteristics, battery degradation, and user charging and swapping needs, specifically: The battery charge and discharge characteristics include: the state of charge of the electric vehicle battery, the charge and discharge constraints of the battery, and the charge and discharge power constraints of the battery. The battery degradation includes battery capacity degradation; and The user's charging and swapping needs include battery inventory.

3. The distributed energy storage scheduling method based on reinforcement learning algorithm according to claim 2, characterized in that, The Markov decision includes: a state space, an action space, state transition probabilities, a reward function, and a discount factor; wherein, the state space includes the real-time electricity price, the number of fully charged batteries, the number of batteries requiring charging, and the charge status of each charging pile battery in the previous time period; the action space includes the charging and discharging power of each charging pile; and the reward function is the negative value of the total operating cost of the battery swapping station.

4. The distributed energy storage scheduling method based on reinforcement learning algorithm according to claim 3, characterized in that, The total operating cost of the battery swapping station includes: Battery degradation costs; and Electricity cost is calculated by comprehensively considering the real-time electricity price of the grid, violations of charge limits and charging and discharging characteristics, and the inability to meet user needs.

5. The distributed energy storage scheduling method based on reinforcement learning algorithm according to any one of claims 1-4, characterized in that, The improved DDPG algorithm network architecture includes: The main network, which has a deep neural network architecture; and The target network has the same deep neural network architecture as the main network; The main network includes an action network and an evaluation network; the target network includes a target action network and a target evaluation network.

6. The distributed energy storage scheduling method based on reinforcement learning algorithm according to claim 1, characterized in that, The process of updating the DDPG algorithm network parameters, optimizing and obtaining the final optimized scheduling strategy includes: Calculate the combined sampling ratio of the first experience replay pool and the second experience replay pool, and calculate the final sampling probability of the sample in combination with the dynamic storage sample priority; Based on the final sampling probability, samples are sampled from the first empirical replay pool and the second empirical replay pool, and the importance weight of the samples is calculated. Update the action network and evaluation network, simultaneously perform soft updates to the target action network and target evaluation network, and dynamically update the sample priority; The optimal scheduling strategy output by the Markov decision is iteratively optimized until the strategy converges, thus obtaining the final optimized scheduling strategy.

7. The distributed energy storage scheduling method based on reinforcement learning algorithm according to claim 6, characterized in that, The iterative optimization process is as follows: Adding Gaussian noise to the output of the action network encourages the exploration of the final optimized scheduling strategy; Repeat the training and evaluate the optimal scheduling strategy obtained in the current training cycle to determine whether it has converged or reached the preset threshold. When the convergence condition or preset threshold is reached, the current agent is output as the final optimized scheduling strategy; if the convergence condition or preset threshold is not reached, new experience is collected and trained on the basis of the original first experience replay pool and second experience replay pool.

8. The distributed energy storage scheduling method based on reinforcement learning algorithm according to claim 1, characterized in that, The updated DDPG algorithm network parameters include: The evaluation network is trained by minimizing the time difference or loss function, and the parameters of the evaluation network are updated based on the learning rate of the evaluation network. Calculate the value gradient to update the action network, and update the action network parameters based on the learning rate of the action network; Based on the updated evaluation network parameters and the updated action network parameters, the target action network parameters and the target evaluation network parameters are updated using a soft update method.

Images