Optimal scheduling method for DDPG energy system based on improved priority experience playback

Abstract

The invention relates to a DDPG energy system optimal scheduling method based on improved priority experience playback, and belongs to the technical field of energy system optimal scheduling. The method comprises the following steps: designing energy exclusive constraints such as power balance, a generator ramp rate, an energy storage system charge state boundary constraint and the like to construct a system model, and defining a reinforcement learning environment at the same time; a multi-dimensional priority evaluation mechanism is constructed through a time sequence difference error, an award and a novelty-priority distribution rule index, and empirical sample utilization is optimized; an improved depth deterministic policy gradient network of the target dual-network architecture is constructed, an actor network outputs continuous actions conforming to constraints, and a commentator network outputs a dual-action value by adopting a state-action separation structure; and circularly training the network to generate an optimal scheduling strategy. The method improves the convergence speed of network training, guarantees that the scheduling strategy gives consideration to economy and safety, and is suitable for real-time scheduling of a complex energy system with distributed energy and energy storage collaboration.

Description

Technical Field

[0001] This invention belongs to the field of energy system optimization and scheduling technology, specifically relating to an optimal scheduling method for DDPG energy systems based on improved priority experience replay. Background Technology

[0002] With the large-scale application of distributed energy and energy storage systems, modern energy systems are gradually transforming from traditional centralized architectures to distributed architectures that coordinate "source-grid-load-storage". While this transformation improves energy utilization efficiency and renewable energy absorption rates, it also significantly increases the complexity of system operation: on the one hand, the output of distributed energy is highly uncertain due to natural conditions, and load demand exhibits significant peak-valley differences and strong randomness; on the other hand, equipment within the system (such as distributed generators and energy storage systems) faces strict physical operating constraints that must be strictly followed during scheduling to avoid equipment damage or system instability. Therefore, achieving real-time optimal scheduling of energy systems under complex constraints has become a core requirement for ensuring the system's economy, security, and stability.

[0003] Traditional energy system dispatching methods primarily rely on optimization techniques such as model predictive control, dynamic programming, and mixed-integer programming. These methods require precise mathematical models of system equipment and accurate forecasts of load and renewable energy output to construct the optimization problem, achieving good results when model accuracy is high and prediction errors are small. However, significant limitations exist in practical applications: the uncertainty of distributed energy resources and loads makes prediction errors unavoidable, easily causing dispatching schemes to deviate from the optimal; aging system equipment and environmental interference can lead to discrepancies between the mathematical model and the actual system, reducing the robustness of the dispatching strategy; and as the system scales up, the computational complexity of the optimization problem increases exponentially, making it difficult to meet the time requirements of real-time dispatching.

[0004] To address these challenges, reinforcement learning techniques, due to their "trial and error learning" characteristics and the advantage of not requiring precise models, have been increasingly applied in the field of energy system dispatching. Among them, the Deep Deterministic Policy Gradient (DDPG) algorithm, with its ability to handle continuous action spaces (such as continuous adjustment of energy storage charging and discharging power and distributed generator output), is highly compatible with the continuous control requirements of multiple devices in energy dispatching, making it a research hotspot in this field. However, traditional DDPG algorithms still face key technical bottlenecks in practical energy dispatching applications: Exploring and utilizing imbalances: Traditional DDPG randomly replays samples, and PER-DDPG relies solely on TD error to assess sample value. Neither can accurately capture the core characteristics of "high-value samples" in the energy system (such as scheduling experience when supply and demand are severely imbalanced), which leads to agents easily overexploring low-value areas or prematurely converging to local optima. Slow convergence speed and poor stability: Although PER-DDPG improves sample utilization, it does not optimize the network structure. The Critic network is not good at fitting the complex relationship between "state and action", resulting in a long training convergence cycle. The scheduling strategy during the testing phase is easily affected by noise, and the operating cost fluctuates greatly. The lack of quantitative supply and demand balance: Existing methods do not define quantifiable "generation-load" balance indicators, and cannot incorporate system imbalance risks (such as overload and undergeneration) into sample priority evaluation and reward functions, making it difficult for dispatch strategies to balance economy and stability. Insufficient adaptability: The lack of customized constraint mechanisms for the physical constraints of the energy system (such as DG ramp rate and energy storage SOC boundary) makes it easy for actions to exceed the operating limits of the equipment, resulting in great difficulty in actual deployment.

[0005] Existing energy system scheduling technologies struggle to simultaneously meet the comprehensive requirements of "uncertainty adaptation, real-time assurance, strict constraint adherence, and multi-objective optimization." A reinforcement learning scheduling algorithm is needed that can combine energy system-specific characteristics, optimize the utilization efficiency of empirical samples, and deeply integrate equipment constraints to improve the optimality, robustness, and security of distributed energy and energy storage collaborative scheduling. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides an optimal scheduling method for DDPG energy systems based on improved priority experience replay. The objective of this invention can be achieved through the following technical solutions: An optimal scheduling method for DDPG energy systems based on improved priority experience replay includes: S1: Design energy-specific constraints, including power balance constraints, distributed generator ramp rate constraints, and energy storage system charge state boundary constraints, and construct an energy system model; construct a reinforcement learning environment and define the state space, action space, and multi-objective reward function; S2: Calculate the novelty-priority allocation rule index, then calculate the priority of experience samples through multi-dimensional evaluation, and combine the smoothing coefficient with the novelty-priority allocation rule index to calculate the time-series difference error priority and reward priority, and weight them to obtain the final priority, thus constructing a multi-dimensional priority evaluation mechanism; S3: Construct an improved deep deterministic policy gradient network based on the main network and the target network, including an actor network and an improved critic network; the actor network processes the state input and outputs the action through a multilayer perceptron structure, and maps it to the action space through a nonlinear activation function and scaling; the improved critic network adopts a state-action separation structure and outputs a dual-action value after processing. S4: Based on the multi-dimensional priority evaluation mechanism, initialize the parameters of the improved deep deterministic strategy gradient network and the priority experience replay buffer, train cyclically, initialize exploration noise and initial state in each cycle, generate actions, store experience, sample and calculate the value of target actions, optimize the main network, softly update the target network and correct sample priorities according to time steps; stop training when the continuous cycle fluctuation amplitude of the multi-objective reward function result is less than the preset threshold or reaches the preset maximum cycle, and generate the optimal scheduling strategy for the energy system.

[0007] Specifically, the power balance constraint process includes: obtaining the current actual output power of all distributed generators in each scheduling time step to obtain the total power generation; determining the positive or negative attribute of the charging and discharging power according to the current operating mode of the energy storage system, and then calculating the difference between the total power generation and the energy storage charging and discharging power; and summing the net load power and grid interaction power at the current moment to obtain the sum of load and interaction power, and maintaining consistency with the difference through regulation.

[0008] Specifically, the distributed generator ramp rate constraint requires that, within a continuous operating time step, the difference between the output power at the current moment and the output power at the previous moment must be within the preset maximum power increase range of the distributed generator, and the difference between the output power at the current moment and the output power at the next moment must be within the preset maximum power decrease range of the distributed generator.

[0009] Specifically, the process of constraining the state of charge (SOC) boundary of the energy storage system includes: when calculating the SOC of the energy storage system at the current moment, first obtain the SOC of the previous moment, then collect the charging and discharging power, charging and discharging efficiency, and scheduling time step at the current moment, and obtain the current SOC by superimposing the charging and discharging efficiency, the current charging and discharging power, and the time step at the previous moment; when generating the safe operating range of the SOC, set the minimum SOC and the maximum SOC based on the type of energy storage battery and the on-site operation and maintenance requirements; when limiting the range of energy storage charging and discharging power, generate the maximum charging power and the maximum discharging power based on the rated capacity of the energy storage converter.

[0010] Specifically, the state space is composed of: the current time step, the current electricity price, the current state of charge of the energy storage system, the current net load power, and the current output power of each distributed generator. Based on spatiotemporal consistency, all state variables are acquired synchronously.

[0011] Specifically, during the operation of the experience sample at a time step, after the main actor network outputs an action and executes it in the energy system, the current state, the executed action, the generated immediate reward, and the state at the next moment corresponding to the time step are collected and combined to generate an experience sample; the final priority of the experience sample is calculated, and the experience sample is bound to the corresponding final priority and stored in the priority experience replay pool, while maintaining the capacity of the experience pool within a preset range. When the experience pool reaches the capacity limit, the old sample is removed according to the principle of replacing the low-priority sample first.

[0012] Specifically, the novelty-priority allocation rule indicators include: daily peak real-time load ratio, daily maximum power generation to real-time power generation ratio, and load conversion coefficient; obtaining the current real-time load and the maximum real-time load monitored on the same day, and dividing the two to obtain the daily peak real-time load ratio; obtaining the current real-time power generation and the maximum power generation monitored on the same day, and calculating the daily maximum power generation to real-time power generation ratio; obtaining the current photovoltaic output and load power, calculating the absolute value of the difference between the two, and dividing it by the maximum absolute value of the difference over all time steps to obtain the load conversion coefficient.

[0013] Specifically, the process of constructing the actor network includes: when designing the fully connected multilayer perceptron structure of the actor network, firstly setting up multiple hidden layers and one output layer, and gradually extracting complex correlation information from the state features through multiple hidden layers; using a linear rectified activation function for the hidden layers and a hyperbolic tangent activation function for the output layer, mapping the output results to a unified interval, and then combining the operating constraints of the energy system equipment to linearly scale the mapped results.

[0014] Specifically, the process of constructing the improved critic network includes: when designing the state-action separation structure, firstly, state branches and action branches are built; for the state branch, after inputting the state quantity, multi-layer fully connected operations are performed, with each layer using a linear rectified activation function, to extract the value information in the state quantity through multi-layer operations; for the action branch, after inputting the action quantity, a single-layer fully connected operation is performed, using a linear rectified activation function to simplify the operation process; then, a shared layer is built, the output results of the state branch and the action branch are merged and input into the shared layer, and the state and action features are fused through multi-layer fully connected operations and a linear rectified activation function in the shared layer, finally outputting two independent action values.

[0015] Specifically, the process of initializing the exploration noise and initial state during the periodic training is as follows: before the start of the training cycle, the intensity of the exploration noise is adjusted according to the current training progress, the intensity of the exploration noise is increased in the early stage of training, and the intensity of the exploration noise is decreased in the later stage of training; when obtaining the initial state, the actual operating data of the energy system at the beginning of the training cycle is obtained, including equipment operating parameters, load data, and electricity price data, and integrated into an initial state vector as the starting input for periodic training.

[0016] Specifically, the process of generating actions and storing experiences at each time step is as follows: When generating an action, the main actor network receives the state vector of the current time step, outputs a basic action vector after internal calculation, and superimposes the exploration noise with the basic action vector to obtain the final action to be executed; after executing the final action, the network obtains real-time feedback data of the energy system after the action is executed, including data on changes in the system's operating state and the corresponding action value, and integrates the state of the current time step, the final action, the action value, and the state of the next time step into an experience sample; when storing the experience sample, the final priority of the experience sample is calculated first, and then the experience sample is associated with the corresponding priority and stored in the priority experience replay buffer.

[0017] Specifically, the sampling process for calculating the target action value is as follows: when sampling experience samples from the priority experience replay buffer, random sampling is performed based on the sampling probability of the samples, and the sampling results cover samples of different priorities; before calculating the target action value, the state of the next time step in the sampled samples is determined. If it is a non-terminating state, the action value of the sample is combined with the discount factor and the smaller of the two action values ​​output by the target commentator network to calculate the target action value; if it is a terminating state, the action value of the sample is directly used as the target action value.

[0018] The DDPG energy system optimal scheduling algorithm based on improved priority experience replay described in this invention addresses the pain points of existing energy scheduling technologies, such as strong model dependence, low efficiency of experience utilization, insufficient constraint fusion, and multi-objective imbalance. Through innovations such as a multi-dimensional priority evaluation mechanism, deep integration of energy-specific constraints, and improved DDPG network design, it achieves multi-dimensional technical advantages, with the following specific beneficial effects: This invention overcomes the limitations of traditional DDPG random replay and single-dimensional priority evaluation by constructing a multi-dimensional priority evaluation mechanism that integrates the N-PAR index (novelty-priority allocation rule index), TD error (time-series differential error), and rewards. The N-PAR index can quantify the real-time dynamics of the energy system (such as load peak proximity and the matching degree between new energy sources and loads), giving higher priority to empirical samples in highly dynamic scenarios (such as sudden load increases and photovoltaic output fluctuations). Combined with the prediction bias reflected by the TD error and the gain / loss degree reflected by the rewards, high-value samples are further accurately identified. This mechanism prioritizes sampling samples that are more critical to the optimization of the scheduling strategy during training, reducing the training resources occupied by invalid samples. This not only improves the convergence speed of network training but also avoids policy bias in key scenarios, ensuring that the scheduling strategy after training can accurately adapt to the highly dynamic operating needs of the energy system.

[0019] This invention embeds energy system-specific constraints and achieves pre-constraint control through Actor network output scaling: distributed generator ramp rate constraints, energy storage SOC boundary constraints, and power balance constraints are no longer used as "post-verification and correction" conditions, but are directly scaled by the physical constraints at the Actor network output, ensuring that the network output action control quantities (such as generator output and energy storage charging and discharging power) conform to the equipment operating limits. Simultaneously, constraint-related parameters such as energy storage SOC and real-time generator output are incorporated into the state space design, enabling the network to dynamically adjust its actions based on the constraint state. This design solves the "invalid action exploration" problem of existing algorithms, avoids equipment damage caused by actions exceeding constraints (such as deep charging and discharging of energy storage, sudden changes in generator power), and reduces the error introduction in the constraint correction stage, ensuring the operational safety of the energy system and the service life of the equipment.

[0020] This invention achieves a dynamic balance between economy and safety by designing a multi-objective reward function with multi-dimensional sub-reward coordination and combining it with an improved Critic network's dual-action value assessment. In the reward function, the energy storage cost sub-reward suppresses frequent charging and discharging losses, the distributed generator cost sub-reward optimizes power generation costs, and the electricity sales / electronic purchase reward reflects market returns. Simultaneously, the power over-limit penalty sub-reward directly constrains grid interaction power and equipment output over-limit risks, avoiding safety hazards caused by solely pursuing cost optimization. The improved Critic network's dual-action value output avoids overestimation of action values ​​by "taking the smaller value," ensuring a more accurate assessment of the "economy-safety" trade-off. Ultimately, this enables the scheduling strategy to reduce electricity sales and purchase costs and equipment operating costs while strictly meeting safety requirements such as power balance and equipment constraints, achieving a synergy between "cost optimization" and "safe operation."

[0021] This invention leverages the "trial and error learning" characteristic of the reinforcement learning DDPG framework, eliminating the need for precise mathematical models of energy system equipment and accurate predictions of load and renewable energy output. The algorithm dynamically learns equipment operating characteristics and load / renewable energy output patterns through real-time interaction with the energy system environment. Even when faced with model biases caused by equipment aging or prediction errors due to changes in natural conditions, it can adaptively adjust scheduling strategies through continuous accumulation of experience samples and priority replay. Compared to traditional methods such as model predictive control and mixed-integer programming that rely on precise models and predictions, this algorithm enhances its adaptability to uncertainty, maintaining the stability and optimality of scheduling strategies even in energy systems with high distributed energy penetration.

[0022] This invention reduces computational complexity during the training and inference phases by optimizing experience replay efficiency and network structure: a multi-dimensional priority evaluation mechanism reduces the computation of invalid samples, and an improved state-action separation structure of the Critic network simplifies the feature extraction process, enabling the algorithm to quickly complete state analysis and action output within the scheduling cycle; simultaneously, the main Actor network after training convergence can directly output scheduling instructions based on real-time state without complex iterative calculations, fully meeting the real-time scheduling requirements of modern "source-grid-load-storage" collaborative systems and adapting to multi-device collaborative control scenarios after large-scale access of distributed energy.

[0023] In summary, this invention, through multi-dimensional technological innovation, achieves a comprehensive improvement in the "economy, security, real-time performance, and robustness" of energy system dispatching, effectively supporting the coordinated operation of distributed energy and energy storage, and providing technical support for the transformation of modern energy systems from "centralized" to "distributed and coordinated". Attached Figure Description

[0024] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0025] Figure 1 This is a flowchart illustrating an optimal scheduling method for a DDPG energy system based on improved priority experience replay, according to the present invention. Figure 2 This is the training logic diagram of NPER-DDPG in this invention. Detailed Implementation

[0026] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0027] Please see Figure 1-2 An optimal scheduling method for DDPG energy systems based on improved priority experience replay includes: S1: Design energy-specific constraints, including power balance constraints, distributed generator ramp rate constraints, and energy storage system charge state boundary constraints, and construct an energy system model; construct a reinforcement learning environment and define the state space, action space, and multi-objective reward function; S2: Calculate the novelty-priority allocation rule index, then calculate the priority of experience samples through multi-dimensional evaluation, and combine the smoothing coefficient with the novelty-priority allocation rule index to calculate the time-series difference error priority and reward priority, and weight them to obtain the final priority, thus constructing a multi-dimensional priority evaluation mechanism; S3: Construct an improved deep deterministic policy gradient network based on the main network and the target network, including an actor network and an improved critic network; the actor network processes the state input and outputs the action through a multilayer perceptron structure, and maps it to the action space through a nonlinear activation function and scaling; the improved critic network adopts a state-action separation structure and outputs a dual-action value after processing. S4: Based on the multi-dimensional priority evaluation mechanism, initialize the parameters of the improved deep deterministic strategy gradient network and the priority experience replay buffer, train cyclically, initialize exploration noise and initial state in each cycle, generate actions, store experience, sample and calculate the value of target actions, optimize the main network, softly update the target network and correct sample priorities according to time steps; stop training when the continuous cycle fluctuation amplitude of the multi-objective reward function result is less than the preset threshold or reaches the preset maximum cycle, and generate the optimal scheduling strategy for the energy system.

[0028] Specifically, the power balance constraint process includes: obtaining the current actual output power of all distributed generators in each scheduling time step to obtain the total power generation; determining the positive or negative attribute of the charging and discharging power according to the current operating mode of the energy storage system, and then calculating the difference between the total power generation and the energy storage charging and discharging power; and summing the net load power and grid interaction power at the current moment to obtain the sum of load and interaction power, and maintaining consistency with the difference through regulation.

[0029] Specifically, the distributed generator ramp rate constraint requires that, within a continuous operating time step, the difference between the output power at the current moment and the output power at the previous moment must be within the preset maximum power increase range of the distributed generator, and the difference between the output power at the current moment and the output power at the next moment must be within the preset maximum power decrease range of the distributed generator.

[0030] Specifically, the process of constraining the energy storage SOC boundary includes: when calculating the current state of charge (SOC) of the energy storage system, first obtain the SOC of the previous moment, then collect the charging and discharging power, charging and discharging efficiency, and scheduling time step of the current moment, and obtain the current SOC by superimposing the charging and discharging efficiency of the previous moment, the current charging and discharging power, and the time step; when generating the safe operating range of the SOC, set the minimum SOC and the maximum SOC according to the type of energy storage battery and the on-site operation and maintenance requirements; when limiting the range of energy storage charging and discharging power, generate the maximum charging power and the maximum discharging power according to the rated capacity of the energy storage converter.

[0031] System constraint modeling Power balance constraint: The sum of the output power of all distributed generators, minus the current charging and discharging power of the energy storage system (charging is positive, discharging is negative), equals the sum of the current net load power (total load power - photovoltaic self-generated and self-consumed power) and the power exchanged with the grid. Distributed generator constraints: in This represents the maximum output power of the m-th distributed generator (DG) (to avoid damage to the equipment from a sudden increase in output). To maximize power reduction (to avoid grid frequency fluctuations caused by a sudden drop in output). For DG set, For time steps; (3) Constraints of energy storage system: Dynamic update of state of charge: in Let t be the state of charge of the stored energy. For charging and discharging efficiency, For charging and discharging power, For time steps; charged state boundaries: ,in To achieve the minimum SOC, This represents the maximum SOC.

[0032] With the core objective of minimizing the total operating cost of the energy system, the total cost includes the operating cost of distributed generators and the grid interaction cost: Distributed generator cost: ,in The cost coefficient for the m-th distributed generator (determined by the equipment model and fuel price) is given. Let be the actual power output of the m-th distributed generator at the current moment. Grid interaction cost in Let t be the electricity price at time t (peak and off-peak prices differ). This is the electricity purchase penalty coefficient (usually taken as 1.2, to discourage electricity purchases during periods of high electricity prices). This indicates selling electricity to the grid (with negative revenue and costs). This indicates that electricity is purchased from the grid (resulting in positive costs).

[0033] Specifically, the state space is composed of: the current time step, the current electricity price, the current state of charge of the energy storage system, the current net load power, and the current output power of each distributed generator. Based on spatiotemporal consistency, all state variables are acquired synchronously.

[0034] Specifically, during the operation of the experience sample at a time step, after the main actor network outputs an action and executes it in the energy system, the current state, the executed action, the generated immediate reward, and the state at the next moment corresponding to the time step are collected and combined to generate an experience sample; the final priority of the experience sample is calculated, and the experience sample is bound to the corresponding final priority and stored in the priority experience replay pool, while maintaining the capacity of the experience pool within a preset range. When the experience pool reaches the capacity limit, the old sample is removed according to the principle of replacing the low-priority sample first.

[0035] Specifically, the novelty-priority allocation rule indicators include: daily peak real-time load ratio, daily maximum power generation to real-time power generation ratio, and load conversion coefficient; obtaining the current real-time load and the maximum real-time load monitored on the same day, and dividing the two to obtain the daily peak real-time load ratio; obtaining the current real-time power generation and the maximum power generation monitored on the same day, and calculating the daily maximum power generation to real-time power generation ratio; obtaining the current photovoltaic output and load power, calculating the absolute value of the difference between the two, and dividing it by the maximum absolute value of the difference over all time steps to obtain the load conversion coefficient.

[0036] A novelty-priority allocation rule index is proposed to quantify the comprehensive balance state of the energy system's "load-generation-energy storage regulation capacity," covering three dimensions: load intensity, generation capacity, and load transfer burden. Daily peak real-time load ratio ( ): ,in Let t be the real-time load. The maximum real-time load for the day, with a value range of [value range missing]. The larger the ratio, the closer the current load is to the peak load of the day, and the more difficult the scheduling is. Daily maximum power generation to real-time power generation ratio : in The real-time power generation at time t (including DG and photovoltaic). The maximum power generation on that day, with a range of values. The smaller the ratio, the weaker the current power generation capacity and the higher the risk of supply-demand imbalance. Load conversion factor : ,in Photovoltaic output at time t Let be the load power at time t. For the entire day's time step, the range of values ​​is... The larger the ratio, the worse the match between photovoltaic output and load, and the heavier the burden of energy storage / grid regulation. N-PAR index synthesis: The smaller the indicator value, the more balanced the supply and demand of the system, and the better the scheduling strategy.

[0037] This indicator can directly reflect the system's balance status. By influencing the sample priority through a scaling factor, the sample with the larger N-PAR (the more unbalanced the supply and demand) has a larger denominator when calculating the priority. This forces the algorithm to prioritize learning scheduling experience in such high-risk scenarios, helping to optimize energy scheduling and electricity purchase and sale strategies, and improve the system's profitability and operational stability.

[0038] By introducing a scaling factor based on N-PAR (load-generation balance) values, and integrating TD error (reflecting prediction deviation), reward (reflecting immediate benefits), and system balance state (reflecting dispatch risk), a multi-dimensional sample priority calculation model is constructed: TD error priority: ,in It is a smoothing coefficient (balancing the priority of historical and current samples). Prioritizing the TD error from the previous time step. Let be the TD error of the i-th sample; Reward Priority ,in The reward priority is based on the previous moment. The immediate reward for the i-th sample; Final experience priority: ,in For weight This ensures that the contribution of TD error and rewards to priority is balanced.

[0039] Specifically, the process of constructing the actor network includes: when designing the fully connected multilayer perceptron structure of the actor network, firstly setting up multiple hidden layers and one output layer, and gradually extracting complex correlation information from the state features through multiple hidden layers; using a linear rectifier function for the hidden layers and a hyperbolic tangent activation function for the output layer, mapping the output results to a unified interval, and then combining the operating constraints of the energy system equipment to linearly scale the mapped results.

[0040] Specifically, the process of constructing the improved Critic network includes: when designing the state-action separation structure, firstly, state branches and action branches are built; for the state branch, after inputting the state quantity, multi-layer fully connected operations are performed, with each layer using a linear rectified activation function, to extract the value information in the state quantity through multi-layer operations; for the action branch, after inputting the action quantity, a single-layer fully connected operation is performed, using a linear rectified activation function to simplify the operation process; then, a shared layer is built, the output results of the state branch and the action branch are merged and input into the shared layer, and the state and action features are fused through multi-layer fully connected operations and a linear rectified activation function in the shared layer, finally outputting two independent action values.

[0041] Enhanced learning environment design State space definition: ,in This represents the current time step (identifying peak / flat / valley periods). Let be the electricity price at time t. For real-time state of charge of energy storage, For real-time net load power, Provides real-time power output to three DG units (covering critical system operating statuses). Action space definition: in For energy storage charging and discharging power control quantity, For the output control of the m-th DG, all actions must meet the physical constraints of the equipment; Reward function design: A multi-objective reward function is adopted, which comprehensively balances cost, benefit, and system stability. The formula is as follows: ,in This is the reward scaling factor (usually set to 0.01 to maintain a uniform reward scale); the definitions of each sub-reward are as follows: Energy storage cost sub-rewards , Changes in energy storage This is the battery cost factor (to suppress frequent charging and discharging). DG cost sub-rewards , (At that time, DG had no downtime costs). Excess electricity: penalty and reward , (Maximum grid interaction power), penalizing power abandonment; : Load lacks punishment sub-reward, Punish those who fail to provide power; Electronic sales rewards , This is the electricity sales revenue coefficient (usually taken as 0.9, taking into account the grid settlement discount). Purchase of electronic items as a reward , This reflects the cost of electricity purchase (as a negative incentive).

[0042] To ensure that the scheduling strategy meets the physical characteristics of the equipment and the requirements for safe system operation, multi-level constraints are set: Energy storage charging and discharging power constraints: ,in The maximum discharge power is determined by the rated capacity of the energy storage converter, usually taken as 100kW. This is the maximum charging power (same as the discharging power). Energy storage SOC constraints: , , ; Distributed generator output constraints: ,in Minimum technical output for DG (e.g., 10kW for DG1). The rated output of DG (e.g., DG1 is 150kW). Power grid interaction constraints: , The value is set by the distribution network access protocol (usually 300kW). Power balance soft constraint: through The punishment-reward system transforms "hard constraints" into "soft constraints," preventing actions from becoming ineffective due to constraint truncation and improving training stability.

[0043] Specifically, the process of initializing the exploration noise and initial state during the periodic training is as follows: before the start of the training cycle, the intensity of the exploration noise is adjusted according to the current training progress, the intensity of the exploration noise is increased in the early stage of training, and the intensity of the exploration noise is decreased in the later stage of training; when obtaining the initial state, the actual operating data of the energy system at the beginning of the training cycle is obtained, including equipment operating parameters, load data, and electricity price data, and integrated into an initial state vector as the starting input for periodic training.

[0044] Specifically, the process of generating actions and storing experiences at each time step is as follows: When generating an action, the main actor network receives the state vector of the current time step, outputs a basic action vector after internal calculation, and superimposes the exploration noise with the basic action vector to obtain the final action to be executed; after executing the final action, the network obtains real-time feedback data of the energy system after the action is executed, including data on changes in the system's operating state and the corresponding action value, and integrates the state of the current time step, the final action, the action value, and the state of the next time step into an experience sample; when storing the experience sample, the final priority of the experience sample is calculated first, and then the experience sample is associated with the corresponding priority and stored in the priority experience replay buffer.

[0045] Specifically, the specific process of sampling to calculate the target action value is as follows: when sampling experience samples from the priority experience replay buffer, random sampling is performed based on the sampling probability of the samples, and the sampling results cover samples of different priorities; before calculating the target action value, the state of the next time step in the sampled sample is determined. If it is a non-terminating state, the action value of the sample is combined with the discount factor and the smaller of the two action values ​​output by the target Critic network to calculate the target action value; if it is a terminating state, the action value of the sample is directly used as the target action value.

[0046] In this embodiment, NPER-DDPG inherits the "Actor-Critic + Main / Target Network" two-layer architecture of DDPG, and the core improvements focus on the multi-dimensional priority experience playback mechanism (NPER) and the reconstruction of the Critic network structure.

[0047] Actor Network (Action Generation) Employing a fully connected multilayer perceptron architecture, it focuses on "accurately outputting continuous actions that meet constraints": Structure: Input layer (7-dimensional state) → Hidden layer 1 (256 neurons, ReLU activation) → Hidden layer 2 (128 neurons, ReLU activation) → Output layer (4-dimensional action, tanh activation); Key processing: After mapping the tanh output to [-1,1], it is linearly scaled according to the device constraints (e.g., energy storage power: -1→-100kW, 1→100kW) to ensure compliance of the operation.

[0048] Improve the Critic network (action value assessment) Breaking away from the traditional "state-action fusion input" model, it adopts a state-action separation structure to avoid overestimating value: State branching: Input 7-dimensional state → Hidden layer 1 (256 neurons, ReLU) → Hidden layer 2 (128 neurons, ReLU), mining the hidden value features of the state; Action branch: Input 4-dimensional action → hidden layer (128 neurons, ReLU), simplifying action feature extraction (avoiding excessive coupling with state); Shared layer: merge state branch and action branch output → hidden layer 1 (256 neurons, ReLU) → hidden layer 2 (128 neurons, ReLU) → output layer (2 independent Q values: Q1, Q2); Valuation: Taking As the final action value, it suppresses the overestimation of value by a single Q network and improves the reliability of the assessment.

[0049] Algorithm training process Initialization phase Network parameters: The parameters of the main Actor / Critic network are randomly initialized using a He normal distribution; the parameters of the target Actor / Critic network are copied from the initial values ​​of the main network. ; Experience replay buffer: Clear and initialize (capacity 50000); Explore noise: Initialize Ornstein-Uhlenbeck noise (initial strength 0.1, linearly decays to 0.01 over training cycles).

[0050] Cyclic training (Episode=1~500) Each cycle corresponds to 1 day (96 time steps), and the process is as follows: Time step initialization: Get the initial state Initialize exploration noise; Action generation and execution: The main Actor network outputs basic actions. The superimposed noise is obtained Execution in the energy system ; Experience Acquisition and Storage: Obtain Instant Rewards Next state Constructing experience tuples ,calculate Then store in the buffer; Sample Sampling and Network Updates: according to Sampling 256 samples (batch size); Calculate the target Q value: Non-termination state ( (Discount factor), Termination state ; Optimize the main Critic network: Minimize loss (Adam optimizer, learning rate 1e-4); Update the main Actor network: Policy gradient ; Target network soft update: , ( (Slowly synchronize the main network); Priority correction: Recalculate samples based on the new TD error of the updated Critic network. And update the buffer; Convergence judgment: If the reward fluctuation is less than 5% for 5 consecutive cycles or reaches the maximum cycle (500), training is stopped, and the output of the main Actor network is the optimal scheduling strategy.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A DDPG energy system optimal scheduling method based on improved priority experience replay, characterized in that, The method comprises the following steps: S1: design energy-specific constraints, including power balance constraints, distributed generator ramp rate constraints, and energy storage system state of charge boundary constraints, to construct an energy system model; construct a reinforcement learning environment, define the state space, action space, and multi-objective reward function; S2: calculate novelty-priority allocation rule indicators, then calculate the priority of experience samples through multi-dimensional evaluation, combine the smoothing coefficient with the novelty-priority allocation rule indicators to calculate the timing difference error priority and the reward priority, and obtain the final priority by weighting, thereby constructing a multi-dimensional priority evaluation mechanism; S3: construct an improved deep deterministic policy gradient network based on a main network and a target network, including an actor network and an improved critic network; the actor network processes state input through a multi-layer perception structure and outputs actions, which are mapped to the action space through a nonlinear activation function and scaling; the improved critic network adopts a state-action separation structure and outputs double-action values after processing; S4: based on the multi-dimensional priority evaluation mechanism, initialize the improved deep deterministic policy gradient network parameters and the priority experience replay buffer, and train in cycles; in each cycle, initialize the exploration noise and the initial state, generate actions, store experiences, sample and calculate target action values, optimize the main network, soft update the target network, and correct the sample priority; stop training when the multi-objective reward function result fluctuates continuously for less than a preset threshold or reaches a preset maximum cycle, and generate an optimal energy system scheduling strategy.

2. The method of claim 1, wherein, In S1, the process of the power balance constraint includes: at each scheduling time step, obtaining the current actual output power of all distributed generators to obtain the total power generation; then determining the positive and negative properties of the charging and discharging power according to the current operating mode of the energy storage system, and then calculating the difference between the total power generation and the energy storage charging and discharging power; based on the net load power and the grid interaction power at the current time, the sum of the load and the interaction power is obtained by adding, and the difference is kept consistent through regulation.

3. The method of claim 1, wherein, In S1, the distributed generator ramp rate constraint is applied to the distributed generator, and the difference between the current time output power and the previous time output power needs to be within the preset maximum power increase range of the distributed generator, and the difference between the current time output power and the next time output power needs to be within the preset maximum power decrease range of the distributed generator.

4. The method of claim 1, wherein, In S1, the process of the energy storage system state of charge boundary constraint includes: when the current state of charge of the energy storage system is calculated, the state of charge at the previous time is first obtained, and then the current charging and discharging power, charging and discharging efficiency, and scheduling time step are collected; the current state of charge is obtained by superimposing the state of charge at the previous time, the charging and discharging efficiency, the product of the current charging and discharging power and the time step; when generating the safe operating interval of the state of charge, the minimum state of charge and the maximum state of charge are set according to the type of energy storage battery and the on-site operation and maintenance requirements; when limiting the energy storage charging and discharging power range, the maximum charging power and the maximum discharging power are generated according to the rated capacity of the energy storage converter.

5. The method of claim 1, wherein, In S1, the specific constitution of the state space includes: current time step, current time price, current state of charge of the energy storage system, current net load power, and current output power of each distributed generator. Based on space-time consistency, all state quantities are synchronously obtained.

6. The method of claim 1, wherein, In S2, during the running of the experience sample at a time step, the current state, the executed action, the instant reward, and the next state corresponding to the time step are collected after the main actor network outputs an action and executes it in the energy system, and the experience sample is generated by combination; The final priority of the experience sample is calculated, the experience sample and the corresponding final priority are bound and stored in the priority experience replay pool, and the capacity of the experience pool is maintained within a preset range. When the experience pool reaches the upper limit of the capacity, old samples are removed according to the principle of replacing low-priority samples first.

7. The method of claim 1, wherein, In S2, the novelty-priority assignment rule index includes: daily peak real-time load ratio, daily maximum generation and real-time generation ratio, and load conversion coefficient. The daily peak real-time load ratio is obtained by dividing the current real-time load by the maximum real-time load monitored on the same day. The daily maximum generation and real-time generation ratio is obtained by dividing the current real-time generation by the maximum generation monitored on the same day. The load conversion coefficient is obtained by calculating the absolute value of the difference between the current photovoltaic output and the load power, and dividing the absolute value by the maximum value of the absolute value of the difference in all time steps.

8. The method of claim 1, wherein, In S3, the process of constructing the actor network includes: when designing the fully connected multi-layer perception structure of the actor network, a plurality of hidden layers and an output layer are first set, and complex correlation information in the state feature is gradually extracted through the plurality of hidden layers; a linear rectifier activation function is used for the hidden layer, and a hyperbolic tangent activation function is used for the output layer, the output result is mapped to a unified interval, and then linear scaling is performed on the mapped result in combination with the operation constraint of the energy system equipment.

9. The method of claim 1, wherein, In S3, the process of constructing the improved critic network includes: when designing the state-action separation structure, a state branch and an action branch are first built; for the state branch, a plurality of fully connected operations are performed after inputting the state quantity, a linear rectifier activation function is used for each layer, and the value information in the state quantity is mined through the plurality of operations; for the action branch, a single-layer fully connected operation is performed after inputting the action quantity, and a linear rectifier activation function is used to simplify the operation process; then, a shared layer is built, the output results of the state branch and the action branch are combined and input into the shared layer, the state and action features are fused through the multi-layer fully connected operation and the linear rectifier activation function of the shared layer, and finally two independent action values are output.

10. The method of claim 1, wherein, In S4, the specific process of initializing the exploration noise and the initial state during the periodic cycle training is: before the start of the training period, the strength of the exploration noise is adjusted according to the current training progress, the exploration noise strength is enhanced at the beginning of the training, and the exploration noise strength is reduced at the later stage of the training; when obtaining the initial state, the actual operation data of the energy system at the starting time of the training period is obtained, including the device operation parameters, the load data, and the price data, which are integrated into an initial state vector as the starting input of the periodic training.

11. The method of claim 1, wherein, In S4, the specific process of the time step generating actions and storing experiences is: when generating actions, the main actor network receives the state vector of the current time step, outputs a basic action vector through internal operation, superimposes the exploration noise on the basic action vector to obtain the final execution action; after executing the final execution action, the real-time feedback data of the energy system after action execution is obtained, including system running state change data and corresponding action value, and the state of the current time step, the final execution action, the action value and the state of the next time step are integrated into an experience sample; when storing the experience sample, the final priority of the experience sample is calculated first, and then the experience sample and the corresponding priority are stored in the priority experience replay buffer.

12. The method of claim 1, wherein, In S4, the specific process of sampling and calculating the target action value is: when sampling the experience sample from the priority experience replay buffer, random sampling is performed based on the sampling probability of the sample, and the sampling result covers samples of different priorities; before calculating the target action value, it is judged that the state of the next time step in the sampled sample is a non-termination state, and the target action value is calculated by combining the action value of the sample, the discount factor and the smaller value of the two action values output by the target critic network; for the termination state, the action value of the sample is directly taken as the target action value.

Images