Micro-grid demand side management method and system based on group reinforcement strategy optimization

Abstract

The application discloses a micro-grid demand side management method and system based on group enhancement strategy optimization, belongs to the technical field of micro-grid management, and comprises the following steps: constructing an environment model based on the state of charge of a battery energy storage system, and the state space is composed of real-time system state data; constructing an adaptive reward function including economic rewards and safety penalties; training an intelligent agent in the environment model by using a group enhancement-based reinforcement learning algorithm, obtaining an optimal energy storage scheduling strategy network model based on the adaptive reward function; and inputting the real-time system state data of the environment model into the intelligent agent, outputting optimal energy storage charging and discharging actions at a time, and realizing intelligent management of the micro-grid demand side. The application realizes real-time, adaptive and high-precision optimal control of the energy storage system without relying on prediction, and improves the operation stability, safety and control intelligent level of the system in the face of renewable energy fluctuation and load change.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of micro-grid management, and particularly relates to a micro-grid demand side management method and system based on group enhancement strategy optimization. BACKGROUND

[0002] The statements herein are provided only to complement the present application and are not necessarily to constitute the prior art.

[0003] With the transformation of global energy structure towards decentralization and low carbonization, the grid-connected scale of renewable energy resources (RERs) such as solar and wind energy is increasingly expanding. As an independent energy system, micro-grid needs to maintain the power balance between power generation and power consumption in real time. Battery energy storage system (BESS) has become a key device for stabilizing the operation of micro-grid due to its fast and flexible power regulation capability.

[0004] However, efficient and safe real-time control of BESS is a complex technical problem. The inventors found that the existing control strategies mainly face the following technical bottlenecks:

[0005] Firstly, traditional methods based on optimization models (such as mixed integer linear programming) or heuristic algorithms rely heavily on accurate prediction of future renewable energy output and load demand. However, in actual operation, prediction errors are inevitable, which will lead the control system to make decisions based on false forward-looking information, not only failing to effectively smooth power fluctuations, but also possibly producing incorrect charging and discharging instructions, exacerbating power imbalance in the system, and even causing bus voltage out-of-limit or frequency fluctuation, threatening the stable operation of micro-grid.

[0006] Secondly, some global state information (such as accurate future electricity price curve, neighborhood load information) relied on during the training phase may not be available in real time during actual deployment, which may cause well-trained algorithms to significantly decrease in performance or completely fail in real environment due to incomplete input information, failing to achieve the expected power regulation function and exposing the system to renewable energy fluctuation risk.

[0007] Thirdly, traditional rule-based strategies, although simple, have fixed logic and are difficult to dynamically adapt to random changes in renewable energy output and load, making it impossible to achieve fine real-time power control. In recent years, multi-agent reinforcement learning and other methods have shown potential in dealing with complex scenarios, but the high computational overhead and delay caused by frequent communication between multiple agents make it difficult to meet the stringent timing requirements of micro-grid real-time control (usually requiring second-level or minute-level response).

[0008] In summary, the existing micro-grid demand side management method has obvious technical defects in real-time, adaptability and robustness to uncertain environment. These defects make it difficult for the micro-grid system to maintain reliable power balance and ensure device safety constraints when facing renewable energy fluctuations, thereby restricting the efficient and safe operation of the micro-grid. SUMMARY

[0009] The purpose of the present application is to overcome the shortcomings of the prior art described above, and to provide a micro-grid demand side management method and system based on group enhancement strategy optimization, which realizes real-time, adaptive and high-precision optimization control of the energy storage system without relying on prediction and ensuring device safety, significantly improving the operation stability, safety and control intelligence level of the micro-grid system when facing renewable energy fluctuations and load changes.

[0010] In order to achieve the above-mentioned purpose, the present application is realized by the following technical solutions:

[0011] On the one hand, the technical scheme of the present application provides a micro-grid demand side management method based on group enhancement strategy optimization, comprising:

[0012] An environment model is constructed based on the state of charge of the battery energy storage system, and the state space thereof is composed of real-time system state data;

[0013] An adaptive reward function including economic rewards and safety penalties is constructed;

[0014] An agent is trained in the environment model using a reinforcement learning algorithm based on group enhancement, and the adaptive reward function guides the update direction of the network model parameters in the agent to obtain an optimal energy storage scheduling strategy network model; the agent takes the real-time system state data in the environment model as input and outputs the optimal energy storage charging and discharging action at the moment, realizing intelligent management of the micro-grid demand side;

[0015] The training includes synchronous group exploration, group-enhanced value function estimation and joint value and advantage estimation.

[0016] In at least one embodiment, the state space of the environment model is a multi-dimensional continuous vector, and the state space of the environment model is represented as: The state vector at the moment t is represented as:

[0017] ;

[0018] Wherein, Each component represents a time index; Represents the real-time local load power; Represents the real-time photovoltaic (PV) power generation; Represents the grid exchange power at the previous moment;​ SoC represents the state of charge of the battery energy storage system at the previous time step; a represents the action taken by the agent at the previous time step; Pgrid represents the real-time grid purchase price; D represents the real-time demand at the previous time step.

[0019] In at least one embodiment, the state space of the environment model is a normalized continuous interval, in a normalized action at the previous time step is mapped to the physical charge and discharge power of the energy storage system, represented as:

[0020]

[0021] wherein, a represents the action taken by the agent at the previous time step, with positive values for charging actions and negative values for discharging actions; is the maximum rated charge and discharge power of the BESS; a represents the charging instruction to the battery energy storage system; a represents the discharging instruction to the battery energy storage system;

[0022] The update rule of the state of charge of the battery energy storage system is:

[0023]

[0024] wherein, SoC represents the state of charge of the battery energy storage system at the previous time step; a represents the charging efficiency; a represents the discharging efficiency; a represents the unit time interval of energy storage scheduling; a represents the energy storage capacity.

[0025] In at least one embodiment, the economic reward is the difference between the baseline operation cost without the energy storage system and the actual operation cost; the safety penalty is a fixed negative value penalty when the state of charge of the energy storage system is out of bounds.

[0026] In at least one embodiment, the actual operation cost is composed of the electricity charge and the demand charge in incremental form;

[0027] The electricity charge is calculated based on the real-time purchase and sale price and the real-time input and output power of the microgrid; the real-time input and output power of the microgrid is calculated based on the real-time external power of the microgrid;

[0028] ​​​Incremental demand charges are obtained by comparing the current demand with the historical highest demand. If the current demand exceeds the historical highest demand, an incremental demand penalty is added as part of the reward function; otherwise, the incremental demand charge is zero.

[0029] In at least one embodiment, synchronized group exploration includes:

[0030] At the start of each training update, the system is initialized with the same random seed. An environment that causes a group of agents to start from the exact same initial state and face the same sequence of external random events;

[0031] Through policy function Randomness of actions, generating Trajectory data with a consistent structure but diverse behaviors.

[0032] In at least one embodiment, group-enhanced value function estimation includes:

[0033] Based on the discount factor, calculate the real-time Monte Carlo reward sequence for each agent's trajectory;

[0034] The mean and standard deviation of the Monte Carlo reward sequence are calculated, and the reward value at each time point within the trajectory is standardized to obtain the standardized reward value as the learning target of the value network.

[0035] In at least one embodiment, the joint value and advantage estimation includes:

[0036] Calculate the average return of all trajectories in the current batch at the same time to obtain the average sequence;

[0037] Standardizing the mean series yields empirical value estimates with low variance;

[0038] Historical value estimates and low-variance empirical value estimates are weighted and mixed using weighting factors to generate a joint baseline;

[0039] Based on the standardized returns and the joint baseline, the final advantage function estimate is calculated.

[0040] In at least one embodiment, the training of the agent is achieved by optimizing a composite loss function that includes policy loss and value loss;

[0041] The strategy loss is specifically represented as follows:

[0042]

[0043] in, Indicates the first One intelligent agent; Indicates time; a probability ratio representing new and old policies; a hyperparameter for defining a range of a credible region; an advantage function estimate;

[0044] The value loss is specifically represented as:

[0045]

[0046] wherein, a total number of time points; a number of parallel environments; a historical value estimate; an empirical value estimate with low variance.

[0047] In another aspect, the technical solution of the present application also provides a micro-grid demand side management system based on group-enhanced policy optimization, comprising:

[0048] An environment model construction module is configured to construct an environment model based on the state of charge of the battery energy storage system, the state space of which is composed of real-time system state data;

[0049] An adaptive reward function construction module is configured to construct an adaptive reward function including economic rewards and safety penalties;

[0050] An optimization training module is configured to train an agent in the environment model using a group-enhanced reinforcement learning algorithm, guide the update direction of network model parameters in the agent based on the adaptive reward function, and obtain an optimal energy storage scheduling strategy network model; the agent takes the real-time system state data in the environment model as input and outputs the optimal energy storage charging and discharging action at the time, thereby realizing intelligent management of the micro-grid demand side.

[0051] The training includes synchronous group exploration, group-enhanced value function estimation, and joint value and advantage estimation.

[0052] The technical solution of the present application has the following advantages:

[0053] 1) The micro-grid demand side management method based on group-enhanced policy optimization of the present application can realize real-time, adaptive, and high-precision optimization control of the energy storage system without relying on prediction and ensuring equipment safety, thereby significantly improving the operation stability, safety, and control intelligence level of the micro-grid system in the face of renewable energy fluctuations and load changes.

[0054] 2) The application fundamentally eliminates the risk of control instability caused by prediction errors by constructing a real-time environment model based on energy storage state of charge (SOC) without prediction; at the same time, the state variables defined by the environment model are all physical quantities that can be directly obtained in real time during deployment within the microgrid, ensuring the consistency of the training environment and the real deployment environment, effectively solving the deployment failure problem caused by "information missing", and improving the robustness and reliability of the microgrid demand side management control system in the face of uncertainty.

[0055] 3) The application avoids control strategies that damage equipment assets by setting a safety penalty term, effectively solving the safety problem of control instability; by setting an economic reward term, the agent learns the smooth control strategy of "peak clipping and valley filling" adaptively, dynamically responding to changes in the system and price, which is more flexible and efficient than traditional fixed rules.

[0056] 4) The GREPO algorithm of the application greatly reduces the variance in the learning process through synchronous group exploration and group-enhanced value estimation, accelerating the convergence speed, which enables the algorithm to train high-performance strategies faster and more stably, overcoming the instability and slow convergence of traditional reinforcement learning in high-dimensional and complex reward environments such as microgrids, effectively improving learning efficiency and stability. BRIEF DESCRIPTION OF DRAWINGS

[0057] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this application. The embodiments of the application illustrated in the drawings, and their description, are presented to add generic scope to the application, and do not in any way limit the application.

[0058] Figure 1 is a step schematic diagram of the microgrid demand side management method based on group-enhanced strategy optimization disclosed in embodiment 1 of the application;

[0059] Figure 2 is a framework schematic diagram of the microgrid demand side management method based on group-enhanced strategy optimization disclosed in embodiment 1 of the application;

[0060] Figure 3 is a schematic diagram of the action of energy storage and the curves of other resources in the park under the control of GREPO disclosed in embodiment 1 of the application;

[0061] Figure 4 is a schematic diagram of the intraday time-of-use price change curve disclosed in embodiment 1 of the application. DETAILED DESCRIPTION

[0062] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise specified, all technical and scientific terms used in the application have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0063] As introduced in the background, the purpose of the present application is to overcome the shortcomings of the prior art, provide a micro-grid demand side management method and system based on group enhancement strategy optimization, realize real-time, adaptive and high-precision optimization control of the energy storage system under the premise of not relying on prediction and ensuring equipment safety, and significantly improve the operation stability, safety and control intelligence level of the micro-grid system in the face of renewable energy fluctuation and load change.

[0064] Embodiment 1

[0065] In a typical embodiment of the present application, as shown in Figure 1 and Figure 2 The present embodiment discloses a micro-grid demand side management method based on group enhancement strategy optimization, comprising the following steps:

[0066] S1. Constructing an environment model based on the state of charge of the battery energy storage system, the state space of which is composed of real-time system state data;

[0067] S2. Constructing an adaptive reward function including economic rewards and safety penalties;

[0068] S3. Training an agent in the environment model using a reinforcement learning algorithm based on group enhancement, guiding the update direction of network model parameters in the agent based on the adaptive reward function, and obtaining an optimal energy storage scheduling strategy network model; the agent takes the real-time system state data in the environment model as input and outputs the optimal energy storage charging and discharging action at the moment, realizing intelligent management of the micro-grid demand side;

[0069] Wherein, the training includes synchronous group exploration, group enhanced value function estimation and joint value and advantage estimation.

[0070] The above micro-grid demand side management method based on group enhancement strategy optimization will be described in detail below in conjunction with specific embodiments.

[0071] S1. Constructing an environment model based on the state of charge of the battery energy storage system, the state space of which is composed of real-time system state data.

[0072] In traditional approaches, the optimization model (e.g., mixed integer linear programming) or heuristic algorithm relies heavily on the accurate prediction of future renewable energy output and load demand. However, in actual operation, prediction errors are inevitable, and such errors will cause the control system to make decisions based on false forward-looking information. To break away from the dependence on prediction, this step builds an environment model centered on the state of charge of the battery energy storage system. The model is built as a constrained Markov decision process (CMDP), and its key feature is that it is based entirely on real-time system data that can be directly measured at the corresponding decision time point, without any prediction information about the future.

[0073] Specifically, the state space of the environment model is defined as an 8-dimensional continuous vector, which comprehensively captures the key information of the microgrid at each time. In this step, one day is divided into time points with granularity, and at each time point, state reading and decision action output occur. The state vector at time is defined as:

[0074]

[0075] wherein represents the time index; represents the real-time local load power, with the unit of ; represents the real-time photovoltaic (PV) power generation, with the unit of ; represents the grid exchange power at the previous time, with the unit of ; represents the state of charge (SOC) of the battery energy storage system (BESS) at the previous time; represents the action taken by the agent at the previous time; represents the real-time grid purchase price, with the unit of yuan / ; represents the instantaneous demand at the previous time. These state vectors need to be normalized before inputting the environment model.

[0076] The action space of the environment model is designed as a normalized continuous interval , which represents the charging or discharging instruction of the BESS. The normalized action at the time is mapped to the physical charging and discharging power of the energy storage system, represented as:

[0077]

[0078] wherein,​​ represents the action taken by the agent at the moment, the charging action takes positive value, and the discharging action takes negative value; is the maximum rated charging and discharging power of the BESS, and the unit is ; represents the instruction to charge the battery energy storage system; represents the instruction to discharge the battery energy storage system.

[0079] The energy storage state of charge of the battery energy storage system is updated in a deterministic manner according to the physical law, while considering the charging efficiency and the discharging efficiency , and the updating rule is as follows:

[0080]

[0081] wherein, represents the energy storage state of charge of the battery energy storage system at the previous moment; represents the charging efficiency; represents the discharging efficiency; represents the unit time interval of energy storage scheduling; represents the energy storage capacity.

[0082] This step constructs a real-time environment model based on the energy storage state of charge (SOC) without prediction, which completely abandons the prediction of future power generation and load, and the state space is entirely composed of real-time measurement data, which fundamentally eliminates the risk of control instability caused by prediction errors. At the same time, the state variables defined by the environment model in this step are all physical quantities that can be directly obtained in real time when deploying the microgrid, which ensures the consistency of the training environment and the real deployment environment, effectively solves the deployment failure problem caused by "information loss", and can improve the robustness and reliability of the microgrid demand side management control system in the face of uncertainty.

[0083] S2. Construct an adaptive reward function including economic rewards and safety penalties.

[0084] In the actual management process of the demand side of the microgrid, the economic optimization goal is often limited by the two-part electricity price structure commonly used in the current electricity market. This electricity price structure divides the total electricity fee into two parts: one is the electricity charge based on the total consumed electricity, usually adopting the time-of-use (TOU) mechanism, i.e., according to the peak and valley rates specified by the user's electricity usage pattern; the other is the demand charge based on the highest average power obtained from the main grid within the billing period (usually 15 minutes), which aims to ensure that users reasonably share the generation and infrastructure costs generated by the grid to meet their maximum power demand, while ensuring the stability of the grid. For most industrial and commercial users, the demand charge often accounts for an important part of the total electricity fee, so effectively reducing this fee also becomes one of the goals of economic optimization of the microgrid.

[0085] This two-part electricity price structure brings double challenges to the economic operation of the microgrid. First, the electricity usage peak of industrial and commercial microgrids usually occurs during the daytime working hours, which frequently overlaps with the high price period of the time-of-use electricity price. Without effective energy storage scheduling or load adjustment capability, the electricity charge will increase significantly. Second, the control of the demand charge faces the risk of renewable energy output fluctuations. Although daytime renewable energy generation can replace high-priced grid purchases to some extent, the inherent intermittency and volatility of renewable energy output bring significant challenges to the precise control and optimization of electricity peak (demand). For example, when weather factors cause a sudden drop in renewable energy output, and local load remains high, the microgrid must instantly purchase a large amount of electricity from the main grid to maintain balance, which is likely to result in a high demand charge.

[0086] Therefore, this step designs a collaborative optimization adaptive reward function to collaboratively optimize the electricity charge and the demand charge while ensuring the safe operation of the energy storage system. Since the monthly demand charge cannot obtain the corresponding demand peak at the end of the settlement period, the feedback of the demand charge part has two characteristics: sparsity and delay. The adaptive reward function in this step decomposes the demand charge target into immediate and intensive reward signals, so that the agent can obtain feedback signals related to demand after each decision in the training phase.

[0087] In this step, the adaptive reward function is composed of economic rewards and safety penalties, which can be specifically represented as:

[0088]

[0089] wherein, represents the economic reward; represents the safety penalty.

[0090] Specifically, the economic reward Defined as the difference between the baseline operating cost without a strategy and the actual operating cost, without an energy storage system, it can be expressed as:

[0091]

[0092] in, This represents the baseline operating cost without a strategy. This represents the actual operating cost. This definition helps guide the agent to be trained with reference to a baseline operating cost, thereby achieving better charging and discharging decision-making performance.

[0093] As a further implementation method, actual operating costs It consists of electricity charges for electricity consumption and incremental demand charges, which can be specifically expressed as:

[0094]

[0095] in, Indicates electricity consumption and electricity cost; This represents the incremental form of demand electricity charges.

[0096] In this step, electricity consumption and electricity costs... Based on real-time electricity purchase and sale prices and the real-time input and output power of the microgrid, the calculation can be specifically expressed as follows:

[0097]

[0098] in, express The input power of the microgrid at any given time, in units of ; express The output power of the microgrid at any given time, in units of ; express The electricity purchase price at any given time, in yuan / ; express The electricity price at any given time, in yuan / ; This indicates the time interval between transactions between the microgrid and the power company, in units of 12000 m / s. It is important to note that only one electricity purchase or sale is allowed per transaction; simultaneous purchase and sale of electricity are not permitted.

[0099] As a further implementation, the real-time power interaction (input / output) between the microgrid and the main grid is calculated by measuring the net external power at its point of connection (PCC). The sign and magnitude of this value directly reflect the power flow direction and amplitude. Specifically, in At any given moment, define the microgrid's net external power. is:

[0100]

[0101] wherein, is the load electric power, in units of ; is the photovoltaic power generation power, in units of ; is the energy storage charging and discharging power, in units of , when is positive, it is the charging power, when is negative, it is the discharging power. When is positive, , ; when is negative, , .

[0102] In this step, the incremental demand charge is obtained by comparing the current demand with the historical maximum demand. Specifically, define the demand at time is the average of the input power of the microgrid in the previous sliding window, which can be specifically expressed as:

[0103]

[0104]

[0105] At time, if the current demand exceeds the historical maximum demand (unit: ), an incremental demand penalty will be added as part of the reward function, i.e. a non-zero demand charge feedback signal; otherwise, the incremental demand charge feedback signal is zero, indicating that no additional demand charge is introduced at the current time. Therefore, the incremental demand charge can be expressed as:

[0106]

[0107] wherein, is a weight factor for balancing the demand cost and the electricity cost.

[0108] In this step, in order to ensure the long-term health of the energy storage system and the safety of the system, when an action of the agent causes the SOC of the energy storage system to exceed the preset safe range , the system will impose a huge fixed negative penalty Safety penalty term in reward function to help the agent understand the safety limit, which can be expressed as:

[0109] .

[0110] where the fixed negative value penalty is an empirical value that can be adjusted according to the energy storage, load, photovoltaic device configuration and electricity price information of the environment. The adaptive reward function of this step effectively guarantees the safety and dynamic adaptability of the control strategy. First, by setting the safety penalty term, a large negative reward is directly applied to the SOC that exceeds the safety range, forcing the agent to consider battery safety as a hard constraint during the learning process, avoiding control strategies that damage device assets, and effectively solving the safety problem of control instability. Second, the economic reward term essentially guides the agent to minimize the fluctuation of grid exchange power. By converting the "demand charge" into an incremental penalty, it can adaptively guide the agent to learn a "peak clipping and valley filling" smoothing control strategy, dynamically respond to changes in the system and price, and be more flexible and efficient than traditional fixed rules.

[0111] S3. Training the agent in the environment model using the group-enhanced reinforcement learning algorithm, guiding the update direction of the network model parameters in the agent based on the adaptive reward function, obtaining the optimal energy storage scheduling strategy network model; the agent takes the real-time system state data in the environment model as input, and outputs the optimal energy storage charging and discharging action at the moment, realizing intelligent management of the demand side of the microgrid;

[0112] In order to achieve stable learning in a high-dimensional random environment without prediction and effectively process complex composite reward signals, this step proposes a novel group-enhanced learning algorithm (GREPO) based on the PPO framework. By introducing a unique group training architecture and hybrid feedback mechanism, the evaluation accuracy of the value function is significantly improved and the variance of the policy gradient is reduced, enabling the trained agent to perform better on microgrid energy management tasks.

[0113] In this step, training the agent in the environment model using the group-enhanced reinforcement learning algorithm includes synchronous group exploration, group-enhanced value function estimation, and joint value and advantage estimation, which are introduced below.

[0114] In this step, synchronous group exploration refers to exploring a group of synchronous parallel environments, rather than a single or multiple asynchronous agents. The implementation process includes: at the beginning of each training update, initializing environments through a same random seed, so that a group of agents start from the same initial state and face the same sequence of external random events (such as load power consumption, photovoltaic power generation, etc.); through the policy function action randomness, generate trajectory data with consistent structure but diverse behaviors .

[0115] based on synchronous group exploration, this difference in agent trajectories is solely due to the randomness in action selection. This synchronous group strategy design has the following advantages: 1) Create a controlled environment: It isolates the impact of policy randomness on results, allowing the final returns of different action sequences to be directly and fairly compared, thereby enhancing the algorithm's ability to identify the causal relationship between actions and long-term returns.

[0116] 2) Generate structured data: It can generate a set of trajectories that are structurally consistent but behaviorally diverse, which helps to build more stable and accurate state value and advantage estimates, laying a solid foundation for subsequent variance reduction mechanisms.

[0117] It should be noted that the present application adopts a standard Actor-Critic architecture, which consists of two independent network models: the policy network (Actor) and the value network (Critic), which are used to approximate the policy function and the value function, respectively. In the Actor-Critic method, the high variance of the value network (Critic) learning objective is the main source of instability in training. Traditional Monte Carlo returns are calculated based on a single random trajectory, which results in a highly noisy estimate. Therefore, in this step, a group-enhanced value estimation mechanism is introduced, which fundamentally changes the Critic's learning objective through trajectory return calculation and trajectory internal return normalization to reduce variance.

[0118] Specifically, trajectory return calculation refers to calculating the Monte Carlo return sequence at time

[0119] for each agent's trajectory based on the discount factor after synchronous group exploration, which is specifically represented as:

[0120]

[0121] where is a temporary index variable.

[0122] To reduce the variance introduced by the difference in reward scales between different trajectories, the return is normalized within each trajectory to prevent outliers with extreme return values from dominating the update process, thereby stabilizing training. ​​​

[0123] For each agent , the mean and standard deviation of its return sequence are calculated, and then the return value at each time point within its trajectory is standardized to obtain the standardized return value , which is used as the learning target of the Critic network. By learning to predict this standardized return value, the Critic network no longer needs to model the high-variance absolute return, but learns a more robust and abstract state-relative value representation, thereby improving the stability of learning. Among them, the mean and standard deviation of the return sequence and the standardized return value are calculated as follows:

[0124]

[0125]

[0126]

[0127] where represents the normalized stability term, which is used to avoid division by zero or numerical instability.

[0128] To further reduce variance and stabilize policy gradient, this step introduces a joint baseline design of historical value estimation and current batch value estimation to estimate joint value and advantage function.

[0129] Specifically, the current batch value estimation is obtained by empirical estimation of the current batch trajectory, which represents the immediate value assessment of the current batch. First, the mean of the return of different groups of agents at each time point is calculated to obtain the mean sequence . The elements of this mean sequence are unbiased Monte Carlo estimates of the return under the current policy, but have large variance. Then, the mean and standard deviation of the mean sequence itself are used for standardization, thereby obtaining a low-variance empirical value estimate , which is specifically represented as:

[0130]

[0131]

[0132]

[0133]

[0134] Subsequently, the historical value estimate is combined with the low-variance empirical value estimate by a weighting factor to generate a joint baseline , which can be expressed as:

[0135]

[0136] where the historical value estimate is the output of the Critic network , which represents a general long-term value expectation learned from the training history, with low variance but possibly biased (especially in the early training stage).

[0137] Finally, the final advantage function estimate is calculated based on the normalized return and the joint baseline , which can be expressed as:

[0138]

[0139] where The function prevents the gradient from affecting the update of the Critic network through the baseline path during backpropagation, ensuring the stability of the training process.

[0140] In this step, both the Actor and Critic networks are implemented using multi-layer perceptrons (MLPs). Each network is designed to process the aforementioned 8-dimensional continuous state vector. The network structure includes an input layer, two hidden layers with 256 and 128 neurons respectively, and an output layer. Both hidden layers use the hyperbolic tangent function (Tanh) as the activation function.

[0141] The output layer of the Actor network calculates the mean of a Gaussian distribution and uses the tangent activation function to limit the mean to the range , corresponding to the normalized action space. The final control action is sampled from this Gaussian distribution, and to ensure consistent exploration behavior and reduce computational overhead during training, the standard deviation of the distribution is set to a fixed value. The output layer of the Critic network is a linear unit, used to output the estimate of the value of the current state.

[0142] As a further implementation, the training of the agent in this step is achieved by optimizing a composite loss function that combines the policy loss of the Actor and the value loss of the Critic.

[0143] Specifically, the update target of the Actor network is to maximize the clipped surrogate objective function of PPO The objective function ensures the stability of training by penalizing updates that deviate too much from the old policy, which can be expressed as:

[0144]

[0145] wherein, represents the i-th agent; represents the time point; represents the probability ratio of the new and old policies, . is a hyperparameter that defines the range of the reliable region, usually . represents the advantage function estimate. The training objective of the Critic network is to make its predicted value

[0146] as close as possible to the low-variance standardized return target calculated in the aforementioned group exploration and group enhancement of the value function estimate . Therefore, the value loss is defined as the mean squared error (MSE) between and , which can be expressed as:

[0147]

[0148] wherein, represents the total number of time points; represents the number of parallel environments; represents the historical value estimate; represents the low-variance empirical value estimate.

[0149] By synchronously optimizing these two loss functions, a policy that can both maximize long-term returns and accurately evaluate state values can be learned stably, thereby achieving efficient and reliable real-time control in complex microgrid environments.

[0150] The GREPO algorithm of this step greatly reduces the variance in the learning process and speeds up the convergence by synchronizing group exploration and group-enhanced value estimation, which enables the algorithm to train a high-performance policy more quickly and stably. This overcomes the problems of instability and slow convergence of traditional reinforcement learning in high-dimensional and complex reward environments such as microgrids, effectively improving learning efficiency and stability. At the same time, the trained policy network (Actor) has extremely small computational complexity when propagating forward (i.e., generating actions based on states), equivalent to the forward calculation of a simple neural network, which is sufficient to meet the requirements of real-time control response at the minute or even second level, effectively solving the problem of large computational overhead and poor real-time performance of complex algorithms.

[0151] ​Under the guidance of the reward function which comprehensively considers the electricity cost, demand cost and SoC constraint, an optimal energy storage scheduling strategy network model can be efficiently and stably trained based on a deep reinforcement learning algorithm GREPO. The agent takes the real-time system state data in the environment model in step S1 as input, trains based on the optimal energy storage scheduling strategy network model, outputs the optimal energy storage charging and discharging action in real time, and applies the optimal energy storage charging and discharging action to the energy storage device in the actual microgrid system, so as to realize intelligent management of the demand side of the microgrid.

[0152] The method is also verified on a dataset of a certain park, and the test results are shown in Table 1.

[0153] Table 1: Demand and electricity cost of each method

[0154]

[0155] The cost saving rate improvement rate is taken as the evaluation benchmark in this verification, and the calculation method of the cost saving rate improvement rate is as follows: taking the daily average cost of the no energy storage scheme as the benchmark, first calculating the cost saving amount realized by the fixed time strategy compared with the no energy storage, then calculating the further reduced cost amount of the GREPO strategy compared with the fixed time strategy, and finally dividing the latter by the former, and the obtained ratio is the relative improvement amplitude of the cost saving rate. The results show that, compared with the existing fixed time charging and discharging method, the cost saving rate of the method is improved by 33.97%.

[0156] In combination with Figure 3 the actual adjustment curve and Figure 4 the price curve, it can be seen that:

[0157] (1) Between 7 o'clock and 10 o'clock, the photovoltaic output is directly used for load power consumption, effectively suppressing the demand.

[0158] (2) Between 10 o'clock and 15 o'clock, the electricity price is in the valley period, the energy storage is charged, and the electricity purchase cost is reduced; at the same time, the photovoltaic output is directly used for load power consumption, effectively suppressing the demand.

[0159] (3) Between 15 o'clock and 17 o'clock, the energy storage discharges almost all the charging in the valley period and is used for load power consumption, reducing the electricity purchase cost in this period.

[0160] (4) Considering the limited capacity of the energy storage, the electricity purchase in the two side periods is directly used for load power consumption, reducing the loss cost caused by inefficient energy storage charging and discharging.

[0161] Embodiment 2

[0162] In a typical embodiment of the present application, the embodiment discloses a microgrid demand side management system based on group enhanced strategy optimization, comprising:

[0163] An environment model construction module is configured to construct an environment model based on the state of charge of the battery energy storage system, a state space of which is constituted by real-time system state data;

[0164] An adaptive reward function construction module is configured to construct an adaptive reward function including economic rewards and safety penalties.

[0165] An optimization training module is configured to train an agent in the environment model by using a reinforcement learning algorithm based on group enhancement, to guide the update direction of network model parameters in the agent based on the adaptive reward function, and to obtain an optimal energy storage scheduling strategy network model; the agent takes the real-time system state data in the environment model as input and outputs an optimal energy storage charging and discharging action at the moment, thereby realizing intelligent management of the demand side of the microgrid.

[0166] The training includes synchronous group exploration, group-enhanced value function estimation, and joint value and advantage estimation.

[0167] The above merely illustrates the preferred embodiments of the present application, but should not be used to limit the present application. Various modifications and changes can be made by those skilled in the art, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the protection scope of the present application.

Claims

1. A microgrid demand-side management method based on group enhancement strategy optimization, characterized in that, include: An environmental model is constructed based on the state of charge of a battery energy storage system, and its state space is composed of real-time system state data. Construct an adaptive reward function that includes economic rewards and security penalties; A group-based reinforcement learning algorithm is used to train an agent in an environment model. An adaptive reward function guides the update direction of the network model parameters in the agent, resulting in an optimal energy storage scheduling strategy network model. The agent takes real-time system state data in the environment model as input and outputs the optimal energy storage charging and discharging action at each moment, realizing intelligent management of the microgrid demand side. The training includes synchronous group exploration, group-enhanced value function estimation, and joint value and advantage estimation. Synchronous group exploration includes: At the start of each training update, the system is initialized with the same random seed. An environment that causes a group of agents to start from the exact same initial state and face the same sequence of external random events; Through policy function Randomness of actions, generating Trajectory data with a consistent structure but diverse behaviors; Group-enhanced value function estimation includes: Based on the discount factor, calculate the real-time Monte Carlo reward sequence for each agent's trajectory; Calculate the mean and standard deviation of the Monte Carlo reward sequence, and standardize the reward value at each time point within the trajectory to obtain the standardized reward value as the learning target of the value network; The combined value and advantage assessment includes: Calculate the average return of all trajectories in the current batch at the same time to obtain the average sequence; Standardizing the mean series yields empirical value estimates with low variance; Historical value estimates and low-variance empirical value estimates are weighted and mixed using weighting factors to generate a joint baseline; Based on the standardized returns and the joint baseline, the final advantage function estimate is calculated.

2. The microgrid demand-side management method based on group enhancement strategy optimization as described in claim 1, characterized in that, The state space of the environment model is a multidimensional continuous vector. State vector at time step Represented as: in, This indicates that each component represents a time index; Indicates real-time local load power; Indicates real-time photovoltaic (PV) power generation; This represents the power exchanged by the power grid at the previous moment; This indicates the state of charge of the battery energy storage system at the previous moment. This indicates the action taken by the agent at the previous moment; This indicates the real-time grid purchase price of electricity; It indicates the immediate needs of the previous moment.

3. The microgrid demand-side management method based on group enhancement strategy optimization as described in claim 1, characterized in that, The action space of the environment model is a normalized continuous interval. Normalized action at time The physical charging and discharging power of the energy storage system is represented as: in, express The actions taken by the intelligent agent at any given time are as follows: charging actions are considered positive, and discharging actions are considered negative. This is the maximum rated charge and discharge power of BESS; This indicates a command to charge the battery energy storage system; This indicates a command to discharge the battery energy storage system; The update rule for the state of charge of a battery energy storage system is as follows: in, This indicates the state of charge of the battery energy storage system at the previous moment. Indicates charging efficiency; Indicates discharge efficiency; Indicates the unit time interval for energy storage dispatch; Indicates energy storage capacity.

4. The microgrid demand-side management method based on group enhancement strategy optimization as described in claim 1, characterized in that, The economic reward is the difference between the base operating cost without a strategy and the actual operating cost when no energy storage system is configured; the safety penalty is a fixed negative penalty imposed when the energy storage's state of charge exceeds the limit.

5. The microgrid demand-side management method based on group enhancement strategy optimization as described in claim 4, characterized in that, The actual operating cost consists of electricity charges for electricity consumption and incremental demand charges. Electricity consumption and charges are calculated based on real-time electricity purchase and sale prices and the real-time input and output power of the microgrid; the real-time input and output power of the microgrid are calculated based on the real-time external power of the microgrid. Incremental demand charges are obtained by comparing the current demand with the historical highest demand. If the current demand exceeds the historical highest demand, an incremental demand penalty is added as part of the reward function; otherwise, the incremental demand charge is zero.

6. The microgrid demand-side management method based on group enhancement strategy optimization as described in claim 1, characterized in that, The training of the agent is achieved by optimizing a composite loss function that includes policy loss and value loss; The strategy loss is specifically represented as follows: in, Indicates the first One intelligent agent; Indicates time; This represents the probability ratio between the old and new strategies; Hyperparameters defining the range of the trusted region; This represents the estimation of the advantage function; The loss of value is specifically expressed as follows: in, Indicates the total number of time points; This represents the number of parallel environments; This indicates an estimated historical value. This represents an empirical value estimate with low variance.

7. A microgrid demand-side management system based on group enhancement strategy optimization, characterized in that, include: The environment model construction module is configured to: construct an environment model based on the state of charge of the battery energy storage system, whose state space is composed of real-time system state data; The adaptive reward function building module is configured to: build an adaptive reward function that includes economic rewards and security penalties; The optimized training module is configured to: train the agent in the environment model using a group-based reinforcement learning algorithm; guide the update direction of the network model parameters in the agent based on an adaptive reward function to obtain the optimal energy storage scheduling strategy network model; the agent takes the real-time system state data in the environment model as input and outputs the optimal energy storage charging and discharging action at the specified time to realize intelligent management of the microgrid demand side. The training includes synchronous group exploration, group-enhanced value function estimation, and joint value and advantage estimation. Synchronous group exploration includes: At the start of each training update, the system is initialized with the same random seed. An environment that causes a group of agents to start from the exact same initial state and face the same sequence of external random events; Through policy function Randomness of actions, generating Trajectory data with a consistent structure but diverse behaviors; Group-enhanced value function estimation includes: Based on the discount factor, calculate the real-time Monte Carlo reward sequence for each agent's trajectory; Calculate the mean and standard deviation of the Monte Carlo reward sequence, and standardize the reward value at each time point within the trajectory to obtain the standardized reward value as the learning target of the value network; The combined value and advantage assessment includes: Calculate the average return of all trajectories in the current batch at the same time to obtain the average sequence; Standardizing the mean series yields empirical value estimates with low variance; Historical value estimates and low-variance empirical value estimates are weighted and mixed using weighting factors to generate a joint baseline; Based on the standardized returns and the joint baseline, the final advantage function estimate is calculated.

Images