A real-time energy management optimization method based on load feedback correction

Abstract

The application provides a real-time energy management optimization method based on load feedback correction, constructs a micro-grid structure model containing renewable energy, defines optimization objectives and operation constraint conditions of the micro-grid, the optimization objectives are minimum system operation cost, and the operation constraint conditions include device power upper and lower limit constraints and system power balance constraints; a MAPPO energy management reinforcement learning model is constructed based on a multi-agent Markov decision process, independent agents are configured for each power supply area of the micro-grid, a state set, an action set and a multi-dimensional reward function of each agent are defined; a load feedback is introduced to perform double-layer correction on a strategy distribution mean value parameter of the MAPPO algorithm, and the strategy distribution mean value parameter is corrected; the MAPPO algorithm is used to update the strategy based on the corrected strategy distribution mean value parameter; whether convergence is judged, if the convergence is judged, the micro-grid real-time energy management optimization strategy is output. The algorithm divergence problem caused by uncertainty is solved, the stability of the algorithm is improved while the energy management optimization is realized.

Description

Technical Field

[0001] This invention relates to the technical field of economic dispatch of smart grids, and in particular to a real-time energy management optimization method for microgrids. Background Technology

[0002] With rapidly increasing energy demand and the growing environmental problems caused by traditional fossil fuel power generation, microgrids with a high proportion of renewable energy penetration have become a new development direction. These microgrids are small-scale power generation and distribution systems composed of various distributed power sources, loads, and energy conversion devices, including renewable energy sources. As a controllable, flexible, economical, and green source-grid-load-storage system, they can not only serve as backup energy during main grid failures, improving the overall reliability of the power system, but also enhance the utilization rate of renewable energy, contributing to the construction of a clean, low-carbon, safe, and efficient new energy system.

[0003] Microgrid optimization methods are mainly divided into model-based methods and data-driven methods. Model-based methods rely on accurate mathematical models, achieving optimized operation by solving these models. For a single microgrid, mixed-integer nonlinear programming, two-layer optimal scheduling considering load characteristics, or two-stage optimal scheduling considering demand-side management for wind-solar-gas-storage microgrids are employed. Distributed model methods are often used for optimizing the operation of multiple integrated energy microgrids. For example, the alternating direction multiplier method considering load demand response is used to achieve distributed optimization of a park's integrated energy system. Another example is the distributed optimization method for heterogeneous integrated energy systems, using the optimal energy market clearing price to ensure maximum benefit. Game theory model methods are often used when various integrated energy microgrids belong to different stakeholders and benefit allocation is required. For example, master-slave game theory is used to achieve optimal operation of multi-stakeholder integrated energy systems. Another example is cooperative game theory to achieve distributed optimization of integrated energy systems under uncertain environments. While these model methods can guarantee optimality of solutions to some extent, they often rely on accurate microgrid parameters to build mathematical models, making modeling difficult. Meanwhile, the model-based approach requires extensive iterative calculations to obtain an optimized operating strategy, resulting in poor real-time performance of decision optimization.

[0004] Data-driven methods, with reinforcement learning at their core, are model-free intelligent optimization methods. They do not require precise mathematical models and obtain optimal strategies through trial and error, making them suitable for microgrid optimization. The literature [Guo, CY. Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning[J]. Energy, 2022, 238: 121873] proposes a centralized PPO algorithm to achieve real-time microgrid energy management. However, for multi-microgrid systems, the centralized PPO algorithm suffers from high requirements for system communication and computing capabilities. Data-driven methods directly output decisions from the network, resulting in faster computation speeds, but require a large amount of actual or simulated operational data for training. Furthermore, the training effect of reinforcement learning is greatly affected by the training design, and compared to model-based methods, its final decision-making performance is more unstable. Summary of the Invention

[0005] To address the issue of poor stability in existing reinforcement learning applications for intelligent dispatching of microgrids, this invention proposes a real-time energy management optimization method based on load feedback correction. By introducing load feedback into the MAPPO algorithm, the median parameters of the algorithm are dynamically adjusted, solving the algorithm divergence problem caused by uncertainty. This improves the stability of the algorithm while achieving energy management optimization.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] A real-time energy management optimization method based on load feedback correction includes the following steps:

[0008] S1: Construct a microgrid structure model containing renewable energy, define the optimization objective and operating constraints of the microgrid, wherein the optimization objective is to minimize the system operating cost, and the operating constraints include upper and lower limits of equipment power and system power balance constraints;

[0009] S2: Construct a MAPPO energy management reinforcement learning model based on a multi-agent Markov decision process, configure independent agents for each power supply area of ​​the microgrid, and define the state set, action set and multi-dimensional reward function of each agent;

[0010] S3: Introduce load feedback to perform a two-level correction on the policy distribution mean parameter of the MAPPO algorithm, and use the MAPPO algorithm to update the policy based on the corrected policy distribution mean parameter;

[0011] S4: Determine if convergence has occurred. If convergence has occurred, output the microgrid real-time energy management optimization strategy. If convergence has not occurred, return to step S3 to continue iterative training.

[0012] Furthermore, the optimization objective function of the microgrid is:

[0013] ;

[0014] in, This represents the operating cost at time t. and These represent the number of generators and photovoltaic units, respectively. and Represent Time of the first The generator and the first The output power of each photovoltaic unit , and This represents the generator's power generation cost coefficient. Indicates the first The cost of generating electricity per unit output power of a photovoltaic system.

[0015] Furthermore, the state set includes load demand, generator start / stop status, and photovoltaic power generation; the action set includes generator power generation; and the multi-dimensional reward function integrates cost rewards, curtailment penalties, power balance penalties, and equipment over-limit penalties.

[0016] Furthermore, the calculation method for the multi-dimensional reward function is as follows:

[0017] Calculate cost incentives: ;

[0018] Calculate the light abandonment penalty: ;

[0019] Calculate the power balance penalty: ;

[0020] Penalties for exceeding computing device limits: ; ;

[0021] The overall reward function for all agents is obtained by combining the results: ;

[0022] in, This is the penalty coefficient for discarded light. for Discarding light at all times This is the power balance penalty factor. for Power imbalance at any moment This is the power over-limit penalty coefficient for the generator. This is the penalty factor for exceeding the power limit in photovoltaic power generation. Penalty for exceeding generator power limits. Penalties for exceeding power limits in photovoltaic systems.

[0023] Furthermore, load feedback is introduced to perform a two-level correction on the policy distribution mean parameter of the MAPPO algorithm, including:

[0024] The first layer calculates the load deviation from the ideal mean coefficient based on load feedback, and dynamically adjusts the global strategy distribution mean based on the net load deviation from the ideal mean coefficient.

[0025] The second layer allocates the net load to each generator set based on the equal incremental rate criterion, and distributes the adjusted global strategy distribution mean according to the ideal mean coefficient of each generator.

[0026] Furthermore, based on load feedback, the load deviation from the ideal mean coefficient is calculated, and the global strategy distribution mean is dynamically adjusted based on the net load deviation from the ideal mean coefficient, including:

[0027] ;

[0028] ;

[0029] in, For system net load, For the policy network The mean of the policy distribution corresponding to each output The net load deviation from the ideal mean coefficient. This represents the adjusted mean of the global policy distribution.

[0030] Furthermore, based on the equal incremental rate criterion, the net load is allocated to each generator unit, and the adjusted global strategy distribution mean is allocated according to the ideal mean coefficient of each generator, including:

[0031] Define the constant incremental rate equation and the load balance equation:

[0032] ; Indicates the first The operating cost of one generator;

[0033] ;

[0034] By simultaneously solving the constant incremental rate equation and the load balance equation, the optimal net load capacity of each generator can be obtained. ;

[0035] Calculate the ideal mean coefficient for each generator based on the ratio of the net load borne by each generator to the net load. ;

[0036] The adjusted global strategy distribution mean is allocated based on the ideal mean coefficient of each generator, resulting in the median of the strategy distribution for each generator set:

[0037] .

[0038] Furthermore, the policy is updated using the MAPPO algorithm based on the corrected policy distribution mean parameter, including:

[0039] S31. Initialize the agent and network parameters. The network parameters include policy network parameters and evaluation network parameters.

[0040] S32. The agent interacts with the environment according to the current strategy and collects experience data into the experience pool.

[0041] S33. Calculate the agent's advantage function based on empirical data and the value function of the evaluation network;

[0042] S34. Calculate the median of the strategy distribution for each generator set. The probability density function of the embedded policy network And calculate the new probability ratio ;

[0043] S35, Based on the new probability ratio Calculate the substitution function and update the parameters of the policy network using the substitution function based on the gradient descent algorithm;

[0044] S36. Calculate the loss function of the evaluation network based on the value function of the evaluation network, and update the parameters of the evaluation network using the loss function of the evaluation network based on the gradient descent algorithm.

[0045] Furthermore, the agent's advantage function is:

[0046] ;

[0047] in, Indicates the first The advantage function of an agent at time t. Indicates the state The following value estimate, This is the discount factor.

[0048] Furthermore, based on the new probability ratio The substitution function calculated is:

[0049] ;

[0050] in, It is the substitution function of the i-th agent at time t; For the clipping function, These are the cutting parameters. The beneficial effects of this invention are:

[0051] By introducing load feedback, the median parameters of the algorithm are corrected in real time, thereby improving the stability of the algorithm and solving the adverse effects of system uncertainty on the stability of MAPPO real-time optimization.

[0052] A two-layer guided median parameter adjustment mechanism is proposed to improve the performance of the MAPPO algorithm in economic dispatch of power systems. A real-time load feedback correction strategy is introduced to the median, ensuring the algorithm quickly tracks random fluctuations. The median parameter is further modified based on the equal incremental rate criterion, achieving a synergistic improvement in dispatch stability and economy. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is the overall flowchart of the present invention.

[0055] Figure 2 The flowchart illustrates the two-level correction of the strategy distribution mean parameter of the MAPPO algorithm by introducing load feedback in this invention.

[0056] Figure 3 This is a diagram of an island microgrid structure according to an embodiment of the present invention.

[0057] Figure 4 This is a graph showing the daily net load variation of the system of the present invention.

[0058] Figure 5 The graph shows the results of the algorithm running without load feedback.

[0059] Figure 6 The graph shows the results of the algorithm with load feedback according to the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] A real-time energy management optimization method based on load feedback correction, such as Figure 1As shown, the steps include:

[0062] S1: Construct a microgrid structure model containing renewable energy, define the optimization objective and operating constraints of the microgrid, the optimization objective is to minimize the system operating cost, and the operating constraints include upper and lower limits of equipment power and system power balance constraints.

[0063] In this embodiment of the application, the microgrid structure model containing renewable energy is an island microgrid, such as... Figure 3 As shown, it consists of three interconnected distributed power supply areas. Area 1 includes load L1, photovoltaic PV1 and diesel generator PG1; Area 2 includes diesel generator PG2, photovoltaic PV2 and load L2; Area 3 includes diesel generator PG3 and load L3. The areas are interconnected by lines to form a distributed power supply system that can operate collaboratively.

[0064] An energy management model based on MAPPO can be obtained based on the optimization objectives and operational constraints.

[0065] In this embodiment, the objective function is established with the goal of minimizing system operating cost:

[0066] ;

[0067] in, and These represent the number of generators and photovoltaic units, respectively. and Represent Time of the first The generator and the first The output power of a single photovoltaic unit, , and This represents the generator's power generation cost coefficient. Indicates the first The cost of generating electricity per unit output power of a photovoltaic system.

[0068] In this embodiment of the application, considering the actual operating limitations of the equipment and the power balance requirements of the microgrid, the upper and lower limits of equipment power and the system power balance constraints are as follows:

[0069] ;

[0070] ;

[0071] in, and The first The lower and upper limits of the generator's output power. and The first The lower and upper limits of photovoltaic output power. for Total load demand at any given time.

[0072] S2: Construct a MAPPO energy management reinforcement learning model based on a multi-agent Markov decision process, configure independent agents for each power supply area of ​​the microgrid, and define the state set, action set, and multi-dimensional reward function of each agent.

[0073] In this embodiment, a reinforcement learning model for MAPPO energy management is constructed based on a multi-agent Markov decision process (MAMDP), including: representing the optimization process of the MAPPO algorithm as tuples. ,in, The variable 'm' represents the number of intelligent agents, specifically referring to adjustable devices in the system; in this example, it represents generators. This section primarily explains the principles of MAMDP, hence the variable 'm' is used to refer to all adjustable devices within the system. Subsequent examples will use 'i', 'k', and 'j' to represent generators, loads, and photovoltaics, respectively. Let m represent the set of states of the m-th agent. express The state of agent m at time t is such that the joint state composed of the states of all agents has the following relationship: , Let represent the joint state space, obtained by the Cartesian product of the individual states of all agents, and represent the global state space of the entire system. express The joint state at a given moment is the global state formed by the individual states of all agents at that moment. Indicates the first A set of actions of an agent. express Time-based intelligent agent The action, for the combined action of all agent actions, has , Represents the joint action space, express The combined actions of moments; Let be the state transition probability, satisfying ; Let be the reward function, representing the state . The following actions are adopted. Instant feedback obtained from the system environment ,and Each agent takes action based on the observed state. It acts on the environment, causing the environmental state to change. Transfer to and get feedback This cycle repeats itself, and through continuous interaction with the environment, strategies are ultimately formed. , Let be the policy function, representing the probability distribution of taking the joint action 'a' given a joint state 's'.

[0074] In this embodiment of the application, based on the established MAPPO-based energy management model and multi-agent Markov process, a MAPPO-based reinforcement learning model for island microgrid energy management can be obtained, including a state set, an action set, and a multi-dimensional reward function.

[0075] The state set of the learning agent mainly corresponds to the observations in the optimization model (i.e., the MAPPO-based energy management model), including load demand, generator start-stop status, and photovoltaic power generation, expressed as:

[0076] ;

[0077] in, For the first Load demand in each region For the first The start / stop status of each generator. For the first The power generation of the photovoltaic system.

[0078] The action set of the learning agent mainly corresponds to the decision variables in the optimization model. In the energy management optimization of the island microgrid, the adjustable decision variable is the power generation of the generator, expressed as:

[0079]

[0080] in, Indicates the first The power generation of each generator.

[0081] The multi-dimensional reward function integrates cost rewards, curtailment penalties, power balancing penalties, and equipment exceeding limits penalties.

[0082] Regarding cost incentives:

[0083] The reward function mainly corresponds to the optimization objective and constraints in the optimization model. In the optimization problem of this invention, the main optimization objective is the system operating cost. Therefore, the cost-related reward function is defined as follows:

[0084] ;

[0085] in, for The cost reward at each time step, i.e., the cost reward function, is the difference in system operating costs between two consecutive time steps. When the operating cost decreases at a later time step, the reward is positive, and vice versa. This causes the reward value to fluctuate around 0, which is more conducive to algorithm convergence.

[0086] Regarding penalties for abandoning light:

[0087] Further considering the "dual-carbon" development orientation, penalties will be imposed on potential curtailment of solar power, as follows:

[0088] ;

[0089] in, for The punishment of abandoning light at all times This is the penalty coefficient for discarded light. for The amount of light discarded at any given time.

[0090] Regarding power balance penalties:

[0091] Considering the power balance constraints for the safe operation of the energy system, the following penalty is given based on the degree of power mismatch:

[0092] ;

[0093] ;

[0094] in, for Power imbalance penalty at any moment, This is the power balance penalty factor. for Power imbalance at any given moment.

[0095] Regarding penalties for exceeding device limits:

[0096] Considering the upper and lower power limits of each device, the safe operation penalties for each device are given. The over-limit power of the generator and photovoltaic system are defined as follows:

[0097] ;

[0098] ;

[0099] in, This refers to the generator's over-limit power. This refers to the over-limit power of photovoltaics.

[0100] Based on the above definitions, the power limit penalties for generators and photovoltaics are as follows:

[0101] ;

[0102] ;

[0103] in, Penalty for exceeding generator power limits. This is the power over-limit penalty coefficient for the generator. Penalties for exceeding power limits in photovoltaic systems. This is the power over-limit penalty factor for photovoltaic power.

[0104] In summary, the total reward function for each agent can be obtained as follows:

[0105] ;

[0106] in, The total reward function for each agent.

[0107] S3: Introduce load feedback to perform a two-level correction on the policy distribution mean parameter of the MAPPO algorithm, and use the MAPPO algorithm to update the policy based on the corrected policy distribution mean parameter.

[0108] The performance of policy gradient algorithms largely depends on the choice of policy distribution, which is a crucial method for addressing the exploration-exploitation balance problem. Therefore, the choice of policy distribution is critical, directly affecting the algorithm's convergence speed and the policy learned by the agent. For the policy network in the MAPPO algorithm, its policy distribution primarily adopts a normal distribution, whose probability density function can be expressed as:

[0109] ;

[0110] in, and Representing the policy distribution The mean and standard deviation of . The policy parameters and their corresponding eigenvectors are expressed as . and , The parameter vector of the mean network. Let be the parameter vector of the standard deviation network. For ease of calculation, , and Use the same set of feature vectors.

[0111] Policy Vector vector ,in:

[0112] Represents the log probability of the policy versus the mean parameter The gradient is used to guide the direction of parameter updates in the mean network;

[0113] The parameter representing the log probability versus standard deviation of the policy. The gradient is used to guide the direction of updating the standard deviation network parameters.

[0114] The relationship between policy distribution and energy management problems, and the problems therein: the mean policy distribution in the above policy network. This is closely related to the range of output variation of each device in the energy management problem. Assume the policy network is... One output ( The upper and lower limits for the number of output quantities are respectively and , then the first The relationship between the mean of the policy distribution corresponding to each output and the upper and lower bounds is as follows:

[0115] ;

[0116] Ideally, the output value of the energy management strategy should be near the mean. However, in actual operation, the stable output value of the system does not always fall near the mean, resulting in the less-than-ideal performance of the MAPPO algorithm. When the stable output value of the system deviates significantly from the mean, the convergence speed of the MAPPO algorithm decreases significantly, and sometimes it may even lead to non-convergence. This instability is very detrimental to the real-time optimization of the system.

[0117] Taking the island microgrid system as an example, the output of each agent's policy network is the power output value of the corresponding device. In an ideal optimization strategy, when the algorithm reaches stability, the photovoltaic power generation should be prioritized to meet load demand; therefore, net load is further defined. as follows:

[0118] ;

[0119] when At that time, the total output value of the generator should be equal to the net load of the system. .when When the output approaches the upper or lower limit of the generator's total output, the output of the policy network often deviates. .when or , , If the upper and lower boundary cutoff values ​​are used for the output of the policy network, the MAPPO algorithm will encounter convergence problems. In this energy management system, when photovoltaic power generation is insufficient, the total load of the system will approach the upper limit of the generator output, and this situation occurs frequently, which will seriously affect the optimal operation of the system.

[0120] Therefore, in this embodiment, load feedback is introduced to perform a two-layer correction on the strategy distribution mean parameter of the MAPPO algorithm.

[0121] When photovoltaic power generation and load fluctuate randomly, the median of the strategy distribution of the original MAPPO algorithm will deviate from the actual load demand, reducing the algorithm's convergence speed and increasing its convergence difficulty. Simultaneously, the resulting optimization will easily deviate from the globally optimal scheduling strategy, leading to poorer operational economy and scheduling power deviations that are detrimental to the safe and stable operation of the system. Therefore, it is necessary to dynamically adjust the strategy distribution of the MAPPO algorithm based on the actual load conditions to ensure the algorithm's stability and speed. Thus, a dynamic strategy distribution adjustment strategy based on generator capacity is proposed, as follows:

[0122] First, calculate the net load deviation from the ideal mean coefficient, and then dynamically adjust the global strategy distribution mean based on this coefficient.

[0123] ;

[0124] ;

[0125] in, For system net load, For the policy network The mean of the policy distribution corresponding to each output The net load deviation from the ideal mean coefficient. This represents the adjusted mean of the global policy distribution.

[0126] The aforementioned strategy for adjusting the mean based on generator capacity has been presented. However, for a multi-agent system containing multiple different generator sets, the regulation capabilities and operating cost curves of each generator are different. To further consider economic efficiency, an ideal mean coefficient allocation strategy based on equal incremental rates is proposed, as follows:

[0127] First, based on the equal incremental rate criterion, the net load is allocated to each generator set to obtain the net load borne by each generator, including:

[0128] Define the constant incremental rate equation and the load balance equation:

[0129] ;

[0130] ;

[0131] Let represent the operating cost of the i-th generator. The general form of generator cost is: .

[0132] By simultaneously solving the constant incremental rate equation and the load balance equation, the optimal net load capacity of each generator can be obtained. .

[0133] Furthermore, the ideal mean coefficient for each generator is calculated based on the ratio of the net load borne by each generator to the net load. .

[0134] Furthermore, the adjusted global strategy distribution mean is allocated according to the ideal mean coefficient of each generator, to obtain the median of the strategy distribution for each generator set:

[0135] .

[0136] In this embodiment, the MAPPO algorithm is used to update the policy based on the corrected policy distribution mean parameter. The MAPPO algorithm uses a policy neural network to approximate the optimized policy and a judgment network to evaluate the quality of the optimized policy. The policy network parameters and evaluation network parameters of each agent are respectively expressed as follows: and .like Figure 2 As shown, the policy update process includes:

[0137] First, initialize the agent and network parameters: this includes initializing policy network parameters, evaluation network parameters, action set, state set, experience pool, learning rate, etc.

[0138] Furthermore, the agent interacts with the environment based on the current policy, collecting experience data (i.e., traces) and storing them in an experience pool. This data can be used multiple times to update the policy network, thereby improving data utilization. Experience data is represented as:

[0139] ;

[0140] in, The first time at time t is respectively The state, actions, and rewards of each agent. During training, the policy network parameters and the evaluator network parameters need to be updated at every step; however, this update method makes the trace... It is extremely difficult to obtain. To solve this problem, an additional parameter was set. The neural network is updated once per round to store the trace.

[0141] Further, status The value function of the lower evaluation network is defined as follows:

[0142] ;

[0143] in, Discount factor, value function Representing state The expected value of the cumulative return can be obtained below. for The instantaneous rewards obtained by the agent at any given moment.

[0144] Based on the value function, we can obtain the first... The advantage function of each agent is as follows:

[0145] ;

[0146] in, Representing state Take action below The advantage function, As the discount factor, the first term of the advantage function reflects the discount factor. Take action in a state The expected value of the actual cumulative reward obtained, the second term reflects all the actions collected in this state. The expected value of the average cumulative return. By subtracting these two values, the result can be used to measure the quality of the current action relative to historical actions.

[0147] The advantage function can be approximated as:

[0148] ;

[0149] in, Indicates the first The advantage function of an agent at time t. Indicates the state The value estimate below.

[0150] Furthermore, the probability ratio is defined as follows:

[0151] ;

[0152] in, Indicating a new strategy Under these conditions, the i-th agent is in state Take action at the time The probability, This indicates the old strategy.

[0153] The median of the policy distribution of each generator unit is embedded into the probability density function of the policy network. Calculate the new probability ratio.

[0154] Furthermore, in updating the policy network, it is necessary to limit the difference between the new and old policies and avoid abrupt changes in the optimization policy. The following substitution function is used to update the network parameters:

[0155]

[0156]

[0157] in, It is the substitution function of the i-th agent at time t; The probability ratio reflects the similarity between the old and new strategies; For the clipping function, It is the clipping parameter, used to control the range of values ​​for the probability ratio.

[0158] Furthermore, the parameter update strategy for the policy network is as follows:

[0159] ;

[0160] in, The learning rate of the action network. This indicates that the gradient of the policy network parameters is calculated.

[0161] Furthermore, the evaluation network also employs gradient descent, with the following rules:

[0162] ;

[0163] ;

[0164] in, Let be the loss function of the evaluation network for the i-th agent. As a discount factor, To evaluate the learning rate of a network, This indicates that the gradient of the evaluation network is calculated.

[0165] S4: Determine if convergence has occurred. If convergence has occurred, output the microgrid real-time energy management optimization strategy. If convergence has not occurred, return to step S3 to continue iterative training.

[0166] The convergence condition is: the difference between the output values ​​of two consecutive rounds is less than the set threshold.

[0167] Simulation analysis:

[0168] To verify the improvement in algorithm stability brought about by introducing load feedback, this paper analyzes the algorithm's stability through simulation under different load conditions throughout the day. The changes in the system's net load throughout the day are shown below. Figure 4 As shown in the figure, the net load ranges from 95 to 274 kW. To more easily analyze the impact of load fluctuations on algorithm stability, Benfa analyzed the impact of a 10 kW increase in net load within the range of 90-280 kW. Each change was tested 20 times repeatedly, and the MAPPO optimization results are shown below. Figure 5 , 6 As shown.

[0169] When no load feedback is introduced, the algorithm's operational statistics are as follows: Figure 5 As shown in the figure, without load feedback, the algorithm's convergence ratio drops significantly when the load approaches the generator's upper and lower output limits. In the power range of [100, 140], the worse the algorithm's stability, the more rounds required for convergence, meaning lower optimization efficiency. At power levels of 90kW and in the range of [250, 280], although the algorithm's convergence ratio doesn't necessarily increase with decreasing stability, the power deviation graph shows that at these points or ranges, the power deviation caused by the algorithm's strategy is large during convergence. This is insufficient to meet the safety requirements of microgrid optimization; therefore, these strategies are essentially unsuccessful.

[0170] After introducing load feedback, the algorithm's operational statistics are as follows: Figure 6 As shown, by introducing load feedback, the algorithm can guarantee convergence and enhance its stability. At the same time, the average number of convergence rounds is generally reduced, mostly stabilizing within 50 rounds. Regarding the power deviation of the strategy, after introducing load feedback, the average power deviation of the results is small, not exceeding 2kW, and can be completely eliminated through subsequent fine-tuning.

[0171] This invention addresses the real-time energy management problem of island microgrid systems with a high proportion of renewable energy penetration, researching and improving the MAPPO-based optimization method. To address the issue of oscillations and policy divergence during algorithm training caused by the uncertainty of renewable energy, a real-time MAPPO optimization algorithm based on load feedback correction is proposed. This algorithm constructs a two-layer parameter adjustment mechanism by introducing real-time load feedback signals: the first layer rapidly adjusts the median policy based on load feedback, enhancing the algorithm's dynamic tracking ability against uncertainty; the second layer further optimizes the median parameters based on the constant incremental rate criterion, achieving a synergistic improvement in operational economy and stability. Simulation results show that the proposed method significantly improves the stability and convergence speed of optimization, effectively solving the policy divergence problem caused by uncertainty.

[0172] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time energy management optimization method based on load feedback correction, characterized in that, Including the following steps: S1: Construct a microgrid structure model containing renewable energy, define the optimization objective and operating constraints of the microgrid, wherein the optimization objective is to minimize the system operating cost, and the operating constraints include upper and lower limits of equipment power and system power balance constraints; S2: Construct a MAPPO energy management reinforcement learning model based on a multi-agent Markov decision process, configure independent agents for each power supply area of ​​the microgrid, and define the state set, action set and multi-dimensional reward function of each agent; S3: Introduce load feedback to perform a two-level correction on the policy distribution mean parameter of the MAPPO algorithm, and use the MAPPO algorithm to update the policy based on the corrected policy distribution mean parameter; S4: Determine if convergence has occurred. If convergence has occurred, output the microgrid real-time energy management optimization strategy. If convergence has not occurred, return to step S3 to continue iterative training.

2. The real-time energy management optimization method based on load feedback correction according to claim 1, characterized in that, The optimization objective function of the microgrid is: ； in, This represents the operating cost at time t. and These represent the number of generators and photovoltaic units, respectively. and Represent Time of the first The generator and the first The output power of each photovoltaic unit , and This represents the generator's power generation cost coefficient. Indicates the first The cost of generating electricity per unit output power of a photovoltaic system.

3. The real-time energy management optimization method based on load feedback correction according to claim 2, characterized in that, The state set includes load demand, generator start / stop status, and photovoltaic power generation; the action set includes generator power generation; and the multi-dimensional reward function integrates cost rewards, curtailment penalties, power balance penalties, and equipment over-limit penalties.

4. The real-time energy management optimization method based on load feedback correction according to any one of claims 1-3, characterized in that, The calculation method for the multi-dimensional reward function is as follows: Calculate cost incentives: ; Calculate the light abandonment penalty: ; Calculate the power balance penalty: ; Penalties for exceeding computing device limits: ; ; The overall reward function for all agents is obtained by combining the results: ; in, This is the penalty coefficient for discarded light. for Discarding light at all times This is the power balance penalty factor. for Power imbalance at any moment This is the power over-limit penalty coefficient for the generator. This is the penalty factor for exceeding the power limit in photovoltaic power generation. Penalty for exceeding generator power limits. Penalties for exceeding power limits in photovoltaic systems.

5. The real-time energy management optimization method based on load feedback correction according to claim 4, characterized in that, A two-level correction is performed on the policy distribution mean parameter of the MAPPO algorithm by introducing load feedback, including: The first layer calculates the load deviation from the ideal mean coefficient based on load feedback, and dynamically adjusts the global strategy distribution mean based on the net load deviation from the ideal mean coefficient. The second layer allocates the net load to each generator set based on the equal incremental rate criterion, and distributes the adjusted global strategy distribution mean according to the ideal mean coefficient of each generator.

6. The real-time energy management optimization method based on load feedback correction according to claim 5, characterized in that, The load deviation from the ideal mean coefficient is calculated based on load feedback, and the global strategy distribution mean is dynamically adjusted based on the net load deviation from the ideal mean coefficient, including: ； ； in, For system net load, For the policy network The mean of the policy distribution corresponding to each output The net load deviation from the ideal mean coefficient. This represents the adjusted mean of the global policy distribution.

7. The real-time energy management optimization method based on load feedback correction according to claim 6, characterized in that, The net load is allocated to each generator set based on the equal incremental rate criterion, and the adjusted global strategy distribution mean is allocated according to the ideal mean coefficient of each generator, including: Define the constant incremental rate equation and the load balance equation: ; Indicates the first The operating cost of one generator; ； By simultaneously solving the constant incremental rate equation and the load balance equation, the optimal net load capacity of each generator can be obtained. ; Calculate the ideal mean coefficient for each generator based on the ratio of the net load borne by each generator to the net load. ; The adjusted global strategy distribution mean is allocated based on the ideal mean coefficient of each generator, resulting in the median of the strategy distribution for each generator set: 。 8. The real-time energy management optimization method based on load feedback correction according to claim 7, characterized in that, Policy updates are performed using the MAPPO algorithm based on the corrected policy distribution mean parameter, including: S31. Initialize the agent and network parameters. The network parameters include policy network parameters and evaluation network parameters. S32. The agent interacts with the environment according to the current strategy and collects experience data into the experience pool. S33. Calculate the agent's advantage function based on empirical data and the value function of the evaluation network; S34. Calculate the median of the strategy distribution for each generator set. The probability density function of the embedded policy network And calculate the new probability ratio ; S35, Based on the new probability ratio Calculate the substitution function and update the parameters of the policy network using the substitution function based on the gradient descent algorithm; S36. Calculate the loss function of the evaluation network based on the value function of the evaluation network, and update the parameters of the evaluation network using the loss function of the evaluation network based on the gradient descent algorithm.

9. The real-time energy management optimization method based on load feedback correction according to claim 8, characterized in that, The agent's advantage function is: ； in, Indicates the first The advantage function of an agent at time t. Indicates the state The following value estimate, This is the discount factor.

10. The real-time energy management optimization method based on load feedback correction according to claim 9, characterized in that, Based on the new probability ratio The substitution function for calculation is: ； in, It is the substitution function of the i-th agent at time t; For the clipping function, These are the trimming parameters.

Images