Multi-mesh system hierarchical reinforcement learning optimization method and system and storage medium

Abstract

The application discloses a multi-microgrid system hierarchical reinforcement learning optimization method and system and a storage medium. The optimization method comprises the following steps: decomposing a multi-microgrid optimization problem into an upper layer and a lower layer; constructing an upper layer intelligent agent model for providing energy storage strategies in each microgrid and power interaction strategies between the microgrids; constructing a lower layer autonomous optimization model for realizing autonomous optimization of equipment output in each microgrid in the lower layer based on the energy storage strategies and the power interaction strategies in the upper layer, and feeding back an optimization target value to the upper layer to guide strategy updating of the upper layer; cooperatively training the lower layer autonomous optimization model and the upper layer intelligent agent model; and using the trained lower layer autonomous optimization model and the upper layer intelligent agent model to instantly give a scheduling strategy according to a system state. The application combines a deep reinforcement learning method with a traditional mathematical programming method, fully utilizes the advantages of strong self-adaptability of the reinforcement learning method and high solution accuracy of the mathematical programming method, and can realize efficient solution of the multi-microgrid optimization problem, and the convergence speed and the accuracy are greatly improved.

Description

Technical Field

[0001] This invention belongs to the field of multi-micronet system optimization and scheduling technology, specifically involving a hierarchical reinforcement learning optimization method, system, and storage medium for multi-micronet systems. Background Technology

[0002] Microgrids, as an effective technology for absorbing renewable energy, have become a focus of attention both domestically and internationally. With the rapid development of microgrids, several microgrids exist within the same region, interconnecting to form multi-microgrid systems. Through the complementary integration of different energy forms within microgrids and the coordinated and optimized scheduling between microgrids, the power supply reliability of the distribution network and the renewable energy absorption rate can be effectively improved. Therefore, research on the optimization of multi-microgrid systems is of great significance for achieving multi-energy complementarity and cascaded energy utilization.

[0003] Currently, numerous scholars have conducted extensive research on the optimization problem of multi-micronet systems. The main solution methods include centralized optimization methods such as mathematical programming, genetic algorithms, and particle swarm optimization; distributed optimization methods such as the alternating direction multiplier method and the target cascade method; and recently, many studies have also employed reinforcement learning methods. The first two methods rely on accurate prediction and refined modeling of source loads, resulting in excessively long solution times and difficulty in meeting real-time requirements. Reinforcement learning, based on data-driven approaches, can provide results instantly after training. However, for multi-micronet systems with numerous variables and a complex action space, constraints are modeled by adding penalty terms to the reward function, leading to difficulties in training convergence, insufficient accuracy, and difficulty in ensuring that the results satisfy all constraints, thus impacting the safe operation of the system.

[0004] Overall, there is currently a lack of an accurate and efficient solution method for the optimization problem of multi-microgrid systems. Summary of the Invention

[0005] The purpose of this invention is to address the problems in the prior art by providing a hierarchical reinforcement learning optimization method, system, and storage medium for multi-micronet systems, which meets the need for accurate and efficient solutions in scheduling high-dimensional nonlinear multi-micronet systems.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] Firstly, a hierarchical reinforcement learning optimization method for multi-micronet systems is provided, including:

[0008] The multi-micronet optimization problem is decomposed into two layers: upper and lower.

[0009] Construct an upper-layer intelligent agent model that provides energy storage strategies within each microgrid and power interaction strategies between microgrids;

[0010] Construct a lower-layer autonomous optimization model that enables each microgrid to autonomously optimize the power output of devices within the microgrid based on the upper-layer energy storage strategy and power interaction strategy, and feeds back the optimization target value to the upper layer to guide the upper-layer strategy update;

[0011] Collaborative training of the lower-level autonomous optimization model and the upper-level intelligent agent model;

[0012] The lower-level autonomous optimization model and the upper-level intelligent agent model, which have been trained, provide scheduling strategies in real time based on the system state.

[0013] As a preferred embodiment of the hierarchical reinforcement learning optimization method for multi-microgrid systems of the present invention, the step of constructing an upper-layer agent model that provides energy storage strategies within each microgrid and power interaction strategies between microgrids includes:

[0014] Construct the state space and action space of the intelligent agent;

[0015] Construct reward and punishment functions for the intelligent agent;

[0016] Construct an intelligent agent network.

[0017] As a preferred embodiment of the hierarchical reinforcement learning optimization method for multi-microgrid systems of the present invention, the steps of constructing the agent state space and action space are as follows: the constructed agent state space includes the predicted renewable energy output, energy storage state of charge, time-of-use electricity price information, and cold, heat, and electrical load power of each microgrid; the constructed agent action space consists of relevant control variables, including the interaction power between each microgrid and the energy storage charging and discharging power of each microgrid.

[0018] As a preferred embodiment of the hierarchical reinforcement learning optimization method for multi-micronet systems of the present invention, the step of constructing the agent reward and punishment function results in the following expression for the agent reward and punishment function:

[0019]

[0020] In the formula, π * The current optimization strategy; γ t s is the discount factor for time rewards; r is the reward function; s t The state at time t; a t The action at time t; α is the temperature coefficient, α > 0, used to determine the weight of the policy entropy term relative to the reward; H is the action at state s. t The entropy of the action taken by the next strategy is calculated as follows:

[0021]

[0022] The reward function r includes two main categories: operating cost and penalty for exceeding action constraints.

[0023] The formula for calculating the operating cost is:

[0024]

[0025] In the formula, f MG,t The overall operating cost of the multi-microgrid system during time period T;

[0026] The penalty for exceeding the action limit applies to the following operational constraints for energy storage: state of charge constraint, charge / discharge power constraint, and energy balance constraint at the beginning and end of the scheduling cycle, expressed as follows:

[0027] SOC min ≤SOC(t)≤SOC max

[0028] 0≤P ch (t)≤λ ch,t P ch,max

[0029] 0≤P dis (t)≤(1-λ ch,t )P dis,max

[0030] λ ch,t ∈{0,1}

[0031] SOC(0)=SOC(T)

[0032] In the formula, SOC min SOC max These represent the upper and lower limits of the battery's state of charge; P ch,max P dis,max These represent the maximum charge and discharge power of the battery; λ ch,t This represents the state of charge / discharge coefficient of the battery during time period t;

[0033] The power interaction between each microgrid satisfies upper and lower limit constraints, expressed as follows:

[0034] P ij,mg,min ≤P ij,mg (t)≤P ij,mg,max

[0035] In the formula, P ij,mg,min P ij,mg,max These are the upper and lower limits of the interaction power between microgrid i and microgrid j, respectively.

[0036] The output action is constrained to the range [-1, 1] by the tanh function, and then the charge state constraint is satisfied by the linear transformation.

[0037] Add the following penalty terms to ensure that the charging and discharging power constraints and the energy balance constraints at the beginning and end of the scheduling cycle are met:

[0038]

[0039] φ2=k2|SOC(0)-SOC(T)|

[0040] In the formula, k1 and k2 are penalty coefficients;

[0041] The reward function for the final agent is as follows:

[0042] R t =-(f+φ1+φ2).

[0043] As a preferred embodiment of the hierarchical reinforcement learning optimization method for multi-micronetwork systems of the present invention, the step of constructing the agent network employs alternating optimization using a Q-function network and a policy distribution network. The network parameters of the Q-function network are obtained by training to minimize the Bellman residual, and the calculation expression is as follows:

[0044]

[0045] In the formula, θ represents the parameters of the Q-function network to be solved. The parameters of the target Q-function network are expressed as follows:

[0046]

[0047] In the formula, τ is the soft update coefficient;

[0048] The two Q-function networks are trained independently, and the smaller value is selected as the parameter value of the target Q-function network during the training process;

[0049] The parameters φ of the policy distribution network are updated by minimizing its KL divergence, as expressed below:

[0050]

[0051] In the formula, the action is reparameterized to reduce the variance of the gradient estimate, as shown in the following expression:

[0052] a t =f φ (ε t ,s t )=f φ μ (s t )+ε t ⊙f φ σ (s t )

[0053] μ and σ are the mean and variance of the policy distribution network output, respectively, and ε t This is noise, sampled from a standard normal distribution;

[0054] The temperature coefficient α in the agent's reward and punishment function is adaptively updated by minimizing the following loss function:

[0055]

[0056] In the formula, κ is a hyperparameter representing the minimum expected entropy.

[0057] As a preferred embodiment of the hierarchical reinforcement learning optimization method for multi-microgrid systems of the present invention, the step of constructing a lower-layer autonomous optimization model that enables each lower-layer microgrid to autonomously optimize the power output of devices within the microgrid based on the upper-layer energy storage strategy and power interaction strategy, and to feed back the optimization target value to the upper layer to guide the upper-layer strategy update, includes:

[0058] To minimize the operating cost of a multi-microgrid system, an objective function is constructed.

[0059] Establish constraints for the autonomous optimization model, including energy balance constraints and operational constraints.

[0060] As a preferred embodiment of the hierarchical reinforcement learning optimization method for multi-micronet systems of the present invention, the step of constructing the objective function with the goal of minimizing the operating cost of the multi-micronet system includes:

[0061] The expression for the objective function is:

[0062] minf MG,i =C i,fuel (t)+C i,mg (t)+C i,grid (t)

[0063] In the formula, C i,fuel (t), C i,mg (t), C i,grid (t) represents the gas cost, adjacent microgrid interaction cost, and distribution network interaction cost of microgrid i during time period t, respectively. The calculation expressions for each cost are as follows:

[0064]

[0065]

[0066] C i,grid (t)=c grid (t)P i,grid (t)

[0067] In the formula, c gas For natural gas prices; P i,MT (t) represents the MT output power of microgrid i during time period t; c mg (t), cgrid (t) represents the electricity trading price between adjacent microgrids and the electricity trading price between a microgrid and the distribution network, respectively, during time period t; P ij,mg (t), P i,grid (t) represents the interaction power between microgrid i and microgrid j, and between microgrid i and the distribution network, respectively, during time period t.

[0068] As a preferred embodiment of the hierarchical reinforcement learning optimization method for multi-microgrid systems of the present invention, the energy balance constraints when establishing the constraints of the autonomous optimization model include cold, heat, and electrical balance constraints within the multi-microgrid system. The expressions for the cold, heat, and electrical balance constraints within the multi-microgrid system are as follows:

[0069] Q i,MT,c (t)+Q i,EC (t)=Q i,load,c (t)

[0070] Q i,MT,h (t)+Q i,GB (t)=Q i,load,h (t)

[0071]

[0072] The operational constraints include equipment operational constraints and power interaction constraints;

[0073] The equipment operation constraints include:

[0074] The operating constraints of the micro gas turbine are expressed as follows:

[0075] P MT,min ≤P MT (t)≤P MT,max

[0076] In the formula, P MT,min P MT,max These represent the upper and lower limits of the gas turbine output, respectively.

[0077] The operating constraints for gas-fired boilers are expressed as follows:

[0078] Q GB,min ≤Q GB (t)≤Q GB,max

[0079] In the formula, Q GB,min Q GB,max These are the upper and lower limits of the output thermal power of the gas-fired boiler, respectively.

[0080] The operating constraints of the electric chiller are expressed as follows:

[0081] P EC,min ≤P EC(t)≤P EC,max

[0082] In the formula, P EC,min P EC,max These are the upper and lower limits of the input electrical power of the electric chiller, respectively.

[0083] The expression for the power interaction constraint is as follows:

[0084] P i,grid,min ≤P i,grid (t)≤P i,grid,max

[0085] In the formula, P i,grid,min P i,grid,max These are the upper and lower limits of the interaction power between the microgrid i and the distribution network, respectively.

[0086] As a preferred embodiment of the hierarchical reinforcement learning optimization method for multi-microgrid systems of the present invention, the steps of collaboratively training the lower-level autonomous optimization model and the upper-level agent model include initializing model parameters; initializing the state of the multi-microgrid system from the state space, providing scheduling actions for the current time segment based on the current state, the scheduling actions including energy storage strategies within each microgrid and power interaction strategies between microgrids, and distributing them to the multi-microgrid system; each lower-level sub-microgrid system receives the energy storage strategies and power interaction strategies from the upper layer, and simultaneously performs autonomous optimization based on its own state information to solve for the optimal output strategy of the devices within the network at the current time segment, updates the system state, and feeds back the objective function value to the upper-level agent; subsequently, storing empirical data samples containing the state at time segment t, the action at time segment t, the reward function at time segment t, and the state at time segment t+1 into the experience replay unit, batch extracting empirical data samples from the experience replay unit, and obtaining the strategy actions at time segment t+1; finally, updating the network parameters; repeating the above process until model training is complete.

[0087] Secondly, a hierarchical reinforcement learning optimization system for multi-micronet systems is provided, including:

[0088] The optimization problem decomposition module is used to decompose the multi-micronet optimization problem into upper and lower layers;

[0089] The upper-layer intelligent agent model construction module is used to construct upper-layer intelligent agent models that provide energy storage strategies within each microgrid and power interaction strategies between microgrids.

[0090] The lower-level autonomous optimization model construction module is used to build a lower-level autonomous optimization model that enables each microgrid in the lower layer to autonomously optimize the output of devices within the microgrid based on the upper-level energy storage strategy and power interaction strategy, and to feed back the optimization target value to the upper layer to guide the upper-level strategy update.

[0091] The collaborative training module is used to collaboratively train the lower-level autonomous optimization model and the upper-level agent model.

[0092] The scheduling strategy module is used to provide a scheduling strategy in real time based on the system state using the trained lower-level autonomous optimization model and the upper-level intelligent agent model.

[0093] As a preferred embodiment of the hierarchical reinforcement learning optimization system for multi-microgrid systems of the present invention, the upper-layer agent model construction module constructs an upper-layer agent model that provides energy storage strategies within each microgrid and power interaction strategies between microgrids, including constructing agent state space and action space, constructing agent reward and punishment functions, and constructing agent network.

[0094] As a preferred embodiment of the hierarchical reinforcement learning optimization system for multi-microgrid systems of the present invention, when the upper-layer intelligent agent model construction module constructs the intelligent agent state space and action space, the constructed intelligent agent state space includes the renewable energy predicted output, energy storage state of charge, time-of-use electricity price information, and cold, heat, and electrical load power of each microgrid; the constructed intelligent agent action space consists of relevant control variables, including the interaction power between each microgrid and the energy storage charging and discharging power of each microgrid.

[0095] As a preferred embodiment of the hierarchical reinforcement learning optimization system for the multi-micronet system of the present invention, the expression of the agent reward and punishment function obtained by the upper-layer agent model construction module when constructing the agent reward and punishment function is as follows:

[0096]

[0097] In the formula, π * The current optimization strategy; γ t s is the discount factor for time rewards; r is the reward function; s t The state at time t; a t The action at time t; α is the temperature coefficient, α > 0, used to determine the weight of the policy entropy term relative to the reward; H is the action at state s. t The entropy of the action taken by the next strategy is calculated as follows:

[0098]

[0099] The reward function r includes two main categories: operating cost and penalty for exceeding action constraints.

[0100] The formula for calculating the operating cost is:

[0101]

[0102] In the formula, f MG,t The overall operating cost of the multi-microgrid system during time period T;

[0103] The penalty for exceeding the action limit applies to the following operational constraints for energy storage: state of charge constraint, charge / discharge power constraint, and energy balance constraint at the beginning and end of the scheduling cycle, expressed as follows:

[0104] SOC min ≤SOC(t)≤SOC max

[0105] 0≤P ch (t)≤λ ch,t P ch,max

[0106] 0≤P dis (t)≤(1-λ ch,t )P dis,max

[0107] λ ch,t ∈{0,1}

[0108] SOC(0)=SOC(T)

[0109] In the formula, SOC min SOC max These represent the upper and lower limits of the battery's state of charge; P ch,max P dis,max These represent the maximum charge and discharge power of the battery; λ ch,t This represents the state of charge / discharge coefficient of the battery during time period t;

[0110] The power interaction between each microgrid satisfies upper and lower limit constraints, expressed as follows:

[0111] P ij,mg,min ≤P ij,mg (t)≤P ij,mg,max

[0112] In the formula, P ij,mg,min P ij,mg,max These are the upper and lower limits of the interaction power between microgrid i and microgrid j, respectively.

[0113] The output action is constrained to the range [-1, 1] by the tanh function, and then the charge state constraint is satisfied by the linear transformation.

[0114] Add the following penalty terms to ensure that the charging and discharging power constraints and the energy balance constraints at the beginning and end of the scheduling cycle are met:

[0115]

[0116] φ2=k2|SOC(0)-SOC(T)|

[0117] In the formula, k1 and k2 are penalty coefficients;

[0118] The reward function for the final agent is as follows:

[0119] R t =-(f+φ1+φ2).

[0120] As a preferred embodiment of the hierarchical reinforcement learning optimization system for the multi-micronetwork system of the present invention, when the upper-layer agent model construction module constructs the agent network, it uses a Q-function network and a policy distribution network for alternating optimization. The network parameters of the Q-function network are obtained by training through minimizing the Bellman residual, and the calculation expression is as follows:

[0121]

[0122] In the formula, θ represents the parameters of the Q-function network to be solved. The parameters of the target Q-function network are expressed as follows:

[0123]

[0124] In the formula, τ is the soft update coefficient;

[0125] The two Q-function networks are trained independently, and the smaller value is selected as the parameter value of the target Q-function network during the training process;

[0126] The parameters φ of the policy distribution network are updated by minimizing its KL divergence, as expressed below:

[0127]

[0128] In the formula, the action is reparameterized to reduce the variance of the gradient estimate, as shown in the following expression:

[0129] a t =f φ (ε t ,s t )=f φ μ (s t )+ε t ⊙f φ σ (s t )

[0130] μ and σ are the mean and variance of the policy distribution network output, respectively, and ε t This is noise, sampled from a standard normal distribution;

[0131] The temperature coefficient α in the agent's reward and punishment function is adaptively updated by minimizing the following loss function:

[0132]

[0133] In the formula, κ is a hyperparameter representing the minimum expected entropy.

[0134] As a preferred embodiment of the hierarchical reinforcement learning optimization system for multi-micronet systems of the present invention, the lower-level autonomous optimization model construction module aims to minimize the operating cost of the multi-micronet system and constructs an objective function; at the same time, it establishes constraints on the autonomous optimization model, including energy balance constraints and operational constraints.

[0135] As a preferred embodiment of the hierarchical reinforcement learning optimization system for multi-micronet systems of the present invention, the lower-level autonomous optimization model construction module aims to minimize the operating cost of the multi-micronet system. When constructing the objective function, the expression of the objective function is:

[0136] minf MG,i =C i,fuel (t)+C i,mg (t)+C i,grid (t)

[0137] In the formula, C i,fuel (t), C i,mg (t), C i,grid (t) represents the gas cost, adjacent microgrid interaction cost, and distribution network interaction cost of microgrid i during time period t, respectively. The calculation expressions for each cost are as follows:

[0138]

[0139]

[0140] C i,grid (t)=c grid (t)P i,grid (t)

[0141] In the formula, c gas For natural gas prices; P i,MT (t) represents the MT output power of microgrid i during time period t; c mg (t), c grid (t) represents the electricity trading price between adjacent microgrids and the electricity trading price between a microgrid and the distribution network, respectively, during time period t; P ij,mg (t), P i,grid (t) represents the interaction power between microgrid i and microgrid j, and between microgrid i and the distribution network, respectively, during time period t.

[0142] As a preferred embodiment of the hierarchical reinforcement learning optimization system for multi-microgrid systems of the present invention, when the lower-level autonomous optimization model construction module establishes the constraints of the autonomous optimization model, the energy balance constraints include cold, heat, and electrical balance constraints within the multi-microgrid system. The expressions for the cold, heat, and electrical balance constraints within the multi-microgrid system are as follows:

[0143] Q i,MT,c (t)+Q i,EC (t)=Q i,load,c (t)

[0144] Q i,MT,h (t)+Q i,GB (t)=Q i,load,h (t)

[0145]

[0146] The operational constraints include equipment operational constraints and power interaction constraints;

[0147] The equipment operation constraints include:

[0148] The operating constraints of the micro gas turbine are expressed as follows:

[0149] P MT,min ≤P MT (t)≤P MT,max

[0150] In the formula, P MT,min P MT,max These represent the upper and lower limits of the gas turbine output, respectively.

[0151] The operating constraints for gas-fired boilers are expressed as follows:

[0152] Q GB,min ≤Q GB (t)≤Q GB,max

[0153] In the formula, Q GB,min Q GB,max These are the upper and lower limits of the output thermal power of the gas-fired boiler, respectively.

[0154] The operating constraints of the electric chiller are expressed as follows:

[0155] P EC,min ≤P EC (t)≤P EC,max

[0156] In the formula, P EC,min P EC,max These are the upper and lower limits of the input electrical power of the electric chiller, respectively.

[0157] The expression for the power interaction constraint is as follows:

[0158] P i,grid,min ≤P i,grid (t)≤P i,grid,max

[0159] In the formula, P i,grid,min Pi,grid,max These are the upper and lower limits of the interaction power between the microgrid i and the distribution network, respectively.

[0160] As a preferred embodiment of the hierarchical reinforcement learning optimization system for multi-microgrid systems of the present invention, the steps of the collaborative training module in collaboratively training the lower-level autonomous optimization model and the upper-level agent model include initializing model parameters; initializing the state of the multi-microgrid system from the state space, providing scheduling actions for the current time segment based on the current state, the scheduling actions including energy storage strategies within each microgrid and power interaction strategies between microgrids, and distributing them to the multi-microgrid system; each lower-level sub-microgrid system receives the energy storage strategies and power interaction strategies from the upper layer, and performs autonomous optimization based on its own state information to solve for the optimal output strategy of the devices within the network at the current time segment, updates the system state, and feeds back the objective function value to the upper-level agent; subsequently, storing the empirical data samples containing the state at time segment t, the action at time segment t, the reward function at time segment t, and the state at time segment t+1 to the empirical replay unit, extracting empirical data samples in batches from the empirical replay unit, and obtaining the strategy actions at time segment t+1; finally, updating the network parameters; repeating the above process until model training is complete.

[0161] Thirdly, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the hierarchical reinforcement learning optimization method for multi-micronet systems.

[0162] Compared with the prior art, the first aspect of the present invention has at least the following beneficial effects:

[0163] This invention decomposes the multi-micronetwork optimization problem into two layers for solution, achieving temporal and spatial decoupling. The upper-layer model is responsible for providing energy storage strategies within each micronetwork and power interaction strategies between micronetworks; the lower-layer micronetworks autonomously optimize the power output of devices within their respective micronetworks based on the upper-layer strategies, while simultaneously feeding back optimization target values ​​to the upper-layer model to guide strategy updates. This collaborative approach between the upper and lower layers achieves global optimization of the multi-micronetwork system. To fully leverage the adaptive source-load randomness of reinforcement learning and the high accuracy of mathematical programming, the upper-layer strategy is formulated by a reinforcement learning agent, while the lower-layer strategy is provided by a mathematical programming solver. This invention combines deep reinforcement learning with traditional mathematical programming, fully utilizing the strong adaptability of reinforcement learning and the high accuracy of mathematical programming. It simplifies the complexity of the reinforcement learning action space and reward design, effectively maintaining the high accuracy of mathematical programming while fully leveraging the rapid solution capability of reinforcement learning. This enables efficient solutions to multi-micronetwork optimization problems, significantly improving both convergence speed and accuracy compared to traditional reinforcement learning methods. The hierarchical reinforcement learning optimization method for multi-micronet systems of this invention allows each micronet to solve independently and in parallel based on its own state information. Then, the overall optimization of the multi-micronet system is achieved by utilizing the collaboration between the upper and lower layers, overcoming the data barrier problem between micronets. The method proposed in this invention has good robustness, can adapt to the random fluctuations of source load, and provide scheduling strategies in real time.

[0164] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0165] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0166] Figure 1 A schematic diagram of the multi-microgrid system structure according to an embodiment of the present invention;

[0167] Figure 2 A schematic diagram of the hierarchical reinforcement learning optimization framework for a multi-microgrid system according to an embodiment of the present invention;

[0168] Figure 3 Flowchart of the hierarchical reinforcement learning optimization method for multi-microgrid systems according to an embodiment of the present invention;

[0169] Figure 4 The structural block diagram of the hierarchical reinforcement learning optimization system for a multi-microgrid system according to an embodiment of the present invention. Detailed Implementation

[0170] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0171] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0172] For the high-dimensional nonlinearity and other characteristics of the optimization scheduling problem in multi-micronet systems, existing solution methods are insufficient to meet the requirements of efficient and accurate solutions. This application proposes a hierarchical reinforcement learning optimization method for multi-micronet systems, which mainly achieves:

[0173] 1) Avoid the detailed modeling and accurate prediction of source loads in traditional centralized and distributed methods. By constructing a data-driven method, the solution time is greatly reduced, meeting the real-time requirements of online computing.

[0174] 2) A multi-micro-network system optimization method based on Bilevel Soft Actor-Critic (Bi-SAC) is proposed to overcome the problems of traditional reinforcement learning methods having difficulty in convergence for large-scale problems, difficulty in fully satisfying a large number of constraints, and low convergence accuracy. Through this hierarchical reinforcement learning method, the advantages of reinforcement learning based on data-driven principle and its ability to adapt to the randomness of source load are fully utilized, while effectively taking into account the solution accuracy of mathematical programming method, thus greatly improving the convergence speed and accuracy of the model.

[0175] 3) By using a hierarchical optimization framework, data privacy within each micronet is protected, effectively solving the data barrier problem between micronets.

[0176] See Figure 3 The hierarchical reinforcement learning optimization method for multi-micronet systems according to embodiments of the present invention specifically includes the following steps:

[0177] The multi-micronet optimization problem is decomposed into two layers: upper and lower.

[0178] Construct an upper-layer intelligent agent model that provides energy storage strategies within each microgrid and power interaction strategies between microgrids;

[0179] Construct a lower-layer autonomous optimization model that enables each microgrid to autonomously optimize the power output of devices within the microgrid based on the upper-layer energy storage strategy and power interaction strategy, and feeds back the optimization target value to the upper layer to guide the upper-layer strategy update;

[0180] Collaborative training of the lower-level autonomous optimization model and the upper-level intelligent agent model;

[0181] The lower-level autonomous optimization model and the upper-level intelligent agent model, which have been trained, provide scheduling strategies in real time based on the system state.

[0182] In one possible implementation, in the step of decomposing the multi-micronet optimization problem into two layers: constructing a hierarchical optimization framework for the multi-micronet system, the structure of the multi-micronet is as follows: Figure 1 As shown, the multi-microgrid optimization problem is decomposed into two layers for solution to achieve temporal and spatial decoupling. The upper-layer model is responsible for providing the energy storage strategy within each microgrid and the power interaction strategy between microgrids; the lower-layer microgrids autonomously optimize the power output of devices within the microgrid based on the upper-layer strategy, while simultaneously feeding back the optimization target value to the upper-layer model to guide the update of the upper-layer strategy. The collaboration between the upper and lower layers achieves global optimization of the multi-microgrid system. To fully utilize the adaptive source-load stochasticity of reinforcement learning and the advantages of mathematical programming in solution accuracy, the upper-layer strategy is formulated by the reinforcement learning agent, and the lower-layer strategy is provided by a mathematical programming solver. The optimization framework is as follows: Figure 2 As shown.

[0183] In one possible implementation, constructing the upper-layer intelligent agent model includes:

[0184] Step 2.1: Construct the agent's state space and action space

[0185] The state space of the agent represents the state information of the environment. It contains the information needed for the agent's decision-making, and the agent takes corresponding actions based on the current state and interacts with the environment. The state space includes the predicted renewable energy output, energy storage state of charge, time-of-use pricing information, and cooling, heating, and electrical load power of each microgrid, expressed as:

[0186] S={P wind ,P solar SOC, C price ,P C ,P H ,P E} (1)

[0187] The action space of the intelligent agent consists of relevant control variables, including the interaction power between microgrids and the energy storage charging and discharging power of each microgrid, as expressed below:

[0188] A = {P} ij,mg ,E} (2)

[0189] Step 2.2: Construct the agent's reward and punishment function

[0190] The reward function is the reward signal that the agent receives after selecting a corresponding action based on the current environmental state and applying it to the environment.

[0191] In this embodiment of the invention, the agent receives a reward proportional to the policy entropy at each step. By introducing an entropy term, the exploratory nature of reinforcement learning in the policy optimization process can be enhanced. The expression is:

[0192]

[0193] In the formula, π represents the current optimization strategy; γ is the reward discount factor; r is the reward function; and s t The state at time t; a t Let be the action at time t; α be the temperature coefficient (α > 0), used to determine the weight of the policy entropy term relative to the reward; H be the action at state s. t The entropy of the action taken by the next strategy is calculated as follows:

[0194]

[0195] The reward function includes two main categories: operating cost and penalty for exceeding the action constraint limit.

[0196] (1) Operating costs:

[0197]

[0198] f MG,t The overall operating cost of the multi-microgrid system during time period T;

[0199] (2) Penalty for exceeding the limits of motion constraints:

[0200] Action constraint violation penalties are used to satisfy the relevant constraints of decision-making actions. Among them, the operational constraints of energy storage include state of charge constraints, charging and discharging power constraints, and energy balance constraints at the beginning and end of the scheduling cycle, with the following expressions:

[0201] SOC min ≤SOC(t)≤SOC max (6)

[0202]

[0203] SOC(0)=SOC(T) (8)

[0204] In the formula, SOC min SOC max These represent the upper and lower limits of the battery's state of charge; P ch,max P dis,max These represent the maximum charge and discharge power of the battery; λ ch,t Let represent the state of charge / discharge coefficient of the battery during time period t.

[0205] The power interaction between microgrids needs to meet upper and lower limit constraints:

[0206] P ij,mg,min ≤P ij,mg (t)≤P ij,mg,max (9)

[0207] In the formula, P ij,mg,min P ij,mg,max These are the upper and lower limits of the interaction power between microgrid i and microgrid j, respectively.

[0208] To ensure the upper and lower bound constraints of the output action, the output action is limited to the range [-1, 1] by the tanh function, and the upper and lower bound constraints of equation (6) can be satisfied by a linear transformation. For equations (7) and (8), the following penalty terms are added to ensure that the action satisfies the constraints:

[0209]

[0210] φ2=k2|SOC(0)-SOC(T)| (11)

[0211] In the formula, k1 and k2 are penalty coefficients.

[0212] The final reward function for the agent is shown below. Since the agent explores with the goal of maximizing the reward, we take the negative value:

[0213] R t =-(f+φ1+φ2) (12)

[0214] Step 2.3: Construction of the Agent Network

[0215] The Q-function and policy distribution are parameterized using neural networks, and the two networks are optimized through alternating optimization.

[0216] The network parameters of the flexible Q-function can be obtained by training by minimizing the Bellman residuals, as shown in the following expression:

[0217]

[0218] In the formula, θ represents the parameters of the Q-function network. The parameters of the target Q-function network are updated using equation (14), where τ is the soft update coefficient, which is 0.005 in this embodiment. To enhance the training speed and stability of the network, this embodiment introduces two Q-function networks and trains them independently, selecting the smaller value as the Q-value of the target Q-network during training.

[0219]

[0220] The parameters φ of the policy distribution network are updated by minimizing its KL divergence:

[0221]

[0222] In the formula, the action is reparameterized to reduce the variance of the gradient estimate:

[0223] a t =f φ (ε t ,s t )=f φ μ (s t )+ε t ⊙f φ σ (s t (16)

[0224] μ and σ are the mean and variance of the policy distribution network output, respectively, and ε t The noise is sampled from a standard normal distribution.

[0225] The temperature coefficient α represents the importance weight of the policy entropy relative to the reward, and is automatically adjusted according to the degree of exploration of the policy space. It is adaptively updated by minimizing the following loss function:

[0226]

[0227] In the formula, κ is a hyperparameter representing the minimum expected entropy.

[0228] In one possible implementation, constructing the lower-level autonomous optimization model includes:

[0229] Step 3.1: Constructing the objective function.

[0230] This patent aims to minimize the operating cost of multi-microgrid systems.

[0231] minf MG,i =C i,fuel (t)+C i,mg (t)+C i,grid (t) (18)

[0232] In the formula, C i,fuel (t), C i,mg (t), C i,grid (t) represents the gas cost, adjacent microgrid interaction cost, and distribution network interaction cost of microgrid i during time period t, respectively. The calculation expressions for each cost are shown below:

[0233]

[0234]

[0235] C i,grid (t)=c grid (t)P i,grid (t) (21)

[0236] In the formula, c gas For natural gas prices; P i,MT (t) represents the MT output power of microgrid i during time period t; c mg (t), c grid (t) represents the electricity trading price between adjacent microgrids and the electricity trading price between a microgrid and the distribution network, respectively, during time period t; P ij,mg (t), P i,grid (t) represents the interaction power between microgrid i and microgrid j, and between microgrid i and the distribution network, respectively, during time period t.

[0237] Step 3.2: Establish the constraints for the autonomous optimization model.

[0238] The model constraints mainly include energy balance constraints and operational constraints.

[0239] (1) Energy balance constraint

[0240] These include the cold, heat, and electrical energy balance constraints within a multi-microgrid system, expressed by the following formulas respectively;

[0241] Q i,MT,c (t)+Q i,EC (t)=Q i,load,c (t) (22)

[0242] Q i,MT,h (t)+Q i,GB (t)=Q i,load,h (t) (23)

[0243]

[0244] Among them, Q i,load,c (t), Q i,load,h (t), P i,load (t) represents the cooling load, heating load, and electrical load of microgrid i during time period t.

[0245] (2) Operational constraints

[0246] To ensure the safe operation of multi-microgrid systems, it is also necessary to meet equipment operation constraints and power interaction constraints.

[0247] Micro gas turbine operating constraints:

[0248] P MT,min ≤P MT (t)≤PMT,max (25)

[0249] In the formula, P MT,min P MT,max These represent the upper and lower limits of the gas turbine's output, respectively.

[0250] Operating constraints of gas-fired boilers:

[0251] Q GB,min ≤Q GB (t)≤Q GB,max (26)

[0252] In the formula, Q GB,min Q GB,max These are the upper and lower limits of the output thermal power of the gas-fired boiler.

[0253] Electric chiller operating constraints:

[0254] P EC,min ≤P EC (t)≤P EC,max (27)

[0255] In the formula, P EC,min P EC,max These are the upper and lower limits of the input electrical power of the electric chiller, respectively.

[0256] Power interaction constraints:

[0257] P i,grid,min ≤P i,grid (t)≤P i,grid,max (28)

[0258] In the formula, P i,grid,min P i,grid,max These are the upper and lower limits of the interaction power between the microgrid i and the distribution network, respectively.

[0259] In one possible implementation, the process of collaboratively training the lower-level autonomous optimization model and the upper-level agent model includes:

[0260] First, initialize the network parameters of the value network (Q-function network) and the policy distribution network, as well as the temperature coefficients θ1 and θ2. φ, α; Initialize the state s0 of the multi-microgrid system from the state space, and give the scheduling action a for the current time segment based on the current state. t (i.e., energy storage optimization strategy and microgrid inter-system interaction strategy), and distribute them to the multi-microgrid system; each lower-level sub-microgrid system receives the upper-level energy storage optimization strategy and microgrid inter-system interaction strategy, and performs autonomous optimization based on its own state information. It then uses a solver to obtain the optimal power output strategy for the devices within the network at the current time segment, and updates the system state to s. t+1The objective function value f is then fed back to the upper-level agent; subsequently, the empirical data (s) are used to... t ,a t ,r t ,s t+1 ) Store to the experience playback unit, and batch extract samples M(s) from the experience playback unit. t ,a t ,r t ,s t+1 ), and obtain s through the policy network. t+1 Policy action a in state t+1 Finally, the value network parameters θ1, θ2 and the target value network parameters are updated using equations (13), (14), (15), and (17), respectively. The policy network parameters φ and the temperature coefficient α are determined; the above process is repeated until the model training is complete. After the model training is complete, a scheduling policy can be provided in real time based on the system state.

[0261] This invention proposes a hierarchical reinforcement learning method for multi-microgrid systems based on Bi-SAC, which is more suitable considering the random fluctuations of source loads. According to the characteristics of the multi-microgrid system optimization problem, it decomposes the problem into two layers, achieving time and space decoupling, simplifying the problem scale, and allowing each microgrid to be solved independently and in parallel. This achieves overall optimization of the multi-microgrid system while protecting the data privacy within each microgrid. The hierarchical reinforcement learning-based optimization method fully leverages the advantages of reinforcement learning's data-driven principle and its ability to adapt to the randomness of source loads, while effectively maintaining the solution accuracy of mathematical programming, significantly improving the model's convergence speed and accuracy.

[0262] See Figure 4 Another embodiment of the present invention also proposes a hierarchical reinforcement learning optimization system for multi-micronet systems, comprising:

[0263] The optimization problem decomposition module 1 is used to decompose the multi-micronet optimization problem into two layers;

[0264] Upper-layer intelligent agent model construction module 2 is used to construct an upper-layer intelligent agent model that provides energy storage strategies within each microgrid and power interaction strategies between microgrids.

[0265] The lower-level autonomous optimization model construction module 3 is used to construct a lower-level autonomous optimization model that enables each microgrid in the lower layer to autonomously optimize the output of devices within the microgrid based on the upper-level energy storage strategy and power interaction strategy, and to feed back the optimization target value to the upper layer to guide the upper-level strategy update.

[0266] Collaborative training module 4 is used for collaborative training of the lower-level autonomous optimization model and the upper-level agent model;

[0267] The scheduling strategy module 5 is used to provide a scheduling strategy in real time based on the system state by utilizing the trained lower-level autonomous optimization model and the upper-level intelligent agent model.

[0268] In one possible implementation, the upper-layer agent model construction module 2 constructs an upper-layer agent model that provides energy storage strategies within each microgrid and power interaction strategies between microgrids, including constructing agent state space and action space, constructing agent reward and punishment functions, and constructing agent network.

[0269] Furthermore, when the upper-layer intelligent agent model construction module 2 constructs the intelligent agent state space and action space, the constructed intelligent agent state space includes the predicted renewable energy output, energy storage state of charge, time-of-use electricity price information, and cold, heat, and electrical load power of each microgrid; the constructed intelligent agent action space consists of relevant control variables, including the interaction power between each microgrid and the energy storage charging and discharging power of each microgrid.

[0270] Furthermore, when the upper-layer agent model construction module 2 constructs the agent reward and punishment function, the resulting agent reward and punishment function expression is as follows:

[0271]

[0272] In the formula, π * The current optimization strategy; γ t s is the discount factor for time rewards; r is the reward function; s t The state at time t; a t The action at time t; α is the temperature coefficient, α > 0, used to determine the weight of the policy entropy term relative to the reward; H is the action at state s. t The entropy of the action taken by the next strategy is calculated as follows:

[0273]

[0274] The reward function r includes two main categories: operating cost and penalty for exceeding action constraints.

[0275] The formula for calculating the operating cost is:

[0276]

[0277] In the formula, f MG,t The overall operating cost of the multi-microgrid system during time period T;

[0278] The penalty for exceeding the action limit applies to the following operational constraints for energy storage: state of charge constraint, charge / discharge power constraint, and energy balance constraint at the beginning and end of the scheduling cycle, expressed as follows:

[0279] SOC min≤SOC(t)≤SOC max

[0280] 0≤P ch (t)≤λ ch,t P ch,max

[0281] 0≤P dis (t)≤(1-λ ch,t )P dis,max

[0282] λ ch,t ∈{0,1}

[0283] SOC(0)=SOC(T)

[0284] In the formula, SOC min SOC max These represent the upper and lower limits of the battery's state of charge; P ch,max P dis,max These represent the maximum charge and discharge power of the battery; λ ch,t This represents the state of charge / discharge coefficient of the battery during time period t;

[0285] The power interaction between each microgrid satisfies upper and lower limit constraints, expressed as follows:

[0286] P ij,mg,min ≤P ij,mg (t)≤P ij,mg,max

[0287] In the formula, P ij,mg,min P ij,mg,max These are the upper and lower limits of the interaction power between microgrid i and microgrid j, respectively.

[0288] The output action is constrained to the range [-1, 1] by the tanh function, and then the charge state constraint is satisfied by the linear transformation.

[0289] Add the following penalty terms to ensure that the charging and discharging power constraints and the energy balance constraints at the beginning and end of the scheduling cycle are met:

[0290]

[0291] φ2=k2|SOC(0)-SOC(T)|

[0292] In the formula, k1 and k2 are penalty coefficients;

[0293] The reward function for the final agent is as follows:

[0294] R t =-(f+φ1+φ2).

[0295] Furthermore, when constructing the agent network, the upper-layer agent model construction module 2 uses a Q-function network and a policy distribution network for alternating optimization. The network parameters of the Q-function network are obtained by minimizing the Bellman residual, and the calculation expression is as follows:

[0296]

[0297] In the formula, θ represents the parameters of the Q-function network to be solved. The parameters of the target Q-function network are expressed as follows:

[0298]

[0299] In the formula, τ is the soft update coefficient;

[0300] The two Q-function networks are trained independently, and the smaller value is selected as the parameter value of the target Q-function network during the training process;

[0301] The parameters φ of the policy distribution network are updated by minimizing its KL divergence, as expressed below:

[0302]

[0303] In the formula, the action is reparameterized to reduce the variance of the gradient estimate, as shown in the following expression:

[0304] a t =f φ (ε t ,s t )=f φ μ (s t )+ε t ⊙f φ σ (s t )

[0305] μ and σ are the mean and variance of the policy distribution network output, respectively, and ε t This is noise, sampled from a standard normal distribution;

[0306] The temperature coefficient α in the agent's reward and punishment function is adaptively updated by minimizing the following loss function:

[0307]

[0308] In the formula, κ is a hyperparameter representing the minimum expected entropy.

[0309] In one possible implementation, the lower-level autonomous optimization model construction module 3 constructs an objective function with the goal of minimizing the operating cost of the multi-microgrid system; at the same time, it establishes the constraints of the autonomous optimization model, including energy balance constraints and operational constraints.

[0310] Furthermore, the lower-level autonomous optimization model construction module 3 aims to minimize the operating cost of the multi-microgrid system. When constructing the objective function, the expression of the objective function is as follows:

[0311] minf MG,i =C i,fuel (t)+C i,mg (t)+C i,grid (t)

[0312] In the formula, C i,fuel (t), C i,mg (t), C i,grid (t) represents the gas cost, adjacent microgrid interaction cost, and distribution network interaction cost of microgrid i during time period t, respectively. The calculation expressions for each cost are as follows:

[0313]

[0314]

[0315] C i,grid (t)=c grid (t)P i,grid (t)

[0316] In the formula, c gas For natural gas prices; P i,MT (t) represents the MT output power of microgrid i during time period t; c mg (t), c grid (t) represents the electricity trading price between adjacent microgrids and the electricity trading price between a microgrid and the distribution network, respectively, during time period t; P ij,mg (t), P i,grid (t) represents the interaction power between microgrid i and microgrid j, and between microgrid i and the distribution network, respectively, during time period t.

[0317] Furthermore, when the lower-level autonomous optimization model construction module 3 establishes the constraints of the autonomous optimization model, the energy balance constraints include cold, heat, and electricity balance constraints within the multi-microgrid system. The expressions for the cold, heat, and electricity balance constraints within the multi-microgrid system are as follows:

[0318] Q i,MT,c (t)+Q i,EC (t)=Q i,load,c (t)

[0319] Q i,MT,h(t)+Q i,GB (t)=Q i,load,h (t)

[0320]

[0321] The operational constraints include equipment operational constraints and power interaction constraints;

[0322] The equipment operation constraints include:

[0323] The operating constraints of the micro gas turbine are expressed as follows:

[0324] P MT,min ≤P MT (t)≤P MT,max

[0325] In the formula, P MT,min P MT,max These represent the upper and lower limits of the gas turbine output, respectively.

[0326] The operating constraints for gas-fired boilers are expressed as follows:

[0327] Q GB,min ≤Q GB (t)≤Q GB,max

[0328] In the formula, Q GB,min Q GB,max These are the upper and lower limits of the output thermal power of the gas-fired boiler, respectively.

[0329] The operating constraints of the electric chiller are expressed as follows:

[0330] P EC,min ≤P EC (t)≤P EC,max

[0331] In the formula, P EC,min P EC,max These are the upper and lower limits of the input electrical power of the electric chiller, respectively.

[0332] The expression for the power interaction constraint is as follows:

[0333] P i,grid,min ≤P i,grid (t)≤P i,grid,max

[0334] In the formula, P i,grid,min P i,grid,max These are the upper and lower limits of the interaction power between the microgrid i and the distribution network, respectively.

[0335] In one possible implementation, the steps of the collaborative training module 4 in collaboratively training the lower-level autonomous optimization model and the upper-level agent model include initializing model parameters; initializing the state of the multi-microgrid system from the state space, providing scheduling actions for the current time segment based on the current state, including energy storage strategies within each microgrid and power interaction strategies between microgrids, and distributing these actions to the multi-microgrid system; each lower-level sub-microgrid system receiving the energy storage strategies and power interaction strategies from the upper layer, and simultaneously performing autonomous optimization based on its own state information to solve for the optimal output strategy of the devices within the network at the current time segment, updating the system state, and feeding back the objective function value to the upper-level agent; subsequently storing empirical data samples containing the state at time segment t, actions at time segment t, reward function at time segment t, and state at time segment t+1 to the experience replay unit, batch extracting empirical data samples from the experience replay unit, and obtaining the strategy actions at time segment t+1; finally, updating the network parameters; repeating the above process until model training is complete.

[0336] The optimization system of this invention addresses the problem that current optimization problems of multi-micronet systems are large in scale and difficult to solve accurately and efficiently. It combines deep reinforcement learning methods with traditional mathematical programming methods, making full use of the advantages of strong adaptive ability of reinforcement learning and high solution accuracy of mathematical programming methods, so as to achieve accurate and efficient solution of optimization problems of multi-micronet systems.

[0337] Another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the hierarchical reinforcement learning optimization method for multi-micronet systems.

[0338] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals. For ease of explanation, the above content only shows the parts related to the embodiments of the present invention; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This computer-readable storage medium is non-transitory and can be stored in storage devices formed by various electronic devices, enabling the execution process described in the method of the embodiments of the present invention.

[0339] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0340] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0341] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0342] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0343] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A hierarchical reinforcement learning optimization method for multi-micronetwork systems, characterized in that, include: The multi-micronet optimization problem is decomposed into two layers: upper and lower. Construct an upper-layer intelligent agent model that provides energy storage strategies within each microgrid and power interaction strategies between microgrids; Construct a lower-layer autonomous optimization model that enables each microgrid to autonomously optimize the power output of devices within the microgrid based on the upper-layer energy storage strategy and power interaction strategy, and feeds back the optimization target value to the upper layer to guide the upper-layer strategy update; Collaborative training of the lower-level autonomous optimization model and the upper-level intelligent agent model; The lower-level autonomous optimization model and the upper-level intelligent agent model, after training, provide scheduling strategies in real time based on the system state; The steps of constructing an upper-layer intelligent agent model that provides energy storage strategies within each microgrid and power interaction strategies between microgrids include: Construct the state space and action space of the intelligent agent; Construct reward and punishment functions for the intelligent agent; Constructing intelligent agent networks; The steps of constructing the state space and action space of the intelligent agent are as follows: the constructed state space of the intelligent agent includes the predicted output of renewable energy, the state of charge of energy storage, the time-of-use electricity price information, and the power of cold, heat and electricity loads of each microgrid; the constructed action space of the intelligent agent consists of relevant control variables, including the interaction power between each microgrid and the energy storage charging and discharging power of each microgrid. The step of constructing the agent's reward and punishment function results in the following expression for the agent's reward and punishment function: In the formula, π * The current optimization strategy; γ t s is the discount factor for time rewards; r is the reward function; s t The state at time t; a t The action at time t; α is the temperature coefficient, α > 0, used to determine the weight of the policy entropy term relative to the reward; H is the action at state s. t The entropy of the action taken by the next strategy is calculated as follows: The reward function r includes two main categories: operating cost and penalty for exceeding action constraints. The formula for calculating the operating cost is: In the formula, f MG,t The overall operating cost of the multi-microgrid system during time period T; The penalty for exceeding the action limit applies to the following operational constraints for energy storage: state of charge constraint, charge / discharge power constraint, and energy balance constraint at the beginning and end of the scheduling cycle, expressed as follows: SOC min ≤SOC(t)≤SOC max 0≤P ch (t)≤λ ch,t P ch,max 0≤P dis (t)≤(1-λ ch,t )P dis,max l ch,t ∈{0,1} SOC(0)=SOC(T) In the formula, SOC min SOC max These represent the upper and lower limits of the battery's state of charge; P ch,max P dis,max These represent the maximum charge and discharge power of the battery; λ ch,t This represents the state of charge / discharge coefficient of the battery during time period t; The power interaction between each microgrid satisfies upper and lower limit constraints, expressed as follows: P ij,mg,min ≤P ij,mg (t)≤P ij,mg,max In the formula, P ij,mg,min P ij,mg,max These are the upper and lower limits of the interaction power between microgrid i and microgrid j, respectively. The output action is constrained to the range [-1, 1] by the tanh function, and then the charge state constraint is satisfied by the linear transformation. Add the following penalty terms to ensure that the charging and discharging power constraints and the energy balance constraints at the beginning and end of the scheduling cycle are met: φ2=k2|SOC(0)-SOC(T)| In the formula, k1 and k2 are penalty coefficients; The reward function for the final agent is as follows: R t =-(f+φ1+φ2)。 2. The hierarchical reinforcement learning optimization method for multi-micronetwork systems according to claim 1, characterized in that, The step of constructing the agent network involves alternating optimization using a Q-function network and a policy distribution network. The network parameters of the Q-function network are obtained by minimizing the Bellman residual, and the calculation expression is as follows: In the formula, θ represents the parameters of the Q-function network to be solved. The parameters of the target Q-function network are expressed as follows: In the formula, τ is the soft update coefficient; The two Q-function networks are trained independently, and the smaller value is selected as the parameter value of the target Q-function network during the training process; The parameters φ of the policy distribution network are updated by minimizing its KL divergence, as expressed below: In the formula, the action is reparameterized to reduce the variance of the gradient estimate, as shown in the following expression: a t =f φ (e t ,s t )=f φ μ (s t )+e t ⊙f φ σ (s t ) μ and σ are the mean and variance of the policy distribution network output, respectively, and ε t This is noise, sampled from a standard normal distribution; The temperature coefficient α in the agent's reward and punishment function is adaptively updated by minimizing the following loss function: In the formula, κ is a hyperparameter representing the minimum expected entropy.

3. The hierarchical reinforcement learning optimization method for multi-micronetwork systems according to claim 1, characterized in that, The steps of constructing a lower-level autonomous optimization model that enables each lower-level microgrid to autonomously optimize the power output of devices within the microgrid based on the upper-level energy storage strategy and power interaction strategy, and to feed back optimization target values ​​to the upper layer to guide the updating of the upper-level strategy, include: To minimize the operating cost of a multi-microgrid system, an objective function is constructed. Establish constraints for the autonomous optimization model, including energy balance constraints and operational constraints.

4. The hierarchical reinforcement learning optimization method for multi-micronetwork systems according to claim 3, characterized in that, The steps for constructing the objective function with the goal of minimizing the operating cost of the multi-microgrid system include: The expression for the objective function is: minf MG,i =C i,fuel (t)+C i,mg (t)+C i,grid (t) In the formula, C i,fuel (t), C i,mg (t), C i,grid (t) represents the gas cost, adjacent microgrid interaction cost, and distribution network interaction cost of microgrid i during time period t, respectively. The calculation expressions for each cost are as follows: C i,grid (t)=c grid (t)P i,grid (t) In the formula, c gas For natural gas prices; P i,MT (t) represents the MT output power of microgrid i during time period t; c mg (t), c grid (t) represents the electricity trading price between adjacent microgrids and the electricity trading price between a microgrid and the distribution network, respectively, during time period t; P ij,mg (t), P i,grid (t) represents the interaction power between microgrid i and microgrid j, and between microgrid i and the distribution network, respectively, during time period t.

5. The hierarchical reinforcement learning optimization method for multi-micronetwork systems according to claim 3, characterized in that, When establishing the constraints of the autonomous optimization model, the energy balance constraints include cold, heat, and electricity balance constraints within the multi-microgrid system. The expressions for the cold, heat, and electricity balance constraints within the multi-microgrid system are as follows: Q i,MT,c (t)+Q i,EC (t)=Q i,load,c (t) Q i,MT,h (t)+Q i,GB (t)=Q i,load,h (t) The operational constraints include equipment operational constraints and power interaction constraints; The equipment operation constraints include: The operating constraints of the micro gas turbine are expressed as follows: P MT,min ≤P MT (t)≤P MT,max In the formula, P MT,min P MT,max These represent the upper and lower limits of the gas turbine output, respectively. The operating constraints for gas-fired boilers are expressed as follows: Q GB,min ≤Q GB (t)≤Q GB,max In the formula, Q GB,min Q GB,max These are the upper and lower limits of the output thermal power of the gas-fired boiler, respectively. The operating constraints of the electric chiller are expressed as follows: P EC,min ≤P EC (t)≤P EC,max In the formula, P EC,min P EC,max These are the upper and lower limits of the input electrical power of the electric chiller, respectively. The expression for the power interaction constraint is as follows: P i,grid,min ≤P i,grid (t)≤P i,grid,max In the formula, P i,grid,min P i,grid,max These are the upper and lower limits of the interaction power between the microgrid i and the distribution network, respectively.

6. The hierarchical reinforcement learning optimization method for multi-micronetwork systems according to claim 1, characterized in that, The steps of collaboratively training the lower-level autonomous optimization model and the upper-level intelligent agent model include initializing model parameters; initializing the state of the multi-microgrid system from the state space; providing scheduling actions for the current time segment based on the current state, including energy storage strategies within each microgrid and power interaction strategies between microgrids, and distributing these actions to the multi-microgrid system; each lower-level sub-microgrid system receiving the energy storage strategies and power interaction strategies from the upper layer, and simultaneously performing autonomous optimization based on its own state information to solve for the optimal power output strategy of the devices within the network at the current time segment, updating the system state, and feeding back the objective function value to the upper-level intelligent agent; Subsequently, the empirical data samples containing the state at time t, the action at time t, the reward function at time t, and the state at time t+1 are stored in the empirical replay unit. Empirical data samples are then extracted in batches from the empirical replay unit to obtain the policy action at time t+1. Finally, the network parameters are updated. The above process is repeated until the model training is complete.

7. A hierarchical reinforcement learning optimization system for multi-micronet systems, characterized in that, include: The optimization problem decomposition module is used to decompose the multi-micronet optimization problem into upper and lower layers; The upper-layer intelligent agent model construction module is used to construct upper-layer intelligent agent models that provide energy storage strategies within each microgrid and power interaction strategies between microgrids. The lower-level autonomous optimization model construction module is used to build a lower-level autonomous optimization model that enables each microgrid in the lower layer to autonomously optimize the output of devices within the microgrid based on the upper-level energy storage strategy and power interaction strategy, and to feed back the optimization target value to the upper layer to guide the upper-level strategy update. The collaborative training module is used to collaboratively train the lower-level autonomous optimization model and the upper-level agent model. The scheduling strategy module is used to provide a scheduling strategy in real time based on the system state using the trained lower-level autonomous optimization model and the upper-level intelligent agent model. The steps of the upper-layer intelligent agent model construction module to construct an upper-layer intelligent agent model that provides energy storage strategies within each microgrid and power interaction strategies between microgrids include constructing an intelligent agent state space and action space, constructing an intelligent agent reward and punishment function, and constructing an intelligent agent network. When the upper-layer intelligent agent model construction module constructs the intelligent agent state space and action space, the constructed intelligent agent state space includes the renewable energy predicted output, energy storage state of charge, time-of-use electricity price information, and cold, heat, and electric load power of each microgrid; the constructed intelligent agent action space consists of relevant control variables, including the interaction power between each microgrid and the energy storage charging and discharging power of each microgrid. When the upper-layer agent model construction module constructs the agent reward and punishment function, the resulting agent reward and punishment function expression is as follows: In the formula, π * The current optimization strategy; γ t s is the discount factor for time rewards; r is the reward function; s t The state at time t; a t The action at time t; α is the temperature coefficient, α > 0, used to determine the weight of the policy entropy term relative to the reward; H is the action at state s. t The entropy of the action taken by the next strategy is calculated as follows: The reward function r includes two main categories: operating cost and penalty for exceeding action constraints. The formula for calculating the operating cost is: In the formula, f MG,t The overall operating cost of the multi-microgrid system during time period T; The penalty for exceeding the action limit applies to the following operational constraints for energy storage: state of charge constraint, charge / discharge power constraint, and energy balance constraint at the beginning and end of the scheduling cycle, expressed as follows: SOC min ≤SOC(t)≤SOC max 0≤P ch (t)≤λ ch,t P ch,max 0≤P dis (t)≤(1-λ ch,t )P dis,max l ch,t ∈{0,1} SOC(0)=SOC(T) In the formula, SOC min SOC max These represent the upper and lower limits of the battery's state of charge; P ch,max P dis,max These represent the maximum charge and discharge power of the battery; λ ch,t This represents the state of charge / discharge coefficient of the battery during time period t; The power interaction between each microgrid satisfies upper and lower limit constraints, expressed as follows: P ij,mg,min ≤P ij,mg (t)≤P ij,mg,max In the formula, P ij,mg,min P ij,mg,max These are the upper and lower limits of the interaction power between microgrid i and microgrid j, respectively. The output action is constrained to the range [-1, 1] by the tanh function, and then the charge state constraint is satisfied by the linear transformation. Add the following penalty terms to ensure that the charging and discharging power constraints and the energy balance constraints at the beginning and end of the scheduling cycle are met: φ2=k2|SOC(0)-SOC(T)| In the formula, k1 and k2 are penalty coefficients; The reward function for the final agent is as follows: R t =-(f+φ1+φ2)。 8. The hierarchical reinforcement learning optimization system for multi-micronet systems according to claim 7, characterized in that, When constructing the agent network, the upper-layer agent model construction module uses a Q-function network and a policy distribution network for alternating optimization. The network parameters of the Q-function network are obtained by minimizing the Bellman residual, and the calculation expression is as follows: In the formula, θ represents the parameters of the Q-function network to be solved. The parameters of the target Q-function network are expressed as follows: In the formula, τ is the soft update coefficient; The two Q-function networks are trained independently, and the smaller value is selected as the parameter value of the target Q-function network during the training process; The parameters φ of the policy distribution network are updated by minimizing its KL divergence, as expressed below: In the formula, the action is reparameterized to reduce the variance of the gradient estimate, as shown in the following expression: a t =f φ (e t ,s t )=f φ μ (s t )+e t ⊙f φ σ (s t ) μ and σ are the mean and variance of the policy distribution network output, respectively, and ε t This is noise, sampled from a standard normal distribution; The temperature coefficient α in the agent's reward and punishment function is adaptively updated by minimizing the following loss function: In the formula, κ is a hyperparameter representing the minimum expected entropy.

9. The hierarchical reinforcement learning optimization system for multi-micronet systems according to claim 7, characterized in that, The lower-level autonomous optimization model construction module aims to minimize the operating cost of the multi-microgrid system by constructing an objective function; at the same time, it establishes the constraints of the autonomous optimization model, including energy balance constraints and operational constraints.

10. The hierarchical reinforcement learning optimization system for multi-micronetwork systems according to claim 9, characterized in that, The lower-level autonomous optimization model construction module aims to minimize the operating cost of the multi-microgrid system. When constructing the objective function, the expression of the objective function is: minf MG,i =C i,fuel (t)+C i,mg (t)+C i,grid (t) In the formula, C i,fuel (t), C i,mg (t), C i,grid (t) represents the gas cost, adjacent microgrid interaction cost, and distribution network interaction cost of microgrid i during time period t, respectively. The calculation expressions for each cost are as follows: C i,grid (t)=c grid (t)P i,grid (t) In the formula, c gas For natural gas prices; P i,MT (t) represents the MT output power of microgrid i during time period t; c mg (t), c grid (t) represents the electricity trading price between adjacent microgrids and the electricity trading price between a microgrid and the distribution network, respectively, during time period t; P ij,mg (t), P i,grid (t) represents the interaction power between microgrid i and microgrid j, and between microgrid i and the distribution network, respectively, during time period t.

11. The hierarchical reinforcement learning optimization system for multi-micronet systems according to claim 9, characterized in that, When the lower-level autonomous optimization model construction module establishes the constraints of the autonomous optimization model, the energy balance constraints include cold, heat, and electricity balance constraints within the multi-microgrid system. The expressions for the cold, heat, and electricity balance constraints within the multi-microgrid system are as follows: Q i,MT,c (t)+Q i,EC (t)=Q i,load,c (t) Q i,MT,h (t)+Q i,GB (t)=Q i,load,h (t) The operational constraints include equipment operational constraints and power interaction constraints; The equipment operation constraints include: The operating constraints of the micro gas turbine are expressed as follows: P MT,min ≤P MT (t)≤P MT,max In the formula, P MT,min P MT,max These represent the upper and lower limits of the gas turbine output, respectively. The operating constraints for gas-fired boilers are expressed as follows: Q GB,min ≤Q GB (t)≤Q GB,max In the formula, Q GB,min Q GB,max These are the upper and lower limits of the output thermal power of the gas-fired boiler, respectively. The operating constraints of the electric chiller are expressed as follows: P EC,min ≤P EC (t)≤P EC,max In the formula, P EC,min P EC,max These are the upper and lower limits of the input electrical power of the electric chiller, respectively. The expression for the power interaction constraint is as follows: P i,grid,min ≤P i,grid (t)≤P i,grid,max In the formula, P i,grid,min P i,grid,max These are the upper and lower limits of the interaction power between the microgrid i and the distribution network, respectively.

12. The hierarchical reinforcement learning optimization system for multi-micronet systems according to claim 7, characterized in that, The steps of the collaborative training module for collaboratively training the lower-level autonomous optimization model and the upper-level intelligent agent model include initializing model parameters; initializing the state of the multi-microgrid system from the state space, providing scheduling actions for the current time segment based on the current state, including energy storage strategies within each microgrid and power interaction strategies between microgrids, and distributing these actions to the multi-microgrid system; each lower-level sub-microgrid system receiving the energy storage strategies and power interaction strategies from the upper layer, and simultaneously performing autonomous optimization based on its own state information to solve for the optimal power output strategy of the devices within the network at the current time segment, updating the system state, and feeding back the objective function value to the upper-level intelligent agent; Subsequently, the empirical data samples containing the state at time t, the action at time t, the reward function at time t, and the state at time t+1 are stored in the empirical replay unit. Empirical data samples are then extracted in batches from the empirical replay unit to obtain the policy action at time t+1. Finally, the network parameters are updated. The above process is repeated until the model training is complete.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the hierarchical reinforcement learning optimization method for multi-micronet systems as described in any one of claims 1 to 6.

Images