Micro-grid energy optimization scheduling method oriented to source grid load storage

Through the microgrid energy optimization scheduling method for source grid load storage, combined with deep reinforcement learning algorithms, the coordinated management of distributed power supplies and energy storage equipment is optimized, and the problems of supply and demand balance and profit maximization of microgrids are solved, and the power system is improved Stability and economy.

CN118971051AActive Publication Date: 2024-11-15HENAN UNIV OF SCI & TECH

Patent Information

Application Number
CN202410956831.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-15
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

The existing microgrid scheduling methods are difficult to achieve supply and demand balance and profit maximization when facing a large number of distributed power generation units and uncertainties, resulting in the impact of the stability and economy of the power system.

Method used

Adopting the microgrid energy optimization scheduling method for source grid load storage, by constructing the microgrid energy scheduling architecture, establishing a mathematical model, combining the TD3 improved algorithm in the deep reinforcement learning algorithm, optimize distributed power supplies, energy storage equipment, temperature control load and Coordinated management between price response loads to achieve real-time decision-making and optimal allocation of resources.

Benefits of technology

It has achieved profit maximization and supply and demand balance for microgrid operators, and improved energy utilization efficiency and the stability and economics of power systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118971051A_ABST
    Figure CN118971051A_ABST
Patent Text Reader

Abstract

The invention discloses a source network load storage oriented microgrid energy optimization scheduling method. The method comprises the following steps: S1, constructing a microgrid energy scheduling architecture; s2, establishing a mathematical model of the micro-grid; s3, establishing an objective function of micro-grid energy optimization scheduling in combination with each mathematical model; and S4, converting an economic profit problem into a Markov decision process, and performing optimal scheduling of the micro-grid based on a TD3 improved algorithm in a deep reinforcement learning algorithm. According to the invention, by implementing an accurate energy management strategy, collaborative optimization is realized among the distributed power supply, the energy storage equipment, the main power grid, the temperature control load and the resident price response load so as to achieve the optimal management effect of local resources, and by means of the excellent generalization ability of the deep reinforcement learning model, a high-quality decision scheme is provided in real time. The objective of the invention is to realize profit maximization of micro-grid operators and maintain a balance state between supply and demand.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of smart grids, and in particular to a microgrid energy optimization scheduling method oriented to source, grid, load and storage. Background Art

[0002] As an independent operating entity, the microgrid exchanges electricity with the main grid and uses advanced energy storage facilities and automatic control systems to optimize internal resource allocation and achieve self-control and off-grid autonomy. Through a unified operating entity, the microgrid organizes energy transactions between distributed power sources, energy storage and user loads, and uses unified coordination and optimized scheduling to ensure the balance of supply and demand within the microgrid. At the same time, as a whole, the microgrid trades electricity with the main grid and provides necessary network-to-network auxiliary services to enhance the stability and reliability of the power system.

[0003] This major move has standardized the development of microgrids, promoted the diversified integration of energy and the interaction between supply and demand, improved energy efficiency, and laid a solid foundation for the goal of green and sustainable energy development. However, the large-scale integration of renewable energy has brought new challenges to energy system scheduling and increased the complexity of the problem. Traditional scheduling methods are stretched when faced with a large number of distributed generation units and increasing uncertainty.

[0004] Therefore, the present invention proposes a microgrid energy optimization scheduling method for source, grid, load and storage. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the existing defects and provide a microgrid energy optimization scheduling method for source, network, load and storage. By implementing precise energy management strategies, the optimal management effect of local resources is achieved. With the help of the excellent generalization ability of deep reinforcement learning models, high-quality decision-making solutions are provided in real time, aiming to maximize the profits of microgrid operators and maintain a balance between supply and demand, realize efficient utilization and flexible scheduling of energy within the microgrid, thereby ensuring the stability and economy of the power system, and can effectively solve the problems in the background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a microgrid energy optimization scheduling method for source-grid-load-storage, comprising the following steps:

[0007] S1. Construct a microgrid energy dispatching architecture. The microgrid energy dispatching architecture includes the main grid, distributed power sources, microgrids, loads, energy storage devices and a microgrid control center. The main grid, distributed power sources, microgrids, loads, energy storage devices and the microgrid control center are connected in communication. The main grid is responsible for selling electricity to the microgrid during peak hours and receiving electricity from the microgrid during low-peak hours. Distributed power sources include photovoltaic power generation, wind power generation and gas turbine power generation. Loads include temperature-controlled loads and price-responsive loads. Temperature-controlled loads change power consumption and operating power by adjusting the temperature of user equipment. Price-responsive loads encourage users to use electricity during low-peak hours through a time-of-use electricity price mechanism. Energy storage devices discharge during peak hours and charge during low-peak hours.

[0008] S2. Establish a mathematical model of the microgrid; the mathematical model of the microgrid includes a photovoltaic power generation model, a wind power generation model, a micro gas turbine model, an energy storage device model, a residential price response load model, a temperature control load model and a main power grid model, and each mathematical model establishes corresponding constraints;

[0009] S3, combining the various mathematical models in step S2 to establish the objective function of microgrid energy optimization scheduling, the objective function is to maximize the economic profit of the microgrid operator, that is, the income obtained from selling electricity to users within the microgrid and the external grid, minus the cost of generating electricity, purchasing electricity from the external grid and transmitting electricity;

[0010] S4. Convert the economic profit problem in step S3 into a Markov decision process, and optimize the scheduling of the microgrid based on the improved TD3 algorithm in the deep reinforcement learning algorithm.

[0011] Preferably, the mathematical model of the microgrid in step S2 is established by the following steps:

[0012] 1) Photovoltaic power generation model:

[0013] Photovoltaic power generation model refers to power generation equipment that directly utilizes light energy from solar energy and converts it into electrical energy. Due to the uncertainty of solar energy intensity, photovoltaic power generation units have obvious intermittent and uncertain characteristics. It is necessary to establish an accurate photovoltaic power generation model. Its output power can be expressed as:

[0014] P pv =ηSI(1-0.005(t0-25)) (1)

[0015] In formula (1): P pv is the photovoltaic power generation power (kW); S is the area of ​​the solar cell array (m 2 );η is the conversion efficiency of the solar cell array (%);I is the solar radiation intensity (kW / m 2 );t0 is the ambient temperature;

[0016] 2) Wind power generation model:

[0017] The wind power generation system has the characteristics of being distributed and can automatically adjust the power generation according to the change of wind speed, thereby maintaining the supply and demand balance of the microgrid. Its output power can be expressed as:

[0018]

[0019] In formula (2): P wt is the fan output power; ρ is the air density; R is the rotation radius of the fan blade; v is the maximum linear velocity on the fan blade; C p is the wind power conversion coefficient;

[0020] 3) Micro gas turbine model:

[0021] Gas turbine power generation converts the energy released by gas combustion into mechanical energy. It is necessary to comprehensively consider the gas turbine performance parameters, fuel consumption and power generation efficiency. Its output power can be expressed as:

[0022] P mt =F mt H ng ξ mt (3)

[0023]

[0024] In formula (3): P mt is the output power of the gas turbine (kW); F mt is the gas consumption (Nm 3 );H ng is the calorific value of natural gas (kWh / Nm 3 );ξ mt is the power generation efficiency (%);

[0025] In formula (4), is the maximum output power of the gas turbine (kW); is the minimum output power of the gas turbine;

[0026] 4) Energy storage equipment model:

[0027] Energy storage equipment is used in microgrids. It can stabilize the fluctuation of distributed energy, stabilize the output, and provide local utilization of distributed energy, avoiding long-distance transmission to the main grid to cause transmission pressure and power loss, which can be expressed as:

[0028]

[0029]

[0030] In formula (5): Bess (t) is the amount of electricity stored in the energy storage device at time t (kWh); δ is the self-discharge rate; P esc (t), P esf (t) is the charging and discharging power of the energy storage device at time t (kW); η esc , η esf is the charging and discharging efficiency of the energy storage device at time t; They are the upper and lower limits of the energy storage device power (kW); is the state of charge (SOC) of the energy storage device at time t; are the upper and lower limits of the state of charge of the energy storage device respectively; Δt is the charging and discharging time (h);

[0031] 5) Residential price response load model:

[0032] According to the long-term distribution characteristics of load, the peak, valley, and normal periods of load curve and electricity prices are reasonably divided, and users are guided to adjust their electricity consumption through intuitive electricity price change signals, so as to achieve the purpose of optimizing load time distribution, which is expressed as:

[0033]

[0034] In formula (9): L i0 , L i is the initial power consumption and the changed power consumption in period i; e is the electricity price elasticity coefficient, which indicates the ability of users on the demand side to transfer and reduce load; e ii is the self-elasticity coefficient. If the electricity price decreases during period i, the user will increase the electricity consumption; e ij is the mutual elasticity coefficient. If the electricity price in period j increases, the electricity consumption of users in period j will decrease and shift to period i. The electricity consumption of users in period i will increase, that is, e ii0 、e ij 0 ;p j0 is the initial electricity price in period j; Δp j It is the changed electricity price minus the initial electricity price;

[0035] 6) Temperature control load model:

[0036] Most households in the microgrid are equipped with temperature-controlled loads, such as air conditioners, heat pumps, water heaters, or refrigerators. These temperature-controlled loads can be directly controlled at each time step t using control signals from the temperature-controlled load aggregator. The temperature-controlled loads are only charged the power generation cost C gen To ensure the comfort of the end users, each temperature-controlled load is equipped with a backup controller to keep the temperature within an acceptable range. The backup controller receives the switching operation from the temperature-controlled load aggregator. Verify the temperature constraints and modify the operation as follows:

[0037]

[0038] In formula (10): is the switch action determined by the backup controller of the temperature control load of group i; T t i is the operating temperature of the temperature-controlled load equipment in the i-th group at time t; Upper and lower temperature limits set by the user;

[0039] In formula (11): W t i For each temperature-controlled load, an average state of charge is defined, which determines T t i Relative position within the desired temperature range;

[0040] 7) Main grid model:

[0041] The operation status of the main grid under different conditions can be simulated, including power supply and demand balance, frequency and voltage stability, etc. When the circuit breaker of the microgrid is closed, the microgrid and the main grid can exchange electric energy, which is expressed as:

[0042]

[0043] S t <B t (13)

[0044] In formula (12): P grid represents the power traded between the microgrid and the main grid (kW); if P grid >0? It means purchasing electricity from the main grid, otherwise, it means selling electricity to the main grid; They are the upper and lower limits of the power traded between the microgrid and the main grid (kW);

[0045] In formula (13): S t represents the price of electricity sold to the main grid at time t when electricity is abundant; B t It represents the price of purchasing electricity from the main power grid at time t when there is a power shortage.

[0046] Preferably, in step S3, economy is an important indicator of the microgrid. It is necessary to reduce costs and improve energy efficiency as much as possible while meeting electricity demand. The economic profit objective function of the microgrid operator is shown in formula (14):

[0047]

[0048]

[0049] Formula (15) represents the income obtained by the microgrid from selling electricity; Pi It represents the price of electricity sold by the microgrid to users; represents the power consumed by the i-th group of temperature control loads; Indicates the amount of electricity sold to the main grid;

[0050] Formula (16) represents the cost of the microgrid; C gen For the power generation cost, it is assumed here that the power generation costs of photovoltaic, wind and gas turbine are the same; is the amount of electricity purchased from the main power grid; C in and C out They represent the transmission costs of electricity imported from and delivered to the external grid respectively;

[0051] Formula (17) represents the power balance constraint condition; P load is the active power consumed by all residential loads in the microgrid; P tcl It is the active power consumed by all temperature-controlled loads in the microgrid.

[0052] Preferably, in step S4, the Markov decision process is that in a round of microgrid energy optimization management, the intelligent agent observes the current environmental state s of the microgrid t ∈S, select an action a according to the action selection strategy π:S→A t ∈A is used to adjust the operating status of the power controllable devices in the microgrid and the amount of power traded with the main grid. The environment will respond to the action of the intelligent agent according to a t ∈A gives feedback r t , and due to the generation of this action, the state s t ∈S transitions to the next state s t+1 ∈S;

[0053] The Markov decision process is defined as: (S, A, P, R, γ); S is a finite set of states; A is a finite set of actions; P is the state transition probability; R is the reward function; γ is the discount factor (used to calculate the cumulative return);

[0054] 1) State space:

[0055] The state space refers to the information used by the microgrid energy optimization management system in the decision-making process at each time step t, denoted by s t ∈S means, which is expressed as:

[0056]

[0057] s t Including current time t, temperature T t , Photovoltaic power generation P pv , Wind power generation P wt 、User required load L t, Real-time electricity price for purchasing electricity from the main power grid B t , the price P of electricity sold by the microgrid to users t , the average charge state W of the temperature control load group t i And the charge state of the energy storage system at the current moment;

[0058] 2) Action Space:

[0059] The action space refers to the set of all actions that can be performed by the microgrid energy optimization management system, represented by a t ∈A means, which can be expressed as:

[0060] a t =[A tcl ,A p ,A b ,A s ] (19)

[0061] The action space consists of four parts: TCL action space A tcl 、Price Action Space A p , insufficient energy action space A b and energy excess action space A s ; A tcl It is to act on the temperature-controlled load. The temperature-controlled load backup controller determines the switch action according to the temperature and the average charge state, and specifies the energy allocation level of TCL; A p The price is set according to the user's electricity consumption, and there are three types of electricity prices: peak, flat and valley. b When the current microgrid is short of electricity, it can purchase electricity from the main grid, discharge energy storage, or generate electricity using gas turbines, and specify the priority of energy storage, gas turbines, or the grid; A s If the microgrid currently has excess electricity, it can sell electricity to the main grid or charge energy storage, and specify the priority of energy storage or the grid;

[0062] 3) State transition probability:

[0063] The state transition probability refers to the probability of moving from a state s t Take an action t Then transfer to the next state s t+ The probability of 1 is expressed as P(s t+1 |s t ,a t ) is expressed as:

[0064]

[0065] The state of charge of the energy storage device The price P of electricity sold by the microgrid to users t , the average charge state W of the temperature control load groupt i is affected by the action, the transfer probability is known, and the operating temperature T of the temperature control load equipment at time t t , Photovoltaic power generation power P pv 、Wind power generation power P wt 、Load power L t and the main grid electricity price B t The state transition probability is unknown, and a large amount of historical data is needed to train the neural network to learn the best strategy;

[0066] 4) Reward function:

[0067] The reward function is an important concept in decision-making problems. It describes the immediate reward or penalty obtained by the microgrid energy optimization management system after taking an action in the current state. The reward function is represented by R(s t ,a t ) is expressed as:

[0068] R(s t ,a t )=λ1R es +λ2F t -μΔP t (twenty one)

[0069]

[0070] The setting of the reward function not only affects the learning efficiency of the system, but also affects the final decision of the system. In the problem of energy optimization management, the reward function should guide the agent to take actions to maximize the operating profit while executing the power balance constraint. An additional energy storage priority reward item is set. When the price is above the price threshold, a positive reward signal is given to discharge and a negative reward signal is given to charge. When the price is below the price threshold, a negative reward signal is given to discharge and a positive reward signal is given to charge to guide the agent to take better actions to maintain power balance and maximize operating profits. λ1, λ2 are weight coefficients, and μ is the penalty coefficient, which is used to control the trade-off between the reward item of the energy storage equipment, profit maximization and the penalty generated when the power is unbalanced.

[0071] 5) Discount Factor:

[0072] The discount factor is a constant between 0 and 1, denoted by γ; in microgrid optimization scheduling, since resource management and energy allocation usually involve long-term planning and multi-stage decision-making, the reward may not only be limited to the immediate return at the current moment, but also need to take into account the impact on the future state. The long-term reward defined by G t It is expressed as:

[0073] G t =R t+γR t+1 +γ 2 R t+2 +…+γ n-1 R n =R t +γG t+1 (twenty three)

[0074] The discount factor γ allows us to find a balance between these immediate rewards and long-term rewards.

[0075] Preferably, the TD3 algorithm in the deep reinforcement learning algorithm in step S4 is improved, and the steps are as follows:

[0076] 1) Replace the neural network of the Actor network with a long short-term memory network (LSTM), and input the state into the LSTM layer. The LSTM layer processes the input sequence data and outputs a hidden state, which contains all the information observed so far. The output of the LSTM layer will be fed into a fully connected layer, which is responsible for mapping the output of the LSTM to the action space to generate the final action. In short, replacing the Actor network in the ITD3 algorithm with the LSTM network can better process the time series data in the microgrid optimization scheduling, capture the time dependency of the state, and generate a more accurate and effective action sequence;

[0077] h t =o t tanh(c t ) (twenty four)

[0078] a t =π(h t ;θ) (25)

[0079] Formula (24) represents the hidden state at the current moment, o t is the switch state of the output gate of LSTM, tanh represents the hyperbolic tangent function, c t is the cell state at the current moment; Equation (25) represents the output action at time t, where π represents the Actor network and θ is the parameter of the network;

[0080] 2) Define two Actor networks and use the standard deviation of the action as the scale of random noise. From the two independent Actor networks defined, output actions according to the current state respectively, compare the two actions and select the larger one as the final action to help the agent quickly converge to a good strategy. Calculate the standard deviation of the action as the scale of random noise, generate random noise with the same shape as the final action, and add random noise to the action to balance exploration and utilization and avoid falling into the local optimal solution.

[0081] noise~N(0,σ 2 )(26)

[0082] Formula (26) is used to generate random noise, where N(0,σ 2 ) means the mean is 0 and the variance is σ 2 The normal distribution of , σ is the standard deviation of the action;

[0083] 3) Use the importance sampling ratio clipping technique to clip the loss of the Actor network. First, calculate the importance ratio r between the current strategy and the old strategy. t (θ), then the importance ratio is trimmed to ensure that the difference between the new and old strategies is not too large, and finally the final Actor loss L is calculated Actor Total (θ), that is, choosing the smaller of the two alternative losses as the final alternative loss and adding an entropy regularization term to encourage exploration

[0084]

[0085] L Actor Total (θ) = L clip (θ)-β·H(π) (29)

[0086] Formula (27) is the importance sampling ratio, which is the ratio of the new strategy distribution function to the old strategy distribution function; π(s t |a t ) indicates the current state s t Next, perform action a t The probability represents the strategy of the agent. Importance sampling helps the algorithm judge the effect of the new strategy (new action) compared with the old strategy (old action). If the new strategy is better than the old one, the algorithm will be more inclined to adopt the new strategy, but at the same time, the algorithm will also try some different strategies to find a possible better solution.

[0087] Formula (28) means that the replacement loss is calculated using the clipped importance ratio and the unclipped importance ratio respectively, and the smaller of the two replacement losses is selected as the final replacement loss. The operator min is used to select the smallest one between the first and second terms. There is a clip function in front of the second term. The clip function means: there are three terms in the brackets. If the first term is less than the second term, 1-ε is output; if the first term is greater than the third term, 1+ε is output, which is responsible for limiting the ratio r t (θ) is within the interval, ε is a hyperparameter that we need to adjust, usually set to 0.1 or 0.2, represents the empirical average over the sample;

[0088] Formula (29) represents the final Actor loss, where H(π) is the entropy of the strategy and β is a hyperparameter that controls the strength of the entropy regularization term.

[0089] 4) Actor network training, first at each time step, output actions from the two Actor networks according to the current state: a1 = π1(s; θ1) and a2 = π2(s; θ2), where θ1 and θ2 are the parameters of the two Actor networks, respectively. After comparing the two actions: select the larger action as the final action, a = max(a1, a2), and add random noise to the calculated final action a, and then use the target Critic network to estimate the Q of the executed action. π (s,a) value, Actor’s loss function L actor Usually based on negative Q π (s, a) value, because our goal is to maximize the expected return, and then use the importance sampling ratio clipping technique to clip the loss of the Actor network to obtain the final loss function L Actor Total (θ) to balance exploration and exploitation, and finally use the gradient descent method to update the parameters θ of the Actor network according to the calculated Actor loss to find a strategy that maximizes the expected return;

[0090]

[0091]

[0092] Formula (30) is the state behavior value function; Formula (31) is the original Actor network loss function, where Q πθ (s t ,a t ) means that when taking strategy π θ Time state t Take action a t The value of (parameterized by the Critic network); N is the number of state-action pairs in the batch; the negative sign is because we usually minimize the loss during the optimization process, while the Q-value function is to be maximized. The action network parameters are updated through formula (32) to obtain a new strategy that meets the truncation requirements, σ A is the learning rate of the Actor network;

[0093] 5) Critic network training. First, a batch of data is randomly sampled from the experience replay buffer, including state (s), action (a), reward (r), and next state (s'). Then, the Critic network is used to estimate the Q value of the current state-action pair: Q1(s,a; φ1) and Q2(s,a; φ2), where φ1 and φ2 are the parameters of the two Critic networks respectively. Then, the target Critic network is used to estimate the Q value of the next state-action pair: Q target' (s',a';φ target' ), the calculation of the target Q value involves the target Actor network and the target Critic network: the target Actor network is used to select the action in the next state s', while the target Critic network is used to estimate the Q value corresponding to this action, and then the mean square error is used to calculate the Critic loss function L(φ), and finally the gradient descent method is used to update the parameters φ of the Critic network;

[0094] Q target' (s',a';φ target' )=Q target' (s',π'(s';θ target' );φ target' ) (33)

[0095] L(φ)=(yQ(s,a;φ)) 2 (34)

[0096] y=r+γmini=1,2Q target,i' (s',π i' (s';θ target' );φ target' ) (35)

[0097]

[0098] Formula (33) first selects action a' through the target Actor network π', and then uses this action and the target Critic network Q target' To estimate the target Q value, where a'=π'(s'; θ target' ) is the action selected by the target Actor network according to the state s′, θ target' is the parameter of the target Actor network, and the parameter of the target Critic network is φ target' ;

[0099] Formula (34) is the loss function of the Critic network. This loss function is used to measure the difference between the Q value estimated by the current Critic network and the target Q value. Q(s, a; φ) represents the Q value estimated by the current Critic network under the given state s and action a using the parameter φ. y represents the target Q value, which is calculated based on the data sampled in the experience replay buffer.

[0100] In formula (35), r is the immediate reward, γ is the discount factor, and mini=1,2 means taking the smaller value of the Q value estimated by the two target critic networks in order to reduce the overoptimistic tendency of the Q value estimation. target,i' and π i' They are the estimates of the target Critic network and the target Actor network respectively;

[0101] Formula (36) uses the gradient descent method to update the size of the Critic network parameters φ, σ C is the learning rate of the Critic network.

[0102] Preferably, the overall process of the microgrid optimization scheduling algorithm using the ITD3 algorithm proposed in step S4 is as follows: in the tth round of energy optimization management, the state s of the microgrid is t Input LSTM to extract time series features, and then send it to the ITD3 network. ITD3 includes two Actor networks and two Critic networks, all of which use a four-layer network structure. The first layer is the input layer, followed by two hidden layers, and finally the output layer. The policy network continuously interacts with the environment, collects the interaction data and temporarily stores it in the trajectory cache pool, and transmits the trajectory data to the policy network and evaluation network to update the network parameters. The updated network has more accurate value assessment and action selection performance. As the interaction between the intelligent agent and the environment gradually deepens, the policy network training gradually stabilizes until convergence. The specific steps are as follows:

[0103] Step 1: Initialization: Set the initial state s0 of the microgrid and initialize the network parameters required by the TD3 algorithm, including two Actor networks and two Critic networks, as well as their respective target networks;

[0104] Step 2: Data collection: In the tth round of energy optimization management, the state of the microgrid s t Input into the online Actor network, the two Actor networks each output an action, and the larger action a is selected t After executing this action, observe the next state s of the microgrid t+1 and the reward r t , will (s t ,a t ,r t,s t+1 ) Such a quadruple is temporarily stored in the experience replay buffer pool;

[0105] Step 3: Network training: Randomly extract a batch of four-tuple data from the experience replay buffer pool to train the online Actor network and Critic network. In the TD3 algorithm, the Critic network is responsible for estimating the state value function Q π (s, a), while the Actor network updates its strategy based on the gradient information provided by the Critic network and clips the loss using the importance sampling ratio clipping technique. During the training process, the target network (a copy of the online network with delayed parameter updates) is used to provide stable target values ​​to enhance the stability of the algorithm.

[0106] Step 4: Delayed update: The parameters of the target network are not updated at every step, but are updated after a certain number of steps of online network training. A soft update of the online network parameters is adopted, using θ′=τθ+(1-τ)θ′, where θ is the parameter of the online network, θ′ is the parameter of the target network, and τ is a small constant;

[0107] Step 5: Strategy improvement: As the online network is trained, the strategy will gradually improve. When the strategy becomes stable or reaches the preset number of training rounds, the algorithm can be considered to have converged.

[0108] Step 6: Output strategy: When the algorithm converges, use the online Actor network as the final energy optimization management strategy and output corresponding actions according to the current state of the microgrid.

[0109] Compared with the prior art, the beneficial effect of the present invention is: by implementing precise energy management strategies, the coordinated optimization between distributed power sources, energy storage equipment, main power grid, temperature control loads and residential price response loads is achieved to achieve the best management effect of local resources. At the same time, with the excellent generalization ability of the deep reinforcement learning model, high-quality decision-making solutions are provided in real time, aiming to maximize the profits of microgrid operators, maintain a balance between supply and demand, and achieve efficient utilization and flexible scheduling of energy within the microgrid, thereby ensuring the stability and economy of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0111] The present invention can be explained in detail through the following embodiments. The purpose of disclosing the present invention is to protect all technical improvements within the scope of the present invention. In the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "front", "back", "left", "right", etc. indicating directions or positional relationships, they only correspond to the drawings of the present application for the convenience of describing the present invention, and do not indicate or imply that the device or element referred to must have a specific direction.

[0112] See also Figure 1 The present invention provides a technical solution: a microgrid energy optimization scheduling method for source-grid-load-storage, comprising the following steps:

[0113] S1. Build a microgrid energy dispatching architecture. The microgrid energy dispatching architecture includes the main grid, distributed power sources, microgrids, loads, energy storage equipment and a microgrid control center. The main grid, distributed power sources, microgrids, loads, energy storage equipment and the microgrid control center are connected in communication.

[0114] Among them, the main grid is responsible for selling electricity to the microgrid during peak electricity consumption and receiving electricity from the microgrid during low electricity consumption; distributed power sources include photovoltaic power generation, wind power generation and gas turbine power generation. These components make full use of renewable energy and provide diversified energy supply for the microgrid; loads include temperature-controlled loads and price-responsive loads. Temperature-controlled loads (air conditioners, refrigerators, water heaters, etc.) change the power consumption and operating power by adjusting the temperature of user equipment to achieve flexible control of power loads. Price-responsive loads encourage users to use electricity during low electricity consumption periods through a time-of-use electricity price mechanism, thereby indirectly participating in the energy management of the microgrid; energy storage devices discharge during peak electricity consumption and charge during low-peak periods, which not only balances the load of the power system, but also effectively smoothes the volatility of renewable energy; energy storage devices discharge during peak electricity consumption and charge during low-peak periods. The real-time status information of these devices is collected and transmitted to the microgrid control center through smart meters. The control center uses a proximal strategy optimization algorithm to send precise control signals based on the current status of the microgrid to achieve efficient use and optimized management of energy.

[0115] S2. Establish a mathematical model of the microgrid; the mathematical model of the microgrid includes a photovoltaic power generation model, a wind power generation model, a micro gas turbine model, an energy storage device model, a residential price response load model, a temperature control load model and a main grid model. Each mathematical model establishes corresponding constraints. The mathematical model is as follows:

[0116] 1) Photovoltaic power generation model:

[0117] Photovoltaic power generation model refers to power generation equipment that directly utilizes light energy from solar energy and converts it into electrical energy. Its power generation principle is to utilize the photovoltaic effect of semiconductor materials. Due to the uncertainty of solar energy intensity, photovoltaic power generation units have obvious intermittent and uncertain characteristics, which brings great challenges to their grid connection and participation in power system scheduling tasks. It is necessary to establish an accurate photovoltaic power generation model and accurately predict the output power of photovoltaic power generation units, which is crucial to promote the research on the optimized operation of microgrids. Its output power can be expressed as:

[0118] P pv =ηSI(1-0.005(t0-25)) (1)

[0119] In formula (1): P pv is the photovoltaic power generation power (kW); S is the area of ​​the solar cell array (m 2 );η is the conversion efficiency of the solar cell array (%);I is the solar radiation intensity (kW / m 2 );t0 is the ambient temperature;

[0120] 2) Wind power generation model:

[0121] The wind power generation system has the characteristics of being distributed and can automatically adjust the power generation according to the change of wind speed, so as to maintain the supply and demand balance of the microgrid. Therefore, it is necessary to predict the output power of wind power generation, maintain the supply and demand balance of the microgrid, and improve the reliability of the microgrid. Its output power can be expressed as:

[0122]

[0123] In formula (2): P wt is the fan output power; ρ is the air density; R is the rotation radius of the fan blade; v is the maximum linear velocity on the fan blade; C p is the wind power conversion coefficient;

[0124] 3) Micro gas turbine model:

[0125] Gas turbine power generation converts the energy released by gas combustion into mechanical energy. In order to effectively dispatch and calculate the cost of microgrids, the output power of micro gas turbines must be accurately calculated. It is necessary to comprehensively consider the performance parameters, fuel consumption and power generation efficiency of gas turbines to ensure the stable operation of microgrids and maximize economic benefits. Its output power can be expressed as:

[0126] P mt =F mt H ng ξ mt (3)

[0127]

[0128] In formula (3): P mt is the output power of the gas turbine (kW); F mt is the gas consumption (Nm 3 );H ng is the calorific value of natural gas (kWh / Nm 3 );ξ mt is the power generation efficiency (%);

[0129] In formula (4), is the maximum output power of the gas turbine (kW); is the minimum output power of the gas turbine;

[0130] 4) Energy storage equipment model:

[0131] Energy storage equipment is used in microgrids. Through the energy management system, distributed energy, energy storage equipment and the main grid are coordinated and controlled to stabilize the fluctuation of distributed energy, stabilize the output, and provide local utilization of distributed energy, avoiding long-distance transmission to the main grid to cause transmission pressure and power loss, which can be expressed as:

[0132]

[0133] In formula (5): B ess (t) is the amount of electricity stored in the energy storage device at time t (kWh); δ is the self-discharge rate; P esc (t), P esf (t) is the charging and discharging power of the energy storage device at time t (kW); η esc , η esf is the charging and discharging efficiency of the energy storage device at time t; They are the upper and lower limits of the energy storage device power (kW); is the state of charge (SOC) of the energy storage device at time t; are the upper and lower limits of the state of charge of the energy storage device respectively; Δt is the charging and discharging time (h);

[0134] 5) Residential price response load model:

[0135] According to the long-term distribution characteristics of the load, the load curve peak, valley, and normal periods and electricity prices are reasonably divided. Through intuitive electricity price change signals, users are guided to adjust their electricity consumption methods, so as to achieve the purpose of optimizing the load time distribution. Through this model, the changes in residents' electricity consumption under different electricity price levels can be quantitatively analyzed, and decision support can be provided for microgrid operators to formulate reasonable scheduling strategies to balance the relationship between electricity supply and demand and improve the power supply reliability and economy of the microgrid, which can be expressed as:

[0136]

[0137] In formula (9): L i0 , L i is the initial power consumption and the changed power consumption in period i; e is the electricity price elasticity coefficient, which indicates the ability of users on the demand side to transfer and reduce load; e ii is the self-elasticity coefficient. If the electricity price decreases during period i, the user will increase the electricity consumption; e ij is the mutual elasticity coefficient. If the electricity price in period j increases, the electricity consumption of users in period j will decrease and shift to period i. The electricity consumption of users in period i will increase, that is, e ii0 、e ij 0 ;p j0 is the initial electricity price in period j; Δp j It is the changed electricity price minus the initial electricity price;

[0138] 6) Temperature control load model:

[0139] Most households in the microgrid are equipped with temperature-controlled loads, such as air conditioners, heat pumps, water heaters, or refrigerators. These temperature-controlled loads can be directly controlled at each time step t using control signals from the temperature-controlled load aggregator. The temperature-controlled loads are only charged the power generation cost C gen To ensure the comfort of the end users, each temperature-controlled load is equipped with a backup controller to keep the temperature within an acceptable range. The backup controller receives the switching operation from the temperature-controlled load aggregator. Verify the temperature constraints and modify the operation as follows:

[0140]

[0141] In formula (10): is the switch action determined by the backup controller of the temperature control load of group i; T t i is the operating temperature of the temperature-controlled load equipment in the i-th group at time t; Upper and lower temperature limits set by the user;

[0142] In formula (11): W t i For each temperature-controlled load, an average state of charge is defined, which determines T t i Relative position within the desired temperature range;

[0143] 7) Main grid model:

[0144] The operation status of the main grid under different conditions can be simulated, including power supply and demand balance, frequency and voltage stability. When the circuit breaker of the microgrid is closed, the microgrid and the main grid can exchange electric energy. By establishing a mathematical model of the main grid, the operation status of the main grid under different conditions can be simulated, including power supply and demand balance, frequency and voltage stability. This helps microgrid operators to formulate more reasonable power dispatching and trading strategies and improve the power supply reliability and economy of the microgrid, which is expressed as:

[0145]

[0146] S t <B t (13)

[0147] In formula (12): P grid represents the power traded between the microgrid and the main grid (kW); if P grid >0? It means purchasing electricity from the main grid, otherwise, it means selling electricity to the main grid; They are the upper and lower limits of the power traded between the microgrid and the main grid (kW);

[0148] In formula (13): S t represents the price of electricity sold to the main grid at time t when electricity is abundant; B t It represents the price of purchasing electricity from the main power grid at time t when there is a power shortage.

[0149] S3, combining the various mathematical models in step S2 to establish the objective function of microgrid energy optimization scheduling, the objective function is to maximize the economic profit of the microgrid operator, that is, the income obtained from selling electricity to users within the microgrid and the external grid, minus the cost of generating electricity, purchasing electricity from the external grid and transmitting electricity;

[0150] The design of a microgrid needs to follow some basic principles and goals. Economic efficiency is an important indicator for designing a microgrid. It is necessary to reduce costs and improve energy efficiency as much as possible while meeting electricity demand. The objective function designed by the present invention is as shown in formula (14), which is to maximize the economic profit of the microgrid operator, that is, the income obtained from selling electricity to users within the microgrid and the external grid, minus the cost of generating electricity, purchasing electricity from the external grid, and transmitting electricity;

[0151]

[0152] Formula (15) represents the income obtained by the microgrid from selling electricity, P i It represents the price of electricity sold by the microgrid to users; represents the power consumed by the i-th group of temperature control loads; represents the amount of electricity sold to the main grid, and formula (16) represents the cost of the microgrid, C gen For the power generation cost, it is assumed here that the power generation costs of photovoltaic, wind, and micro gas turbines are the same; is the amount of electricity purchased from the main power grid; C in and C out They represent the transmission costs from the external grid and to the external grid respectively. Formula (17) represents the power balance constraint condition, P load is the active power consumed by all residential loads in the microgrid, P tcl It is the active power consumed by all temperature-controlled loads in the microgrid.

[0153] S4. Convert the economic profit problem in step S3 into a Markov decision process, and optimize the scheduling of the microgrid based on the improved TD3 algorithm in the deep reinforcement learning algorithm.

[0154] Convert the profit problem into a Markov decision process:

[0155] The goal of reinforcement learning is to find the optimal strategy given a Markov decision process. In a round of microgrid energy optimization management, the agent observes the current environmental state of the microgrid s t ∈S, select an action a according to the action selection strategy π:S→A t ∈A is used to adjust the operating status of the power controllable devices in the microgrid and the amount of power traded with the main grid. The environment will respond to the action of the intelligent agent according to a t ∈A gives feedback r t , and due to the generation of this action, the state s t ∈S transitions to the next state s t+1 ∈S. The Markov decision process describes the above process mathematically. The Markov decision process is defined as: (S, A, P, R, γ), where S is a finite state set; A is a finite action set; P is the state transition probability; R is the reward function; γ is the discount factor (used to calculate the cumulative return).

[0156] 1) State space:

[0157] The state space refers to the information used by the microgrid energy optimization management system in the decision-making process at each time step t, denoted by s t ∈S is expressed as:

[0158]

[0159] s t Including current time t, temperature T t , Photovoltaic power generation P pv , Wind power generation P wt 、User required load Lt , Real-time electricity price for purchasing electricity from the main power grid B t , the price P of electricity sold by the microgrid to users t , the average charge state W of the temperature control load group t i And the current state of charge of the energy storage system

[0160] 2) Action Space:

[0161] The action space refers to the set of all actions that can be performed by the microgrid energy optimization management system, represented by a t ∈A is expressed as:

[0162] a t =[A tcl ,A p ,A b ,A s ] (19)

[0163] The action space consists of four parts: TCL action space A tcl 、Price Action Space A p , insufficient energy action space A b and energy excess action space A s ; A tcl It is to act on the temperature-controlled load. The temperature-controlled load backup controller determines the switch action according to the temperature and the average charge state, and specifies the energy allocation level of TCL; A p The price is set according to the user's electricity consumption, and there are three types of electricity prices: peak, flat and valley. b When the current microgrid is short of electricity, it can purchase electricity from the main grid, discharge energy storage, or generate electricity using gas turbines, and specify the priority of energy storage, gas turbines, or the grid; A s If the microgrid currently has excess electricity, it can sell electricity to the main grid or charge energy storage, and specify the priority of energy storage or the grid;

[0164] 3) State transition probability:

[0165] The state transition probability refers to the probability of moving from a state s t Take an action t Then transfer to the next state s t+1 The probability of t+1 |s t ,a t ) is expressed as:

[0166]

[0167] The state of charge of the energy storage device The price P of electricity sold by the microgrid to users t, the average charge state W of the temperature control load group t i is affected by the action, the transfer probability is known, and the operating temperature T of the temperature control load equipment at time t t , Photovoltaic power generation power P pv 、Wind power generation power P wt 、Load power L t and the main grid electricity price B t The state transition probability is unknown, so a large amount of historical data is needed to train the neural network to learn the best strategy;

[0168] 4) Reward function:

[0169] The reward function is an important concept in decision-making problems. It describes the immediate reward or penalty obtained by the microgrid energy optimization management system after taking an action in the current state. The reward function is represented by R(s t ,a t ) is expressed as:

[0170] R(s t ,a t )=λ1R es +λ2F t -μΔP t (twenty one)

[0171]

[0172] The setting of the reward function not only affects the learning efficiency of the system, but also affects the final decision of the system. In the energy optimization management problem, the reward function should guide the agent to take actions to maximize the operating profit while executing the power balance constraint. Here, an additional energy storage priority reward item is set. When the price is above the price threshold, a positive reward signal is given to discharge and a negative reward signal is given to charge. When the price is below the price threshold, a negative reward signal is given to discharge and a positive reward signal is given to charge to guide the agent to take better actions to maintain power balance and maximize operating profits. λ1, λ2 are weight coefficients, and μ is the penalty coefficient, which is used to control the trade-off between the reward item of the energy storage equipment, profit maximization and the penalty generated when the power is unbalanced.

[0173] 5) Discount Factor:

[0174] The discount factor is a constant between 0 and 1, denoted by γ. In microgrid optimization scheduling, since resource management and energy allocation usually involve long-term planning and multi-stage decision-making, the reward may not only be limited to the immediate return at the current moment, but also need to take into account the impact on future states. The long-term reward defined by G t It is expressed as formula (23):

[0175] Gt =R t +γR t+1 +γ 2 R t+2 +…+γ n-1 R n =R t +γG t+1 (twenty three)

[0176] The discount factor γ allows us to find a balance between these immediate rewards and long-term rewards.

[0177] Microgrid energy optimization management algorithm design:

[0178] This application uses the TD3 algorithm in the deep reinforcement learning algorithm to optimize the scheduling of the microgrid, and improves the algorithm to optimize the results. The specific improvements to TD3 are as follows:

[0179] 1. Replace the Actor network’s neural network with a Long Short-Term Memory (LSTM) network and replace the state s t Input to the LSTM layer, the LSTM layer processes the input sequence data and outputs a hidden state h t , which contains all the information observed so far, and the output h of the LSTM layer t will be fed into a fully connected layer, which is responsible for mapping the output of LSTM to the action space to produce the final action a t In summary, replacing the Actor network in the ITD3 algorithm with an LSTM network can better process time series data in microgrid optimization scheduling, capture the time dependency of the state, and generate more accurate and effective action sequences, which will help improve the performance and reliability of the microgrid.

[0180] h t =o t tanh(c t ) (twenty four)

[0181] a t =π(h t ;θ) (25)

[0182] Formula (24) represents the hidden state at the current moment, o t is the switch state of the output gate of LSTM, tanh represents the hyperbolic tangent function, c t is the cell state at the current moment;

[0183] Formula (25) represents the output action at time t, where π represents the Actor network and θ is the parameter of the network.

[0184] 2. Define two Actor networks, and use the standard deviation of the action as the scale of random noise. From the two independent Actor networks defined, output actions according to the current state respectively, compare the two actions and select the larger one as the final action to help the agent quickly converge to a good strategy, and calculate the standard deviation of the action as the scale of random noise, generate random noise with the same shape as the final action, and add random noise to the action to balance exploration and utilization and avoid falling into the local optimal solution.

[0185] noise~N(0,σ 2 ) (26)

[0186] Formula (26) is used to generate random noise, where N(0,σ 2 ) means the mean is 0 and the variance is σ 2 is a normal distribution, σ is the standard deviation of the action.

[0187] 3. Use the importance sampling ratio clipping technique to clip the loss of the Actor network. First, calculate the importance ratio r between the current strategy and the old strategy. t (θ), then the importance ratio is trimmed to ensure that the difference between the new and old strategies is not too large, and finally the final Actor loss L is calculated Actor Total (θ), that is, selecting the smaller of the two alternative losses as the final alternative loss and adding an entropy regularization term to encourage exploration. By introducing the importance ratio clipping technique, the amplitude of the strategy update can be limited, which may improve the stability and convergence speed of training.

[0188]

[0189] L Actor Total (θ) = L clip (θ)-β·H(π) (29)

[0190] Formula (27) is the importance sampling ratio, which is the ratio of the new strategy distribution function to the old strategy distribution function; π(s t |a t ) indicates the current state s t Next, perform action a t The probability of , which represents the strategy of the agent; importance sampling helps the algorithm determine the effect of the new strategy (new action) compared with the old strategy (old action); if the new strategy is better than the old one, the algorithm will be more inclined to adopt the new strategy; but at the same time, the algorithm will also try some different strategies to find a possible better solution;

[0191] Formula (28) means that the replacement loss is calculated using the clipped importance ratio and the unclipped importance ratio respectively, and the smaller of the two replacement losses is selected as the final replacement loss. The operator min is used to select the smallest one between the first and second terms. There is a clip function in front of the second term. The clip function means: there are three terms in the brackets. If the first term is less than the second term, 1-ε is output; if the first term is greater than the third term, 1+ε is output, which is responsible for limiting the ratio r t (θ) is within the interval, ε is a hyperparameter that we need to adjust, usually set to 0.1 or 0.2, represents the empirical average over the samples; Formula (29) represents the final Actor loss, where H(π) is the entropy of the strategy and β is a hyperparameter that controls the strength of the entropy regularization term.

[0192] 4.Actor Network Training

[0193] First, at each time step, the two Actor networks output actions according to the current state: a1 = π1(s; θ1) and a2 = π2(s; θ2), where θ1 and θ2 are the parameters of the two Actor networks, respectively. After comparing the two actions, the larger action is selected as the final action, a = max(a1, a2), and random noise is added to the calculated final action a. Then, the target Critic network is used to estimate the Q of the executed action. π (s,a) value, Actor’s loss function L actor Usually based on negative Q π (s, a) value, because our goal is to maximize the expected return, and then use the importance sampling ratio clipping technique to clip the loss of the Actor network to obtain the final loss function L Actor Total (θ) to balance exploration and exploitation, and finally use the gradient descent method to update the parameters θ of the Actor network according to the calculated Actor loss. This process aims to find a strategy that maximizes the expected return.

[0194]

[0195] Formula (30) is the state behavior value function, and Formula (31) is the original Actor network loss function. Here, Q πθ (s t ,a t ) means that when taking strategy π θ Time state t Take action a tThe value of (parameterized by the Critic network); N is the number of state-action pairs in the batch. Note that the negative sign here is because we usually minimize the loss in the optimization process, while the Q-value function is to be maximized. The action network parameters are updated through formula (32) to obtain a new strategy that meets the truncation requirements, σ A is the learning rate of the Actor network.

[0196] 5. Critic Network Training

[0197] First, a batch of data is randomly sampled from the experience replay buffer, including state (s), action (a), reward (r), and next state (s'). Then, the Critic network is used to estimate the Q value of the current state-action pair: Q1(s,a; φ1) and Q2(s,a; φ2), where φ1 and φ2 are the parameters of the two Critic networks respectively. Then, the target Critic network is used to estimate the Q value of the next state-action pair: Q target' (s',a';φ target' ), the calculation of the target Q value involves the target Actor network and the target Critic network. The target Actor network is used to select the action in the next state s', while the target Critic network is used to estimate the Q value corresponding to this action. The mean square error is then used to calculate the Critic loss function L(φ), and finally the gradient descent method is used to update the parameters φ of the Critic network.

[0198] Q target' (s',a';φ target' )=Q target' (s',π'(s';θ target' );φ target' ) (33)

[0199] L(φ)=(yQ(s,a;φ)) 2 (34)

[0200] y=r+γmini=1,2Q target,i' (s',π i' (s';θ target' );φ target' ) (35)

[0201]

[0202] Formula (33) first selects action a' through the target Actor network π', and then uses this action and the target Critic network Q target' To estimate the target Q value, where a'=π'(s'; θ target' ) is the action selected by the target Actor network according to the state s′, θtarget' is the parameter of the target Actor network, and the parameter of the target Critic network is φ target' ;

[0203] Formula (34) is the loss function of the Critic network. This loss function is used to measure the difference between the Q value estimated by the current Critic network and the target Q value. Q(s, a; φ) represents the Q value estimated by the current Critic network under the given state s and action a using the parameter φ. y represents the target Q value, which is calculated based on the data sampled in the experience replay buffer.

[0204] In formula (35), r is the immediate reward, γ is the discount factor, and mini=1,2 means taking the smaller value of the Q value estimated by the two target critic networks. This is to reduce the overoptimistic tendency of the Q value estimate. target,i' and π i' They are the estimates of the target Critic network and the target Actor network respectively;

[0205] Formula (36) uses the gradient descent method to update the size of the Critic network parameters φ, σ C is the learning rate of the Critic network.

[0206] 6. Algorithm steps

[0207] The overall process of the microgrid energy optimization management algorithm based on ITD3 proposed in this application is as follows: in the tth round of energy optimization management, the state s of the microgrid is t Input LSTM to extract time series features, and then send it to the ITD3 network. ITD3 includes two Actor networks and two Critic networks, all of which use a four-layer network structure. The first layer is the input layer, followed by two hidden layers, and finally the output layer. The policy network continuously interacts with the environment, collects the interaction data and temporarily stores it in the trajectory cache pool, and transmits the trajectory data to the policy network and evaluation network to update the network parameters. The updated network has more accurate value assessment and action selection performance. As the interaction between the intelligent agent and the environment gradually deepens, the policy network training gradually stabilizes until convergence. The specific steps are as follows:

[0208] Step 1: Initialization: Set the initial state s0 of the microgrid and initialize the network parameters required by the TD3 algorithm, including two Actor networks and two Critic networks, as well as their respective target networks;

[0209] Step 2: Data collection: In the tth round of energy optimization management, the state of the microgrid s t Input into the online Actor network, the two Actor networks each output an action, and the larger action a is selected tAfter executing this action, observe the next state s of the microgrid t+1 and the reward r t , will (s t ,a t ,r t ,s t+1 ) Such a quadruple is temporarily stored in the experience replay buffer pool;

[0210] Step 3: Network training: Randomly extract a batch of four-tuple data from the experience replay buffer pool to train the online Actor network and Critic network. In the TD3 algorithm, the Critic network is responsible for estimating the state value function Q π (s, a), while the Actor network updates its strategy based on the gradient information provided by the Critic network and clips the loss using the importance sampling ratio clipping technique. During the training process, the target network (a copy of the online network with delayed parameter updates) is used to provide stable target values ​​to enhance the stability of the algorithm.

[0211] Step 4: Delayed update: The parameters of the target network are not updated at every step, but are updated after a certain number of steps of online network training. A soft update of the online network parameters is adopted, using θ′=τθ+(1-τ)θ′, where θ is the parameter of the online network, θ′ is the parameter of the target network, and τ is a small constant;

[0212] Step 5: Strategy improvement: As the online network is trained, the strategy will gradually improve. When the strategy becomes stable or reaches the preset number of training rounds, the algorithm can be considered to have converged.

[0213] Step 6: Output strategy: When the algorithm converges, use the online Actor network as the final energy optimization management strategy and output corresponding actions according to the current state of the microgrid.

[0214] The parts of the present invention that are not described in detail are prior art. It is obvious to those skilled in the art that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention; therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and it is intended that all changes that fall within the meaning and scope of equivalent elements are included in the content of the present invention.

Claims

1. A microgrid energy optimization scheduling method for source, grid, load and storage, characterized by: The following steps are involved: S1. Build a microgrid energy dispatching architecture; The microgrid energy dispatching architecture includes the main grid, distributed power sources, microgrids, loads, energy storage devices and a microgrid control center. The main grid, distributed power sources, microgrids, loads, energy storage devices and the microgrid control center are connected in communication. The main grid is responsible for selling electricity to the microgrid during peak hours and receiving electricity from the microgrid during off-peak hours. Distributed power sources include photovoltaic power generation, wind power generation and gas turbine power generation. Loads include temperature-controlled loads and price-responsive loads. Temperature-controlled loads change power consumption and operating power by adjusting the temperature of user equipment. Price-responsive loads encourage users to use electricity during off-peak hours through a time-of-use electricity price mechanism. Energy storage devices discharge during peak hours and charge during off-peak hours. S2. Establish a mathematical model of the microgrid; the mathematical model of the microgrid includes a photovoltaic power generation model, a wind power generation model, a micro gas turbine model, an energy storage device model, a residential price response load model, a temperature control load model and a main grid model, and each mathematical model establishes corresponding constraints; S3, combining the various mathematical models in step S2 to establish the objective function of microgrid energy optimization scheduling, the objective function is to maximize the economic profit of the microgrid operator, that is, the income obtained from selling electricity to users within the microgrid and the external grid, minus the cost of generating electricity, purchasing electricity from the external grid and transmitting electricity; S4. Convert the economic profit problem in step S3 into a Markov decision process, and optimize the scheduling of the microgrid based on the improved TD3 algorithm in the deep reinforcement learning algorithm.

2. According to a method for optimizing the energy dispatch of a microgrid for source, grid, load and storage according to claim 1, it is characterized by: The mathematical model of the microgrid in step S2 is established as follows: 1) Photovoltaic power generation model: Photovoltaic power generation model refers to power generation equipment that directly utilizes light energy from solar energy and converts it into electrical energy. Due to the uncertainty of solar energy intensity, photovoltaic power generation units have obvious intermittent and uncertain characteristics. It is necessary to establish an accurate photovoltaic power generation model. Its output power can be expressed as: P pv =ηSI(1-0.005(t0-25)) (1) In formula (1): P pv is the photovoltaic power generation power (kW); S is the area of ​​the solar cell array (m 2 );η is the conversion efficiency of the solar cell array (%);I is the solar radiation intensity (kW / m 2 );t0 is the ambient temperature; 2) Wind power generation model: The wind power generation system has the characteristics of being distributed and can automatically adjust the power generation according to the change of wind speed, thereby maintaining the supply and demand balance of the microgrid. Its output power can be expressed as: In formula (2): P wt is the fan output power; ρ is the air density; R is the rotation radius of the fan blade; v is the maximum linear velocity on the fan blade; C p is the wind power conversion coefficient; 3) Micro gas turbine model: Gas turbine power generation converts the energy released by gas combustion into mechanical energy. It is necessary to comprehensively consider the gas turbine performance parameters, fuel consumption and power generation efficiency. Its output power can be expressed as: P mt =F mt H ng ξ mt (3) In formula (3): P mt is the output power of the gas turbine (kW); F mt is the gas consumption (Nm 3 );H ng is the calorific value of natural gas (kWh / Nm 3 );ξ mt is the power generation efficiency (%); In formula (4), is the maximum output power of the gas turbine (kW); is the minimum output power of the gas turbine; 4) Energy storage equipment model: Energy storage equipment is used in microgrids. It can stabilize the fluctuation of distributed energy, stabilize the output, and provide local utilization of distributed energy, avoiding long-distance transmission to the main grid to cause transmission pressure and power loss, which can be expressed as: In formula (5): B ess (t) is the amount of electricity stored in the energy storage device at time t (kWh); δ is the self-discharge rate; P esc (t), P esf (t) is the charging and discharging power of the energy storage device at time t (kW); η esc , η esf is the charging and discharging efficiency of the energy storage device at time t; They are the upper and lower limits of the energy storage device power (kW); is the state of charge (SOC) of the energy storage device at time t; are the upper and lower limits of the state of charge of the energy storage device respectively; Δt is the charging and discharging time (h); 5) Residential price response load model: According to the long-term distribution characteristics of load, the peak, valley, and normal periods of load curve and electricity prices are reasonably divided, and users are guided to adjust their electricity consumption through intuitive electricity price change signals, so as to achieve the purpose of optimizing load time distribution, which is expressed as: In formula (9): L i0 , L i is the initial power consumption and the changed power consumption in period i; e is the electricity price elasticity coefficient, which indicates the ability of users on the demand side to transfer and reduce load; ii is the self-elasticity coefficient. If the electricity price decreases during period i, the user will increase the electricity consumption; e ij is the mutual elasticity coefficient. If the electricity price in period j increases, the electricity consumption of users in period j will decrease and shift to period i. The electricity consumption of users in period i will increase, that is, e ii0 、e ij0 ;p j0 is the initial electricity price in period j; Δp j It is the changed electricity price minus the initial electricity price; 6) Temperature control load model: Most households in the microgrid are equipped with temperature-controlled loads, such as air conditioners, heat pumps, water heaters, or refrigerators. These temperature-controlled loads can be directly controlled at each time step t using control signals from the temperature-controlled load aggregator. The temperature-controlled loads are only charged the power generation cost C gen To ensure the comfort of the end users, each temperature-controlled load is equipped with a backup controller to keep the temperature within an acceptable range. The backup controller receives the switching operation from the temperature-controlled load aggregator. Verify the temperature constraints and modify the operation as follows: In formula (10): The switch action determined by the backup controller of the temperature control load of group i; is the operating temperature of the temperature-controlled load equipment in the i-th group at time t; Upper and lower temperature limits set by the user; In formula (11): For each temperature-controlled load, an average state of charge is defined, which determines T t i's relative position within the desired temperature range; 7) Main grid model: The operation status of the main grid under different conditions can be simulated, including power supply and demand balance, frequency and voltage stability, etc. When the circuit breaker of the microgrid is closed, the microgrid and the main grid can exchange electric energy, which is expressed as: S t <B t (13) In formula (12): P grid represents the power traded between the microgrid and the main grid (kW); if P grid >0? It means purchasing electricity from the main grid, otherwise, it means selling electricity to the main grid; They are the upper and lower limits of the power traded between the microgrid and the main grid (kW); In formula (13): S t Represents the price of electricity sold to the main grid at time t when electricity is abundant; B t It represents the price of purchasing electricity from the main power grid at time t when there is a power shortage.

3. The method for optimizing the energy dispatch of a microgrid for source, grid, load and storage according to claim 2, characterized in that: In step S3, economy is an important indicator of the microgrid. It is necessary to reduce costs and improve energy efficiency as much as possible while meeting electricity demand. The economic profit objective function of the microgrid operator is shown in formula (14): Formula (15) represents the income obtained by the microgrid from selling electricity; P i It represents the price of electricity sold by the microgrid to users; represents the power consumed by the i-th group of temperature control loads; Indicates the amount of electricity sold to the main grid; Formula (16) represents the cost of the microgrid; C gen For the power generation cost, it is assumed here that the power generation costs of photovoltaic, wind and gas turbine are the same; is the amount of electricity purchased from the main power grid; C in and C out They represent the transmission costs of electricity imported from and delivered to the external grid respectively; Formula (17) represents the power balance constraint condition; P load is the active power consumed by all residential loads in the microgrid; P tcl It is the active power consumed by all temperature-controlled loads in the microgrid.

4. The method for optimizing the energy dispatch of a microgrid for source, grid, load and storage according to claim 2, characterized in that: In step S4, the Markov decision process is that in a round of microgrid energy optimization management, the intelligent agent observes the current environmental state s of the microgrid t ∈S, select an action a according to the action selection strategy π:S→A t ∈A is used to adjust the operating status of the power controllable devices in the microgrid and the amount of power traded with the main grid. The environment will respond to the action of the intelligent agent according to a t ∈A gives feedback r t , and due to the generation of this action, the state s t ∈S transitions to the next state s t+1 ∈S; The Markov decision process is defined as: (S, A, P, R, γ); S is a finite set of states; A is a finite set of actions; P is the state transition probability; R is the reward function; γ is the discount factor (used to calculate the cumulative return); 1) State space: The state space refers to the information used by the microgrid energy optimization management system in the decision-making process at each time step t, denoted by s t ∈S means, which is expressed as: s t Including current time t, temperature T t , Photovoltaic power generation P pv , Wind power generation P wt 、User required load L t , Real-time electricity price for purchasing electricity from the main power grid B t , the price P of electricity sold by the microgrid to users t , the average charge state W of the temperature control load group t i And the charge state of the energy storage system at the current moment; 2) Action Space: The action space refers to the set of all actions that can be performed by the microgrid energy optimization management system, represented by a t ∈A means, which can be expressed as: a t =[A tcl ,A p ,A b ,A s ] (19) The action space consists of four parts: TCL action space A tcl 、Price Action Space A p , insufficient energy action space A b and energy excess action space A s ; A tcl It is to act on the temperature-controlled load. The temperature-controlled load backup controller determines the switch action according to the temperature and the average charge state, and specifies the energy allocation level of TCL; A p The price is set according to the user's electricity consumption, and there are three types of electricity prices: peak, flat and valley. b When the current microgrid is short of electricity, it can purchase electricity from the main grid, discharge energy storage, or generate electricity using gas turbines, and specify the priority of energy storage, gas turbines, or the grid; A s If the microgrid currently has excess electricity, it can sell electricity to the main grid or charge energy storage, and specify the priority of energy storage or the grid; 3) State transition probability: The state transition probability refers to the probability of moving from a state s t Take an action t Then transfer to the next state s t+ The probability of 1 is expressed as P(s t+1 |s t ,a t ) is expressed as: The state of charge of the energy storage device The price P of electricity sold by the microgrid to users t , the average charge state of the temperature control load group is affected by the action, the transfer probability is known, and the operating temperature T of the temperature control load equipment at time t t , Photovoltaic power generation power P pv 、Wind power generation power P wt 、Load power L t and the main grid electricity price B t The state transition probability is unknown, and a large amount of historical data is needed to train the neural network to learn the best strategy; 4) Reward function: The reward function is an important concept in decision-making problems. It describes the immediate reward or penalty obtained by the microgrid energy optimization management system after taking an action in the current state. The reward function is represented by R(s t ,a t ) is expressed as: R(s t ,a t )=λ1R es +λ2F t -μΔP t (21) The setting of the reward function not only affects the learning efficiency of the system, but also affects the final decision of the system. In the problem of energy optimization management, the reward function should guide the agent to take actions to maximize the operating profit while executing the power balance constraint. An additional energy storage priority reward item is set. When the price is above the price threshold, a positive reward signal is given to discharge and a negative reward signal is given to charge. When the price is below the price threshold, a negative reward signal is given to discharge and a positive reward signal is given to charge to guide the agent to take better actions to maintain power balance and maximize operating profits. λ1, λ2 are weight coefficients, and μ is the penalty coefficient, which is used to control the trade-off between the reward item of the energy storage equipment, profit maximization and the penalty generated when the power is unbalanced. 5) Discount Factor: The discount factor is a constant between 0 and 1, denoted by γ; in microgrid optimization scheduling, since resource management and energy allocation usually involve long-term planning and multi-stage decision-making, the reward may not only be limited to the immediate return at the current moment, but also need to take into account the impact on the future state. The long-term reward defined by G t It is expressed as: G t =R t +γR t+1 +g 2 R t+2 +…+c n-1 R n =R t +γG t+1 (23) The discount factor γ allows us to find a balance between these immediate rewards and long-term rewards.

5. The method for optimizing the energy dispatch of a microgrid for source, grid, load and storage according to claim 4 is characterized by: The TD3 algorithm in the deep reinforcement learning algorithm in step S4 is improved as follows: 1) Replace the neural network of the Actor network with a long short-term memory network (LSTM), and input the state into the LSTM layer. The LSTM layer processes the input sequence data and outputs a hidden state, which contains all the information observed so far. The output of the LSTM layer will be sent to a fully connected layer, which is responsible for mapping the output of the LSTM to the action space to generate the final action. In short, replacing the Actor network in the ITD3 algorithm with the LSTM network can better process the time series data in the microgrid optimization scheduling, capture the time dependency of the state, and generate a more accurate and effective action sequence; h t =o t fishy t ) (24) a t =π(h t (i) (25) Formula (24) represents the hidden state at the current moment, o t is the switch state of the output gate of LSTM, tanh represents the hyperbolic tangent function, c t is the cell state at the current moment; Equation (25) represents the output action at time t, where π represents the Actor network and θ is the parameter of the network; 2) Define two Actor networks and use the standard deviation of the action as the scale of random noise. From the two independent Actor networks defined, output actions according to the current state respectively, compare the two actions and select the larger one as the final action to help the agent quickly converge to a good strategy. Calculate the standard deviation of the action as the scale of random noise, generate random noise with the same shape as the final action, and add random noise to the action to balance exploration and utilization and avoid falling into the local optimal solution. noise~N(0,σ 2 ) (26) Formula (26) is used to generate random noise, where N(0,σ 2 ) means the mean is 0 and the variance is σ 2 The normal distribution of , σ is the standard deviation of the action; 3) Use the importance sampling ratio clipping technique to clip the loss of the Actor network. First, calculate the importance ratio r between the current strategy and the old strategy. t (θ), then the importance ratio is trimmed to ensure that the difference between the new and old strategies is not too large, and finally the final Actor loss L is calculated ActorTotal (θ), that is, choosing the smaller of the two alternative losses as the final alternative loss and adding an entropy regularization term to encourage exploration; L ActorTotal (θ)=L clip (θ)-β·H(π) (29) Formula (27) is the importance sampling ratio, which is the ratio of the new strategy distribution function to the old strategy distribution function; π(s t |a t ) indicates the current state s t Next, perform action a t The probability represents the strategy of the agent. Importance sampling helps the algorithm judge the effect of the new strategy (new action) compared with the old strategy (old action). If the new strategy is better than the old one, the algorithm will be more inclined to adopt the new strategy. Formula (28) means that the replacement loss is calculated using the clipped importance ratio and the unclipped importance ratio respectively, and the smaller of the two replacement losses is selected as the final replacement loss. The operator min is used to select the smallest one between the first and second terms. There is a clip function in front of the second term. The clip function means: there are three terms in the brackets. If the first term is less than the second term, 1-ε is output; if the first term is greater than the third term, 1+ε is output, which is responsible for limiting the ratio r t (θ) is within the interval, ε is a hyperparameter that we need to adjust, usually set to 0.1 or 0.2, represents the empirical average over the sample; Formula (29) represents the final Actor loss, where H(π) is the entropy of the strategy and β is a hyperparameter that controls the strength of the entropy regularization term. 4) Actor network training: First, at each time step, the two Actor networks output actions according to the current state: a1 = π1(s; θ1) and a2 = π2(s; θ2), where θ1 and θ2 are the parameters of the two Actor networks, respectively. Then, the two actions are compared and the larger action is selected as the final action, a = max(a1, a2). Random noise is added to the calculated final action a, and then the target Critic network is used to estimate the Q of the executed action. π (s,a) value, Actor’s loss function L actor Usually based on negative Q π (s, a) value, because the goal is to maximize the expected return, and then use the importance sampling ratio clipping technique to clip the loss of the Actor network to obtain the final loss function L ActorTotal (θ) to balance exploration and exploitation, and finally use the gradient descent method to update the parameters θ of the Actor network according to the calculated Actor loss to find a strategy that maximizes the expected return; Formula (30) is the state behavior value function; Formula (31) is the original Actor network loss function, where Indicates that when adopting strategy π θ Time state t Take action a t The value of (parameterized by the Critic network); N is the number of state-action pairs in the batch; the negative sign is because we usually minimize the loss during the optimization process, while the Q-value function is to be maximized. The action network parameters are updated through formula (32) to obtain a new strategy that meets the truncation requirements, σ A is the learning rate of the Actor network; 5) Critic network training. First, a batch of data is randomly sampled from the experience replay buffer, including state (s), action (a), reward (r), and next state (s'). Then, the Critic network is used to estimate the Q value of the current state-action pair: Q1(s,a; φ1) and Q2(s,a; φ2), where φ1 and φ2 are the parameters of the two Critic networks respectively. Then, the target Critic network is used to estimate the Q value of the next state-action pair: Q target' (s',a';φ target' ), the calculation of the target Q value involves the target Actor network and the target Critic network. The target Actor network is used to select the action in the next state s', while the target Critic network is used to estimate the Q value corresponding to this action. Then the mean square error is used to calculate the Critic loss function L(φ), and finally the gradient descent method is used to update the parameters φ of the Critic network. Q target' (s',a';φ target' )=Q target' (s',π'(s';θ target' );f target' ) (33) L(φ)=(yQ(s,a;φ)) 2 (34) y=r+γmini=1.2Q target,i' (s',π i' (s';θ target' );f target' ) (35) Formula (33) first selects action a' through the target Actor network π', and then uses this action and the target Critic network Q target' To estimate the target Q value, where a'=π'(s'; θ target' ) is the action selected by the target Actor network according to the state s′, θ target' is the parameter of the target Actor network, and the parameter of the target Critic network is φ target' ; Formula (34) is the loss function of the Critic network. This loss function is used to measure the difference between the Q value estimated by the current Critic network and the target Q value. Q(s, a; φ) represents the Q value estimated by the current Critic network under the given state s and action a using the parameter φ. y represents the target Q value, which is calculated based on the data sampled in the experience replay buffer. In formula (35), r is the immediate reward, γ is the discount factor, and mini=1,2 means taking the smaller value of the Q value estimated by the two target critic networks in order to reduce the overoptimistic tendency of the Q value estimation. target,i' and π i' They are the estimates of the target Critic network and the target Actor network respectively; Formula (36) uses the gradient descent method to update the size of the Critic network parameters φ, σ C is the learning rate of the Critic network.

6. A microgrid energy optimization scheduling method for source-grid-load-storage according to claim 5, characterized in that: The overall process of the microgrid optimization scheduling algorithm using the ITD3 algorithm proposed in step S4 is as follows: in the tth round of energy optimization management, the state s of the microgrid is t Input LSTM to extract time series features, and then send it to the ITD3 network. ITD3 includes two Actor networks and two Critic networks, all of which use a four-layer network structure. The first layer is the input layer, followed by two hidden layers, and finally the output layer. The policy network continuously interacts with the environment, collects the interaction data and temporarily stores it in the trajectory cache pool, and transmits the trajectory data to the policy network and evaluation network to update the network parameters. The updated network has more accurate value assessment and action selection performance. As the interaction between the intelligent agent and the environment gradually deepens, the policy network training gradually stabilizes until convergence. The specific steps are as follows: Step 1: Initialization: Set the initial state s0 of the microgrid and initialize the network parameters required by the TD3 algorithm, including two Actor networks and two Critic networks, as well as their respective target networks; Step 2: Data collection: In the tth round of energy optimization management, the state of the microgrid s t Input into the online Actor network, the two Actor networks each output an action, and the larger action a is selected t After executing this action, observe the next state s of the microgrid t+1 and the reward r t , will (s t ,a t ,r t ,s t+1 ) Such a quadruple is temporarily stored in the experience replay buffer pool; Step 3: Network training: Randomly extract a batch of four-tuple data from the experience replay buffer pool to train the online Actor network and Critic network. In the TD3 algorithm, the Critic network is responsible for estimating the state value function Q π (s, a), while the Actor network updates its strategy based on the gradient information provided by the Critic network and clips the loss using the importance sampling ratio clipping technique. During the training process, the target network (a copy of the online network with delayed parameter updates) is used to provide stable target values ​​to enhance the stability of the algorithm. Step 4: Delayed update: The parameters of the target network are not updated at every step, but are updated after a certain number of steps of online network training. A soft update of the online network parameters is adopted, using θ′=τθ+(1-τ)θ′, where θ is the parameter of the online network, θ′ is the parameter of the target network, and τ is a small constant; Step 5: Strategy improvement: As the online network is trained, the strategy will gradually improve. When the strategy becomes stable or reaches the preset number of training rounds, the algorithm can be considered to have converged. Step 6: Output strategy: When the algorithm converges, use the online Actor network as the final energy optimization management strategy and output corresponding actions according to the current state of the microgrid.

Citation Information

Patent Citations

  • Micro-grid space-time perception energy management method based on security deep reinforcement learning

    CN115731072A

  • Low-carbon dispatching method and system for cogeneration system based on deep reinforcement learning

    CN116914732A

  • Intelligent community energy optimization scheduling method and system based on reinforcement learning, and storage medium

    CN117172499A

  • Scheduling decision model establishment method based on SumTree-TD3 algorithm

    CN117291390A

  • Independent micro-grid multi-objective optimization scheduling method and system considering power supply reliability

    CN117639078A

Cited By

  • Advanced on-orbit system intelligent framing method based on near-end strategy optimization

    CN119450578A

  • Power dispatching optimization method and system based on reinforcement learning

    CN119647898A

  • Electric energy demand optimization operation method based on pelican optimization algorithm

    CN119671109A

  • Optimized dispatching method for electricity-hydrogen-heat comprehensive energy microgrid based on improved TD3 algorithm

    CN119787352A

  • Active power distribution network multi-device cooperation reactive power optimization control method and system

    CN119944717A