Micro-grid optimization operation method based on elastic step length depth Q network

Abstract

The invention discloses a micro-grid optimization operation method based on an elastic step length depth Q network. The micro-grid optimization operation method comprises the following steps: constructing a multi-objective optimization model taking the minimization of total operation cost and environmental cost as an objective; a micro-grid multi-objective optimization problem is converted into a Markov game process; a hidden feature clustering driving mechanism and a neighbor-based variable density clustering algorithm are combined, an elastic step length adjusting mechanism is designed, and the step length of deep reinforcement learning is adjusted according to state similarity so as to relieve the problem of Q value overestimation; and optimizing the micro-grid intelligent agent by using an ES-DQN method. According to the method provided by the invention, the state similarity is effectively judged by combining an elastic step length mechanism and deep reinforcement learning and utilizing clustering analysis, so that the optimization efficiency is improved while the problem of Q value over-estimation in the traditional DQN is relieved, and the micro-grid optimization scheduling is more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of microgrid optimization operation, and in particular to a microgrid optimization operation method based on an elastic step-length deep Q network. Background Art

[0002] With the rapid development of renewable energy technology and the increasing demand for environmental protection, microgrids, as a flexible energy management system that integrates distributed energy, energy storage systems and loads, have received widespread attention and application. Microgrids can operate in grid-connected or off-grid modes, and by optimizing the scheduling of their internal resources, they can achieve efficient energy utilization and minimize operating costs. However, due to the significant intermittency and volatility of renewable energy (such as photovoltaics and wind power), the optimized operation of microgrids faces huge challenges. Therefore, it is particularly important to optimize the operation of microgrids.

[0003] In the existing technical literature, document [1]: "Microgrid Optimization Operation Based on Improved Particle Swarm Algorithm" (Zhang Shaoming, Sheng Siqing. Microgrid Optimization Operation Based on Improved Particle Swarm Algorithm [J]. China Electric Power, 2020, 53(5): 24-31.) introduces elite reverse learning strategy and worst particle exclusion method to improve the particle swarm algorithm, and enables the microgrid to achieve better economic benefits. However, the use of intelligent algorithms to optimize the operation of microgrids is prone to fall into local optimal solutions.

[0004] Reference [2]: "Energy management strategy for wind, solar, diesel and storage microgrids based on deep Q network" (Liu Xiangjie, Liu Zian, Kong Xiaobing, et al. Energy management strategy for wind, solar, diesel and storage microgrids based on deep Q network [J]. Control Engineering, 2023, 30(08): 1538-1547.) Combining random planning with deep Q network improves the adaptability to random renewable energy output. Reference [3]: "A novel energy management method for networked multi-energy microgridsbased on improved DQN" (Xiao H, Pu X, Pei W, et al. A novel energy management method for networked multi-energy microgrids based on improved DQN [J]. IEEE Transactions on Smart Grid, 2023, 14(6): 4912-4926.) Combining k-cross sampling strategy and gated recurrent unit-temporal convolutional network to improve DQN effectively improves the efficiency of energy management and increases economic benefits.

[0005] However, the intelligent algorithms used in the above literature are prone to fall into local optimal solutions. The DQN-based method solves this problem to a certain extent, but the DQN-based method is prone to overestimation of the Q value, which leads to unstable convergence or unsatisfactory results. Summary of the invention

[0006] In view of the above shortcomings, the present invention discloses a method for optimizing the operation of a microgrid based on a deep Q network with an elastic step size. First, a multi-objective optimization model with the goal of minimizing the total operating cost and environmental cost is constructed in view of the economic and environmental protection requirements of the microgrid operation; secondly, in order to effectively solve the problem of optimizing the operation of the microgrid, the multi-objective optimization problem of the microgrid is converted into a Markov game process; then, a flexible step size adjustment mechanism is designed by combining the hidden feature clustering driving mechanism and the neighbor-based variable density clustering algorithm, and the step size of deep reinforcement learning is adjusted according to the state similarity to alleviate the problem of overestimation of the Q value; finally, the proposed ES-DQN method is used to optimize the microgrid intelligent body. The method proposed in the present invention can effectively improve the optimization quality and decision-making efficiency.

[0007] The technical solution adopted by the present invention is:

[0008] The microgrid optimization operation method based on the elastic step-length deep Q network includes the following steps:

[0009] Step 1: Construct a multi-objective optimization model with the goal of minimizing total operating cost and environmental cost;

[0010] Step 2: Convert the microgrid multi-objective optimization problem into a Markov game process;

[0011] Step 3: Combining the hidden feature clustering driving mechanism and the variable density clustering algorithm based on nearest neighbors, a flexible step adjustment mechanism is designed to adjust the step size of deep reinforcement learning according to state similarity to alleviate the problem of overestimation of Q value;

[0012] Step 4: Use the ES-DQN method to optimize the microgrid agent.

[0013] In the step 1, a multi-objective optimization model is constructed with the goal of minimizing the total operating cost and the environmental cost in view of the economic and environmental protection requirements of the microgrid operation;

[0014] (1): The objective function f1 is expressed as minimizing the total operating cost and environmental cost of the microgrid system. The total operating cost includes the operation and maintenance cost and fuel cost of the micro-turbine and energy storage system as well as the cost of interaction with the microgrid. The environmental cost is the conversion cost of pollutants generated when the power generation device in the microgrid outputs electricity. Since WT and PV are renewable energy generation, their costs are ignored.

[0015] f1=min(Cec +C en );

[0016]

[0017] Where: C ec is the total operating cost; C en is the environmental protection cost; T is the total number of optimal scheduling time periods, which is 24; t is the time period; The operation and maintenance cost of the micro gas turbine; The operation and maintenance cost of the energy storage system; is the fuel cost of the micro-turbine; C g is the interaction cost between microgrid and distribution network; I MT ,I ESS ,I FU are the collection of micro-turbine, energy storage system and fuel in the microgrid; k MT , k E are the unit electricity maintenance cost coefficients of micro-turbine and energy storage respectively; P MT,i (t), P B,i (t) are the outputs of the i-th micro-turbine and the energy storage system respectively; k c,i is the amount of fuel consumed per unit output of the micro-turbine; η i is the power generation efficiency of the micro-turbine; ρ in (t),ρ out (t) are the electricity price of the microgrid purchased from the distribution network and the electricity price of the microgrid sold to the distribution network during period t; P in (t), P out (t) are the power purchased from the distribution network and the power sold to the distribution network by the microgrid during period t; M is the type of pollutant: NO X , CO2, CO, SO2, where subscript X = 1, 2; Y j (t) is the amount of the jth pollutant in period t; D j (t) is the conversion cost of the jth pollutant in time period t; Δt represents the interval of each time step, which is used to calculate the cost incurred in each time step;

[0018] (2): The multi-objective optimization model of the microgrid needs to consider power balance, upper and lower limits of equipment operation, energy storage operation constraints, etc.

[0019] 1) Power balance constraints:

[0020] P L (t) = P WT (t)+P PV (t)+P MT (t)+P B (t)+P Grid (t);

[0021] Where: P L (t) is the load power during period t; P WT (t), P PV (t), P MT (t), P B (t), P Grid (t) are the power of wind turbine, photovoltaic, micro-turbine, energy storage system and the power interacting with the distribution network during period t.

[0022] 2) Micro turbine operation constraints:

[0023] P MT,min P MT (t)P MT,max ;

[0024] Where P MT,min and P MT,max They are respectively the upper and lower limits of the active output of the micro gas turbine.

[0025] 3) Constraints on electricity purchase and sales prices:

[0026] The electricity purchase and sale price ρ(t) between the microgrid and the distribution network satisfies the following relationship:

[0027] ρ out (t)<ρ(t)<ρ in (t);

[0028] Among them, ρ in (t),ρ out (t) are the electricity purchase and sales prices of the microgrid and distribution network during period t.

[0029] 4) Energy storage operation constraints:

[0030] 0≤|P B (t)∣≤P B,max ;

[0031] Where P B,max is the maximum charge and discharge power.

[0032] The state of charge of the energy storage system is an important decision variable to prevent overcharging or over-discharging of the energy storage. The value is the ratio of the remaining dischargeable capacity of the energy storage to its rated capacity.

[0033]

[0034] b SOC,min ≤ b SOC (t)≤b SOC,max ;

[0035] Where b SOC (t+1) means t+1 Time storage charge state; bSOC (t) is the energy storage charge state during period t; η N is the energy storage charging and discharging efficiency; B is the rated capacity of energy storage; b SOC,max and b SOC,min They are the upper and lower limits of the energy storage state of charge respectively.

[0036] In step 2, in order to effectively solve the problem of microgrid optimization operation, the multi-objective optimization problem of the microgrid is converted into a Markov game process, which specifically includes:

[0037] (1) State space description:

[0038] The information provided by the environment to the microgrid intelligent agent includes the state of charge SOC(t) of the energy storage, the system load power P L (t), the electricity purchase and sale price between the microgrid and the distribution network ρ in (t),ρ out (t) and the power of photovoltaic and wind turbines. Therefore, the state space is expressed as:

[0039] s=[SOC(t),P L (t),ρ in (t),ρ out (t),P PV (t),P WT (t)];

[0040] Where s represents the state space; P PV (t) represents the photovoltaic output power; P WT (t) represents the fan output power.

[0041] (2) Action space description:

[0042] The dispatching action of the microgrid consists of the power output related to the flexible dispatching unit, which includes the micro-turbines and the energy storage device; its action space is expressed as:

[0043] a=[P MT (t),P ESS (t),P Grid (t)];

[0044] Where a represents the action space; P MT (t) represents the output power of the micro-turbine; P ESS (t) represents the charging / discharging power of the energy storage system; P Grid (t) represents the power of interaction between the microgrid and the distribution network.

[0045] Since the ES-DQN adopted in the present invention is a deep reinforcement learning algorithm based on the discretized actions of DQN, the above three actions are discretized as follows:

[0046] P MT (t)∈{30,60,90,120,150,180}

[0047] The above values ​​represent the discrete values ​​of the micro-turbine output power in time period t, that is, the output power that can be selected when scheduling the output of the micro-turbine, in kilowatts (kW).

[0048] P ESS (t)∈{-200,-150,0,150,200}

[0049] The above values ​​represent the discrete values ​​of the energy storage system's charge / discharge power in time period t, that is, the output power can be selected when scheduling the output of the energy storage system. Positive values ​​represent the energy storage system's discharge, and negative values ​​represent the energy storage system's charge, in kilowatts (kW).

[0050] P Grid (t)∈{-300,-150,-90,60,0,60,90,120,150,300}

[0051] The above values ​​represent the discrete values ​​of the interactive power between the microgrid and the distribution network in time period t, that is, the power can be selected when scheduling the interactive power between the microgrid and the distribution network. The positive value indicates that the microgrid purchases power from the distribution network, and the negative value indicates that the microgrid sells power to the distribution network, in kilowatts (kW).

[0052] (3) Reward function construction:

[0053] In the microgrid state t Next, the agent performs action a t After that, the reward r is obtained; the purpose of multi-objective optimization of microgrid is to minimize the total operating cost and environmental cost, so the reward function can be expressed as:

[0054] r=-(C ec +C en )

[0055] In the formula, r represents the reward function; C ec represents the operating cost of the microgrid; C en Environmental protection costs.

[0056] The step 3 comprises:

[0057] (1) Design a hidden feature clustering driving mechanism:

[0058] The method proposed in this invention constructs a hidden feature clustering driving mechanism in DQN, which uses the output features of the hidden layer of the deep neural network as the input of the unsupervised clustering algorithm instead of the original state value, such as Figure 2As shown, the details are as follows:

[0059] First, the original data (i.e. Figure 2 The input state in the input layer) is then input into the input layer, and then the high-quality feature information is captured through the nonlinear mapping and transformation of the hidden layer, and the feature information is used as the input of the subsequent NNVDC clustering algorithm, thereby driving the NNVDC clustering algorithm to perform clustering analysis, instead of the most original data (i.e. Figure 2 The input state in ) is used as the input of the NNVDC clustering algorithm.

[0060] Functional modules and calculation formulas involved:

[0061] 1. Input layer: The input layer is used to receive external input data, which refers to the input state in the figure, and pass it to the hidden layer.

[0062] 2. Hidden layer: The hidden layer processes the input data through nonlinear mapping and transformation to capture complex state-space relationships and high-dimensional feature information, thereby generating features that are more representative than the original state, and uses the above-generated features as the input of the NNVDC clustering algorithm, thereby driving the clustering algorithm to perform clustering analysis.

[0063] In feedforward networks, nonlinear mapping and transformation are usually implemented through activation functions. The present invention uses the ReLU activation function, which introduces nonlinearity and enables the network to learn complex features in the input data. The output features of the hidden layer are:

[0064] Q h (s)=ReLU(WiA+b)=max(0,WiA+b)

[0065] Among them, the expression of the ReLU function is ReLU(x)=max(0,x); Q h (s) represents the output features of the hidden layer; W represents the weight parameter matrix; A represents the input feature vector, which contains all the information of the input state; b represents the bias vector, which is an additional parameter in the neural network, used to adjust the output of the neuron so that the neural network can learn more complex functional relationships.

[0066] The hidden layer features are nonlinearly mapped and transformed by the feedforward network, which can capture complex state space relationships and high-dimensional feature information, thereby generating features with more expressive power than the original state.

[0067] The clustering results of the original state and hidden layer output as the input of the unsupervised clustering algorithm were compared in classic reinforcement learning tasks such as Cartpole, Acrobot and Mountain Car. The results show that the proposed hidden feature clustering driving mechanism significantly improves the training efficiency and convergence performance.

[0068] (2) Variable density clustering algorithm based on nearest neighbor:

[0069] The density clustering algorithm DBSCAN was proposed by Ester et al. in 1996. It uses global parameters and therefore cannot handle clusters of variable density. In order to solve this problem, the present invention proposes a nearest neighbor variable density clustering method (NNVDC), which is based on DBSCAN and uses k-nearest neighbor (kNN) distance to identify clusters of variable density in the data set. This method can accurately capture state similarity in complex operation scenarios of microgrids and effectively improve the accuracy and efficiency of scheduling optimization.

[0070] The core of the proposed method is to replace the distance-based density concept in DBSCAN with k-nearest neighbor distance, namely, ε, where ε represents a key parameter in the DBSCAN algorithm, which represents the neighborhood range of a point and is used to determine the density of a point.

[0071] And in each iteration, a set of clusters is identified;

[0072] In the algorithm, two points o i and j The distance between them can be defined as dist(o i ,o j ), o i represents the i-th point in the data set D; o j The jth point in the data set D. Among them, o i ,o j ∈D, D represents the set of points in the data set, and the present invention uses Euclidean distance as the distance metric for all data sets.

[0073] The neighbor-based variable density clustering algorithm relies on the following definition to form clusters;

[0074] 1) Click o i The set of k-nearest neighbors is defined as in, Indicates that point o is removed from the data set D i A subset of the set formed later; D represents the data set; {o i} represents the set consisting of the i-th point in the data set.

[0075] and meet and In the formula, k Indicates the size of the collection; express It is the dataset D with o removed i A subset of itself; Indicates o m is a point in dataset D, but it does not belong to Collection, not point o i ;dist(o i ,o j )≤dist(o i ,o m )express From point to o i The distance between o and other points in the data set is less than or equal to i distance, ensuring Contains i Recent k Points; i represents the i-th point in the data set D; o j represents the jth point in the data set D; o m The dataset D represents the set All points in and point o i A point in the resulting set.

[0076] 2) Click o i The k-nearest neighbor distance is defined as o i The maximum distance in the set of k-nearest neighbors is mathematically defined as:

[0077]

[0078] In the formula, NND(o i ) is the k-nearest neighbor distance, indicating o i In the k-nearest neighbor set of i The farthest point from o i distance.

[0079] 3) Define the global average density through k-nearest neighbor distance ( AVG ) and variance ( NNVariance ), as a reference to distinguish high-density from low-density areas:

[0080]

[0081] In the formula, A VG represents the average k-nearest neighbor distance of all points in the data set; n represents the total number of points in the data set;NNVariance Represents the variance of the k-nearest neighbor distances of all points in the data set, which is used to measure the dispersion of the k-nearest neighbor distances.

[0082] 4) The minimum k-nearest neighbor distance, a reference value for detecting outliers, is defined as:

[0083]

[0084] In the formula, MinimumNND represents the minimum k-nearest neighbor distance.

[0085] 5) Cluster initialization and expansion mechanism

[0086] If you click o i The k-nearest neighbor distance is less than or equal to the global average distance A VG , then i is the starting point of the cluster; if o i is the starting point, and point o j In o i Within the k-nearest neighbor range, then o j for o i neighbors; if o j Yes i Neighbors of i is the starting point, then o j From i Directly reachable; if point o p There is a path connecting to the origin point o i , and all points on the path are directly reachable, then o p From i Density can reach.

[0087] 6) Cluster merging and structure definition:

[0088] If you click o p and q From the starting point o i The density can reach, then o p and q Density connected; a cluster is composed of all mutually reachable and density connected points; if point o p is a density-reachable point but not a starting point, it is defined as a boundary point; if point o p If there are less than k points in the neighborhood of p It is a noise point.

[0089] The above sections 1) to 4) define the dynamic k-nearest neighbor density, which enables NNVDC to adapt to the changes in local density in the data set, thereby detecting clusters with different density distributions. At the same time, using the global indicator ( AVG and NNVariance ), realizes the dynamic division of state density and accurately captures the similarities and changes between states.

[0090] The above two designs of parts 5) and 6) enable NNVDC to accurately detect and expand clusters in complex data distribution, which is particularly suitable for dynamic environments such as microgrid optimization operation.

[0091] (3) DQN algorithm based on elastic step size:

[0092] The main goal of ES-DQN is to integrate the ideas of coarse Q-learning and multi-step DQN. The algorithm will use the special properties of both to reduce overestimation and improve the overall performance of DQN. The proposed algorithm ES-DQN chooses whether to guide a specific step during each time step based on the similarity between the current state and the next state of the agent. The central idea is that similar states can be aggregated into a single update, while different states should be updated separately;

[0093] After the introduction of elastic step length, the multi-step cumulative reward R t It can be expressed as:

[0094]

[0095] In the formula, γ is the discount factor; d is the step length; r t+a+1 For immediate rewards.

[0096] The step 4 comprises the following steps:

[0097] Step 4.1: Initialize the step counter d = 0, the sampling value u, the maximum training round M, the maximum time step C; and the target network update interval; initialize the weight θ of the main network Q, the target network The weight θ - =θ, and an experience replay buffer with capacity N Initialize the state memory B (with a capacity of H) and the nearest neighbor-based variable density clustering algorithm for clustering;

[0098] Step 4.2: The microgrid agent interacts with the environment and generates a batch of experience replay buffers The corresponding storage For each experience in , calculate its hidden layer output in the main network Q and store these hidden layer outputs in B.

[0099] Step 4.3: Model training:

[0100] ①. Reset the environment and observe the initial state;

[0101] ②. For each time step t to T, perform the following steps:

[0102] Use the greedy strategy to select action a t ; Perform action a in the environment t , observe the next state s t+1 and instant rewards t ; Set the state s t and t+1 Input to the hidden layer Q h Extract the output and store it in B; randomly sample from B with sampling value u as interval; cluster the sampled data using NNVDC and generate Q h (s t ) and Q h (s t+d+1 ) assign cluster labels; if the cluster labels are different: transfer the state (s t ,a t ,R t ,s t+d+1 ,γ d ) is stored in D, and d is set to zero, s t+d+1 represents the state at time step t+d+1; γ d represents the d-th power of the discount factor, d is the step counter in step 4.1; if the cluster labels are the same, update the cumulative reward R t =γ d r t+d+1 +R t , r t+d+1 Represents the reward at time step t+d+1; and increases the step counter d=d+1;

[0103] ③. Randomly sample small batches of data from D for training; for the terminal state s t+d+1 : The target value is y t+d+1 =R t ; For non-terminal state s t+d+1 : The target value is Indicates that in state s t+d+1 Take action a t The corresponding Q value.

[0104] ④. Update the main network weights θ using gradient descent.

[0105] ⑤. Update the target network weight θ according to the target network update interval - =θ.

[0106] Step 4.4: Determine whether the maximum number of training times has been reached. If not, return to step ② of step 4.3 until the maximum number of rounds M is reached, otherwise terminate the training.

[0107] Step 4.5: Use the trained model to optimize the microgrid multi-objective optimization problem online and obtain the optimal operation plan for the microgrid.

[0108] The present invention provides a microgrid optimization operation method based on elastic step-length deep Q network, and the technical effects are as follows:

[0109] 1) Step 1 of the present invention constructs a multi-objective optimization model with the goal of minimizing the total operating cost and environmental cost. It can systematically analyze and optimize the microgrid operating cost from an economic perspective, including the cost of power generation, energy storage, etc., to reduce energy waste, and it can also consider environmental costs, quantify and reduce carbon dioxide and other pollutant emissions. This reflects the dual focus on economic benefits and environmental benefits, and is in line with the concept of sustainable development.

[0110] 2) The advantage of converting the multi-objective optimization problem of the microgrid into a Markov game process in step 2 of the present invention is that it can provide a framework to simultaneously deal with the cost-effectiveness and environmental impact of microgrid operations, optimize resource allocation by simulating the decision-making behavior of intelligent agents in uncertain environments, improve the efficiency and sustainability of energy management, and take into account the dynamic and real-time requirements of the system. Different from the existing method of converting multi-objective optimization problems into single-objective problems and then applying single-objective optimization methods to solve them, this method ignores the trade-offs between different objectives, resulting in the solution being unable to fully reflect the complexity of the problem. Step 2 converts the multi-objective optimization problem into a Markov game process. On the one hand, the Markov game process can handle multiple objectives at the same time, rather than simplifying them into a single objective, so that the trade-offs between different objectives can be considered more comprehensively. On the other hand, the uncertainty and changes in the microgrid can be dynamically considered, so that the decision-making process can adapt to real-time changing environmental conditions. In addition, step 2 also provides convenience for solving using the ES-DQN algorithm.

[0111] 3) In step 3 of the present invention, hidden feature clustering driving mechanism: The core component of ES-DQN is to use NNVDC to assign cluster labels to states. High-quality features usually play an important role in the quality of the generated clusters. Therefore, we propose a hidden feature clustering driving mechanism. In traditional clustering methods, the original input state is directly used as the feature input. This method is often difficult to capture the deep features of the data, which will lead to unsatisfactory clustering effects and affect the accuracy of state classification. Different from the above method, the proposed method extracts the hidden layer output as the feature input of the NNVDC clustering algorithm, which can reveal the deeper features of the data, thereby significantly improving the quality of clustering and the accuracy of state classification.

[0112] Nearest Neighbor-Based Variable Density Clustering (NNVDC): The NNVDC algorithm is able to detect clusters of different densities, shapes, and sizes, even in the presence of noise. This enables the algorithm to accurately capture state similarities and improve the accuracy and efficiency of scheduling optimization when dealing with complex operating scenarios of microgrids. Unlike traditional density clustering algorithms such as DBSCAN, NNVDC only requires one input parameter (k value), which greatly simplifies the parameter selection process and makes the algorithm easier to use and adjust.

[0113] Flexible step-size adjustment mechanism: Combining the hidden feature clustering driving mechanism and the neighbor-based variable density clustering algorithm (NNVDC), first, the hidden feature clustering driving mechanism is used to obtain high-quality state features, and then the obtained high-quality state features are input into the NNVDC clustering algorithm. The NNVDC clustering algorithm is used to systematically distinguish the similarities between states. Finally, the step size is dynamically adjusted according to the similarity of the states. Unlike traditional multi-step update methods that usually rely on fixed step size parameters for updates, when faced with complex and changing environments, the fixed step size has a weak ability to adapt to the dynamic characteristics of the environment, resulting in poor training results. The proposed flexible step size adjustment mechanism dynamically adjusts the step size according to state similarity through a clustering algorithm, which enables the agent to flexibly select the appropriate step size according to environmental changes, better capture key information in the environment, and avoid falling into local optimality, thereby improving the stability and efficiency of training.

[0114] 4) In step 4 of the present invention, the operating environment of the microgrid is complex and changeable, and contains a variety of energy components (such as solar energy, wind energy, energy storage equipment, etc.), and its power generation and load demand will fluctuate greatly with time, weather and other factors. The proposed ES-DQN method is used to train the microgrid intelligent agent, which effectively avoids the decision-making errors caused by the traditional method under a fixed step size, significantly improves the decision-making accuracy of the intelligent agent in a complex dynamic environment, and ensures that the operation of the microgrid is always developing in the optimal direction. For example, when the solar power generation power fluctuates drastically due to weather changes, the intelligent agent can quickly adjust the step size and select the appropriate energy management strategy based on the comparison between the current state and the historical similar state, such as adjusting the charging and discharging of the energy storage equipment and the output of the micro-turbine; thereby more accurately responding to complex dynamic environments and improving the accuracy and efficiency of decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] The present invention will be further described below in conjunction with the accompanying drawings and examples;

[0116] Figure 1 It is a flow chart of the microgrid optimization operation method based on elastic step-size deep Q network.

[0117] Figure 2 It is a network structure diagram of the hidden feature clustering driving mechanism in the present invention.

[0118] Figure 3 It is a microgrid optimization dispatch diagram based on elastic step-size deep Q network. DETAILED DESCRIPTION

[0119] Figure 1 The flowchart of the microgrid optimization operation method based on the elastic step-size deep Q network. First, in view of the economic and environmental protection requirements of the microgrid operation, a multi-objective optimization model with the goal of minimizing the total operating cost and environmental cost is constructed. Secondly, in order to effectively solve the problem of microgrid optimization operation, the multi-objective optimization problem of the microgrid is converted into a Markov game process. Then, a flexible step-size adjustment mechanism is designed by combining the hidden feature clustering driving mechanism and the neighbor-based variable density clustering algorithm, and the step size of deep reinforcement learning is adjusted according to the state similarity to alleviate the problem of overestimation of Q value. Finally, the microgrid intelligent body is optimized based on the method proposed in the present invention. The method proposed in the present invention can effectively improve the optimization quality and decision-making efficiency.

[0120] Figure 2 The network structure diagram of the hidden feature clustering driving mechanism in this invention. The core component of ES-DQN is to assign cluster labels to states using NNVDC. High-quality features usually play an important role in the quality of the generated clusters. Therefore, we design to extract the hidden layer output as the feature input of the unsupervised algorithm.

[0121] Figure 3 It is a microgrid optimization dispatch diagram based on elastic step-size deep Q network. Figure 3 The power distribution of each distributed energy source in the microgrid system based on the proposed optimization scheduling method within 24 hours a day is demonstrated. The charge state of the energy storage is set to be the same at the beginning and end of the day, and is 0. In the time period of 1:00-6:00, the only new energy source is wind power output, which cannot meet the load demand. At this time, the micro-gas turbine output is dispatched and the shortfall is purchased from the power grid to meet the electricity demand. During the period of 7:00-18:00, both photovoltaic and wind power are output. Around 13:00 is the time of the day when the output of new energy is the strongest. At this time, the output of new energy is enough to meet the electricity demand. At this time, the surplus electricity is used to charge the energy storage or sell it to the main power grid. In the peak period of electricity consumption from 19:00 to 20:00, the electricity price is high, and the energy storage makes a profit by discharging, thereby reducing costs. In the time period of 21:00-24:00, the electricity demand is met by the output of the micro-gas turbine and the purchase of electricity from the power grid.

[0122] Table 1 Comparison of optimized operation costs of microgrids based on different algorithms

[0123]

[0124] Table 1 is a comparison of the optimized operation costs of microgrids based on different algorithms. The present invention designs three benchmark methods for comparative analysis.

[0125] Benchmark 1: Microgrid Optimization Operation Method Based on DQN:

[0126] Benchmark 2: Microgrid Optimal Operation Method Based on Average DQN;

[0127] Benchmark 3: Microgrid optimization operation method based on ES-DQN.

[0128] As can be seen from Table 1, the total cost of the ES-DQN algorithm is the lowest at 2636.819 yuan, which is 12.27% lower than the total cost of the DQN algorithm; 9.64% lower than the Average DQN. The operating cost and environmental protection cost of the proposed method are 2291.8584 yuan and 344.961 yuan respectively, which are significantly lower than those of DQN and Average DQN, indicating that it performs better in reducing energy use and pollution emissions. This is because the method proposed in the present invention combines the elastic step size mechanism with deep reinforcement learning, and uses cluster analysis to effectively judge state similarity, which alleviates the problem of Q value overestimation in traditional DQN while improving the optimization efficiency, making the microgrid optimization scheduling more accurate.

Claims

1. A microgrid optimization operation method based on elastic step-length deep Q network, characterized by The following steps are involved: Step 1: Construct a multi-objective optimization model with the goal of minimizing total operating cost and environmental cost; Step 2: Convert the microgrid multi-objective optimization problem into a Markov game process; Step 3: Combining the hidden feature clustering driving mechanism and the variable density clustering algorithm based on nearest neighbors, a flexible step adjustment mechanism is designed to adjust the step size of deep reinforcement learning according to state similarity to alleviate the problem of overestimation of Q value; Step 4: Use the ES-DQN method to optimize the microgrid agent.

2. The microgrid optimization operation method based on elastic step-length deep Q network according to claim 1 is characterized in that: In the step 1, a multi-objective optimization model is constructed with the goal of minimizing the total operating cost and the environmental cost, based on the economic and environmental protection requirements of the microgrid operation; including the objective function f1; The objective function f1 is expressed as minimizing the total operating cost and environmental cost of the microgrid system. The total operating cost includes the operation and maintenance cost and fuel cost of the micro-turbine and energy storage system, as well as the cost of interaction with the microgrid. The environmental cost is the conversion cost of pollutants generated when the power generation device in the microgrid outputs electricity. f1=min(C ec +C en ); Where: C ec is the total operating cost; C en is the environmental protection cost; T is the total number of optimal scheduling time periods; t is the time period; The operation and maintenance cost of the micro gas turbine; The operation and maintenance cost of the energy storage system; is the fuel cost of the micro-turbine; C g is the interaction cost between microgrid and distribution network; I MT ,I ESS ,I FU are the collection of micro-turbine, energy storage system and fuel in the microgrid; k MT , k E are the unit electricity maintenance cost coefficients of micro gas turbine and energy storage respectively; P MT,i (t), P B,i (t) are the outputs of the i-th micro-turbine and the energy storage system respectively; k c,i The amount of fuel consumed by the micro-turbine per unit output; η i is the power generation efficiency of the micro-turbine; ρ in (t),ρ out (t) are the electricity price of the microgrid purchased from the distribution network and the electricity price of the microgrid sold to the distribution network during period t; P in (t), P out (t) are the power purchased from the distribution network and the power sold to the distribution network by the microgrid during period t; M is the type of pollutant: NO X , CO2, CO, SO2, where subscript X = 1, 2; Y j (t) is the amount of the jth pollutant in period t; D j (t) is the conversion cost of the j-th pollutant in period t; Δt represents the interval of each time step, which is used to calculate the cost incurred in each time step.

3. The microgrid optimization operation method based on elastic step-length deep Q network according to claim 2 is characterized in that: The multi-objective optimization model of the microgrid considers power balance, upper and lower limits of equipment operation, and energy storage operation constraints; 1) Power balance constraints: P L (t)=P WT (t)+P PV (t)+P MT (t)+P B (t)+P Grid (t); Where: P L (t) is the load power during period t; P WT (t), P PV (t), P MT (t), P B (t), P Grid (t) are the power of wind turbine, photovoltaic, micro-turbine, energy storage system and the power interacting with the distribution network during period t; 2) Micro turbine operation constraints: P MT,min ≤P MT (t)≤P MT,max ; Where P MT,min and P MT,max They are the upper and lower limits of the active output of the micro-turbine respectively; 3) Constraints on electricity purchase and sales prices: The electricity purchase and sale price ρ(t) between the microgrid and the distribution network satisfies the following relationship: ρ out (t)<ρ(t)<ρ in (t); Among them, ρ in (t),ρ out (t) are the electricity purchase and sales prices of the microgrid and distribution network in period t; 4) Energy storage operation constraints: 0≤∣P B (t)∣≤P B,max ; Where P B,max is the maximum charge and discharge power; The state of charge value of the energy storage system is the ratio of the remaining dischargeable capacity of the energy storage to its rated capacity; b SOC,min ≤b SOC (t)≤b SOC,max ; Where b SOC (t+1) means t+1 Energy storage charge state during the period; b SOC (t) is the energy storage charge state during period t; η N is the energy storage charging and discharging efficiency; B is the rated capacity of energy storage; b SOC,max and b SOC,min They are the upper and lower limits of the energy storage state of charge respectively.

4. The microgrid optimization operation method based on elastic step-length deep Q network according to claim 1 is characterized in that: In step 2, in order to effectively solve the problem of microgrid optimization operation, the multi-objective optimization problem of the microgrid is converted into a Markov game process, which specifically includes: (1) State space description: The information provided by the environment to the microgrid intelligent agent includes the state of charge SOC(t) of the energy storage, the system load power P L (t), the electricity purchase and sale price between the microgrid and the distribution network ρ in (t),ρ out (t) and the power of photovoltaic and wind turbines; therefore, the state space is expressed as: s=[SOC(t),P L (t),ρ in (t),ρ out (t),P PV (t),P WT (t)]; Where s represents the state space; P PV (t) represents the photovoltaic output power; P WT (t) represents the fan output power; (2) Action space description: The dispatching action of the microgrid consists of the power output related to the flexible dispatching unit, which includes the micro-turbines and the energy storage device; its action space is expressed as: a=[P MT (t),P ESS (t),P Grid (t)]; Where a represents the action space; P MT (t) represents the output power of the micro-turbine; P ESS (t) represents the charging / discharging power of the energy storage system; P Grid (t) represents the power of interaction between the microgrid and the distribution network; (3) Reward function construction: In the microgrid state t Next, the agent performs action a t After that, the reward r is obtained; the purpose of multi-objective optimization of microgrid is to minimize the total operating cost and environmental cost, so the reward function is expressed as: r=-(C ec +C en ) In the formula, r represents the reward function; C ec represents the operating cost of the microgrid; C en Environmental protection costs.

5. The microgrid optimization operation method based on elastic step-length deep Q network according to claim 1 is characterized in that: The step 3 comprises: (1) Design a hidden feature clustering driving mechanism: A hidden feature clustering driving mechanism is constructed in DQN, which uses the output features of the hidden layer of the deep neural network as the input of the unsupervised clustering algorithm instead of the original state value; specifically, it includes: Input layer: The input layer is used to receive external input data and pass it to the hidden layer; Hidden layer: The hidden layer processes the input data through nonlinear mapping and transformation to capture complex state space relationships and high-dimensional feature information, thereby generating features with more representation capabilities than the original state, and uses the generated features as the input of the NNVDC clustering algorithm, thereby driving the clustering algorithm to perform clustering analysis; In the feedforward network, nonlinear mapping and transformation are realized through activation functions. The ReLU activation function is used. The ReLU activation function introduces nonlinearity, which enables the network to learn the complex features in the input data. The output features of the hidden layer are: Q h (s)=ReLU(WiA+b)=max(0,WiA+b) Among them, the expression of the ReLU function is ReLU(x)=max(0,x); Q h (s) represents the output features of the hidden layer; W represents the weight parameter matrix; A represents the input feature vector, which contains all the information of the input state; b represents the bias vector, which is an additional parameter in the neural network and is used to adjust the output of the neuron so that the neural network can learn more complex functional relationships; The hidden layer features are nonlinearly mapped and transformed by the feedforward network, which can capture complex state space relationships and high-dimensional feature information, thereby generating features with more expressive power than the original state; (2) Neighbor-based variable density clustering method NNVDC: The algorithm is based on DBSCAN and uses k-nearest neighbor distance to identify clusters of variable density in the data set. The k-nearest neighbor distance replaces the distance-based density concept in DBSCAN, i.e., ε, which represents a key parameter in the DBSCAN algorithm. It represents the neighborhood range of a point and is used to determine the density of a point. A set of clusters are identified in each iteration. In the algorithm, two points o i and j The distance between them is defined as dist(o i ,o j ), o i represents the i-th point in the data set D; o j The jth point in the data set D; where o i ,o j ∈D, D represents the set of points in the data set, and the Euclidean distance is used as the distance metric for all data sets; The neighbor-based variable density clustering algorithm relies on the following definition to form clusters; 1) Click o i The set of k-nearest neighbors is defined as in, Indicates that point o is removed from the data set D i A subset of the set formed later; D represents the data set; {o i } represents the set consisting of the i-th point in the data set; and satisfies and In the formula, k Indicates the size of the collection; express It is the dataset D with o removed i A subset of itself; Indicates o m is a point in dataset D, but it does not belong to Collection, not point o i ;dist(o i ,o j )≤dist(o i ,o m )express From point to o i The distance between o and other points in the data set is less than or equal to i distance, ensuring Contains o i The nearest k points; o i represents the i-th point in the data set D; o j represents the jth point in the data set D; o m The dataset D represents the set All points in and point o i A point in the set obtained later; 2) Click o i The k-nearest neighbor distance is defined as o i The maximum distance in the set of k-nearest neighbors is mathematically defined as: In the formula, NND(o i ) is the k-nearest neighbor distance, indicating o i In the k-nearest neighbor set of i The farthest point from o i distance; 3) Define the global average density through k-nearest neighbor distance ( AVG ) and variance ( NNVariance ), as a reference to distinguish high-density from low-density areas: In the formula, AVG represents the average k-nearest neighbor distance of all points in the data set; n represents the total number of points in the data set; NNVariance Represents the variance of the k-nearest neighbor distances of all points in the data set, which is used to measure the dispersion of the k-nearest neighbor distances; 4) The minimum k-nearest neighbor distance, a reference value for detecting outliers, is defined as: In the formula, MinimumNND represents the minimum k-neighbor distance; 5) Cluster initialization and expansion mechanism: If you click o i The k-nearest neighbor distance is less than or equal to the global average distance AVG , then i is the starting point of the cluster; if o i is the starting point, and point o j In o i Within the k-nearest neighbor range, then o j for o i neighbors; if o j Yes i Neighbors of i is the starting point, then o j From i Directly reachable; if point o p There is a path connecting to the origin point o i , and all points on the path are directly reachable, then o p From i Density can reach; 6) Cluster merging and structure definition: If you click o p and q From the starting point o i The density can reach, then o p and q Density connected; a cluster is composed of all mutually reachable and density connected points; if point o p is a density-reachable point but not a starting point, it is defined as a boundary point; if point o p If there are less than k points in the neighborhood of p It is a noise point; (3) DQN algorithm based on elastic step size: ES-DQN chooses whether to guide a specific step during each time step based on the similarity between the current state the agent is in and the next state; After the introduction of elastic step length, the multi-step cumulative reward R t It is expressed as: In the formula, γ is the discount factor; d is the step size; r t+a+1 For immediate rewards.

6. The microgrid optimization operation method based on elastic step-length deep Q network according to claim 5 is characterized in that: The step 4 comprises the following steps: Step 4.1: Initialize the step counter d = 0, the sampling value u, the maximum training round M, the maximum time step C; and the target network update interval; initialize the weight θ of the main network Q, the target network The weight θ - =θ, and the capacity is N Experience Replay Buffer Initialize the state memory B and the nearest neighbor-based variable density clustering algorithm for clustering; Step 4.2: The microgrid agent interacts with the environment and generates a batch of experience replay buffers The corresponding storage For each experience in, calculate its hidden layer output in the main network Q and store these hidden layer outputs in B; Step 4.3: Model training: ①. Reset the environment and observe the initial state; ②. For each time step t to T, perform the following steps: Use the greedy strategy to select action a t ; Perform action a in the environment t , observe the next state s t+1 and instant rewards t ; Set the state s t and t+1 Input to the hidden layer Q h Extract the output and store it in B; randomly sample from B with sampling value u as interval; cluster the sampled data using NNVDC and generate Q h (s t ) and Q h (s t+d+1 ) assign cluster labels; if the cluster labels are different: transfer the state (s t ,a t ,R t ,s t+d+1 ,γ d ) is stored in D, and d is set to zero, s t+d+1 represents the state at time step t+d+1; γ d represents the d-th power of the discount factor, d is the step counter in step 4.1; if the cluster labels are the same, update the cumulative reward R t =γ d r t+d+1 +R t , r t+d+1 represents the reward at time step t+d+1; and increases the step counter d=d+1; ③. Randomly sample small batches of data from D for training; for the terminal state s t+d+1 : The target value is y t+d+1 =R t ; For non-terminal state s t+d+1 : The target value is Indicates that in state s t+d+1 Take action a t The corresponding Q value; ④. Update the main network weight θ using gradient descent; ⑤. Update the target network weight θ according to the target network update interval - =θ; Step 4.4: Determine whether the maximum number of training times has been reached. If not, return to step ② of step 4.3 until the maximum number of rounds M is reached, otherwise terminate the training; Step 4.5: Use the trained model to optimize the microgrid multi-objective optimization problem online and obtain the optimal operation plan for the microgrid.

Images