Multi-objective security optimization method for microgrid energy system based on deep reinforcement learning
Through deep reinforcement learning and particle swarm optimization technology, the multi-target problems of the microgrid energy system are decomposed and optimized, and the economic, environmental protection and safe operation of the microgrid is achieved, and the problems that are difficult to meet the multi-target needs in the existing technology are solved.
Patent Information
- Application Number
- CN202210662643.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-13
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-06-13
AI Technical Summary
The existing microgrid energy system optimization methods are difficult to achieve economical, environmentally friendly and safe simultaneously, and cannot effectively meet multi-target needs.
Adopt multi-objective security optimization method based on deep reinforcement learning, using DDPG algorithm and PSO algorithm, and promote solutions by decomposing multi-objective problems into scalar problems, and through penalizing factor optimization strategies, combining deep deterministic strategy gradient algorithm and particle swarm optimization, Approaching the real Pareto frontier.
The economic, environmentally friendly and safe operation of the microgrid energy system has been achieved, and the efficiency and accuracy of multi-objective optimization scheduling have been improved.
Smart Images

Figure CN114897266B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of power system automation, and specifically relates to a multi-objective security optimization method for a microgrid energy system based on deep reinforcement learning. Background Art
[0002] With the continuous advancement of power grid technology, microgrids have gradually become complex independent systems. The main difference between them and large power grids is that they can operate as independent systems or be connected to large power grids for coordinated operation.
[0003] ZL2021109555161 discloses an optimization scheduling method for isolated microgrids based on multi-period prediction of automatic reinforcement learning. First, an advanced prediction model PER-AutoRL is proposed, and a multi-period single-step prediction method based on PER-AutoRL is designed. In addition, the influence of prediction error on prediction accuracy is considered. The prediction value is corrected according to the prediction error distribution and the rotating reserve capacity is determined. Then, the prediction method is combined with scheduling to construct a scheduling model that takes demand response into consideration with the goal of minimizing the total operating cost of the microgrid. Finally, the scheduling model is converted into a mixed integer linear programming problem through SOT and the CPLEX solver is used to solve the model to obtain the global optimal solution. It effectively solves the impact of new energy output and load uncertainty on the day-ahead scheduling of the microgrid, greatly reduces the total operating cost of the microgrid by improving the prediction accuracy, and greatly simplifies the modeling difficulty of the prediction model. It has the advantages of scientific and reasonable methods, strong applicability, and good results.
[0004] ZL2021115787151 discloses a microgrid energy online optimization method based on distributed deep reinforcement learning. It uses the Actor network to interact with the local environment to obtain the corresponding action strategy, and then judges whether the constraints are met based on the environmental parameters and the selected action, and calculates the reward value. Then the Critic network learns the neural network parameters of the local and other intelligent agents. Finally, based on the learned model, it provides feedback on the actions selected by the Actor network to guide the Actor network to pursue higher reward values. It can quickly give optimization strategies and reasonably allocate the power generation output of each unit, ultimately effectively solving the energy optimization problem in the microgrid.
[0005] Since the optimal configuration of the energy system has a variety of target requirements, only considering the single target requirement of its multi-agent characteristics cannot meet its actual engineering needs. For example, economy and environmental protection are the goals to be pursued in microgrid operation. Summary of the invention
[0006] In order to overcome the above-mentioned technical defects, the present invention provides a multi-objective safety optimization method for a microgrid energy system based on deep reinforcement learning. On the basis of the reinforcement learning DDPG algorithm, the aggregation efficiency of the PBI algorithm is improved, and finally the PSO algorithm is used to optimize the sub-population to push the solution to approach the real Pareto frontier, so that the microgrid energy system can operate economically, environmentally friendly and safely, realize the optimal configuration of the energy system, and solve the multi-objective optimization scheduling problem of the microgrid energy system.
[0007] In order to achieve the above object, the present invention is achieved through the following technical solutions:
[0008] The present invention is a multi-objective security optimization method for a microgrid energy system based on deep reinforcement learning, comprising the following steps:
[0009] Step 1: For the multi-energy system consisting of fuel, operation and maintenance, power outage compensation, CHP system, power interaction and environmental protection management of each micro-power source during microgrid operation, a multi-objective safety optimization model of the microgrid energy system based on deep reinforcement learning is established by taking into account the economic cost, environmental protection cost and power supply reliability within one day as the objective function, and considering the microgrid power balance constraint, the output power constraint of each micro-power source, the ramp rate constraint and the tie line power constraint:
[0010] Multiple Targets:
[0011] Electric power balance safety constraints:
[0012] Output power constraints of each micro power source: P imin ≤P it ≤P imax ,
[0013] Climbing rate constraint: -H MT,d Δt≤P MT,t -P MT,t-1 ≤H MT,u Δt,
[0014] Tie line power constraint: P lmin ≤P gt ≤P lmax
[0015] in:
[0016] ① Fuel cost S f (t)
[0017]
[0018] S fi (P it) is the fuel cost of the i-th micro power source in the t-th scheduling period; N is the number of micro power sources.
[0019] ②Operation and maintenance cost S om (t)
[0020]
[0021] K om,i is the operation and maintenance cost coefficient of the i-th micro power source; P it It is the active power output of the i-th micro-source in the t-th scheduling period.
[0022] ③Power outage compensation cost S L (t)
[0023] S L (t) = S bu (t)×P Lt
[0024] S bu P is the compensation paid by the microgrid system owner to the user for the unit power outage; Lt is the load shedding amount of the microgrid in the tth scheduling period.
[0025] ④ The cost of electricity interaction with the large power grid S g (t)
[0026] S g (t) = S gt (t)×P gt
[0027] S gt P is the electricity purchase and sale price of the microgrid in the tth dispatch period; gt is the power injected from the large grid to the microgrid in the tth dispatch period.
[0028] ⑤ Heating revenue S of CHP system s
[0029] S s =Q ht ×K ph
[0030] S s is the heating benefit of the micro gas turbine in the tth scheduling period; Q ht is the heat load demand in the tth scheduling period; K ph is the selling price per unit of thermal energy;
[0031] F2 is the environmental protection cost of the next day when the microgrid is connected to the grid; α im is the emission of pollutant m when the i-th micro power source produces unit electricity; α gmIt is the emission of pollutant m by thermal power units in the large grid when the large grid transmits unit electric energy to the microgrid. F3 is the load power shortage rate in the tth dispatching period when the microgrid is running, which is used as a measure of power supply reliability. Dt is the electric load demand of the microgrid in the tth scheduling period.
[0032] P imin , P imax are the upper and lower limits of the output power of each micro-power source in the microgrid, H MT,d , H MT,u are the downward and upward ramp rates of the micro gas turbine, P lmin , P lmax They are the upper and lower limits of the maximum power on the tie line respectively.
[0033] Step 2: According to the decomposition-based multi-objective evolutionary algorithm (MOEAD), the multi-objective optimization model in step 1 is decomposed into a set of scalar sub-problems by presetting a set of weight vectors uniformly distributed in the target space, thereby establishing a multi-objective safety optimization model with economic and environmental protection as the goals. When the weight vector is generated, N neighbors are assigned to each sub-problem according to the Euclidean distance between the weight vectors, that is, each particle and the N adjacent particles constitute a sub-population of the particle, and the optimization of each sub-problem is optimized in its sub-population through a replacement strategy based on the penalty boundary intersection method (PBI); when the weight vector is generated, the decomposition-based multi-objective evolutionary algorithm will assign a solution to each weight vector, and there will be a replacement strategy every time a new solution is generated. What principle is the replacement based on and how to compare the advantages and disadvantages between two solutions for a specified sub-problem involves the use of aggregation methods.
[0034] First, we need to generate a set of uniformly distributed weight vectors, where N is the number of weight vectors, i.e. the size of the population:
[0035] ω=(ω1,ω2,ω3...ω N ) T
[0036]
[0037] minF(x)=(f1(x),...f m (x) T
[0038] stx∈Ω
[0039] F(x) is the original multi-objective problem, which is a mapping from decision space to objective space. The multi-objective problem is decomposed into multiple scalar sub-problems. The solutions of the sub-problems are:
[0040] The PBI method uses the penalty factor θ to construct a single-objective optimization subproblem consisting of the vertical distance from the solution to the current search direction and the distance from the solution to the reference point, and then promotes the solution to approach the true Pareto frontier by optimizing the subproblem.
[0041] The PBI decomposition method is as follows:
[0042] minimize g pbi (x|ω,z * )=d1+θd2
[0043] subject to x∈Ω
[0044] In the formula
[0045]
[0046]
[0047] ω=(ω1,ω2,ω3...ω N ) T is the search direction of this subproblem, θ>0 is the penalty factor; is the ideal point in the target space. d1 is the relationship between F(x) and z * The projection of the line is used to evaluate the convergence of x to the solution set. d2 is the distance between F(x) and the projection point, reflecting diversity. Before the search, z * It is generally unknown, and the algorithm uses a search process to find the smallest f i (x) value to replace The purpose of this method is to minimize g pbi The function is used to push F(x) as low as possible, so that it can approach the edge of the target solution set. Therefore, in the PBI method, the value of θ has a crucial impact on the optimization performance. Therefore, the reinforcement learning algorithm is used to train the θ value to achieve the best performance.
[0048] Step 3: Based on the multi-objective evolutionary algorithm (MOEAD), the deep deterministic policy gradient algorithm (DDPG) in reinforcement learning is used to train the penalty factor θ in PBI. The concentration of solutions around the subproblem is defined as the state, and the change in the value of the penalty factor θ is defined as the action. The current state, through the allowed action strategy, can obtain the next state and feedback, and form an experience tuple. The experience tuples obtained under different action strategies are then combined to form the experience pool required for learning the DDPG algorithm in deep reinforcement learning, so that a large amount of training data is stored in the experience pool.
[0049] Step 4: Using the data in the experience pool obtained in step 3, through the neural network in the DDPG algorithm, that is, through a large amount of data, learn the Critic network while learning the Actor network, where the weight of the Critic network is represented by w and the weight of the Actor network is represented by b. Through online learning, the relevant weights in the neural network are trained, and a complete neural network is constructed to find the optimal penalty factor θ, which improves the aggregation efficiency of the PBI algorithm. Finally, the PSO algorithm is used to optimize the subpopulation to promote the solution to approach the real Pareto frontier.
[0050] The specific method of neural network weight training in the DDPG algorithm is:
[0051] D1. The Actor network outputs the corresponding action according to the current state. The Critic network evaluates the action output by the Actor network at each step, that is, estimates the Q value (Q w (s,a)).
[0052] D2. The Actor network updates the policy network weight b according to the Q value evaluated by the Critic network. The Critic network updates the weight w of the Critic network according to the feedback from the environment r, thereby maximizing the Q value.
[0053] D3. At the beginning of training, the weights of the two neural networks are random. However, since the present invention has the environment feedback r, and uses the gradient algorithm to update the optimization weights, the weights of the two neural networks will be trained better and better, and finally the desired weight values will be obtained. With the weights of the trained neural network, when a state is input, the optimal action will be obtained, that is, the optimal θ value will be found. Thereby improving the aggregation efficiency of the PBI algorithm, and finally optimizing the subpopulation through the PSO algorithm to promote the solution to approach the real Pareto frontier.
[0054] The beneficial effects of the present invention are as follows: the multi-objective safety optimization method of a microgrid energy system based on deep reinforcement learning proposed in the present invention not only takes into account the multi-objective characteristics but also takes into account the independent system operation of the microgrid. First, based on the multi-objective overall optimization model of the microgrid energy system, according to the decomposition-based multi-objective evolutionary algorithm, a multi-objective problem is decomposed into multiple scalar subproblems, and then a number of neighbors are assigned to each subproblem. Then, each particle and a number of neighboring particles constitute a subpopulation of the particle, and the penalty-based boundary intersection method (PBI) is used for optimization; and the deep deterministic policy gradient algorithm (DDPG) in reinforcement learning is used to learn the penalty factor θ in the PBI, and the operation of the microgrid is regarded as a continuous process, and the system state and action strategy are accurately set. The current state, through the allowed action strategy, can obtain the next state and feedback, and form an experience tuple; then the other experience tuples obtained under different action strategies are combined to form the experience pool required for DDPG algorithm learning in reinforcement learning; using the data in the experience pool, through the neural network in the DDPG algorithm, learning can be carried out, the relevant weights in the neural network can be trained, and the optimal penalty factor θ can be found, thereby improving the aggregation efficiency of the PBI algorithm, and finally the PSO algorithm is used to optimize the subpopulation to promote the solution to approach the real Pareto frontier. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is a block diagram of the multi-objective security optimization method of a microgrid energy system based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION
[0056] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is, in some embodiments of the present invention, these practical details are not necessary. In addition, for the purpose of simplifying the drawings, some conventional structures and components will be depicted in a simple schematic manner in the drawings.
[0057] like Figure 1 As shown, the present invention is a multi-objective security optimization method for a microgrid energy system based on deep reinforcement learning, and the security optimization method includes:
[0058] (I) Combining the economic benefits of the energy system and the target requirements of environmental protection and low emissions, the multi-energy system consisting of fuel, operation and maintenance, power outage compensation, CHP system, power interaction and environmental protection management of each micro-power source during the operation of the microgrid is comprehensively considered. Taking the economic cost, environmental protection cost and power supply reliability within one day as the objective function, the multi-objective safety optimization model of the microgrid energy system based on deep reinforcement learning is established by considering the microgrid power balance constraint, the output power constraint of each micro-power source, the climbing rate constraint and the interconnection line power constraint:
[0059] (1) Objectives:
[0060] Economic benefits:
[0061] Environmental costs:
[0062] Power supply reliability indicators:
[0063] in:
[0064] ① Fuel cost S f (t)
[0065]
[0066] S fi (P it ) is the fuel cost of the i-th micro power source in the t-th scheduling period; N is the number of micro power sources.
[0067] ②Operation and maintenance cost S om (t)
[0068]
[0069] K om,i is the operation and maintenance cost coefficient of the i-th micro power source; P it It is the active power output of the i-th micro-source in the t-th scheduling period.
[0070] ③Power outage compensation cost S L (t)
[0071] S L (t) = S bu (t)×P Lt
[0072] S bu P is the compensation paid by the microgrid system owner to the user for the unit power outage; Lt is the load shedding amount of the microgrid in the tth scheduling period.
[0073] ④ The cost of electricity interaction with the large power grid S g (t)
[0074] S g (t) = S gt (t)×P gt
[0075] S gt P is the electricity purchase and sale price of the microgrid in the tth dispatch period; gt is the power injected from the large grid to the microgrid in the tth dispatch period.
[0076] ⑤ Heating revenue S of CHP system s
[0077] S s =Q ht ×K ph
[0078] S s is the heating benefit of the micro gas turbine in the tth scheduling period; Q ht is the heat load demand in the tth scheduling period; K ph is the selling price per unit of thermal energy;
[0079] F2 is the environmental protection cost of the next day when the microgrid is connected to the grid; α im is the emission of pollutant m when the i-th micro power source produces unit electricity; α gm It is the emission of pollutant m by thermal power units in the large grid when the large grid transmits unit electric energy to the microgrid. F3 is the load power shortage rate in the tth dispatching period when the microgrid is running, which is used as a measure of power supply reliability. Dt is the electric load demand of the microgrid in the tth scheduling period.
[0080] P imin , P imax are the upper and lower limits of the output power of each micro-power source in the microgrid, H MT,d , H MT,u are the downward and upward ramp rates of the micro gas turbine, P lmin , P lmax They are the upper and lower limits of the maximum power on the tie line respectively.
[0081] (2) Constraints
[0082] Electric power balance safety constraints:
[0083] Output power constraints of each micro power source: P imin ≤P it ≤P imax ,
[0084] Climbing rate constraint: -H MT,d Δt≤P MT,t -P MT,t-1 ≤H MT,u Δt,
[0085] Tie line power constraint: P lmin ≤P gt ≤P lmax
[0086] (ii) According to the multi-objective evolutionary algorithm based on decomposition (MOEAD), a multi-objective optimization model in step 1 is decomposed into a set of scalar subproblems by presetting a set of weight vectors uniformly distributed in the target space, thereby establishing a multi-objective safety optimization model with economic and environmental protection as the goals. After the weight vectors are generated, N neighbors are assigned to each subproblem according to the Euclidean distance between the weight vectors, that is, each particle and its N neighboring particles constitute a subpopulation of the particle. The optimization of each subproblem is optimized in its subpopulation through a replacement strategy based on the penalty boundary intersection method (PBI).
[0087] First, we need to generate a set of uniformly distributed weight vectors, where N is the number of weight vectors, i.e. the size of the population:
[0088] ω=(ω1,ω2,ω3...ω N ) T
[0089]
[0090] minF(x)=(f1(x),...f m (x) T
[0091] stx∈Ω
[0092] F(x) is the original multi-objective problem, which is a mapping from decision space to objective space. The multi-objective problem is decomposed into multiple scalar sub-problems. The solutions of the sub-problems are:
[0093] The PBI method uses the penalty factor θ to construct a single-objective optimization subproblem consisting of the vertical distance from the solution to the current search direction and the distance from the solution to the reference point, and then promotes the solution to approach the true Pareto frontier by optimizing the subproblem.
[0094] The PBI decomposition method is as follows:
[0095] minimize g pbi (x|ω,z * )=d1+θd2
[0096] subject to x∈Ω
[0097] In the formula
[0098]
[0099]
[0100] ω=(ω1,ω2,ω3...ω N ) Tis the search direction of this subproblem, θ>0 is the penalty factor; is the ideal point in the target space. d1 is the relationship between F(x) and z * The projection of the line is used to evaluate the convergence of x to the solution set. d2 is the distance between F(x) and the projection point, reflecting diversity. Before the search, z * It is generally unknown, and the algorithm uses a search process to find the smallest f i (x) value to replace The purpose of this method is to minimize g pbi The function is used to push F(x) as low as possible, so that it can approach the edge of the target solution set. Therefore, in the PBI method, the value of θ has a crucial impact on the optimization performance. Therefore, the reinforcement learning algorithm is used to train the θ value to achieve the best performance.
[0101] (III) Based on the system model obtained above, the optimization of the particle swarm is regarded as a continuous process, the weight vector, i.e., the concentration of solutions around the subproblem, is defined as the state, and the change of the value of θ is defined as the action. t , the action strategy a is allowed t , it can transfer to the next state s t+1 , and whether the feedback value r reaches a balance between convergence and diversity, and forms an experience tuple t ,a t ,s t+1 ,r t >, and stored in the experience pool R. Then, other experience sets obtained from the initial state under different action strategies are also stored in the experience pool R, so that a large amount of training data is stored in the experience pool.
[0102]
[0103] Based on the DDPG algorithm, the concentration S of the solution around the weight vector con Defined as the current state s t , the change of θ value is defined as action a t , whether the balance between convergence and diversity is achieved is defined as the feedback value r. Because the optimal solution of the subproblem should be located around its direction vector, if the number of solutions gathered around some direction vectors is greater than that of other direction vectors, it indicates that the current solution is unevenly distributed. In the evolutionary process, when the penalty parameter is too small, the solution of the boundary subproblem is easily replaced by a solution with better convergence in the neighborhood, resulting in a loss of diversity; when the penalty parameter is too large, the convergence speed of the population will slow down, resulting in a failure to approach the true Pareto frontier. Therefore, the penalty factor can be dynamically adjusted according to the concentration of solutions around the weight vector. The specific steps are:
[0104] (1) Initialize s1 as the first state of the current state sequence;
[0105] (2) Select an action to explore by adding Gaussian perturbation to the untrained network. The network parameters w are:
[0106] a t =μ(s|b)+N t ,
[0107] Where N t is a Gaussian perturbation, and we get action a1 based on state s1;
[0108] (3) Execute action a1 and obtain the new state s2 and reward r1;
[0109] (4) The four elements s1, a1, s2, and r1 form an experience set<s1,a1,s2,r1> , and stored in the experience pool R;
[0110] (5) Randomly select other states and repeat the above steps to obtain the corresponding experience set i ,a i ,s i+1 ,r i >, i = 1, 2, ..., T, and stored in the experience pool R;
[0111] (IV) In the multi-objective safety optimization method of the microgrid energy system based on deep reinforcement learning, the data in the experience pool is used to learn the Critic network and the Actor network through the neural network in the DDPG algorithm, that is, through a large amount of data, where the weight of the Critic network is represented by w and the weight of the Actor network is represented by b. The present invention is to train the weight value of the network through learning, build a complete neural network, and thus find the optimal θ value. The specific training adopts the following method:
[0112] (1) Construct a network consisting of two parts: the Critic network and the Actor network, with parameter weights w and b respectively. The Actor network performs Q function calculation to obtain the Q value: Q w (s, a), the Critic network maps the state to the action to obtain μ(s|ω), and randomly initializes the weights w, b.
[0113] (2) Initialize the weights w', b' of the target network to be solved through the weights w, b.
[0114] (3) The iterative solution operation is as follows:
[0115] ① Update the weights. t ,a t |b) Estimate, and at the same time randomly select a small batch of experience sets from the experience pool R in (iii), and estimate Q through the Bellman equation. The result is
[0116] Assume that y i If it is expressed, then:
[0117] y i =r i +γQ'(s i+1 ,μ'(s i+1 |ω')|b'),
[0118] ②The weight update is based on the gradient descent of TD-error. The policy network uses the value Q(s) of the next moment fitted by the target network. i ,a i |b), and the real return r, we can get Q target , let Q target Subtract the current Q to find the mean square error, and then the Loss function can be constructed.
[0119]
[0120] This loss function minimizes the weights b used to update the policy network.
[0121] ③After the policy network is updated, the Q network is updated. The update mainly adopts the policy gradient method, that is:
[0122]
[0123] All weights ω of the Q network are updated by backpropagating the gradient of the neural network.
[0124] ④Finally, update the weights:
[0125]
[0126] After training the weights of the neural network in this DDPG algorithm, the ideal weights are obtained. With the trained weights of the neural network, when a state is input, the optimal action, that is, the optimal penalty factor θ, will be obtained, thereby improving the aggregation efficiency of the PBI algorithm, and finally the PSO algorithm optimizes the subpopulation to push the solution close to the real Pareto frontier.
[0127] The present invention proposes a multi-objective safety optimization method based on deep reinforcement learning for the multi-objective and multi-constraint characteristics of a microgrid energy system. According to the microgrid energy system's pursuit of economic benefits and environmental protection costs and other target requirements, the multi-objective safety optimization of the energy system is established in combination with the power and load constraints of the microgrid energy system itself. Based on the multi-objective optimization decomposition theory, the multi-objective problem is decomposed into a set of scalar subproblems, which are aggregated using the penalty boundary intersection method (PBI), and these sub-populations are optimized using a population-based stochastic optimization technology algorithm (PSO). In the optimization process, considering that the relevant parameters in the PSO algorithm need to be optimized and adjusted and the reasonable value of the penalty factor θ in the PBI algorithm will improve the efficiency of the optimization algorithm, a deep reinforcement learning DDPG algorithm is used to train the evolutionary parameters in the optimization algorithm, thereby improving the optimization ability of the algorithm, and finally the algorithm optimizes the sub-population to promote the solution to approach the real Pareto frontier, so that the microgrid energy system can operate economically, environmentally friendly and safely.
[0128] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims
1. A multi-objective security optimization method for microgrid energy system based on deep reinforcement learning, Features: The safety optimization method comprises the following steps: Step 1: Establish a multi-objective optimization model of microgrid energy system based on deep reinforcement learning; Step 2: According to the decomposition-based multi-objective evolutionary algorithm MOEAD, the multi-objective optimization model in step 1 is decomposed into a set of scalar sub-problems by presetting a set of weight vectors uniformly distributed in the target space, thereby establishing a multi-objective safety optimization model with economic and environmental protection as the goals. After the weight vectors are generated, N neighbors are assigned to each sub-problem according to the Euclidean distance between the weight vectors, that is, each particle and its N neighboring particles constitute a sub-population of the particle. The optimization of each sub-problem is optimized in its sub-population through the replacement strategy of the penalty-based boundary intersection method PBI; Step 3: Based on the multi-objective evolutionary algorithm MOEAD, the deep deterministic policy gradient algorithm DDPG in reinforcement learning is used to train the penalty factor θ in PBI. The concentration of solutions around the subproblem is defined as the state, and the change in the value of the penalty factor θ is defined as the action. The current state, through the allowed action strategy, can derive the next state and feedback, and form an experience tuple. Then, the experience tuples obtained under different action strategies together form the experience pool required for learning the DDPG algorithm in deep reinforcement learning. Step 4: Using the data in the experience pool obtained in step 3, the neural network in the DDPG algorithm is used to train the relevant weights in the neural network through online learning, so as to find the optimal penalty factor θ, improve the aggregation efficiency of the PBI algorithm, and finally optimize the subpopulation through the PSO algorithm to promote the solution to approach the real Pareto frontier, where: The multi-objective security optimization model of the microgrid energy system based on deep reinforcement learning in step 1 is: Multiple Targets: in: ① Fuel cost S f (t) S fi (P it ) is the fuel cost of the i-th micro power source in the t-th scheduling period; N is the number of micro power sources, ②Operation and maintenance cost S om (t) K om,i is the operation and maintenance cost coefficient of the i-th micro power source; P it is the active power output of the i-th micro-source in the t-th dispatch period; ③Power outage compensation cost S L (t) S L (t)=S bu (t)×P Lt S bu P is the compensation paid by the microgrid system owner to the user for the unit power outage; Lt is the load shedding amount of the microgrid in the tth dispatching period; ④ The cost of electricity interaction with the large power grid S g (t) S g (t)=S gt (t)×P gt S gt P is the electricity purchase and sale price of the microgrid in the tth dispatch period; gt is the power injected from the large grid to the microgrid in the tth dispatch period; ⑤ Heating revenue S of CHP system s S s =Q ht ×K ph S s is the heating benefit of the micro gas turbine in the tth scheduling period; Q ht is the heat load demand in the tth scheduling period; K ph is the selling price per unit of thermal energy; F 2 is the environmental protection cost of the next day when the microgrid is connected to the grid, α im is the emission of pollutant m when the i-th micro power source produces unit electricity, α gm is the emission of pollutants m by thermal power units in the large grid when the large grid transmits unit electric energy to the microgrid, F 3 is the load power shortage rate in the tth dispatching period when the microgrid is running, which is used as a measure of power supply reliability. Dt is the electric load demand of the microgrid in the tth dispatching period, P imin , P imax are the upper and lower limits of the output power of each micro-power source in the microgrid, H MT,d , H MT,u are the downward and upward ramp rates of the micro gas turbine, P lmin , P lmax are the upper and lower limits of the ultimate power on the tie line respectively; The constraints of the multi-objective safety optimization model are: Electric power balance safety constraints: Output power constraints of each micro power source: P imin ≤P it ≤P imax , Climbing rate constraint: -H MT,d Δt≤P MT,t -P MT,t-1 ≤H MT,u Δt, Tie line power constraint: P lmin ≤P gt ≤P lmax ; The steps of optimizing the replacement strategy of the penalty-based boundary intersection method PBI in the subpopulation in step 2 are as follows: Step 2-1: First, you need to generate a set of uniformly distributed weight vectors. N is the number of weight vectors, that is, the size of the population: ω=(ω 1 ,oh 2 ,oh 3 ...oh N ) T minF(x)=(f 1 (x),...f m (x)) T stx∈Ω F(x) is the original multi-objective problem, which is a mapping from decision space to target space. The multi-objective problem is decomposed into multiple scalar sub-problems, ω = (ω 1 ,ω 2 ,ω 3 ... N ) T is the search direction for this subproblem, is an ideal point in the target space, and the solution to the subproblem is Step 2-2: The penalty-based boundary intersection method PBI uses the penalty factor θ to construct a single-objective optimization subproblem consisting of the vertical distance from the solution to the current search direction and the distance from the solution to the reference point, and then optimizes the subproblem to push the solution to the true Pareto frontier.
2. According to the multi-objective security optimization method of microgrid energy system based on deep reinforcement learning according to claim 1, Features: The specific PBI decomposition method in step 2-2 is as follows: minimize g pbi (x|ω,z * )=d 1 +θd 2 subject to x∈Ω In the formula θ>0 is the penalty factor, d 1 For F(x) and z * The projection of the connecting line is used to evaluate the convergence of x to the solution set, d 2 It is the distance between F(x) and the projection point, reflecting diversity.
3. According to the multi-objective security optimization method of the microgrid energy system based on deep reinforcement learning in claim 1, Features: In step 3, the deep deterministic policy gradient algorithm DDPG in reinforcement learning is used to train the penalty factor θ in PBI, which specifically includes the following steps: Step 3-1: Build a network consisting of two parts: the Critic network and the Actor network. The Actor network outputs the corresponding action according to the current state. The Critic network evaluates the action output by the Actor network at each step, that is, estimates the Q value Q of the action output by the Actor network. w (s,a); Step 3-2: The Actor network updates the policy network weight b according to the Q value evaluated by the Critic network. The Critic network updates the weight w of the Critic network according to the feedback from the environment r, thereby maximizing the Q value and finally obtaining the desired weight value.
