Micro-grid optimization scheduling method of improved multi-target particle swarm optimization algorithm in combination with Q-learning

Abstract

The invention provides a micro-grid optimal scheduling method of an improved multi-target particle swarm algorithm in combination with Q-learning, and the method comprises the steps: building a micro-grid model containing renewable energy sources according to the load demand characteristics of an e-commerce park, and building a target function in consideration of the economic cost and carbon emission optimization of a micro-grid system; the multi-target particle swarm optimization algorithm is combined with Q-learning to improve the algorithm performance, the improved multi-target particle swarm optimization algorithm combined with Q-learning utilizes environment information generated in an iteration process, an optimal strategy is adaptively selected in combination with the current state of a population, the improved multi-target particle swarm optimization algorithm combined with Q-learning is adopted to solve a micro-grid system target function, and the optimal strategy is selected in combination with the current state of the population. And an excellent micro-grid dispatching scheme can be obtained. According to the invention, the adaptability and robustness of the system in response to renewable energy power generation fluctuation can be improved, the operation cost of the micro-grid system is reduced, and the emission of pollution gas is reduced.

Description

Technical Field

[0001] The invention relates to a microgrid optimization dispatching method combining an improved multi-objective particle swarm algorithm with Q-learning, and belongs to the technical field of microgrids. Background Art

[0002] Under the background of the "dual carbon" strategy, the power system needs to improve the utilization efficiency of primary energy and optimize the energy structure. As part of the power system, microgrids can effectively absorb photovoltaic power and improve the economy of microgrids by adding photovoltaic power generation and equipping them with a certain capacity of energy storage and backup units. When photovoltaic, energy storage and backup units are added to microgrids, the optimal scheduling of microgrids is a nonlinear, multi-constrained optimization problem. Therefore, a high-performance improved particle swarm optimization algorithm is proposed to effectively solve the microgrid objective function and obtain a set of excellent solution sets, so as to find a microgrid scheduling scheme that takes into account both economic cost and environmental protection.

[0003] Multi-Objective Particle Swarm Optimization (MOPSO) is an intelligent algorithm developed on the basis of Particle Swarm Optimization (Particle Swarm Optimization) to solve multi-objective optimization problems. The multi-objective particle swarm optimization algorithm has the characteristics of few parameters, simple programming, strong optimization ability and the ability to explore in complex multi-objective search space, so the algorithm is often used in the optimization scheduling problem of microgrids. The microgrid system involves more equipment, which brings more variables and constraints, and it is easy to cause the multi-objective particle swarm algorithm to fall into the local optimal solution. In view of the shortcomings of the multi-objective particle swarm optimization algorithm in terms of insufficient optimization ability and convergence ability, it is necessary to improve it, so as to improve the quality of the microgrid optimization scheduling solution. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a microgrid optimization scheduling method based on an improved multi-objective particle swarm algorithm combined with Q-learning, through which a better solution set is obtained, and a set of microgrid scheduling schemes that take into account both economic costs and carbon emissions are obtained.

[0005] The present invention provides a microgrid optimization scheduling method based on an improved multi-objective particle swarm algorithm combined with Q-learning, comprising the following steps:

[0006] S1. Determine the selection of photovoltaic, energy storage and backup units in the microgrid system according to the demand characteristics of the e-commerce park load, achieve the purpose of absorbing renewable energy power generation and improving the efficiency of primary energy utilization, and establish the optimal objective function of economic cost and carbon emissions;

[0007] S2. Propose a constraint processing method to ensure that the operation of the microgrid system meets the constraint conditions, which include the operation constraints of each unit, the battery capacity constraints, and the load supply and demand balance constraints when the microgrid system is running;

[0008] S3, divide the multi-objective particle swarm algorithm into three stages: exploration, exploration and development, and improve the multi-objective particle swarm algorithm by combining the reinforcement learning algorithm Q-learning. According to the value of the Q-table table, the improved multi-objective particle swarm algorithm is judged to enter which stage, and then the population is updated according to the corresponding position update formula;

[0009] S4. The improved particle swarm algorithm combined with Q-1earning is used to solve the microgrid objective function and obtain the optimal solution for microgrid scheduling.

[0010] The present invention establishes a microgrid model including renewable energy based on the load demand characteristics of the e-commerce park, takes into account the economic cost and carbon emission of the microgrid system to optimally establish the objective function, combines the multi-objective particle swarm algorithm with Q-learning to improve the algorithm performance, and uses the improved multi-objective particle swarm algorithm combined with Q-learning to utilize the environmental information generated by the iterative process, and adaptively selects the optimal strategy in combination with the current state of the population. The improved multi-objective particle swarm algorithm combined with Q-learning is used to solve the objective function of the microgrid system, and an excellent microgrid scheduling scheme can be obtained. The present invention can improve the adaptability and robustness of the system in dealing with fluctuations in renewable energy generation, reduce the operating cost of the microgrid system, and reduce the emission of polluting gases.

[0011] The technical solution further optimized as the present invention is as follows:

[0012] In step S1, according to the demand characteristics of the e-commerce park load, an economic cost and carbon emission objective function is constructed, as shown below:

[0013] (1) Economic cost objective function of microgrid

[0014]

[0015] In the formula, C gas is the unit price of natural gas per cubic meter, P MT (t) is the electric power output by the micro gas turbine in the time period t, L NG is the lower calorific value of natural gas, η MTE is the power generation efficiency of the micro gas turbine, P M_P (t) is the power purchased by the large power grid, P M_s (t) is the power sold by the large power grid, C p is the unit price of electricity purchased by the large power grid, C Sis the unit price of electricity sold by the large power grid, μ i is the operating cost coefficient of different units in the system, such as photovoltaic, battery, electric boiler and electric air conditioner, P i (t) is the output power of different units in the system, such as photovoltaic, battery, electric boiler and electric air conditioner, and N is the number of units that produce gaseous pollutants;

[0016] (2) Carbon emission objective function of microgrid

[0017]

[0018] Where N is the number of units producing gaseous pollutants; i,1 is the emission coefficient of carbon dioxide corresponding to the output of electrical power by the micro gas turbine, λ i,2 The emission factor of carbon dioxide generated by exporting electrical power to the large power grid.

[0019] In step S2, the constraints mainly include:

[0020] (1) Electric power balance constraints

[0021] P MT (t)+P M (t)+P PV (t)+P BA (t)-P EC (t)-P EH (t) = P load (t)

[0022] Where P MT (t) is the power generation of the micro gas turbine, P M (t) is the interaction power of the large power grid, P PV (t) is the photovoltaic power generation power, P BA (t) is the charging and discharging power of the battery, P EC (t) is the power consumption of the air conditioner, P EH (t) is the power consumption of the electric boiler, P load (t) is the user's electricity load demand;

[0023] (2) Thermal power balance constraints

[0024]

[0025] In the formula, Q S (t) is the thermal power output of the heat storage tank, COP EH is the heating efficiency of the electric boiler, Q H load (t) is the heat load required by the e-commerce park;

[0026] (3) Cold power balance constraint

[0027]

[0028] In the formula, COP EC is the cooling efficiency of the electric refrigerator, Q C load (t) is the cooling load required by the e-commerce park;

[0029] (4) Output constraints of controllable units

[0030] P i,min ≤P i (t)≤P i,max ,i=1,...,N;t=1,...,24

[0031] Where P i,max , P i,min are the upper and lower bounds of the output of the ith unit respectively;

[0032] (5) Climbing constraints of micro power supply

[0033] -DR i ≤P i (t)-P i (t-1)≤UR i ,i=1,...,N;t=1,...,24

[0034] In the formula, DR i is the upper limit of the upward climbing of the i-th unit, UR i is the lower limit of the downward climbing of the i-th unit;

[0035] (6) Battery capacity constraints

[0036] E BA (t) = (1-σ BA )E BA (t-1)+[p BA_C (t)η BA -p BA_d (t) / η BA ]Δt

[0037] In the formula, E BA (t) is the electric load stored in the battery during the period t, E BA (t-1) is the electric load stored by the battery in the t-1 period, p BA_C (t) is the output power of the battery in period t, p BA_d (t) is the power stored in the battery during the period t, σ BA is the battery self-discharge efficiency, η BA is the charge and discharge efficiency of the battery, and Δt is the adjacent time period.

[0038] In step S2, a constraint processing method is proposed to ensure that the operation of the microgrid system meets the constraint conditions, and the specific operation steps are as follows:

[0039] S201, the output of the micro gas turbine, battery and large power grid at time t is randomly generated. When the unit output exceeds the upper and lower limits, it is repaired according to the unit output repair formula. The unit output repair formula is:

[0040] S202. When the output of the unit exceeds the climbing limit in the adjacent time period, the unit climbing repair formula is used for repair. The unit climbing repair formula is:

[0041]

[0042] S203. After determining the output power of the micro gas turbine, determine the waste heat value of the micro gas turbine. According to the heat load and cold load requirements, the power consumption of the electric boiler and the electric air conditioner can be obtained. R(t) is a variable that violates the power balance constraint. The power balance constraint repair formula can be used to adjust the output of the large power grid. Through the above steps, the power balance of the system can be strictly achieved. The variable formula that violates the power balance constraint is

[0043] R(t)=P load (t)-(P MT (t)+P M (t)+P PV (t)+P BA (t)-P EC (t)-P EH (t))

[0044] The power balance constraint repair formula is P M (t) = P M (t)+R(t)

[0045] Where R(t) is the variable that violates the electric power balance constraint;

[0046] S204: Determine the waste heat of the micro gas turbine in step S203, and adjust the thermal power output of the electric boiler according to the thermal load demand of the e-commerce park, so as to strictly ensure the balance of thermal power. Since the cooling power of the system is only provided by the electric air conditioner, the cooling power balance can also be ensured.

[0047] In step S3, the multi-objective particle swarm algorithm is improved by combining Q-1earning, and the specific method is as follows:

[0048] S301, set the population size and the maximum number of iterations Tmax, initialize the position and velocity of each particle in the population, and initialize the size of the external archive Archive to Amax;

[0049] S302, set Q-table to a 3×3 zero matrix, and set reward table to

[0050] S303, dividing the particle swarm algorithm into three main stages: exploration, exploration and development, and setting different position update formulas in the three stages, so as to continuously adjust the search focus of the population according to the search state of the population, and improve the global search performance of the particle swarm algorithm;

[0051] S304, combined with the Q-learning algorithm, the intelligent agent determines which stage the particle swarm should enter according to the values ​​in the Q-table, and updates the population using the corresponding speed and position update formula;

[0052] S305, executing the selected position update formula, if the newly obtained solution set is in a dominant position, giving the solution set a positive reward (Reward) of +1, otherwise giving the solution set a negative reward (Reward) of -1;

[0053] S306. Update the corresponding Q-table according to the Bellman equation, the formula is:

[0054] Q(s t+1 ,a t+1 )=Q(s t ,a t )+α[r t+1 +γmax(Q(s',a'))-Q(s t ,a t )]

[0055] In the formula, Q(s t+1 ,a t+1 ) is the expected return of taking action a in state s in the t+1th generation, Q(s t ,a t ) is the expected return of taking action d in state s in the tth generation, α is the learning rate, γ is the discount factor, and r t+1 is the actual benefit received from taking action a in state s, and Q(s',a') is the expected benefit of all actions a' taken in state s';

[0056] S307, determine whether the current number of iterations is greater than the maximum number of iterations, if so, output the result, otherwise return to step S303 to enter the loop.

[0057] In step S303, the specific method for improving the global search performance of the particle swarm algorithm is as follows:

[0058] S303-1. Exploration phase: The population explores a wider range of target space at a faster speed to find the potential best solution. The speed and position update formulas of the population in the exploration phase are:

[0059] w=w max +(w max -w min )×rand(0,1)

[0060] X i (t+1)=wX i (t)+c1r1(X i (t)-pbest i )+c2r2(X i (t)-lb)+c3r3(X i (t)-Ub)

[0061] In the formula, w is the weight parameter, w min is the minimum weight parameter, w max is the maximum weight parameter, X i (t+1) The position of the ith particle in generation t+1, X i (t) is the position of the ith particle in generation t, pbest i is the optimal value of the i-th particle, Ub and lb are the upper and lower bounds of the particle group position, c1, c2, and c3 are learning factors, and r1, r2, and r3 are random numbers between 0 and 1;

[0062] S303-2, Exploration stage - In order to increase the complexity of the population and reduce the particle swarm from falling into the local optimum, the tangent search strategy is used to expand the search space. The tangent search strategy controls the movement of particles through the angle value, and the tangent function helps to expand the exploration space of the population. The speed and position update formulas of the population in the exploration stage are:

[0063]

[0064] V i (t+1)=wX i (t)+c1r1(X i (t)-Gbest)×tan(θ)

[0065] X i (t+1)=X i (t)+V i (t+1)

[0066] Where AF is the adaptive parameter that changes with the number of iterations, t is the current number of iterations, Gbest is the global optimal value, θ is the radian value, V i (t+1) is the velocity of the ith particle in generation t+1;

[0067] S303-3, Development stage - Use Gaussian distribution function to perturb the existing solution set, so that the solutions in the solution set are more evenly distributed in the target space. The speed and position update formulas of the population in the development stage are:

[0068]

[0069] Wherein, the population mean μ is at 0, the population variance σ decreases with the increase of the number of iterations, r4 is a random number between 0 and 1, GD is the Gaussian distribution function, mean(X) is the average position of the population, and |mean(X)-Gbest| is the matrix of the mean values ​​of each dimension of the population center.

[0070] In step S304, the specific method for the agent to determine which stage the particle swarm should enter according to the value in the Q-table is as follows:

[0071] like Maximum, enter the exploration phase, and use the speed and position update formula of the exploration phase;

[0072] like Maximum, enter the exploration phase, and use the speed and position update formula of the exploration phase;

[0073] If Q(s t ,a3 t ) is maximum, enter the development stage, and select the speed and position update formula of the development stage.

[0074] In step S4, the objective function is solved by using an improved multi-objective particle swarm algorithm to obtain a Pareto solution set, and an external archive Archive is output.

[0075] In step S4, the specific method of solving the objective function using the improved multi-objective particle swarm algorithm is as follows:

[0076] Step S401, collecting load demand data of the e-commerce park;

[0077] Step S402, initializing the multi-objective particle swarm parameters, initializing the Q-table, reward table, and external archive size;

[0078] Step S403: Use the improved multi-objective particle swarm algorithm combined with Q-learning to solve the microgrid objective function, obtain the individual optimal solution Pbest based on non-dominated sorting, update the Pareto non-dominated solution of the population and store it in Archive. If the number of its solution sets exceeds the maximum capacity Amax of the external archive Archive, select the first Amax particles according to the crowding distance sorting;

[0079] Step S404: selecting the global optimal solution Gbest from the external archive Archive based on the roulette strategy;

[0080] Step S405, first update the reward table, and then update the corresponding Q-table according to the Bellman equation;

[0081] Step S406, determine whether the algorithm has reached the maximum number of iterations, if not, return to step S403, if reached, go to step S407;

[0082] Step S407, outputting the external archive Archive;

[0083] Step S408, end the operation.

[0084] Compared with the prior art, the present invention takes the economic cost and carbon emissions of the microgrid as the objective function, adopts an improved multi-objective particle swarm algorithm, and combines it with the reinforcement learning algorithm Q-learning to realize the use of Agent intelligent body to select the optimal position update formula, make full use of the information of the previous iteration of the population, and take into account the exploration, exploration and development of the population, thereby obtaining a set of excellent Pareto solutions, which not only fully absorbs the photovoltaic power, but also reduces the operating cost through the charging and discharging of the battery, improves the economic benefits of the microgrid, and reduces carbon emissions. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 This is a flow chart of the microgrid optimization scheduling of the present invention.

[0086] Figure 2 It is a structural diagram of the microgrid system of the present invention.

[0087] Figure 3 The figure is a flow chart of the improved multi-objective particle swarm algorithm combined with Q-learning of the present invention. DETAILED DESCRIPTION

[0088] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and specific operation process are given, but the protection authority of the present invention is not limited to the following embodiments.

[0089] Example 1

[0090] A microgrid optimization scheduling method based on an improved multi-objective particle swarm algorithm combined with Q-learning, such as Figure 1 As shown, the following steps are included:

[0091] S1. Determine the selection of photovoltaic, energy storage and standby units in the microgrid system according to the demand characteristics of the e-commerce park load, achieve the purpose of absorbing renewable energy power generation and improving the efficiency of primary energy utilization, and establish the optimal objective function of economic cost and carbon emissions.

[0092] Taking into account the load demand characteristics of the e-commerce park, the selection of each unit in the microgrid is determined, and the economic cost and carbon emission objective function is constructed, as shown below:

[0093] (1) Economic cost objective function of microgrid

[0094]

[0095] In the formula, C gas is the unit price of natural gas per cubic meter, P MT (t) is the electric power output by the micro gas turbine in the time period t, L NG is the lower calorific value of natural gas, η MTE is the power generation efficiency of the micro gas turbine, P M_P (t) is the power purchased by the large power grid, P M_s (t) is the power sold by the large power grid, C p is the unit price of electricity sold by the large power grid, C S is the unit price of electricity sold by the large power grid, μ i is the operating cost coefficient of different units in the system, such as photovoltaic, battery, electric boiler and electric air conditioner, P i (t) is the output power of different units in the system, such as photovoltaic, battery, electric boiler and electric air conditioner, and N is the number of units that produce gaseous pollutants.

[0096] (2) Carbon emission objective function of microgrid

[0097]

[0098] Where N is the number of units producing gaseous pollutants; i,1 is the emission coefficient of carbon dioxide corresponding to the output of electrical power by the micro gas turbine, λ i,2 is the emission coefficient of carbon dioxide corresponding to the output of electric power from the large power grid. At this time, P i (t) is the electric power output by the micro gas turbine and the large power grid, where P1(t) is the electric power output by the micro gas turbine and P2(t) is the electric power output by the large power grid.

[0099] S2. A constraint processing method is proposed to ensure that the operation of the microgrid system meets the constraint conditions, which include the operation constraints of each unit, the battery capacity constraints, and the load supply and demand balance constraints during the operation of the microgrid system.

[0100] The constraints mainly include:

[0101] (1) Electric power balance constraints

[0102] P MT (t)+P M (t)+P PV (t)+P BA (t)-P EC (t)-P EH (t) = P load (t)

[0103] Where P MT (t) is the power generation of the micro gas turbine, P M (t) is the interaction power of the large power grid, P PV (t) is the photovoltaic power generation power, P BA (t) is the charging and discharging power of the battery, P EC (t) is the power consumption of the air conditioner, P EH (t) is the power consumption of the electric boiler, P load (t) is the user's electricity load demand;

[0104] (2) Thermal power balance constraints

[0105]

[0106] In the formula, Q S (t) is the thermal power output of the heat storage tank, COP EH is the heating efficiency of the electric boiler, Q H load (t) is the heat load required by the e-commerce park;

[0107] (3) Cold power balance constraint

[0108]

[0109] In the formula, COP EC is the cooling efficiency of the electric refrigerator, Q C load (t) is the cooling load required by the e-commerce park;

[0110] (4) Output constraints of controllable units

[0111] P i,min ≤P i (t)≤Pi,max ,i=1,...,N;t=1,...,24

[0112] Where P i,max , P i,min are the upper and lower bounds of the output of the i-th unit, P i (t) is the output of the i-th unit in period t;

[0113] (5) Climbing constraints of micro power supply

[0114] -DR i ≤P i (t)-P i (t-1)≤UR i ,i=1,...,N;t=1,...,24

[0115] In the formula, DR i is the upper limit of the upward climbing of the i-th unit, P i (t-1) is the output of the i-th unit in the t-1 period, UR i is the lower limit of the downward climbing of the i-th unit;

[0116] (6) Battery capacity constraints

[0117] E BA (t) = (1-σ BA )E BA (t-1)+[p BA_C (t)η BA -p BA_d (t) / η BA ]Δt

[0118] In the formula, E BA (t) is the electric load stored in the battery during the period t, E BA (t-1) is the electric load stored by the battery in the t-1 period, p BA_C (t) is the output power of the battery in period t, p BA_d (t) is the power stored in the battery during the period t, σ BA is the battery self-discharge efficiency, η BA is the charge and discharge efficiency of the battery, and Δt is the adjacent time period.

[0119] The specific operation steps of a constraint processing method are as follows:

[0120] S201, the outputs of the micro gas turbine, battery and large power grid at time t are randomly generated. When the unit output exceeds the upper and lower limits, it is repaired according to the unit output repair formula.

[0121] The unit output repair formula is:

[0122] S202. When the output of the unit exceeds the ramp limit in the adjacent time period, the unit is repaired according to the ramp repair formula. The ramp repair formula is:

[0123]

[0124] S203. After determining the output power of the micro gas turbine, determine the waste heat value of the micro gas turbine. According to the heat load and cold load requirements, the power consumption of the electric boiler and the electric air conditioner can be obtained. R(t) is a variable that violates the power balance constraint. The power balance constraint repair formula can be used to adjust the output of the large power grid. Through the above steps, the power balance of the system can be strictly achieved. The variable formula that violates the power balance constraint is

[0125] R(t)=P load (t)-(P MT (t)+P M (t)+P PV (t)+P BA (t)-P EC (t)-P EH (t))

[0126] The power balance constraint repair formula is P M (t) = P M (t)+R(t)

[0127] Where R(t) is the variable that violates the electric power balance constraint.

[0128] S204: Determine the waste heat of the micro gas turbine in step S203, and adjust the thermal power output of the electric boiler according to the thermal load demand of the e-commerce park, so as to strictly ensure the balance of thermal power. Since the cooling power of the system is only provided by the electric air conditioner, the cooling power balance can also be ensured.

[0129] S3. The multi-objective particle swarm algorithm is divided into three stages: exploration, exploration and development. The multi-objective particle swarm algorithm is improved by combining the reinforcement learning algorithm Q-learning. The Agent determines which stage to enter according to the value of the Q-table, and then the population is updated according to the corresponding position update formula.

[0130] The improved multi-objective particle swarm algorithm combined with Q-1earning relies on the Agent intelligent body to judge the value of Q-table and select an optimal operation considering the current state of the population. Its main steps are:

[0131] S301, set the population size and the maximum number of iterations Tmax, initialize the position and velocity of each particle in the population, and initialize the size of the external archive Archive to Amax;

[0132] S302, set Q-table to a 3×3 zero matrix, and set reward table to

[0133] S303, dividing the particle swarm algorithm into three main stages: exploration, exploration and development, and setting different position update formulas in the three stages, so as to continuously adjust the search focus of the population according to the search state of the population, and improve the global search performance of the particle swarm algorithm;

[0134] The specific methods to improve the global search performance of the particle swarm algorithm are as follows:

[0135] S303-1. Exploration phase: The population explores a wider range of target space at a faster speed to find the potential best solution. The speed and position update formulas of the population in the exploration phase are:

[0136] w=w max +(w max -w min )×rand(0,1)

[0137] X i (t+1)=wX i (t)+c1r1(X i (t)-pbest i )+c2r2(X i (t)-lb)+c3r3(X i (t)-Ub)

[0138] In the formula, w is the weight parameter, w min is the minimum weight parameter, w max is the maximum weight parameter, X i (t+1) The position of the ith particle in generation t+1, X i (t) is the position of the ith particle in generation t, pbest i is the optimal value of the i-th particle, Ub and lb are the upper and lower bounds of the particle group position, c1, c2, and c3 are learning factors, and r1, r2, and r3 are random numbers between 0 and 1;

[0139] S303-2, Exploration stage - In order to increase the complexity of the population and reduce the particle swarm from falling into the local optimum, the tangent search strategy is used to expand the search space. The tangent search strategy controls the movement of particles through the angle value. The tangent function helps to expand the exploration space of the population. The speed and position update formulas of the population in the exploration stage are:

[0140]

[0141] V i (t+1)=wX i (t)+c1r1(X i (t)-Gbest)×tan(θ)

[0142] X i (t+1)=X i (t)+V i (t+1)

[0143] Where AF is the adaptive parameter that changes with the number of iterations, t is the current number of iterations, Gbest is the global optimal value, θ is the radian value, V i (t+1) is the velocity of the ith particle in generation t+1;

[0144] S303-3, Development stage - Use Gaussian distribution function to perturb the existing solution set, so that the solutions in the solution set are more evenly distributed in the target space. The speed and position update formulas of the population in the development stage are:

[0145]

[0146] Where, the population mean μ is at 0, the population variance σ decreases with the increase of the number of iterations, r4 is a random number between 0 and 1, GD is the Gaussian distribution function, mean(X) is the average position of the population, |mean(X)-Gbest| is the matrix of the mean values ​​of each dimension of the population center;

[0147] S304, combined with the Q-learning algorithm, the intelligent agent determines which stage the particle swarm should enter according to the values ​​in the Q-table, and updates the population using the corresponding speed and position update formula;

[0148] The specific method for the agent to determine which stage the particle swarm should enter based on the values ​​in the Q-table is as follows:

[0149] like Maximum, enter the exploration phase, and use the speed and position update formula of the exploration phase;

[0150] like Maximum, enter the exploration phase, and use the speed and position update formula of the exploration phase;

[0151] like Maximum, cut into the development stage, and use the speed and position update formula of the development stage;

[0152] in, is the expected return of taking action a in state s in the tth generation of the corresponding exploration phase, is the expected return of taking action a in state s in the tth generation of the corresponding exploration stage, is the expected return of taking action a in state s at the tth generation of the corresponding development stage;

[0153] S305, executing the selected position update formula, if the newly obtained solution set is in a dominant position, giving the solution set a positive reward (Reward) of +1, otherwise giving the solution set a negative reward (Reward) of -1;

[0154] S306. Update the corresponding Q-table according to the Bellman equation, the formula is:

[0155] Q(s t+1 ,a t+1 )=Q(s t ,a t )+α[r t+1 +γmax(Q(s',a'))-Q(s t ,a t )]

[0156] In the formula, Q(s t+1 ,a t+1 ) is the expected return of taking action a in state s in the t+1th generation, Q(s t ,a t ) is the expected return of taking action d in state s in the tth generation, α is the learning rate, γ is the discount factor, and r t+1 is the actual benefit received from taking action a in state s, and Q(s',a') is the expected benefit of all actions a' taken in state s';

[0157] S307, determine whether the current number of iterations is greater than the maximum number of iterations, if so, output the result, otherwise return to step S303 to enter the loop.

[0158] S4. The improved particle swarm algorithm combined with Q-1earning is used to solve the microgrid objective function and obtain the optimal solution for microgrid scheduling.

[0159] The improved multi-objective particle swarm algorithm combined with the Q-learning algorithm solves the objective function of the microgrid, obtains the Pareto solution set stored in the external archive, and outputs the external archive. The specific process is as follows:

[0160] Step S401, collecting load demand data of the e-commerce park;

[0161] Step S402, initializing the multi-objective particle swarm parameters, initializing the Q-table, reward table, and external archive size;

[0162] Step S403: Use the improved multi-objective particle swarm algorithm combined with Q-learning to solve the microgrid objective function, obtain the individual optimal solution Pbest based on non-dominated sorting, update the Pareto non-dominated solution of the population and store it in Archive. If the number of its solution sets exceeds the maximum capacity Amax of the external archive Archive, select the first Amax particles according to the crowding distance sorting;

[0163] Step S404: selecting the global optimal solution Gbest from the external archive Archive based on the roulette strategy;

[0164] Step S405, first update the reward table, and then update the corresponding Q-table according to the Bellman equation;

[0165] Step S406, determine whether the algorithm has reached the maximum number of iterations, if not, return to step S403, if reached, go to step S407;

[0166] Step S407, outputting the external archive Archive;

[0167] Step S408, end the operation.

[0168] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technology can understand and think of any changes or substitutions within the technical scope disclosed by the present invention, which should be included in the scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A microgrid optimization scheduling method based on an improved multi-objective particle swarm algorithm combined with Q-learning, characterized in that: The following steps are involved: S1. Determine the selection of photovoltaic, energy storage and backup units in the microgrid system according to the demand characteristics of the e-commerce park load, and establish the optimal objective function of economic cost and carbon emissions; S2. Propose a constraint processing method to ensure that the operation of the microgrid system meets the constraint conditions, which include the operation constraints of each unit, the battery capacity constraints, and the load supply and demand balance constraints when the microgrid system is running; S3, divide the multi-objective particle swarm algorithm into three stages: exploration, exploration and development, and improve the multi-objective particle swarm algorithm by combining the reinforcement learning algorithm Q-learning. According to the value of the Q-table table, the improved multi-objective particle swarm algorithm is judged to enter which stage, and then the population is updated according to the corresponding position update formula; S4. The improved particle swarm algorithm combined with Q-1earning is used to solve the microgrid objective function and obtain the optimal solution for microgrid scheduling.

2. According to claim 1, a microgrid optimization scheduling method based on an improved multi-objective particle swarm algorithm combined with Q-learning is characterized in that: In step S1, according to the demand characteristics of the e-commerce park load, an economic cost and carbon emission objective function is constructed, as shown below: (1) Economic cost objective function of microgrid In the formula, C gas is the unit price of natural gas per cubic meter, P MT (t) is the electric power output by the micro gas turbine in the time period t, L NG is the lower calorific value of natural gas, η MTE is the power generation efficiency of the micro gas turbine, P M_P (t) is the power purchased by the large power grid, P M_s (t) is the power sold by the large power grid, C p is the unit price of electricity purchased by the large power grid, C S is the unit price of electricity sold by the large power grid, μ i is the operating cost coefficient of different units, P i (t) is the output power of different units, and N is the number of units producing gaseous pollutants; (2) Carbon emission objective function of microgrid Where N is the number of units producing gaseous pollutants; i,1 is the emission coefficient of carbon dioxide corresponding to the output of electrical power by the micro gas turbine, λ i,2 The emission factor of carbon dioxide generated by exporting electrical power to the large power grid.

3. According to claim 2, a microgrid optimization scheduling method based on an improved multi-objective particle swarm algorithm combined with Q-learning is characterized in that: In step S2, the constraints mainly include: (1) Electric power balance constraints P MT (t)+P M (t)+P PV (t)+P BA (t)-P EC (t)-P EH (t)=P load (t) Where P MT (t) is the power generation of the micro gas turbine, P M (t) is the interaction power of the large power grid, P PV (t) is the photovoltaic power generation power, P BA (t) is the charging and discharging power of the battery, P EC (t) is the power consumption of the air conditioner, P EH (t) is the power consumption of the electric boiler, P load (t) is the user's electricity load demand; (2) Thermal power balance constraints In the formula, Q S (t) is the thermal power output of the heat storage tank, COP EH is the heating efficiency of the electric boiler, Q H load (t) is the heat load required by the e-commerce park; (3) Cold power balance constraint In the formula, COP EC is the cooling efficiency of the electric refrigerator, Q C load (t) is the cooling load required by the e-commerce park; (4) Output constraints of controllable units P i,min ≤P i (t)≤P i,max ,i=1,...,N;t=1,...,24 Where P i,max , P i,min are the upper and lower bounds of the output of the ith unit respectively; (5) Climbing constraints of micro power supply -DR i ≤P i (t)-P i (t-1)≤UR i ,i=1,...,N;t=1,...,24 In the formula, DR i is the upper limit of the upward climbing of the i-th unit, UR i is the lower limit of the downward climbing of the i-th unit; (6) Battery capacity constraints From BA (t)=(1-σ BA )E BA (t-1)+[p BA_C (t)η BA -p BA_d (t) / η BA ]Δt In the formula, E BA (t) is the electric load stored in the battery during the period t, E BA (t-1) is the electric load stored by the battery in the t-1 period, p BA_C (t) is the output power of the battery in period t, p BA_d (t) is the power stored in the battery during the period t, σ BA is the battery self-discharge efficiency, η BA is the charge and discharge efficiency of the battery, and Δt is the adjacent time period.

4. According to claim 3, a microgrid optimization scheduling method combining an improved multi-objective particle swarm algorithm with Q-learning is characterized in that: In step S2, a constraint processing method is proposed to ensure that the operation of the microgrid system meets the constraint conditions, and the specific operation steps are as follows: S201, the output of the micro gas turbine, battery and large power grid at time t is randomly generated. When the unit output exceeds the upper and lower limits, it is repaired according to the unit output repair formula. The unit output repair formula is: S202. When the output of the unit exceeds the climbing limit in the adjacent time period, the unit climbing repair formula is used for repair. The unit climbing repair formula is: S203. After determining the output power of the micro gas turbine, determine the waste heat value of the micro gas turbine. According to the heat load and cold load requirements, the power consumption of the electric boiler and the electric air conditioner can be obtained. The variable formula that violates the power balance constraint is: R(t)=P load (t)-(P MT (t)+P M (t)+P PV (t)+P BA (t)-P EC (t)-P EH (t)) The power balance constraint repair formula is P M (t) = P M (t)+R(t) Where R(t) is the variable that violates the electric power balance constraint; S204. In step S203, the waste heat of the micro gas turbine is determined, and the thermal power output by the electric boiler is adjusted according to the thermal load demand of the e-commerce park, so as to strictly ensure the thermal power balance.

5. The microgrid optimization scheduling method based on the improved multi-objective particle swarm algorithm combined with Q-learning according to claim 4 is characterized in that: In step S3, the multi-objective particle swarm algorithm is improved by combining Q-1earning, and the specific method is as follows: S301, set the population size and the maximum number of iterations Tmax, initialize the position and velocity of each particle in the population, and initialize the size of the external archive Archive to Amax; S302, set Q-table to a 3×3 zero matrix, set the reward table to S303, dividing the particle swarm algorithm into three main stages: exploration, exploration and development, and setting different position update formulas in the three stages, so as to continuously adjust the search focus of the population according to the search state of the population, and improve the global search performance of the particle swarm algorithm; S304, combining the Q-learning algorithm, judging which stage the particle swarm should enter according to the values ​​in the Q-table, and then using the corresponding speed and position update formula to update the population; S305, executing the selected position update formula, if the newly obtained solution set is in a dominant position, giving the solution set a positive reward of +1, otherwise giving the solution set a negative reward of -1; S306. Update the corresponding Q-table according to the Bellman equation, the formula is: Q(s t+1 ,a t+1 )=Q(s t ,a t )+α[r t+1 +γmax(Q(s',a'))-Q(s t ,a t )] In the formula, Q(s t+1 ,a t+1 ) is the expected return of taking action a in state s in the t+1th generation, Q(s t ,a t ) is the expected return of taking action d in state s in the tth generation, α is the learning rate, γ is the discount factor, and r t+1 is the actual benefit received from taking action a in state s, and Q(s',a') is the expected benefit of all actions a' taken in state s'; S307, determine whether the current number of iterations is greater than the maximum number of iterations, if so, output the result, otherwise return to step S303 to enter the loop.

6. The microgrid optimization scheduling method based on the improved multi-objective particle swarm algorithm combined with Q-learning according to claim 5 is characterized in that: In step S303, the specific method for improving the global search performance of the particle swarm algorithm is as follows: S303-1. The speed and position update formulas of the population in the exploration phase are: in=in max +(in max -In min )×rand(0,1) X i (t+1)=wX i (t)+c1r1(X i (t)-pbest i )+c2r2(X i (t)-lb)+c3r3(X i (t)-Ub) In the formula, w is the weight parameter, w min is the minimum weight parameter, w max is the maximum weight parameter, X i (t+1) The position of the ith particle in generation t+1, X i (t) is the position of the ith particle in generation t, pbest i is the optimal value of the i-th particle, Ub and lb are the upper and lower bounds of the particle group position, c1, c2, and c3 are learning factors, and r1, r2, and r3 are random numbers between 0 and 1; S303-2. The speed and position update formulas of the population in the exploration phase are: V i (t+1)=wX i (t)+c1r1(X i (t)-Gbest)×tan(θ) X i (t+1)=X i (t)+V i (t+1) Where AF is the adaptive parameter that changes with the number of iterations, t is the current number of iterations, Gbest is the global optimal value, θ is the radian value, V i (t+1) is the velocity of the ith particle in generation t+1; S303-3. The speed and position update formulas of the population in the development stage are: Wherein, the population mean μ is at 0, the population variance σ decreases with the increase of the number of iterations, r4 is a random number between 0 and 1, GD is the Gaussian distribution function, mean(X) is the average position of the population, and |mean(X)-Gbest| is the matrix of the mean values ​​of each dimension of the population center.

7. The microgrid optimization scheduling method based on the improved multi-objective particle swarm algorithm combined with Q-learning according to claim 6 is characterized in that: In step S304, the specific method for the agent to determine which stage the particle swarm should enter according to the value in the Q-table is as follows: like Maximum, enter the exploration phase, and use the speed and position update formula of the exploration phase; like Maximum, enter the exploration phase, and use the speed and position update formula of the exploration phase; like Maximum, enter the development stage, and select the speed and position update formula of the development stage.

8. The microgrid optimization scheduling method based on the improved multi-objective particle swarm algorithm combined with Q-learning according to claim 7 is characterized in that: In step S4, the objective function is solved by using an improved multi-objective particle swarm algorithm to obtain a Pareto solution set, and an external archive Archive is output.

9. The microgrid optimization scheduling method based on the improved multi-objective particle swarm algorithm combined with Q-learning according to claim 8 is characterized in that: In step S4, the specific method of solving the objective function using the improved multi-objective particle swarm algorithm is as follows: Step S401, collecting load demand data of the e-commerce park; Step S402, initialize the multi-objective particle swarm parameters, initialize the Q-table, reward table and external archive size; Step S403: Use the improved multi-objective particle swarm algorithm combined with Q-learning to solve the microgrid objective function, obtain the individual optimal solution Pbest based on non-dominated sorting, update the Pareto non-dominated solution of the population and store it in Archive. If the number of its solution sets exceeds the maximum capacity Amax of the external archive Archive, select the first Amax particles according to the crowding distance sorting; Step S404: selecting the global optimal solution Gbest from the external archive Archive based on the roulette strategy; Step S405, first update the reward table, and then update the corresponding Q-table according to the Bellman equation; Step S406, determine whether the algorithm has reached the maximum number of iterations, if not, return to step S403, if reached, go to step S407; Step S407, outputting the external archive Archive; Step S408, end the operation.

Images