Post-disaster emergency energy supply optimization scheduling method for multi-energy coupling distribution system

Abstract

The present invention relates to a post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling distribution system, which belongs to the technical field of post-disaster resource utilization of distribution systems, and solves the problem of the lack of effective related optimization scheduling methods in the prior art. The method includes: taking the maximization of the energy supply benefit of the multi-energy coupling distribution system in the period of time when the external energy supply is lost after the disaster as the goal, constructing the objective function of the post-disaster emergency energy supply optimization scheduling model; constructing the constraint conditions of the model based on the constraints such as the constraints related to the multi-energy coupling system and the distribution network operation constraints; constructing the solution problem of the above model as a Markov decision process, determining the state vector, action vector and reward function corresponding to each moment; and using the improved reinforcement learning algorithm for training to obtain a reinforcement learning intelligent agent adapted to the post-disaster emergency energy supply optimization scheduling; and applying the intelligent agent to the post-disaster emergency energy supply optimization scheduling process to obtain the corresponding optimization scheduling strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of post-disaster resource utilization of power distribution systems, and in particular to a post-disaster emergency energy supply optimization scheduling method for multi-energy coupling power distribution systems. Background Art

[0002] The post-disaster emergency energy supply optimization scheduling method for multi-energy coupled distribution systems is a complex issue, which involves how to effectively utilize and schedule various energy resources after a disaster to ensure the power supply restoration of critical loads and the stable operation of the system.

[0003] In actual application, the post-disaster emergency energy supply optimization scheduling problem for multi-energy coupling distribution systems requires accurate positioning of scheduling problems and the provision of relevant model representations. At the same time, the optimization scheduling process involves both load switching and multi-energy coupling system output control, that is, it involves solving both discrete variables and continuous variables, which is difficult to solve.

[0004] Therefore, how to accurately construct a post-disaster emergency energy supply optimization scheduling model for a multi-energy coupled distribution system and solve the model involving both discrete variables and continuous variables to guide the actual post-disaster emergency energy supply optimization scheduling process is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling distribution system, so as to solve the problem that there is a lack of effective post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling distribution system in the prior art.

[0006] The present invention discloses a post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling power distribution system, the method comprising:

[0007] Taking the maximization of the energy supply benefits of the multi-energy coupling distribution system during the period of time when the external energy supply is lost after the disaster as the goal, the objective function of the post-disaster emergency energy supply optimization scheduling model is constructed;

[0008] The constraints of the post-disaster emergency energy supply optimization scheduling model are constructed by satisfying the constraints related to the multi-energy coupling system, the distribution network operation constraints, the gas network operation constraints, the electric energy storage constraints and the gas tank constraints;

[0009] The problem of solving the post-disaster emergency energy supply optimization scheduling model is constructed as a Markov decision process to determine the state vector, action vector and reward function corresponding to each moment; the Markov decision process is solved using an improved reinforcement learning algorithm, and a reinforcement learning agent that is adapted to the post-disaster emergency energy supply optimization scheduling is trained; the reinforcement learning agent is applied to the post-disaster emergency energy supply optimization scheduling process to obtain the post-disaster emergency energy supply optimization scheduling strategy at each moment.

[0010] On the basis of the above scheme, the present invention also makes the following improvements:

[0011] Furthermore, the objective function of the post-disaster emergency energy supply optimization scheduling model is expressed as:

[0012] (1);

[0013] in, , Respectively represent the park At the moment Actual electricity load supply, actual heat load supply; , Respectively represent the park The importance of electrical load and thermal load; , Respectively represent the park At the moment The electrical load removal state and thermal load removal state; among them, Indicates the park At the moment The electrical load is not removed. Indicates the park At the moment The electrical load is removed; Indicates the park At the moment The heat load is not removed. Indicates the park At the moment Heat load removal; represents the post-disaster optimization scheduling cycle, Indicates the total number of parks;

[0014] (2);

[0015] (3);

[0016] in, Indicates the park Energy hub at all times The electrical energy input power; Indicates the park Energy hub at all times The input power of the combined heat and power unit, Indicates the park Energy hub at all times Input power of gas boiler; , , , Respectively represent the park The power generation efficiency of the transformers connected to the energy hub, the power generation efficiency of the cogeneration unit, the heat generation efficiency of the cogeneration unit, and the heat generation efficiency of the gas boiler.

[0017] Furthermore, the constraints related to the multi-energy coupling system are expressed as:

[0018] (4);

[0019] (5);

[0020] in, , Respectively represent the park Energy hub at all times Electrical energy output and thermal energy output.

[0021] Furthermore, the distribution network operation constraints include active power balance constraints, reactive power balance constraints, voltage drop constraints, relaxed branch power balance relationship constraints, and voltage deviation range constraints allowed by distribution network buses; wherein,

[0022] The active power balance constraint of the distribution network node is expressed as:

[0023] (6);

[0024] in, Indicates that it is mounted on the distribution network node The electric energy storage under The meritorious contribution, Indicates that it is mounted on the distribution network node The photovoltaic power station under The meritorious contribution, Represents the distribution network node The connected multi-energy coupling system at time The active power demand of the electrical load, Represents a distribution network node The root node is at time Flow to distribution network nodes The active power, Represents a distribution network node At the moment Flow to distribution network nodes The distribution network node is the root node Active power of Indicates line At the moment The current, Indicates line The resistance on , represents the total number of distribution network nodes, Represents a distribution network node A collection of distribution network nodes which is the root node;

[0025] The reactive power balance constraint of the distribution network node is expressed as:

[0026] (7);

[0027] in, Indicates that it is mounted on the distribution network node The electric energy storage under The reactive power output, Represents a distribution network node The root node is at time Flow to Node The reactive power, Represents a distribution network node At the moment Flow to distribution network nodes The distribution network node is the root node The reactive power, Indicates line The reactance on

[0028] The voltage drop constraint of the distribution network line is expressed as:

[0029] (8);

[0030] in, , Represents the distribution network nodes , distribution network nodes At the moment Voltage;

[0031] The voltage deviation range constraint allowed by the distribution network bus is expressed as:

[0032] (9);

[0033] in, , Represents the distribution network nodes The upper and lower limits of the voltage.

[0034] Furthermore, the electric energy storage constraint is expressed as:

[0035] (10);

[0036] (11);

[0037] (12);

[0038] in, Represents electrical energy storage At the moment of efforts, , Represents electrical energy storage The upper and lower limits of output; Represents electrical energy storage At the moment The storage capacity, , Represents electrical energy storage The upper and lower limits of the storage capacity; , Represents the total amount of electrical energy stored.

[0039] Furthermore, the gas network operation constraints are expressed as:

[0040] (13);

[0041] (14);

[0042] (15);

[0043] (16);

[0044] in, Indicates natural gas pipeline At the moment The amount of natural gas delivered, Indicates natural gas pipeline The transmission parameters of , Represents the gas network nodes , Gas Network Node At the moment Air pressure; , Represents the gas network nodes The upper and lower limits of air pressure; , Indicates the total number of gas network nodes; Indicates the gas network node The root node is at time Flow to gas network node The amount of natural gas delivered, Indicates the gas network node The mounted gas tank is at The output of natural gas, Indicates the gas network node The natural gas demand of the mounted multi-energy coupling system, Indicates the gas network node The collection of downstream gas network nodes.

[0045] Further, the gas tank constraint is expressed as:

[0046] (17);

[0047] (18);

[0048] (19);

[0049] in, Indicates gas tank At the moment The output of natural gas, , Respectively represent gas tanks The upper and lower limits of natural gas output; Indicates gas tank At the moment of natural gas storage capacity; , Respectively represent gas tanks The upper and lower limits of natural gas storage capacity; , Indicates the total number of gas tanks.

[0050] Further, time The state vector It is expressed as:

[0051] (20);

[0052] in, Indicates the current moment; Indicates that the electric storage can The power storage vector, Indicates the photovoltaic power station at time The output vector of Indicates that the gas tank is at time of natural gas storage, Represents the multi-energy coupling system at time The electric load demand vector; Represents the multi-energy coupling system at time The heat load demand vector; Indicates that the park is at The state vector of electric load removal; Indicates that the park is at The heat load removal state vector;

[0053] time The motion vector It is expressed as:

[0054] (twenty one);

[0055] Indicates that The corresponding maximum value , Indicates that The corresponding maximum value ,

[0056] , ,

[0057] At this time express In certain cases , A set of values ​​for ;

[0058] (twenty two);

[0059] Indicates the time of each park The electric loads are cut off in order from low to high importance. Parks with the same electric load importance are arranged in adjacent numbers.

[0060] , express The sequence number is The importance of the park's electrical load, ,and There is a one-to-one correspondence between the elements in ; Indicates the time of each park The electrical load is cut off in order from low to high heat load importance. Parks with the same heat load importance are arranged in adjacent numbers. , express The sequence number is The importance of the heat load of the park, ,and There is a one-to-one correspondence between the elements in ; express The state transition probability function is express The state transition probability function of Represents an integer greater than 1;

[0061] time The reward function It is expressed as:

[0062] (twenty three);

[0063] in, Indicates time The penalty for violating the constraint.

[0064] Furthermore, the Markov decision process is solved using an improved reinforcement learning algorithm, and a reinforcement learning agent adapted to post-disaster emergency energy supply optimization scheduling is trained; executing:

[0065] The agent outputs the corresponding action vector according to the current state vector; the output action vector is applied to the environment and the gurobi solver is used to solve and determine , and ; Obtain the matching reward function, update the state vector at the next moment, and jump to the step where the agent outputs the corresponding action according to the state vector at the next moment;

[0066] After the iterative training is completed, a reinforcement learning agent that can adapt to the optimal scheduling of post-disaster emergency energy supply is obtained.

[0067] Furthermore, the reinforcement learning agent is applied to the post-disaster emergency energy supply optimization scheduling process to obtain the post-disaster emergency energy supply optimization scheduling strategy at each moment, and execute:

[0068] The reinforcement learning agent outputs the corresponding action according to the current state vector; the output action is applied to the environment, and the gurobi solver is called to solve and determine , and , and obtain the decision variables at the current moment as the optimal scheduling strategy for post-disaster emergency energy supply at the current moment.

[0069] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0070] The post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling power distribution system provided by the present invention is improved in the following aspects:

[0071] (1) A post-disaster emergency energy supply optimization scheduling model for a multi-energy coupled distribution system was established, with the goal of maximizing the energy supply benefits during the period of loss of external energy supply, and the constraints of the model were set according to the actual functional conditions;

[0072] (2) In order to solve the problem of solving both discrete and continuous quantities in the post-disaster emergency energy supply optimization scheduling model, a hybrid action reinforcement learning solution algorithm is proposed to solve the post-disaster emergency energy supply optimization scheduling model. After the model is solved, the discrete action output vector is used to guide the load shedding strategy, and the continuous action output vector is used to guide the output setting of gas storage, electric energy storage, etc., thereby achieving the optimal scheduling of post-disaster emergency energy supply.

[0073] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0075] Figure 1 A flow chart of a post-disaster emergency energy supply optimization and scheduling method for a multi-energy coupling power distribution system provided by an embodiment of the present invention;

[0076] Figure 2 A schematic diagram of the algorithm structure of a discrete-continuous hybrid action reinforcement learning algorithm provided in an embodiment of the present invention;

[0077] Figure 3 A schematic diagram of a circuit topology structure of a multi-energy coupling power distribution system provided by an embodiment of the present invention;

[0078] Figure 4 Training data provided by the embodiment of the present invention;

[0079] Figure 5 Test data provided for the embodiments of the present invention;

[0080] Figure 6 A schematic diagram of a reward convergence curve for a training process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0082] A specific embodiment of the present invention discloses a post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling distribution system, the flow chart of which is as follows: Figure 1 As shown, the specific implementation steps of this method are described as follows:

[0083] Step S1: Taking the energy supply benefit maximization of the multi-energy coupling distribution system during the period of time when the external energy supply is lost after the disaster as the goal, construct the objective function of the post-disaster emergency energy supply optimization scheduling model;

[0084] Step S2: Constructing the constraint conditions of the post-disaster emergency energy supply optimization scheduling model by satisfying the constraints related to the multi-energy coupling system, the distribution network operation constraints, the gas network operation constraints, the electric energy storage constraints and the gas storage tank constraints;

[0085] Step S3: Construct the problem of solving the post-disaster emergency energy supply optimization scheduling model as a Markov decision process, determine the state vector, action vector and reward function corresponding to each moment; use the improved reinforcement learning algorithm to solve the Markov decision process, and train a reinforcement learning agent that is adapted to the post-disaster emergency energy supply optimization scheduling; apply the reinforcement learning agent to the post-disaster emergency energy supply optimization scheduling process to obtain the post-disaster emergency energy supply optimization scheduling strategy at each moment.

[0086] It should be noted that the following assumptions are made for the post-disaster emergency energy supply optimization scheduling problem for a multi-energy coupling distribution system proposed in this embodiment:

[0087] (1) Assuming that after the disaster weather, the multi-energy coupling distribution system loses external energy supply and the internal energy storage is limited, the multi-energy coupling distribution system is in an island operation state;

[0088] (2) Assuming that after disaster weather, the time for the multi-energy coupling distribution system to restore external energy supply is predictable;

[0089] Based on the above assumptions, this embodiment provides the following post-disaster emergency energy supply optimization scheduling strategy for a multi-energy coupled distribution system.

[0090] In this embodiment, the objective function of the post-disaster emergency energy supply optimization scheduling model is constructed with the goal of maximizing the energy supply benefit of the multi-energy coupling distribution system during the period of time when the external energy supply is lost after the disaster. The objective function is expressed as:

[0091] (1);

[0092] in, , Respectively represent the park At the moment Actual electricity load supply, actual heat load supply; , Respectively represent the park The importance of electrical load and thermal load; , Respectively represent the park At the moment The electrical load removal state and thermal load removal state; among them, Indicates the park At the moment The electrical load is not removed. Indicates the park At the moment The electrical load is removed; Indicates the park At the moment The heat load is not removed. Indicates the park At the moment Heat load removal; represents the post-disaster optimization scheduling cycle, Indicates the total number of parks.

[0093] (2);

[0094] (3);

[0095] in, Indicates the park Energy hub at all times The electrical energy input power; Indicates the park Energy hub at all times The input power of the combined heat and power unit, Indicates the park Energy hub at all times Input power of gas boiler; , , , Respectively represent the park The power generation efficiency of the transformers connected to the energy hub, the power generation efficiency of the cogeneration unit, the heat generation efficiency of the cogeneration unit, and the heat generation efficiency of the gas boiler.

[0096] In this embodiment, the constraints of the post-disaster emergency energy supply optimization scheduling model are constructed by satisfying the constraints related to the multi-energy coupling system, the distribution network operation constraints, the gas network operation constraints, the electric energy storage constraints and the gas tank constraints. The specific introduction is as follows;

[0097] (1) Constraints related to multi-energy coupling systems:

[0098] Considering the limited resource reserves of the system, it is necessary to cut off the loads with lower importance at the appropriate time to achieve the goal of supplying energy to the critical loads. Therefore, the relevant constraints of the multi-energy coupling system are as follows:

[0099] (4);

[0100] (5);

[0101] in, , Respectively represent the park Energy hub at all times The electrical energy output and thermal energy output;

[0102] (2) Distribution network operation constraints

[0103] Specifically, the distribution network operation constraints include active power balance constraints, reactive power balance constraints, voltage drop constraints, and voltage deviation range constraints allowed by distribution network buses.

[0104] Since the distribution network line is short and the voltage level is low, its parallel capacitance can be ignored. The DistFlow model is used to describe the power flow constraints of the distribution network in the distribution network related constraints, and the second-order cone relaxation is used to deal with the non-convex terms in the power flow constraints. The distribution network operation constraints in the second-order cone form are established as follows.

[0105] The active power balance constraint of the distribution network node is expressed as:

[0106] (6);

[0107] in, Indicates that it is mounted on the distribution network node The electric energy storage under The meritorious contribution, Indicates that it is mounted on the distribution network node The photovoltaic power station under The meritorious contribution, Represents the distribution network node The connected multi-energy coupling system at time The active power demand of the electrical load, Represents a distribution network node The root node is at time Flow to distribution network nodes The active power, Represents a distribution network node At the moment Flow to distribution network nodes The distribution network node is the root node Active power of Indicates line At the moment The current, Indicates line The resistance on , represents the total number of distribution network nodes, Represents a distribution network node A collection of distribution network nodes with the root node as the root node.

[0108] The reactive power balance constraint of the distribution network node is expressed as:

[0109] (7);

[0110] in, Indicates that it is mounted on the distribution network node The electric energy storage under The reactive power output, Represents a distribution network node The root node is at time Flow to Node The reactive power, Represents a distribution network node At the moment Flow to distribution network nodes The distribution network node is the root node The reactive power, Indicates line The reactance on.

[0111] The voltage drop constraint of the distribution network line is expressed as: (8) ;

[0112] , Represents the distribution network nodes , distribution network nodes At the moment voltage.

[0113] The relaxed branch power balance constraint is expressed as:

[0114] (9);

[0115] The voltage deviation range constraint allowed by the distribution network bus is expressed as:

[0116] (10);

[0117] in, , Represents the distribution network nodes The upper and lower limits of the voltage.

[0118] In order to facilitate the calculation of power flow, the network loss is ignored and LinDistFlow is introduced. The voltage drop constraint (8) in the power flow constraint of the corresponding distribution network can be written as formula (11), and the second-order cone constraint (9) can be ignored.

[0119] (11);

[0120] (3) Constraints on electric energy storage

[0121] The amount of electricity that can be stored in electric energy storage is limited, and the relevant constraints are as follows.

[0122] (12);

[0123] (13);

[0124] (14);

[0125] Represents electrical energy storage At the moment of efforts, , Represents electrical energy storage The upper and lower limits of output; Represents electrical energy storage At the moment The storage capacity, , Represents electrical energy storage The upper and lower limits of the storage capacity; , In the above constraints, equation (12) represents the total amount of electric energy storage The upper and lower limits of the output are constrained; Formula (13) represents the storage capacity of the electric energy storage The expression of the relationship between the power output at the current stage, formula (14) represents the electric energy storage The upper and lower limits of storage capacity.

[0126] (4) Gas network operation constraints

[0127] The gas grid operation constraints established based on the steady-state flow of natural gas are shown below.

[0128] (15);

[0129] (16);

[0130] (17);

[0131] (18);

[0132] in, Indicates natural gas pipeline At the moment The amount of natural gas delivered, Indicates natural gas pipeline The transmission parameters of , Represents the gas network nodes , Gas Network Node At the moment Air pressure; , Represents the gas network nodes The upper and lower limits of air pressure; , Indicates the total number of gas network nodes; Indicates the gas network node The root node is at time Flow to gas network node The amount of natural gas delivered, Indicates the gas network node The mounted gas tank is at The output of natural gas, Indicates the gas network node The natural gas demand of the mounted multi-energy coupling system, Indicates the gas network node The collection of downstream gas network nodes.

[0133] In the above formula, formula (15) represents the natural gas pipeline The natural gas flow equation is: The upper and lower limits of the gas pressure are constrained. Formula (17) represents the upstream gas network node The gas pressure is higher than the downstream gas network node This embodiment assumes that the gas network is a radial gas network, and the flow in the pipeline will maintain a constant flow direction. Therefore, the natural gas pipeline Midstream and upstream gas network nodes The gas pressure is higher than that of the downstream gas network node The air pressure of the gas network node. The natural gas balance equation.

[0134] (5) Gas tank constraints

[0135] The natural gas stored in the gas tank is limited. Equations (19)-(21) represent the constraints related to the distributed gas tank.

[0136] (19);

[0137] (20);

[0138] (twenty one);

[0139] in, Indicates gas tank At the moment The output of natural gas, , Respectively represent gas tanks The upper and lower limits of natural gas output; Indicates gas tank At the moment of natural gas storage capacity; , Respectively represent gas tanks The upper and lower limits of natural gas storage capacity; , Indicates the total number of gas tanks.

[0140] In the above formula, formula (19) represents the gas tank The upper and lower limit constraints of the natural gas output. Formula (20) represents the expression of the relationship between the remaining natural gas in the gas tank and the current consumption, and formula (21) represents the upper and lower limit constraints of the natural gas storage capacity in the gas tank.

[0141] The decision variables in the above post-disaster emergency energy supply optimization scheduling model include: , , , , , , , where the discrete variables are , , continuous variables have , , , , .

[0142] In the post-disaster emergency energy supply optimization scheduling model established in this embodiment, the gas network constraints contain non-convex constraints, the decision variables include both continuous variables and discrete variables, and the multi-energy load and new energy output are uncertain. Therefore, it is difficult to analytically solve this mixed integer nonlinear optimization problem containing multiple uncertainties. The data-driven reinforcement learning method can be trained offline based on historical data and simulation data, and the trained model can be applied to actual scenarios for rapid decision-making. It can handle problems containing uncertainties while having high online solution efficiency. Therefore, this embodiment improves the existing reinforcement learning algorithm and obtains a solution algorithm suitable for the post-disaster emergency energy supply optimization scheduling model in this application, which is specifically introduced as follows.

[0143] First, the optimization problem of the post-disaster emergency energy supply optimization scheduling model is defined as a Markov decision process (MDP). At each moment, the system state is represented by a state vector, and is transferred to the system state at the next moment according to the action taken by the agent and the probability transfer function. Taking certain actions in the system state will generate rewards, which can be defined as the energy supply benefit at each moment according to the objective function. The MDP defined in this embodiment consists of the following parts:

[0144] (1) State vector

[0145] time The state vector is represented as:

[0146] (twenty two);

[0147] in, Indicates the current moment; Indicates that the electric storage can The power storage vector, ; Indicates the photovoltaic power station at time The output vector of ; Indicates that the gas tank is at time of natural gas storage, ; Represents the multi-energy coupling system at time The electric load demand vector, , Representation and Park The multi-energy coupling system connected to the energy hub is Active power demand of electrical load; Represents the multi-energy coupling system at time The heat load demand vector,

[0148] , Representation and Park The multi-energy coupling system connected to the energy hub is The heat load active power demand; Indicates that the park is at The load shedding state vector is ; Indicates that the park is at The heat load removal state vector is .

[0149] (2) Action vector

[0150] time The action vector is expressed as:

[0151] (twenty three);

[0152] in, Indicates that the electric storage can The output vector of ;

[0153] Indicates that the gas tank is at time The output gas vector, ; Indicates that the park is at The load shedding state vector is Indicates that the park is at The heat load removal state vector.

[0154] In order to reduce the dimension of the action vector and thus reduce the complexity of the reinforcement learning decision, the action vector designed in this embodiment does not include , and , when the action vector is determined , the solution of the remaining actions can be placed in the environment. Among them, Indicates that the park's energy hub is at all times The electrical energy input power vector, ; Indicates that the park's energy hub is at all times The input power vector of the CHP unit is, ; Indicates the park Energy hub at all times The input power vector of the gas boiler; .

[0155] In order to reflect the difference in load importance in the load shedding action, this embodiment adjusts the load shedding action, as shown in formula (24), where: Indicates the time of each park The electric loads are cut off in order from low to high importance. Parks with the same electric load importance are arranged in adjacent numbers. , express The sequence number is The importance of the park's electrical load, ,and There is a one-to-one correspondence between the elements in ; The action space is expressed as:

[0156] ,

[0157] This action space has discrete actions can be selected. For example, if any , Take 1 to , it means that at time All electrical loads must be powered; It means that the electric load is removed from Park 1 (the least important); This means that Park 2 (the second least important) will remove the electrical load based on Park 1 (the least important). The thermal load removal action is similar. Indicates the time of each park The electrical load is cut off in order from low to high heat load importance. Parks with the same heat load importance are arranged in adjacent numbers. , express The sequence number is The importance of the heat load of the park, ,and There is a one-to-one correspondence between the elements in ; The action space is expressed as:

[0158] For related meanings, refer to the above electrical load switching content.

[0159] From the above explanation, it can be concluded that the action space in this embodiment is a mixed action space, which has both continuous action , , and discrete actions Therefore, the moment The motion vector It is expressed as:

[0160] (twenty four);

[0161] (3) Reward Function

[0162] time The reward function is expressed as:

[0163] (25);

[0164] The reward function is set based on the objective function, and the moment is added on this basis Constraint violation penalty .

[0165] For the established MDP model, there are two difficulties in directly using the existing reinforcement learning algorithm:

[0166] Difficulty 1: Discrete-mixed actions make it difficult to solve reinforcement learning algorithms. There are several traditional processing methods for mixed action spaces:

[0167] (1) Discretization of continuous actions: The continuous action space is binned / bucketed into a discrete space. Then a discrete action algorithm, such as DQN, is used.

[0168] (2) Continuation of discrete actions: The discrete action space is mapped to the continuous action space through some transformations. Then a continuous action algorithm such as DDPG is used.

[0169] (3) Use multiple agents to optimize discrete actions and continuous actions respectively: for discrete actions, use a discrete algorithm; for continuous actions, use a continuous algorithm. The two algorithms are updated alternately, and each calculates its own gradient. Each finds the optimal discrete action and continuous action.

[0170] There are many problems with the above traditional solutions. Among them, discretization / continuation will change the original action space structure; and the individual optimization will lose the intrinsic connection between discrete actions and continuous parameters; at the same time, the reward function needs to be decomposed into reward values, and improper decomposition will affect the training effect. Based on the above problems, this embodiment proposes a discrete-continuous hybrid action reinforcement learning algorithm for the emergency energy supply scheduling problem of multi-energy coupling distribution system. The algorithm structure diagram is shown in the figure. Figure 2 That is, according to the state variables at the current moment, the action space corresponding to the current moment will be obtained. A set of action spaces is composed of Composition, among which, , , express In certain cases , A set of values ​​for ; express The state transition probability function is express The state transition probability function is a random function between 0 and 1.

[0171] Difficulty 2: The action space is not fixed. , the action space of the load shedding action is not fixed and is related to the state. When some loads have been cut off, the load shedding or energy supply action should no longer be performed. Therefore, the action space of the reinforcement learning agent under different states is not fixed. To address this problem, this embodiment sets an action mask layer (Action Mask Layer) in the neural network to handle the action screening before the discrete action output. The action mask layer is set according to the system state at the moment, and its rules are shown in formula (26). That is, after the first several loads are cut off, the first several discrete action output values ​​generated by the Actor network are set to a large negative number, and will not be selected in the action selection, thereby realizing the adjustment of the action space. The position of the action mask layer in the proposed algorithm is shown in Figure 2 As shown, the action output after adding the action shielding layer is shown in formula (26).

[0172] (26);

[0173] Represents an integer greater than 1.

[0174] At this moment The action vector becomes:

[0175] (27);

[0176] , ,

[0177] At this time express In certain cases , A set of values ​​for ; argmax represents a function that finds the parameter with the maximum score, Indicates that The corresponding maximum value , Indicates that The corresponding maximum value , Indicated in In certain cases , A set of values; at this time Indicates , It is the action vector filtered out after the action shielding layer.

[0178] There are many uncertainties in the post-disaster multi-energy coupled distribution system, which brings great challenges to its optimal scheduling problem. These uncertainties mainly come from the output of new energy, the electric and thermal loads of each park, the initial energy storage content in the system and the initial system topology.

[0179] In step S3, the Markov decision process is solved using the improved reinforcement learning algorithm, and a reinforcement learning agent adapted to post-disaster emergency energy supply optimization scheduling is trained; execute:

[0180] The agent outputs the corresponding action vector according to the current state vector; the output action vector is applied to the environment, and the gurobi solver is used to solve and determine the remaining decision variables ( , and ); find the matching reward function, update the state at the next moment, and jump to the step where the agent outputs the corresponding action according to the state at the next moment;

[0181] After the iterative training is completed, a reinforcement learning agent that can adapt to the optimal scheduling of post-disaster emergency energy supply is obtained.

[0182] The reinforcement learning agent is applied to the post-disaster emergency energy supply optimization scheduling process to obtain the post-disaster emergency energy supply optimization scheduling strategy at each moment; the specific process is as follows:

[0183] The reinforcement learning agent outputs the corresponding action vector according to the current state vector; the output action vector is applied to the environment, and the gurobi solver is called to solve and determine , and , and obtain the decision variables at the current moment as the optimal scheduling strategy for post-disaster emergency energy supply at the current moment.

[0184] This example also designs a 13bus-6node-3ehs multi-energy coupling distribution system to verify the effectiveness of the proposed emergency energy supply scheduling model based on reinforcement learning. The circuit topology diagram of the multi-energy coupling distribution system in this example is shown in Figure 3 As shown in Table 1, the natural gas network is a ringless radial network, the flow direction in the pipeline is given in advance, and the power network is also a ringless radial network. The system contains 2 electric energy storage systems, 12 transmission lines, 1 natural gas storage system and 5 gas pipelines. The parameters of the multi-energy coupling distribution system are shown in Table 1.

[0185] Table 1 Equipment information of the park multi-energy coupling system

[0186]

[0187] Note: kcf is cubic feet. According to the calorific value conversion, this embodiment converts the unit of natural gas power into kW in the park energy conversion equipment CHP and GB, that is, .

[0188] The energy supply interruption time of the multi-energy coupling system is affected by many factors such as the system emergency repair capability and the severity of the disaster. This example sets the expected energy supply interruption time to 30 hours.

[0189] The load importance of each park is different. For example, the load importance of hospital parks is higher, and the load importance of residential parks is lower. The importance of electricity and heat loads of the three parks in the example are shown in Table 2 and Table 3.

[0190] Table 2 Importance of power load, power supply benefits and power outage losses in each park

[0191]

[0192] Table 3 Importance of heat load, heating benefits and power outage losses of each park

[0193]

[0194] The example generates multiple fitting curves by sampling historical data, using 60 sets of curves as test sets and the rest as training sets. The training data and test data are as follows: Figure 4 , Figure 5 shown.

[0195] In order to verify the effectiveness of the reinforcement learning algorithm proposed in this embodiment, this embodiment sets the following four strategies:

[0196] 1) Comparison strategy 1 (Policy-1): Source-side action: give priority to using electric energy storage output to meet the power load demand of each park, and the remaining demand is met by the gas grid; Load-side action: do not actively cut off any load, and passively cut the load when the energy supply is insufficient.

[0197] 2) Comparative Strategy 2 (Policy-2): Source-side action: give priority to using electric energy storage output to meet the power load demand of each park, and the remaining demand is met by the gas grid; Load-side action: actively cut off the least important electric load, and passively cut the load when the energy supply is insufficient.

[0198] 3) Comparison strategy 3 (Policy-opt): Considering the photovoltaic output and the electric and thermal loads of each park during the future external energy supply interruption period, an analytical optimization algorithm is used to solve the action vector. The goal is to maximize the energy supply benefit during the energy supply interruption period. In theory, this method can obtain the optimal solution and maximize the energy supply benefit. Therefore, this embodiment uses this algorithm to verify the performance of the proposed reinforcement learning algorithm.

[0199] 4) Reinforcement Learning Strategy (Policy-RL): The algorithm proposed in this embodiment generates the action vector for this period according to the real-time state vector, with the goal of maximizing the energy supply benefit in all periods. There is no need to obtain prediction information for subsequent periods during the online execution stage.

[0200] The reward convergence curve diagram of the training process is as follows Figure 6 The average energy supply benefits of each strategy on the test set are shown in Table 4. Among them, Policy-1 represents the average energy supply benefit of the comparison strategy 1 on the test set, Policy-2 represents the average energy supply benefit of the comparison strategy 2 on the test set, Policy-opt represents the average energy supply benefit of the comparison strategy 3 on the test set, and Policy-RL represents the average energy supply benefit of the reinforcement learning algorithm proposed in this embodiment on the test set. It can be seen from the reward convergence curve and the average energy supply benefits of each strategy that the reinforcement learning algorithm gradually converges during the training process, and the energy supply benefits are significantly higher than those of the comparison strategies 1 and 2, and the gap from the optimal solution is small.

[0201] Table 4 Average energy supply benefits of different strategies

[0202]

[0203] Next, we select the scenario in the test set as the online execution scenario to analyze the effect of the load shedding action and energy storage resource scheduling strategy formulated by the reinforcement learning algorithm. This scenario is a 30-hour power outage (18:00 to 24:00 the next day). Table 5 shows the energy supply benefits of each strategy in this scenario. It can be seen from the table that the energy supply benefits of the comparison strategy 1 for the third-level load are higher because it does not actively cut off the load, but the energy supply benefits for the first and second-level loads are lower; the comparison strategy 2 actively cuts off the third-level load, but still cannot guarantee the energy supply of the first-level load; compared with the comparison strategy 1 and comparison strategy 2, the benefits of the reinforcement learning strategy are significantly higher than the two comparison strategies, close to the total energy supply benefits of the optimal strategy (comparison strategy 3), and the energy supply benefits of the first-level load of the optimal strategy and the reinforcement learning algorithm are equal, reflecting that the strategy specified by the reinforcement learning algorithm can give priority to guaranteeing the energy supply for the first-level load like the optimal strategy.

[0204] Table 5 Energy supply benefits of different strategies

[0205]

[0206] From the above experimental results, it can be seen that Policy-1 will not actively cut off the load. This strategy will supply energy to all three levels of loads from the 1st hour to the 8th hour, so its profit growth during this period is the fastest among all strategies. In the 8th hour, the electric energy stored in the system under this strategy has been completely consumed. Since there is still a certain amount of natural gas stored at this time, it can still supply the first and second level electric loads. In the 12th hour, the natural gas is also completely consumed, and all the electric and thermal loads of the system are passively cut off. The cumulative profit of this strategy reaches the maximum and begins to decline rapidly.

[0207] Policy-2 actively cuts off the tertiary electricity and heat loads. Therefore, in the early stage of emergency energy supply, the cumulative benefit curve of Policy-2 is significantly lower than that of Policy-1. At the 14th hour, the cumulative benefit of Policy-2 begins to exceed that of Policy-1, but at this time, the power storage under Policy-2 has been consumed, and the secondary electricity load is passively cut off. By the 20th hour, the natural gas is also consumed, and all heat loads are passively cut off. The primary electricity load relies on the photovoltaic field for power supply. At this time, the cumulative benefit curve of Policy-2 turns and begins to decline. By the 24th hour, the photovoltaic output cannot meet the power supply of the primary electricity load, and all loads are cut off.

[0208] Similar to the Policy-2 strategy, the RL strategy also proactively cuts off the third-level electric load and the third-level heat load at the beginning of the emergency energy supply, and cuts off the second-level electric load in advance at the third hour, and cuts off the second-level heat load in advance at the tenth hour, retaining enough electricity and natural gas to supply the first-level load. Compared with the Policy-1 strategy and the Policy-2 strategy, the RL strategy ensures the energy supply to the first-level electric load and the first-level heat load.

[0209] In summary, through typical scenarios, it can be concluded that the RL strategy has a stronger energy supply capability than Policy-1 and Policy-2, and the gap from the optimal strategy (Policy-3) is small, which can ensure continuous energy supply to important loads in the scenario of losing external energy supply.

[0210] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0211] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling distribution system, characterized in that: The method comprises: Taking the maximization of the energy supply benefits of the multi-energy coupling distribution system during the period of time when the external energy supply is lost after the disaster as the goal, the objective function of the post-disaster emergency energy supply optimization scheduling model is constructed; The constraints of the post-disaster emergency energy supply optimization scheduling model are constructed by satisfying the constraints related to the multi-energy coupling system, the distribution network operation constraints, the gas network operation constraints, the electric energy storage constraints and the gas storage tank constraints; The problem of solving the post-disaster emergency energy supply optimization scheduling model is constructed as a Markov decision process to determine the state vector, action vector and reward function corresponding to each moment; the Markov decision process is solved by using an improved reinforcement learning algorithm, and a reinforcement learning agent adapted to the post-disaster emergency energy supply optimization scheduling is trained; the reinforcement learning agent is applied to the post-disaster emergency energy supply optimization scheduling process to obtain the post-disaster emergency energy supply optimization scheduling strategy at each moment; The objective function of the post-disaster emergency energy supply optimization scheduling model is expressed as: (1)； in, , Respectively represent the park At the moment Actual electricity load supply, actual heat load supply; , Respectively represent the park The importance of electrical load and thermal load; , Respectively represent the park At the moment The electrical load removal state and thermal load removal state; among them, Indicates the park At the moment The electrical load is not removed. Indicates the park At the moment The electrical load is removed; Indicates the park At the moment The heat load is not removed. Indicates the park At the moment Heat load removal; represents the post-disaster optimization scheduling cycle, Indicates the total number of parks; (2)； (3)； in, Indicates the park Energy hub at all times The electrical energy input power; Indicates the park Energy hub at all times The input power of the combined heat and power unit, Indicates the park Energy hub at all times Input power of gas boiler; , , , Respectively represent the park The power generation efficiency of transformers connected to the energy hub, the power generation efficiency of cogeneration units, the heat generation efficiency of cogeneration units, and the heat generation efficiency of gas boilers; The constraints related to the multi-energy coupling system are expressed as: (4)； (5)； in, , Respectively represent the park Energy hub at all times The electrical energy output and thermal energy output; time The state vector It is expressed as: (6)； in, Indicates the current moment; Indicates that the electric storage can The power storage vector, Indicates the photovoltaic power station at time The output vector of Indicates that the gas tank is at time of natural gas storage, Represents the multi-energy coupling system at time The electric load demand vector; Represents the multi-energy coupling system at time The heat load demand vector; Indicates that the park is at The state vector of electric load removal; Indicates that the park is at The heat load removal state vector; time The motion vector It is expressed as: (7)； Indicates that The corresponding maximum value , Indicates that The corresponding maximum value , ， ， At this time express In certain cases , A set of values ​​for ; (8)； Indicates the time of each park The electric loads are cut off in order from low to high importance. Parks with the same electric load importance are arranged in adjacent numbers. , express The sequence number is The importance of the park's electrical load, ,and There is a one-to-one correspondence between the elements in ; Indicates the time of each park The electrical load is cut off in order from low to high heat load importance. Parks with the same heat load importance are arranged in adjacent numbers. , express The sequence number is The importance of the heat load of the park, ,and There is a one-to-one correspondence between the elements in ; express The state transition probability function is express The state transition probability function of Represents an integer greater than 1; time The reward function It is expressed as: (9)； in, Indicates time The penalty for violating the constraint.

2. The post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling distribution system according to claim 1 is characterized in that: The distribution network operation constraints include active power balance constraints, reactive power balance constraints, voltage drop constraints, relaxed branch power balance relationship constraints, and voltage deviation range constraints allowed by distribution network buses; wherein, The active power balance constraint of the distribution network node is expressed as: (10)； in, Indicates that it is mounted on the distribution network node The electric energy storage under The meritorious contribution, Indicates that it is mounted on the distribution network node The photovoltaic power station under The meritorious contribution, Represents the distribution network node The connected multi-energy coupling system at time The active power demand of the electrical load, Represents a distribution network node The root node is at time Flow to distribution network nodes The active power, Represents a distribution network node At the moment Flow to distribution network nodes The distribution network node is the root node Active power of Indicates line At the moment The current, Indicates line The resistance on , represents the total number of distribution network nodes, Represents a distribution network node A collection of distribution network nodes which is the root node; The reactive power balance constraint of the distribution network node is expressed as: (11)； in, Indicates that it is mounted on the distribution network node The electric energy storage under The reactive power output, Represents a distribution network node The root node is at time Flow to Node The reactive power, Represents a distribution network node At the moment Flow to distribution network nodes The distribution network node is the root node The reactive power, Indicates line The reactance on The voltage drop constraint of the distribution network line is expressed as: (12)； in, , Represents the distribution network nodes , distribution network nodes At the moment Voltage; The voltage deviation range constraint allowed by the distribution network bus is expressed as: (13)； in, , Represents the distribution network nodes The upper and lower limits of the voltage.

3. The post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling distribution system according to claim 2 is characterized in that: The electric energy storage constraint is expressed as: (14)； (15)； (16)； in, Represents electrical energy storage At the moment of effort, , Represents electrical energy storage The upper and lower limits of output; Represents electrical energy storage At the moment The storage capacity, , Represents electrical energy storage The upper and lower limits of the storage capacity; , Represents the total amount of electrical energy stored.

4. The post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling distribution system according to claim 3 is characterized in that: The gas network operation constraints are expressed as: (17)； (18)； (19)； (20)； in, Indicates natural gas pipeline At the moment The amount of natural gas delivered, Indicates natural gas pipeline The transmission parameters of , Represents the gas network nodes , Gas Network Node At the moment Air pressure; , Represents the gas network nodes The upper and lower limits of air pressure; , Indicates the total number of gas network nodes; Indicates the gas network node The root node is at time Flow to gas network node The amount of natural gas delivered, Indicates the gas network node The mounted gas tank is at The output of natural gas, Indicates the gas network node The natural gas demand of the mounted multi-energy coupling system, Indicates the gas network node The collection of downstream gas network nodes.

5. The post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling distribution system according to claim 4 is characterized in that: The gas tank constraint is expressed as: (21)； (22)； (23)； in, Indicates gas tank At the moment The output of natural gas, , Respectively represent gas tanks The upper and lower limits of natural gas output; Indicates gas tank At the moment of natural gas storage capacity; , Respectively represent gas tanks The upper and lower limits of natural gas storage capacity; , Indicates the total number of gas tanks.

6. The post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling distribution system according to claim 5 is characterized in that: The Markov decision process is solved by using an improved reinforcement learning algorithm, and a reinforcement learning agent adapted to post-disaster emergency energy supply optimization scheduling is trained; implement: The agent outputs the corresponding action vector according to the current state vector; the output action vector is applied to the environment and the gurobi solver is used to solve and determine , and ; Obtain the matching reward function, update the state vector at the next moment, and jump to the step where the agent outputs the corresponding action according to the state vector at the next moment; After the iterative training is completed, a reinforcement learning agent that adapts to the optimal scheduling of post-disaster emergency energy supply is obtained; in, Indicates that the park's energy hub is at all times The electrical energy input power vector, ; Indicates that the park's energy hub is at all times The input power vector of the CHP unit is, ; Indicates the park Energy hub at all times The input power vector of the gas boiler; .

7. The post-disaster emergency energy supply optimization scheduling method for a multi-energy coupling distribution system according to claim 6 is characterized in that: The reinforcement learning agent is applied to the post-disaster emergency energy supply optimization scheduling process to obtain the post-disaster emergency energy supply optimization scheduling strategy at each moment and execute: The reinforcement learning agent outputs the corresponding action according to the current state vector; the output action is applied to the environment, and the gurobi solver is called to solve and determine , and , and obtain the decision variables at the current moment as the optimal scheduling strategy for post-disaster emergency energy supply at the current moment.

Images