Graph reinforcement learning-based wind and light uncertain building integrated energy system optimization method

Abstract

The invention discloses a method for optimizing a wind-solar uncertain building integrated energy system based on graph reinforcement learning. The method comprises the following steps: initializing a feature matrix and an adjacent matrix of system topological graph nodes in an intelligent scheduling environment of the building integrated energy system; the scene analysis method is combined to elaborate the wind and light output uncertainty, and a building integrated energy system master-slave game bilateral optimization model based on the wind and light uncertainty is established; constructing a Markov decision process by considering node topological structure features; and an energy scheduling and pricing strategy is decided according to the topology state information by adopting a depth deterministic strategy gradient algorithm of a multi-head graph attention network of TopkPooling. Aiming at random dynamic events or uncertain factors occurring in the optimization process of the building integrated energy system, the optimal scheduling scheme and pricing scheme can be generated in time according to the change of the environment state.

Description

Technical Field

[0001] The present invention relates to the technical fields of optimization of integrated energy systems for buildings with uncertain wind and light and graph reinforcement learning, and particularly relates to an optimization method for an integrated energy system for buildings with uncertain wind and light based on graph reinforcement learning. Background Art

[0002] Considering the uncertainty of renewable energy is an important energy-saving approach and an effective means to promote the safe and stable operation of integrated building energy systems, which can better reflect the actual situation of the system. However, with the large-scale integration of renewable energy, especially photovoltaic and wind power, due to the combined effects of natural characteristics, environmental factors, geographical conditions and other factors, the output of wind and light has significant random and volatile characteristics, which poses many problems for the power system, such as the inability to effectively regulate the energy supply-demand balance, high energy curtailment costs, and increased operation safety risks. Therefore, considering the impact of its uncertainty is of great significance for optimizing system configuration and operation. In response to the scheduling challenges brought about by the energy uncertainty of integrated energy systems, current research has formed three mainstream methodologies: robust optimization, interval optimization, and stochastic optimization. Among them, robust optimization describes the parameter fluctuation range by constructing an uncertainty set. By defining a robust feasible region, this method limits the fluctuation of uncertainty parameters within a preset fuzzy set to ensure that the optimization solution still satisfies all constraint conditions under the worst perturbation, thus demonstrating excellent robust performance in uncertainty scenarios such as renewable energy output fluctuations and load prediction errors. Some researchers have proposed an optimization strategy based on robust model predictive control to handle the multiple uncertainty problems of sources and loads, and established an integrated rolling optimization framework that combines predicted probability distribution and uncertainty optimization using the Gaussian process regression method to obtain a robust scheduling strategy that minimizes the operating cost. Some researchers have classified the levels of uncertainty scenarios for historical data sets, calculated the initial probability distribution corresponding to each scenario level, randomly selected scenario representatives from each scenario level, and constructed a discrete uncertainty scenario set; although this method fully considers the probability information of the occurrence of uncertainty events, it largely depends on the definition of the uncertainty set. Some researchers have designed a multi-energy management strategy for IES considering the uncertainty of sources and loads with the participation of integrated demand response, and proposed a two-stage scheduling scheme for IES, using the robust optimization method to achieve the optimal economic scheduling of integrated energy microgrids; however, this scheme has a high computational complexity and is difficult to solve the model. Some researchers have proposed a hydrogen refueling station model relying on the integration of new energy hydrogen production and hydrogenation, and used interval optimization improved by a polygon uncertainty set to handle the uncertainty of new energy power generation, but the interval definition cannot adapt to the dynamic changes of the system. In recent years, stochastic optimization scheduling based on scenario analysis methods has received increasing attention. Scenario analysis is a way to analyze the uncertainty problems of energy systems by constructing deterministic scenarios, including scenario generation and scenario reduction. Some researchers have determined the optimal number of clusters by the Elbow method and obtained the typical scenarios of wind speed using the K-means clustering algorithm. Some researchers have proposed a scenario reduction method for wind power output based on the combination of improved K-means clustering and synchronous back substitution elimination algorithm; however, they also have the problem that the deviation between the photovoltaic scenario and the actual scenario is relatively large due to the influence of cloud movement on the light intensity.Other researchers used five Copula functions to establish the joint distribution relationships of wind power and photovoltaic random variables respectively; the Euclidean distance method was used to judge the goodness of fit of each model and determine the optimal Copula function; based on the optimal Copula function, typical application scenarios were generated after sampling. There are also researchers who used the distributionally robust optimization method that combines robust optimization and stochastic optimization to deal with the uncertainty problems of wind and light, effectively ensuring the balance between the robustness and economy of the scheduling plan, but there are limitations that are too conservative in the characterization of flexible supply and demand. In addition, the above-mentioned technical solutions do not consider that the photovoltaic output shows a certain regularity along with the day-night alternation. There are also researchers who mentioned a master-slave game-based optimal scheduling of building integrated energy systems using deep reinforcement learning. Combining the master-slave game framework and the stepped reward and punishment carbon trading mechanism, an optimization model of the building integrated energy system was established, and the DDPG algorithm with an adaptive action exploration mechanism was designed to solve the established model; the experimental results show that the improvement mechanisms in the model and algorithm of this method are effective. Compared with the representative algorithms, the proposed algorithm can obtain better convergence accuracy, stability and total revenue. However, this algorithm does not consider the topological structure of the distribution of energy equipment, and the system revenue and carbon emissions have not reached the best results yet.

[0003] In recent years, graph neural networks have become a popular research direction. It has the ability to fully exploit the topological structure information of the system, can effectively describe the complex non-linear relationships between nodes, and is especially good at feature extraction and training for non-Euclidean data such as trees, graphs, and sequences that cannot be represented by vectors. However, currently, since graph neural networks are mostly only applied in power systems and do not consider the coupling effect of the thermal system on the topological structure, the Graph Convolutional Network (GCN), a classic GNN, extends the convolutional neural network commonly used in the field of deep learning for images to graph data processing by defining the Fourier transform and inverse transform formulas on graphs. However, GCN has certain limitations. On the one hand, within the same-order neighborhood, when too many network layers are stacked, the training effect will decline because the node representations tend to be the same, and this phenomenon is called "over-smoothing"; on the other hand, the aggregation of GCN's neighbor node features is closely related to the specific graph structure, which limits the generalization ability of the trained model on other different graph structure data and is difficult to effectively apply it to the optimization of the wind-light uncertain building integrated energy system. Summary of the Invention

[0004] The object of the present invention is to provide an optimization method for a wind-solar uncertain building integrated energy system based on graph reinforcement learning. After modeling the output of each device, a topology graph of the building integrated energy system is constructed, and then a supply-demand bilateral optimization model is established based on the master-slave game framework; finally, a Markov decision process is constructed for the characteristics of the provided supply-demand bilateral optimization model, and an improvement mechanism is designed and solved based on the graph reinforcement learning algorithm. The present invention can timely generate an optimal scheduling plan and pricing plan according to the change of the environmental state for the random dynamic events or uncertain factors occurring in the optimization process of the building integrated energy system, which plays a great role in promoting the research and development of the building integrated energy system.

[0005] To achieve the above technical object, the technical solution adopted by the present invention is as follows:

[0006] The present invention discloses an optimization method for a wind-solar uncertain building integrated energy system based on graph reinforcement learning, and the method includes the following steps:

[0007] S1: Describe the building integrated energy system as a topological graph structure with energy transmission and distribution units as nodes and energy flow paths as edges, initialize the feature matrix and adjacency matrix of the nodes in the topological graph, and construct a topological graph structure of the building integrated energy system based on graph reinforcement learning;

[0008] S2: Establish a wind-solar output uncertainty model based on the scenario analysis method, comprehensively use the LSTM neural network and the Latin hypercube sampling technique to carry out the multi-scenario generation of the wind-solar output; then use the K-means++ algorithm and the synchronous iteration elimination method to perform scenario reduction processing on the generated several scenarios, and finally obtain multiple typical scenarios of the wind-solar combined output with different probabilities respectively;

[0009] S3: Construct a Markov decision process considering the node topological structure characteristics based on the optimization model of the wind-solar uncertain building integrated energy system based on graph reinforcement learning;

[0010] S4: Use a multi-head graph attention network introducing the TopKPooling pooling layer to represent the topological graph structure, and based on the deep deterministic policy gradient algorithm, decide the energy scheduling and pricing strategies according to the topological state information; specifically, the demand side adjusts the user load according to the master-slave game framework and feeds it back to the supplier to optimize the pricing strategy; among them, the sample information generated during the sampling scheduling process is recorded in the experience replay pool, and until the number of samples in the experience replay pool reaches the standard, the training stage is started to update the network parameters;

[0011] S5: Judge whether the cumulative total revenue of the supplier is stable. If it is stable, output the optimal scheduling plan and pricing plan, and end the process; otherwise, return to step S4.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0013] First, the optimization method for the integrated energy system of wind-solar uncertain buildings based on graph reinforcement learning of the present invention combines the randomness and volatility characteristics of wind energy and solar energy. Based on the scenario analysis method, it comprehensively uses the LSTM neural network and Latin hypercube sampling technology to carry out the multi-scenario generation of wind-solar power output, and establishes a master-slave game bilateral optimization model for the integrated energy system of buildings considering wind-solar uncertainty, so as to be able to more truly reflect the operation of the integrated energy system of buildings and improve the system's response ability.

[0014] Second, the optimization method for the integrated energy system of wind-solar uncertain buildings based on graph reinforcement learning of the present invention proposes a deep deterministic policy gradient algorithm MHGAT-DDPG based on a multi-head graph attention network. According to the characteristics of the problem, it constructs an energy system topology state graph, designs a new Markov decision process according to the topological structure of the nodes, defines the state space, action space and reward function considering the state graph, and uses a multi-head graph attention network introducing a TopKPooling pooling layer to represent the topological graph structure, and fuses the multi-head graph attention network with the deep deterministic policy gradient (Deep Deterministic Policy Gradient, abbreviated as DDPG) algorithm to improve the performance of the algorithm for solving the model, so as to effectively depict the topological structure of the system itself and represent the complex non-linear relationship between nodes.

[0015] Third, compared with the existing representative algorithms, the proposed algorithm MHGAT-DDPG of the optimization method for the integrated energy system of wind-solar uncertain buildings based on graph reinforcement learning of the present invention can obtain better convergence accuracy, convergence speed, stability and total revenue. Description of the Drawings

[0016] Figure 1 is the flow chart of the optimization method for the integrated energy system of wind-solar uncertain buildings based on graph reinforcement learning of the present invention;

[0017] Figure 2 is the topological structure diagram of a 14-node electro-thermal coupled building integrated energy system;

[0018] Figure 3 is the schematic diagram of the MHGAT-DDPG network structure based on the multi-head graph attention mechanism;

[0019] Figure 4 is the solution framework diagram of the MHGAT-DDPG algorithm;

[0020] Figure 5Schematic diagram of the power scheduling result for the energy operator; the line graph shows the user load corresponding to the demand side in each time period, and the bar graph shows the energy scheduling plan for the supply side in each time period;

[0021] Figure 6 For the heat energy scheduling result of the energy operator; the line graph shows the user load corresponding to the demand side in each time period, and the bar graph shows the energy scheduling plan for the supply side in each time period. Specific implementation manner

[0022] The following further describes the embodiments of the present invention in detail with reference to the accompanying drawings.

[0023] See Figure 1 , the present invention discloses an optimization method for a wind-solar uncertain building integrated energy system based on graph reinforcement learning, and the method includes the following steps:

[0024] S1: Describe the building integrated energy system as a topological graph structure with energy transmission and distribution units as nodes and energy flow paths as edges, initialize the feature matrix and adjacency matrix of the nodes in the topological graph, and construct a topological graph structure of the building integrated energy system based on graph reinforcement learning;

[0025] S2: Establish a wind-solar output uncertainty model based on the scenario analysis method, comprehensively use the LSTM neural network and the Latin hypercube sampling technology to carry out the multi-scenario generation of wind-solar output; then use the K-means++ algorithm and the synchronous iteration elimination method to perform scenario reduction processing on the generated several scenarios, and finally obtain multiple typical scenarios of wind-solar combined output with different probabilities respectively;

[0026] S3: Based on the optimization model of the wind-solar uncertain building integrated energy system based on graph reinforcement learning, construct a Markov decision process considering the characteristics of the node topological structure;

[0027] S4: Use a multi-head graph attention network with an introduced TopKPooling pooling layer to represent the topological graph structure, and based on the deep deterministic policy gradient algorithm, make decisions on energy scheduling and pricing strategies according to the topological state information; specifically, the demand side adjusts the user load according to the master-slave game framework and feeds it back to the supplier to optimize the pricing strategy; among them, the sample information generated during the sampling scheduling process is recorded in the experience replay pool, and until the number of samples in the experience replay pool reaches the standard, the training stage is started to update the network parameters;

[0028] S5: Judge whether the cumulative total revenue of the supplier is stable. If it is stable, output the optimal scheduling plan and pricing plan, and end the process. Otherwise, return to step S4.

[0029] See Figure 1 , the method specifically includes the following steps:

[0030] Step S1, construct a topological graph structure of the building integrated energy system based on graph reinforcement learning.

[0031] Building integrated energy system is required to complete the scheduling within a limited time (e.g., a scheduling point), and for each scheduling, perform the scheduling of the power system and the thermal system, as well as formulate the electricity price and the heat price. According to the actual situation of the building area, collect the historical comprehensive data (including photovoltaic historical data) of the target area, perform data cleaning and normalization processing on the historical comprehensive data, establish a wind-solar power output uncertainty model by using the scenario analysis method, and determine the characteristic data related to the variation law of the node scheduling data and price information of the power system and the thermal system with wind and solar.

[0032] Essentially, the building integrated energy system can be described as a topological graph structure with energy transmission and distribution units (such as power generation nodes, intermediate nodes) as nodes and energy flow paths as edges. This node topological graph structure can clearly represent the spatial relationship of energy during production, transmission, and distribution. After considering the node topological structure of the building integrated energy system, it provides an intuitive modeling basis for the optimal scheduling and operation management of the supply-side system.

[0033] Step S2, establish a wind-solar power output uncertainty model based on the scenario analysis method to obtain multiple typical scenarios of wind-solar combined power output with different probabilities respectively.

[0034] Specifically, the uncertainty of wind-solar power output is described in combination with the scenario analysis method, including the following steps:

[0035] S201, obtain the historical photovoltaic power output data, predict N groups of photovoltaic power output scenarios through LSTM training, and sample M groups of wind power output scenarios based on the Latin hypercube sampling method;

[0036] S202, obtain the initial clustering centers of M wind power output scenarios according to the maximum-minimum distance principle and use the clustering algorithm to reduce them to obtain K groups of wind power output scenarios;

[0037] S203, perform a Cartesian connection on the generated N groups of photovoltaic power output scenarios and K groups of wind power output scenarios to generate N*K groups of wind-solar combined power generation scenarios;

[0038] In S204, through the synchronous iterative elimination method, the generated N*K sets of wind-solar combined power generation scenarios are reduced, and 5 sets of final wind-solar combined power generation scenarios and their probabilities are output. The master-slave game bilateral optimization model of the building integrated energy system based on wind-solar uncertainty established in this step includes the objective function of the cumulative total revenue of the energy supplier in the building integrated energy system, the power balance constraints of electricity / heat, the resource conversion constraints, the price constraints, the power system power flow constraints, the thermal system balance constraints, the node voltage over-limit constraints, and the operation constraints of each device and network.

[0039] In step S201, the M sets of wind power output scenarios sampled based on the Latin hypercube sampling method are obtained by sampling the wind speed probability distribution using the Latin hypercube sampling method to obtain M sets of wind speed samples, and then inputting the wind speed samples into the wind turbine to convert them into wind power output, resulting in M sets of discrete wind power output scenarios, as shown in Equation (1):

[0040] The probability density function of the Weibull distribution of wind speed is:

[0041] (1);

[0042] Among them, is the actual wind speed, is the scale parameter, is the shape parameter.

[0043] The output model of the wind turbine is:

[0044] (2);

[0045] Among them, is the wind power output at time t, is the rated output power of the wind power; , , and are the actual wind speed, cut-in wind speed, cut-out wind speed, and rated wind speed at time t, respectively.

[0046] In this embodiment, the supply-side optimization model considering the node topology structure is as Figure 2 shown, which is a topological structure diagram of a 14-node electro-thermal coupled building integrated energy system. Figure 2In it, the blue part is the power system. Nodes 1 to 6 are respectively connected to a distributed generation unit (abbreviated as DG unit), a photovoltaic generation unit (abbreviated as PV unit), an electric boiler (abbreviated as EB unit), a wind turbine unit (abbreviated as WT unit), an energy storage system (abbreviated as ESS unit), and a combined heat and power unit (abbreviated as CHP unit). The red part is the thermal system. Node 7 is connected to a gas-fired boiler unit (abbreviated as GB unit) and the CHP unit, and node 10 is connected to a thermal energy storage system (TES unit). After considering the node topology of the building integrated energy system, the operation constraints on the supply side will increase, and the state space in the Markov decision process will also change accordingly, while the demand side is not affected.

[0047] Under the constraints of the nodes in the topology diagram, a reasonable energy scheduling strategy and price-making strategy are designed to maximize the total revenue of the Building Energy Operator (abbreviated as BEO). The objective function is shown in Equation (3): (3); Where, is the total number of time periods in a day, is the revenue of the BEO from selling electric energy and thermal energy to users at time t , is the revenue of the BEO from selling the surplus electric energy to the superior power grid at time t , is the carbon trading cost of the BEO at time t ; , and are respectively the energy purchase cost, operation and maintenance cost, and energy storage cost of the BEO from the superior power grid and gas grid at time t .

[0048] (4);

[0049] Where, i represents electric energy, thermal energy, and cooling energy, represents the corresponding user load, represents the selling price of the i-th type of energy, represents the power of the BEO selling electricity to the superior power grid, Represents the price of selling electricity to the superior power grid. The objective function of the lower-layer model of the building integrated energy system generated is shown in Equation (5):

[0050] (5);

[0051] Among them, and respectively represent the power of purchasing electricity and gas from the superior power grid, and are the prices of purchasing electricity and gas respectively.

[0052] (6);

[0053] Among them, represents the unit operation and maintenance cost of the equipment, represents the active power of node i at time t.

[0054] The constraint conditions of the generated building integrated energy system are as follows:

[0055] (4-1) Energy and power balance constraints

[0056] (7);

[0057] (8);

[0058] (9);

[0059] Among them, and respectively represent the total charging power and total discharging power of the produced electric energy; and respectively represent the total charging power and total discharging power of the produced thermal energy; , and are the distribution coefficients of the directly supplied electric load, thermal load, and cooling load of the produced electric energy respectively; and are the produced electric energy and thermal energy respectively; and are the distribution coefficients of the directly supplied thermal load and cooling load of the produced thermal energy respectively; and respectively represent the electric load and thermal load of the demand-side response at time t; , and are the transferable electric load, transferable-out electric load, and reducible electric load respectively; and respectively represent the total charging power and total discharging power of the produced cooling energy; Represents the cooling load corresponding to the demand side at time t.

[0060] (4-2) Equipment power constraint

[0061] (10);

[0062] (11);

[0063] Among them, and respectively represent the power of equipment m at time t and t-1, represents the maximum power of equipment m, represents the maximum ramp power of equipment m.

[0064] (4-3) Charging and discharging power limits of the ES unit

[0065] (12);

[0066] (13);

[0067] (14);

[0068] Among them, represents the total charging power of the energy storage device, represents the total discharging power of the energy storage device, and respectively represent the maximum charging power and the maximum discharging power, and respectively represent the equipment energy storage at time t and t-1, and respectively represent the charging and discharging efficiencies.

[0069] (4-4) Demand response constraint

[0070] (15);

[0071] (16);

[0072] (17)

[0073] Among them, represents the proportion of the maximum transferable load of users. Demand response itself is a classic application of dynamic scheduling. With the time-varying time-of-use electricity price fluctuating up and down, the transferable load of users is adjusted accordingly, reflecting dynamic optimization.

[0074] (4-5) User load constraint (18);

[0076] (19);

[0077] Among them, represents the discomfort coefficient for transferring and reducing the i-th type of energy, represents the ideal load for the user's most comfortable state, represents the actual load. For transferable loads, there is:

[0078] (20);

[0079] (21);

[0080] (22);

[0081] Among them, represents the upper limit value of the transferable load, represents the total amount of transferable load.

[0082] (4 - 6) AC power system power flow constraints

[0083] (23);

[0084] (24);

[0085] (25);

[0086] Among them, and respectively represent the active power and reactive power of node i at time t, represents the voltage amplitude of node i at time t, , , and respectively represent the active power generation outputs of the DG unit, CHP unit, PV unit, and WT unit at node i at time t, , respectively represent the charging and discharging power of the energy storage device at node i at time t, represents the active power of the electric heating of the EB unit at node i at time t; is the active load at node i at time t, , , and respectively represent the reactive power generation outputs of the DG unit, CHP unit, PV unit, and WT unit at node i at time t, is the reactive load at node i at time t, Denotes the conductance between node i and node j, Denotes the susceptance between node i and node j, Denotes the phase angle difference between node i and node j at time t , and respectively denote the power sold by the BEO to the power grid and the power purchased from the power grid at time t.

[0087] (4 - 7) Thermal system balance constraint:

[0088] (26);

[0089] Among them, Denotes the thermal power of node i at time t, Is the specific heat capacity of water, Denotes the injection flow rate of node i at time t, , Are respectively the supply water temperature and the return water temperature of node i at time t, and Respectively denote the water temperatures at the i - end and the j - end of the pipeline branch ij at time t, Is the external environmental temperature at time t, λ is the thermal conductivity, Denotes the length of the pipeline branch ij, Denotes the mass flow rate of the pipeline branch ij, Denotes the mass flow rate flowing from node k to node i, Denotes the water temperature when flowing from node k to node i at time t, Denotes the water temperature after mixing of the return water at node i at time t, Denotes the total number of nodes flowing to node i.

[0090] (4 - 8) Node voltage violation constraint

[0091] (27);

[0092] Among them, and Are respectively the minimum and maximum values of the safe voltage of the system nodes.

[0093] Step S3, based on the wind - solar uncertain building integrated energy system optimization model of graph reinforcement learning, construct a Markov decision process considering the node topological structure characteristics.

[0094] Based on the topological graph in step S1, establish a Markov decision process represented by the quadruple Which respectively represent the state space, action space, reward function, and state transition probability. ​

[0095] The state - space design process is as follows:

[0096] Model the building integrated energy system as a graph , and the system operation state information is reflected by the node features and edge features in the graph. The state includes the graph adjacency matrix and the node feature matrix , where n represents the number of nodes, and f represents the number of node features. is an -dimensional graph adjacency matrix, representing the connection relationship between n nodes in the topological graph. Indicates that there is an edge connection between node and node , while indicates that there is no edge connection. Since the node features include the node voltage at time , the node active power , the node reactive power , the node load , the electricity purchase price of users and the gas purchase price , so f = 6. The i - th row in the node feature matrix represents the 6 - dimensional feature vector of the i - th node , as shown in Equation (28).

[0097] (28);

[0098] where, at the i - th node at time t, is the node feature vector, is the node voltage, is the node active power, is the node reactive power, is the node load, is the electricity purchase price of users, is the gas purchase price of users.

[0099] The reward function is designed as follows:

[0100] The operation constraints of each device in the system, the node voltage constraints with power flow balance, the energy network constraints, etc. can be added to the reward function in the form of penalty functions, as shown in Equation (29).

[0101] (29);

[0102] where, is the total penalty function for all nodes in the system violating the constraints at time t, , , , , , and are the penalty functions for the DG unit, CHP unit, PV unit, WT unit, EB unit, power energy storage system, and heat energy storage system violating the constraints at the th node at time t, represents the penalty function for the voltage violating the constraints at the

[0103] The penalty function for the DG unit violating the equipment operation constraints at time t is expressed as Equation (30).

[0104] (30);

[0105] where represents the active power generation output of the DG unit at node i at time t, represents the minimum value of the active power generation output of the DG unit at node i, represents the maximum value of the active power generation output of the DG unit at node i; the penalty function terms for other units violating the operation constraints at time t have the same form as that of the DG unit, and when the constraints are satisfied, the penalty term is 0.

[0106] Therefore, the immediate reward function of the agent at time t

[0107] (31);

[0108] where is the penalty factor. To ensure that the training results satisfy the constraint conditions as much as possible, a relatively large penalty factor is generally set. In this embodiment, is taken as 100.

[0109] The trajectory information is a five-tuple ; where is the flag for judging whether the scheduling is terminated. When the current timestamp reaches 24 hours, the optimization ends, ; otherwise, .

[0110] Since the graph data in the multi-head graph attention network layer usually contains a large number of nodes and edges, the computational complexity for processing is huge. At this time, the TopKPooling pooling layer is introduced, which can downsample the graph by controlling the ratio parameter, reduce the number of nodes, and thus reduce the computational complexity of the subsequent network layer.

[0111] Figure 3The network structures of the actor current network and the critic current network that introduce the multi-head graph attention mechanism in the proposed algorithm MHGAT-DDPG are given. They consist of two layers of multi-head graph attention layers (MHGAT layer1 and MHGAT layer2), two layers of TopKPooling layers, and a multi-layer perceptron (MLP, Multilayer Perceptron) with four fully-connected layers.

[0112] The main role of the multi-head graph attention layer is to enhance the model's ability to represent graph-structured data through the multi-head attention mechanism, so as to more comprehensively capture the relationships between BIES nodes and the topological structure of the graph. The TopKPooling layer embedded after the multi-head graph attention layer can help the model reduce the complexity of the graph, focus on the most representative parts of the graph, extract more valuable features, and thus improve the performance and generalization ability of the model. The role of finally connecting the MLP is to further process and transform the node features, adjust the dimensions and apply them to the subsequent processes of DDPG, while improving the interpretability and stability of the model. Since the input of the critic network in the original DDPG process includes the state and action, in this chapter, the output (action ) of the multi-head graph attention actor current network is sent to the input of the MLP of the critic current network and concatenated with the current state to calculate the Q value. Finally, the Q value is returned to the actor current network to guide the update of the parameters. The network structures of the actor target network and the critic target network that introduce the multi-head graph attention mechanism are similar to Figure 1 , the difference is that their inputs and outputs are Markov information at the next moment, aiming to improve the stability of the MHGAT-DDPG algorithm.

[0113] The solution framework of the proposed algorithm MHGAT-DDPG in this embodiment is as shown in Figure 4 , which consists of a multi-head graph attention network (Multi-Head Graph Attention Network, abbreviated as MHGAT) and a deep deterministic policy gradient DDPG. Figure 4 In, first construct four graph attention networks: (1) The multi-head graph attention actor current network, whose input is the state at time , that is, the graph adjacency matrix mapped through the topological state graph and the node feature matrix containing information such as node voltage and power at time t. (2) The multi-head graph attention actor target network, whose input is the state at time , including and Node feature matrix at a moment The outputs of the above two actor networks are the states and a certain action in the lower action space respectively. This action is a 10-dimensional vector, including the energy scheduling strategy and the energy pricing strategy. (3) The multi-head graph attention critic current network. (4) The multi-head graph attention critic target network. After the two critic networks extract the corresponding matrices of the states and through the graph attention layer and the pooling layer, they are concatenated with the actions and output by the corresponding actor network respectively, that is, the input is and . The final outputs of the two critic networks are the current Q value and the target Q value respectively, that is, the expected discounted reward value that can be obtained after executing a certain action in a given state, which is used to evaluate the action.

[0114] Step S4, use the multi-head graph attention network introducing the TopKPooling pooling layer to represent the topological graph structure, and based on the deep deterministic policy gradient algorithm, make decisions on the energy scheduling and pricing strategies according to the topological state information.

[0115] The graph attention network (Graph Attention Network, abbreviated as GAT) introduces an attention mechanism on the basis of GCN, thus improving the way of GCN operation. GAT uses the attention mechanism to perform weighted summation on the features of neighboring nodes. In this process, the weights of the features of neighboring nodes are completely determined by the features of the nodes themselves, and do not depend on the specific structure of the graph. Specifically, GAT linearly accumulates the feature representations of the node neighborhood according to the calculated attention weights to obtain the final output result. In order to reduce the impact of a single attention head on the overall performance of the model and improve the stability and robustness of the model, the multi-head attention mechanism is introduced. The multi-head graph attention network uses multiple independent attention heads in parallel, and each attention head can learn different structural information and feature patterns in the graph data. Through the combination of multiple attention heads, the model can capture richer and more comprehensive node state information in the BEO topological graph, thereby enhancing the feature expression ability of the graph attention network. In addition, since the graph data in the multi-head graph attention network layer usually contains a large number of nodes and edges, the computational complexity is huge when processing. At this time, the TopKPooling pooling layer is introduced, which can downsample the graph by controlling parameters to reduce the number of nodes, thereby reducing the computational complexity of the subsequent network layer.

[0116] The decision-making model is trained using a deep deterministic policy gradient algorithm based on a multi-head graph attention network. The algorithm framework consists of a multi-head graph attention network and the deep deterministic policy gradient DDPG, including a multi-head graph attention actor current network, a multi-head graph attention actor target network, a multi-head graph attention critic current network, and a multi-head graph attention critic target network.

[0117] The graph attention network uses an attention mechanism to perform weighted summation on the features of neighboring nodes. In order to accurately extract the weight information between nodes in the energy topology graph, first, through the weight parameter matrix of the neural network the feature vectors of nodes i and j and are linearly transformed, and the transformed feature vectors and are concatenated to form a new node feature matrix. Secondly, the new node feature matrix is mapped to the real number space using a feed-forward neural network and activated through the function to obtain the attention score of node j for node i, as shown in Eqs. (32)-(33).

[0118] (32);

[0119] (33);

[0120] Among them, represents the independent variable of the LeakyRelu function, and the score reflects the relative importance of neighboring node j to node i, is a learnable attention vector, which serves to linearly combine the concatenated feature vectors; is the vector concatenation operator, and the superscript T represents the transpose; is a positive constant less than 1, which determines the slope for negative inputs. In this embodiment, the value is 0.01.

[0121] After that, the function is used to normalize the attention scores to obtain the attention coefficients , as shown in Eq. (34).

[0122] (34);

[0123] In the training of the GAT model, using the above-obtained attention coefficients , the nodes are updated through information transfer aggregation between nodes, as shown in Eq. (35).

[0124] (35);

[0125] Among them, represents the feature vector of node i in the l-th layer network, represents the set of neighbor nodes of node i, is the activation function, and the commonly used one is the ReLu function; is the weight matrix.

[0126] Use the multi-head attention mechanism to obtain the feature vector The calculation method of is as shown in Equation (36):

[0127] (36);

[0128] Among them, K is the number of attention heads, is the weight parameter matrix of the k-th attention head, is the node in the k-th attention head for node attention coefficient.

[0129] Step S4 includes two stages: the sampling stage and the training stage. In the sampling stage, randomly take the topological state and input it into the multi-head graph attention actor in the current network. The actor in the current network selects an action using the adaptive action exploration mechanism according to the state, outputs the corresponding reward after interacting with the BEO environment, and observes the next state, forming a quadruple sample and storing it in the experience replay pool D; repeat the loop until the number of samples in the experience replay pool reaches the scale L, and enter the training stage. In the training stage, the agent executes the Markov decision process through continuous interaction with the BEO environment, and samples the smallest batch of samples from the experience replay pool and inputs them into 4 networks to continuously update the parameters of the networks. The specific update process includes:

[0130] On the basis of step S2, collect the variance of the rewards in the first 10 generations and the average loss of the critic network. Normalize the variance of the rewards in the first 10 generations and the average loss of the critic network to unify the dimension. Construct a decreasing function based on the variance of the rewards in the first 10 generations and the average loss of the critic network as the adaptive noise coefficient before the agent selects an action. The user adjusts the electricity load according to their own best response strategy, updates the load state information for the next step, and realizes the interaction between the user and the energy operator. The agent uses the improved adaptive noise mechanism to select actions, generates a series of trajectory information such as rewards, and stores them in the experience replay pool.

[0131] When the number of trajectories in the experience replay pool reaches the limit, sample the trajectories in the pool according to the minimum sampling batch to supply the current critic network and the critic target network for Q-value prediction. The target critic network will pass the predicted Q-values to the current critic network for gradient descent training. The current critic network with trained neural network parameters will regularly copy the parameters to the target critic network to improve the prediction ability of the target critic network. Update the current Actor network using the gradient ascent method according to the current Q-values, and update the current Actor network to maximize the Q-value output. The current Actor network will regularly copy the parameters to the target Actor network. The current Actor network will select the action with the maximum Q-value according to the state features and execute it.

[0132] The present invention combines the randomness and volatility characteristics of wind energy and solar energy to establish a master-slave game bilateral optimization model for a building integrated energy system considering the uncertainty of wind and light. Based on the scenario analysis method, comprehensively using the LSTM neural network and the Latin hypercube sampling technology, carry out the multi-scenario generation of wind and light output. Subsequently, use the K-means++ algorithm and the synchronous iteration elimination method to perform scenario reduction processing on the generated numerous scenarios, and finally obtain a small number of typical scenarios of wind-light combined output with different probabilities and analyze them. To effectively characterize the topological structure of the system itself and represent the complex non-linear relationship between nodes, a deep deterministic policy gradient algorithm MHGAT-DDPG based on a multi-head graph attention network is proposed. First, construct an energy system topological state graph according to the problem characteristics. Second, design a new Markov decision process according to the topological structure of the nodes, and define the state space, action space, and reward function considering the state graph. Third, use a multi-head graph attention network with an introduced TopKPooling pooling layer to represent the topological graph structure. Finally, fuse the multi-head graph attention network with the DDPG algorithm to improve the performance of the algorithm for solving the model. The experimental results show that the designed scenario analysis method and the multi-head graph attention network are effective; compared with the representative algorithms, the proposed algorithm MHGAT-DDPG can obtain better convergence accuracy, convergence speed, stability, and total revenue.

[0133] Based on the scenario analysis method, this invention combines the LSTM neural network to model the historical photovoltaic output data of the past year and predicts the typical scenarios of photovoltaic output. At the same time, the Latin hypercube sampling technique is used to generate the typical scenarios of wind power output. On this basis, the k-means++ clustering algorithm and the synchronous iteration elimination method are adopted to reduce the typical scenarios of wind-solar output, and finally a small number of wind-solar combined typical scenarios with different probabilities are obtained. By calculating the total BEO revenue of each entity in each typical scenario and accumulating them after multiplying by the corresponding probabilities, a comprehensive analysis of the system is carried out. Thus, a master-slave game two-sided optimization model for building integrated energy systems considering the uncertainty of wind-solar output is constructed. In addition, this invention combines the graph neural network and deep reinforcement learning, and proposes a graph deep reinforcement learning algorithm to solve the established model. The proposed algorithm represents the state information of the building integrated energy system in the form of a state graph containing an adjacency matrix and a node feature matrix, enabling the intelligent agent to fully utilize the topological structure information of the system for real-time decision-making and optimization when facing a complex and changeable energy scheduling environment, so as to accurately capture the dynamic relationships and energy flow characteristics between nodes in the energy network, and further improve the overall efficiency of the system and reduce carbon emissions, providing strong technical support for the realization of a green and efficient building integrated energy system.

[0134] Example

[0135] The parameters of the wind-solar uncertainty model and the two-sided optimization scheduling model considering the node topological structure in this example are as follows:

[0136] , , , , , , , , , , , .

[0137] In the system corresponding to the topological structure of this example, the reference capacity S = 100 MVA and the reference voltage U = 23 KV are set. The relevant parameters of the power system lines are shown in Table 1. Figure 2 The red part in represents the thermal system. This thermal system maintains the water supply temperature constant at 100 °C and the external pipe environment temperature constant at 10 °C. The specific parameters of the thermal network pipes are shown in Table 2.

[0138] Table 1 Power System Parameters Branch Number Busbar Start Point Busbar End Point Resistance / p.u Reactance / p.u 1 1 2 0.11 0.17 2 1 4 0.075 0.26 3 2 3 0.09 0.20 4 4 5 0.08 0.04 5 5 6 0.04 0.14

[0139] Table 2 Thermal System Parameters Pipeline Number Pipeline Start Point Pipeline End Point Length / m Flow Rate (kg / s) Thermal Conductivity (W / m*K) 1 1 2 350 265.8 0.12 2 2 3 175 241.4 0.12 3 3 4 175 143.58 0.12 4 2 5 75 24.4 0.12 5 3 6 175 97.82 0.12 6 7 1 350 265.8 0.12 7 5 7 75 24.4 0.12 8 8 7 175 241.4 0.12 9 6 8 175 97.82 0.12 10 4 8 175 143.58 0.12

[0140] Figure 5 and Figure 6 shows the scheduling results of BEO's electric energy and heat energy. The line chart gives the user load corresponding to the demand side in each time period, and the bar chart gives the energy scheduling plan of the supply side in each time period.

[0141] In Figure 5 , taking 0:00 - 1:00 as an example: the load on the demand side is 204 kwh; BEO sells 70 kwh of electric energy to the superior power grid, and generates 40 kwh of electric energy through the EB unit. The ESS unit is in the charging state and charges 60 kwh of electric energy; at the same time, the DG unit, CHP unit and WT unit respectively provide 60 kwh, 80 kwh and 234 kwh of electric energy to the demand side. The electric load on the demand side is in the low valley state during the night time period; during the peak electricity consumption periods at noon and in the evening, the electric load on the demand side increases significantly, reaches the highest value at 14:00, and after 18:00, the electric load on the demand side shows a decreasing trend. It should be noted that during the night, the energy storage system ESS is in the charging mode and does not start discharging until the peak electricity consumption period during the daytime working hours. This embodiment takes into account the uncertainty of wind and light. Since the profit obtained from selling electricity to the superior power grid at noon is higher, the surplus electricity (ranging from 20 - 100 kwh) after the output of wind and light at noon is sold to the superior power grid in time (at 15:00), which can enable BEO to obtain higher profits while reducing the cost of abandoning wind and light. On the contrary, Figure 6 the peak heat consumption period on the demand side is at night, and the heat load on the demand side during the day is relatively low, reaching the lowest value at 11:00. This is because the sunlight is sufficient and the temperature is high during the noon period, and users do not have too much additional heat energy demand.

[0142] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be interpreted to include the preferred embodiments as well as all changes and modifications falling within the scope of the present application.

[0143] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these changes and modifications therein.

Claims

1. An optimization method for a wind-solar uncertain building integrated energy system based on graph reinforcement learning, characterized in that, The method includes the following steps: S1: Describe the building integrated energy system as a topological graph structure with energy transmission and distribution units as nodes and energy flow paths as edges. Initialize the feature matrix and adjacency matrix of the nodes in the topological graph, and construct a topological graph structure of the building integrated energy system based on graph reinforcement learning. S2: Establish an uncertainty model for the output of wind and light based on the scenario analysis method. Comprehensively use the LSTM neural network and Latin hypercube sampling technology to carry out the multi-scenario generation of the output of wind and light. Then, use the K-means++ algorithm and synchronous iteration elimination method to perform scenario reduction processing on the generated scenarios, and finally obtain multiple typical scenarios of the combined output of wind and light with different probabilities. S3: Based on the optimization model of the building integrated energy system with uncertain wind and light based on graph reinforcement learning, construct a Markov decision process considering the characteristics of the node topological structure. S4: Use a multi-head graph attention network with a TopKPooling pooling layer to represent the topological graph structure. Based on the deep deterministic policy gradient algorithm, make decisions on energy scheduling and pricing strategies according to the topological state information. Specifically, the demand side adjusts the user load according to the master-slave game framework and feeds it back to the supplier to optimize the pricing strategy. Among them, the sample information generated during the sampling scheduling process is recorded in the experience replay pool. After the number of samples in the experience replay pool reaches the standard, the training phase is started to update the network parameters. S5: Judge whether the cumulative total revenue of the supplier is stable. If it is stable, output the optimal scheduling plan and pricing plan, and end the process. Otherwise, return to step S4.

2. The optimization method for a wind-solar uncertain building integrated energy system based on graph reinforcement learning according to claim 1, wherein Step S1 further includes: According to the actual situation of the building area, collect the historical comprehensive data of the target area, perform data cleaning and normalization processing on the historical comprehensive data, describe the building integrated energy system as a topological graph structure with energy transmission and distribution units as nodes and energy flow paths as edges, and the generated topological graph is used to represent the spatial relationship of energy in the production, transmission, and distribution processes.

3. The optimization method for the integrated energy system of wind-solar uncertain buildings based on graph reinforcement learning according to claim 1, wherein In step S2, the description of the uncertainty of the output of wind and light in combination with the scenario analysis method includes the following steps: Obtain the historical data of the photovoltaic output, predict N groups of photovoltaic output scenarios through LSTM training, and sample M groups of wind power output scenarios based on the Latin hypercube sampling method. Obtain the initial clustering centers of the M wind power output scenarios according to the maximum minimum distance principle and use the clustering algorithm for reduction to obtain K groups of wind power output scenarios. Perform a Cartesian connection on the generated N groups of photovoltaic output scenarios and K groups of wind power output scenarios to generate N*K groups of combined wind and light power generation scenarios. Through the synchronous iteration elimination method, perform scenario reduction on the generated N*K groups of combined wind and light power generation scenarios, and output 5 groups of final combined wind and light power generation scenarios and their probabilities.

4. The optimization method for a wind-solar uncertain building integrated energy system based on graph reinforcement learning according to claim 1, wherein, In step S2, the established master-slave game bilateral optimization model of the building integrated energy system based on the uncertainty of wind and light includes the target function of the cumulative total revenue of the energy supplier in the building integrated energy system, the power balance constraint of electricity / heat, the resource conversion constraint, the price constraint, the power system power flow constraint, the thermal system balance constraint, the node voltage over-limit constraint, and the operation constraints of each device and network.

5. The optimization method for a wind-solar uncertain building integrated energy system based on graph reinforcement learning according to claim 1, characterized in that, Step S3 further includes: Based on the topological graph in step S1, a quadruple The Markov decision process represented by Represent the state space, action space, reward function and state transition probability respectively; Specifically, the building integrated energy system is modeled as a graph ,The system operation status information is reflected by the node features and edge features in the graph; Including graph adjacency matrix and node feature matrix ,in n represents the number of nodes, f Represents the number of node features; graph adjacency matrix In the topology diagram n The connection relationship between nodes, Representation node and nodes There are edges connecting them. Indicates that there is no edge connection; node feature matrix The i Row represents i nodes The 6-dimensional feature vector of: ;for t Moment i nodes, is the node feature vector, is the node voltage, is the node active power, is the node reactive power, is the node load, The electricity purchase price for users, The gas purchase price for users; The operation constraints of each device in the system, the node voltage constraints with power flow balance, and the energy network constraints are added to the reward function in the form of penalty functions, obtaining: ； Among them, is t the total penalty function for all nodes in the system violating constraints at a certain moment, , , , , , and are the penalty functions for the distributed generation unit, the combined heat and power unit, the photovoltaic generation unit, the wind turbine unit, the electric boiler unit, the power energy storage system, and the heat energy storage system violating constraints at the t th moment at the i th node, represents the penalty function for the voltage violating constraints at the i-th node at time t; The immediate reward function of the agent at time t It is the difference between the net income of the building energy operator and the penalty function for violating the constraints: ； Among them, is the penalty factor; is t the revenue of the building energy operator from selling electric energy and heat energy to users at time is t the revenue of the building energy operator from selling the surplus electric energy to the superior power grid at time is t the energy purchase cost of the building energy operator from the superior power grid and gas grid at time is the operation and maintenance cost, is the energy storage cost.

6. The optimization method for a wind-solar uncertain building integrated energy system based on graph reinforcement learning according to claim 5, wherein In step S3, the trajectory information of the Markov decision process is a five-tuple ; among which, is a flag for judging whether the scheduling terminates. When the current timestamp reaches 24 hours, the optimization ends, ; otherwise, .

7. The optimization method for the integrated energy system of wind-solar uncertain buildings based on graph reinforcement learning according to claim 5, wherein Penalty function for distributed power generation units violating equipment operation constraints at time t It is expressed as: ； Among them, represents the active power generation output of the DG unit at node i at time t, represents the minimum value of the active power generation output of the DG unit at node i, represents the maximum value of the active power generation output of the DG unit at node i; when the constraint is satisfied, the penalty term is 0.

8. The optimization method for a wind-solar uncertain building integrated energy system based on graph reinforcement learning according to claim 1, wherein In step S4, a decision model is constructed and trained using the deep deterministic policy gradient algorithm based on the multi-head graph attention network. The decision model includes the multi-head graph attention actor current network, the multi-head graph attention actor target network, the multi-head graph attention critic current network, and the multi-head graph attention critic target network; In the sampling phase, a topological state is randomly selected and input into the multi-head graph attention actor current network. The actor current network selects an action using the adaptive action exploration mechanism according to the state, outputs the corresponding reward after interacting with the building energy operator, and observes the next state, forming a quadruple sample and storing it in the experience replay pool D; the loop is repeated until the number of samples in the experience replay pool reaches the scale L, and then the training phase is entered; In the training phase, the agent executes the Markov decision process through continuous interaction with the building energy operator, samples a minimum batch of samples from the experience replay pool, and inputs them into the four networks to continuously update the parameters of the networks; Among them, the input of the multi-head graph attention actor current network is the state at time , including the graph adjacency matrix mapped through the BIES topological state graph and the node feature matrix containing node voltage and power information at time t , and the output is an action in the action space at state ; the input of the multi-head graph attention actor target network is the state at time , including and the node feature matrix at time , and the output is an action in the action space at state ; among them, the output action is a 10-dimensional vector, including the energy scheduling strategy and the energy pricing strategy; The multi-head graph attention critic current network extracts the state through the graph attention layer and pooling layer The corresponding matrix and output action Splice and get As the network input data, the current Q value is output; the multi-head graph attention critic target network extracts the state through the graph attention layer and pooling layer The corresponding matrix and output action Splice and get As the network input data, it outputs the target Q value; the Q value refers to the expected discounted reward value that can be obtained after performing the corresponding action in a given state, which is used to evaluate the action.

9. The optimization method for a wind-solar uncertain building integrated energy system based on graph reinforcement learning according to claim 8, wherein The process of continuously updating the parameters of the networks includes the following steps: The attention mechanism is used to perform weighted summation on the features of neighboring nodes to obtain the attention score of node j for node i ; Utilize the function to normalize the attention scores to obtain the attention coefficients : ； Using the obtained attention coefficients , update the nodes by aggregating information transmission between nodes: ； Among them, represents the feature vector of node i in the l-th layer network, represents the set of neighbor nodes of node i, is the activation function; is the weight parameter matrix; Obtaining feature vectors using the multi-head attention mechanism : ； Among them, K is the number of attention heads, is the weight parameter matrix of the k-th attention head, is the node in the k-th attention head for the node attention coefficient.

10. The optimization method for a wind-solar uncertain building integrated energy system based on graph reinforcement learning according to claim 9, wherein, The process by which the multi-head graph attention network uses the attention mechanism to perform weighted summation of the features of neighboring nodes includes: The weight parameter matrix of the neural network The feature vectors of nodes i and j and Perform a linear transformation on the transformed feature vectors and Concatenate them to form a new node feature matrix; The new node feature matrix is mapped to the real number space using a feedforward neural network and activated through function to obtain the attention score of node j for node i : ； ； Among them, represents the independent variable of the LeakyRelu function, and the fraction reflects the relative importance of neighbor node j to node i. is a learnable attention vector, which is used to perform a linear combination of the concatenated feature vectors. The superscript T represents the transpose. is the vector concatenation operator; is a positive constant less than 1, which is used to determine the slope when the input is negative.

Images