Comprehensive energy system dynamic scheduling method based on multi-agent deep reinforcement learning

Abstract

The invention provides an integrated energy system dynamic scheduling method based on multi-agent deep reinforcement learning, and the method comprises the steps: constructing an agent for the scheduling of an integrated energy system based on a Markov decision interaction architecture, and the agent comprises a state space, an action space and a reward function; dividing the integrated energy system into multiple environments with different characteristics in different time periods by using K-means clustering; capturing environmental characteristics of the integrated energy system in a scheduling period by using a prediction model, and matching an intelligent agent of a corresponding environment; setting an economical depth deterministic strategy gradient model for each environment to carry out interactive learning and decision making, and forming a distributed learning-distributed execution architecture; and carrying out dynamic scheduling on the integrated energy system by utilizing the distributed learning-distributed execution architecture. Compared with other deep reinforcement learning methods and traditional optimization methods, the method provided by the invention can obtain a more economic and stable scheduling result.

Description

Technical Field

[0001] The present invention relates to the technical field of computer reinforcement learning and power system dispatching, and in particular to a dynamic dispatching method for an integrated energy system based on multi-agent deep reinforcement learning, as well as a corresponding system, a computer terminal and a computer-readable storage medium. Background Art

[0002] In recent years, as the severity of climate change has become increasingly prominent, the international community has become increasingly aware of the urgency of transitioning to a low-carbon economy. Achieving the "dual carbon" goal requires the vigorous development of renewable energy, which means that wind power (WT) and photovoltaic power generation (PV) will increasingly penetrate into the integrated energy system (IES). However, the intermittent nature of renewable energy and the uncertainty of load demand, such as electricity, hydrogen, and thermal energy, pose challenges to the low-carbon economic operation of the integrated energy system through optimized scheduling.

[0003] To meet this challenge, researchers have designed various uncertainty mathematical models (UMM), such as stochastic programming (SP), interval linear programming (ILP), robust optimization (RO), distributionally robust optimization (DRO), and information-gap decision technology (IGDT). Compared with deterministic mathematical models (CMM) such as energy hub (EH) or model predictive control (MPC), UMM does not rely on the accuracy of predictions. It improves the feasibility of the optimal solution by modeling the uncertainty factors in the energy system. It has been applied in the mid-term energy hub management, the analysis of the uncertainty impact caused by the deviation of the unit operation efficiency, and the installed capacity configuration planning of system equipment. However, the above model-based optimization methods still have the following limitations: 1) The problems of insufficient scenarios or conservative solutions are difficult to eliminate; 2) The highly random and dynamic real environment makes it difficult to clearly construct and obtain the mathematical model and technical parameters of the system; 3) Assumptions are required, which makes it difficult to characterize the real environment; 4) After the scene changes or data updates, the optimization model needs to be re-called for solution, and there is a lack of the ability to use historical optimization results for autonomous learning.

[0004] In view of the shortcomings of model-based methods, Deep Reinforcement Learning (DRL) has become a research hotspot in recent years and has been used to solve optimization scheduling problems in the fields of microgrids, home energy management, and multi-energy systems. This technology does not optimize decisions through mathematical models, but through repeated interactions between decision-making subjects (agents) and system states (environment), gradually maximizing the target benefits (reward function) and making optimal control strategies (behavior strategies). The DRL algorithm has three advantages: 1) There is no need to model the research object; 2) The powerful fitting ability of the neural network can effectively learn the uncertainty characteristics in the multi-energy system; 3) The trained DRL algorithm can be directly used for online real-time scheduling decisions without solving any optimization.

[0005] According to the number of decision-making entities, DRL can be divided into single-agent DRL (SADRL) and multi-agent DRL (MADRL). SADRL algorithms can also be divided into value-based algorithms and policy-based algorithms according to the discreteness or continuity of the action space of the problem to be solved. Deep Q-network (DQN) and its derivative Double DQN (DDQN) are the most representative DRL algorithms based on value estimation. As the number of decision variables in the system increases, the SADRL algorithm faces the problem of high dimensions of action and state space, which leads to difficulty in training convergence. To this end, MADRL converts the problem into a cooperative game between different agents or between different agents and the environment to promote training convergence. Based on the differences in training and execution of agents, the MADRL algorithm framework can be roughly divided into centralized learning-centralized execution (CLCE), distributed learning-distributed execution (DLDE) and centralized learning-distributed execution (CLDE). The CLCE framework requires that different agents have complete information exchange during learning and execution, and use a centralized executor to learn the joint strategy distribution of all agents. Therefore, MADRL based on the CLCE framework is similar to SADRL in that it is difficult to completely overcome the dimensionality curse caused by the increase in the number of environments and agents, and it is difficult to adapt to the interconnected multi-system environment. There is little research on it in the IES field. MADRL combines the advantages of multi-agent system and data-driven, and can efficiently optimize decisions in an environment with uncertainty and complexity.

[0006] According to the number of optimization objectives, DRL can be divided into single-objective deep reinforcement learning (SODRL) and multi-objective deep reinforcement learning (MODRL). In the field of IES, SODRL focuses on economic objectives. Due to the increasingly multi-dimensional needs of IES, it is necessary to consider objectives such as economy, safety, low carbon emissions, high renewable energy consumption, and high energy utilization. Therefore, it is necessary to study the multi-objective optimization problem of IES. At present, the relevant methods mainly include mathematical programming techniques (such as nonlinear programming), heuristic algorithms (such as multi-objective particle swarm algorithms, multi-objective genetic algorithms), and data-driven methods (such as MODRL). Compared with mathematical programming techniques, heuristic algorithms are superior in solving nonlinear optimization problems and are widely used in various fields to deal with multi-objective optimization problems. However, as an emerging technology, MODRL has little related research in the field of IES. At present, no description or report of similar technologies to the present invention has been found, and similar materials at home and abroad have not yet been collected. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a dynamic scheduling method for an integrated energy system based on multi-agent deep reinforcement learning, and at the same time provide a corresponding system, a computer terminal and a computer-readable storage medium.

[0008] According to one aspect of the present invention, a method for dynamic scheduling of an integrated energy system based on multi-agent deep reinforcement learning is provided, comprising:

[0009] Based on the Markov decision interaction (MDP) architecture, an intelligent agent for integrated energy system (IES) scheduling is constructed, wherein the intelligent agent includes: a state space, an action space and a reward function;

[0010] Using K-means clustering, the integrated energy system (IES) is divided into multiple environments with different characteristics at different time periods;

[0011] A prediction model is used to capture the environmental characteristics of the integrated energy system during the scheduling period and match the intelligent agent with the corresponding environment;

[0012] An Economic Deep Deterministic Policy Gradient (EDDPG) model is set up for each environment for interactive learning and decision-making, forming a distributed learning-distributed execution (DLDE) architecture;

[0013] The distributed learning-distributed execution (DLDE) architecture is used to dynamically schedule the integrated energy system.

[0014] According to another aspect of the present invention, a comprehensive energy system dynamic scheduling system based on multi-agent deep reinforcement learning is provided, comprising:

[0015] An agent building module, which builds an agent for integrated energy system (IES) scheduling based on a Markov decision interaction (MDP) architecture, wherein the agent includes: a state space, an action space, and a reward function;

[0016] The environment classification module uses K-means clustering to divide the integrated energy system (IES) into multiple environments with different characteristics at different time periods;

[0017] An environment matching module, which uses a prediction model to capture the environmental characteristics of the integrated energy system scheduling period and matches the intelligent agent with the corresponding environment;

[0018] The DLDE architecture building module sets up an economic deep deterministic policy gradient (EDDPG) model for each environment for interactive learning and decision-making, forming a distributed learning-distributed execution (DLDE) architecture;

[0019] A dynamic scheduling module utilizes the distributed learning-distributed execution (DLDE) architecture to dynamically schedule the integrated energy system.

[0020] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can be used to execute the method described above in the present invention, or to run the system described above in the present invention.

[0021] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute the method described above in the present invention, or to run the system described above in the present invention.

[0022] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0023] The present invention introduces the EDDPG model of the economic guidance network, integrates the economic state-action value function into the action network of DDPG, and guides the policy gradient to optimize in the direction of economy; the model can self-learn economic rewards when dealing with multi-objective optimization problems, which to a certain extent alleviates the local optimal problem caused by the design of the reward function; there is no need to adjust the weight coefficient of the preferred target in the reward function, and better economic scheduling can be achieved while taking other goals into consideration.

[0024] The present invention combines K-means clustering, prediction model and EDDPG MADRL framework to form a DLDE architecture based on multiple environments; based on the clustering algorithm, the environment is diversified and stabilized, which is conducive to the convergence of the DRL algorithm; the differences in scenarios in the system are fully considered, and the stability and optimality of asynchronous solving are improved by matching interactive learning agents one by one for different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0026] Figure 1 This is a workflow diagram of a dynamic scheduling method for an integrated energy system based on multi-agent deep reinforcement learning in one embodiment of the present invention;

[0027] Figure 2 A schematic diagram of the component modules of a comprehensive energy system dynamic scheduling system based on multi-agent deep reinforcement learning in one embodiment of the present invention;

[0028] Figure 3 A schematic diagram of the working framework of a dynamic scheduling system for an integrated energy system based on multi-agent deep reinforcement learning in a preferred embodiment of the present invention;

[0029] Figure 4 A schematic diagram of a prediction model structure based on CNN-GRU in a preferred embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of the EDDPG model structure in a preferred embodiment of the present invention;

[0031] Figure 6 It is a schematic diagram of the structure of the IEEE-33-node power network and 16-node thermal network system in a simulation example of the present invention;

[0032] Figure 7 A schematic diagram of time-of-use electricity prices used in a simulation example of the present invention;

[0033] FIG8(a) is a line graph of normalized renewable energy data in a simulation example of the present invention, and FIG8(b) is a line graph of normalized energy demand data in a simulation example of the present invention;

[0034] Fig. 9 A schematic diagram of the division of a training set and a test set in a simulation example of the present invention;

[0035] Fig.10 It is a line chart of the cumulative daily costs of different methods in 30 test days in a simulation example of the present invention;

[0036] Fig.11A bar graph of the electric power of the system tested in a simulation example of the present invention;

[0037] Fig.12 A line graph showing the dispatching results of renewable energy and electric energy storage of a system tested in a simulation example of the present invention;

[0038] Fig.13 It is a bar graph of the thermal power of the system tested in a simulation example of the present invention;

[0039] Fig.14 It is a line graph of the scheduling results of hydrogen energy storage and electricity purchase of the system tested in a simulation example of the present invention. DETAILED DESCRIPTION

[0040] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0041] An embodiment of the present invention provides a method for dynamic scheduling of an integrated energy system based on multi-agent deep reinforcement learning. The method uses a multi-agent deep reinforcement learning architecture based on a DLDE framework of K-means clustering for dynamic scheduling of an integrated energy system, with the goal of improving the efficiency and economy of dynamic scheduling of the integrated energy system, formulates a Markov decision process for the integrated energy system, and realizes real-time dynamic scheduling based on model-free; introduces an economic guidance network model to guide the policy gradient to optimize in the direction of economy, alleviates the local optimal problem caused by the design of the reward function, and does not need to adjust the weight coefficient of the preferred target in the reward function, while taking into account other goals to achieve better economic scheduling; adopts a MADRL framework that combines K-means clustering, a prediction module and EDDPG, fully considers the differences in scenarios in the system, and improves the stability and optimality of asynchronous solving by matching interactive learning agents one by one for different environments.

[0042] Specifically, Figure 1 As shown, the integrated energy system dynamic scheduling method based on multi-agent deep reinforcement learning provided in this embodiment may include the following operations:

[0043] S1, based on the Markov decision interaction (MDP) architecture, builds an intelligent agent for integrated energy system (IES) scheduling, which includes: state space, action space and reward function;

[0044] S2, using K-means clustering, the integrated energy system (IES) is divided into multiple environments with different characteristics at different time periods;

[0045] S3, using a prediction model to capture the environmental characteristics of the integrated energy system scheduling period and matching the intelligent agent with the corresponding environment;

[0046] S4, an economic deep deterministic policy gradient (EDDPG) model is set for each environment for interactive learning and decision-making, forming a distributed learning-distributed execution (DLDE) architecture;

[0047] S5, uses the distributed learning-distributed execution (DLDE) architecture to dynamically schedule the integrated energy system.

[0048] In some preferred implementations, the above S1, based on the Markov decision interaction (MDP) architecture, constructs an intelligent agent for integrated energy system (IES) scheduling, including:

[0049] The Markov decision interaction framework includes: the i-th environment With Agent i The interaction process is defined as:

[0050]

[0051] Among them, S i for The state space of state s i ∈S i ; A i for The action space of action a i ∈A i ; P is the conditional probability function; R i for Critic network instant reward space; E i for The economic network instant reward space of ; γ is the discount coefficient. Specifically, Agent i According to the strategy i (a i |s i ) produces a that satisfies the safety constraints i Then, based on a i , Agent i Feedback Critic network instant reward r i ∈R i and instant rewards for economic networks i ∈E i , and follows the conditional probability function P(s i ′|s i ,a i ) Transition to the next state s i′. At each time step t, Agent i The goal is to maximize its cumulative discount reward and Where γ∈[0,1) is the discount factor and T is the time range of batch training;

[0052] Construct the state space s of the agent i,t , the state space s i,t It is composed of 8-dimensional vectors and is expressed as:

[0053]

[0054] In the formula, s i,t It includes two parts: endogenous state and exogenous state; the endogenous state includes the battery storage capacity. and the hydrogen storage capacity of the hydrogen storage tank and is affected by the execution action; the exogenous state includes the electricity price in period t and carbon emission penalty factor Electricity demand P during period t t e,load and the heat demand P during period t t h,load And the wind power generation power P in period t t wt and photovoltaic power generation power P during period t t pv , and is not affected by the execution action;

[0055] Construct the action space a of the intelligent agent i,t , expressed as:

[0056]

[0057] Where P t grid represents the power purchased during period t, P t e,ED Indicates the power consumption of hydrogen production equipment, P t e,HFC Represents the output electrical power, Indicates the electrical power consumed by the electric boiler. The range is [-1,1], which represents the actions of the battery and the hydrogen storage tank respectively. and When it is greater than or equal to zero, or otherwise or Indicates the hydrogen release power, Indicates the discharge power of the electric energy storage, Indicates hydrogen charging power, Indicates the charging power of the electric energy storage;

[0058] Construct the agent's reward function, including the reward function e of the economic guidance network i,t and the reward function r of the evaluation network i,t , respectively expressed as:

[0059]

[0060] In the formula, It represents the operation and maintenance cost of system equipment, and β represents the coefficient of converting purchased electricity into carbon emissions.

[0061] In some preferred implementations, the above S2 uses K-means clustering to divide the integrated energy system (IES) into multiple environments with different characteristics at different time periods, including:

[0062] Data normalization: Eliminate the dimensions of the photovoltaic output variable and the power demand variable to obtain the normalized photovoltaic output variable and the power demand variable;

[0063] Batch cutting: The normalized photovoltaic output variables and power demand variables are cut in batches to obtain subsets of normalized photovoltaic output variables and power demand variables with a time step of t;

[0064] Grouping and combining: Grouping and combining subsets of normalized PV output and power demand variables in time series order;

[0065] Initialize cluster centers: Calculate the sum of elements in each subset and sort the subsets by size, then divide the subsets into K groups, and select the subset in the middle of each group as the initial cluster center;

[0066] Implement clustering: Calculate the distance between each subset and each cluster center according to formula (5), and assign each subset to the cluster center with the smallest distance to obtain a new cluster center:

[0067]

[0068] Where, X j is, X is, are the cluster centers of the low-K class, and the subscripts i and j represent them respectively;

[0069] Move the cluster center: move the new cluster center to the average value of all subsets of this cluster;

[0070] Repeated movement: Repeatedly move the new cluster center until the cluster center no longer moves or the maximum number of iterations is reached;

[0071] Determine the environment: K clusters correspond to K environments.

[0072] In some preferred implementations, the above S3, using a prediction model to capture the environmental characteristics of the integrated energy system scheduling period and matching the intelligent agent corresponding to the environment, includes:

[0073] Based on the CNN-GRU model, meteorological variables (such as temperature, light intensity, humidity, etc.) are input to the CNN Layer, and feature information is extracted from the meteorological variables using the CNN Layer. When photovoltaic power generation is predicted, photovoltaic power samples P are input to the GRU Layer. pv , and output the photovoltaic power prediction value When predicting power demand, the power demand sample P is input to the GRU Layer. e,load , and output the power demand forecast value The advantage of GRU in effectively capturing time series features is used to improve prediction accuracy.

[0074] Prediction-based and Constructing the environmental characteristic matrix of the integrated energy system dispatch period Then calculate separately The distance from the K cluster centers is calculated, and the cluster (i.e., environment) corresponding to the minimum distance is selected. Finally, the agent corresponding to the environment is taken as the target agent to achieve the agent matching the corresponding environment.

[0075] In some preferred implementations, the above S4 sets an economic deep deterministic policy gradient (EDDPG) model for each environment for interactive learning and decision-making, forming a distributed learning-distributed execution (DLDE) architecture, including:

[0076] An Economic Deep Deterministic Policy Gradient (EDDPG) model is set for each environment. The Economic Deep Deterministic Policy Gradient (EDDPG) model includes: Online Critic Network, Target Critic Network, Online Actor Network, Target Actor Network, Online Economy Network and Target Economy Network; among which:

[0077] Online evaluation network, used to calculate the evaluation value Q of (s,a) θ (s i ,a i );

[0078] The target evaluation network is used to calculate the target Q value y of the current state-action (s, a)Q ;

[0079] Online action network, used to provide the strategy or action a=μ for the current state φ (s i )

[0080] The target action network is used to provide the strategy or action for the next state;

[0081] Online economic network, used to calculate the evaluation value of (s,a)

[0082] Target economic network, used to calculate the target E value y of the current state-action (s, a) E ;

[0083] For each time step t, a small batch of experience is uniformly sampled from the experience replay buffer D, and the experience is input into the action network, evaluation network and economic network to update the network parameters, forming a distributed learning-distributed execution architecture; among them, the action network updates the parameters by maximizing the expected value of the cumulative reward, and the evaluation network and economic network update the parameters by minimizing the error between the evaluation value and the target value.

[0084] In some preferred implementations, the above S5 utilizes a distributed learning-distributed execution (DLDE) architecture to dynamically schedule the integrated energy system, including:

[0085] Based on the multi-environment obtained by K-means clustering, the predicted photovoltaic output and power demand variables are input into the distributed learning-distributed execution architecture, and an intelligent agent for realizing dynamic scheduling learning and decision-making is selected; the state space parameters are input into the intelligent agent for learning, and the action space of the device is output through the economic deep deterministic policy gradient model to dynamically schedule the integrated energy system.

[0086] Based on the same inventive concept, an embodiment of the present invention also provides a dynamic scheduling system for an integrated energy system based on multi-agent deep reinforcement learning.

[0087] Specifically, Figure 2 As shown, the integrated energy system dynamic scheduling system based on multi-agent deep reinforcement learning provided by this embodiment may include the following modules:

[0088] The intelligent agent building module is based on the Markov decision interaction (MDP) architecture to build an intelligent agent for integrated energy system (IES) scheduling. The intelligent agent includes: state space, action space and reward function;

[0089] The environment classification module uses K-means clustering to divide the integrated energy system (IES) into multiple environments with different characteristics at different time periods;

[0090] An environment matching module, which uses a prediction model to capture the environmental characteristics of the integrated energy system scheduling period and matches the intelligent agent with the corresponding environment;

[0091] The DLDE architecture building module sets up an economic deep deterministic policy gradient (EDDPG) model for each environment for interactive learning and decision-making, forming a distributed learning-distributed execution (DLDE) architecture;

[0092] Dynamic scheduling module, which uses the distributed learning-distributed execution (DLDE) architecture to dynamically schedule the integrated energy system.

[0093] The integrated energy system dynamic scheduling system based on multi-agent deep reinforcement learning provided by the above embodiment of the present invention has a working framework as follows: Figure 3 As shown. This framework is a new DLDE framework based on K-means clustering, which mainly includes three parts: 1) the environment classification and matching part based on K-means clustering, which is composed of the agent building module, the environment classification module and the environment matching module; 2) the environment feature capture part based on the prediction model for environment matching; 3) the EDDPG part based on DRL, which is composed of the DLDE architecture building module. Specifically, K-means clustering is first used to divide the IES environment into multiple environments with different characteristics in different time periods. Then, the prediction model is used to capture the environmental characteristics of the scheduling period to match the agent of the corresponding environment. Finally, an EDDPG model is set for each environment for interactive learning and decision-making to form a distributed learning-distributed execution architecture.

[0094] The working process of the integrated energy system dynamic scheduling system based on multi-agent deep reinforcement learning provided by the above embodiment of the present invention mainly includes the following steps:

[0095] Step 1: Using the agent building module, based on the Markov decision interaction (MDP) architecture, design an agent for integrated energy system (IES) scheduling. The agent includes state space, action space and reward function:

[0096] The state space s of the agent i,t It is composed of 8-dimensional vectors:

[0097]

[0098] In the formula, s i,t It consists of two parts: endogenous state and exogenous state. The endogenous state is affected by the execution action, such as the battery storage capacity St BT and the hydrogen storage capacity S of the hydrogen storage tank t HS . Exogenous states are not affected by the actions taken, such as price (λ t e ,λ t CO2 ), load demand (P t e,load , P t h,load ) and renewable energy generation (P t pv , P t wt );

[0099] The action space a of the agent i,t It is expressed as:

[0100]

[0101] In the formula, a t BT and a t HS The range is [-1,1], indicating the action of the battery and hydrogen storage tank. t BT and a t HS When a is greater than or equal to zero, t HS =P HS,t dis or a t BT =P BT,t dis , otherwise a t HS =-P HS,t cha or a t BT =P BT,t cha ;

[0102] The reward function of the agent consists of two parts: the economic guidance network and the evaluation (Critic) network. The reward function of the two is e i,t and r i,t The form is:

[0103] e i,t = -ε(λ t e P t grid +C t o&m ) (13)

[0104]

[0105] Step 2: Using the environment classification module, the integrated energy system (IES) is divided into multiple environments with different characteristics at different time periods based on K-means clustering:

[0106] Data normalization: Eliminate the dimensions of the photovoltaic output variable and the power demand variable to obtain the normalized photovoltaic output variable and the power demand variable;

[0107] Batch cutting: obtain the normalized photovoltaic output variable and power demand variable with a time step of t;

[0108] Grouping and combining: Grouping and combining subsets of normalized PV output and power demand variables in time series order;

[0109] Initialize cluster centers: Calculate the sum of elements in each subset and sort the subsets by size, then divide the subsets into K groups, and select the subset in the middle of each group as the initial cluster center;

[0110] Implement clustering: Calculate the distance between each subset and each cluster center according to (5), and assign each subset to the cluster center with the smallest distance to obtain a new cluster center:

[0111]

[0112] Where X i K is the cluster center of the low K class;

[0113] Moving cluster center: The new cluster center moves to the average value of all subsets of this cluster;

[0114] Repeated movement: until the cluster center no longer moves or the maximum number of iterations is reached;

[0115] Determine the environment: K clusters correspond to K environments.

[0116] Step 3: Use the environment matching module to capture the environment characteristics of the scheduling period through a prediction model to match the agent of the corresponding environment:

[0117] Use as Figure 4 The CNN-GRU prediction model shown in the figure obtains the photovoltaic power generation P during the scheduling period. pv and power demand P e ,load ; Input meteorological variables (such as temperature, light intensity, humidity, etc.) to CNN, and use CNN to extract feature information from complex meteorological data; When predicting photovoltaic power generation, input photovoltaic power samples P to GRU pvWhen predicting power demand, the power demand sample P is input to GRU e,load . The advantage of GRU in effectively capturing time series features is used to improve prediction accuracy.

[0118] Step 4: Use the DLDE architecture to build modules and set up an economic deep deterministic policy gradient (EDDPG) model for each environment for interactive learning and decision-making, forming a distributed learning-distributed execution (DLDE) architecture:

[0119] Specific structure such as Figure 5 As shown in the figure; the economic deep deterministic policy gradient (EDDPG) model of each environment contains six networks: Online Critic Network, Target Critic Network, Online Actor Network, Target Actor Network, Online Economy Network, Target Economy Network; Among them, the action network updates the parameters by maximizing the expected value of the cumulative reward, and the evaluation network and the economic network update the parameters by minimizing the error between the evaluation value and the target value; in the training and testing phases, for each time step t, a small batch of experience is uniformly sampled from the experience replay buffer D; it is input into the action, evaluation and economic networks to update the network parameters of the learning-distributed execution (DLDE) architecture.

[0120] Step 5: Use the dynamic scheduling module to dynamically schedule the integrated energy system through the distributed learning-distributed execution (DLDE) architecture.

[0121] It should be noted that the steps in the method provided by the present invention can be implemented by using the corresponding components in the system, and those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, and can also refer to the technical solution of the method to implement the composition of the system, that is, the embodiments in the system and the embodiments in the method can be understood as preferred examples of each other, which will not be elaborated here.

[0122] The effect of the technical solution provided by the above embodiment of the present invention is further illustrated by a simulation example below.

[0123] First, the system is set up. A modified IEEE-33-node power network and a 16-node thermal network are selected for case study. Figure 6 The system parameters are shown in Table 1. The electricity purchase price is as follows Figure 7The time-of-use electricity prices shown are: 0.4￥ / kWh for the low peak period of 23:00-7:00, 0.75￥ / kWh for the medium peak periods of 07:00-11:00 and 19:00-23:00, and 1.2￥ / kWh for the peak period of 11:00-19:00.

[0124] Then select the input data. The uncertainty factor data comes from the dataset of China State Grid, which includes renewable energy and energy demand for a whole year. The dataset is a time series data with a resolution of 1 hour. To train the model, it needs to be mapped to the interval [0,1] to achieve normalization, as shown in Figure 8(a) and Figure 8(b). The dataset is divided proportionally to achieve model training and testing with a scheduling cycle of 24 hours a day. Taking the power demand dataset as an example, the training set and test set are divided according to the ratio, such as Figure 7 As shown in Figure 2, January to May and July to November are selected as training sets, and June and December are selected as test sets.

[0125] Then the method is set up. The designed EDDPG network structure is as follows Fig. 9 As shown. The input layer of the Actor network is set with 8 neurons, corresponding to the 8-dimensional state space set, and its output layer is set with 6 neurons using the Sigmoid activation function to generate action values. The input of the Critic network and the Economy network is a set of states s and actions a. First, s and a are concatenated after passing through a neural network layer and applying the activation function, and then passing through 2 hidden layers using the ReLU activation function, and finally passing through 1 neuron using the linear activation function to obtain the Q and E values ​​of the state-action pair. The EDDPG network selects Adam as the optimizer, and other hyperparameter settings are shown in Table 2.

[0126] Table 1. System parameters of simulation example

[0127]

[0128] Table 2 Hyperparameter settings of EDDPG network

[0129]

[0130]

[0131] First, the advantages of the proposed K-means-based multi-agent architecture are evaluated. DDPG, EDDPG, K-means-DDPG and K-means-EDDPG (the proposed method) are selected for comparative analysis. The cumulative daily cost of the four methods for a continuous month (30 days) is as follows: Fig.10The optimization cost of each method per day was statistically analyzed to evaluate the stability of the method. The selected evaluation indicators include range (R), mean absolute deviation (MAD), standard deviation (SD) and coefficient of variation (CV). The statistical results are shown in Table 3. The smaller the value of the above indicators, the smaller the degree of dispersion of the data and the better the stability of the method. Fig.10 It can be seen that although the cumulative daily costs of the four methods are slightly different, the proposed K-means-EDDPG has the lowest cumulative cost. This verifies the significance of building a multi-agent architecture based on K-means. K-means-based clustering makes the environment more stable, which helps the agents to learn interactively and make more accurate decisions. As can be seen from Table 4, the proposed method obtains the smallest index value in the four index evaluations, which means that its decision-making stability is the best. This further shows that the decision-making environment can be stabilized based on the K-means architecture, which helps the agents to make more robust decisions based on specific environments.

[0132] Table 3 Stability evaluation of four methods

[0133]

[0134] Then the scheduling effect is evaluated. Based on the proposed method, the scheduling of the test set is implemented on June 1. In IES, the scheduling results of equipment, renewable energy, and load demand are as follows: Figures 9 to 12 As shown. Fig.11 In the proposed method, when renewable energy generation cannot meet the power demand of the system, a large amount of power purchase and a small amount of BT discharge and HFC generation are dispatched to maintain the power balance of the system. More specifically, Fig.12 As shown in the figure, during the periods of 00:00-08:00 and 17:00-24:00, the power generation of renewable energy is relatively small. During this period, BT promotes the consumption of renewable energy by discharging on a smaller scale; this effect is more obvious during the period of 09:00-16:00. The power generation of renewable energy during this period is significantly increased, and BT reduces the abandonment of light and wind by charging on a larger scale to improve the economy of system operation. In IES, EB and HFC are the sources of heat supply, and HFC heat generation is the main source, such as Fig.13 As shown. When the system has sufficient power and the heat load demand is large, the conversion of power to heat through EB is also an important means for the system to maintain thermal power balance. In order to reduce the operating cost of the system, the proposed method arranges the purchase of electricity during the period of lower electricity prices, such as Fig.14 In addition, hydrogen energy storage also indirectly consumes or stores electrical energy to a certain extent to coordinate the energy balance of the system.

[0135] Finally, the generalization performance of the proposed method is evaluated. Based on the two scenarios of summer and winter, SMILP, DQN, K-means-DDPG and the proposed method are selected to compare and test the optimization effect. The summer scenario selects the test set from June 1 to June 10, when the electricity demand and photovoltaic power generation are high, while the thermal energy demand is low. The winter scenario selects the test set from December 1 to December 10, when the electricity demand and photovoltaic power generation are low, while the thermal energy demand is high. The average cost of the four methods for 10 consecutive days in summer or winter is shown in Table 4. It can be seen from Table 4 that the proposed method has the best performance in the two scenarios of summer and winter with large differences, which shows that it has good generalization.

[0136] Table 4 Average cost of four methods

[0137]

[0138] An embodiment of the present invention further provides a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any method of the above-mentioned embodiments of the present invention, or to run any system of the above-mentioned embodiments of the present invention.

[0139] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0140] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0141] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0142] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0143] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0144] The above-mentioned embodiments of the present invention provide a method, system, computer terminal and computer-readable storage medium for dynamic scheduling of an integrated energy system based on multi-agent deep reinforcement learning. First, based on the Markov decision interaction (MDP) architecture, an intelligence for scheduling an integrated energy system (IES) is designed; then, the integrated energy system (IES) is divided into multiple environments with different characteristics in different time periods by using K-means clustering, and the environmental characteristics of the scheduling period are captured by a prediction model to match the agents of the corresponding environment; finally, an economic deep deterministic policy gradient (EDDPG) model is set for each environment for interactive learning and decision-making, forming a distributed learning-distributed execution (DLDE) architecture, and the DLDE architecture is used to dynamically schedule the integrated energy system. The above-mentioned embodiments of the present invention implement an innovative multi-environment multi-agent interactive learning framework based on the K-means clustering algorithm, train specific agents for specific environments or scenarios, and weaken the influence of uncertain factors on agent decision-making. Aiming at the problem of target preference in multi-objective reinforcement learning (MODRL), an economic deep deterministic policy gradient (EDDPG) model is designed. By integrating the economic guidance network into the deep deterministic policy gradient (DDPG) model, the economic weight of the decision is improved. The analysis of simulation examples shows that the technical solution provided by the above embodiment of the present invention can obtain more economical and stable scheduling results compared with other deep reinforcement learning methods and traditional optimization methods.

[0145] All matters not covered in the above embodiments of the present invention are well known in the art.

[0146] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A dynamic scheduling method for an integrated energy system based on multi-agent deep reinforcement learning, characterized in that: include: Based on the Markov decision interaction architecture, an intelligent agent for integrated energy system scheduling is constructed, wherein the intelligent agent includes: a state space, an action space and a reward function; Using K-means clustering, the integrated energy system is divided into multiple environments with different characteristics at different time periods; A prediction model is used to capture the environmental characteristics of the integrated energy system during the scheduling period and match the intelligent agent with the corresponding environment; An economic deep deterministic policy gradient model is set up for each environment for interactive learning and decision-making, forming a distributed learning-distributed execution architecture; The distributed learning-distributed execution architecture is used to dynamically schedule the integrated energy system.

2. The method for dynamic scheduling of integrated energy systems based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The intelligent agent for integrated energy system scheduling is constructed based on the Markov decision interaction architecture, including: The Markov decision interaction framework includes: the i-th environment With Agent i The interaction process is defined as: Among them, S i for The state space of state s i ∈S i ; A i for The action space of action a i ∈A i ; P is the conditional probability function; R i for Critic network instant reward space; E i for The economic network instant reward space; γ is the discount coefficient; Construct the state space s of the agent i,t , the state space s i,t It is composed of 8-dimensional vectors and is expressed as: In the formula, s i,t It includes two parts: endogenous state and exogenous state; the endogenous state includes the battery storage capacity. and the hydrogen storage capacity of the hydrogen storage tank and is affected by the execution action; the exogenous state includes the electricity price in period t and carbon emission penalty factor Electricity demand during period t and heat demand during period t And the wind power generation power in period t and photovoltaic power generation power during period t and is not affected by the execution action; Construct the action space a of the intelligent agent i,t , expressed as: In the formula, represents the power purchased during period t, Indicates the power consumption of hydrogen production equipment, Represents the output electrical power, Indicates the electric power consumed by the electric boiler. The range is [-1,1], which represents the actions of the battery and the hydrogen storage tank respectively. and When greater than or equal to zero, or otherwise or Indicates the hydrogen release power, Indicates the discharge power of the electric energy storage, Indicates hydrogen charging power, Indicates the charging power of the electric energy storage; Construct the agent's reward function, including the reward function e of the economic guidance network i,t and the reward function r of the evaluation network i,t , respectively expressed as: In the formula, It represents the operation and maintenance cost of system equipment, and β represents the coefficient of converting purchased electricity into carbon emissions.

3. The method for dynamic scheduling of integrated energy systems based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The K-means clustering is used to divide the integrated energy system into multiple environments with different characteristics at different time periods, including: Eliminate the dimensions of the photovoltaic output variable and the power demand variable to obtain normalized photovoltaic output variable and power demand variable; Cutting the normalized photovoltaic output variable and power demand variable in batches to obtain subsets of the normalized photovoltaic output variable and power demand variable with a time step of t; Grouping and combining subsets of the normalized photovoltaic output and power demand variables in time series order; Calculate the sum of the elements in each subset and sort the subsets by size, then divide the subsets into K groups, and select the subset in the middle of each group as the initial cluster center; According to formula (5), the distance between each subset and each cluster center is calculated, and each subset is assigned to the cluster center with the smallest distance to obtain a new cluster center: Where, X j is the jth data, X is the data set, is the cluster center of the low K class, and the subscripts i and j represent the i-th and j-th subsets respectively; Move the new cluster center to the average value of all subsets of this cluster; Repeatedly move the new cluster center until the cluster center no longer moves or the maximum number of iterations is reached; K clusters correspond to K environments.

4. The method for dynamic scheduling of integrated energy systems based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The method of using a prediction model to capture the environmental characteristics of the integrated energy system scheduling period and matching the intelligent agent of the corresponding environment includes: Based on the CNN-GRU model, the meteorological factor variables are input to the CNN Layer, and the CNN Layer is used to extract feature information from the meteorological factor variables; when photovoltaic power generation prediction is performed, the photovoltaic power sample P is input to the GRU Layer. pv , and output the photovoltaic power prediction value When predicting power demand, the power demand sample P is input to the GRU Layer. e,load , and output the power demand forecast value Prediction-based and Constructing the environmental characteristic matrix of the integrated energy system dispatch period Then calculate separately The distance from the K cluster centers is calculated, and the cluster corresponding to the minimum distance is selected as the environment. Finally, the agent corresponding to the environment is taken as the target agent to achieve the agent matching the corresponding environment.

5. The method for dynamic scheduling of integrated energy systems based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The economic deep deterministic policy gradient model is set for each environment for interactive learning and decision-making, forming a distributed learning-distributed execution architecture, including: An economic deep deterministic policy gradient model is set for each environment, wherein the economic deep deterministic policy gradient model includes: an online evaluation network, a target evaluation network, an online action network, a target action network, an online economic network, and a target economic network; wherein: The online evaluation network is used to calculate the evaluation value Q of (s, a) θ (s i ,a i ); The target evaluation network is used to calculate the target Q value y of the current state-action (s, a) Q ; The online action network is used to provide the strategy or action a=μ for the current state φ (s i ); The target action network is used to provide a strategy or action for the next state; The online economic network is used to calculate the evaluation value of (s, a) The target economic network is used to calculate the target E value y of the current state-action (s, a) E ; For each time step t, experience is uniformly sampled from the experience replay buffer D, and the experience is input into the action network, evaluation network and economic network to update the network parameters, forming a distributed learning-distributed execution architecture; wherein the action network updates the parameters by maximizing the expected value of the cumulative reward, and the evaluation network and economic network update the parameters by minimizing the error between the evaluation value and the target value.

6. The method for dynamic scheduling of integrated energy systems based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The method of utilizing the distributed learning-distributed execution architecture to dynamically schedule the integrated energy system includes: Based on the multi-environment obtained by K-means clustering, the predicted photovoltaic output and power demand variables are input into the distributed learning-distributed execution architecture, and an intelligent agent for realizing dynamic scheduling learning and decision-making is selected; the state space parameters are input into the intelligent agent for learning, and the action space of the device is output through the economic deep deterministic policy gradient model to dynamically schedule the integrated energy system.

7. A dynamic scheduling method for an integrated energy system based on multi-agent deep reinforcement learning, characterized in that: include: An agent building module, which builds an agent for integrated energy system scheduling based on a Markov decision interaction architecture, wherein the agent includes: a state space, an action space, and a reward function; The environment classification module uses K-means clustering to divide the integrated energy system into multiple environments with different characteristics at different time periods; An environment matching module, which uses a prediction model to capture the environmental characteristics of the integrated energy system scheduling period and matches the intelligent agent with the corresponding environment; DLDE architecture building module, which sets an economic deep deterministic policy gradient model for each environment for interactive learning and decision-making, forming a distributed learning-distributed execution architecture; A dynamic scheduling module utilizes the distributed learning-distributed execution architecture to dynamically schedule the integrated energy system.

8. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it can be used to perform the method described in any one of claims 1 to 6, or to run the system described in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method described in any one of claims 1 to 6, or to run the system described in claim 7.

Images