A scheduling method and device of an integrated energy system, a computer device and a medium

Abstract

The application provides a scheduling method and device of a comprehensive energy system, computer equipment and a medium, and belongs to the field of energy scheduling. The method comprises the following steps: obtaining state information of the comprehensive energy system and operation information of equipment operation; constructing a Markov decision process according to the comprehensive energy system; constructing an agent of a model-based policy optimization algorithm (MBPO) according to the Markov decision process, constructing an environment model by using a neural network, and optimizing the policy of the MBPO agent based on experience samples simulated by the environment model; and scheduling the operation of each device in the comprehensive energy system by the agent according to the optimized policy. In this way, the simulated experience generated quickly by the environment model and the agent allowed to explore possible states and actions more widely can adapt to the complex and changeable comprehensive energy system environment, improve the multi-energy scheduling capability of the comprehensive energy system, reduce the number of interactions between the agent and the real world, improve the sampling efficiency, and reduce the operation cost of the comprehensive energy system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of energy scheduling, and particularly relates to a scheduling method and device for a comprehensive energy system, a computer device and a medium. BACKGROUND

[0002] Under the background of close coupling between the power grid and the natural gas pipeline network, multiple types of energy are complex and interlaced. In order to solve the scheduling problem of multiple types of energy, the existing technology proposes a comprehensive energy system. The comprehensive energy system can realize energy efficient utilization and sustainable development, meet diversified load demand by coordinating different types of energy, and realize integrated operation of energy utilization, energy conversion and energy storage.

[0003] There are many studies on the operation optimization of the comprehensive energy system in the existing method, such as using particle swarm algorithm and linear programming to reduce cost and carbon emission. These methods are mainly divided into heuristic algorithm and mathematical optimization method, but both of them have certain deficiencies in practical application. The heuristic algorithm can solve nonlinear, multimodal and high-dimensional problems, but it has problems of high calculation cost, convergence difficulty and inability to guarantee finding the global optimal solution, resulting in poor scheduling optimization effect of the comprehensive energy system. The mathematical optimization method relies on accurate mathematical models, but with the access of renewable energy and multiple loads, accurate modeling becomes increasingly difficult, and it is difficult to adapt to the comprehensive energy system. Therefore, the above existing technology based on the scheduling method optimization of the comprehensive energy system has limitations, which makes it difficult to adapt to the comprehensive energy system in a complex multi-energy high-dimensional environment. SUMMARY

[0004] In order to solve the problem that the existing technology is difficult to adapt to the comprehensive energy system scheduling in a complex high-dimensional environment, the present application provides a scheduling method, device, computer device and medium for a comprehensive energy system.

[0005] In order to achieve the above purpose, the present application provides the following technical scheme:

[0006] Firstly, a scheduling method for a comprehensive energy system is provided, comprising:

[0007] obtaining state information of the comprehensive energy system and operation actions of device operation;

[0008] constructing a Markov decision process according to the comprehensive energy system; the Markov decision process comprises a state space, an action space and a reward function; wherein the state space is a state information set constructed according to the state information, the action space is an operation action set constructed according to the operation action, and the reward function generates a reward according to the corresponding state space and action space, and the reward is negatively correlated with the operation cost of the comprehensive energy system;

[0009] The agent is constructed according to a Markov decision process, an environment model is constructed by using a neural network, the agent is interacted with the environment model, and the strategy of the agent is optimized by maximizing the reward; wherein the environment model is used to predict the future value of an exogenous state and a reward, and the exogenous state includes load demand, energy price and wind speed.

[0010] According to the optimized strategy, the operation of each device in the integrated energy system is scheduled by the agent.

[0011] Optionally, the environment model is constructed by using a neural network, and the environment model comprises:

[0012] The environment model is constructed by using a set of deep neural networks, each deep neural network in the set is used for independent prediction, and the set of deep neural networks is composed of a plurality of deep neural networks (DNNs).

[0013] Optionally, the agent is interacted with the environment model, and the strategy of the agent is optimized by maximizing the reward, and the strategy optimization comprises:

[0014] Based on the real experience samples obtained in advance, the environment model is trained through the interaction between the agent and the environment model.

[0015] The environment model is trained through the interaction between the agent and the environment model.

[0016] Optionally, the environment model is trained through the interaction between the agent and the environment model, and the strategy optimization comprises:

[0017] The strategy network Actor and the Q value network Critic of the MBPO are initialized, and two experience replay buffers are initialized, which are used to store real experience samples and simulated experience samples respectively;

[0018] The agent is interacted with the environment model, including: the agent inputs the current state of the integrated energy system into the strategy network, simulates the state change of the integrated energy system based on the environment model, determines the probability distribution of each continuous behavior of the agent, randomly samples an action from the probability distribution, and transfers the integrated energy system simulated by the environment model to the next state and receives a reward;

[0019] A plurality of time steps of the interaction process are cycled, a plurality of simulated experience samples composed of states, actions, rewards and next states are generated, and the plurality of simulated experience samples are stored in the experience replay buffer.

[0020] The environment model is trained by minimizing the difference between the next state in the real experience sample and the next state in the simulated experience sample.

[0021] Optionally, the difference between the simulated experience sample and the next state in the real experience sample is characterized by mean square error (MSE), and the DNN is trained to minimize the difference.

[0022] Optionally, the policy optimization of the agent to maximize the reward comprises:

[0023] The policy optimization of the agent is achieved by minimizing the real-time operation cost of the integrated energy system in each time slot.

[0024] Secondly, a scheduling device of an integrated energy system is provided, comprising:

[0025] An acquisition module is configured to acquire state information of the integrated energy system and operation actions of device operation;

[0026] A construction module is configured to construct a Markov decision process according to the integrated energy system; the Markov decision process comprises a state space, an action space and a reward function; wherein the state space is a state information set constructed according to the state information, the action space is an operation action set constructed according to the operation actions, and the reward function generates a reward according to the corresponding state space and action space, and the reward is negatively correlated with the operation cost of the integrated energy system;

[0027] An optimization module is configured to construct an agent of a model-based policy optimization algorithm (MBPO) according to the Markov decision process, construct an environment model by using a neural network, interact the agent with the environment model, and optimize the policy of the agent by maximizing the reward; wherein the environment model is configured to predict future values and rewards of exogenous states, and the exogenous states comprise load demand, energy price and wind speed;

[0028] A scheduling module is configured to schedule the operation of devices in the integrated energy system by the agent according to the optimized policy.

[0029] In addition, a computer readable storage medium is provided, and the storage medium stores a computer program; when the computer program is executed by a processor, the above-mentioned scheduling method of an integrated energy system is implemented.

[0030] Finally, a computer device is provided, comprising a memory, a processor and a computer program stored in the memory and executable on the processor; when the processor executes the program, the above-mentioned scheduling method of an integrated energy system is implemented.

[0031] The scheduling method of an integrated energy system provided by the application has the following beneficial effects:

[0032] By abstracting the scheduling problem of the integrated energy system into a Markov decision process, a basis is provided for the learning and decision-making of the agent, facilitating optimization using reinforcement learning and other algorithms, and on this basis, the agent and environment model of the model-based strategy optimization algorithm are constructed, the simulation experience generated by the environment model is used, and the agent is allowed to explore a wider range of possible states and actions, which can adapt to the complex and variable integrated energy system environment, improve the scheduling capability of multiple energies in the integrated energy system, enhance the generalization ability of the agent, ensure the accuracy and reliability of the agent in scheduling the integrated energy system, reduce the number of interactions between the agent and the real world, improve the sampling efficiency, and thus be more efficient than learning only from real experience, while also reducing the operating cost of the integrated energy system and the frequency and cost of human intervention. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application and the design thereof, the drawings required by the present embodiments will be briefly introduced as follows. The drawings in the following description are only part of the embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0034] Figure 1 A flowchart of a scheduling method of an integrated energy system according to an exemplary embodiment of the present application is provided.

[0035] Figure 2 A structural diagram of an integrated energy system according to an exemplary embodiment of the present application is provided.

[0036] Figure 3 An interaction diagram of an agent and an environment according to an exemplary embodiment of the present application is provided.

[0037] Figure 4 A parameter training diagram of a scheduling algorithm according to an exemplary embodiment of the present application is provided.

[0038] Figure 5 A training flowchart according to an exemplary embodiment of the present application is provided.

[0039] Figure 6 A block diagram of a scheduling device of an integrated energy system according to an exemplary embodiment of the present application is provided. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the technical solutions of the present application and to implement them, the present application will be described in detail below in conjunction with the drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

[0041] The application is a comprehensive energy system optimization scheduling method based on model reinforcement learning, which comprises the following steps:

[0042] Step 1: a comprehensive energy system model containing electric heating natural gas is established, and renewable energy equipment models, combined heat and power equipment models, gas turbine equipment models, electric boiler equipment models, heat boiler equipment models and energy storage equipment models are obtained.

[0043] Step 2: considering the models of various devices in step 1, an operation cost model is constructed, system constraints are defined, and an operation cost model is constructed according to the indexes and parameters of energy consumption and conversion of various devices.

[0044] Step 3: based on the optimization target of the cost in step 2, the energy optimization scheduling problem is mapped to a Markov decision process model, and a reinforcement learning state space, an action space and a reward function are designed.

[0045] Step 4: the MBPO algorithm is used to solve the Markov decision model problem of continuous behavior, and the optimal scheme of scheduling is solved based on the expansion and optimization of network parameter training samples, taking the operation cost at each time as the target, and the network including Actor network and Critic network is optimized.

[0046] The technical solutions provided by the embodiments of the application are described in detail below with reference to the accompanying drawings.

[0047] Firstly, the application provides a scheduling method of a comprehensive energy system, specifically as shown in Figure 1 , comprising the following steps:

[0048] S101, obtaining state information of a comprehensive energy system and operation actions of device operation.

[0049] The state information includes load, consumption, conversion, storage and price information of energy.

[0050] The IES structure in the application is as shown in Figure 2As shown, considering the widespread application of renewable energy such as wind and solar, the introduction of photovoltaic and wind turbine power generation equipment in the architecture can help the system reduce dependence on fossil fuels. In addition, the IES architecture also includes gas turbines, combined heat and power units, gas boilers, electric boilers, energy storage devices, and heat storage devices. These can flexibly adjust output power, quickly respond to demand changes, make up for gaps caused by renewable energy fluctuations, and enhance system flexibility. Electric boilers and gas boilers as heat source equipment can provide heat energy and work with the power system to choose different forms of energy consumption at different times to optimize the comprehensive utilization of energy. Combined heat and power simultaneously produces heat and electricity, improving energy utilization efficiency. Energy storage devices can store excess energy and release it during peak demand to balance load fluctuations, improving system stability and reliability. In addition, energy can be purchased directly from the main power grid and natural gas suppliers, and the electricity and heat produced by the system are delivered to users through the power grid and district heating network, respectively.

[0051] For example, the integrated energy system can include photovoltaic (PV), wind turbine (WT), combined heat and power (CHP), gas boiler (GB), electric boiler (EB), gas turbine (GT), energy storage (ES), heat storage (HS), electric load (EL), and thermal load (HL). Electricity and natural gas can be purchased directly from the power grid and natural gas network and then delivered to users through regional networks.

[0052] Among them, the state information includes but is not limited to various energy demand, storage and price information, such as electric load and thermal load, which reflect the energy demand of the system at different time periods; energy storage information covers the storage state of energy storage (ES) and heat storage (HS) devices; price information includes real-time electricity and natural gas prices, which are crucial for optimizing energy purchase strategies.

[0053] and represent the amount of electricity produced by PV and WT, respectively, calculated from wind speed. CHP generates heat by consuming natural gas and electricity GB and GT output heat and electricity by burning natural gas, respectively and is the energy change of the storage device within time period t, and if it is positive, it means charging, otherwise it means discharging.

[0054] S102, constructing a Markov decision process according to the integrated energy system.

[0055] The Markov decision process MDP includes a state space, an action space and a reward function; wherein the state space is a state information set constructed according to the state information, the action space is an operation action set constructed according to the operation action, and the reward function generates a reward according to the corresponding state space and action space, and the reward is negatively correlated with the operation cost of the integrated energy system.

[0056] Specifically, the state space is composed of state information, and the action space includes related parameters of operation actions of devices in the integrated energy system, such as power generation of PV and WT, natural gas consumption of CHP, heat and power output of GB and GT, power consumption of EB, and charging / discharging and heat power of ES and HS; the reward function is calculated according to the total cost of the integrated energy system, the total cost includes the cost of purchasing energy and the cost of device maintenance, and the reward function is designed as the negative value of the total cost, so as to maximize the reward and minimize the total cost.

[0057] For example, in the MDP, the state space can include power load, heat load, electricity price, natural gas price, wind speed, photovoltaic power generation, ES power and HS heat, etc.; the action space can include power generation of PV, natural gas consumption of CHP, and charging / discharging power of ES; and the reward function is designed as: -(energy purchase cost + device maintenance cost).

[0058] In an embodiment, the Markov decision process generally includes a tuple (S, A, P, r, γ), wherein S, A and γ represent state space, action space and discount factor of future reward respectively. IES is the environment of the present application, which provides its own state s t ∈S to the agent at time slot t. The agent determines the action a t according to the state s t ∈A. In addition, P(s t+1 |s t ,a t ) is a state transition function, and r t (s t ,a t ) is a reward function. The Markov decision process is as follows:

[0059] Definition 1 (state space). At time t, the state space of the integrated energy system is defined as follows:

[0060]

[0061] wherein, and are the demand sizes of power and heat; and are the electricity price and natural gas price; v t is the wind speed, and the corresponding power output can be calculated through the wind speed ; Photovoltaic power generation; and is the energy storage state of the energy storage device and the heat storage device, expressed in percentage.

[0062] Definition 2 (Action Space). At time t, the action space of the integrated energy system is defined as follows:

[0063]

[0064] where, Electricity output of the combined heat and power device; is the heat output of the gas boiler; is the electricity output of the gas turbine; is the heat output of the electric boiler; P t ES and P t HS respectively represent the charging and discharging capacity of the energy storage device and the heat storage device.

[0065] Definition 3 (Reward Function). The optimization objective of the present invention is to minimize the energy purchase cost and equipment maintenance cost of the integrated energy system. Therefore, the reward function at time step t can be calculated as:

[0066]

[0067]

[0068] where, represents the total cost of the integrated energy system; and respectively represent the cost of purchasing energy and the cost of equipment maintenance; represents the amount of electricity purchased from the main grid; represents the amount of natural gas purchased from the natural gas website, represents the maintenance price coefficient of equipment j, is the output of the equipment at time t, where j = {WT, PV, CHP, EB, GB, GT, ES, HS}.

[0069] Definition 4 (Optimization Objective). The optimization objective of the present invention is as follows:

[0070]

[0071] subject to

[0072]

[0073]

[0074]

[0075] where the discount factor is denoted by symbol γ.Formula (1.6), (1.7) and (1.8) are power constraints, heat constraints and natural gas constraints of the integrated energy system, aiming to ensure that the integrated energy system realizes safe and efficient operation on the premise of meeting demand.

[0076] S103, according to the Markov decision process, an agent based on a model-based policy optimization algorithm MBPO is constructed, an environment model is constructed by using a neural network, the agent is interacted with the environment model, and the policy of the agent is optimized by maximizing the reward.

[0077] wherein the environment model is used to predict future values of exogenous states and rewards, and the exogenous states include variables with uncertainty such as load demand, energy price and wind speed. The environment model is constructed by a neural network, which can learn and simulate the dynamic behavior of the integrated energy system. The MBPO agent optimizes the policy by using these simulated experience samples, reduces the demand for interaction with the real environment, and improves the learning efficiency.

[0078] Specifically, based on the real experience samples obtained in advance, the environment model is trained through the interaction of the agent and the environment model; the simulated experience samples are generated by using the trained environment model, and the real experience samples are combined to optimize the policy of the agent to maximize the reward, and the optimized agent policy is obtained.

[0079] In the present application, MBPO uses a neural network to construct a model that approximates the dynamics of the environment and uses the model to generate simulated experience samples to optimize the policy. Generally, the state transition function of the Markov decision process is unknown. In the present application, the state is divided into two types according to whether a neural network is needed for prediction:

[0080] Endogenous states are internal states of the IES, such as and which can be directly calculated in the environment according to the previous state and action, without the need for neural network prediction.

[0081] Exogenous states include random variables such as load demand, energy price and wind speed, such as v t and Since these exogenous random variables have strong uncertainty, an environment model is needed to predict them to generate experience samples for policy optimization.

[0082] The application adopts a model-based policy optimization MBPO algorithm to solve the IES energy scheduling problem. An agent interacts with an environment to obtain simulated experience samples, which are used to train an environment model, as shown in Figure 3 The simulated experience samples generated by the trained environment model can be used to more effectively explore the state and action, rather than relying solely on real experience samples for policy optimization.

[0083] Among them, the training of the environment model can be as follows: initialize the policy network Actor and the Q value network Critic of MBPO, and two experience replay buffers for storing real experience samples and simulated experience samples respectively; interact the agent with the environment model, including: the agent inputs the current state of the integrated energy system into the policy network, determines the probability distribution of each continuous behavior of the agent based on the state change of the integrated energy system simulated by the environment model, randomly samples an action from the probability distribution, and moves the integrated energy system simulated by the environment model to the next state and receives a reward; cycle the interaction process for multiple time steps, generate multiple simulated experience samples composed of state, action, reward and next state, and store the multiple simulated experience samples in the experience replay buffer; train the environment model by minimizing the difference between the next state in the real experience sample and the next state in the simulated experience sample, or minimizing the difference between the reward in the real experience sample and the reward in the simulated experience sample.

[0084] In order to solve the problem of performance degradation caused by model bias, MBPO adopts a set of deep neural networks (the size of the set is N) to construct the environment model Each deep neural network in the set makes an independent prediction, and the set of deep neural networks is composed of multiple deep neural networks DNN. The ensemble model in the application is composed of seven deep neural networks (DNN), which increases the complexity and computational cost of the model, but it can effectively cope with the uncertainty brought by environmental noise and the uncertainty brought by insufficient model parameters and data. The ensemble model can maintain high prediction accuracy in the case of data scarcity or severe environmental changes, and reduce the negative impact on scheduling decisions by suppressing the accumulation of prediction bias.

[0085] As shown in the following formula, the ensemble model is trained by the mean squared error (MSE) loss function, which represents the difference between the next state in the simulated experience sample and the real experience sample, and the DNN is trained to minimize the difference:

[0086]

[0087] Where ω is the neural network parameter of DNNs; N is the sample size; is the predicted value of DNNs; e n is the true state; σ n is the predicted standard deviation.

[0088] The SAC (Soft Actor-Critic, a deep learning-based reinforcement learning algorithm) algorithm uses a mixture of simulated samples and real experience samples to optimize the policy, and adds an entropy term in the reward to ensure a high level of exploration and robustness when facing random factors. SAC contains five networks: one policy network (Actor), two Q-value networks (current Critic), and two target Q-value networks (target Critic). The parameters of these networks are φ, θ m＝1,2 and The Actor is updated by minimizing the soft Bellman residual:

[0089]

[0090] where, is the experience replay buffer, used to store experience samples, is the state s t and action a t at time t corresponding to the Q value of the current Critic algorithm, is the state s t and action a t at time t corresponding to the Q value of the target Critic algorithm. The parameters of the target Critic are updated by soft updating the old target Critic network parameters by a small part of the current Critic network parameters, which makes the changes of the target Critic network more smooth, thus helping to stabilize the training process. During the policy improvement process of the algorithm, the update of the agent network is based on the maximum expected value of the weighted Q value and entropy, and π is the current policy:

[0091]

[0092] where, logπ φ (a t |s t ) is the entropy, and the higher the entropy, the more the agent can find a more effective policy through continuous exploration; α is a hyperparameter used to control the weight of the entropy term. Using the minimum of the two Q values can stabilize the training process and reduce estimation bias.

[0093] The network parameters φ and θ are updated by gradient descent method:

[0094]

[0095]

[0096] where λ π and λ Q is the number of update steps for parameters φ and θ

[0097] To further improve sample efficiency, the baseline algorithm SAC in the present application adopts a priority experience replay (PER) mechanism. By assigning higher priorities to experience samples that have a greater impact on policy improvement, usually determined by time errors, it ensures that more impactful samples are used more frequently, thereby speeding up the learning process.

[0098] In an embodiment, the scheduling algorithm of the present application improves sample efficiency and policy learning rate by using two types of experience samples, thereby controlling the device action of the system to achieve cost minimization, as shown in Figure 4 (1) Real experience samples. Real experience refers to the experience of current state, current action, reward and next state obtained in the interaction with the integrated energy system, which is directly used to train the prediction model and subsequent policy optimization. (2) Simulated experience samples. Simulated experience is generated by the model and used to learn the optimal policy. It is worth noting that this scheme can reduce the number of real interactions and improve sampling efficiency. The training process of MBPO in the entire workflow is shown in Figure 5 and the algorithm in Table 1 below.

[0099] Table 1 Training process algorithm of MBPO

[0100]

[0101]

[0102] At each time step t, the environment experience replay pool stores real experience samples obtained by interacting with the environment, which are regularly used to train the environment model After training, the model selects N historical states as the initial states for prediction. After determining the state and action, each DNN predicts the Gaussian distribution of the next state and reward. Then, the agent generates the next action according to the next state and repeats the above process k times. Finally, the ensemble model generates N x k simulated experience samples, which are stored in the model replay experience pool . Simulated samples reduce the need for real environment interaction during policy learning.

[0103] Model updating usually involves procedures such as calculating complex gradients, refitting data, etc. If the model is updated too frequently, the computational overhead can be high, resulting in slower training speed. To balance the model accuracy and learning efficiency, the model in the present application is retrained every 250 steps (model update frequency) to ensure the stability of the learning process. Finally, the simulated experience samples from are combined with the real experience samples from combined with SAC algorithm for policy optimization.

[0104] Specifically, first, to store real experience and simulated experience, two experience replay buffers are initialized. To realize the selection, evaluation and optimization of the selected action, the policy network Actor and the Q-value network Critic are initialized, (1) the policy network π φ for making decisions in the system; (2) the Q-value network Q θ for evaluating the decision made by the policy network π φ in the current state.

[0105] Subsequently, the agent observes the current state s t of the integrated energy system and inputs it into the policy network π φ , which outputs the probability distribution a t ~ π (·|s t ) of each continuous action. The agent randomly samples a t from the probability distribution and returns to the integrated energy system environment, which moves to the next state and receives a reward. The above interaction process is looped for several time steps to generate a series of data quadruples composed of state, action, reward and next state (s t , a t , r t , s t+1 ), which are then stored in the environment experience replay buffer D env . D env is used to train the environment prediction model and collect experience for training the model in the initial stage (lines 2-6 of Algorithm 1).

[0106] Secondly, the experience samples in D env are used to train the environment model regularly (lines 7-9 of Algorithm 1). During the training of the prediction model, the following defined loss function is used for optimization:

[0107]

[0108] where ω is the parameter of the DNN; N is the size of the sample; is the predicted mean value of the DNN; e i is the real state and reward value; σ i is the predicted standard deviation.

[0109] At the same time, experience samples (s t , a t , r t , s t+1 ) are randomly selected from D env , and st This serves as the initial state for the model's rolling prediction. From s t Let's start with the environment integration model. The model uses N deep neural networks to perform k-step rolling predictions, and stores the predicted results in the model experience replay buffer. (Lines 10-14 of Algorithm 1)

[0110] Subsequently, and D env The empirical samples are mixed, and then these samples are used to update the policy network π in a gradient-based manner. φ The parameters φ and Q-value of the network Q θ The parameter θ (lines 15-18 of Algorithm 1) is defined as follows:

[0111]

[0112]

[0113] Where, λ π and λ Q These are the update step sizes for parameters φ and θ, respectively. J π (φ t ) and J Q (θ t The objective functions for the policy network and the Q-value network are defined as follows:

[0114]

[0115]

[0116] The policy learning and optimization parts of this invention are implemented using the model-free reinforcement learning Soft Actor-Critic (SAC) algorithm. SAC is an algorithm based on entropy maximization, whose objective function adds a state information entropy term αlogπ(a) to the traditional cumulative reward. t |s t ), consisting of the adaptive entropy coefficient α and the selected action a t logarithmic probability logπ(a) t |s t The system is composed of five deep neural networks: a policy network, two Q-value networks, and two target Q-value networks. This ensures high exploratory power and robustness even when faced with random factors such as electricity price fluctuations or load changes. To improve the model's adaptive learning and generalization capabilities, SAC includes five deep neural networks: one policy network, two Q-value networks, and two target Q-value networks, such as... Figure 4 As shown on the left, its parameters are φ and θ respectively. m＝1,2 and

[0117] is the improvement process of the Actor network in the algorithm, that is, the Actor network is updated by maximizing the expectation of the weighted Q value and the entropy. In the formula, , that is, the experience replay buffer, stores a mixed sample of real experience and simulated experience (s t , a t , r t , s t+1 ). , that is, the Critic network, concatenates the state and action as input, and outputs a Q value estimate through a series of linear transformations and activation functions in the neural network. The minimum value operation of the two Q values improves the stability of the training.

[0118] Equation (1.13) is the parameter update method of the Critic network in the algorithm, that is, the Critic network updates the parameters by minimizing the mean square error (MSE) loss function of the target Q value y t and the real Q value . The purpose of the target Critic network is to constantly update the parameters of the Q value network. If the Q value output by the current Q value network is directly used to update itself, it will lead to unstable training process. The calculation method of the target Q value is defined as follows:

[0119]

[0120] , where r t is the current reward, and a t+1 ~ π (· | s t+1 ) is the probability distribution of selecting action a t+1 according to the next state. By selecting the minimum value from the Q value estimates of the next state output by the two target Q value networks, the training stability is improved. The influence of the policy entropy is also considered, that is, the logarithmic probability of the policy network π φ adopting action a t+1 .

[0121] The parameter updating method of the target Critic network is as follows:

[0122]

[0123] , where τ is a coefficient less than 1, used to control the update speed. The parameters of the target Critic network are updated by soft updating (Soft Update) by weighting and mixing a small part of the parameters of the current Critic network into the old target Critic network parameters. This method makes the update of the target Critic network more smooth, which helps to stabilize the training process.

[0124] S104, according to the optimized strategy, the agent schedules the operation of each device in the integrated energy system.

[0125] The agent adjusts the operation parameters of each device in the integrated energy system in real time according to the optimized strategy to meet the load demand while minimizing the total cost. These adjustments may include changing the power generation of PV and WT, adjusting the natural gas consumption of CHP, controlling the charge / discharge / thermal power of ES and HS, etc.

[0126] For example, under the optimized strategy, the agent may decide to increase the amount of electricity purchased when the electricity price is low, while reducing the power generation of PV and WT to save costs; when the heat load is high, the natural gas consumption of CHP may be increased to provide more heat.

[0127] By abstracting the scheduling problem of the integrated energy system as a Markov decision process, the above method provides a basis for the learning and decision-making of the agent, facilitating optimization using reinforcement learning algorithms, and building an agent and environment model based on model-based policy optimization algorithm, using the simulation experience generated by the environment model and allowing the agent to explore a wider range of possible states and actions. The agent can adapt to the complex and variable integrated energy system environment, improve the multi-energy scheduling capability in the integrated energy system, enhance the generalization ability of the agent, ensure the accuracy and reliability of the agent in scheduling the integrated energy system, reduce the number of interactions between the agent and the real world, improve the sampling efficiency, and thus be more efficient than learning from real experience alone. It also reduces the operating cost of the integrated energy system and the frequency and cost of human intervention.

[0128] Secondly, the present application also provides a kind of scheduling device of integrated energy system, as shown in Figure Figure 6 It includes:

[0129] The acquisition module 601 is used to acquire the state information of the integrated energy system and the operation action of the device operation.

[0130] The construction module 602 is used to construct a Markov decision process according to the integrated energy system; the Markov decision process includes a state space, an action space and a reward function; wherein the state space is a set of state information constructed according to the state information, the action space is a set of operation actions constructed according to the operation action, and the reward function generates a reward according to the corresponding state space and action space, which is negatively correlated with the operating cost of the integrated energy system.

[0131] The optimization module 603 is used to construct an agent based on model-based policy optimization algorithm MBPO according to the Markov decision process, construct an environment model using a neural network, interact the agent with the environment model, and optimize the strategy of the agent by maximizing the reward; wherein the environment model is used to predict the future value of the exogenous state and the reward, and the exogenous state includes load demand, energy price and wind speed.

[0132] The scheduling module 604 is configured to schedule the operation of each device in the integrated energy system according to the optimized strategy through the agent.

[0133] The scheduling process implemented by each module of the integrated energy system is as follows:

[0134] 1. The current integrated energy system environment state is taken as the environment for interacting with the reinforcement learning algorithm. The acquisition module 601 collects various state information from the integrated energy system, including energy load information (such as the load size of each energy at different times), consumption information (the consumption amount of various energies), conversion information (such as the conversion of energy conversion devices such as electric-gas and cogeneration), storage information (the energy storage level of energy storage devices), and price information (the market price of each type of energy).

[0135] 2. After integrating the key states, the construction module 602 constructs a Markov decision process model according to the acquired integrated energy system information. The state space contains the state information obtained from the acquisition module, which comprehensively reflects the current state of the integrated energy system. The action space includes the operation information of the devices, which represents the operations that can be taken in the system. The calculation basis of the reward function is the total cost of the integrated energy system, which takes into account the cost of purchasing energy (such as the cost of purchasing electricity, natural gas, etc.) and the cost of device maintenance (the cost required for maintaining various devices). The design of the reward function aims to guide the system to optimize the scheduling in the direction of cost optimization. When the total cost decreases, a higher reward is given, and vice versa, providing a quantitative evaluation standard for subsequent optimization.

[0136] 3. Based on the constructed Markov decision process model, the optimization module 603 begins to construct the agent of the model-based policy optimization algorithm MBPO. An environment model is constructed using a neural network, which can predict the future value of exogenous states and rewards, including load demand (such as future changes in electricity load, heat load, etc.), energy prices (future price fluctuations of different energies), and wind speed (particularly important for systems that rely on wind energy). Using the experience samples simulated by the environment model, the strategy of the MBPO agent is optimized. Through continuous simulation and optimization, the agent can learn what actions (device operation) to take in different states to achieve better rewards, and gradually adjust and optimize its strategy to achieve better scheduling performance.

[0137] 4. When the optimization module 603 completes the policy optimization of the MBPO agent, the scheduling module 604 will use the optimized policy. The agent will schedule the operation of each device in the integrated energy system according to the current state of the integrated energy system and its optimized policy, and determine the output size operation of the device, so as to achieve the optimized operation of the system as a whole.

[0138] In actual operation, the above modules may continuously cycle to achieve dynamic and continuous optimization of integrated energy system scheduling according to the feedback and new state information of the system. The acquisition module continuously updates the state and operation information, the optimization module continuously optimizes the agent policy, and the scheduling module adjusts the device operation according to the new optimized policy, forming a closed-loop optimization scheduling process.

[0139] By using the above device, the scheduling problem of the integrated energy system is abstracted into a Markov decision process, which provides a basis for the learning and decision-making of the agent, facilitates the optimization using reinforcement learning algorithms, and builds an agent and an environment model based on the model-based policy optimization algorithm. The simulation experience generated by the environment model and the agent's ability to explore a wider range of possible states and actions can adapt to complex and variable integrated energy system environments, improve the multi-energy scheduling capability of the integrated energy system, enhance the generalization ability of the agent, ensure the accuracy and reliability of the agent's integrated energy system scheduling, reduce the number of interactions between the agent and the real world, improve the sampling efficiency, and thus be more efficient than learning from real experience alone. At the same time, it can also reduce the operation cost of the integrated energy system and the frequency and cost of manual intervention.

[0140] The application also provides a computer readable storage medium, which stores a computer program, and the computer program can be used to execute the steps of the scheduling method of the integrated energy system. Figure 1 The steps of the scheduling method of the integrated energy system are provided.

[0141] The application also provides a computer device, which comprises a processor, an internal bus, a network interface, a memory and a non-volatile memory at the hardware level. Of course, it can also comprise other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the steps of the scheduling method of the integrated energy system. Figure 1 The steps of the scheduling method of the integrated energy system are provided.

[0142] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0143] The present application is described in reference to the flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for carrying out each of the one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0144] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for carrying out each of the one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. ​ means for carrying out each of the one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0146] It should be noted that the above detailed description of the specific embodiments of the present application is not intended to limit the scope of the present application. Therefore, although the present application has been described in detail with reference to the specific embodiments, it should be understood that modifications and equivalents thereof can be made by those skilled in the art without departing from the spirit and scope of the present application. All technical solutions and improvements that do not depart from the spirit and scope of the present application are encompassed in the protection scope of the patent of the present application. Any reference signs in the claims should not be considered as limiting the claims.

Claims

1. A scheduling method for an integrated energy system, characterized in that, include: To acquire status information and operational actions of the integrated energy system; Construct a Markov decision process based on the integrated energy system described above; The Markov decision process includes a state space, a behavior space, and a reward function; wherein, the state space is a set of state information constructed based on the state information, the behavior space is a set of operation actions constructed based on the operation actions, and the reward function generates a reward based on the corresponding state space and behavior space, and the reward is negatively correlated with the operating cost of the integrated energy system; A model-based policy optimization algorithm (MBPO) is constructed based on Markov decision processes. An environment model is built using a neural network, and the agent interacts with the environment model to optimize the agent's policy by maximizing rewards. The environment model is used to predict the future values ​​and rewards of exogenous states, including load demand, energy prices, and wind speed. Based on the optimized strategy, the operation of each device in the integrated energy system is scheduled by an intelligent agent. The method of constructing an environment model using neural networks includes: An environment model is constructed using a deep neural network ensemble, where each deep neural network in the ensemble makes independent predictions. The deep neural network ensemble consists of multiple deep neural networks (DNNs).

2. The scheduling method for an integrated energy system according to claim 1, characterized in that, The process of interacting the agent with the environment model and optimizing the MBPO agent strategy by maximizing rewards includes: Based on pre-acquired real-world experience samples, the environment model is trained through the interaction between the agent and the environment model; Simulated experience samples are generated using the trained environment model and combined with real experience samples to optimize the agent's policy with the goal of maximizing rewards, resulting in an optimized agent policy.

3. The scheduling method for an integrated energy system according to claim 2, characterized in that, The training of the environment model through the interaction between the agent and the environment model includes: Initialize the MBPO policy network Actor and Q-value network Critic, as well as two experience replay buffers, which are used to store real experience samples and simulated experience samples, respectively. Interacting with the environment model involves the agent inputting the current state of the integrated energy system into the policy network, simulating the state changes of the integrated energy system based on the environment model, determining the probability distribution of each continuous action of the agent, randomly sampling actions from the probability distribution, and transferring the integrated energy system simulated by the environment model to the next state and receiving a reward. Multiple time steps in the cyclical interaction process generate multiple simulated experience samples consisting of state, behavior, reward and next state, and store multiple simulated experience samples in the experience replay buffer; The environment model is trained by minimizing the difference between the next state in real experience samples and the next state in simulated experience samples.

4. The scheduling method for an integrated energy system according to claim 3, characterized in that, The DNN is trained with the goal of minimizing the difference between the next state in simulated experience samples and real experience samples, characterized by the mean squared error (MSE).

5. The scheduling method for an integrated energy system according to claim 2, characterized in that, The strategy optimization of the agent with the goal of maximizing rewards includes: The strategy optimization of the agent is achieved by minimizing the real-time operating cost of the integrated energy system in each time slot.

6. A dispatching device for an integrated energy system, characterized in that, include: The acquisition module is used to acquire the status information and operational actions of the integrated energy system and its equipment. A construction module is used to construct a Markov decision process based on the integrated energy system; The Markov decision process includes a state space, a behavior space, and a reward function; wherein, the state space is a set of state information constructed based on the state information, the behavior space is a set of operation actions constructed based on the operation actions, and the reward function generates a reward based on the corresponding state space and behavior space, and the reward is negatively correlated with the operating cost of the integrated energy system; The optimization module is used to construct an agent based on the Model-Based Policy Optimization (MBPO) algorithm according to the Markov Decision Process. It utilizes neural networks to build an environment model, allowing the agent to interact with the environment model and optimize the agent's policy by maximizing rewards. The environment model is used to predict the future values ​​and rewards of exogenous states, including load demand, energy prices, and wind speed. The construction of the environment model using neural networks includes: using an ensemble of deep neural networks to construct the environment model, where each deep neural network in the ensemble performs independent predictions. The ensemble of deep neural networks consists of multiple deep neural networks (DNNs). The scheduling module is used to schedule the operation of various devices in the integrated energy system through an intelligent agent, based on the optimized strategy.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 5.

8. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 5.

Images