Energy scheduling method for integrated multi-energy system

Abstract

The invention belongs to the technical field of multi-energy system optimization scheduling, and relates to an integrated multi-energy system energy scheduling method comprising the following steps: S1, constructing a system framework; s2, constructing a system model on the basis of the system framework; s3, converting a scheduling problem of the system model into a Markov decision process containing uncertain factors; and S4, improving the optimization capability of the Markov decision process by adopting a TPO-VRN algorithm so as to obtain an optimal scheduling strategy. Compared with the prior art, the cooperative scheduling of the high-dimensional energy system is realized, a new thought is provided for the optimal scheduling of the high-proportion renewable energy multi-energy system, powerful technical support is provided for improving the economy and the scheduling stability of the multi-energy system, the optimization capability of the scheduling strategy can be improved, and the scheduling efficiency of the multi-energy system can be improved. And better performance can be shown in a dynamic and complex multi-energy system, and challenges brought by future uncertainty can be adapted.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-energy system optimal scheduling, and relates to an integrated multi-energy system energy scheduling method. BACKGROUND

[0002] CIES is an advanced energy system architecture integrated by diversified distributed renewable energy generation units, advanced energy storage systems (ESS) and intelligent load systems. Compared with traditional energy supply modes, CIES shows significant advantages in economic benefits, environmental friendliness and sustainable development in multiple dimensions. In this system, energy storage technology as an innovative energy regulation mechanism can effectively cope with the volatility challenge of renewable energy. However, traditional energy storage mainly solves the short-term power balance problem of system scale, and it is difficult to manage long-term energy imbalance problem. Therefore, compressed air energy storage technology (CAES) is widely used, which is a zero-carbon, high-energy-density storage medium that can support large-scale and longer duration energy transfer and improve the utilization rate of renewable energy.

[0003] Currently, the main types of CAES technology include: isothermal compressed air energy storage (I-CAES), thermal storage compressed air energy storage (TS-CAES), advanced adiabatic compressed air energy storage (AA-CAES) and supercritical compressed air energy storage (SC-CAES). Among them, the AA-CAES system is particularly outstanding, as it does not rely on fossil fuels, reduces carbon emissions through heat recovery and storage technology, and has significant environmental benefits. However, the uncertainty of resource output may affect the stable operation of the system, and different constraint conditions also increase the complexity of the economic management of the system. The inherent volatility of renewable energy hinders the system to achieve real-time balance between energy supply and demand, so integrating energy storage systems and developing optimal energy scheduling schemes to achieve expected operation goals have become a research hotspot of current academia and industry.

[0004] In the transformation process of modern industrial ecosystems, the synergy and intelligent scheduling of multi-energy have become the key path to break through the traditional single energy paradigm. The complexity and inherent uncertainty of multi-energy pose unprecedented theoretical and practical challenges to system optimization design. Current academic research on multi-energy system optimization mainly focuses on the following core areas: equipment type modeling, scheduling strategy, optimization operation, and uncertainty management.

[0005] In the aspect of modeling, there is a scheme of integrating the combined refrigeration, heating and power (CCHP) system with the renewable energy optimization model, which innovatively adopts the operation strategy of electrical load following thermal load, so that the pollutant emission reduction rate is increased by 5.13%, but the energy conversion and storage unit is not integrated, and there is still some limitation. In the field of scheduling strategy, the existing technology proposes an energy management strategy based on TD3 to improve the training efficiency and optimization ability, and the results show that TD3 improves the training efficiency by 56.7% compared with DDPG, and reduces the total operation cost by 8.58%. But this method may cause the problem of gradient disappearance or gradient explosion due to unstable training. In terms of handling the uncertainty of renewable energy, the existing technology uses the dimension reduction clustering algorithm to capture the uncertainty of renewable energy and energy demand, although this method has certain advantages in dimension reduction and feature extraction, but there is a potential risk of losing key features. SUMMARY

[0006] The purpose of the present application is to provide an integrated multi-energy system energy scheduling method, which aims to realize the collaborative scheduling of high-dimensional energy systems, provide a new idea for the optimization scheduling of high-proportion renewable energy multi-energy systems, provide strong technical support for improving the economic efficiency and scheduling stability of multi-energy systems, and improve the optimization ability of the scheduling strategy, and exhibit better performance in dynamic and complex multi-energy systems, and adapt to the challenges brought by future uncertainties.

[0007] Based on the above invention purpose, the present application provides an integrated multi-energy system energy scheduling method, comprising the following steps:

[0008] S1, constructing a system framework: constructing a collaborative scheduling framework for high-dimensional energy systems, covering the complete link of energy input, energy conversion, energy storage and energy consumption, forming a "source-grid-load-storage" structure;

[0009] S2, constructing a system model based on the system framework, the system model comprising an energy input model, an energy conversion model, an energy storage model and an objective function, the objective function being constructed to minimize the total operation cost;

[0010] S3, converting the scheduling problem of the system model into a Markov decision process containing uncertain factors;

[0011] S4, using the TPPO-VRN algorithm to improve the optimization ability of the Markov decision process, the TPPO-VRN algorithm being an algorithm that introduces a variational autoencoder (VAE) and a residual network (ResNet) in the Markov decision process, effectively reducing the complexity of the high-dimensional state space, improving the feature extraction and representation ability, and then using the proximal policy optimization (TPPO) algorithm for policy optimization to obtain the optimal scheduling strategy.

[0012] Preferably, in step S1, the system framework integrates renewable energy sources including wind power and photovoltaic power, and ensures the stability and reliability of energy supply through public grid access, and is equipped with energy storage devices including compressed air storage and water storage tanks for storing excess energy and releasing when needed, so as to realize the "peak load shifting" of the system, balance energy supply and demand, and realize the conversion and optimal utilization of energy by using absorption chillers and electric heat pumps.

[0013] Preferably, in step S2, the objective function is expressed as follows:

[0014]

[0015] C em (t) = C AA-CAES (t) + C EHP (t) + C AC (t) + C WST (t)

[0016] In the formula: C total is the total operating cost of the system framework; C grid (t) is the cost of purchasing electricity from the grid; C em (t) is the total operating cost of each device, C AA-CAES (t), C EHP (t), C AC (t), C WST (t) respectively represent the operating costs of compressed air storage, electric heat pump, absorption chiller and water storage tank.

[0017] Preferably, in step S2, the objective function ensures the feasibility and safety of the scheduling scheme by adding operating constraints, including energy balance constraints, renewable energy constraints and operation constraints, wherein the goal of the energy balance constraint is to ensure that the supply and demand of various energies in the system framework are balanced at any time, the renewable energy constraint includes the output power constraint of photovoltaic power and the output power constraint of wind power, and the operation constraint limits the output power of each device in the system framework within a range that ensures the safe operation of the entire system framework.

[0018] Preferably, in step S3, in the Markov decision process, the state space is used to describe the operating state of the system framework, the action space represents the scheduling strategy, the transition probability describes the state evolution, and the reward function quantifies the performance of the strategy.

[0019] Preferably, in step S4, in the TPPO-VRN algorithm, the variational autoencoder (VAE) can learn to compress the high-dimensional state space into a low-dimensional latent space, which can capture the key features in the state space, thereby reducing the difficulty of learning the strategy; the residual network (ResNet) extracts useful features from the high-dimensional input and serves as the encoder and decoder of the variational autoencoder (VAE), improving the reconstruction and generation capabilities of the variational autoencoder (VAE), thereby compressing the high-dimensional vector in the action space into a low-dimensional latent vector, and then the proximal policy optimization (TPPO) algorithm learns the strategy based on the latent vector.

[0020] Preferably, in step S4, the proximal policy optimization (TPPO) algorithm realizes more stable and efficient strategy learning through the clipping mechanism and the KL divergence constraint.

[0021] Preferably, in step S2, the energy input model includes an uncertainty factor model, and for wind power generation and photovoltaic power generation, the uncertainty factors include wind speed, light intensity, weather conditions and geographical location; the uncertainty factor model uses robust optimization theory to accurately model the uncertainty of wind power and photovoltaic power generation output, and uses a probabilistic classification-based chance constraint method to establish a hierarchical adjustable uncertainty set of wind power and photovoltaic power generation output, which can distinguish wind power and photovoltaic output at different probability levels through hierarchical processing, thereby realizing fine control of the risk level of the dispatching strategy.

[0022] Compared with the prior art, the present application has the following beneficial effects:

[0023] (1) A collaborative dispatching framework for high-dimensional energy systems is constructed, covering the complete link of energy input, conversion, storage and consumption, breaking through the limitations of traditional single energy system modeling.

[0024] (2) An interval prediction method is used to accurately depict the uncertainty of wind power and photovoltaic power generation, and the multi-dimensional influence of system constraints and renewable energy fluctuations on dispatching results and economy is analyzed in depth.

[0025] (3) ResNet is introduced into VAE, which significantly reduces the dimension of the state space, solves the sample efficiency problem of reinforcement learning, and improves the generalization performance of the model.

[0026] (4) Reinforcement learning and generative model are innovatively combined, and TPPO, VAE and ResNet are deeply combined, which effectively reduces the dimension of the high-dimensional state space, enhances the perception and extraction ability of the model to key information in complex environments, and improves the training stability.

[0027] In conclusion, the present application realizes the coordinated scheduling of high-dimensional energy systems, and provides a new idea for the optimal scheduling of high-proportion renewable energy multi-energy systems, and provides strong technical support for improving the economy and scheduling stability of multi-energy systems, the present application can not only improve the optimization ability of the scheduling strategy, but also can exhibit better performance in dynamic and complex multi-energy systems, and adapt to the challenges brought by future uncertainties. BRIEF DESCRIPTION OF DRAWINGS

[0028] Other features, objects, and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, read in conjunction with the accompanying drawings:

[0029] Figure 1 A system framework diagram in the embodiment of the present application is shown;

[0030] Figure 2 A Markov decision process in the embodiment of the present application is shown;

[0031] Figure 3 A principle diagram of a clipping mechanism in the embodiment of the present application is shown;

[0032] Figure 4 A flowchart of a TPPO algorithm in the embodiment of the present application is shown;

[0033] Figure 5 A block diagram of a VAE in the embodiment of the present application is shown;

[0034] Figure 6 A structure diagram of a residual module in the embodiment of the present application is shown;

[0035] Figure 7 A comprehensive block diagram of a TPPO-VRN algorithm in the embodiment of the present application is shown;

[0036] Figure 8 A learning performance comparison diagram of TPPO, TPPO-VAE and TPPO-VRN algorithms in the embodiment of the present application is shown;

[0037] Figure 9 An influence diagram of different constraint values on system operation cost in the embodiment of the present application is shown;

[0038] Figure 10 A load demand curve of a typical summer in the embodiment of the present application is shown;

[0039] Figure 11 A cost comparison diagram of TPPO, TPPO-VAE and TPPO-VRN algorithms in the embodiment of the present application is shown;

[0040] Figure 12 An electric energy power balance diagram in the embodiment of the present application is shown;

[0041] Figure 13 A thermal power balance diagram is shown in an embodiment of the present invention;

[0042] Figure 14 A cold energy power balance diagram is shown in an embodiment of the present invention;

[0043] Figure 15 A statistical chart illustrating the impact of a single uncertainty on the scheduling results is shown in an embodiment of the present invention.

[0044] Figure 16 A statistical chart showing the impact of multiple uncertainties on scheduling results in an embodiment of the present invention is shown. Detailed Implementation

[0045] The following will be combined with the appendix Figures 1-16 The technical solutions of the present invention have been clearly and completely described. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Example

[0047] This embodiment proposes an integrated multi-energy system energy dispatching method, including the following steps:

[0048] S1. System Framework Construction: Construct a collaborative scheduling framework for high-dimensional energy systems, covering the complete chain of energy input, energy conversion, energy storage and energy consumption, forming a "source-grid-load-storage" structure; the system framework aims to effectively meet users' needs for electricity, heat and cooling loads, and efficiently provide electricity, cooling and heating services to various users.

[0049] like Figure 1 As shown, the system framework integrates renewable energy sources, including wind and photovoltaic power generation, and ensures the stability and reliability of energy supply through grid connection. To optimize energy utilization, the system framework is equipped with energy storage devices, including compressed air storage and water storage tanks, to store excess energy and release it when needed, thereby achieving peak shaving and valley filling, balancing energy supply and demand, and improving energy efficiency. This not only reduces the system's dependence on the grid but also contributes to the stable operation of the grid. Furthermore, the system framework uses absorption chillers and electric heat pumps to achieve energy conversion and optimized utilization; the output of the electric heat pump is used to generate heat and cold energy, while the absorption chiller converts heat energy into cold energy. Through the coordinated operation of various devices in the system, the diverse energy needs of different users can be met.

[0050] S2, build a system model on the basis of the system framework, the system model comprising an energy input model, an energy conversion model, an energy storage model and an objective function, the objective function being built with the goal of minimizing total operating cost;

[0051] The energy input model comprises a PV model, a WT model and an uncertain factor model.

[0052] In order to accurately evaluate the performance of a photovoltaic power generation system, it needs to be accurately modeled. The power generation efficiency of photovoltaics is affected by many factors, of which temperature is one of the most critical factors. When the temperature of photovoltaics rises, its power generation efficiency will decrease. Therefore, when modeling a photovoltaic system, the effect of temperature needs to be considered. The output power of photovoltaic power generation is calculated using the following formula:

[0053] η PV = η PV，0 [1- γ(T(t)- T ref )], t ∈ H (1)

[0054] P PV (t) = η PV · S PV · G (2)

[0055] In the above formula, η PV is the power generation efficiency of the photovoltaic panel system; η PV,0 is the power generation efficiency of the photovoltaic panel at standard temperature; T(t) is the actual temperature of the photovoltaic panel at time t, in ℃; T ref is the reference temperature (usually 25℃); γ is the temperature coefficient; H is the time set; P PV (t) is the output power of the photovoltaic panel; S PV is the area of the photovoltaic panel; G represents the light intensity, in kw / m 2 .

[0056] As the core component of a wind power generation system, the performance of a wind turbine affects the power generation efficiency and operational stability of the entire system. Wind speed not only determines the amount of wind energy that can be captured by the wind turbine, but also directly affects the operational state of the wind turbine. Depending on the wind speed, the wind turbine can be in different operating modes, such as starting, variable-speed operation, constant-power operation or shutdown state. Therefore, the present application adopts a piecewise modeling method for the wind turbine. When the wind speed is within the range of rated wind speed to cut-out wind speed, the output power of the unit is constant at the rated value; when the wind speed reaches the cut-in wind speed, the unit starts to generate electricity and the power generation increases with the wind speed; when the wind speed exceeds the range of cut-in wind speed and cut-out wind speed, the unit stops generating electricity. The output power P WT (t) of the wind turbine can be represented by the following formula:

[0057]

[0058] where η is the efficiency of the WT; p is the air density; A is the swept area of the rotor; C WT is the power coefficient; and v is the wind speed. p (λ, β) is the wind energy utilization coefficient, which is a function of the tip speed ratio λ and the pitch angle β; v(t) represents the wind speed at time t, v c-in v c-out and v rated are the cut-in speed, the cut-out speed, and the rated speed (m / s), respectively. WT-rated P(t) is the rated output power of the wind turbine.

[0059] Wind power and photovoltaic power generation, as intermittent renewable energy sources, are susceptible to various complex factors such as wind speed, light intensity, weather conditions, and geographical location. This leads to problems such as low prediction accuracy, strong volatility, and large uncertainty in actual operation. Moreover, the output of wind power and photovoltaic power generation is usually a constraint condition for power system operation, and its uncertainty directly relates to the safety and economy of system operation. Therefore, the present application uses robust optimization theory to accurately model the uncertainty of wind power and photovoltaic power generation output to ensure the optimal dispatch and safe and stable operation of the power system. Unlike traditional stochastic optimization methods, robust optimization can ensure that the solution to the optimization problem satisfies the constraint condition regardless of how the uncertain parameters change within their uncertainty set.

[0060] In view of the differences in the random distribution of the uncertainty sources of wind power and photovoltaic power generation, the present application uses a probability classification-based chance constraint method to establish a hierarchical uncertainty set for the output of wind power and photovoltaic power generation. This strategy not only effectively alleviates the conservatism of the robust optimization result, but also more accurately depicts the robust performance of the optimal dispatch scheme. Through hierarchical processing, the wind power and photovoltaic output at different probability levels can be distinguished, thereby achieving fine control of the risk level of the dispatch strategy, including:

[0061] (1) The wind power uncertainty output set is:

[0062]

[0063] where P(t) is the wind power output at time t, P WT (t) is the predicted wind power output at time t, 1-ε(t) is the wind power output confidence level at time t, and P and P are the upper and lower limits of the wind power output confidence interval.

[0064] (2) The photovoltaic power generation uncertainty output set is:

[0065] ​

[0066] where P PV (t) is the PV output at time t, is the predicted PV output at time t, and denote the upper and lower bounds of the PV output confidence interval at confidence level 1 - ε(t), respectively.

[0067] where the energy conversion model includes an EHP model and an AC model;

[0068] In the EHP model, an electric heat pump (EHP) is a highly efficient energy conversion device that can convert electrical energy into thermal energy for heating or cooling. In a multi-energy system, an electric heat pump (EHP) can serve as a bridge connecting the electrical and thermal systems, improving energy utilization efficiency. The performance of an electric heat pump (EHP) is typically measured by the coefficient of performance (COP), which is defined as the ratio of the thermal energy output provided by the electric heat pump (EHP) to the electrical energy consumed. The thermal output power Q EHP of an electric heat pump (EHP) can be represented as:

[0069] R EHP (t) = P EHP (t) · COP EHP (6)

[0070] H EHP (t) = P EHP (t) · COP EHP (7)

[0071] where R EHP (t) and H EHP (t) represent the cooling energy and thermal load generated by the electric heat pump (EHP), respectively, P EHP (t) is the electrical power consumed by the electric heat pump (EHP), and COP EHP is the coefficient of performance of the electric heat pump (EHP).

[0072] In addition, in the AC model, an absorption chiller (AC) uses thermal energy to drive the cooling process. This process can be represented as

[0073] R AC (t) = H AC (t) · COP AC (8)

[0074] where R AC (t) is the cooling power output of the absorption chiller, with units of KW; COP AC is the cooling coefficient; and H AC (t) is the heat absorbed by the absorption chiller.

[0075] wherein the energy storage model comprises an AA-CAES model and a WST model;

[0076] In the AA-CAES model, the AA-CAES system can store excess electrical energy during off-peak periods and release it during peak periods, thereby improving the stability and economy of the power grid. The AA-CAES system stores compressed heat through an adiabatic process, avoiding the problem of energy loss in the form of waste heat in traditional compressed air energy storage (CAES) systems, thereby significantly improving energy storage efficiency. The system uses a multi-stage compression method to gradually increase the pressure of the air, and sets coolers between stages to reduce the temperature of the compression process and improve compression efficiency. During discharge, the system uses multi-stage expansion and uses heat storage devices between stages to release heat to heat the air, thereby improving the output power and efficiency of the expander.

[0077] In the WST model, a water storage tank (WST) is used to store excess heat, which can be released during periods of high heat demand. This energy storage and release process achieves peak load shifting, improves energy storage efficiency, and thus improves the overall energy utilization efficiency of the system. The mathematical model can be expressed as:

[0078]

[0079] s WST (t)=H WST (t) / IC WST (10)

[0080] In the above formula: t is the time variable; H WST (t) is the heat power of the water storage tank at time t; δ is the heat loss coefficient of the water storage tank; and is the heat power input and output of the water storage tank at time t; and is the heat charging and heat exchange efficiency of the water storage tank; s WST (t) is the proportion of heat in the water storage tank at time t; IC WST is the installed capacity of the water storage tank.

[0081] The objective of this optimization problem is to minimize the total operating cost of the system, which consists of two parts: the cost of purchasing electricity from the grid and the operating cost of each device. By reasonably scheduling the operating state of each device in the system, the total operating cost can be minimized. Therefore, the objective function can be expressed as:

[0082]

[0083] C em (t)=CAA-CAES (t)+C EHP (t)+C AC (t)+C WST (t) (12)

[0084] wherein C total is the total operating cost of the system framework; C grid (t) is the cost of electricity purchased from the grid; C em (t) is the total operating cost of each device, C AA-CAES (t), C EHP (t), C AC (t), C WST (t) respectively represent the operating cost of compressed air energy storage, electric heat pump, absorption chiller and water tank.

[0085] The objective function ensures the feasibility and safety of the scheduling scheme by adding operating constraints, including energy balance constraints, renewable energy constraints and operation constraints, wherein the objective of the energy balance constraints is to ensure that at any time, the supply and demand of various energies in the system framework are balanced, the renewable energy constraints include output power constraints of photovoltaic power generation and output power constraints of wind power generation, and the operation constraints limit the output power of each device in the system framework to a range that ensures safe operation of the entire system framework.

[0086] In the energy balance constraints, energy balance is the basis of the operation of the system framework, and must ensure that at any time, the supply and demand of various energies in the system are balanced. That is, at any time, the generation, conversion and consumption of energy must strictly comply with the law of conservation of energy. The mathematical expression is as follows:

[0087] P Load (t)+P EHP (t)=P grid (t)+P PV (t)+P WT (t)+P AA-CAES (t) (13)

[0088] H Load (t)+H AC (t)=H AA-CAES (t)+H EHP (t)+H WST (t) (14)

[0089] R Load (t)=R AC (t)+R EHP (t)+R AA-CAES (t) (15)

[0090] wherein PLoad (t), H Load (t), R Load (t) respectively represent the power, heat and refrigeration load demand at time t.

[0091] In this application, the renewable energy constraint is wind and photovoltaic power generation constraint. The output of renewable energy has volatility and uncertainty, which brings challenges to the stable operation of the system framework. In order to ensure the stable operation of the system and maximize the consumption of renewable energy, it is necessary to reasonably constrain it. The formula is as follows:

[0092]

[0093] In the formula: P WT,max (t) is the maximum output power of the wind turbine; P PV,max (t) is the maximum output power of the photovoltaic.

[0094] In the operation constraint, each device in the system should limit the output power in a specific valid range to ensure safe operation. Among them, EHP, AC and AA-CAES devices need to meet the following limit conditions (min represents the upper limit, max represents the lower limit, and PHR represents the power of electricity, heat and cold respectively) :

[0095]

[0096] AA-CAES requires the power, energy allocation ratio and conversion efficiency of each device to vary within a reasonable range. To this end, the constraint condition must be met. Considering that the actual heat storage capacity of the water storage tank (WST) is much smaller than its capacity, and the SOC constraint has little effect on the optimal scheduling of the system, this application ignores the constraint of the state of charge of WST. In order to ensure the safe operation of AA-CAES and prevent the energy storage system from being overcharged or discharged, it is necessary to limit its state of charge (SOC) within the following specific safe range:

[0097] SOC min ≤ SOC (t) ≤ SOC max (18)

[0098] In the formula: SOC min and SOC max are the minimum and maximum state of charge of compressed air energy storage respectively.

[0099] S3, the scheduling problem of the system model is converted into a Markov decision process containing uncertain factors; such as Figure 2As shown, Markov Decision Process (MDP) provides a powerful framework for describing and solving sequential decision problems with Markov property. In Markov Decision Process, the state space describes the running states of the system framework, the action space represents the scheduling strategy, the transition probability characterizes the state evolution, the reward function quantifies the performance of the strategy, and the energy scheduling problem with randomness and dynamics can be handled. Therefore, the energy scheduling problem of the system framework is described as an MDP in the present application. The traditional MDP assumes that the environment is completely observable and the state transition probability is known, which is often difficult to meet in real applications, so the influence of uncertain factors is also considered in the present application.

[0100] In MDP, the state space is like a map that marks all the "positions" the agent can encounter. Each "position" represents a specific state of the environment, containing all the relevant information that affects the agent's decision. The state space of the present application is represented as a seven-dimensional vector: real-time electricity demand price; wind power; photovoltaic power; electricity, heat and cold load demand of combined cooling heating and power (CCHP); and state of charge (SOC) of compressed air energy storage system (CAES).

[0101] s t = [P Load (t), H Load (t), R Load (t), C b (t), P WT (t), P PV (t), SOC(t)] (19)

[0102] In the formula, C b (t) is the real-time electricity demand price; P WT (t) is the wind power; P PV (t) is the photovoltaic power; P Load (t), H Load (t), R Load (t) are the load demand of CCHP, respectively; and SOC(t) is the state of charge of CAES.

[0103] The action space defines all possible actions the agent can take in each state. The goal of the agent is to learn how to choose the appropriate action in different states to maximize the cumulative reward. The action space can be represented by a five-dimensional vector: P AA-CAES (t) is the power output of AA-CAES; is the charging and discharging state of AA-CAES; P EHP (t) is the output power of EHP; P AC (t) is the output power of AC; and P WST (t) is the output power of WST.

[0104]

[0105] A state transition function describes how the environment state evolves given the current state and action. This application employs an uncertain state transition function, where, given the current state s and action a, the next state s′ is not uniquely determined, but rather appears with a certain probability distribution. These probabilities are used to calculate the expected reward, thereby guiding the choice of strategy. The formula is as follows:

[0106] P(s′|s,a)=Pr(S t+1 =s′|S t =s,A t =a) (21)

[0107] Where P(s′|s,a) can be any value between 0 and 1, and satisfies the normalization condition that, given the current state and action, the sum of the probabilities of transitioning to all possible states must be equal to 1.

[0108] The reward function quantifies energy scheduling objectives as numerical signals, and the agent learns how to achieve these objectives by maximizing these signals. Actions that reduce costs receive positive rewards, encouraging the agent to repeat them; actions that increase costs receive negative rewards, prompting the agent to avoid them. The objective of this application is to minimize operating costs, while the agent seeks to maximize rewards; therefore, the reward function must take the negative value of the cost to guide the agent to make optimal decisions. Thus, the reward function is designed as follows:

[0109] r t =-[C grid (t)+C em (t)] (22)

[0110] C grid =P buy (t)·C b (t)+P sell (t)·C s (t) (23)S4. The TPPO-VRN algorithm is used to improve the optimization capability of the Markov decision process. The TPPO-VRN algorithm effectively reduces the complexity of the high-dimensional state space and improves the feature extraction and representation capabilities by introducing a variational autoencoder (VAE) and a residual network (ResNet) in the Markov decision process. Then, the proximal policy optimization (TPPO) algorithm is used to optimize the policy to obtain the optimal scheduling policy. Furthermore, the TPPO algorithm achieves more stable and efficient policy learning through pruning mechanism and KL divergence constraint.

[0111] In multi-energy systems, the state space is high-dimensional, and it is difficult for agents to learn strategies directly in high-dimensional state space. VAE can learn to compress high-dimensional state space into a low-dimensional latent space, which can capture the key features in the state space, thereby reducing the difficulty of learning strategies. ResNet is a powerful feature extractor that can effectively extract useful features from high-dimensional inputs. Using ResNet as the encoder and decoder of VAE can improve the reconstruction and generation capabilities of VAE. VAE-ResNet can compress high-dimensional data into low-dimensional latent vectors, and then TPPO learns strategies based on these latent vectors. This can greatly reduce computational complexity and improve learning efficiency. The generation capability of VAE can be used to generate new states and encourage agents to explore unknown state space.

[0112] The proximal policy optimization (TPPO) algorithm is a policy optimization method based on the traditional policy gradient method and the trust region policy optimization (TRPO) for simplification and improvement, and is also an actor-critic architecture-based reinforcement learning algorithm (RL). In order to solve the problem of KL divergence restriction, TPPO introduces clip probability ratio (clip), and designs a new objective function to limit the amplitude of policy update, thereby maintaining the stability of the policy. A clipping term is added to the objective function, and the advantages of the current policy and the old policy are combined, so that the change between the new and old policies is kept within a controllable range, aiming to achieve a balance between training stability and sample efficiency. As shown in the clipping mechanism Figure 3 When the ratio between the new policy and the old policy exceeds the set threshold, the policy is not updated, thereby ensuring that the policy update is performed within a safe range. The core idea of TPPO is to limit the step size of policy update to prevent the dramatic change of policy that may lead to performance degradation. The process of TPPO algorithm is shown in Figure 4 TPPO also converts the online training mode based on policy gradient into offline training mode, which can simultaneously collect experience and train the algorithm, store the collected experience in the experience pool, and update the parameters after a certain number of iterations.

[0113] In the TPPO algorithm, the KL divergence is introduced as a constraint to constrain the distribution difference between the new and old policies, and the formula is as follows:

[0114]

[0115] The core of TPPO is the objective function, which balances the needs of policy optimization and policy stability. The objective function L CLIP (θ) can be obtained by the following formula:

[0116]

[0117] where r t (θ) is the probability ratio of the new policy to the old policy; A θ′ is the advantage function, which can be calculated by GAE clip(r t (θ),1-∈,1+∈) is the Clipping function, which limits r t (θ) within the range of [1-∈,1+∈]; D KL (π θ ||π θ′ ) is the KL divergence, which measures the difference between the new and old policies; β is the hyperparameter of the KL divergence penalty term, which is used to control the magnitude of policy update. ∈ is a small constant, usually 0.1 or 0.2, which is used to limit the ratio of policy update.

[0118] The entropy term can encourage the randomness of the policy, so as not to converge to a deterministic policy too early. High-entropy policy can increase exploration, which is particularly important when facing sparse rewards, local optimum or large state space. Usually set c2∈[0.001,0.01] to balance the interference of this term to policy optimization. The advantage function of TPPO algorithm and the entropy term S[π θ ](s t ) are as follows:

[0119]

[0120] where γ is the discount factor; λ is a parameter in GAE, which is used to balance the bias and variance; δ t is the time difference error. By setting a suitable β value, a good balance point can be found between the clipping mechanism and the KL divergence penalty to ensure the stability of training and the efficiency of sampling. The objective function of TPPO algorithm is its policy loss function, and the total loss is as follows:

[0121] L(θ)=E t [L CLIP (θ)-c1L VF (θ)+c2S[π θ ](s t )] (29)

[0122] L VF (θ)=(V θ (s t )-V t target ) 2 (30)

[0123] where L CLIP (θ) is the policy loss function, L VF (θ) is the value function loss function, and S[πθ ](s t ) is the entropy of the policy at state st, used to encourage randomness of the policy and improve exploration ability; c1, c2 are hyperparameters, respectively controlling the weights of the value function loss and the entropy regularizer in the overall objective; V t target is the target value function.

[0124] Specifically, VAE is a powerful generative model, aiming to learn the latent distribution of input data and introduce an inference mechanism that can be trained in the generative model. The core idea is to map the data x into a latent space z, and reconstruct the data through the decoder. Unlike traditional autoencoders, VAE introduces probabilistic modeling in the latent feature space, learning the probability distribution of latent variables, thus possessing the ability of generation. To train the model, VAE adopts variational inference to approximate the true posterior distribution p(z|x) with the approximate distribution q(z|x), and learns the posterior distribution of latent variables z by maximizing the variational lower bound. In practical applications, VAE assumes that the latent variables follow a Gaussian distribution, and the encoder network outputs the mean and variance, while the decoder network learns to reconstruct the input from the latent variables. Through the reparameterization trick, VAE converts the sampling operation into a differentiable operation, so that end-to-end training can be performed using gradient descent. The structural diagram of VAE is shown in FIG. 1. Its objective function can be expressed as: Figure 5

[0125]

[0126] z = μ + σ ⊙ ε (32)

[0127] where q φ (z|x) is the approximate posterior distribution of latent variables generated by the encoder (inference network); p θ (x|z) is the conditional distribution of input data generated by the decoder (generative network); D KL is the KL divergence, measuring the closeness of the two. To sample from the latent distribution, VAE introduces the reparameterization trick, letting the distribution of z be a normal distribution, and ε be a standard normal distribution random variable, μ and σ be the mean and standard deviation output by the encoder, respectively. The reconstruction error formula can be obtained from the mean and standard deviation:

[0128]

[0129] ​Following the above examples, ResNet is a very popular convolutional neural network (CNN) architecture that introduces residual connections to address the representation learning degradation bottleneck in deep neural networks, enabling progressive nonlinear mapping of features across scales. However, as the number of neural network layers increases, these problems become more apparent. Therefore, ResNet introduces the idea of "jump connections" or "short-cut paths" that allow input signals to bypass multiple neural layers and be passed directly to deeper layers, thereby alleviating the above problems. This structure does not require the network to learn the complete mapping function, but only the residual between the input and the desired output, reducing the training difficulty and improving the convergence speed and performance of the network. This residual learning approach makes it easier for the network to learn the identity mapping and effectively transmit information. The residual block structure learns the residual in the input and adds it to the input, achieving a balance between efficiency and performance, effectively alleviating the representation learning degradation bottleneck in deep neural networks and enabling progressive nonlinear mapping of features across scales, as shown in the following equations:

[0130] F(x) = H(x) - x (34)

[0131] H(x) = F(x) + x (35) H(x) is the output of the residual block, and F(x) is the output after processing by the convolutional layer and activation function. This structure allows the network to learn the residual rather than the direct mapping, making it easier to train the network.

[0132] Figure 6 is the structure of a residual block, with the specific sequence of convolutional layer - batch normalization - activation function - second convolutional layer - again batch normalization - residual connection. The input is x, and normalization is used to improve the stability and convergence speed of the model. The activation function introduces nonlinearity, allowing the network to learn complex functions.

[0133] z1 = Conv(x, W1), z2 = BN(z1), z3 = ReLU(z2) (36)

[0134] z4 = Conv(z3, W2), F(x) = BN(z4), H(x) = F(x) + x (37)

[0135] The present application uses three residual modules in the encoder part, each consisting of two fully connected layers (FC), with ReLU as the activation function. The input state vector is first upgraded to 64 dimensions by a FC layer, then passes through multiple residual blocks to extract features, and finally is downgraded to 32 dimensions to output the mean and log variance of the latent variable.

[0136] The TPPO-VRN algorithm in this embodiment can not only improve the optimization ability of the scheduling strategy, but also enable the model to exhibit better performance in dynamic and complex multi-energy systems and adapt to the challenges brought by future uncertainties. Table 1 shows the pseudo code of the algorithm, and the comprehensive block diagram of the algorithm is as shown in Figure 7

[0137] Pseudo code:

[0138] Table 1 Algorithm Pseudo Code

[0139]

[0140]

[0141] The following solves the specific application scenario to further illustrate this embodiment:

[0142] The electric energy storage capacity of the compressed air energy storage device in this embodiment is 50 megawatt-hours. The data set used includes one year of demand data from a multi-functional community (including residential and commercial buildings) in China and a wind and photovoltaic power generation data set from a power plant. During data processing, 80% of the data was allocated as a test set for algorithm training and hyperparameter optimization, and 20% of the data was designated as a training set. To confirm the effectiveness of model training and testing, an outlier check was performed on the data. After normalizing, cleaning and preprocessing the historical scheduling data, the high-quality data set generated was used as the expert data set. To ensure the performance of the algorithm, key hyperparameters were carefully adjusted and experimented. The optimal hyperparameter configuration was finally determined, as shown in Table 2.

[0143] Table 2 Hyperparameters of the Algorithm

[0144]

[0145]

[0146] Figure 8 The learning performance of the proposed TPPO-VRN algorithm and two benchmark methods is shown. The graph shows the average reward obtained by each algorithm in 800 training rounds, and the shaded area represents the fluctuation of the reward. As Figure 8 ​As shown, all three algorithms exhibit an initial fast learning phase. However, the TPPO-VRN algorithm significantly outperforms the other two algorithms in terms of reward. Specifically, at the 400th episode, the average reward of TPPO-VRN is approximately -12000, while the rewards of TPPO and TPPO-VAE are approximately -17000 and -15000, respectively. This indicates that TPPO-VRN can achieve higher performance within the same training time. In the early stage of training, all methods exhibit similar learning curves, indicating that they are all exploring the environment and gradually improving the scheduling policy. However, as training progresses, TPPO-VRN is able to converge to a higher reward level more quickly and exhibits smaller volatility, indicating that it can more stably cope with the uncertainty factors in the system, reflecting its more stable learning process. Compared with TPPO-VAE, the addition of ResNet makes TPPO-VRN perform better, indicating that the ResNet structure can effectively improve the utilization of the feature representation learned by VAE and further improve the optimization ability of the policy. By comparing the performance of TPPO-VAE and standard TPPO, it can be found that the introduction of VAE can improve the performance of the scheduling policy. This verifies the feasibility of using deep learning techniques to extract useful information from energy system data. However, simply using VAE may not be able to fully utilize its potential, so ResNet is used to further improve performance. The results show that combining variational autoencoder with residual network can effectively improve the performance and stability of TPPO algorithm in complex multi-energy system scheduling problems.

[0147] Figure 9 The impact of different constraint values on system operating costs is analyzed. The results show that as the values of maximum storage capacity and maximum load constraints increase, the system's cost tends to decrease. More specifically, as the system power (MW) increases, the operating cost of the system tends to decrease regardless of whether the maximum load constraint or the maximum capacity constraint is used. However, at the same power level, the operating cost with the maximum capacity constraint is slightly higher than that with the maximum load constraint. This reduction in operating cost can be attributed to higher constraint settings, which allow the system to more flexibly schedule resources and more effectively meet load demands. Increasing the constraint value can reduce operating costs, thereby reducing energy expenditures and improving the economic efficiency of the system. Figure 10 A typical summer load demand curve is shown. As can be seen from Figure 10 , the power load exhibits a clear peak-valley characteristic throughout the day, gradually increasing in the morning, peaking at noon, and then gradually decreasing, maintaining a relatively stable level at night. The heating load has two peaks in the morning and evening, which may be related to residents' daily life habits. The cooling load is mainly concentrated in the noon period, which is consistent with the trend of air temperature, indicating that cooling demand is significantly affected by environmental temperature.

[0148] Figure 11 The cost comparison results of different strategies are shown. It can be seen that the TPPO-VRN strategy achieves the lowest cost of 28755 yuan, while the TPPO strategy has the highest cost of 31734 yuan. The results show that the TPPO-VRN strategy saves about 9.39% of the cost relative to the TPPO strategy, and the TPPO-VAE strategy saves about 7.29% of the cost. In addition, the TPPO-VRN strategy also saves about 2.26% of the cost relative to the TPPO-VAE strategy. These results indicate that the TPPO-VRN strategy is superior to the other two strategies in terms of cost-effectiveness.

[0149] Figure 12 The power output of wind turbines (WT) and photovoltaics (PV) during the day is clearly shown, both of which share a portion of the electrical load, reflecting the use of renewable energy. Additionally, the grid's power fluctuates between positive and negative, reflecting the energy exchange behavior between the system and the grid. The power curve of the advanced adiabatic compressed air energy storage (AA-CAES) shows a clear charging and discharging process, especially during peak and trough periods, indicating its important role in balancing supply and demand in the system. The negative power output of the electric heat pump (EHP) indicates that it is operating in cooling mode to meet the cooling load demand of the system. Figure 12 The electrical dispatch results for a typical summer day are shown. From the graph, it can be seen that wind turbines (WT) and photovoltaics (PV) contribute a portion of the electricity during the day, indicating a preference for using renewable energy sources for power generation. The grid's power output fluctuates between positive and negative, reflecting the two-way interaction between the system and the grid, possibly to balance supply and demand or participate in the electricity market. The advanced adiabatic compressed air energy storage (AA-CAES) charges during periods of low load and discharges during peak periods, exhibiting a peak shaving and filling mode of operation. In addition, the electric heat pump (EHP) consumes electricity at certain times, possibly for cooling or heating, depending on the load demand at that time.

[0150] Figure 13 In this case, the electric heat pump (EHP) seems to be taking on the main heating task, with its power output relatively stable and maintaining a high level throughout the day. Wind power (WT) contributes less, indicating that heat production mainly relies on the electric heat pump. The absorption chiller (AC) operates at certain times, but its power output is relatively low, indicating that the cooling demand is not high. The advanced adiabatic compressed air energy storage (AA-CAES) has a relatively moderate charging and discharging behavior, which may be mainly used to smooth the fluctuations in heat load. Overall, the heat dispatch strategy focuses more on meeting the basic heat load demand and uses less of other forms of energy.

[0151] Figure 14In this case, the absorption chiller (AC) dominates and operates at a high power most of the time, indicating that the system is primarily targeting to meet the cooling load demand. The electric heat pump (EHP) also operates continuously, possibly providing a portion of the cooling or assisting in maintaining the system temperature. The advanced adiabatic compressed air energy storage (AA-CAES) has a relatively small variation in power output, indicating that its role in cooling dispatch is limited and can only be used to provide a base load. Therefore, the main feature of this cooling dispatch strategy is the reliance on the absorption chiller to meet the cooling load demand.

[0152] In the uncertainty analysis of renewable energy systems, the inventors explored the impact of wind and photovoltaic output on daily operating costs under different fluctuation rates through systematic experiments. The study set a fluctuation range of ±2% to ±10%, and examined the dispatch characteristics of single and multiple uncertainty factors, respectively, as shown in Figure 15 and Figure 16 .

[0153] Figure 15 The system dynamics of single uncertainty factor are revealed: as the fluctuation rate increases, the daily operating cost exhibits significant fluctuation characteristics, and the economic benefit shows a continuous downward trend. In contrast, Figure 16 The system resilience of multiple uncertainty factors is demonstrated: the fluctuation range of the comprehensive operating cost is significantly narrowed, and different uncertainty factors achieve mutual offset through complex interactions, enabling the system to maintain economic stability in the face of increasing fluctuation rates. This phenomenon highlights the unique value of multi-factor coupling analysis: compared to linear analysis of single factors, multi-dimensional uncertainty assessment can reveal deeper adjustment mechanisms of the system, effectively reduce the fluctuation range of the dispatch results, and maintain the overall economic benefit of the system in the context of continuously increasing fluctuation rates. Therefore, considering multiple uncertainty factors simultaneously better reflects the adaptive ability and robustness of the system in complex environments. In addition, the cost fluctuation range of wind power is significantly larger than that of photovoltaic power. Under the ±10% fluctuation scenario, the operating cost of the wind power system is close to 2.7×10 4 , and the operating cost of the photovoltaic system is about 2.6×10 4 , indicating that the operating cost of wind power is more sensitive to uncertainty, which may be due to the more complex randomness and multi-factor coupling characteristics of wind power systems. These results indicate that the impact of uncertainty factors needs to be fully considered when evaluating the economic performance of wind and photovoltaic projects, especially in high volatility environments.

[0154] The embodiment aims at the energy scheduling complexity and uncertainty problem existing in multi-energy system, and proposes an innovative scheduling strategy TPPO-VRN algorithm based on reinforcement learning. By constructing a comprehensive multi-energy system model, the uncertainty of renewable energy is accurately modeled by using interval prediction method and robust optimization theory, and the multi-dimensional influence of system constraint conditions and renewable energy fluctuation on system economy is analyzed in depth. The research results show that, with the increase of renewable energy fluctuation rate, the system operation cost shows an upward trend, and the cost fluctuation range of wind power is significantly larger than that of photovoltaic, which provides important insights for the economic evaluation of multi-energy system.

[0155] As can be seen from the above, the TPPO-VRN algorithm proposed in the embodiment solves the problems of strategy sparsity and dimension disaster faced by traditional reinforcement learning in complex energy scenarios. By introducing variational autoencoder (VAE) and residual network (ResNet), the complexity of high-dimensional state space is effectively reduced, and the feature extraction and representation ability is improved. Combined with the proximal policy optimization (TPPO) algorithm, through the clipping mechanism and KL divergence constraint, more stable and efficient policy learning is realized, and the accuracy and robustness of energy scheduling are improved. The research results show that compared with the traditional TPPO algorithm, the system operation cost can be reduced by 7.29%, and an intelligent scheduling scheme considering economy and reliability is provided for multi-energy system with high proportion of renewable energy.

[0156] It is apparent for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all aspects as illustrative and not restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and range of equivalents of the elements of the claims are intended to be embraced therein. Any reference signs in the claims should not be considered as limiting the claims involved. Furthermore, it is apparent that the word "comprise" does not exclude other elements or steps, and the singular does not exclude the plural. The plurality of units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to express names and not to express any particular order.

Claims

1. An integrated multi-energy system energy scheduling method, characterized in that, Comprise the following steps: S1, constructing a system framework: constructing a collaborative scheduling framework for high-dimensional energy systems, covering the complete link of energy input, energy conversion, energy storage and energy consumption, forming a "source-network-load-storage" structure; S2, constructing a system model on the basis of the system framework, the system model comprising an energy input model, an energy conversion model, an energy storage model and an objective function, the objective function being constructed with the goal of minimizing total operating cost; S3, converting the scheduling problem of the system model into a Markov decision process containing uncertain factors; S4, using a TPPO-VRN algorithm to improve the optimization capability of the Markov decision process, the TPPO-VRN algorithm being an algorithm that, in the Markov decision process, effectively reduces the complexity of the high-dimensional state space by introducing a variational autoencoder (VAE) and a residual network (ResNet), improves the feature extraction and representation capability, and then uses a temporal difference (TD) algorithm to optimize the strategy to obtain the optimal scheduling strategy.

2. The method of claim 1, wherein: In step S1, the system framework integrates renewable energy sources including wind power generation and photovoltaic power generation, and ensures the stability and reliability of energy supply through public grid access, and is equipped with energy storage devices including compressed air energy storage and water storage tanks for storing excess energy and releasing it when needed, thereby realizing "peak load shifting" of the system, balancing energy supply and demand, and the system framework uses absorption chillers and electric heat pumps to realize energy conversion and optimal utilization.

3. The method of claim 2, wherein, In step S2, the objective function is described as follows: C em (t) = C AA-CAES (t) + C EHP (t) + C AC (t) + C WST (t) where: C total Ctotalis the total operating cost of the system framework; C grid Cgrid(t) is the cost of electricity purchased from the grid; C em Ctotal(t) is the total operating cost of each device, C AA-CAES Cgrid(t), C EHP Cgrid(t), C AC Cgrid(t), C WST Ccaes(t), Cehp(t), Cabs(t), and Cwater(t) represent the operating costs of compressed air energy storage, electric heat pump, absorption chiller, and water tank, respectively.

4. The method of claim 3, wherein: In step S2, the objective function ensures the feasibility and safety of the scheduling scheme by adding operating constraints, including energy balance constraints, renewable energy constraints and operation constraints, wherein the goal of the energy balance constraints is to ensure that the supply and demand of various energies in the system framework are balanced at any time, the renewable energy constraints include output power constraints for photovoltaic power generation and output power constraints for wind power generation, and the operation constraints limit the output power of each device in the system framework to a range that ensures safe operation of the entire system framework.

5. The method of claim 4, wherein: In step S3, in the Markov decision process, the state space is used to describe the operating state of the system framework, the action space represents the scheduling strategy, the transition probability describes the state evolution, and the reward function quantifies the performance of the strategy.

6. The method of claim 5, wherein: In step S4, in the TPPO-VRN algorithm, the variational autoencoder (VAE) can learn to compress the high-dimensional state space into a low-dimensional hidden space, which can capture the key features in the state space, thereby reducing the difficulty of learning the strategy; the residual network (ResNet) extracts useful features from the high-dimensional input and serves as the encoder and decoder of the variational autoencoder (VAE), improving the reconstruction and generation capabilities of the variational autoencoder (VAE), thereby compressing the high-dimensional vector in the action space into a low-dimensional hidden vector, and then the temporal difference (TD) algorithm learns the strategy based on these hidden vectors.

7. The method of claim 6, wherein: In step S4, the proximal policy optimization (TPPO) algorithm realizes more stable and efficient policy learning through the clipping mechanism and the KL divergence constraint.

8. The energy scheduling method of the integrated multi-energy system according to any one of claims 2-7, characterized in that: In step S2, the energy input model includes an uncertainty factor model, and for wind power generation and photovoltaic power generation, the uncertainty factors include wind speed, light intensity, weather conditions and geographical location; the uncertainty factor model uses the robust optimization theory to accurately model the uncertainty of wind power and photovoltaic power generation output, and uses the opportunity constraint method based on probability classification to establish a hierarchical adjustable uncertainty set of wind power and photovoltaic power generation output, so that different probability levels of wind power and photovoltaic output can be distinguished through hierarchical processing, and then the fine control of the risk level of the dispatching strategy can be realized.

Images