Method and system for optimizing an integrated energy system under multiple uncertain environmental conditions

Abstract

The present invention relates to the technical field of integrated energy systems and provides a method and system for optimizing an integrated energy system under multiple uncertain environments, the method comprising the steps of acquiring the state of the integrated energy system and acquiring the actions of a multi-energy flow facility through an actor network, incorporating an MPC actor into a double-delay deep deterministic policy gradient architecture, storing the transition tuples of the actor network and MPC actor in an agent experience regeneration pool and an expert experience regeneration pool, respectively, assigning a priority to each sample based on the time difference error of the samples in both experience regeneration pools, determining a mixing ratio according to the training time step, and calculating the sampling probability of the sample by combining the mixing ratio and priority, thereby controlling the updating of the critical network to rely more heavily on the expert experience regeneration pool in the early stages of training and more heavily on the agent experience regeneration pool in the later stages of training. Better performance is obtained in IES environments with multiple uncertainties combined.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This invention claims priority to the Chinese patent application filed with the China National Intellectual Property Administration on September 16, 2025, with application number 202511315858.1, titled "Method and System for Optimizing an Integrated Energy System under Multiple Uncertain Environments," the entirety of which is incorporated as part of this invention and constitutes part of this invention for all purposes.

[0002] The present invention belongs to the technical field of integrated energy systems, and more particularly to a method and system for optimizing integrated energy systems under multiple uncertain environments. [Background technology]

[0003] This section is intended solely to provide background information related to the present invention and does not necessarily constitute prior art.

[0004] Compared to conventional independent and distributed energy systems, Integrated Energy Systems (IESs) leverage the fundamental principles of multi-energy complementarity and coordinated optimization to overcome operational barriers between different energy categories and improve renewable energy consumption and overall system energy efficiency. However, their multiphysics coupling characteristics, while enabling the complementarity of multiple energy sources, introduce significant risks to the transmission of multiple uncertain domains, directly threatening the operational safety of the system.

[0005] In conventional research, external uncertainties in IESs are generally separated into three categories: renewable energy sources such as wind and solar power, diverse loads such as cooling, heating, and electricity, and energy prices. Conventional scheduling methods often focus on a single or local uncertainty factor. Based on conventional mechanisms such as robust optimization, stochastic optimization, and information gap decision-making, pseudo-joint modeling is achieved by simple linear superposition of constraint boundaries or stochastic scenario enumeration, but it is difficult to characterize the essential relationships between multiple uncertainties. As a direct result, covering all scenarios requires introducing numerous boundary constraints and redundant scenarios, resulting in overly conservative operational strategies and significant resource waste when dealing with combinations of low-probability or impossible scenarios. Furthermore, the above methods usually require mathematical models and forecasting information, and the reliability of the final decision depends heavily on the accuracy of the system model and forecasting method used, making them difficult to adapt to high-dimensional and non-convex uncertainty spaces such as wind, solar power, cooling, heating, electricity, and prices.

[0006] Artificial intelligence is fundamentally transforming how we solve energy problems. A prime example is Deep Reinforcement Learning (DRL), with its end-to-end model-free learning properties, which overcomes the dimensional curse of traditional mechanistic modeling and can mine implicit correlation rules in a data-driven manner in high-dimensional, non-convex, and uncertain spaces. However, despite the significant theoretical advantages of DRL in energy management, several drawbacks remain for practical application. Firstly, typical algorithms such as the double-delay policy gradient (TD3) are highly susceptible to hyperparameter influences, are difficult to tune, and have poor generalization capabilities. Although their upper limit is high, their lower limit is also low, resulting in insufficient stability. Secondly, designing the reward function is extremely complex, and directly using traditional operating costs as rewards makes it easier for rewards to converge while agent actions violate the system's hard constraints. Even if a penalty term is introduced to represent the constraints, it remains difficult to effectively guarantee the feasibility of those constraints. Thirdly, the power, load, and price environmental parameters in IES are only partially observable and do not fully satisfy the Markov decision assumption, which limits agent learning and leads to unstable performance. Finally, because IES has a high-dimensional joint state-action space, very sparse rewards are obtained in the early stages of training. Without prior knowledge, the agent often converges action values ​​to the boundary region and is unable to adequately explore the optimal policy. Furthermore, the training process is slow to converge and inefficient.

[0007] However, this does not mean that DRLs have fundamental limitations in the field of scheduling; their powerful adaptive environment representation capabilities offer invaluable advantages in addressing multiple uncertainties across multiple domains. The central challenge lies in how to improve the agent-environment interaction in DRLs in line with the essence of IES optimization problems, while maintaining the advantages of artificial intelligence. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] To address the technical challenges in the background technologies described above, the present invention provides a method and system for optimizing integrated energy systems under multiple uncertain environments. A collaborative optimization framework of Twin Delayed Deep Deterministic Policy Gradient (TD3) and Model Predict Control (MPC) is proposed. Based on this, a hybrid preferred experience replay mechanism is designed to balance expert experience and autonomous exploration. Through dynamic adjustment of sampling weights in a double buffer, the system prioritizes learning from demonstration experiences generated by MPC actors in the early stages of training, avoiding the blind spots of random policies. As interactive data accumulates, the autonomous sampling weights gradually increase, encouraging the agent to autonomously explore and adapt to cross-domain uncertainty correlation rules for power, load, and price. This results in superior performance in IES environments with multiple combined uncertainties.

[0009] To achieve the above objectives, the present invention employs the following technical solutions.

[0010] In a first aspect of the present invention, a method and system for optimizing an integrated energy system under multiple uncertain environments are provided, the method being: Steps to obtain the status of the integrated energy system, This includes the step of obtaining the action of a multi-energy flow device through an actor network based on its status, During the actor network training process, MPC actors are incorporated into a dual-delay deep deterministic policy gradient architecture. Transition tuples of the actor network and MPC actors are stored in the agent experience replay pool and expert experience replay pool, respectively. Priorities are assigned to each sample based on the time difference error of samples in the two experience replay pools. A mixing ratio is determined according to the training time step. The sampling probability of a sample is calculated by combining the mixing ratio and priority. This controls the updates of the critic network to heavily rely on the expert experience replay pool in the early stages of training and heavily rely on the agent experience replay pool in the later stages of training, guiding the actor network to optimize its output actions via the value gradient of actions by the critic network.

[0011] Furthermore, the aforementioned state includes wind power, solar power, heat load, electrical load, gas load, electricity price, output power of combined heat and power (CH) equipment, input power of power-to-gas equipment, and the charge state of energy storage within a time segment that has the current time period as its midpoint.

[0012] Furthermore, the actions of the multi-energy flow equipment include changes in charging power, changes in discharging power, changes in purchased power, changes in input gas power of the combined heat and power unit, changes in input power of the power-to-gas unit, changes in input power of the gas boiler, and changes in purchased gas power.

[0013] Furthermore, during the training of the actor network, at each time step, The MPC actor selects an action based on the current time step's state, interacts with the environment to obtain a reward and the state for the next time step, saves the MPC actor's transition tuple to the expert experience replay pool, and sets the priority of each sample in the expert experience replay pool. The actor network selects an action based on the current time step's state, interacts with the environment to obtain a reward and the state for the next time step, saves the actor network's transition tuple to the agent experience replay pool, and sets the priority of each sample in the agent experience replay pool. If the current time step is an integer multiple of the experience replay interval, the following steps are taken: Based on the expert experience replay pool and the agent experience replay pool, in combination with the target network, multiple mini-batch update operations are performed on the critical network to update the priority of samples in the experience replay pool. If the current time step is an integer multiple of the set interval, the step to update the actor network and target network is executed.

[0014] Furthermore, the aforementioned compensation is the inverse of the sum of the operating costs of the integrated energy system, the penalties for curtailing wind and solar power generation output, and the penalties for exceeding equipment limits.

[0015] Furthermore, the aforementioned mini-batch update operation is The steps include sampling transition data from an experience-recreated pool according to the sampling probability of the sample, The steps include: calculating importance sampling weights based on sampling probabilities for sampled transition data; In the current time step, the steps involve generating a target action using a target actor network, adding Gaussian noise to the target action, calculating a time-difference target based on the target critic network, calculating the difference between the time-difference target and the result of the critic network, and obtaining the time-difference error. A step to update the priority of samples in the experience regeneration pool based on time difference error, This includes the steps of calculating the loss of the critical network based on the time difference error and importance sampling weights, and updating the critical network parameters.

[0016] Furthermore, the sampling probability is as follows: [Number] Here, P j is the sampling probability of the j-th sample, and the linear weight decay coefficient is [Number] where X init and X final are the initial and final decay coefficients respectively, T is the total number of training steps, l represents the current time step, type j =MPC indicates that the j-th sample is obtained from the expert experience replay pool, type j =Agent indicates that the j-th sample is obtained from the agent experience replay pool, D agent and D mpc are the number of samples in the agent experience replay pool and the expert experience replay pool respectively, β is the priority intensity coefficient, p j is the priority of the j-th sample, p z and p d represent the priorities of samples from the agent experience replay pool and the expert experience replay pool respectively.

[0017] In the second aspect of the present invention, an optimization system for an integrated energy system in a plurality of uncertain environments is provided, and the system includes a data acquisition module configured to acquire the state of the integrated energy system, and an optimization module configured to acquire the actions of the multi-energy flow devices through the actor network based on the state. During the actor network training process, MPC actors are incorporated into a dual-delay deep deterministic policy gradient architecture. Transition tuples of the actor network and MPC actors are stored in the agent experience replay pool and expert experience replay pool, respectively. Priorities are assigned to each sample based on the time difference error of samples in the two experience replay pools. A mixing ratio is determined according to the training time step. The sampling probability of a sample is calculated by combining the mixing ratio and priority. This controls the updates of the critic network to heavily rely on the expert experience replay pool in the early stages of training and heavily rely on the agent experience replay pool in the later stages of training, guiding the actor network to optimize its output actions via the value gradient of actions by the critic network.

[0018] Furthermore, the aforementioned state includes wind power, solar power, heat load, electrical load, gas load, electricity price, output power of combined heat and power (CH) equipment, input power of power-to-gas equipment, and the charge state of energy storage within a time segment that has the current time period as its midpoint.

[0019] Furthermore, the actions of the multi-energy flow equipment include changes in charging power, changes in discharging power, changes in purchased power, changes in input gas power of the combined heat and power unit, changes in input power of the power-to-gas unit, changes in input power of the gas boiler, and changes in purchased gas power.

[0020] Compared to conventional technology, the beneficial effects of the present invention are as follows: This invention proposes a collaborative optimization framework for TD3 (Twin Delayed Deep Deterministic Policy Gradient) and MPC. Based on this, a hybrid priority experience playback mechanism is designed to balance expert experience and autonomous exploration. Dynamic adjustment of sampling weights in a double buffer prioritizes learning demonstration experiences generated by MPC actors in the early stages of training, avoiding the blind spots of random policies. As interactive data accumulates, the autonomous sampling weights gradually increase, encouraging the agent to explore autonomously and adapt to cross-domain uncertainty correlation rules for power, load, and price.

[0021] The TD3 and MPC collaborative optimization framework proposed in this invention inherits the DRL's ability to learn from the environment and address uncertainty, and the MPC's ability to provide basic performance using a physical model. Compared to conventional TD3, the TD3MPC reward function is simpler to design, converges faster, yields better results, and significantly reduces the difficulty of parameter tuning.

[0022] This invention enhances TD3's ability to generalize to complex energy problems by extending a single-time state to an extended state space that includes past and future information, and by modeling the state transition characteristics of a Markov decision process based on probabilistic scenario techniques.

[0023] The drawings of the specification, which constitute part of the present invention, are provided for further understanding of the present invention, and the exemplary embodiments and descriptions of the present invention are used to illustrate the present invention and do not unduly limit the present invention. [Brief explanation of the drawing]

[0024] [Figure 1] Figure 1 is a diagram of the TD3MPC architecture of Embodiment 1 of the present invention. [Figure 2] Figure 2 is a diagram of the IES structure of Embodiment 1 of the present invention. [Figure 3]Figure 3 shows the convergence result of TD3MPC in Example 1 of the present invention. [Figure 4] Figure 4 shows the comparison results of convergence in Example 1 of the present invention. [Figure 5] Figure 5 shows the multi-scenario scheduling results of Embodiment 1 of the present invention. [Modes for carrying out the invention]

[0025] To further clarify the object, technical solution, and advantages of the embodiments of the present invention, the technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention.

[0026] It should be noted that the following detailed description is illustrative and intended to further illustrate the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the invention belongs.

[0027] Example 1 This embodiment provides a method for optimizing an integrated energy system under multiple uncertain environmental conditions.

[0028] A fundamental solution to the problems described in the background technology section lies in compressing the agent's search space into a physically feasible domain and a high-value decision domain, thereby increasing reward density. One feasible approach is to introduce an expert guidance mechanism into DRL. Model Predictive Control (MPC) is a preferred choice for experts due to its open architecture and mature theoretical system. In practice, MPC and DRL each have their strengths and weaknesses and complement each other. The collaboration between the two can not only solve problems such as DRL's hyperparameter sensitivity, reward sparsity, and low interpretability through physical guidance, but also compensate for MPC's inherent shortcomings such as model dependency and myopia. Given the multiple uncertainties faced by IES, the synergistic effect of MPC and DRL is expected to play a crucial role.

[0029] The method for optimizing an integrated energy system under multiple uncertain environments provided by this embodiment proposes a collaborative optimization framework for TD3 and MPC. In this framework, Model Predict Control (MPC) is incorporated as an expert actor into the Twin Delayed Deep Deterministic policy gradient (TD3) architecture, providing a security baseline action sequence that conforms to physical constraints through rolling time-domain optimization. Based on data, TD3 adapts to environments with multiple uncertainties by adaptively exploring within its neighborhood to compensate for model inconsistencies. This design incorporates a hybrid preferred experience retrieval mechanism to balance expert experience and autonomous exploration. Based on this, the generalizability of TD3 to complex energy problems is enhanced by extending the time state of a single time point to an extended state space that includes past and future information, and by modeling the state transition characteristics of a Markov decision process based on probabilistic scenario techniques.

[0030] The method for optimizing an integrated energy system under multiple uncertain environments provided by this embodiment includes the following steps: Step 1: Obtain historical power source, load, and price data for the integrated energy system, establish a physical model of Park that includes the objective function (reward function), equality constraints, and inequality constraints, and convert it into a Markov decision process.

[0031] As shown in Figure 2, the IES demonstration park in northern China, which was the subject of this study, includes three different energy sources: electricity, heat, and gas, all of which are centrally managed by a unified operating entity. The park acquires energy from the upstream power grid, gas distribution network, wind power plants, and solar power plants, and interconnects them through coupling devices such as combined heat and power (CHP), gas boilers (GB), and power-to-gas (P2G) to form a multi-energy flow interconnected network that meets the diverse energy needs of end users. Renewable energy, multi-energy loads, and random fluctuations in price signals propagate through physical coupling and market mechanisms, seriously impacting the stable operation of the IES. The integrated energy system targeted in this study is a typical low-voltage distribution network architecture, and because it has a compact electricity, heat, and gas network topology structure and short transmission distances, the impact of line parameters and pipeline pressure losses on the network's power flow distribution can be ignored. The integrated energy system is modeled according to the structural diagram of the integrated energy system.

[0032] (1) Energy conversion equipment. Energy conversion equipment in an integrated energy system includes CHP, P2G, and GB. The input / output power balance equation is as follows:

number

number

number

number

number

number

[0033] (2) Energy storage systems. The integrated energy system includes an energy storage system. The energy charge / discharge balance model is as follows:

number

number

number

[0034] (3) Rated power and lamp power constraints.

number

number

number

number

[0035] (4) Power balance constraints.

number

number

number

number

number

number

number

number

number

number

[0036] Step 2: To ensure consistency in the Markov decision process, the conventional TD3 state space is adaptively improved by designing an extended state space that includes past and future information. Based on probabilistic scenario generation techniques, the transition probabilities of random state variables are modeled to prevent anomalous probabilistic scenarios from interfering with the overall policy. Based on a conventional actor network, an MPC Actor is established based on MPC, explicitly considering the state and input physical constraints to generate a baseline control action sequence with safety and economy, which is stored in the Expert Experience Replay Pool (Demo buffer) along with the state-reward signal. Its objective function is co-designed with an improved TD3 reward function to ensure consistency in the optimization target. The improved TD3 agent further explores the globally optimal policy in the neighborhood of the baseline control actions obtained from the Demo buffer, and stores the exploration experience in the Agent Experience Replay Pool (Agent buffer).

[0037] The main advantage of MPC lies in its ability to generate optimal control sequences by leveraging dynamically updated system state and environmental information and explicitly solving constrained optimization problems online. It is noteworthy that MPC and DRL are mathematically remarkably isomorphic in that they use state variables, control variables, and objective functions as their fundamental framework. This formal similarity is another reason to choose MPC for physical induction. MPC Actor inherits the standard MPC framework, using a state-space model to describe the system's dynamic characteristics and generating a baseline control strategy in deterministic scenarios. Unlike traditional MPC, which uses only the control variables of the first step, MPCActor stores the optimal control sequence and its corresponding state-reward information in a full time-domain Demo buffer, building a training sample set that matches the extended state-space. Within each rolling time domain, it solves the following optimization problems:

number

[0038] Here, z is the predicted time-domain output, and z max and z min δ and λ represent the maximum and minimum values ​​of the predicted time-domain output, respectively, w is the reference trajectory obtained from one optimization run on the previous day, and the optimization objective function is the same as the TD3 reward function. δ and λ are weight coefficients, and x(t) is the state variable matrix of the multi-energy flow coupling device.

number

number

number

number

number

[0039] The single-step state-space model can be represented as follows:

number

number

number

number

number

number

number

[0040] The TD3 algorithm is an improved Actor-Critic framework proposed for control problems in high-dimensional continuous action spaces. Compared to conventional deep deterministic policy gradient methods, the TD3 algorithm significantly improves policy stability and convergence efficiency through three innovative techniques: a dual-Critic network, delayed policy updating, and target policy smoothing. Modeling the IES as a partially observable Markov decision process, it can be represented by five tuples <S, A, P, R, Γ>. Except for several random variables, the other state transition probabilities P are implicitly defined by the dynamic equations.

number

[0041] In state-space S:DRL, it is typically assumed that the system satisfies a Markov decision process, meaning that future states depend only on the current state and are independent of previous states. However, the power, load, and price random variables in IES often exhibit periodic or inertial characteristics. This causes the Markov property assumption of the conventional TD3 algorithm to fail, and policies that depend only on the current state tend to produce suboptimal actions. Therefore, in this embodiment, an extended state-space is constructed to represent the system state using temporal state fragments containing past and future information, rather than a single state variable. This improves the conventional one-to-one state-action mapping to a many-to-one mapping, thereby improving the completeness of state information. The extended state-space for time zone l includes wind and solar power, heat, electricity, gas loads, electricity prices, and the operating state of the multi-energy flow coupling device, and can be represented as follows:

number

number

number

number

number

number

number

number

number

number

number

[0042] Action Space A: An action refers to a tunable variable. In this embodiment, power balance within the integrated energy system is ensured through the actions of a multi-energy flow device. The TD3 action space corresponds to the action sequence output by the MPC Actor and can be described as follows:

number

number

number

number

number

number

number

number

[0043] State transition probability P: Conventional TD3 algorithms are based on the assumption of Markov decision processes and model state transitions using only a single sample trajectory, making it difficult to characterize the randomness of state transitions caused by multiple uncertainties in the power, load, and price of the IES. Specifically, after action a1 is performed in time zone l, it is not possible to accurately obtain the random state variable in time zone l+1. Relying solely on deterministic predictions leads to insufficient adaptability of the policy network to a stochastic environment.

[0044] In this example, we simulate uncertainty in the Markov decision process using a probabilistic scenario generation technique and probabilistically model the transition characteristics of the random state space. The transition characteristics of other states can be obtained from the system dynamics of the MPC Actor. The TD target is updated using the expected value of the action value function of the random state. The update process can be expressed as follows:

number

[0045] Here, y l This represents the TD target,

number

number

number

number

[0046] Here, s k,l+1 is the kth possible scenario in time zone l+1, and p(s k,l+1 |s l ,a l ) is a scenario k,l+1 This is the probability that occurs, where M is the number of clustered scenarios, and a k,l+1 is the action at time point l+1 under the k-th scenario. To facilitate analysis, the random state space is used.

number

[0047] The reward function R:DRL maximizes long-term rewards through continuous trial-feedback-evaluation, and the inverse of the optimization target is the immediate reward r. l It can be set as follows and expressed as follows:

number

number

number

number

number

number

number

number

number

[0048] Step 4: To balance expert experience and autonomous exploration, a hybrid priority experience playback mechanism is designed, prioritizing the learning of demonstration experiences generated by the MPC Actor in the early stages of training through dynamic adjustment of sampling weights in a double buffer, thus avoiding the blind spots of a random policy. As interactive data accumulates, the autonomous sampling weights are gradually increased, encouraging the agent to explore autonomously and adapt to the cross-domain uncertainty correlation rules of power, load, and price.

[0049] Experience Recall Mechanism: Prioritized experience recall improves upon conventional experience recall by introducing a priority and importance weighting mechanism, prioritizing the learning of samples with high TD errors to improve the efficiency of policy improvement. Considering that the experience recall pool of the proposed framework consists of two parts, the Agent buffer and the Demo buffer, this embodiment designs a new hybrid prioritized experience recall mechanism to balance the contradiction between policy search and knowledge transfer. Note that the prioritized experience recall mechanism is not required when migrating and applying this framework. If the target problem has a dense reward signal and the environment is dynamically stable, additional computational overhead can be avoided and training complexity reduced by directly using uniform recall.

[0050] Prioritize each sample based on the TD error:

number

number

number

number

[0051] Define a linear weight decay coefficient X(l) to control the mixing ratio of expert experience and autonomous exploration experience.

number

[0052] Combining the mixing ratio and priority, the sampling probability of sample j is defined as follows:

number

number

[0053] In response to that,

number

[0054] Thus, a collaborative optimization framework based on TD3MPC is constructed, and the overall framework is shown in Figure 1. It includes the following steps: (1) Initialization of parameters such as mini-batch data size k, experience playback interval K, and total number of training steps T. (2) Network initialization: Two Critic networks Q and Actor networks π are initialized with random parameters θ1 and θ2 respectively.

number

Number

Number

Number

number

number

number

number

number

[0055] To verify the effectiveness of the proposed method, simulation validation was conducted at the IES Demonstration Park in northern China. The hardware environment used for the simulation example was a laptop equipped with an Intel(R) Core(TM) i9-14900HX CPU and 32GB of RAM. Model construction and training were performed using the PyTorch deep learning framework.

[0056] Figure 3 shows the convergence of the agent during training. In the early stages of training, the algorithm is in the exploration phase. Because the initial action selection is based on a deterministic environment, the reward value the agent receives after making a decision is relatively small. After approximately 3000 epochs of training, the agent gradually converges to a stable reward range, indicating that it has learned the optimal scheduling policy. Due to multiple uncertainties in the power, load, and price environments, as well as the algorithm's noise mechanism, the agent's reward value inevitably oscillated during the training process.

[0057] To further highlight the convergence performance of TD3MPC, Figure 4 shows the convergence of the average reward of different algorithms using 10 random seeds. The shaded areas in the figure represent the range of maximum and minimum training results, and the dark dashed lines represent the average value of the 10 training results after smoothing. To clarify the characteristics of the TD3 algorithm itself and the advantages of state-space adaptive improvements, an improved TD3 incorporating state probability transitions and time segment representations was compared with the conventional TD3. From the convergence results, it was found that the improved TD3 stabilized after 9000 steps, while the conventional TD3 only gradually stabilized after 14000 steps, and that the improved TD3 had a slightly higher final convergence reward. This is because the improved TD3 effectively avoids irrational state transitions by modeling the random transition characteristics of the state space, obtaining more accurate Q-value estimates and ensuring the direction of algorithm updates. Furthermore, compared to the conventional single-time state variables, the extended state space provides richer temporal information through environment-to-policy mapping. To demonstrate the positive effects of MPC in DRL, an improved TD3 was further compared with TD3MPC. TD3MPC, guided by the experience of an MPC expert, converged rapidly after approximately 4000 epochs. Compared to the initial completely random search mode, the introduction of the MPC Actor significantly improved the starting point of the search, greatly increased the initial reward, avoided blind trial and error in ineffective action areas, and accelerated the rate of increase in cumulative reward in the early stages. In later stages, the agent continuously learned through hybrid preferred experience retrieval and interaction with the environment, conducting further exploration based on the expert's experience. This allowed the agent to move out of the sample space with a small range of variation and explore towards the optimal solution as the reward function approached a local optimum, resulting in a significant improvement in convergence reward in the later stages.

[0058] The test site is blessed with wind and solar resources, so the majority of the load is supplied by renewable energy. However, fluctuations in wind and solar power significantly impact the stability of the power supply, and furthermore, the uncertainty of electricity prices and diverse loads makes it difficult to maintain a power balance. To verify the effectiveness of the proposed algorithm, one day was randomly selected from winter, summer, and the transition period as a test set, and the scheduling results were analyzed. As shown in Figure 5, the proposed strategy was basically able to maintain the balance of power supply and demand. Energy purchase costs and operation and maintenance costs were lowest in summer because there was no heat load demand. Since the randomly selected test day in the transition period was close to winter, both costs were similar.

[0059] From the perspective of equipment output, the low-cost effect of combined heat and power (CHP) meant that the majority of the power load gap was covered by CHP, and the power grid functioned as a backup resource during periods of low electricity prices. The operating mode of the energy storage system was almost the same in winter, summer, and autumn, in all cases charging when wind and solar resources were abundant and discharging when electricity prices were high. It should be noted that the TD3MPC agents did not blindly choose to charge the energy storage system during periods of abundant renewable energy. Based solely on real-time information, they comprehensively evaluated electricity and gas prices and future trends of various loads, and jointly controlled P2G and the energy storage system to consume renewable energy, thereby enabling the batteries to complete energy storage before peak demand, demonstrating a powerful global capability. Because P2G is expensive, during periods of low renewable energy consumption pressure, the gas demand gap was covered by the gas distribution network. During autumn and winter, the heat load was mainly supplied by CHP. Considering that its operation is affected by multiple energy flows including electricity, gas, and heat, the agent flexibly controls its operation in either a heat-driven power supply mode or an electrically driven heat supply mode, with the remaining thermal load gap being filled by the GB (Ground Booster). [Table 1]

[0060] As can be seen from Table 1, TD3MPC outperforms the other models in all metrics and most closely approximates the ideal situation. Compared to TD3, Improved TD3 reduced wind and solar power curtailment costs, load curtailment costs, and operating costs by 53.93%, 42.86%, and 4.02%, respectively. This means that TD3 tends to adopt more conservative operational strategies and invest more controllable equipment to support the load. However, the cost increase did not improve the situation of wind and solar power curtailment and load curtailment. This is because irrational state transitions rarely occur in actual operation, and over-focusing on them leads to meaningless cost increases. Furthermore, the strong temporal correlation of state parameters in IES violates the Markov principle, resulting in insufficient criteria for decision-making. Improved TD3 addresses these shortcomings by improving the state space and significantly enhances the agent's adaptability to environments with multiple uncertainties in the power, load, and price domains. Compared to Improved TD3, the output curtailment costs, load reduction costs, and total costs of wind and solar power generation in TD3MPC were reduced by 17.28%, 23.74%, and 3.97%, respectively. This indicates that Improved TD3 can converge to the optimal solution by providing a baseline policy that satisfies physical constraints with the MPC Actor. In fact, in this study, we attempted to use RMPC and stochastic MPC as expert policies to induce DRL in the preliminary research stage. However, these MPC variants, which take environmental uncertainty into account, showed significant performance degradation in IES scenarios with multidimensional uncertainty and could not provide a stable and accurate starting point for exploration. In contrast, deterministic MPC was more efficient due to its superior stability. Without subsequent adjustments to Improved TD3, it is difficult to obtain a reliable operational policy with MPC alone. Although the operational costs are low, it is myopic and susceptible to environmental noise, failing to avoid multiple uncertainties and resulting in extremely high output curtailment and load penalties for wind and solar power generation.In summary, the TD3MPC framework, which combines Improved TD3 and MPC, delivers superior performance in IES environments where multiple uncertainties are present.

[0061] RMPC and RO result in lower output curtailment and load curtailment rates for wind and solar power in worst-case scenarios, similar to when faced with local uncertainties. However, in environments with multiple uncertainties such as power sources, loads, and prices, individually modeling multiple worst-case scenarios is extremely costly. Because it is difficult to characterize the combined probability distribution of multiple random variables, RO and CCP fail to provide reliable operational strategies despite their high cost. Compared to conventional optimization methods, DRL adapts to environments with multiple uncertainties by leveraging the powerful representational capabilities of deep learning, and TD3MPC further enhances this characteristic.

[0062] In this embodiment, MPC and DRL exhibit important complementary properties in IES scheduling. The TD3MPC framework inherits the DRL's ability to learn from the environment and address uncertainty, and the MPC's ability to provide basic performance using a physical model. Compared to conventional TD3, the TD3MPC reward function is simpler to design, converges faster, yields better results, and significantly reduces the difficulty of parameter tuning.

[0063] In this embodiment, the proposed hybrid priority experience retrieval mechanism, based on the priority experience retrieval mechanism, achieves a balance between the efficiency of expert experience utilization and the diversity of exploration through a dual-weight priority design. This makes it more suitable for training TD3MPC, reducing training time by 20.87% and yielding better results.

[0064] In this embodiment, the strong temporal correlation of the IES state variables causes the conventional Markov decision assumption of DRL to fail. By extending the state space with a sequence of state variables whose ACF exceeds 0.7 instead of a single-time random variable, pseudo-Markov interference can be effectively eliminated. Based on this, explicitly modeling the transition properties of the extended state space using a probabilistic scenario generation technique significantly improves the agent's ability to estimate uncertainty.

[0065] Example 2 This embodiment provides an optimization system for an integrated energy system under multiple uncertain environmental conditions, and the system specifically provides: A data acquisition module configured to obtain the status of an integrated energy system, Includes an optimization module configured to acquire actions of multi-energy flow equipment through an actor network based on the state, During the actor network training process, MPC actors are incorporated into a dual-delay deep deterministic policy gradient architecture. Transition tuples of the actor network and MPC actors are stored in the agent experience replay pool and expert experience replay pool, respectively. Priorities are assigned to each sample based on the time difference error of samples in the two experience replay pools. A mixing ratio is determined according to the training time step. The sampling probability of a sample is calculated by combining the mixing ratio and priority. This controls the updates of the critic network to heavily rely on the expert experience replay pool in the early stages of training and heavily rely on the agent experience replay pool in the later stages of training, guiding the actor network to optimize its output actions via the value gradient of actions by the critic network.

[0066] Furthermore, the aforementioned state includes wind power, solar power, heat load, electrical load, gas load, electricity price, output power of combined heat and power (CH) equipment, input power of power-to-gas equipment, and the charge state of energy storage within a time segment that has the current time period as its midpoint.

[0067] Furthermore, the actions of the multi-energy flow equipment include changes in charging power, changes in discharging power, changes in purchased power, changes in input gas power of the combined heat and power unit, changes in input power of the power-to-gas unit, changes in input power of the gas boiler, and changes in purchased gas power.

[0068] Note that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and the specific implementation process is the same, so it will not be repeated here.

[0069] The foregoing describes only preferred embodiments of the present invention and is not intended to limit the invention. Those skilled in the art will know that the present invention can be modified and altered in various ways. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for optimizing an integrated energy system under multiple uncertain environmental conditions, The steps include obtaining historical power source-load-price data for an integrated energy system, converting it into a Markov decision process, and obtaining the state of the integrated energy system. This includes the step of obtaining actions for a multi-energy flow facility through an actor network based on its status, During the actor network training process, MPC actors are incorporated into a dual-delay deep deterministic policy gradient architecture. Transition tuples of the actor network and MPC actors are stored in the agent experience replay pool and expert experience replay pool, respectively. Priorities are assigned to each sample based on the time difference error of samples in both experience replay pools. A mixing ratio is determined according to the training time step. The sampling probability of a sample is calculated by combining the mixing ratio and priority. This controls the updates of the critic network so that it heavily relies on the expert experience replay pool in the early stages of training and heavily relies on the agent experience replay pool in the later stages of training. Through the value gradient of actions by the critic network, the actor network is guided to optimize its output actions. During training of the actor network, at each time step, the following steps are performed: the MPC actor selects an action based on the current time step's state, interacts with the environment to obtain a reward and the state for the next time step, saves the MPC actor's transition tuple to the expert experience replay pool, and sets the priority of each sample in the expert experience replay pool; the actor network selects an action based on the current time step's state, interacts with the environment to obtain a reward and the state for the next time step, saves the actor network's transition tuple to the agent experience replay pool, and sets the priority of each sample in the agent experience replay pool; if the current time step is an integer multiple of the experience replay interval, multiple mini-batch update operations are performed on the critic network in combination with the target network based on the expert experience replay pool and the agent experience replay pool to update the priority of samples in the experience replay pool; and if the current time step is an integer multiple of the set interval, the actor network and the target network are updated. The mini-batch update operation includes the steps of: sampling transition data from the experience replay pool according to the sampling probability of the sample; calculating importance sampling weights for the sampled transition data based on the sampling probability; generating target actions using the target actor network in the current time step state; adding Gaussian noise to the target actions; calculating a time-difference target based on the target critic network; calculating the difference between the time-difference target and the result of the critic network to obtain a time-difference error; updating the priority of the samples in the experience replay pool based on the time-difference error; and calculating the loss of the critic network based on the time-difference error and importance sampling weights, and updating the parameters of the critic network. The sampling probabilities are as follows: [Number 91] Here, P ｊ is the sampling probability of the j-th sample, and the linear weight decay coefficient is [Number 92] and X ｉｎｉｔ and X ｆｉｎａｌ are the initial and final attenuation coefficients respectively, T is the total number of training steps, l represents the current time step, type ｊ = MPC indicates that the j-th sample is obtained from the expert experience replay pool, type ｊ = Agent indicates that the j-th sample is obtained from the agent experience replay pool, D ａｇｅｎｔ and D ｍｐｃ are the number of samples in the agent experience replay pool and the expert experience replay pool respectively, β is the priority intensity coefficient, p ｊ is the priority of the j-th sample, p ｚ and p ｄ represent the priorities of the samples from the agent experience replay pool and the expert experience replay pool respectively. A method for optimizing an integrated energy system under multiple uncertain environments is characterized by this.

2. The method for optimizing an integrated energy system under multiple uncertain conditions according to claim 1, characterized in that the state includes wind power generation, solar power generation, heat load, electrical load, gas load, electricity charges, output power of a combined heat and power supply (CH) device, input power for electricity-to-gas conversion, and the charge state of energy storage within a time segment configured with the current time period as the midpoint.

3. The method for optimizing an integrated energy system under multiple uncertain conditions according to claim 1, characterized in that the actions of the multi-energy flow equipment include a change in charging power, a change in discharging power, a change in purchased power, a change in input gas power of a combined heat and power device, a change in input power of a power-to-gas device, a change in input power of a gas boiler, and a change in purchased gas power.

4. The method for optimizing an integrated energy system under multiple uncertain environments according to claim 1, characterized in that the reward is the inverse of the sum of the operating costs of the integrated energy system, the penalties for curtailing wind and solar power output, and the penalties for exceeding equipment limits.

5. An optimization system for an integrated energy system under multiple uncertain environmental conditions, wherein the system uses the method for optimizing an integrated energy system under multiple uncertain environmental conditions described in any one of claims 1 to 4, and the system is A data acquisition module configured to obtain the status of an integrated energy system, Includes an optimization module configured to acquire actions for multi-energy flow equipment through an actor network based on the state, An integrated energy system optimization system under multiple uncertain environments, characterized by incorporating MPC actors into a dual-delay deep deterministic policy gradient architecture during the actor network training process, storing transition tuples of actor networks and MPC actors in an agent experience replay pool and an expert experience replay pool, respectively, assigning a priority to each sample based on the time difference error of samples in both experience replay pools, determining a mixing ratio according to the training time step, calculating the sampling probability of a sample by combining the mixing ratio and priority, thereby controlling the updates of the critic network to heavily rely on the expert experience replay pool in the early stages of training and heavily rely on the agent experience replay pool in the later stages of training, and guiding the actor network to optimize its output actions via the value gradient of actions by the critic network.

6. The optimization system for an integrated energy system under multiple uncertain environments according to claim 5, characterized in that the state includes wind power, solar power, heat load, electrical load, gas load, electricity charges, output power of a combined heat and power system, input power of a power-to-gas system, and the charge state of energy storage within a time segment configured with the current time period as the midpoint.

7. The optimization system for an integrated energy system under multiple uncertain conditions according to claim 5, characterized in that the actions of the multi-energy flow equipment include a change in charging power, a change in discharging power, a change in purchased power, a change in input gas power for a combined heat and power device, a change in input power for a power-to-gas device, a change in input power for a gas boiler, and a change in purchased gas power.

Images