Improved deep reinforcement learning based ies multi-objective dynamic optimization scheduling method

Abstract

This invention discloses a multi-objective dynamic optimization scheduling method for Integrated Energy Systems (IES) based on improved deep reinforcement learning, belonging to the field of optimization scheduling technology for integrated energy systems. This method constructs a multi-objective optimization model that simultaneously considers operating costs and carbon emission costs, and designs an improved DDPG algorithm that integrates a conditional knowledge representation network, a feature attention mechanism, and a curriculum learning strategy. In an integrated energy system simulation environment, following the curriculum learning strategy, an experience replay mechanism and a network parameter update mechanism are used to train the deep deterministic policy gradient algorithm integrating the conditional knowledge representation network and the feature attention mechanism until a convergent and stable optimal scheduling strategy model is obtained and deployed to a real integrated energy system. Experiments show that this invention can significantly reduce carbon emissions while lowering system operating costs, providing reliable and scalable technical support for the safe, economical, and low-carbon operation of IES with a high proportion of renewable energy access.

Description

Technical Field

[0001] This invention belongs to the field of optimization scheduling technology for integrated energy systems, specifically involving an IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning. Background Technology

[0002] With the accelerated pace of global energy transition and the in-depth implementation of the "dual carbon" target, Integrated Energy Systems (IES), as a key carrier for achieving multi-energy complementarity and improving the overall efficiency of energy utilization, have received widespread attention from academia and industry. By coordinating and optimizing the conversion, storage, and distribution of various energy forms such as electricity, heat, cooling, and gas, this system can effectively promote the large-scale consumption of renewable energy, improve the overall energy efficiency of the system, and provide important support for building a clean, low-carbon, safe, and efficient modern energy system. However, the optimal scheduling of IES still faces significant challenges. On the one hand, the output of renewable energy sources (such as photovoltaic and wind power) in the system is highly intermittent and fluctuating; on the other hand, user energy demand exhibits random and time-varying characteristics. This uncertainty on both the source and load sides poses serious problems to the safe and economical operation of the system, making traditional scheduling methods difficult to apply directly.

[0003] Several methods have been proposed for the IES scheduling problem. The first is traditional scheduling methods, such as optimization models based on deterministic prediction and stochastic programming. These methods typically rely heavily on accurate predictions or probability distribution knowledge of uncertainties, and have limited adaptability in real-world dynamic environments, struggling to cope with real-time changes in source-load uncertainty. The second is reinforcement learning methods. As a model-free optimization method, reinforcement learning autonomously learns optimal policies through agent-environment interaction, without relying on accurate prediction models, thus offering advantages in handling uncertainty problems. However, most existing reinforcement learning methods require discretization of the continuous action space, easily leading to the curse of dimensionality and making it difficult to achieve fine and continuous adjustment of equipment output. The third is deep reinforcement learning methods, such as those based on Deep Deterministic Policy Gradient (DDPG). These methods have significant advantages in handling continuous action spaces. However, most existing research focuses on a single economic objective, failing to comprehensively consider environmental impact and the full utilization of renewable energy. For example, the cost of photovoltaic curtailment is often ignored, resulting in insufficient system environmental performance and renewable energy absorption levels. Summary of the Invention

[0004] The purpose of this invention is to provide an improved deep reinforcement learning-based multi-objective dynamic optimization scheduling method for IES, which can effectively solve the problems of weak adaptability to environmental changes, limited decision accuracy, and single optimization objective of existing IES optimization scheduling methods.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: an IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning, comprising the following steps: S1. Establish a multi-objective optimization scheduling model for the integrated energy system to obtain the minimum total operating cost of the integrated energy system; S2. Establish a multi-objective optimization scheduling reinforcement learning model for the integrated energy system, transform the multi-objective optimization scheduling model constructed in step S1 into a Markov decision process, and define the state variables, action variables, and reward function in the reinforcement learning process. S3. Construct and initialize a deep deterministic policy gradient algorithm network, which includes an Actor policy network for generating continuous actions, a Critic value network for evaluating the value of state-action pairs, and their respective target networks. S4. Based on the Actor policy network and Critic value network constructed in step S3, a conditional knowledge representation network is integrated. The conditional knowledge representation network uses historical running data to pre-train an independent feature extraction network, enabling it to output an embedding vector representing the essential features of the current running scenario based on the system's input state data. S5. Embedded feature attention mechanism: After the input layers of the Actor policy network and the Critic value network, an attention weighting layer is added to each of them. This layer is used to dynamically generate attention weights for each feature dimension based on the current input, and then feed the weighted features into the subsequent layers of the network for calculation. S6. Formulate a course learning strategy and train the agent. In the integrated energy system simulation environment, according to the course learning strategy, use the experience playback mechanism and network parameter update mechanism to train the deep deterministic policy gradient algorithm that integrates the conditional knowledge representation network and feature attention mechanism until a convergent and stable optimal scheduling policy model is obtained. S7. Deploy the trained optimal scheduling strategy model to the actual integrated energy system.

[0006] Furthermore, the objective function of the multi-objective optimization scheduling model in step S1 includes operating cost and carbon emission cost, and the constraints include power balance constraints, power interaction constraints with the main grid, and equipment operation constraints; the operating cost includes energy procurement cost, depreciation cost of battery energy storage system, and curtailment cost of renewable energy, and carbon emission cost. Based on the carbon price mechanism and the total carbon emissions of the system, the calculation formula is as follows: In the formula, Indicates time carbon prices, This represents the total carbon emissions during that period. This indicates the carbon emissions generated from purchasing electricity from the main grid. and These represent the carbon emissions of CHP and GB, respectively, both of which originate from natural gas consumption.

[0007] Furthermore, the conditional knowledge representation network in step S4 includes an encoder network and a projection network. The encoder network employs a long short-term memory network, and its expression is: In the formula, and These represent the hidden state and cell state of the LSTM, respectively. Indicates from time arrive The state sequence, where The length of the historical state sequence; The projection network employs a multilayer perceptron structure, mapping the extracted features to a low-dimensional embedding space, and undergoes self-supervised pre-training using a contrastive loss based on InfoNCE. Its expression is as follows: In the formula, Let the conditional representation vector be... This is the weight matrix. and is the bias term, and ReLU is the activation function.

[0008] Further, the attention weighting layer receives an input feature vector composed of the current environment state vector and the conditional representation vector; it calculates the attention weight of each input feature using a weight matrix and a sigmoid activation function; the input feature vector is element-wise multiplied with the attention weight vector to obtain a refined feature representation, which is then input into the Actor policy network and the Critic value network. The attention weight is calculated as follows: In the formula, The feature vector is formed by concatenating the current environment state vector and the condition representation vector. The weight matrix is ​​a learnable matrix. For bias terms, For query vector, My key vector, This represents the Sigmoid activation function. Attention weights for each feature dimension.

[0009] Furthermore, the learning process in step S6 includes two alternating exploration schemes to fill the experience replay buffer: one scheme uses a traditional action selection mechanism that injects exploration noise, and the other scheme uses random action selection until convergence.

[0010] Furthermore, the expression for the traditional action selection mechanism is: in, The shearing function restricts the injection of noise to between its specified upper and lower bounds.

[0011] Furthermore, the training process of the intelligent agent specifically includes the following steps: S61. Initialize the experience replay buffer, all network parameters, and training hyperparameters; S62. Enter the course learning phase. For each training round, reset the environment to its initial state. S63. Select actions based on strategies for the current course stage. Environmental execution actions , transition to a new state And calculate the reward , experience tuple Store in the experience replay buffer; S64. Sample small batches of data from the buffer, update the Actor policy network and Critic value network according to the DDPG algorithm update rules, combine the unique loss function in the conditional knowledge representation network, and softly update the target network parameters. S65: Repeat steps S62 to S64 until all course phases are completed, then proceed to the regular training phase until the policy model converges.

[0012] Furthermore, at each scheduling moment in step S7, the real-time status of the system is collected, input into the optimal scheduling strategy model to obtain the optimal scheduling action instruction, and then sent to each controllable device in the system for execution.

[0013] The beneficial effects of the above technical solution are as follows: This invention constructs a multi-objective optimization model that simultaneously considers operating costs and carbon emission costs, and designs an improved DDPG algorithm framework that integrates conditional knowledge representation networks, feature attention mechanisms, and curriculum learning strategies. This not only effectively extracts essential environmental features to enhance adaptability and generalization capabilities across different operating scenarios, but also dynamically focuses key state information through the attention mechanism to improve scheduling decision accuracy. Furthermore, curriculum learning ensures stable and efficient convergence during the training process. Ultimately, this invention significantly reduces carbon emissions while lowering system operating costs, providing reliable and scalable technical support for the safe, economical, and low-carbon operation of IES with a high proportion of renewable energy access. Attached Figure Description

[0014] Figure 1 An improved deep reinforcement learning framework; Figure 2 Diagram of integrated energy system architecture; Figure 3 For comparison of experimental results; Figure 4 The diagram shows the results of electrical and thermal power scheduling. Detailed Implementation

[0015] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0016] It should be noted that, unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0017] To address the problems of weak adaptability to environmental changes, limited decision-making accuracy, and single optimization objective in existing IES optimization scheduling methods, this invention designs an IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning.

[0018] The IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning includes the following steps: S1. Establish a multi-objective optimization scheduling model for the integrated energy system to obtain the minimum total operating cost of the integrated energy system; S2. Establish a multi-objective optimization scheduling reinforcement learning model for the integrated energy system, transform the multi-objective optimization scheduling model constructed in step S1 into a Markov decision process, and define the state variables, action variables, and reward function in the reinforcement learning process. S3. Construct and initialize a deep deterministic policy gradient algorithm network, which includes an Actor policy network for generating continuous actions, a Critic value network for evaluating the value of state-action pairs, and their respective target networks. S4. Based on the Actor policy network and Critic value network constructed in step S3, a conditional knowledge representation network is integrated. The conditional knowledge representation network uses historical running data to pre-train an independent feature extraction network, enabling it to output an embedding vector representing the essential features of the current running scenario based on the system's input state data. S5. Embedded feature attention mechanism: After the input layers of the Actor policy network and the Critic value network, an attention weighting layer is added to each of them. This layer is used to dynamically generate attention weights for each feature dimension based on the current input, and then feed the weighted features into the subsequent layers of the network for calculation. S6. Formulate a course learning strategy and train the agent. In the integrated energy system simulation environment, according to the course learning strategy, use the experience playback mechanism and network parameter update mechanism to train the deep deterministic policy gradient algorithm that integrates the conditional knowledge representation network and feature attention mechanism until a convergent and stable optimal scheduling policy model is obtained. S7. Deploy the trained optimal scheduling strategy model to the actual integrated energy system.

[0019] The steps of this invention will be described in detail below: like Figure 2 As shown, the integrated energy system includes a combined heat and power (CHP) unit, a photovoltaic (PV) power generation device, a battery energy storage device (BES), a gas-fired boiler (GB), an electric boiler (EB), and user electric-heat loads. These devices are coupled through power, heat, and gas networks to achieve the coordinated conversion, storage, and distribution of electrical and thermal energy.

[0020] Step S1: Establish a multi-objective optimization scheduling model for the integrated energy system to obtain the minimum total operating cost of the integrated energy system.

[0021] For an integrated energy system, its total cost Mainly includes operating costs and carbon emission costs Therefore, the objective function of the multi-objective optimization scheduling model includes minimizing the operating cost. and carbon emission costs Its expression is: (1) Among them, operating costs Includes energy procurement costs Depreciation cost of battery energy storage system And the cost of curtailment of renewable energy (photovoltaics) Its expression is: (2) (3) (4) (5) In the formula, and They represent time respectively Electricity and natural gas prices The duration of each time step, The total number of scheduling cycles. This indicates the exchange of electricity with the main power grid. and These represent the electricity generation of CHP and the heat output of GB, respectively. and These are the electrical efficiency of its CHP and the efficiency of its GB, respectively. The depreciation factor for energy storage charging and discharging is... Indicates BES power; This is the light rejection factor. This indicates the amount of light discarded.

[0022] And carbon emission costs It is calculated based on the carbon price mechanism and the total carbon emissions of the system, and the calculation formula is as follows: (6) (7) In the formula, Indicates time carbon prices, This represents the total carbon emissions during that period. This indicates the carbon emissions generated from purchasing electricity from the main grid. and These represent the carbon emissions of CHP and GB, respectively, both of which originate from natural gas consumption.

[0023] Meanwhile, the IES multi-objective optimization scheduling model also needs to meet power balance constraints, interactive power constraints with the main power grid, and equipment operation constraints to ensure the safe and stable operation of the system.

[0024] Among them, the power balance constraint is used to ensure the real-time balance of power and heat supply and demand, including electrical power balance constraint and thermal power balance constraint, and its expression is: (8) (9) In the formula, and Separation time For electrical load and thermal load, Contribute to photovoltaic power For the electrical power input of EB, For the thermal power output of CHP This is the heat output of GB. Simultaneously, the CHP unit must adhere to a constant heat-to-power ratio operating mode: (10) In the formula, This refers to the thermoelectric ratio of the CHP unit.

[0025] The interaction power constraints and equipment operation constraints between the IES multi-objective optimization scheduling model and the main power grid are mainly constrained by setting upper and lower limits, and their expressions are as follows: (11) (12) (13) (14) (15) In the formula, the superscripts min and max represent the lower and upper limits of power, respectively. Furthermore, energy storage devices must also meet state of charge (SOC) constraints to avoid damage to the equipment from deep charge / discharge cycles. (16) (17) In the formula, For time The state of charge of energy storage, and These are its lower and upper limits, respectively. For the overall efficiency of electric energy storage, This refers to the energy storage capacity.

[0026] S2. Establish a reinforcement learning model for multi-objective optimization scheduling of integrated energy systems.

[0027] To achieve optimal scheduling of IES under complex and uncertain environments, this invention transforms the multi-objective optimization scheduling model in step S1 into a sequential decision problem and constructs a corresponding reinforcement learning solution framework. This method models the dynamic scheduling decision process as a Markov decision process involving continuous interaction between the agent and the environment, thereby achieving real-time response and adaptive optimization to uncertainties. Specifically, the framework mainly includes a state space, an action space, and a reward function r.

[0028] The state space is based on an in-depth analysis of the operating characteristics of IES. The changes in its state variable S include seven key variables, and its expression is as follows: (18) In the formula, and They are respectively Time-based electrical and thermal load demands express Time-based photovoltaic power output Indicates time The BES state of charge, and Indicates time Electricity price and carbon price, Indicates the current scheduling time.

[0029] The action space defines the scheduling operations that the agent can execute at each decision moment. Considering the operational characteristics of each device in the IES, its action variable A contains three continuous control variables, expressed as follows: (19) In the formula, Indicates CHP during the time period Electrical power output, Indicates BES during the time period The charging and discharging power, Indicates EB during the time period Thermal power output, other variables such as , , and It can be derived from constraints and energy balance relationships.

[0030] Since the goal of reinforcement learning is to maximize cumulative reward, and the goal of IES scheduling is to minimize total cost, the reward function r is defined as a negative cost function, expressed as: (20) Wherein, scaling factor This ensures numerical stability during training, and the weight coefficients... and This helps to achieve an adjustable trade-off between operational economics and environmental impact.

[0031] In a certain state of IES When determined, IES dynamic economic scheduling actions The quality of an action can be evaluated using an action-value function, i.e.: (twenty one) In the formula, Representation Strategy The expected value under, The discount factor represents the proportion of the reward at a future point in time in the total accumulated reward. The larger the value, the more emphasis is placed on future rewards. The ultimate goal of IES dynamic economic scheduling is to find the optimal strategy. To maximize the action value function over all state-action pairs: (twenty two) S3. Build and initialize a network based on the deep deterministic policy gradient algorithm.

[0032] The DDPG algorithm is a deep reinforcement learning method based on the Actor-Critic framework, specifically designed to handle control problems in continuous action spaces. This algorithm employs two main neural networks: an Actor policy network and a Critic value network. The Actor policy network is responsible for generating continuous control actions based on the current environment state, and its expression is: (twenty three) in, The parameter is The deterministic strategy function, For time The environmental conditions, This is the output of a continuous action vector. The Critic value network is responsible for evaluating the long-term expected cumulative reward of the state-action pair, i.e., the action-value function: (twenty four) In the formula, For Critic value network parameters, This is the discount factor.

[0033] The Critic value network is trained by minimizing the temporal difference error: (25) Among them, the target value The calculation is as follows: (26) The Actor policy network updates via deterministic policy gradients, expressed as: (27) The target network is updated using soft updates to ensure training stability: (28) (29) in, This is the soft update coefficient.

[0034] S4. Based on the network constructed in step S3, integrate the conditional knowledge representation network.

[0035] To overcome the difficulties in feature extraction caused by the high dimensionality, coupling, and time-varying nature of the source-load state in IES, this invention, based on the DDPG algorithm, pre-trains an independent feature extraction network using historical operating data. This network can output embedding vectors representing the essential features of the current operating scenario based on the system's input state data. .

[0036] like Figure 1 As shown, the conditional knowledge representation network consists of an encoder network and a projection network.

[0037] The encoder network employs a Long Short-Term Memory (LSTM) mechanism to extract time dependencies from state variables within a typical operating day, expressed as follows: (30) In the formula, and These represent the hidden state and cell state of the LSTM, respectively. Indicates from time arrive The state sequence, where The length of the historical state sequence.

[0038] The projection network employs a multilayer perceptron (MLP) structure, mapping extracted features to a low-dimensional embedding space. This network enhances the discriminative power of features while reducing data complexity and redundancy, thereby achieving better separation between similar and dissimilar operating states. Its expression is: (31) In the formula, Let the conditional representation vector be... This is the weight matrix. and is the bias term, and ReLU is the activation function.

[0039] Furthermore, the training of this network does not depend on a specific scheduling task. Instead, it employs self-supervised pre-training using contrastive loss based on InfoNCE, aiming to maximize the similarity of positive sample pairs between states at different time points within the same runtime scenario, while minimizing the similarity between states across different scenarios. The resulting... As a supplementary representation of the environmental state, it is input into the subsequent decision network along with the original observed state.

[0040] S5, Embedded Feature Attention Mechanism.

[0041] To address the dynamic changes in the importance of various state features during IES operation (e.g., during peak electricity price periods, features such as electricity load and energy storage SOC have a greater impact on decision-making; while during peak carbon price periods, carbon price signals and photovoltaic power generation features have higher decision-making value), this invention adds an attention weighting layer after the input layers of the Actor policy network and the Critic value network. This layer dynamically generates attention weights for each feature dimension based on the current input, and its expression is as follows: (32) (33) (34) (35) In the formula, The feature vector is formed by concatenating the current environment state vector and the condition representation vector. Here, is the learnable weight matrix, and represents the bias term. For query vector, My key vector, This represents the Sigmoid activation function. Attention weights for each feature dimension; these attention weights reflect the importance of each original feature dimension in the current state.

[0042] After obtaining the attention weights, attention-weighted feature representations are generated by weighting the original features element by element. (36) Among them, the weighted feature vector It amplifies features that are crucial to the current decision and suppresses secondary features.

[0043] For Actor policy networks: (37) For Critic value networks: (38) (39) S6. Develop course learning strategies and train the agent.

[0044] To alleviate the problems of training instability and slow convergence caused by frequent violations of physical constraints (such as power imbalance and device over-limit) in the early stage of action space exploration, this invention adopts a course learning strategy to guide the agent's progressive learning. In the course learning stage, the experience replay buffer is filled with two alternating exploration schemes. One scheme is to use the traditional action selection mechanism that injects exploration noise, as shown in formula (40), and the other scheme is to use random action selection until convergence.

[0045] (40) in, The shearing function restricts the injection of noise to between its specified upper and lower bounds.

[0046] The steps for training an agent include: S61. Initialize the experience replay buffer, all network parameters, and training hyperparameters; S62. Enter the course learning phase. For each training round, reset the environment to its initial state. S63. Select actions based on strategies for the current course stage. Environmental execution actions , transition to a new state And calculate the reward , experience tuple Store in the experience replay buffer; S64. Sample small batches of data from the buffer, update the Actor policy network and Critic value network according to the DDPG algorithm update rules, combine the unique loss function in the conditional knowledge representation network, and softly update the target network parameters. S65: Repeat steps S62 to S64 until all course phases are completed, then proceed to the regular training phase until the policy model converges.

[0047] The training method described above can gradually cultivate the agent's ability to identify appropriate operation points, enabling it to minimize total cost while meeting basic constraints. This progressive training strategy allows the agent to first establish a feasible baseline scheduling scheme, and then gradually learn and achieve more complex optimization goals, thus achieving a natural progression from feasible to efficient.

[0048] S7. Deploy the trained optimal scheduling strategy model to the actual integrated energy system. At each scheduling moment, the real-time status of the system is collected and input into the optimal scheduling strategy model to obtain the optimal scheduling action instruction, which is then sent to each controllable device in the system for execution, thereby realizing the online real-time optimization scheduling of the system.

[0049] To comprehensively evaluate the effectiveness of the proposed method, a systematic comparative experiment was conducted with five mainstream deep reinforcement learning algorithms, including traditional deep deterministic policy gradient (DDPG), deep Q-network (DQN), soft Actor-Critic (SAC), dual-delay deep deterministic policy gradient (TD3), and proximal policy optimization (PPO). The comparison results of each algorithm in terms of economy and environmental friendliness are shown in Table 1.

[0050] Table 1 Comparison results of different methods method Average daily cost (yuan) Daily carbon emissions (kg) SAC 42897.75 55.71 TD3 43138.80 53.73 PPO 50867.62 46.08 DQN 46801.78 54.96 DDPG 40724.53 58.76 Ours 40389.83 55.12 From an economic perspective, this invention demonstrates a significant advantage in average daily operating costs. Specifically, it achieves the lowest average daily cost of 40,389.8 yuan, a reduction of 0.8% compared to the second-best performing traditional DDPG algorithm (40,724.5 yuan). Compared to other comparative algorithms, this invention reduces costs by 13.8%, 5.8%, 6.3%, and 20.6% compared to DQN (46,801.8 yuan), SAC (42,897.8 yuan), TD3 (43,138.8 yuan), and PPO (50,867.6 yuan), respectively. This result fully validates the effectiveness of the proposed improvements in reducing operating costs, particularly in enhancing the accuracy of continuous action space processing and long-term value estimation.

[0051] Regarding environmental performance indicators, the algorithms exhibit different optimization characteristics. While the PPO algorithm achieved the lowest daily carbon emissions (46.1 kg), its operating cost was the highest, indicating an inadequate trade-off between environmental friendliness and economic efficiency. The present invention achieves daily carbon emissions of 55.1 kg, achieving optimal economic efficiency while maintaining a carbon emission level comparable to SAC (55.7 kg) and DQN (55.0 kg), and significantly lower than the traditional DDPG algorithm (58.8 kg). Notably, the TD3 algorithm performs well in terms of carbon emissions (53.7 kg), but its cost is relatively high, and its training stability is inferior to that of the present invention.

[0052] Figure 3 The dynamic performance comparison of each algorithm during training is further demonstrated. From the training results, this invention exhibits the best stability and convergence: the reward curve rises smoothly and stabilizes at a high level after approximately 3000 rounds, while the cost curve and carbon emission curve are superior to other methods.

[0053] Figure 4 This paper presents the optimal scheduling results of a typical daily integrated energy system based on the improved deep reinforcement learning algorithm proposed in this invention. Regarding power scheduling (… Figure 4 a) The system exhibits excellent multi-energy complementary operation characteristics. Specifically, the BES (Battery Storage System) primarily operates in charging mode during off-peak electricity price periods (23:00-07:00), effectively utilizing low-priced electricity to store energy; during peak electricity price periods (12:00-19:00), it switches to discharging mode, actively participating in system peak regulation, demonstrating the effective guidance of price signals on energy storage operation strategies. The CHP (Cyclic Photovoltaic Power Generation) unit significantly increases power output during peak electricity price periods, reducing the need for high-priced external power purchases by increasing self-generated power, a strategy that effectively lowers system operating costs. Particularly noteworthy is that during daytime periods with sufficient photovoltaic output, the system prioritizes the consumption of renewable energy, significantly reducing purchased electricity and related carbon emissions.

[0054] In terms of thermal power dispatch ( Figure 4 (b) The system achieves efficient heat load balance through the coordinated operation of multiple heat sources. The EB unit undertakes the main heating task during periods of low electricity prices, fully utilizing low-priced electricity for heating, thus reducing heating costs and promoting grid load balance. The GB unit operates as a supplementary heat source during peak heat load periods or periods of normal electricity prices, ensuring the reliability and flexibility of the heating system. The thermoelectric coupling characteristics of the CHP unit are fully utilized, providing both electricity and heat output during peak electricity prices, achieving cascaded energy utilization and significantly improving the overall energy efficiency of the system. This dispatching scheme fully verifies the synergistic optimization potential of the integrated energy system. Through the coordinated control of the electro-thermal energy conversion and storage equipment, the system achieves the dual goals of economic operation and low-carbon environmental protection while ensuring the system's energy needs are met.

[0055] This invention comprehensively considers economic performance, environmental performance, and the cost of photovoltaic curtailment, achieving a holistic optimization of economic costs, environmental emissions, and photovoltaic curtailment penalties, and providing an effective solution for the safe, economical, and green operation of IES under a high proportion of renewable energy access.

[0056] Finally, it should be noted that any parts of this invention not described in detail are prior art. Those skilled in the art will understand that the above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. An IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning, characterized in that, Includes the following steps: S1. Establish a multi-objective optimization scheduling model for the integrated energy system to obtain the minimum total operating cost of the integrated energy system; S2. Establish a multi-objective optimization scheduling reinforcement learning model for the integrated energy system, transform the multi-objective optimization scheduling model constructed in step S1 into a Markov decision process, and define the state variables, action variables, and reward function in the reinforcement learning process. S3. Construct and initialize a deep deterministic policy gradient algorithm network, which includes an Actor policy network for generating continuous actions, a Critic value network for evaluating the value of state-action pairs, and their respective target networks. S4. Based on the Actor policy network and Critic value network constructed in step S3, a conditional knowledge representation network is integrated. The conditional knowledge representation network uses historical running data to pre-train an independent feature extraction network, enabling it to output an embedding vector representing the essential features of the current running scenario based on the system's input state data. S5. Embedded feature attention mechanism: After the input layers of the Actor policy network and the Critic value network, an attention weighting layer is added to each of them. This layer is used to dynamically generate attention weights for each feature dimension based on the current input, and then feed the weighted features into the subsequent layers of the network for calculation. S6. Formulate a course learning strategy and train the agent. In the integrated energy system simulation environment, according to the course learning strategy, use the experience playback mechanism and network parameter update mechanism to train the deep deterministic policy gradient algorithm that integrates the conditional knowledge representation network and feature attention mechanism until a convergent and stable optimal scheduling policy model is obtained. S7. Deploy the trained optimal scheduling strategy model to the actual integrated energy system.

2. The IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning according to claim 1, characterized in that, The objective function of the multi-objective optimization scheduling model in step S1 includes operating cost and carbon emission cost. The constraints include power balance constraints, power interaction constraints with the main grid, and equipment operation constraints. The operating cost includes energy procurement cost, depreciation cost of the battery storage system, and the cost of curtailing renewable energy. The carbon emission cost... Based on the carbon price mechanism and the total carbon emissions of the system, the calculation formula is as follows: In the formula, Indicates time carbon prices, This represents the total carbon emissions during that period. This indicates the carbon emissions generated from purchasing electricity from the main grid. and These represent the carbon emissions of CHP and GB, respectively, both of which originate from natural gas consumption.

3. The IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning according to claim 1, characterized in that, The conditional knowledge representation network in step S4 includes an encoder network and a projection network. The encoder network uses a long short-term memory network, and its expression is: In the formula, and These represent the hidden state and cell state of the LSTM, respectively. Indicates from time arrive The state sequence, where The length of the historical state sequence; The projection network employs a multilayer perceptron structure, mapping the extracted features to a low-dimensional embedding space, and undergoes self-supervised pre-training using a contrastive loss based on InfoNCE. Its expression is as follows: In the formula, Let the conditional representation vector be... This is the weight matrix. and is the bias term, and ReLU is the activation function.

4. The IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning according to claim 1, characterized in that, The attention weighting layer receives an input feature vector composed of the current environment state vector and the conditional representation vector; and calculates the attention weight of each input feature using a weight matrix and a sigmoid activation function. The input feature vector is element-wise multiplied with the attention weight vector to obtain a refined feature representation, which is then input into the Actor policy network and the Critic value network. The attention weights are calculated as follows: In the formula, The feature vector is formed by concatenating the current environment state vector and the condition representation vector. The weight matrix is ​​a learnable matrix. For bias terms, For query vector, My key vector, This represents the Sigmoid activation function. Attention weights for each feature dimension.

5. The IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning according to claim 1, characterized in that, The learning process in step S6 includes two alternating exploration schemes to fill the experience replay buffer: one scheme uses a traditional action selection mechanism that injects exploration noise, and the other scheme uses random action selection until convergence.

6. The IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning according to claim 5, characterized in that, The expression for the traditional action selection mechanism is: in, The shearing function restricts the injection of noise to between its specified upper and lower bounds.

7. The IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning according to claim 1, characterized in that, The training process of the intelligent agent specifically includes the following steps: S61. Initialize the experience replay buffer, all network parameters, and training hyperparameters; S62. Enter the course learning phase. For each training round, reset the environment to its initial state. S63. Select actions based on strategies for the current course stage. Environmental execution actions transition to a new state And calculate the reward , experience tuple Store in the experience replay buffer; S64. Sample small batches of data from the buffer, update the Actor policy network and Critic value network according to the DDPG algorithm update rules, combine the unique loss function in the conditional knowledge representation network, and softly update the target network parameters. S65: Repeat steps S62 to S64 until all course phases are completed, then proceed to the regular training phase until the policy model converges.

8. The IES multi-objective dynamic optimization scheduling method based on improved deep reinforcement learning according to claim 1, characterized in that, At each scheduling moment in step S7, the real-time status of the system is collected, input into the optimal scheduling strategy model to obtain the optimal scheduling action instruction, and then sent to each controllable device in the system for execution.

Images