A deep reinforcement learning method for optimal scheduling of integrated energy systems based on graph neural networks

Abstract

The application provides a deep reinforcement learning comprehensive energy system optimization scheduling method based on a graph neural network, which mathematically describes the optimization scheduling problem of the comprehensive energy system with the overall operation economy of the system as the target, expresses the optimization scheduling problem as a Markov decision process, first designs a reasonable state space, an action space and a reward function, then introduces a graph neural network in the spatial dimension to improve the deep reinforcement learning algorithm, and explicitly models the electric and heat network topology to optimize the energy flow distribution, and then uses the improved model for training until a stable optimization scheduling strategy model is obtained. After the training is completed, the model can make scheduling decisions according to the state of the system and output scheduling instructions of each device. The method provided by the application can optimally schedule according to the electric and heat load demand and the renewable energy output of the system, so that the operation economy of the entire comprehensive energy system is realized.

Description

Technical Field

[0001] This invention belongs to the field of optimization scheduling technology for integrated energy systems, specifically relating to a deep reinforcement learning-based optimization scheduling method for integrated energy systems based on graph neural networks. Background Technology

[0002] With the rapid development of wind power and photovoltaic technologies, the penetration rate of renewable energy in the power system is constantly increasing, providing important support for promoting energy structure transformation. Integrated energy systems have proven to be a highly promising solution, effectively integrating and utilizing various energy devices to improve energy efficiency and the absorption capacity of renewable energy. Currently, existing research on optimization scheduling methods for integrated energy systems mainly focuses on traditional modeling approaches such as mathematical programming and mixed-integer optimization. However, with the increasing complexity and scale of integrated energy systems, these methods rely on precise linearization and convex approximation of the model for solution accuracy. This necessitates equivalent approximation linearization of the nonlinear problems in the integrated energy system model, leading to insufficient accuracy in the scheduling model. Furthermore, in terms of computational efficiency, with the integration of various energy units, the system optimization problem exhibits high-dimensional and strongly nonlinear characteristics, causing the solution time of the mixed-integer programming problem to explode exponentially. The computational time of the above optimization algorithms is insufficient to meet the system's timeliness requirements.

[0003] With the development of artificial intelligence, deep reinforcement learning has provided a solution to the bottlenecks of traditional models in high-dimensional, nonlinear, and highly uncertain problems. It trains agents to make optimal decisions through continuous interaction with the environment, maximizing long-term cumulative rewards and effectively addressing uncertainties such as renewable energy fluctuations and load changes. Some literature has used various deep reinforcement learning algorithms to study the optimal scheduling of integrated energy systems. By modeling the optimal scheduling problem of integrated energy systems as a Markov decision process and selecting appropriate algorithms based on the characteristics of the action space, the potential of deep reinforcement learning algorithms in solving optimal scheduling problems has been verified. However, existing deep reinforcement learning algorithms typically use the node states of the integrated energy system as flattened vector inputs, completely ignoring the topological structure and physical transmission characteristics between electrical and thermal energy flows. This prevents the agent from learning the optimal transmission path and dynamic balance strategy of electrical-thermal energy flows in the spatial dimension, limiting the economy and security of the scheduling scheme.

[0004] Therefore, there is an urgent need for a deep reinforcement learning-based method for optimizing and scheduling integrated energy systems based on graph neural networks, in order to address the problem that traditional deep reinforcement learning methods are insufficient in extracting spatial topological features of multi-energy networks, thereby improving the algorithm's scheduling decision-making ability for the system. Summary of the Invention

[0005] To address the problem of optimal scheduling of integrated energy systems, this invention provides a deep reinforcement learning-based method for optimal scheduling of integrated energy systems using graph neural networks.

[0006] The technical solution of this invention is as follows:

[0007] A deep reinforcement learning-based method for optimizing the scheduling of integrated energy systems using graph neural networks includes:

[0008] S1. First, a comprehensive energy system incorporating both electrical and thermal energy flows is constructed, including wind turbines, photovoltaic generators, combined heat and power (CHP) units, gas-fired boilers, electric boilers, and battery storage units. Based on the mathematical models of each unit, the optimal scheduling problem of the comprehensive energy system is mathematically described with the goal of improving system operating economy.

[0009] S2. Next, the optimization scheduling problem of the integrated energy system is expressed as a Markov decision process, and the state space, action space and reward function of the system are designed in detail.

[0010] S3. Then, the SAC algorithm is improved by combining graph neural networks to explicitly model the topology of electric and thermal networks to optimize energy flow distribution.

[0011] S4. Finally, the improved deep reinforcement learning algorithm is used to make optimal scheduling decisions for the integrated energy system. After training, a stable optimal scheduling strategy model is obtained.

[0012] Furthermore, the optimization problem of the integrated energy system is a sequential decision problem, where the optimal scheduling aims to minimize the system's operating cost by determining the output power of each unit at each time step. The interaction between the agent and the environment in the proposed improved deep reinforcement learning algorithm is a Markov decision process. The system environment includes electrical and thermal energy flows. The agent makes scheduling decisions by observing the real-time state of the environment and generates corresponding actions for each unit to adjust its power output. The environment then transitions to the next state, and the agent uses this new state as the new state to update the actions of each unit.

[0013] Furthermore, the aforementioned integrated energy system is a combined electricity and heat energy system, primarily comprising electricity and heat, and meeting load demands through various energy conversion devices such as combined heat and power (CHP) units, gas-fired boilers, and electric boilers. Energy supply includes energy from the upstream power grid, wind turbines, and photovoltaic power generation. CHP units, electric boilers, and gas-fired boilers are used for coordinated energy conversion. Batteries serve as energy storage devices, storing energy when supply exceeds load demand and releasing it when supply falls short. Energy demand includes both electrical and thermal loads on the user side. This system achieves supply-demand balance and economical operation through multi-energy flow coupling and coordinated operation among various devices.

[0014] Furthermore, the mathematical modeling of each energy device is as follows:

[0015] (1) Modeling of cogeneration units:

[0016] A combined heat and power (CHP) unit generates electricity and heat by consuming natural gas. Its physical model is as follows:

[0017] ;

[0018] ;

[0019] In the formula, P CHP,t H CHP,t and G CHP,t Represent the electrical power, thermal power, and input power of the combined heat and power unit at time t, respectively; η CHP_e and η CHP_h These represent power generation efficiency and heating efficiency, respectively.

[0020] (2) Modeling of gas-fired boilers:

[0021] A gas-fired boiler is a device that generates heat power by burning natural gas. Its physical model is as follows:

[0022] ;

[0023] In the formula, η GB G represents conversion efficiency; GB,t H represents the input power of the gas-fired boiler. GB,t This represents the thermal power generated by the gas-fired boiler.

[0024] (3) Modeling of electric boiler:

[0025] Electric boilers convert electricity generated from renewable energy sources such as wind and solar power into heat energy without the need for fuel combustion, reducing carbon emissions while also improving the utilization of renewable energy. Their physical model is as follows:

[0026] ;

[0027] In the formula, η EB P represents the electro-thermal efficiency of an electric boiler. EB,t H represents the electricity consumption of the electric boiler. EB,t This represents the heating capacity of the electric boiler.

[0028] (4) Battery modeling;

[0029] As an electrical energy storage device, the battery meets the supply and demand balance by storing and releasing electrical energy. The State of Charge (SOC) at each time t can be expressed as:

[0030] ;

[0031] In the formula, SOC BESS,t Represents the SOC state of the battery; P cha,t , P dis,t , η cha_e,t and η dis_e,t Represents the charging / discharging power and efficiency of the battery; E BESS This represents the rated capacity of the battery.

[0032] Furthermore, the constraints of the constructed integrated energy system include:

[0033] (1) Power balance constraint;

[0034] The power balance equations for the constructed integrated energy system are as follows:

[0035] ;

[0036] In the formula, P Load,t P represents the power required by the system's electrical load. grid,t It is the electrical power purchased by the integrated energy system from the grid; P PV,t and P WT,t P represents the electrical power generated by photovoltaic and wind power generation. CHP,t P represents the electrical power output of a combined heat and power (CHP) unit. EB,t P represents the electrical power required by the electric boiler. dis,t P represents the electrical power released by the battery. cha,t This represents the electrical power stored in the battery.

[0037] (2) Thermal power balance constraint;

[0038] The heat power balance equation of the constructed integrated energy system is as follows:

[0039] ;

[0040] In the formula, H Load,t H represents the power required for the system's heat load. CHP,t H represents the thermal power output of a combined heat and power unit. GB,t H represents the thermal power output of a gas-fired boiler. EB,t This represents the thermal power output of the electric boiler.

[0041] (3) Constraints;

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] In the formula, P grid,max G GB,max G CHP,max P EB,max P grid,min G GB,min G CHP,min and P EB,min These represent the upper and lower limits of the power purchased from the grid and the power of each generating unit, respectively; SOC BESS,max and SOC BESS,min The upper and lower limits of the SOC of the battery are respectively.

[0048] (4) Climbing constraint;

[0049] ;

[0050] ;

[0051] ;

[0052] In the formula, ΔG GB,up ΔG CHP,up ΔP EB,up ΔG GB,down ΔG CHP,down and ΔP EB,down These represent the upper and lower limits of the ramp power for each unit's equipment, respectively.

[0053] Furthermore, the goal of optimal operation of the integrated energy system is to reduce costs while ensuring the safe and stable operation of each system device within a specified capacity range. Therefore, the core objective of the integrated energy system optimization scheduling problem in this invention is to minimize the total operating cost of the system while maintaining normal system operation, where the objective function is expressed as follows:

[0054] ;

[0055] In the formula, F is the total operating cost of the system, T is the scheduling period, and C is the total operating cost of the system. om,t For the operation and maintenance costs of the equipment, C grid,t The cost of purchasing electricity from the grid.

[0056] Furthermore, a deep reinforcement learning model for optimizing the operation of integrated energy systems is constructed as follows:

[0057] This invention optimizes integrated energy scheduling using a deep reinforcement learning method based on the SAC algorithm framework. The algorithm comprises one Actor network, two Critic networks, and two corresponding target Critic networks. The Actor network samples actions based on the current state and interacts with the environment to obtain immediate rewards and the next-time state. The two Critic networks are responsible for evaluating the value of state-action pairs. The two target Critic networks generate stable value estimates through soft updates, specifically including:

[0058] (1) The optimal policy function of the SAC algorithm can be expressed as:

[0059] ;

[0060] In the formula, π* represents the strategy function that maximizes the expected cumulative return; s t It is the environmental state at time t; a t It represents the action at time t; α is the temperature coefficient, used to balance the trade-off between expected reward and entropy; It is the policy entropy term; ρ π The state-action trajectory distribution formed by strategy π; r(s) t ,a t ) represents the instantaneous reward value at time step t;

[0061] The Q-value function and state-value function of the SAC algorithm are defined as follows:

[0062] ;

[0063] ;

[0064] In the formula, p represents the state transition probability; γ represents the discount factor, which takes a value in the range [0,1], indicating the importance of future rewards;

[0065] In the process of agent policy improvement, the difference between the two distributions is minimized by minimizing the KL divergence, which is expressed as:

[0066] ;

[0067] In the formula, D KL () represents the KL divergence; Π represents the policy set; This is the allocation function under the old strategy, which is used to normalize the distribution of Q-values; Represented as the old strategy π old Value functions;

[0068] Adaptive parameter updates are achieved by minimizing the residuals, as described below:

[0069] ;

[0070] ;

[0071] ;

[0072] In the formula, J Q (θ) represents the loss function of the Q-network, and θ and Let represent the parameters of the value network and its corresponding target network, respectively. During training, these parameters are updated as follows:

[0073] ;

[0074] In the formula, k represents the update parameter.

[0075] The policy network is updated as follows:

[0076] ;

[0077] In the formula, ϕ is the parameter of the policy network.

[0078] (2) State-space design;

[0079] By observing the system's state, the agent determines the optimal output for each device in the system. The states of photovoltaic power, wind power, electricity price, electrical load, heat load, and battery are all contained in the state space, which is defined at time t as follows:

[0080] ;

[0081] In the formula, P PV,t and P WT,t Represents the electrical power generated by photovoltaic and wind power; C grid,t It is time-of-use electricity pricing; H Load,t P represents the system's heat load demand; Load,t Represents the electrical load demand of the system; SOC BESS,t This represents the battery's SOC (State of Charge) status.

[0082] (3) Motion space design;

[0083] The motion space design of this invention is as follows:

[0084] ;

[0085] In the formula, P EB,t P represents the electrical power consumed by the electric boiler. grid,t It is the electrical power that the system purchases from the upstream power grid; H CHP,t H represents the thermal power output of a combined heat and power unit. GB,t P represents the thermal power output of the gas-fired boiler.BESS,t This represents the charge / discharge power of the BESS at time t.

[0086] (4) Reward function design;

[0087] The goal of the integrated energy system constructed in this invention is to minimize the operating costs and electricity purchase costs of each unit while meeting electricity and heat load demands. The reward function of the intelligent agent comprises three parts: operating cost, electricity purchase cost, and constraint penalty, and its formula is as follows:

[0088] ;

[0089] Among them, C om,t F1 is the operating cost of each unit in the system; β1 is the penalty cost for thermal imbalance; F2 is the penalty cost for electrical imbalance; β2 is the penalty coefficient for electrical imbalance.

[0090] Furthermore, this invention employs graph neural networks to further improve the SAC algorithm:

[0091] This invention considers the heterogeneity and topological complexity of devices in an integrated energy system, thus incorporating graph neural networks into the SAC algorithm and making targeted improvements. First, a learnable 3D node type embedding vector is introduced to enhance the representation of device functional heterogeneity; second, deep information aggregation with structure awareness is achieved through stacked GCNs and residual connections; finally, a flattening strategy that preserves node order is adopted to adapt to the output requirements of node-level scheduling tasks. The improved architecture can provide accurate spatial topology and state fusion features, specifically including:

[0092] The improved algorithm of this invention uses graph features extracted by a graph neural network as state input. The improved dual-Critic network achieves adaptive parameter update by minimizing the residual, as shown in the formula: ;

[0093] In the formula, G t Indicates the state of the input graph;

[0094] The policy network update of the improved algorithm is shown below:

[0095] ;

[0096] In the formula, ϕ represents the parameters of the policy network.

[0097] The technical effects and advantages of this invention are as follows:

[0098] 1. This invention considers the complex coupling characteristics of heterogeneous energy networks in a comprehensive energy system on a spatial scale, adopts the SAC algorithm, and introduces a graph neural network in the spatial dimension and makes targeted improvements to explicitly model the topology of the electric and thermal networks to optimize energy flow distribution.

[0099] 2. This invention aims to reduce the overall operating cost of the system by constructing the action space, state space, and reward function in detail. Simulation results demonstrate that the proposed method has good scheduling capabilities and can further improve the system's economic efficiency.

[0100] 3. The proposed method for optimizing the scheduling of integrated energy systems based on graph neural networks improves the Actor and Critic networks of SAC by introducing graph neural networks. It transforms the flattened node state vectors of traditional deep reinforcement learning into a graph structure containing node attributes and edge associations, explicitly models the spatial distribution and supply-demand coupling relationship between units, and further improves the optimization of energy flow transmission and distribution of electrical and thermal energy by the agent, thereby improving the overall economy of the system. Attached Figure Description

[0101] Figure 1 This is a diagram of the integrated energy system of the present invention;

[0102] Figure 2 This is a graph showing the results of power dispatching;

[0103] Figure 3 This is a graph showing the results of thermal power scheduling;

[0104] Figure 4 This is a flowchart of the method of the present invention. Detailed Implementation

[0105] The following will combine Figures 1-4 The technical solution of the present invention will be further illustrated through specific embodiments.

[0106] Example 1:

[0107] Description of the test scenario;

[0108] To verify, as Figure 4 The effectiveness of the deep reinforcement learning-based integrated energy system optimization scheduling method based on graph neural networks is demonstrated. Figure 1The simulation analysis is performed using the integrated energy system operation scenario shown. This invention constructs a one-month dataset of electricity load, heat load, photovoltaic power, and wind power to train the algorithm. The system's equipment operating parameters are shown in Table 1, and the time-of-use electricity price is shown in Table 2. In this invention's improved algorithm, both the Actor and Critic networks use the Adam optimizer to update network weights. The training iterations are set to 2000 epochs, with the Actor learning rate set to 2e-4 and the Critic learning rate set to 4e-4 for both layers. Furthermore, to account for future rewards, the discount factor is set to 0.95.

[0109] The system's equipment operating parameters are shown in Table 1.

[0110] Table 1

[0111]

[0112] The time-of-use electricity prices are shown in Table 2.

[0113] Table 2

[0114]

[0115] Results analysis:

[0116] To verify the effectiveness of the proposed integrated energy system optimization scheduling method based on improved deep reinforcement learning, historical datasets were used for training and testing to analyze the overall operating cost of the system.

[0117] Scheduling result analysis:

[0118] After training the algorithm network using the constructed historical dataset, the trained network is used for the optimized scheduling of the constructed integrated energy system. To verify the effectiveness of the proposed method, this invention randomly selects a day from the test set as validation data.

[0119] Figure 2The figure illustrates the hourly power balance results over 24 hours using the method proposed in this invention. As shown, the overall power load is stably supplied by renewable energy units such as wind power and photovoltaics, as well as combined heat and power (CHP) units. Batteries serve as energy storage devices providing backup support, while electric boilers convert excess electricity into heat, thus absorbing surplus renewable energy while maintaining power balance. During the night, the overall power load is low, with no photovoltaic output and sufficient wind power output, and electricity prices are low. At this time, the system purchases electricity from the grid and stores excess energy in batteries. When the batteries cannot fully absorb the excess energy, it is fed into the electric boilers and converted into heat. During the daytime period from 8:00 AM to 3:00 PM, user power load rapidly increases, and the batteries switch to discharge mode, releasing the energy stored overnight to work with the CHP units to supplement the power supply gap. Furthermore, the combined wind and solar power supply capacity is significantly enhanced during this period, and excess energy is also converted into heat by the electric boilers, further improving the system's operational economy.

[0120] Figure 3 The figure illustrates the hourly heat power balance results over 24 hours using the method proposed in this invention. As can be seen from the figure, the heat load is mainly supplied by the combined heat and power (CHP) unit and the gas-fired boiler, with the electric boiler serving as an auxiliary heating method, converting some of the renewable energy into heat energy to supplement the heat load demand. During the nighttime hours, the overall heat load remains relatively high, with the CHP unit and gas-fired boiler acting as the core heating providers, dominating the output and sharing the basic heat load, while the electric boiler provides only a small amount of supplementary heat energy during these periods. From 9:00 AM to 3:00 PM during the day, the heat load drops significantly compared to the nighttime. To adapt to the decrease in load, the gas-fired boiler significantly reduces its heating output, and the CHP unit also reduces its output from 9:00 AM to 12:00 PM. Although the output of the electric boiler accounts for a very small proportion during this period, the entire system, through the coordinated scheduling of the CHP unit and the gas-fired boiler, can still efficiently ensure the real-time heat supply and demand balance throughout the day, effectively improving the system's operational economy while meeting heat demand.

[0121] The above descriptions are merely specific embodiments of the present invention, and are intended only to explain the principles of the invention, and should not be construed as limiting the scope of protection of the invention in any way. Based on this explanation, those skilled in the art can readily conceive of other specific embodiments of the invention without any creative effort, and these embodiments will all fall within the scope of protection of the present invention.

Claims

1. A deep reinforcement learning-based integrated energy system optimization scheduling method based on graph neural networks, characterized in that, include: S1. First, construct an integrated energy system that includes two energy flows: electricity and heat, including wind turbines, photovoltaic generators, combined heat and power units, gas boilers, electric boilers, and battery units. Based on the mathematical models of each unit, mathematically describe the optimization scheduling problem of the integrated energy system with the goal of system operation economy. S2. Next, the optimization scheduling problem is formulated as a Markov decision process, and the state space, action space and reward function of the system are constructed. S3. Then, the SAC algorithm is improved by combining graph neural networks to explicitly model the topology of electric and thermal networks to optimize energy flow distribution. S4. Finally, the improved deep reinforcement learning algorithm is used to make optimal scheduling decisions for the integrated energy system. After training, a stable optimal scheduling strategy model is obtained.

2. The method for optimal scheduling of a comprehensive energy system based on graph neural networks and deep reinforcement learning, as described in claim 1, is characterized in that: The optimization scheduling problem of the integrated energy system is a sequential decision problem. Its optimization scheduling aims to minimize the operating cost of the system and determine the output power of each unit at each time step.

3. The method for optimal scheduling of a comprehensive energy system based on graph neural networks and deep reinforcement learning, as described in claim 1, is characterized in that: The integrated energy system described is a combined energy system that integrates electricity and heat, comprising: The energy supply layer integrates the upstream power grid, wind turbines, and photovoltaic power generation. The energy conversion layer integrates combined heat and power units, electric boilers, and gas-fired boilers to coordinate energy conversion between them. The energy storage layer consists of battery units, which use batteries as energy storage devices. When the energy supply exceeds the load demand, the energy is stored, and when the energy supply is less than the load demand, the energy is released. The energy demand layer includes the electrical load and heat load on the user side.

4. The method for optimal scheduling of a comprehensive energy system based on graph neural networks and deep reinforcement learning, as described in claim 3, is characterized in that: The mathematical model for each unit is as follows: A combined heat and power (CHP) unit generates electricity and heat by consuming natural gas. Its physical model is as follows: ； ； In the formula, P CHP,t H CHP,t and G CHP,t Represent the electrical power, thermal power, and input power of the combined heat and power unit at time t, respectively; η CHP_e and η CHP_h These represent power generation efficiency and heating efficiency, respectively. The physical model of a gas-fired boiler is as follows: ； In the formula, η GB G represents conversion efficiency; GB,t H represents the input power of the gas-fired boiler. GB,t This represents the thermal power generated by the gas-fired boiler. The physical model of the electric boiler is as follows: ； In the formula, η EB P represents the electro-thermal efficiency of an electric boiler. EB,t H represents the electricity consumption of the electric boiler. EB,t This represents the heating capacity of the electric boiler. Battery storage units achieve supply and demand balance by storing and releasing electrical energy. The State of Charge (SOC) at each time t is represented as follows: ； In the formula, SOC BESS,t Represents the SOC state of the battery; P cha,t , P dis,t , η cha_e,t and η dis_e,t Represents the charging / discharging power and efficiency of the battery; E BESS This represents the rated capacity of the battery.

5. The method for optimal scheduling of a comprehensive energy system based on graph neural networks and deep reinforcement learning, as described in claim 3, is characterized in that: The integrated energy system described above is also subject to the following system constraints: The power balance equations for the constructed integrated energy system are as follows: ； In the formula, P Load,t P represents the power required by the system's electrical load. grid,t It is the electrical power purchased by the integrated energy system from the grid; P PV,t and P WT,t P represents the electrical power generated by photovoltaic power generation and wind power generation, respectively; CHP,t P represents the electrical power output of a combined heat and power (CHP) unit. EB,t P represents the electrical power required by the electric boiler. dis,t P represents the electrical power released by the battery. cha,t Represents the electrical power stored in the battery; The heat power balance equation of the constructed integrated energy system is as follows: ； In the formula, H Load,t H represents the power required for the system's heat load. CHP,t H represents the thermal power output of a combined heat and power unit. GB,t H represents the thermal power output of a gas-fired boiler. EB,t This represents the thermal power output of the electric boiler; The constraints include: ； ； ； ； ； In the formula, P grid,max G GB,max G CHP,max P EB,max P represents the upper limits of the power purchased from the grid, the power of the gas-fired boiler, the power of the combined heat and power unit, and the power of the electric boiler, respectively. grid,min G GB,min G CHP,min and P EB,min These represent the lower limits of the power purchased from the grid, the power of the gas-fired boiler, the power of the combined heat and power unit, and the power of the electric boiler, respectively; SOC BESS,max and SOC BESS,min These represent the upper and lower limits of the battery's State of Charge (SOC), respectively. Climbing constraints include: ； ； ； In the formula, ΔG GB,up ΔG CHP,up ΔP EB,up ΔG represents the maximum ramp-up power of gas-fired boilers, combined heat and power units, and electric boilers, respectively. GB,down ΔG CHP,down and ΔP EB,down These represent the lower limits of ramp power for gas-fired boilers, combined heat and power units, and electric boilers, respectively.

6. The method for optimal scheduling of a comprehensive energy system based on graph neural networks and deep reinforcement learning, as described in claim 3, is characterized in that: The objective function for the optimal scheduling problem of the integrated energy system is as follows: ； In the formula, F is the total operating cost of the system, T is the scheduling period, and C is the total operating cost of the system. om,t For the operation and maintenance costs of the equipment, C grid,t The cost of purchasing electricity from the grid.

7. The method for optimal scheduling of a comprehensive energy system based on graph neural networks according to claim 1, characterized in that: The SAC algorithm in step S3 includes an Actor network, two Critic networks, and two corresponding target Critic networks. The Actor network samples actions based on the current state and interacts with the environment to obtain immediate rewards and the state at the next moment. The two Critic networks are responsible for evaluating the value of state-action pairs. The two target Critic networks generate stable value estimates through soft updates.

8. The method for optimal scheduling of a comprehensive energy system based on graph neural networks and deep reinforcement learning, as described in claim 7, is characterized in that: The SAC algorithm is updated and trained in the following manner: The optimal policy function of the SAC algorithm can be expressed as: ； In the formula, π* represents the strategy function that maximizes the expected cumulative return; s t It is the environmental state at time t; a t It represents the action at time t; α is the temperature coefficient, used to balance the trade-off between expected reward and entropy; It is the policy entropy term; ρ π The state-action trajectory distribution formed by strategy π; r(s) t ,a t ) represents the instantaneous reward value at time step t; The Q-value function and state-value function of the SAC algorithm are defined as follows: ； ； In the formula, p represents the state transition probability; γ represents the discount factor, which takes a value in the range [0,1], indicating the importance of future rewards; In the process of agent policy improvement, the difference between the two distributions is minimized by minimizing the KL divergence, which is expressed as: ； In the formula, D KL () represents the KL divergence; Π represents the policy set; This is the allocation function under the old strategy, used to normalize the distribution of Q-values; Represented as the old strategy π old Value functions; Adaptive parameter updates are achieved by minimizing the residuals, as described below: ； ； ； In the formula, J Q (θ) represents the loss function of the Q-network, and θ and Let represent the parameters of the value network and its corresponding target network, respectively. During training, these parameters are updated as follows: ； In the formula, k represents the update parameter; The policy network is updated as follows: ； In the formula, ϕ is the parameter of the policy network.

9. The method for optimal scheduling of a comprehensive energy system based on graph neural networks and deep reinforcement learning according to claim 1, characterized in that: In step S2 The state space is defined as: ； In the formula, P PV,t and P WT,t These represent the electrical power generated by photovoltaic and wind power, respectively; C grid,t It is time-of-use electricity pricing; H Load,t P represents the system's heat load demand; Load,t Represents the electrical load demand of the system; SOC BESS,t This represents the battery's SOC (State of Charge) status. The motion space design is as follows: ； In the formula, P EB,t P represents the electrical power consumed by the electric boiler. grid,t It is the electrical power that the system purchases from the upstream power grid; H CHP,t H represents the thermal power output of a combined heat and power unit. GB,t P represents the thermal power output of the gas-fired boiler. BESS,t This represents the charge / discharge power of the BESS at time t; The reward function is: ； In the formula, C om,t F1 is the operating cost of each unit in the system; β1 is the penalty cost for thermal imbalance; F2 is the penalty cost for electrical imbalance; β2 is the penalty coefficient for electrical imbalance.

10. The method for optimal scheduling of a comprehensive energy system based on graph neural networks using deep reinforcement learning, as described in claim 8, is characterized in that: This also includes further improvements to the SAC algorithm using graph neural networks: First, learnable 3D node type embedding vectors are introduced to enhance the representation of device functional heterogeneity; second, structure-aware deep information aggregation is achieved by stacking GCNs and residual connections. Finally, a flattening strategy that preserves node order is adopted to adapt to the output requirements of node-level scheduling tasks. The improved architecture provides accurate spatial topology and state fusion features, specifically including: Using graph features extracted by a graph neural network as state input, the improved dual-Critic network achieves adaptive parameter update by minimizing the residuals, as shown in the formula: ； In the formula, G t Indicates the state of the input graph; The policy network update of the improved algorithm is shown below: ； In the formula, ϕ represents the parameters of the policy network.

Images