Electric heating comprehensive energy system energy management method, system, equipment and medium

Abstract

The invention relates to the technical field of electric heating integrated energy system energy management, and discloses an electric heating integrated energy system energy management method, system, device and medium, and the method comprises the steps: obtaining first operation data of a target electric heating integrated energy system, and building a first energy management research model based on the data; establishing a Markov decision process framework for the model; constructing a first improved algorithm to optimize the framework to obtain an optimal strategy framework model; and solving the model to obtain an energy management optimization strategy of the electric heating integrated energy system. According to the method, a mathematical model is established and expressed as a Markov decision process, so that display modeling of random variables is avoided, and optimization solution complexity is reduced; by designing a return mechanism and improving a PPO algorithm, the search efficiency and the solving precision are improved, effective decision can be made according to real-time observation data, the rapidity and stability of intelligent agent learning are improved, calculation is reduced, and the operation cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of energy management technology for integrated electrothermal energy systems, and in particular to an energy management method, system, device, and medium for integrated electrothermal energy systems. Background Technology

[0002] Integrated electrothermal energy systems, as a cutting-edge development in the energy sector, play a crucial role in promoting the global low-carbon transition. Their core lies in breaking down the barriers of traditional energy systems that rely on a single energy source operating independently. Through multi-energy synergy and intelligent regulation, they achieve the dual goals of improving energy efficiency and ensuring economic operation, becoming a key technological path for building a new energy system.

[0003] However, the energy management problem of this system faces multiple complex challenges: From a technical perspective, the integrated electric and thermal energy system model has high-dimensionality and non-convexity characteristics, and there are strong nonlinear coupling relationships between different energy networks. This complex mathematical characteristic makes traditional optimization algorithms face the risk of combinatorial explosion. From the perspective of information management architecture, the power network and the heating network are usually operated by different entities, resulting in information silos among multiple stakeholders. Data privacy barriers severely restrict the construction and solution of the global optimization model.

[0004] Current research suffers from two main limitations: Firstly, in terms of model building, the uncertainty, volatility, and large-scale integration of renewable energy sources increase the complexity of energy management models for integrated electro-thermal energy systems, leading to unsatisfactory scheduling performance and low search efficiency for model-based optimization methods such as metaheuristic algorithms. Secondly, regarding reinforcement learning-based methods, existing research often suffers from low search efficiency, poor applicability, and susceptibility to local optima. Furthermore, these algorithms are sensitive to hyperparameter selection, requiring extensive parameter tuning. Therefore, it is urgent to further consider the complexity of mathematical modeling for integrated electro-thermal energy systems and, considering the uncertainty of renewable energy output and load consumption, propose an energy management method for integrated electro-thermal energy systems that avoids the need for explicit modeling of random variables in the system and achieves high solution accuracy. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides an energy management method, system, device and medium for an integrated electric and thermal energy system, which can solve the problems of complex models, low search efficiency and easy getting trapped in local optima in the energy management of integrated electric and thermal energy systems.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides an energy management method for an integrated electrothermal energy system, comprising:

[0009] Acquire the first operating data of the target integrated electrothermal energy system and establish a first energy management research model based on the first operating data;

[0010] The first energy management research model includes an objective function and several constraints;

[0011] Establish a Markov decision process framework for the first energy management research model;

[0012] A first improved algorithm is constructed to optimize the Markov decision process framework and obtain the optimal policy framework model.

[0013] The first improved algorithm includes a dynamic normalization strategy and deterministic training;

[0014] The optimal strategy framework model is solved to obtain the energy management optimization strategy for the integrated electric and thermal energy system.

[0015] As a preferred embodiment of the energy management method for the integrated electrothermal energy system described in this invention, the establishment of a Markov decision process framework for the first energy management research model includes:

[0016] The first energy management research model is treated as an intelligent agent;

[0017] Establish a Markov decision process framework based on the aforementioned agent;

[0018] In the Markov decision process framework, the reward function is an arbitrary function that characterizes the agent's search for the optimal strategy within the feasible domain of the target electrothermal integrated energy system.

[0019] This preferred approach simplifies the problem-solving process by abstracting the complex energy management problem into an interaction between an agent and its environment. The flexible design of the reward function enables the agent to selectively explore better energy management strategies, improving search efficiency and the ability to obtain the global optimum. Furthermore, this framework lays a solid foundation for subsequent policy optimization using advanced reinforcement learning methods.

[0020] As a preferred embodiment of the energy management method for the integrated electrothermal energy system described in this invention, the objective function includes:

[0021] Establish an arbitrary function to characterize the total cost of the target integrated electrothermal energy system that minimizes the total cost;

[0022] The total cost includes the individual operating costs of several devices that make up the target integrated electric and thermal energy system, as well as the electricity purchase and sales costs of the target integrated electric and thermal energy system.

[0023] This preferred solution, by explicitly setting the objective of minimizing total cost, allows energy management strategies to focus more on economic efficiency. The total cost composition considers both the individual operating costs of equipment and the costs of purchasing and selling electricity; this comprehensive cost consideration helps achieve economical and efficient energy use. The benefits of this preferred solution are that it not only improves the overall operating efficiency of the integrated electric and thermal energy system but also promotes the sustainable development of the energy system through refined cost management, providing energy managers with a clearer and more specific direction for optimization.

[0024] As a preferred embodiment of the energy management method for the integrated electrothermal energy system described in this invention, the establishment of the Markov decision process framework based on the intelligent agent includes:

[0025] The output and load status of several devices in the target electrothermal integrated energy system are used as states in the Markov decision process framework.

[0026] The output increment of several devices in the target integrated electrothermal energy system is used as a decision variable;

[0027] The decision variables are treated as actions within the Markov decision process framework.

[0028] As a preferred embodiment of the energy management method for the integrated electrothermal energy system described in this invention, the step of solving the optimal strategy framework model includes:

[0029] Acquire renewable energy output data and load consumption data from the target integrated electric and thermal energy system;

[0030] Based on the renewable energy output data and load consumption data, the optimal strategy framework model is solved using a solver.

[0031] The solution results will be used as an energy management optimization strategy for the integrated electrothermal energy system.

[0032] As a preferred embodiment of the energy management method for the integrated electric and thermal energy system described in this invention, the plurality of constraints include power balance constraints, electric boiler operation constraints, combined heat and power unit operation constraints, electrochemical energy storage system operation constraints, and electricity market exchange constraints.

[0033] As a preferred embodiment of the energy management method for the integrated electric and thermal energy system described in this invention, the reward function includes a penalty function, and the penalty terms of the penalty function include a penalty term for electricity market exchange constraints and a penalty term for the thermal output value of the electric boiler.

[0034] Secondly, the present invention provides an energy management system for an integrated electrothermal energy system, comprising:

[0035] The model building module is used to acquire the first operating data of the target integrated electrothermal energy system and to build a first energy management research model based on the first operating data.

[0036] The first energy management research model includes an objective function and several constraints;

[0037] The framework building module is used to establish a Markov decision process framework for the first energy management research model;

[0038] The optimization module is used to construct the first improved algorithm to optimize the Markov decision process framework and obtain the optimal policy framework model.

[0039] The first improved algorithm includes a dynamic normalization strategy and deterministic training;

[0040] The solution module is used to solve the optimal strategy framework model to obtain the energy management optimization strategy of the integrated electric and thermal energy system.

[0041] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0042] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0043] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention proposes an energy management method for an integrated electrothermal energy system. It acquires the first operating data of the target integrated electrothermal energy system and establishes a first energy management research model based on this data. It then establishes a Markov decision process framework for this model; constructs a first improved algorithm to optimize the framework, obtaining the optimal strategy framework model; and solves the model to obtain the energy management optimization strategy for the integrated electrothermal energy system. This invention avoids explicit modeling of random variables and reduces the complexity of optimization by establishing a mathematical model and expressing it as a Markov decision process. By designing a reward mechanism and improving the PPO algorithm, it enhances search efficiency and solution accuracy, providing the system with an energy management optimization method that is both economical and reliable. It can effectively make decisions based on real-time observation data, improving the speed and stability of agent learning, reducing computation, and lowering operating costs. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart of an energy management method for an integrated electrothermal energy system provided in one embodiment of the present invention.

[0046] Figure 2 The diagram shows an integrated electric and thermal energy system in schemes 1 to 4 of an energy management method for an integrated electric and thermal energy system provided in an embodiment of the present invention.

[0047] Figure 3 This is a schematic diagram of the return curves of different algorithms in Scheme 1 of an energy management method for an integrated electrothermal energy system provided in an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram of the test daily load curve in Scheme 2 of an energy management method for an integrated electric and thermal energy system provided in an embodiment of the present invention.

[0049] Figure 5 This is a schematic diagram of the test daytime wind and solar power output in Scheme 2 of an energy management method for an integrated electric and thermal energy system provided in an embodiment of the present invention.

[0050] Figure 6 This is a schematic diagram of the test results of the daily load optimization in Scheme 2 of an energy management method for an integrated electric and thermal energy system provided in an embodiment of the present invention.

[0051] Figure 7 This is a schematic diagram of the test results of daily heat load optimization in Scheme 2 of an energy management method for an integrated electric and thermal energy system provided in an embodiment of the present invention.

[0052] Figure 8 This is a schematic diagram of a typical wind power output scenario in Scheme 3 of an energy management method for an integrated electrothermal energy system provided in an embodiment of the present invention.

[0053] Figure 9 This is a schematic diagram of a typical photovoltaic output scenario in Scheme 3 of an energy management method for an integrated electrothermal energy system provided in an embodiment of the present invention.

[0054] Figure 10 This diagram illustrates the operating costs of different algorithms in Scheme 3 of an energy management method for an integrated electrothermal energy system, as provided in an embodiment of the present invention.

[0055] Figure 11This is a schematic diagram of the average reward value of the agent under different cost discount coefficients in Scheme 4 of an energy management method for an integrated electrothermal energy system provided in an embodiment of the present invention.

[0056] Figure 12 This is a schematic diagram of the average reward value of the agent under different penalty coefficients in Scheme 4 of an energy management method for an integrated electrothermal energy system provided in an embodiment of the present invention.

[0057] Figure 13 This is an internal structural diagram of an electronic device for an energy management method of an integrated electrothermal energy system, provided as an embodiment of the present invention. Detailed Implementation

[0058] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0059] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides an energy management method for an integrated electrothermal energy system, comprising:

[0060] Existing technologies have several drawbacks. For instance, energy management in integrated electrothermal energy systems often lacks efficiency and flexibility, making it difficult to meet energy demands across different scenarios. Traditional management methods may overlook the complementarity and synergy between energy sources, leading to low energy utilization efficiency. Furthermore, existing energy management systems may lack intelligence and automation, requiring complex manual operations and decision-making, which not only increases operating costs but may also reduce system stability and reliability.

[0061] This invention provides a method that can effectively solve the problems mentioned above. The following sections will elaborate on how to implement this energy management method for an integrated electrothermal energy system using multiple embodiments.

[0062] Figure 1 A flowchart of an energy management method for an integrated electrothermal energy system is shown, including:

[0063] S101, Obtain the first operating data of the target integrated electrothermal energy system, and establish a first energy management research model based on the first operating data, wherein:

[0064] It should be noted that the integrated electric and thermal energy system provided in this embodiment of the invention consists of an electric boiler, an electric load, a thermal storage load, a power grid, a combined heat and power unit, an electrochemical energy storage system, and a renewable energy generator set.

[0065] In some specific implementations, the integrated electrothermal energy system may also include other auxiliary equipment, such as heat load controllers and power regulators, to further enhance the system's flexibility and controllability.

[0066] It should be noted that any auxiliary equipment added to the integrated electric and thermal energy system structure of this embodiment, or any simple improvement to the integrated electric and thermal energy system structure of this embodiment, can achieve the final energy management by adaptively modifying the specific method of this invention. Therefore, these improvements should also be within the scope of protection of this invention.

[0067] In some specific implementations, the first operating data refers to real-time monitoring data of the integrated electrothermal energy system within a specific time period, including but not limited to the output status of each device, load status, real-time power generation from renewable energy sources, and grid-connected power. This data is acquired through sensors, metering instruments, and other data collection devices, and after preprocessing, is input into the energy management system. The specific first operating data can be selected by those skilled in the art based on actual needs; this invention does not impose any limitations.

[0068] In some specific implementations, the first energy management research model can be established using machine learning algorithms, statistical methods, or optimization theory, aiming to reflect the operating characteristics and energy flow relationships of the integrated electrothermal energy system. For example, specific machine learning algorithms may include support vector machines, neural networks, and decision trees; statistical methods may include regression analysis and principal component analysis; and optimization theory may involve linear programming and integer programming. The choice of these models and methods depends on the complexity of the integrated electrothermal energy system, data availability, and management objectives.

[0069] In other specific implementations, the first energy management research model can also incorporate deep learning techniques, such as neural network models, to capture the complex nonlinear relationships in the integrated electrothermal energy system, further improving the model's prediction accuracy and generalization ability. For example, a deep neural network can be constructed and trained using a large amount of historical operating data, enabling it to learn the potential correlations and changing patterns among various variables in the integrated electrothermal energy system.

[0070] In this embodiment of the invention, the first energy management research model includes an objective function and several constraints;

[0071] In this embodiment of the invention, the objective function includes:

[0072] Establish an arbitrary function to characterize the total cost of the target integrated electrothermal energy system that minimizes the total cost;

[0073] The total cost includes the individual operating costs of several devices that make up the target integrated electric and thermal energy system, as well as the electricity purchase and sales costs of the target integrated electric and thermal energy system.

[0074] It should be noted that cost is a key indicator for measuring the economics of an energy system. By minimizing total cost, it is possible to ensure that the energy system operates both efficiently and economically.

[0075] In some specific implementations, various methods can be used to establish a function that characterizes the minimization of the total cost of the target integrated electrothermal energy system, including but not limited to mathematical optimization techniques such as linear programming, quadratic programming, or nonlinear programming. These methods allow for the precise calculation of the energy allocation strategy that minimizes the total cost based on the actual operating conditions and constraints of the integrated electrothermal energy system.

[0076] In other specific implementations, heuristic or metaheuristic algorithms, such as genetic algorithms and particle swarm optimization algorithms, can be introduced to find near-optimal solutions within a reasonable timeframe, especially when dealing with large-scale or highly complex integrated electrothermal energy systems. These algorithms can explore the solution space to find solutions close to the global optimum while reducing computational complexity and time costs.

[0077] In this embodiment of the invention, the expression for the objective function is:

[0078]

[0079] In the formula, t represents the operating time of the integrated electric and thermal energy system; T represents the total operating time of the integrated electric and thermal energy system; C(t) represents the total cost of the integrated electric and thermal energy system; C grid (t) represents the cost of the integrated energy system purchasing and selling electricity to the electricity market during time period t; C chp (t) represents the operating cost of the cogeneration unit during time period t; C eb (t) represents the boiler's operating cost during time period t; C bess (t) represents the operating cost of the electrochemical energy storage system during time period t.

[0080] In some specific implementations, the cost C of the integrated energy system purchasing and selling electricity from the electricity market during time period t is... grid (t) is obtained from the following formula:

[0081] C grid (t)=λ e (t)P grid (t)λ disc Δh

[0082] In the formula, λe (t) represents the electricity price during time period t; λ disc The electricity sales discount factor is represented by Δh; the time step of period t is represented by P. grid (t) represents the power exchanged between the system and the electricity market during time period t, when P grid When (t)>0, it represents the system purchasing electricity from the electricity market, λ disc The value is 1, otherwise it is electricity sales, λ disc ∈[0,1].

[0083] In some specific implementations, the operating cost C of the combined heat and power unit over time period t is... chp (t) is obtained from the following formula:

[0084] C chp (t)=(a c (P chp (t)) 2 +b c P chp (t)+c c +d c (Q chp (t)) 2 +e c Q chp (t)+f c )Δh

[0085] In the formula, P chp (t) represents the power generation of the cogeneration unit during time period t; Q chp (t) represents the heat output of the cogeneration unit during time period t; a c ,b c ,c c ,d c ,e c ,f c These all represent the cost coefficients of a combined heat and power (CHP) unit. For simplicity, the cost calculation of a CHP unit can be directly replaced by the price of natural gas, defined as follows:

[0086]

[0087] In the formula, λ gas (t) represents the unit calorific value price of natural gas during time period t; η1 and η2 are the unit cost coefficients of the combined heat and power unit.

[0088] In some specific implementations, the boiler operating cost C over time period t is... eb (t) is obtained from the following formula:

[0089] C eb (t)=κ eb (t)P eb(t)Δh

[0090] In the formula, κ eb (t) represents the unit cost coefficient of the electric boiler during time period t; P eb (t) represents the electrical power consumed by the electric boiler during the time period t.

[0091] In some specific implementations, the operating cost C of the electrochemical energy storage system over time period t is... bess (t) is obtained from the following formula:

[0092] C bess (t)=κ bess (t)|P bess (t)|Δh

[0093] In the formula, κ bess (t) represents the depreciation unit operating cost coefficient of the electrochemical energy storage system during time period t; P bess (t) represents the electrical power exchanged between the system and the electrochemical energy storage system during time period t, when P bess When (t) > 0, it represents the electrochemical energy storage system discharging; otherwise, it represents charging.

[0094] It should be noted that the total cost may also include some additional costs related to the operation of the integrated electric and thermal energy system, such as equipment maintenance costs, downtime losses due to malfunctions, and environmental impact costs. Since these costs do not meet the primary focus of this embodiment, they will not be discussed in detail here. However, in practical applications, these additional costs should also be considered to comprehensively evaluate the economics of the integrated electric and thermal energy system.

[0095] In some specific implementations, constraints can be established for the aforementioned objective function to achieve stable operation and optimized scheduling of the energy system. These constraints ensure that the integrated electric and thermal energy system can meet energy demand while maintaining safe and stable operation of equipment and optimizing energy utilization efficiency.

[0096] In some specific implementations, constraints may include power balance constraints, meaning that the power and heat supply and demand of the integrated electrothermal energy system should remain balanced throughout the various time periods to ensure stable system operation. Additionally, constraints may include upper and lower output limits for equipment to ensure that each device operates within a safe range, avoiding overload or underload. Furthermore, considering the potential presence of energy storage devices in the integrated electrothermal energy system, constraints should also include charge / discharge state constraints and capacity constraints for these devices to rationally manage their charging and discharging processes and optimize energy storage and utilization.

[0097] In some specific implementations, constraints may also include reliability and stability constraints on the energy system, ensuring that the integrated electric and thermal energy system maintains a continuous and stable energy supply under various operating scenarios, avoiding system interruptions due to equipment failure or energy shortages. These constraints help improve the system's resilience, ensuring its normal operation in the face of emergencies or extreme weather conditions, and meeting users' energy needs.

[0098] In this embodiment of the invention, in order to meet the requirements of the integrated electric and thermal energy system of the present invention, several constraints include power balance constraints, electric boiler operation constraints, cogeneration unit operation constraints, electrochemical energy storage system operation constraints, and electricity market exchange constraints.

[0099] In this embodiment of the invention, the power balance constraint is:

[0100] P rew (t)+P chp (t)+P bess (t)+P grid (t)-P eb (t)=P load (t)

[0101] Q chp (t)+Q eb (t)=Q load (t)

[0102] Q eb =η eb P eb (t)

[0103] In the formula, P rew (t) represents the renewable energy output of the system during time period t, including wind power output and photovoltaic power output; P load (t) represents the electrical load of the system during time period t; Q load (t) represents the system's heat load during time period t; Q eb (t) represents the thermal output value of the electric boiler during time period t; η eb This indicates the operating efficiency parameters of the electric boiler.

[0104] In this embodiment of the invention, the operating constraints of the electric boiler are as follows:

[0105] P eb,min ≤P eb (t)≤P eb,max

[0106] In the formula, P eb,min P eb,max These represent the minimum and maximum power consumption during the operation of the electric boiler, respectively.

[0107] In this embodiment of the invention, the operating constraints of the combined heat and power unit are as follows:

[0108] |P chp (t)-P chp (t-1)|≤P climb,max

[0109] |Q chp (t)-Q chp (t-1)|≤Q climb,max

[0110] In the formula, P climb,max Q represents the maximum power output ramp-up of the unit per unit time; climb,max This indicates the maximum thermal output ramp-up of the unit per unit time.

[0111] In this embodiment of the invention, the operating constraints of the electrochemical energy storage system are:

[0112]

[0113] |P bess (t)|≤P bess,max

[0114] SOC min ≤SOC(t)≤SOC max

[0115] In the formula, η ch η dis These represent the charging and discharging power of the electrochemical energy storage system, respectively; SOC(t) represents the state of charge of the electrochemical energy storage system during time period t; E cap Rated capacity of the electrochemical energy storage system; P bess,max The maximum charge / discharge power of the electrochemical energy storage system; SOC min SOC max These represent the minimum and maximum states of charge of the electrochemical energy storage system, respectively.

[0116] In this embodiment of the invention, the electricity market exchange constraints are as follows:

[0117] |P grid (t)|≤P grid,max

[0118] In the formula, P grid,max This represents the maximum power exchanged between the integrated energy system and the electricity market.

[0119] It should be noted that relevant technical personnel can flexibly adjust and expand these constraints according to the specific needs and operating environment of the integrated electrothermal energy system to ensure the system's economy and reliability. For example, in specific application scenarios, the environmental performance of the energy system may need to be considered. In this case, carbon emissions can be introduced as one of the constraints, and carbon emissions can be reduced by optimizing scheduling strategies to achieve green and sustainable energy utilization.

[0120] It should also be noted that obtaining the initial operating data of the target integrated electrothermal energy system and establishing a first energy management research model based on this data can more accurately simulate and predict the actual operation of the integrated electrothermal energy system, providing strong data support for subsequent energy management strategy formulation. In-depth analysis of the first energy management research model can identify energy efficiency bottlenecks and optimization potential within the system, leading to targeted improvement measures. Furthermore, the first energy management research model can serve as the basis for subsequent optimization algorithms and strategy verification, gradually improving the overall energy efficiency and economy of the integrated electrothermal energy system through continuous iteration and optimization.

[0121] S102, Establish a Markov decision process framework for the first energy management research model, wherein:

[0122] It should be noted that after obtaining the first energy management research model, it is necessary to solve the first energy management research model to obtain the optimal energy management strategy, and the Markov decision process framework is an effective solution method. It can transform the energy management problem of the integrated electrothermal energy system into a sequential decision problem. At each decision moment, the system selects an optimal action based on the current state and the available action set to maximize long-term benefits or minimize long-term costs.

[0123] In this framework, the system's state can include the operating status of each device, energy market price information, and the system's energy demand, while actions can include adjusting the output of devices and exchanging electricity with the energy market.

[0124] In some specific implementations, a complete Markov decision process model can be constructed by defining appropriate reward or cost functions and transition probabilities. When solving this model, methods such as dynamic programming and reinforcement learning can be used to obtain the optimal energy management strategy. These strategies can guide the integrated electrothermal energy system on how to rationally allocate energy and optimize equipment operation under different operating scenarios, thereby reducing operating costs and improving energy efficiency.

[0125] In other specific implementations, solutions can be found in different ways, such as using heuristic algorithms or intelligent optimization algorithms. Specifically, these include simulated annealing, genetic algorithms, or particle swarm optimization. These algorithms can find approximate optimal solutions to complex energy management problems. Heuristic algorithms are typically based on empirical rules or intuitive judgments and can provide good solutions within an acceptable timeframe, although they do not necessarily guarantee finding the global optimum. Intelligent optimization algorithms, on the other hand, utilize certain phenomena or principles in nature for searching and optimization, exhibiting strong global search capabilities and robustness.

[0126] However, the content of this invention is relatively complex and requires a lot of optimization or calculation. Therefore, it is more appropriate to use the Markov decision process framework to solve it.

[0127] In this embodiment of the invention, establishing a Markov decision process framework for the first energy management research model includes:

[0128] The first energy management research model is treated as an intelligent agent;

[0129] Establish an agent-based Markov decision process framework;

[0130] In the Markov decision process framework, the reward function is an arbitrary function that characterizes the agent's search for the optimal strategy within the feasible domain of the target electrothermal integrated energy system.

[0131] In this embodiment of the invention, the reward function includes a penalty function, and the penalty terms of the penalty function include a penalty term for electricity market exchange constraints and a penalty term for the thermal output value of the electric boiler.

[0132] In this embodiment of the invention, establishing an agent-based Markov decision process framework includes:

[0133] The output and load status of several devices in the target integrated electrothermal energy system are used as states in the Markov decision process framework.

[0134] The output increment of several devices in the target integrated electrothermal energy system is used as a decision variable;

[0135] The decision variables are treated as actions within the Markov decision process framework.

[0136] Specifically, in this embodiment of the invention, the energy management system of the integrated electrothermal energy system is treated as an intelligent agent. Within the framework of deep reinforcement learning, the Markov decision mechanism for establishing the energy management optimization model of the integrated electrothermal energy system includes the following steps:

[0137] The energy management system of the integrated electric and thermal energy system is treated as an intelligent agent. It can observe the output, electrical load, and thermal load of renewable energy units that are not controlled by the system, as well as the output of units controlled by the system at the previous moment and the state of charge of the electrochemical energy storage system at the previous moment. The state of the system is described as follows:

[0138] s t =(P rew (t),P load (t),Q load (t),λ e (t),λ gas (t),SOC(t-1),P chp (t-1),Q chp (t-1))

[0139] In some specific implementations, the output of each device is used as a decision variable. Considering the temporal coupling of unit output, the output increment of the unit is used as a decision variable. The system's action is described as follows:

[0140] a t =(ΔP) chp (t),ΔQ chp (t),P bess (t))

[0141] In some specific implementations, during the time period t, the system determines the current state s. t and current action a t Transition to the next state s t+1 The state transition model of the system is defined as follows:

[0142]

[0143] In the formula, Let be the set of system state transition probabilities.

[0144] In some specific implementations, a reward function is designed to guide the agent to find the optimal strategy within the feasible region to achieve the system's operational goal, and a penalty function is designed to ensure that no constraint violations occur during operation. The reward function is expressed as follows:

[0145] r t =-(a1C(t)+a2C) ex (t))

[0146] p en (v x )=|v x -v min |+|v x -v max |-|v max -vmin |

[0147] In the formula, C ex (t) represents the constraint penalty term, which will set P... grid and Q eb The operational constraints are defined as p en (P grid ) and p en (Q eb The penalty items are described as follows:

[0148] C ex (t)=p en (P grid (t))+p en (Q eb )

[0149] In the formula, a1 is the cost scaling factor and a2 is the penalty scaling factor.

[0150] It should be noted that establishing a Markov decision process framework for the first energy management research model provides a structured framework that transforms the complex energy management problem of an integrated electrothermal energy system into a solvable sequential decision problem. By defining states, actions, and reward functions within this framework, we can utilize Markov decision process theory and methods to find the optimal energy management strategy. This method considers not only the current state of the system but also possible future state transitions and long-term benefits, thus enabling the formulation of more comprehensive and effective energy management strategies. Furthermore, this framework possesses good scalability and flexibility, allowing for adjustment and optimization based on the specific needs and operating environment of the integrated electrothermal energy system to adapt to different application scenarios and constraints.

[0151] S103, Construct the first improved algorithm to optimize the Markov decision process framework and obtain the optimal policy framework model, where:

[0152] It should be noted that once the Markov decision process framework is obtained, it is necessary to perform framework optimization on the Markov decision process framework in order to find the optimal energy management strategy.

[0153] In some specific implementations, deep reinforcement learning algorithms can be used for optimization. Deep reinforcement learning combines the perceptual capabilities of deep learning with the decision-making capabilities of reinforcement learning, enabling it to effectively learn optimal policies in high-dimensional state spaces and continuous action spaces.

[0154] In some specific implementations, suitable deep reinforcement learning algorithms, such as Deep Q-Networks (DQN), policy gradient algorithms, or actor-critic algorithms, can be selected to train and optimize the Markov decision process framework. During training, the agent interacts with the environment, continuously tries different actions, and receives feedback based on the reward function, thereby gradually adjusting and optimizing its policy. After multiple iterations of training, the agent will be able to learn the policy of selecting the optimal action in a given state, i.e., the optimal energy management policy. This policy can guide the integrated electrothermal energy system on how to rationally allocate energy and optimize equipment operation under different operating scenarios, thereby reducing operating costs and improving energy efficiency.

[0155] In other specific implementations, optimization can be achieved using heuristic algorithms or intelligent optimization algorithms. These algorithms can find near-optimal solutions to complex energy management problems, thus meeting the practical needs of integrated electrothermal energy systems. For example, simulated annealing can be used to optimize a Markov decision process framework. By simulating the annealing process, the temperature is gradually reduced, allowing the agent to search the solution space and approach the optimal solution. Alternatively, genetic algorithms can be used, continuously evolving the population through selection, crossover, and mutation to find the individual with the highest fitness, i.e., the optimal energy management strategy. Furthermore, particle swarm optimization is also a viable option. It simulates the foraging behavior of birds, gradually converging to the optimal solution through information sharing and cooperation among particles. These algorithms each have their own characteristics, and the appropriate algorithm can be selected for optimization based on the specific circumstances and needs of the integrated electrothermal energy system.

[0156] However, existing optimization methods are difficult to apply directly to this invention because they are not well-suited. For example, although deep reinforcement learning algorithms are powerful, their training process requires high computational resources. Furthermore, in large-scale integrated electrothermal energy systems, the state space and action space may be extremely large, leading to a significant increase in training difficulty and time cost.

[0157] Therefore, in this embodiment of the invention, a first improved algorithm for the characteristics of integrated electrothermal energy systems is proposed. While maintaining the advantages of deep reinforcement learning algorithms, this algorithm improves its applicability and efficiency in energy management problems of integrated electrothermal energy systems by introducing a series of optimization measures.

[0158] In this embodiment of the invention, the first improved algorithm includes a dynamic normalization strategy and deterministic training;

[0159] The improved PPO algorithm first adopts a dynamic normalization update method. The mean and variance update methods under the dynamic normalization strategy are as follows:

[0160]

[0161] In the formula, s t The state at time t; and State s t Mean and variance before update; and State s t The updated mean and variance; n is the total number of states currently in existence.

[0162] Secondly, deterministic training is introduced into the improved PPO algorithm to enhance convergence performance in the later stages of training. Referring to the greedy strategy, the action with the highest probability is selected deterministically according to a certain probability, as defined below:

[0163]

[0164] In the formula, P is the probability of action a being selected; a is the selected action; a′ is all candidate actions; s is the environment state; θ is the policy network parameter; π(a′|s,θ) is the policy function; and ∈ is the greedy parameter.

[0165] In this embodiment of the invention, the intelligent agent comprises two parts: a policy network and a value network. The method for updating the value network using temporal difference theory is as follows:

[0166]

[0167] θ c =θ c +α c ▽L(θ c )

[0168] In the formula, L(θ) c ) represents the loss function of the value network; Let θ be the expected function; c For value network parameters; r t For state s t Perform action a t The immediate reward; γ is the decay coefficient of future rewards; α c V represents the learning rate of the value network. π (s t ) represents the current state s t The value function under; For gradient operators; s t+1 This represents the state at the next moment.

[0169] In some specific implementations, an advantage function A is defined during policy network training. π Let (s,a) represent the current state s. t Selected action a tThe advantage of other optional actions, if the selected action a t If the performance is above average, then the advantage function A π (s,a) are positive numbers, otherwise negative numbers. The dominance function is defined as:

[0170] A π (s,a)=Q π (s,a)-V π (s)

[0171] In the formula, A π (s,a) is the dominance function; Q π (s,a) is the action state value function.

[0172] In some specific implementations, the policy network is updated as follows;

[0173]

[0174] In the formula, L(θ) a θ is the loss function of the policy network; a θ represents the current policy network parameters. a2 For old strategy network parameters; Let be the expectation function; ε∈(0,1) are parameters that restrict the change of the old and new strategy functions; This is a generalized estimate of the advantage function; ρ(θ) is the probability ratio of the new and old policy networks; clip(ρ(θ), 1-ε, 1+ε) is a function that limits the range of the probability ratio of the new and old policy networks, where (1-ε, 1+ε) is the range of the probability ratio; π(a t |s t ,θ a ) represents the current strategy; π(a) t |s t ,θ a2 ) represents the old strategy; α a is the learning rate of the policy network.

[0175] In some specific implementations, generalized dominance estimation is introduced. The calculation method for estimating the dominance function is as follows:

[0176]

[0177] δ t =r t +γV π (s t+1 )-V π (s t )

[0178] In the formula, δ tLet be the value prediction bias at time t; γ be the future reward decay coefficient; λ be the hyperparameter of the balance bias and variance; r t For state s t Perform action a t Instant rewards; V π (s t ) represents the state s under policy π. t State value function; s t+1 This is the state at the next moment.

[0179] After multiple updates to the policy network, the differences between the old and new policy networks gradually increase. The old policy network will be updated periodically to adapt to the new policy network. The update method is as follows:

[0180] θ a2 ←θ a

[0181] In the formula, θ a θ represents the current policy network parameters. a2 These are the network parameters for the old strategy.

[0182] It should be noted that after completing the first improved algorithm optimization, the optimal energy management strategy framework model of the integrated electrothermal energy system can be obtained. This model can intelligently adjust the output and load distribution of the equipment according to different operating states and environmental conditions to achieve efficient energy utilization and cost minimization.

[0183] It should also be noted that constructing the first improved algorithm to optimize the Markov decision process framework and obtain the optimal strategy framework model can significantly improve the efficiency and effectiveness of energy management in integrated electrothermal energy systems. Through optimization using the first improved algorithm, integrated electrothermal energy systems can better cope with various operating scenarios and constraints, and formulate more precise and effective energy management strategies. This not only helps reduce system operating costs and improve energy efficiency, but also enhances system stability and reliability, providing strong support for the sustainable development of integrated electrothermal energy systems. Simultaneously, this optimal strategy framework model also has good scalability and portability, adaptable to integrated electrothermal energy systems of different scales and types, bringing more innovation and application possibilities to the field of energy management.

[0184] S104, Solve the optimal strategy framework model to obtain the energy management optimization strategy for the integrated electric and thermal energy system, where:

[0185] It should be noted that appropriate numerical methods and optimization techniques are required in the process of solving the optimal strategy framework model to ensure the accuracy and efficiency of the solution.

[0186] In some specific implementations, advanced numerical solvers, such as nonlinear programming solvers or mixed-integer programming solvers, can be used to solve the optimal policy framework model. These solvers can handle complex constraints and objective functions, and gradually approximate the optimal solution through iterative calculations.

[0187] During the solution process, it is also necessary to consider the actual operating data and historical information of the integrated electrothermal energy system in order to calibrate and verify the model. By comparing and analyzing with actual data, the model parameters can be further adjusted and optimized to improve the model's accuracy and practicality.

[0188] It should be noted that the solution will yield an energy management optimization strategy for the integrated electrothermal energy system. This strategy will detail how to rationally allocate energy and optimize equipment operation under different operating scenarios to reduce operating costs and improve energy efficiency. The strategy will fully consider the system's economics and environmental friendliness, ensuring that energy demand is met while achieving efficient energy utilization and sustainable development.

[0189] In this embodiment of the invention, solving the optimal strategy framework model includes:

[0190] Acquire renewable energy output data and load consumption data from the target integrated electric and thermal energy system;

[0191] Based on renewable energy output data and load consumption data, the optimal strategy framework model is solved using a solver.

[0192] The solution results will be used as an energy management optimization strategy for the integrated electrothermal energy system.

[0193] Specifically, the renewable energy output data and load consumption data of the target integrated electric and thermal energy system, including historical data on renewable energy output, electrical load and heat load, are divided into training set and test set;

[0194] The training set and test set are sequentially input into the integrated electrothermal energy system, and the solution is obtained by using a solver to obtain the energy management optimization strategy of the integrated electrothermal energy system.

[0195] In summary, this invention proposes an energy management method for an integrated electrothermal energy system. It acquires the first operational data of the target integrated electrothermal energy system and establishes a first energy management research model based on this data. A Markov decision process framework is then established for this model. A first improved algorithm is constructed to optimize the framework, obtaining the optimal strategy framework model. Finally, the model is solved to obtain the energy management optimization strategy for the integrated electrothermal energy system. This invention avoids explicit modeling of random variables and reduces the complexity of optimization by establishing a mathematical model and expressing it as a Markov decision process. By designing a reward mechanism and improving the PPO algorithm, it enhances search efficiency and solution accuracy, providing the system with an energy management optimization method that is both economical and reliable. This method enables effective decision-making based on real-time observation data, improves the speed and stability of agent learning, and reduces computation and operating costs.

[0196] Example 2, refer to Figures 2-3 In a preferred embodiment, the present invention also provides a specific method, employing... Figure 2 The integrated electrothermal energy system shown is used as a case study. The system's scheduling period is 24 hours, with a scheduling interval of 1 hour. The system adopts a time-of-use pricing mechanism with a sales discount factor of 0.5. The natural gas price is fixed at 0.4 yuan / kWh. The rated and initial capacities of the electrochemical energy storage system are 2000kWh and 1000kWh, respectively. Other parameters of the system are shown in Tables 1-5.

[0197] Table 1 Time-of-use pricing mechanism

[0198] Time period (h) Electricity price (yuan / kWh) Valley period 23:00～6:00 0.48 Normal period 7:00、11:00～17:00 0.90 Peak hours 8:00～10:00、18:00～22:00 1.35

[0199] Table 2 System Equipment Operating Parameters

[0200] parameter value parameter value <![CDATA[P grid,max ]]> 1000kW <![CDATA[η eb ]]> 0.95 <![CDATA[P eb.min / P eb.max ]]> 0 / 1200kW <![CDATA[κ eb ]]> 0.05 <![CDATA[P bess,max ]]> 200kW <![CDATA[κ bess ]]> 0.05 <![CDATA[SOC min / SOC max ]]> 0.2 / 0.9 <![CDATA[η ch / or dis ]]> 0.95 / 0.95 <![CDATA[P climb,max ]]> 300kW <![CDATA[Q climb,max ]]> 360kW <![CDATA[η1]]> 0.8 <![CDATA[η2]]> 0.75 <![CDATA[λ disc ]]> 0.5 A (0,0) B (960,800) C (960,400) D (200,0) E (0,0)

[0201] Table 3 Strategy Network Structure

[0202]

[0203] Table 4 Value Network Structure

[0204]

[0205] Table 5 Training parameters for deep neural networks

[0206] Hyperparameters value Hyperparameters value Policy Network Learning Rate 3e-4 Policy Network Learning Rate 3e-4 Return Discount Factor 0.99 Truncation parameters 0.2 Greedy parameters 0.2 Replay experience pool size 10000 Small batch sampling quantity 128 Maximum number of training rounds 1500

[0207] To verify the effectiveness of the energy management method for the integrated electrothermal energy system in this embodiment, four schemes were selected for verification:

[0208] Option 1: Train the proposed improved PPO algorithm and the PPO algorithm using historical data, compare the reward functions during the training process, and analyze the learning and training effects of the two algorithms.

[0209] Option 2: Based on Option 1, the saved improved PPO algorithm model is used for the energy management optimization problem of the integrated electric and thermal energy system. Historical data is randomly selected from the test set as test data to optimize the system's energy management.

[0210] Option 3: Based on Option 1, consider using model-based improved particle swarm optimization algorithm, stochastic programming algorithm and mixed integer linear programming method to optimize the system energy management, and compare the results with the proposed method.

[0211] Option 4: Based on Option 1, consider the impact of parameters in the reward function on the agent's accumulation of environmental experience. Process the problem by setting different parameter values ​​and verify the robustness of the proposed method.

[0212] Option 1: This option does not consider the integrated electric and thermal energy system, and is... Figure 3 It can be seen that the improved PPO algorithm exhibits superior stability compared to the standard PPO (-800 to -1300) in the initial exploration phase (-520 to -580), and converges rapidly within 550 rounds, maintaining stability in the later stages; while the standard PPO requires 750 rounds to converge and still exhibits fluctuations. Furthermore, dynamic state normalization effectively mitigates early fluctuations, and the deterministic strategy avoids a sharp performance drop in the later stages (e.g., at 786 and 944 rounds), indicating that the improved PPO significantly improves both convergence speed and stability.

[0213] Option 2: This option uses the saved improved PPO algorithm model for the energy management optimization problem of the integrated electric and thermal energy system. Figures 4-7 It can be seen that the CHP unit bears the main electricity / heat load. Under the time-of-use pricing mechanism, the system achieves economic dispatch by purchasing electricity during low-price periods (0-6h) and selling electricity during high-price periods (8-10h, 18-22h). Electrochemical energy storage participates in peak shaving and valley filling, while electric boilers supplement heating when CHP output is limited. This scheme significantly improves overall economic efficiency while meeting system constraints through multi-device collaborative optimization.

[0214] Option 3: The proposed improved PPO algorithm is compared with other algorithms. The improved particle swarm optimization algorithm is set with a particle count of 300, a maximum number of iterations of 1000, individual learning coefficients and swarm learning coefficients of 1.6 and 2 respectively, an inertia weight parameter of 0.9, and a decay coefficient of 0.8. The result obtained by the MILP method is considered the theoretically optimal result, and the system operating cost error is defined as follows:

[0215]

[0216] In the formula, c i For the system running cost obtained by other optimization algorithms, c MILP This represents the theoretically optimal system operating cost obtained using the mixed-integer linear programming method.

[0217] Table 6 Evaluation Indicators for Copula Functions

[0218]

[0219]

[0220] Table 7 Probability of Occurrence for Each Scenario

[0221] Scene Number Scene 1 Scene 2 Scene 3 Scene 4 Scene 5 probability 0.243 0.167 0.201 0.198 0.191

[0222] Table 8 Online Execution Time

[0223] method Online execution time MILP 14.65 IPSO 13.25 SP 27.45 PPO 0.33 DDPG 0.35 IPPO 0.34

[0224] Table 6 shows that wind power and photovoltaic output are negatively correlated (correlation coefficient < 0). t-Copula was selected as the best correlation description method because its correlation coefficient is closest to the original data and its empirical Copula distance is the smallest. After constructing a joint probability distribution based on t-Copula, Monte Carlo simulation was used to generate 2000 output scenarios, which were then reduced to 5 typical scenarios using k-means clustering. Figure 8 , 9 The probability distributions are shown in Table 7. Finally, the system models for each scenario are constructed as MILP optimization problems. Figure 10 The table compares the operating costs of five algorithms: the baseline cost of MILP is 14598.58 yuan, while the improved PPO algorithm (14786.33 yuan) only increases by 1.29%, significantly outperforming PPO (4.97%), DDPG (5.03%), stochastic optimization (3.69%), and improved particle swarm optimization (5.44%). Table 8 shows that the online execution time of the improved PPO algorithm is only 0.01 seconds slower than that of the PPO algorithm, and far superior to MILP (14.65s), IPSO (13.25s), and SP (27.45s). This demonstrates that the improved PPO algorithm not only has the lowest average cost but also the smallest error fluctuation range and a significant advantage in online execution time, indicating that this algorithm can effectively address the uncertainties of integrated electrothermal energy systems and achieve optimal decision-making.

[0225] Option 4: Consider the impact of parameters in the reward function on the agent's accumulation of environmental experience, and process it by setting different parameter values. Figure 11-12The analysis of the reward curves shows that systematically adjusting key parameters can effectively optimize the training performance of the agent. The optimal convergence effect is achieved when the cost discount coefficient a1 is 0.01, while the best stability is observed when the penalty discount coefficient a2 is 100. A value that is too large (150) leads to increased fluctuations in the reward function, while a value that is too small (50) affects the convergence performance. Experimental results show that under different parameter combinations, the improved PPO algorithm (first improved algorithm) proposed in this invention maintains stable convergence characteristics. Its reward curve shows a steady upward trend and eventually converges to a better range, verifying that the method has good robustness and adaptability in parameter selection and can effectively balance the relationship between convergence speed and training stability.

[0226] This invention establishes a mathematical model for energy management in an integrated electrothermal energy system and expresses it as a Markov decision process, avoiding explicit modeling of random variables such as renewable energy sources, reducing the complexity of optimization solutions. By designing a reward mechanism and improving the PPO algorithm, it significantly enhances the search efficiency and solution accuracy of the optimization process, providing an economical and reliable energy management optimization method for integrated electrothermal energy systems. This method can make effective decisions based on real-time system observation data, improve the speed and stability of agent learning, reduce complex calculations, and effectively lower system operating costs.

[0227] Example 3, referring to Figure 13 This embodiment also provides an energy management system for an integrated electrothermal energy system, including:

[0228] The model building module is used to acquire the first operating data of the target integrated electrothermal energy system and to build a first energy management research model based on the first operating data.

[0229] The first energy management research model includes an objective function and several constraints;

[0230] The framework building module is used to establish a Markov decision process framework for the first energy management research model;

[0231] The optimization module is used to construct the first improved algorithm, optimize the Markov decision process framework, and obtain the optimal policy framework model.

[0232] The first improved algorithm includes a dynamic normalization strategy and deterministic training;

[0233] The solution module is used to solve the optimal strategy framework model to obtain the energy management optimization strategy of the integrated electric and thermal energy system.

[0234] The above-mentioned unit modules can be embedded in the processor of the electronic device in hardware form or independent of it, or they can be stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0235] This embodiment also provides an electronic device, which can be a terminal, and its internal structure diagram can be as follows. Figure 13 As shown, the electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an energy management method for an integrated electrothermal energy system. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the device's casing, or an external keyboard, touchpad, or mouse.

[0236] This embodiment also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it performs the following steps:

[0237] Obtain the first operating data of the target integrated electrothermal energy system and establish a first energy management research model based on the first operating data;

[0238] The first energy management research model includes an objective function and several constraints;

[0239] Establish a Markov decision process framework for the first energy management research model;

[0240] The first improved algorithm is constructed to optimize the Markov decision process framework and obtain the optimal policy framework model;

[0241] The first improved algorithm includes a dynamic normalization strategy and deterministic training;

[0242] The optimal strategy framework model is solved to obtain the energy management optimization strategy for the integrated electric and thermal energy system.

[0243] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0244] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0245] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An energy management method for an integrated electrothermal energy system, characterized in that, include: Acquire the first operating data of the target integrated electrothermal energy system and establish a first energy management research model based on the first operating data; The first energy management research model includes an objective function and several constraints; Establish a Markov decision process framework for the first energy management research model; A first improved algorithm is constructed to optimize the Markov decision process framework and obtain the optimal policy framework model. The first improved algorithm includes a dynamic normalization strategy and deterministic training; The optimal strategy framework model is solved to obtain the energy management optimization strategy for the integrated electric and thermal energy system.

2. The energy management method for an integrated electrothermal energy system as described in claim 1, characterized in that, The establishment of the Markov decision process framework for the first energy management research model includes: The first energy management research model is treated as an intelligent agent; Establish a Markov decision process framework based on the aforementioned agent; In the Markov decision process framework, the reward function is an arbitrary function that characterizes the agent's search for the optimal strategy within the feasible domain of the target electrothermal integrated energy system.

3. The energy management method for an integrated electrothermal energy system as described in claim 2, characterized in that, The objective function includes: Establish an arbitrary function to characterize the total cost of the target integrated electrothermal energy system that minimizes the total cost; The total cost includes the individual operating costs of several devices that make up the target integrated electric and thermal energy system, as well as the electricity purchase and sales costs of the target integrated electric and thermal energy system.

4. The energy management method for an integrated electrothermal energy system as described in claim 3, characterized in that, The establishment of the Markov decision process framework based on the intelligent agent includes: The output and load status of several devices in the target electrothermal integrated energy system are used as states in the Markov decision process framework. The output increment of several devices in the target integrated electrothermal energy system is used as a decision variable; The decision variables are treated as actions within the Markov decision process framework.

5. The energy management method for an integrated electrothermal energy system as described in claim 4, characterized in that, Solving the optimal strategy framework model includes: Acquire renewable energy output data and load consumption data from the target integrated electric and thermal energy system; Based on the renewable energy output data and load consumption data, the optimal strategy framework model is solved using a solver. The solution results will be used as an energy management optimization strategy for the integrated electrothermal energy system.

6. The energy management method for an integrated electrothermal energy system as described in claim 5, characterized in that, The constraints include power balance constraints, electric boiler operation constraints, combined heat and power unit operation constraints, electrochemical energy storage system operation constraints, and electricity market exchange constraints.

7. The energy management method for an integrated electrothermal energy system as described in claim 6, characterized in that, The reward function includes a penalty function, and the penalty terms of the penalty function include a penalty term for electricity market exchange constraints and a penalty term for the thermal output value of the electric boiler.

8. An energy management system for an integrated electrothermal energy system, employing the method described in any one of claims 1 to 7, characterized in that, include: The model building module is used to acquire the first operating data of the target integrated electrothermal energy system and to build a first energy management research model based on the first operating data. The first energy management research model includes an objective function and several constraints; The framework building module is used to establish a Markov decision process framework for the first energy management research model; The optimization module is used to construct the first improved algorithm to optimize the Markov decision process framework and obtain the optimal policy framework model. The first improved algorithm includes a dynamic normalization strategy and deterministic training; The solution module is used to solve the optimal strategy framework model to obtain the energy management optimization strategy of the integrated electric and thermal energy system.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the energy management method for an integrated electrothermal energy system according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the energy management method for an integrated electrothermal energy system according to any one of claims 1 to 7.

Images