Low-carbon building cluster energy management optimization method and system based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based low-carbon building cluster energy management optimization method and system, and relates to the technical field of building cluster comprehensive energy system management, and the method comprises the steps: building a single building environment; the current environment state of single building energy consumption is obtained; outputting actions of energy storage and an energy conversion device according to the current environment state; acquiring an action reward of a single building and an environment state of a next time step; putting a training sample into a buffer area, and maximizing a reward function by using a near-end strategy optimization algorithm; and the comprehensive energy optimization and low-carbon management of the building cluster are converged through a federal reinforcement learning framework. According to the invention, low-carbon management is carried out on a comprehensive energy management system of a single building based on a near-end strategy optimization algorithm. The energy management method can cope with uncertainty caused by environmental changes, and information of any uncertain system parameters and a building thermodynamic model do not need to be known.

Description

Technical Field

[0001] The present invention relates to the technical field of building cluster integrated energy system management, and in particular to a low-carbon building cluster energy management optimization method and system based on reinforcement learning. Background Art

[0002] With the acceleration of urbanization and the growing demand for energy, buildings, as the main part of urban energy consumption, have become increasingly prominent in terms of energy management. Although modern technology has made many advances in the design and management of building energy systems, there are still many challenges and problems, especially in the energy management of building clusters. First, there are many complex and random factors and uncertain system parameters in the comprehensive energy management of a single building. It is often difficult to develop an accurate and efficient building model for energy optimization. Secondly, in the building cluster system, it is necessary to consider the energy flow and optimization between buildings, including but not limited to energy sharing across buildings, load transfer under peak and valley electricity prices, etc. Coordinating the energy demand and supply of each building and optimizing the overall energy efficiency have become a major challenge. Third, multi-building systems involve a large amount of data collection, processing and analysis. Since different buildings may use different energy management systems, the integration of data and operating platforms between these systems is a major problem, which requires complex system integration technology, and when traditional optimization methods are used, it is difficult to solve the problem of large-scale building energy consumption optimization in real time. Fourth, although economies of scale can reduce unit energy costs to a certain extent, integrating and maintaining energy systems for multiple buildings may also lead to increased total costs, especially in the initial stage. Based on the above problems and challenges, a low-carbon building cluster integrated energy system management optimization framework based on reinforcement learning is proposed, aiming to use advanced algorithms and technologies, especially the proximal policy optimization algorithm and the framework of federated learning to achieve intelligent energy management. This framework can optimize energy distribution, reduce carbon emissions, and improve the economy and environmental protection of the system through real-time data analysis and automatic learning adjustment strategies, while providing the necessary flexibility and scalability to adapt to changing needs. Summary of the invention

[0003] In view of the above-mentioned problems, the present invention is proposed.

[0004] Therefore, the technical problem solved by the present invention is to optimize the integrated energy system management of a building cluster through a reinforcement learning algorithm to achieve low-carbon operation and maximize cost-effectiveness.

[0005] To solve the above technical problems, the present invention provides the following technical solution: a low-carbon building cluster energy management optimization method based on reinforcement learning, which comprises the following steps:

[0006] Establish a single building environment; obtain the current environmental status of a single building's energy consumption; output the actions of energy storage and energy conversion devices based on the current environmental status; obtain the action reward of a single building and the environmental status of the next time step by interacting with the environment; put the training samples into the buffer and use the proximal policy optimization algorithm to maximize the action reward; converge the comprehensive energy optimization and low-carbon management of building clusters through the federated reinforcement learning framework.

[0007] As a preferred scheme of the low-carbon building cluster energy management optimization method based on reinforcement learning described in the present invention, the single building environment is a simulation environment for comprehensive building energy management, and the system operation constraints in the simulation environment are determined, including the building energy balance equation and the operation constraint equations of the energy devices.

[0008] The building energy balance equation is expressed as,

[0009] PB t +PV t +P t eesd +P t chp =EL t +P t eesc

[0010]

[0011] Among them, PB t is the electricity purchased from the grid at time t, PV t is the distributed photovoltaic output power at time t, EL t is the electricity load of the building at time t, HL t is the heat load of the building at time t, P t chp , are the electricity output and heat output of CHP at time t, P t eesd , P t eesc are the charging power and discharging power of the electric energy storage EES respectively, are the heat release and heat absorption powers of the thermal energy storage unit TES at time t, respectively.

[0012] The operation constraint equation is expressed as,

[0013]

[0014]

[0015] and

[0016] and

[0017] Among them, η chpe and η chpq are the conversion efficiency of natural gas into electricity and heat energy in cogeneration, respectively, t eesc Represents the charging power of the energy storage device, P t eesd Indicates the discharge power of the energy storage device, represents the charging power of the thermal energy storage device, represents the discharge power of the thermal energy storage device, Represents the maximum power capacity of the electric energy storage system, Indicates the maximum power capacity of the thermal energy storage system.

[0018] As a preferred solution of the low-carbon building cluster energy management optimization method based on reinforcement learning described in the present invention, wherein: the acquisition of the current environmental state of energy consumption of a single building includes modeling the management of the integrated energy system of a single building as a Markov decision process, and acquiring the environmental state of energy consumption of the current building, expressed as,

[0019] S t =(λ t ,EL t ,HL t ,PV t ,ES t )

[0020] Among them, S t Indicates the current environmental status of building energy consumption, λ t is the time-of-use electricity price of the building at time t, EL t is the electricity load of the building at time t, HL t is the heat load of the building at time t, PV t is the distributed photovoltaic output power at time t, ES t is the energy storage state at time t.

[0021] As a preferred solution of the low-carbon building cluster energy management optimization method based on reinforcement learning described in the present invention, wherein: the output of the action of the energy storage and energy conversion device includes, according to the current action of the energy storage and energy conversion device, the control instruction of the energy storage system and the energy conversion device, wherein the action A of the energy storage system and the energy conversion performed by the intelligent agent at time t t It is expressed as,

[0022]

[0023] in, is the output action of the photovoltaic panel at time t, A is the action of the power system energy storage at time t, t CHP is the action of the cogeneration device at time t.

[0024] As a preferred solution of the low-carbon building cluster energy management optimization method based on reinforcement learning described in the present invention, the action reward for obtaining a single building includes using the carbon price in the carbon market to evaluate the cost of carbon emissions generated during the operation of the building, expressed as:

[0025] R t =-λ t PB t -λ G GB t -λ C CE t

[0026] Among them, R t represents the action reward of the building at time t, PB t is the amount of electricity purchased from the grid at time t, G is the unit price of natural gas, GB t is the total amount of natural gas purchased by the building at time t, λ C is the carbon price in the carbon market, CE t is the carbon emission of the building at time t, where carbon emission CE t Calculated from the electric carbon factor and the average carbon density of natural gas, it is expressed as,

[0027]

[0028] Among them, P t is the electricity consumption of the building at time t, F t is the node dynamic electric carbon factor, F G is the average carbon density of natural gas.

[0029] As a preferred solution of the low-carbon building cluster energy management optimization method based on reinforcement learning described in the present invention, the training samples include the current environment state, the current action, the action reward, and the environment state at the next time step.

[0030] As a preferred solution of the low-carbon building cluster energy management optimization method based on reinforcement learning described in the present invention, the comprehensive energy optimization and low-carbon management of the convergent building cluster includes obtaining the global strategy parameter θ by aggregating the updates of each intelligent agent in federated reinforcement learning, which is expressed as:

[0031]

[0032] Among them, n i is the amount of data or weight of agent i, η is the learning rate, k is the number of iterations, N is the number of buildings, represents the strategy of the agent represented by each building after local training, θ i Represents the policy parameters of agent i.

[0033] Another object of the present invention is to provide a low-carbon building cluster energy management optimization system based on reinforcement learning, which can intelligently optimize energy utilization efficiency by dynamically adjusting the operating strategies of building energy equipment in real time, thereby solving the existing problems of energy waste and excessive carbon emissions.

[0034] To solve the above technical problems, the present invention provides the following technical solutions: a low-carbon building cluster energy management optimization system based on reinforcement learning, including a building environment modeling module, an environment state acquisition module, an action output and control module, and an action reward calculation module.

[0035] The building environment modeling module is responsible for establishing a comprehensive energy management simulation environment for each building and defining system operation constraints in the environment.

[0036] The environment status acquisition module is responsible for acquiring the current energy usage status from the simulated building environment;

[0037] The action output and control module is responsible for calculating and outputting the optimal operation strategy of the energy storage and energy conversion equipment according to the current environmental status.

[0038] The action reward calculation module is responsible for calculating the reward value of each action according to the actual benefits generated by the operation of the building.

[0039] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the low-carbon building cluster energy management optimization method based on reinforcement learning as described above are implemented.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the low-carbon building cluster energy management optimization method based on reinforcement learning as described above.

[0041] Beneficial effects of the present invention: The present invention firstly adopts a proximal strategy optimization algorithm to perform low-carbon management on the integrated energy management system of a single building. The energy management method can cope with the uncertainty brought by environmental changes, and does not need to know any information of uncertainty system parameters and building thermodynamic models; at the same time, it takes into account the carbon dioxide generated by energy load, and can effectively reduce the operating cost and carbon emissions of buildings while improving the energy utilization rate and scheduling flexibility of buildings.

[0042] Federated reinforcement learning is a method that combines the characteristics of reinforcement learning and federated learning. It enables each building to independently train its own policy model based on local data. Then, only the model parameters or updates need to be sent to a central server, which aggregates these updates to improve the global model, and then sends this updated global model back to each building. It has an efficient management function for scenarios where multiple objects are jointly optimized, while also protecting the privacy and security of data. Based on the energy management model of a single building, by constructing a federated reinforcement learning framework, efficient collaboration and optimization of building clusters in the federated reinforcement learning framework can be achieved, maximizing energy efficiency while minimizing operating costs and carbon emissions. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0044] Figure 1 An overall flow chart of a low-carbon building cluster energy management optimization method based on reinforcement learning provided in the first embodiment of the present invention.

[0045] Figure 2 The overall framework diagram of the low-carbon building cluster energy management optimization system based on reinforcement learning provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0047] Example 1, reference Figure 1 , is an embodiment of the present invention, and provides a low-carbon building cluster energy management optimization method based on reinforcement learning, characterized in that:

[0048] S1: Establish a single building environment.

[0049] A single building environment is a simulation environment for integrated building energy management, and the system operation constraints in the simulation environment are determined, including the building energy balance equation and the operation constraint equations of the energy devices.

[0050] The building energy balance equation is expressed as,

[0051] PB t +PV t +P t eesd +P t chp =EL t +P t eesc

[0052]

[0053] Among them, PB t is the electricity purchased from the grid at time t, PV t is the distributed photovoltaic output power at time t, EL t is the electricity load of the building at time t, HL t is the heat load of the building at time t, P t chp , are the electricity output and heat output of CHP at time t, P t eesd , P t eesc are the charging power and discharging power of the electric energy storage EES respectively, are the heat release and heat absorption powers of the thermal energy storage unit TES at time t, respectively.

[0054] The running constraint equation is expressed as,

[0055]

[0056]

[0057] and

[0058] and

[0059] Among them, η chpe and η chpq are the conversion efficiency of natural gas into electricity and heat energy in cogeneration, respectively, t eesc Represents the charging power of the energy storage device, P t eesd Indicates the discharge power of the energy storage device, represents the charging power of the thermal energy storage device, represents the discharge power of the thermal energy storage device, Represents the maximum power capacity of the electric energy storage system, Indicates the maximum power capacity of the thermal energy storage system.

[0060] Step 1 establishes an environment that can simulate the energy consumption and generation behavior of buildings by treating each building as an independent intelligent agent. This step can flexibly perform personalized management and optimization based on the unique energy needs and characteristics of each building, ensuring that the model can accurately reflect the energy balance and equipment constraints of each building, thereby more accurately controlling energy flow and use.

[0061] S2: Obtain the current environmental status of energy consumption of a single building.

[0062] Obtaining the current environmental status of energy consumption of a single building includes modeling the integrated energy system management of a single building as a Markov decision process and obtaining the current environmental status of energy consumption of the building, expressed as,

[0063] S t =(λ t ,EL t ,HL t ,PV t ,ES t )

[0064] Among them, S t Indicates the current environmental status of building energy consumption, λ t is the time-of-use electricity price of the building at time t, EL t is the electricity load of the building at time t, HL t is the heat load of the building at time t, PV t is the distributed photovoltaic output power at time t, ES t is the energy storage state at time t.

[0065] Step 2: By modeling the building energy management problem as a Markov decision process, the state (such as electricity price, energy demand) and possible actions (such as charging, discharging) can be clearly defined at each time step. This modeling method enables systematic processing of complex multivariate factors, achieves efficient matching between state and action, and lays the foundation for subsequent strategy optimization.

[0066] S3: Output the actions of energy storage and energy conversion devices according to the current environmental status.

[0067] Outputting the actions of the energy storage and energy conversion devices includes issuing control instructions to the energy storage system and energy conversion devices according to the current actions of the energy storage and energy conversion devices, wherein the action A of the energy storage system and energy conversion performed by the intelligent agent at time t t It is expressed as,

[0068]

[0069] in, is the output action of the photovoltaic panel at time t, is the action of the power system energy storage at time t, is the action of the cogeneration device at time t.

[0070] Step three enables the intelligent agent to select the optimal action (such as energy storage and conversion) according to the current environmental status (such as grid prices and current energy usage), and then dynamically adjust the building's energy usage strategy to cope with real-time changes in energy prices and demand, maximizing economic benefits and energy efficiency.

[0071] S4: Obtain the action reward of a single building and the environment state of the next time step by interacting with the environment.

[0072] Obtaining the action reward for a single building includes using the carbon price in the carbon market to evaluate the cost of carbon emissions generated during building operation, expressed as,

[0073] R t =-λ t PB t -λ G GB t -λ C CE t

[0074] Among them, R t represents the action reward of the building at time t, PB t is the amount of electricity purchased from the grid at time t, G is the unit price of natural gas, GB t is the total amount of natural gas purchased by the building at time t, λ C is the carbon price in the carbon market, CE t is the carbon emission of the building at time t, where carbon emission CE t Calculated from the electric carbon factor and the average carbon density of natural gas, it is expressed as,

[0075]

[0076] Among them, P t is the electricity consumption of the building at time t, F t is the node dynamic electric carbon factor, F G is the average carbon density of natural gas.

[0077] Step 4 can obtain the reward value of each action (such as cost savings or reduced carbon emissions) in real time, as well as the environmental state of the next time step, so that the system can effectively feedback and adjust future strategies. It utilizes real-time data and feedback in the environment to ensure that the agent continuously optimizes its decision-making to achieve better energy efficiency and low carbon emission goals.

[0078] S5: Put the training samples into the buffer and use the proximal policy optimization algorithm to maximize the action reward.

[0079] The training samples include the current environment state, current action, action reward, and the environment state at the next time step.

[0080] Step 5 uses the proximal policy optimization algorithm (PPO) to train the optimal energy management action sequence. This step can converge to the optimal strategy more quickly and accurately, adapt to complex and dynamically changing environments, and ensure that the system can make the best decision when facing different energy usage scenarios.

[0081] S6: Converging integrated energy optimization and low-carbon management of building clusters via a federated reinforcement learning framework.

[0082] The comprehensive energy optimization and low-carbon management of convergent building clusters includes obtaining the global strategy parameter θ by aggregating the updates of each agent in federated reinforcement learning, which is expressed as:

[0083]

[0084] Among them, n i is the amount of data or weight of agent i, η is the learning rate, k is the number of iterations, N is the number of buildings, represents the strategy of the agent represented by each building after local training, θ i Represents the policy parameters of agent i.

[0085] To further explain why it is the amount of data or weight, it depends on the application scenario and optimization goal. For example, when the amount of data owned by the agents is different, these data are the basis for the agents to learn strategies in their local environments. In order to fairly integrate the contributions of these agents, the amount of data used by each agent needs to be considered in the aggregation process. A larger amount of data usually means a more reliable or more comprehensive learning result. If the amount of data is used as a weight, the central server will use the local model updates of each agent to perform a weighted average according to the amount of data when calculating the global model parameter update; however, although the amount of data of some agents may be small, their data may have higher quality or representativeness. For example, an agent's building is located in a very representative area, and its data may have a higher contribution value to the global model. Therefore, at this time, a weight can be assigned to each agent to ensure the stability of the global model update.

[0086] After each update, the agent needs to use the global policy Synchronizing local strategies to ensure that all agents are optimized towards a common goal, combined with the previous energy management strategy for the low-carbon integrated system of a single building, can achieve efficient collaboration and optimization of building clusters in the federated learning framework, maximize energy utilization efficiency, and minimize operating costs and carbon emissions.

[0087] Step 6: Through the local independent learning and global aggregation of multiple agents, we can better utilize the data characteristics and distribution differences of each building to achieve more efficient collaborative optimization, further improving the energy efficiency and operating cost reduction of the overall system. At the same time, federated learning performs calculations in a distributed manner, greatly reducing the centralized consumption of computing resources, improving the computing efficiency and response speed of the entire system, and adapting to the energy consumption management needs of large-scale building clusters.

[0088] Example 2, reference Figure 2 , which is an embodiment of the present invention, provides a system for a low-carbon building cluster energy management optimization method based on reinforcement learning, characterized in that it includes a building environment modeling module, an environment state acquisition module, an action output and control module, and an action reward calculation module.

[0089] The building environment modeling module is responsible for establishing a comprehensive energy management simulation environment for each building and defining the system operation constraints in the environment.

[0090] The environmental status acquisition module is responsible for obtaining the current energy consumption status from the simulated building environment.

[0091] The action output and control module is responsible for calculating and outputting the optimal operation strategy of energy storage and energy conversion equipment according to the current environmental status.

[0092] The action reward calculation module is responsible for calculating the reward value of each action based on the actual benefits generated by the building's operation.

[0093] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0094] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0095] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0096] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0097] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A low-carbon building cluster energy management optimization method based on reinforcement learning, characterized in that: include: Establish a single building environment; Obtain the current environmental status of energy consumption of a single building; Output the actions of energy storage and energy conversion devices according to the current environmental status; Obtain action rewards for a single building and the state of the environment at the next time step by interacting with the environment; Put the training samples into the buffer and use the proximal policy optimization algorithm to maximize the action reward; Converging integrated energy optimization and low-carbon management of building clusters via a federated reinforcement learning framework.

2. The low-carbon building cluster energy management optimization method based on reinforcement learning as claimed in claim 1, characterized in that: The single building environment is a simulation environment for building integrated energy management, and system operation constraints in the simulation environment are determined, including a building energy balance equation and an operation constraint equation for energy devices; The building energy balance equation is expressed as, PB t +PV t +P t eesd +P t chp =EL t +P t eesc Among them, PB t is the electricity purchased from the grid at time t, PV t is the distributed photovoltaic output power at time t, EL t is the electricity load of the building at time t, HL t is the heat load of the building at time t, P t chp , are the electricity output and heat output of CHP at time t, P t eesd , P t eesc are the charging power and discharging power of the electric energy storage EES respectively, are the heat release and heat absorption powers of the thermal energy storage unit TES at time t, respectively; The operation constraint equation is expressed as, Among them, η chpe and η chpq are the conversion efficiency of natural gas into electricity and heat energy in cogeneration, respectively, t eesc Represents the charging power of the energy storage device, P t eesd Indicates the discharge power of the energy storage device, represents the charging power of the thermal energy storage device, represents the discharge power of the thermal energy storage device, Represents the maximum power capacity of the electric energy storage system, Indicates the maximum power capacity of the thermal energy storage system.

3. The low-carbon building cluster energy management optimization method based on reinforcement learning as claimed in claim 2, characterized in that: The method of obtaining the current environmental state of energy consumption of a single building includes modeling the management of the integrated energy system of a single building as a Markov decision process, and obtaining the current environmental state of energy consumption of the building, expressed as: S t =(λ t ,HE t ,HL t ,PV t ,IS t ) Among them, S t Indicates the current environmental status of building energy consumption, λ t is the time-of-use electricity price of the building at time t, EL t is the electricity load of the building at time t, HL t is the heat load of the building at time t, PV t is the distributed photovoltaic output power at time t, ES t is the energy storage state at time t.

4. The low-carbon building cluster energy management optimization method based on reinforcement learning as claimed in claim 3 is characterized by: The output energy storage and energy conversion device actions include: according to the current energy storage and energy conversion device actions, the energy storage system and energy conversion device are controlled. The action A of the energy storage system and energy conversion device performed by the intelligent agent at time t t It is expressed as, in, is the output action of the photovoltaic panel at time t, is the action of the power system energy storage at time t, is the action of the cogeneration device at time t.

5. The low-carbon building cluster energy management optimization method based on reinforcement learning as claimed in claim 4, characterized in that: The action reward for obtaining a single building includes using the carbon price in the carbon market to evaluate the cost of carbon emissions generated during the operation of the building and using it as an action reward, where the action reward is expressed as: R t =-λ t PB t -l G GB t -l C CE t Among them, R t represents the action reward of the building at time t, PB t is the amount of electricity purchased from the grid at time t, G is the unit price of natural gas, GB t is the total amount of natural gas purchased by the building at time t, λ C is the carbon price in the carbon market, CE t is the carbon emission of the building at time t, where carbon emission CE t Calculated from the electric carbon factor and the average carbon density of natural gas, it is expressed as, Among them, P t is the electricity consumption of the building at time t, F t is the node dynamic electric carbon factor, F G is the average carbon density of natural gas.

6. The low-carbon building cluster energy management optimization method based on reinforcement learning as claimed in claim 5, characterized in that: The training samples include the current environment state, the current action, the action reward, and the environment state at the next time step.

7. The low-carbon building cluster energy management optimization method based on reinforcement learning as claimed in claim 6, characterized in that: The comprehensive energy optimization and low-carbon management of the convergent building cluster includes obtaining the global strategy parameter θ by aggregating the updates of each agent in federated reinforcement learning, which is expressed as: Among them, n i is the amount of data or weight of agent i, η is the learning rate, k is the number of iterations, N is the number of buildings, represents the strategy of the agent represented by each building after local training, θ i Represents the policy parameters of agent i.

8. A system using the low-carbon building cluster energy management optimization method based on reinforcement learning as claimed in any one of claims 1 to 7, characterized in that: It includes building environment modeling module, environment state acquisition module, action output and control module and action reward calculation module; The building environment modeling module is responsible for establishing a comprehensive energy management simulation environment for each building and defining system operation constraints in the environment; The environment status acquisition module is responsible for acquiring the current energy usage status from the simulated building environment; The action output and control module is responsible for calculating and outputting the optimal operation strategy of the energy storage and energy conversion equipment according to the current environmental status; The action reward calculation module is responsible for calculating the reward value of each action according to the actual benefits generated by the operation of the building.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the low-carbon building cluster energy management optimization method based on reinforcement learning described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the low-carbon building cluster energy management optimization method based on reinforcement learning as described in any one of claims 1 to 7 are implemented.

Images