Optimal control method for residential integrated energy system based on multi-agent reinforcement learning

Abstract

The present invention provides a residential integrated energy system optimization control method based on multi-agent reinforcement learning, which belongs to the field of building environment control technology. The method obtains relevant data from the actual operation of the building, and uses the collected data as training sample data to preprocess the data; constructs a multi-agent environmental model to simulate and constrain the operation of the system; constructs a multi-agent IPPO model, uses the IPPO algorithm to train through the policy gradient method, and generates the optimal control strategy of the building energy system after multiple iterations; uses the trained model to achieve efficient energy allocation and reduce operating costs. The present invention adopts a model-based and data-driven method, and overcomes the limitations of traditional model-based control methods by considering the nonlinear characteristics of heat pump heating and fuel cell power generation behaviors, and can accurately describe the complex dynamic behavior of the system and make fast and efficient decisions in highly complex scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of building environment control, and in particular to an optimization control method for a residential integrated energy system based on multi-agent reinforcement learning. Background Art

[0002] At present, the carbon emissions related to the construction and operation of buildings in my country during their entire life cycle can account for more than 40% of the total carbon emissions of the whole society. With the improvement of economic income and electrification level, the energy consumption of building operation in my country will further increase. Promoting the clean and efficient use of energy and accelerating the planning and construction of a new energy system cannot be separated from the support of the construction sector. Since 2010, the heat consumption of the national building sector has increased by 75%, and the carbon emissions related to heat consumption in the building sector in my country account for about 25% of the world. my country has the largest scale of urban centralized heating in the world, and the main heat source is the cogeneration and boiler system based on fossil fuels. There is great potential for promoting the decarbonization of distributed renewable energy and electric heat pumps in the building regional energy system. A feasible way to decarbonize buildings is to adopt renewable energy and electrification. With the continuous improvement of the integration of distributed energy, the centralized energy system is transitioning to a decentralized energy system. Renewable energy generation is greatly affected by weather conditions and has the characteristics of intermittent. The integration of various distributed energy sources further increases the complexity of the energy system and also poses challenges to real-time energy balance. The development of energy storage systems and coordinated demand energy management have great potential in solving energy mismatches, achieving energy conservation and cost savings.

[0003] Residential integrated energy systems can integrate and dispatch multiple energy flows, including electricity, natural gas, and thermal energy. However, the integration of multiple energy sources increases the complexity of managing different energy flows. In addition, residential integrated energy systems are composed of various components, often involving nonlinear constraints and requiring complex modeling. Model-based nonlinear optimization usually requires a combination of advanced optimization algorithms, specific problem modeling techniques, and heuristic optimization methods. Traditional model-based optimization methods require accurate system modeling, and this process requires a lot of expertise and is time-consuming, so this method is difficult to promote. Nonlinear models can more accurately describe the complex dynamic behavior of the system. However, solving these nonlinear fitting relationships in model-based algorithms is very complex, which further increases the difficulty of system modeling and optimization. In order to solve the nonlinearity and implicit operating constraints of multi-energy systems, linearization processes have been widely introduced, but simplifying models to improve computational efficiency may reduce the reliability of optimized energy systems. Summary of the invention

[0004] The purpose of the present invention is to provide a residential integrated energy system optimization control method based on multi-agent reinforcement learning to make up for the shortcomings of the prior art.

[0005] The goal of this invention is to minimize household energy costs while meeting the energy balance in terms of electricity prices and on-site photovoltaic power generation. A model-based and data-driven approach is adopted to overcome the limitations of traditional model-based control methods by considering the nonlinear characteristics of heat pump heating and fuel cell power generation behavior, accurately describe the complex dynamic behavior of the system, and make fast and efficient decisions in highly complex scenarios.

[0006] To achieve the above object, the present invention is achieved through the following technical solutions:

[0007] A residential integrated energy system optimization control method based on multi-agent reinforcement learning, the method comprising the following steps:

[0008] S1: Obtain relevant data from the actual operation of the building, and use the collected data as training sample data for data preprocessing;

[0009] S2: Construct a multi-agent environment model to simulate and constrain the operation of the system; in this environment, the system imposes constraints on the actions of each agent to ensure that the operation of the system complies with physical and practical conditions;

[0010] S3: Construct a multi-agent IPPO model, clarify the state variables and action variables to be regulated, and optimize the strategies of each agent by designing a reward and punishment mechanism;

[0011] S4: Each agent independently maintains and updates its strategy, and uses the IPPO algorithm to train through the policy gradient method to generate the optimal control strategy of the building energy system after multiple iterations;

[0012] S5: Use the trained multi-agent IPPO model to achieve efficient energy allocation and reduce operating costs.

[0013] Furthermore, in S1, the collected relevant data include: photovoltaic power generation, user power load, hot water demand, real-time electricity price, month information, hour information, outdoor temperature, and heat pump performance coefficient; in order to facilitate the learning of the reinforcement learning model, the time data is converted into sine and cosine form.

[0014] Furthermore, in S1, a preprocessing method for the collected data is:

[0015] (1) Filling missing data values: During the data collection process, the temperature, photovoltaic power generation and other information at some time points may be lost or missing due to various reasons such as equipment failure. Forward / backward filling is used to fill in this part of the data;

[0016] (2) Treatment of outliers: Outliers are data that deviate from the majority of data in a data set. There are many methods to treat outliers, such as direct deletion, mean or other statistical replacement. Here, they are regarded as missing values ​​for filling. In order to identify outliers, select The standard deviation method is used to confirm the reasonable range of the collected data. ,in represents the mean value of the data set, Indicates standard deviation, and selects data out of range for replacement and filling.

[0017] Furthermore, in S2:

[0018] S2-1: First, the goal is to minimize household energy costs while meeting the energy balance in terms of electricity prices and on-site photovoltaic power generation. The energy management strategy developed focuses on regulating the charging and discharging power of the hydrogen storage equipment and the hot water storage tank, as well as the power generation of the fuel cell and heat pump system; the optimization target can be described by formula (1).

[0019] ;

[0020] In the formula, For household energy costs, is the electricity and gas purchase cost at time step t; T is the time step of a single scheduling cycle.

[0021] S2-2: Construct a fuel cell flexibility optimization model; the operation of the fuel cell system generates electricity and hot water at the same time, and the electrical efficiency is an indicator to measure its power generation performance; the nonlinear relationship between the fuel cell load ratio and the electrical efficiency is described by formula (2):

[0022] ;

[0023] in, is the fuel cell electrical efficiency at time t, , are the power generation power of the fuel cell at time step t and the maximum rated output power of the fuel cell, in kW; a, b, c, d are the correlation coefficients of the electrical efficiency of the fuel cell.

[0024] The power output of a fuel cell depends on the maximum power generation:

[0025] ;

[0026] The operation of the fuel cell should always satisfy the power balance of the system, as shown in equations (4)-(5):

[0027] ;

[0028] ;

[0029] In the formula, represents the power generated by hydrogen, represents the natural gas power generation capacity, is the lower calorific value of natural gas, Indicates the volume of natural gas;

[0030] S2-3: Construct equipment flexibility optimization model. Hydrogen storage equipment is used to store hydrogen produced by electrolyzers and provide fuel for fuel cells. The energy storage system can store excess electricity during peak power generation and release it during peak demand, which can better cope with electricity price volatility and maximize the economic benefits of the energy system.

[0031] In the electrolytic cell, electricity is used to produce hydrogen through electrolysis and then stored. The electricity-to-hydrogen relationship can be expressed as formula (6):

[0032] ;

[0033] In the formula, Hydrogen production power of electrolytic cell; is the power consumption of the electrolytic cell, is the electricity-to-hydrogen efficiency.

[0034] The hydrogen storage device is used to store the hydrogen produced by the electrolyzer and provide fuel for the fuel cell. When the photovoltaic power generation exceeds the load, the hydrogen storage device is charged; otherwise, it is discharged. The hydrogen storage device is limited by the maximum power during operation. Since the efficiency of the hydrogen storage device is relatively low, losses will be generated during the charging and discharging process. Its operating constraints are as shown in equations (7)-(9):

[0035] ;

[0036] ;

[0037] ;

[0038] in, The real-time electric power of the hydrogen tank, Charging power for hydrogen tanks, is the discharge power of the hydrogen tank, , They are the maximum discharge power and maximum charging power of the hydrogen tank, is the photovoltaic power generation power; is the capacity of the hydrogen tank.

[0039] S2-4: Construct an air source heat pump flexibility optimization model. Heat storage tanks have been widely used in residential hot water systems. When the heat recovered from the heat pump and fuel cell is sufficient to meet the user's heat demand, the hot water can be stored in the heat storage tank for backup. The operating constraints of the heat storage tank are shown in equations (10)-(12):

[0040] ;

[0041] ;

[0042] ;

[0043] in, is the real-time thermal power of the water storage tank, To charge the hot water tank with heat power, The heat release power of the hot water storage tank, , They are the maximum heat dissipation power and the maximum heat charging power of the hot water storage tank respectively; is the capacity of the hot water tank, is the charging and discharging efficiency of the hot water storage tank, represent the waste heat recovery rate and the thermal efficiency of the fuel cell respectively.

[0044] The operation of the heat pump should always satisfy the power balance of the system, which is expressed as follows:

[0045] ;

[0046] In the formula, The electrical power consumed by the heat pump; The amount imported into the power grid; Photovoltaic energy is consumed for heat pumps.

[0047] Heat energy generated by air source heat pump It can be determined by formula (15):

[0048] ;

[0049] The coefficient of performance (COP) is defined as the ratio of the heat generated to the electricity consumed. COP has a nonlinear relationship with the outdoor air temperature T and is calculated by equation (16):

[0050] ;

[0051] in, is the correlation coefficient of the COP of the air source heat pump.

[0052] Furthermore, the S3 specifically includes:

[0053] Neural network construction. Select a multi-agent parallel training architecture and use the IPPO algorithm in the XuanCe library to build the input and output model of the system. Each agent uses an independent strategy network and value network, with the input being the state information of the environment and the output being the control strategy.

[0054] Furthermore, in the above neural network model, a multilayer perceptron (MLP) is used to model the strategy. The specific configuration of the hidden layer and the output layer depends on the default settings of the IPPO algorithm in xuance. The MLP is configured with two hidden layers, each containing 128 nodes, and the activation function is ReLU. The input layer parameters include the photovoltaic power generation at the current moment. , excluding power demand for heat pumps , hot water demand , outdoor temperature information , Seasonal information , half-hour time information , electricity price information , heat pump performance coefficient The input features are 8 dimensions, t represents the current moment; the control strategies of various energy devices at the next moment are output, such as the fuel cell power , heat pump power and the power of hydrogen storage equipment energy storage system The policy network uses the ReLU activation function, and the output layer is a Sigmoid function that gradually converges and generates a dynamic estimate of the policy. The loss of the policy network is based on policy gradient optimization, and the loss estimate error of the value network is obtained through error calculation. In order to improve the convergence speed and model performance, the optimizer uses the Adam gradient descent algorithm to find the optimal solution.

[0055] Furthermore, the multi-agent IPPO model specifically models the control method operation optimization problem as a Markov decision process:

[0056] (1) State observation space Each agent is assigned an independent state space. Describes the state information characteristics of each agent at time t. , these characteristics can be divided into photovoltaic power generation , excluding power demand for heat pumps , hot water demand , outdoor temperature information , Seasonal information , half-hour time information , electricity price information , heat pump performance coefficient . These state spaces are independent but have the same dimensional structure.

[0057] (2) Setting of action space A. Each agent corresponds to a device and sets the corresponding action range according to the function and constraints of the device. The value range for hydrogen storage device control is [-1, 1], and it is scaled by multiplying the difference between charge and discharge. Negative values ​​indicate discharge of hydrogen storage device, and positive values ​​indicate charge. The calculation of charge and discharge power of hydrogen storage device is subject to strict constraint control. The action space of fuel cell and heat pump has a value range of [0, 1]. If the action is 0, it means the device is not working, otherwise it is working.

[0058] (3) Setting of reward function R. This paper adopts a shared reward function design, that is, all agents receive the same reward regardless of the performance of a single agent, which directs actions to a common goal. This can promote collaboration by defining the overall state and the behavior of each agent. Considering the formulated optimization problem, the developed reward function consists of four parts: (1) Reward for economic cost , (2 Power supply execution action to obtain rewards , (3 Rewards for heating execution actions , (4 fuel cell electrical efficiency bonus , the reward function The definition is as follows:

[0059] ;

[0060] Rewards with economic costs , considering the cost of purchasing electricity and gas as shown in formula (18):

[0061] ;

[0062] In the formula, are the electricity purchased and sold from the public grid, represents the electricity purchase price and electricity sales price, For natural gas prices.

[0063] The heat storage and heat release is based on the amount of waste heat recovered from the fuel cell, the action of the heat pump, the demand for hot water and the remaining heat storage space. Update accordingly based on the operation of the hydrogen storage equipment and the remaining space. In order to ensure the normal operation of the system, It is constrained to prevent overcharge and over discharge. The constraint expression is as follows:

[0064] ;

[0065] Optimizing electrical efficiency helps improve energy efficiency;

[0066] .

[0067] Furthermore, the training of the multi-agent IPPO model described in S4: each agent continuously updates the weights of the policy network and the value network according to the feedback from the environment. The agent gradually learns the optimal strategy through interaction with the environment to maximize the sum of the reward function. During the training process, the cumulative reward curve is used to monitor the learning progress of the model to ensure that the agent strategy converges and continuously improves performance.

[0068] Assume that the number of selected generations is M, the maximum length of the sampled sequence is T, and the learning rate is , the cropping factor is , the discount factor is , the entropy coefficient is . The state space is S and the action space is A:

[0069] Step 1: Initialize the policy network and value network of each agent and set the initial weights and ;

[0070] Step 2: Create a parallel environment and assign initial states to each agent ;

[0071] Step 3: Loop through each event ;

[0072] Step 4: Initialize the state of each agent , and input into the neural network to extract state features;

[0073] Step 5: Loop through each step of each agent. ;

[0074] Step 6: Based on the current status , policy network Generate Action ;

[0075] Step 7: Execute the action , the environment returns an immediate reward and the next moment state ;

[0076] Step 8: Calculate the advantage function ;

[0077] Step 9: Calculate the strategy loss ;

[0078] Step 10: Calculate the value network loss ;

[0079] Step 11: Calculate entropy loss ;

[0080] Step 12: Combined total loss function: ;

[0081] Step 13: Update the policy network using the Adam optimizer and value network ;

[0082] Step 14: Update Status , and record the cumulative rewards of each round;

[0083] Step 15: Repeat the above steps M times until you learn a strategy that maximizes the cumulative reward value R .

[0084] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0085] This invention proposes an optimization control method for a residential integrated energy system based on multi-agent reinforcement learning, which overcomes the limitations of traditional model-based control methods, can accurately describe the complex dynamic behavior of the system, and achieve efficient collaboration of multiple energy flows. This invention achieves real-time dynamic decision-making, significantly reduces household energy costs, improves energy utilization efficiency, has economic and environmental benefits, and has obvious advantages over traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a diagram of the basic principles of reinforcement learning, (a) is single-agent reinforcement learning, and (b) is multi-agent reinforcement learning.

[0087] Figure 2 It is a framework structure diagram of the present invention.

[0088] Figure 3 It is a structural diagram of the residential integrated energy system.

[0089] Figure 4 It is a graph showing the relationship between the fuel cell electrical efficiency and the load ratio.

[0090] Figure 5 This is a graph showing the relationship between heat pump COP and outdoor air temperature.

[0091] Figure 6It is the flow chart of IPPO algorithm.

[0092] Figure 7 It is the convergence curve diagram of IPPO model.

[0093] Figure 8 This is the IPPO model optimization result diagram, (a) is summer, (b) is transition season, and (c) is winter. DETAILED DESCRIPTION

[0094] The technical solution of the present invention is further described below in conjunction with embodiments and drawings.

[0095] The reinforcement learning process is expressed as a Markov decision process, which can handle complex nonlinear features well. It can learn complex decision strategies through trial and error and adapt to different environments without detailed physical models, which greatly simplifies the complexity of modeling (the principle of reinforcement learning is as follows Figure 1 ). Model-free reinforcement learning methods require a large amount of training data to achieve good performance in building energy system control. The quality and quantity of training data have a significant impact on the results. Seasonal variations in multiple energy loads in a building, dynamic changes in system components, and changes in renewable energy generation can greatly affect the computational efficiency of reinforcement learning methods in multi-building energy system optimization. Compared to model-free methods, model-based methods can achieve higher sample efficiency by leveraging a dynamic model of the environment to learn effective policies with fewer interactions with the environment.

[0096] Embodiment 1:

[0097] A residential integrated energy system optimization control method based on multi-agent reinforcement learning, such as Figure 2 As shown, the method comprises the following steps:

[0098] (1) In this embodiment, the data comes from a detached house in the Kyushu smart community in Japan. The collected relevant data include: photovoltaic power generation, user power load, hot water demand, real-time electricity price, month information, hour information, outdoor temperature, and heat pump performance coefficient. In order to facilitate the learning of the reinforcement learning model, the time data is converted into sine and cosine forms.

[0099] (2) Construct a multi-agent environment model, such as Figure 3 As shown, the operation of the system is simulated and constrained.

[0100] First, the goal is to minimize household energy costs while meeting the energy balance in terms of electricity prices and on-site photovoltaic power generation. The energy management strategy developed focuses on regulating the charging and discharging power of the hydrogen storage equipment and the hot water storage tank, as well as the power generation of the fuel cell and heat pump system. The optimization goal can be described by formula (1):

[0101] ;

[0102] In the formula, For household energy costs, is the electricity and gas purchase cost at time step t; T is the time step of a single scheduling cycle.

[0103] Secondly, in this environment, the system imposes constraints on the actions of each agent to ensure that the system operates in accordance with physical and practical conditions;

[0104] Construct a fuel cell flexibility optimization model. The operation of the fuel cell system generates electricity and hot water at the same time. The electrical efficiency is an indicator to measure its power generation performance (the relationship between the fuel cell electrical efficiency and the load ratio is as follows Figure 4 The nonlinear relationship between fuel cell load ratio and electrical efficiency is described by formula (2):

[0105] ;

[0106] in, is the fuel cell electrical efficiency at time t, , are the power generation power of the fuel cell at time step t, kW, and the maximum rated output power of the fuel cell, kW; a, b, c, d are the correlation coefficients of the electrical efficiency of the fuel cell.

[0107] The power output of a fuel cell depends on the maximum power generation:

[0108] ;

[0109] The operation of the fuel cell should always satisfy the power balance of the system, as shown in equations (4)-(5):

[0110] ;

[0111] ;

[0112] In the formula, represents the power generated by hydrogen, represents the natural gas power generation capacity, is the lower calorific value of natural gas, Indicates the volume of natural gas;

[0113] Build an equipment flexibility optimization model. Hydrogen storage equipment is used to store hydrogen produced by electrolyzers and provide fuel for fuel cells. The energy storage system can store excess electricity during peak power generation and release it during peak demand, which can better cope with electricity price volatility and maximize the economic benefits of the energy system.

[0114] In the electrolytic cell, electricity is used to produce hydrogen through electrolysis and then stored. The electricity-to-hydrogen relationship can be expressed as formula (6):

[0115] ;

[0116] In the formula, Hydrogen production power of electrolytic cell; is the power consumption of the electrolytic cell, is the electricity-to-hydrogen efficiency.

[0117] The hydrogen storage device is used to store the hydrogen produced by the electrolyzer and provide fuel for the fuel cell. When the photovoltaic power generation exceeds the load, the hydrogen storage device is charged; otherwise, it is discharged. The hydrogen storage device is limited by the maximum power during operation. Since the efficiency of the hydrogen storage device is relatively low, losses will be generated during the charging and discharging process. Its operating constraints are as shown in equations (7)-(9):

[0118] ;

[0119] ;

[0120] ;

[0121] in, The real-time electric power of the hydrogen tank, Charging power for hydrogen tanks, is the discharge power of the hydrogen tank, , They are the maximum discharge power and maximum charging power of the hydrogen tank, is the photovoltaic power generation power; is the capacity of the hydrogen tank.

[0122] Constructing an air source heat pump flexibility optimization model, the hot water storage tank has been widely used in residential hot water systems. In this embodiment, the hot water storage tank is used to store hot water generated by heat pump and fuel cell waste heat recovery. When the heat recovered from the heat pump and fuel cell is sufficient to meet the user's heat demand, the hot water can be stored in the hot water storage tank for standby use. The operating constraints of the hot water storage tank are shown in equations (10)-(12):

[0123] ;

[0124] ;

[0125] ;

[0126] in, is the real-time thermal power of the water storage tank, To charge the hot water tank with heat power, The heat release power of the hot water storage tank, , They are the maximum heat dissipation power and the maximum heat charging power of the hot water storage tank respectively; is the capacity of the hot water tank, is the charging and discharging efficiency of the hot water storage tank, represent the waste heat recovery rate and the thermal efficiency of the fuel cell respectively.

[0127] The operation of the heat pump should always satisfy the power balance of the system, which is expressed as follows:

[0128] ;

[0129] In the formula, The electrical power consumed by the heat pump; The amount imported into the power grid; Photovoltaic energy is consumed for heat pumps.

[0130] Heat energy generated by air source heat pump It can be determined by formula (15):

[0131] ;

[0132] The coefficient of performance (COP) is defined as the ratio of the heat generated to the electricity consumed. COP has a nonlinear relationship with the outdoor air temperature T and is calculated by equation (16):

[0133] ;

[0134] in, is the correlation coefficient of the COP of the air source heat pump.

[0135] (3) For the construction of neural network. A multi-agent parallel training architecture is selected, and the input and output model of the system is established through the IPPO algorithm in the XuanCe library. Each agent uses an independent strategy network and value network, with the input being the state information of the environment and the output being the control strategy.

[0136] Furthermore, in the above neural network model, a multilayer perceptron (MLP) is used to model the strategy. The specific configuration of the hidden layer and the output layer depends on the default settings of the IPPO algorithm in xuance. The MLP is configured with two hidden layers, each containing 128 nodes, and the activation function is ReLU. The input layer parameters include the photovoltaic power generation at the current moment. , excluding power demand for heat pumps , hot water demand , outdoor temperature information , Seasonal information , half-hour time information , electricity price information , heat pump performance coefficient The input features are 8 dimensions, t represents the current moment; the control strategies of various energy devices at the next moment are output, such as the fuel cell power , heat pump power and the power of hydrogen storage equipment energy storage system The policy network uses the ReLU activation function, and the output is a Sigmoid function that gradually converges and generates a dynamic estimate of the policy. The loss of the policy network is based on policy gradient optimization, and the loss estimate error of the value network is obtained through error calculation. In order to improve the convergence speed and model performance, the optimizer uses the Adam gradient descent algorithm to find the optimal solution.

[0137] Secondly, a multi-agent IPPO model is constructed, the control system optimization operation problem is modeled as a Markov decision process, and the strategy of each agent is optimized by designing a reward and punishment mechanism.

[0138] Each agent is assigned an independent state space. Describes the state information characteristics of each agent at time t. , these characteristics can be divided into photovoltaic power generation , excluding power demand for heat pumps , hot water demand , outdoor temperature information , Seasonal information , half-hour time information , electricity price information , heat pump performance coefficient . These state spaces are independent but have the same dimensional structure.

[0139] Each agent corresponds to a device and sets the corresponding action range according to the function and constraints of the device. The value range for hydrogen storage device control is [-1, 1], and it is scaled by multiplying the difference between charge and discharge. Negative values ​​indicate discharge of hydrogen storage device, and positive values ​​indicate charge. The calculation of charge and discharge power of hydrogen storage device is subject to strict constraint control. The action space of fuel cell and heat pump has a value range of [0, 1]. If the action is 0, it means the device is not working, otherwise it is working.

[0140] Setting of reward function R, the present invention adopts a shared reward function design, that is, all agents receive the same reward regardless of the performance of a single agent, which directs actions to a common goal. This can promote collaboration by defining the overall state and the behavior of each agent. Considering the formulated optimization problem, the developed reward function consists of four parts: (1) Reward with economic cost , (2 Power supply execution action to obtain rewards , (3 Rewards for heating execution actions , (4 fuel cell electrical efficiency bonus , the reward function The definition is as follows:

[0141] ;

[0142] Rewards with economic costs , considering the cost of purchasing electricity and gas as shown in formula (18):

[0143] ;

[0144] In the formula, are the electricity purchased and sold from the public grid, represents the electricity purchase price and electricity sales price, For natural gas prices.

[0145] The heat storage and heat release is based on the amount of waste heat recovered from the fuel cell, the action of the heat pump, the demand for hot water and the remaining heat storage space. Update accordingly based on the operation of the hydrogen storage equipment and the remaining space. In order to ensure the normal operation of the system, It is constrained to prevent overcharge and over discharge. The constraint expression is as follows:

[0146] ;

[0147] Optimizing electrical efficiency helps improve energy efficiency:

[0148] .

[0149] Finally, using the IPPO algorithm (such as Figure 6 As shown in Figure 2, the agent is trained through the policy gradient method. The agent gradually learns the optimal strategy through interaction with the environment to maximize the sum of the reward functions.

[0150] During the training process, the cumulative reward curve is used to monitor the learning progress of the model to ensure that the agent strategy converges and continuously improves performance ( Figure 7 Demonstrates IPPO convergence).

[0151] After the model training is completed, the model is evaluated using the test environment. The evaluation process includes multiple test rounds to measure the stability of the model. Focus on observing whether the agent can make reasonable decisions in an unseen environment, and evaluate the model performance based on the average and standard deviation of the accumulated rewards. If the model performance is not ideal during the evaluation process, the model can be tuned by adjusting the hyperparameters. For example, the learning rate, the number of layers of the neural network, the number of neurons in the hidden layer, the exploration rate, etc. can be adjusted to further improve the model performance.

[0152] Focus on observing whether the agent can make reasonable decisions in an unseen environment, and evaluate the model performance based on the average and standard deviation of the accumulated rewards. If the model performance is not ideal during the evaluation process, you can adjust the model by adjusting the hyperparameters. For example, you can adjust the learning rate, the number of neural network layers, the number of neurons in the hidden layer, the exploration rate, etc. to further improve the model performance.

[0153] Figure 8 The strategic scheduling results of the optimization model in typical weeks in different quarters are shown, which shows how the optimization model dynamically adjusts energy scheduling according to the external environment and electricity prices in different seasons to achieve the goal of reducing operating costs and improving energy utilization efficiency.

[0154] On the basis of the above embodiments, the present invention continues to describe in detail the technical features involved therein and the functions and roles played by the technical features in the present invention, so as to help technicians in this field fully understand the technical solution of the present invention and reproduce it.

[0155] Finally, although this specification is described according to implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A residential integrated energy system optimization control method based on multi-agent reinforcement learning, characterized in that: The method comprises the following steps: S1: Obtain relevant data from the actual operation of the building, and use the collected data as training sample data for data preprocessing; S2: Construct a multi-agent environment model to simulate and constrain the operation of the system; in this environment, the system imposes constraints on the actions of each agent to ensure that the operation of the system complies with physical and practical conditions; S3: Construct a multi-agent IPPO model, clarify the state variables and action variables of the control, and optimize the strategy of each agent by designing a reward and punishment mechanism; the multi-agent IPPO model specifically models the control method operation optimization problem as a Markov decision process: (1) Setting of state observation space S: Each agent is assigned an independent state space. Describes the state information characteristics of each agent at time t; These characteristics are divided into photovoltaic power generation PV t , excluding the power demand of heat pump Hot water demand Outdoor temperature information Season Information t , half-hour time information hour t , electricity price information price t , heat pump performance coefficient COP t ; (2) Setting of action space A: Each agent corresponds to a device and sets the corresponding action range according to the function and constraints of the device. The value range for hydrogen storage device control is [-1, 1], and it is scaled by multiplying the difference between charge and discharge. Negative values ​​indicate discharge of hydrogen storage devices, and positive values ​​indicate charging. The action space range for fuel cells and heat pumps is [0, 1]. If the action is 0, it means the device is not working, otherwise it is working. (3) The setting of the reward function R promotes collaboration by defining the overall state and the behavior of each agent; considering the formulated optimization problem, the developed reward function consists of four parts: the reward of economic cost Rewards for powering up actions Rewards for performing heating actions Rewards for fuel cell electrical efficiency Reward function R t The definition is as follows: Rewards with economic costs Considering the cost of purchasing electricity and gas as shown in formula (18): In the formula, are the electricity purchased and sold from the public grid, represents the electricity purchase price and electricity sales price, k is the natural gas price; The heat storage and heat release is based on the amount of waste heat recovered from the fuel cell, the action of the heat pump, the demand for hot water and the remaining heat storage space. According to the operation of hydrogen storage equipment and the remaining space, corresponding updates are carried out. In order to ensure the normal operation of the system, It is constrained to prevent overcharge and over discharge. The constraint expression is as follows: Optimizing electrical efficiency helps improve energy efficiency: S4: Each agent independently maintains and updates its strategy, and uses the IPPO algorithm to train through the policy gradient method to generate the optimal control strategy of the building energy system after multiple iterations; S5: Use the trained multi-agent IPPO model to achieve efficient energy allocation and reduce operating costs.

2. The method for optimizing and controlling a residential integrated energy system based on multi-agent reinforcement learning according to claim 1, characterized in that: In S1, the collected relevant data include: photovoltaic power generation, user power load, hot water demand, real-time electricity price, month information, hour information, outdoor temperature, and heat pump performance coefficient.

3. The residential integrated energy system optimization control method based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: In S1, the data preprocessing method is: filling in missing data values ​​or outliers.

4. The method for optimizing and controlling a residential integrated energy system based on multi-agent reinforcement learning according to claim 1, characterized in that: In S2: S2-1: First, the goal is to minimize household energy costs while meeting the energy balance in terms of electricity prices and on-site photovoltaic power generation. The optimization goal is described by formula (1): In the formula, F C For home energy costs, is the electricity and gas purchase cost at time step t; T is the time step of a single scheduling cycle; S2-2: Construct a fuel cell flexibility optimization model. The nonlinear relationship between the fuel cell load ratio and electrical efficiency is described by formula (2): in, is the fuel cell electrical efficiency at time t, P max，fc are the power generation power of the fuel cell at time step t and the maximum rated output power of the fuel cell, in kW; a, b, c, d are the correlation coefficients of the fuel cell electrical efficiency; The power output of a fuel cell depends on the maximum power generation: The operation of the fuel cell should always satisfy the power balance of the system, as shown in equations (4)-(5): In the formula, represents the power generated by hydrogen, Indicates natural gas power generation, LHV Gas is the lower calorific value of natural gas, V Gas Indicates the volume of natural gas; S2-3: Construct an equipment flexibility optimization model. In the electrolytic cell, electricity is used to produce hydrogen through electrolysis and store it. The electricity-to-hydrogen relationship is expressed as formula (6): In the formula, Hydrogen production power of electrolytic cell; is the power consumption of the electrolytic cell, ∑ el is the power-to-hydrogen efficiency; Hydrogen storage equipment will generate losses during the charging and discharging process, and its operation constraints are as shown in equations (7)-(9): in, The real-time electric power of the hydrogen tank, Charging power for hydrogen tanks, is the discharge power of the hydrogen tank, P MaxdischargeHT , P MaxchargeHT They are the maximum discharge power and maximum charging power of the hydrogen tank, is the photovoltaic power generation power; is the hydrogen tank capacity; S2-4: Construct an air source heat pump flexibility optimization model. The operating constraints of the hot water storage tank are shown in equations (10)-(12): in, is the real-time thermal power of the water storage tank, To charge the hot water tank with heat power, is the heat release power of the hot water storage tank, Q MaxdischargeHWT , Q MaxchargeHWT They are the maximum heat dissipation power and the maximum heat charging power of the hot water storage tank respectively; is the capacity of the hot water tank, η HWT is the charging and discharging efficiency of the hot water storage tank, η Thermal represent the waste heat recovery rate and the thermal efficiency of the fuel cell respectively; The operation of the heat pump should always satisfy the power balance of the system, which is expressed as follows: In the formula, The electrical power consumed by the heat pump; The amount imported into the power grid; PV consumption for heat pumps; Heat energy generated by air source heat pump Determine by formula (15): The coefficient of performance (COP) is defined as the ratio of the heat generated to the power consumed. The COP has a nonlinear relationship with the outdoor air temperature T and is calculated by formula (16): Among them, α, β, and γ are the correlation coefficients of the COP of the air source heat pump.

5. The residential integrated energy system optimization control method based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: In the S3, a multi-agent parallel training architecture is selected, and the input and output model of the system is established through the IPPO algorithm in the XuanCe library; each agent uses an independent strategy network and value network, the input is the state information of the environment, and the output is the control strategy.

6. The method for optimizing and controlling a residential integrated energy system based on multi-agent reinforcement learning according to claim 1, characterized in that: In the multi-agent IPPO model, a multi-layer perceptron MLP is used to model the strategy. The specific configuration of the hidden layer and the output layer depends on the default settings of the IPPO algorithm in xuance. The MLP is configured with two hidden layers, each containing 128 nodes, and the activation function is ReLU. The input layer parameters include the photovoltaic power generation PV at the current moment. t , excluding the power demand of heat pump Hot water demand Outdoor temperature information Season Information t , half-hour time information hour t , electricity price information price t , heat pump performance coefficient COP t The input features are 8 dimensions, t represents the current moment; the control strategies of various energy devices at the next moment are output, such as the fuel cell power Heat pump power and the power of hydrogen storage equipment energy storage system The policy network uses the ReLU activation function, and the output layer is a Sigmoid function that gradually converges and generates a dynamic valuation of the policy. The loss of the policy network is based on policy gradient optimization, and the loss estimation error of the value network is obtained through error calculation. In order to improve the convergence speed and model performance, the optimizer uses the Adam gradient descent algorithm to find the optimal solution.

7. The residential integrated energy system optimization control method based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: The training of the multi-agent IPPO model described in S4 is as follows: each agent continuously updates the weights of the policy network and the value network according to environmental feedback; the agent gradually learns the optimal strategy through interaction with the environment to maximize the sum of the reward function. During the training process, the cumulative reward curve is used to monitor the learning progress of the model to ensure that the agent strategy converges and continuously improves performance.

8. The method for optimizing and controlling a residential integrated energy system based on multi-agent reinforcement learning according to claim 7, characterized in that: Assume that the number of selected generations is M, the maximum length of the sampled sequence is T, the learning rate is α, the clipping coefficient is ∈, the discount factor is γ, the entropy coefficient is β, the state space is S, and the action space is A; Step 1: Initialize the policy network and value network of each agent, and set the initial weights θ and φ; Step 2: Create a parallel environment and assign an initial state s0 to each agent; Step 3: Loop through each event episode = 1, 2, ..., T; Step 4: Initialize the state s1 of each agent and input it into the neural network to extract state features; Step 5: Loop through each step of each agent, t=1,2,...,T; Step 6: Based on the current state s t , policy network π θ (a t |s t ) Generate action a t ; Step 7: Execute action a t , the environment returns the immediate reward r t and the next state s t+1 ; Step 8: Calculate the advantage function A t =r t +γV φ (s t+1 )-V φ (s t ); Step 9: Calculate the policy loss L clip (θ) = E t [min(r t (θ)A t ,clip(r t (θ), 1-∈, 1+∈)A t )]; Step 10: Calculate the value network loss L value (φ)=E t [(V φ (s t )-R t ) 2 ]; Step 11: Calculate the entropy loss L entropy (θ) = βH(π θ (s t )); Step 12: Combined total loss function: L(θ, φ) = L clip (θ)+c1Lv alue (φ)-c2L entropy *(θ); Step 13: Use Adam optimizer to update the policy network θ and value network φ; Step 14: Update status s t+1 =s t , and record the cumulative rewards of each round; Step 15: Repeat the above steps M times until a strategy π that maximizes the cumulative reward value R is learned.

Images