A multi-building energy management method and system combining robust Markov and SAC algorithms

Abstract

The present invention discloses a multi-building energy management method and system combining robust Markov and SAC algorithms, including: obtaining building energy consumption, energy storage status and environmental information to form a multi-building environment simulation system; modeling the multi-building environment, and optimizing the energy management strategy by combining robust SAC (soft actor critic algorithm); processing observations, rewards and actions through a data processing module, and inputting them into a reinforcement learning agent module for training; the reinforcement learning agent module includes an improved neural network model, an experience replay buffer and an optimized strategy algorithm, and optimizes the energy management strategy through training; the invention provides a method for realizing intelligent dynamic energy management in a complex and changeable environment by combining robust Markov and SAC algorithms, effectively improving the overall energy efficiency and stability of the system, and introducing external data sources such as meteorological data and market electricity prices to improve the accuracy and economy of decision-making.

Description

Technical Field

[0001] The present invention belongs to the field of multi-building environment intelligent energy management systems, and in particular relates to a multi-building energy management method and system combining robust Markov and SAC algorithms. Background Art

[0002] In modern society, to achieve sustainable development goals and address the challenges of global climate change, including reducing energy consumption, improving energy efficiency, and promoting environmental protection, developing intelligent buildings has become a choice for many countries around the world. Simultaneously, with the advancement of information and communication technologies, an increasing number of intelligent devices are enabling remote control within building energy management systems (BEMs), particularly facilitating the management of distributed energy resources and energy storage systems. These factors are driving the gradual transition of modern BEMs toward distributed, intelligent, cyber-physical (C-P) energy management systems. In C-P-BEMs, the intelligence of energy management devices is closely linked to the overall energy efficiency of the system. Due to the high real-time requirements of physical systems during operation, many devices often overlook intelligent optimization at the outset of their design. This, in turn, necessitates more intelligent management strategies when transitioning to C-P-BEMs. Furthermore, the complexity of BEMs requires consideration not only of optimizing traditional energy management strategies but also of novel energy management approaches that incorporate the physical characteristics of buildings. Currently, there is limited research on intelligent optimization for multi-building energy management, and the corresponding optimization strategies are understudied. Therefore, research on integrated C-P-BEM optimization for efficient energy management is crucial to help EEM operators better operate and optimize energy usage.

[0003] Existing research on building energy management system optimization primarily consists of traditional control algorithms and simple machine learning algorithms. While simple and easy to implement, traditional control algorithms such as PID control and rule-based control methods struggle to achieve good performance in complex environments. While simple machine learning algorithms such as linear regression and decision trees have shown some improvement, they often perform poorly in highly nonlinear and dynamically changing multi-building environments.

[0004] Therefore, it is an urgent problem for those skilled in the art to propose a SAC algorithm based on Markov game to solve the difficulties existing in the prior art. Summary of the Invention

[0005] In view of this, the object of the present invention is to provide a multi-building energy management method and system combining robust Markov and SAC algorithms to optimize strategies and cope with various uncertainties, thereby improving robustness.

[0006] To achieve the above objectives, the present invention provides the following technical solutions: In a first aspect, a multi-building energy management method combining robust Markov and SAC algorithms comprises the following steps:

[0007] Step 1: Model a multi-building environment, collect historical data and construct a building energy performance model;

[0008] Step 2: Define the state space, reward value, and action space through the data processing module, train the gradient boosted decision tree (GBDT) for the building, input it into the reinforcement learning agent module for training, and introduce the natural agent to model uncertainty;

[0009] Step 3: Based on the modeling results of the residual distribution, a dynamic environment with uncertainty is constructed for training the reinforcement learning agent;

[0010] Step 4: Build and train a low-carbon energy scheduling model based on the SAC algorithm. The reinforcement learning agent generates actions based on observations and the policy model, obtains rewards and next states from the environment, stores experience in a replay buffer, and regularly updates the policy. Finally, it is applied to an environment constructed based on real data.

[0011] Furthermore, in step 1, the building environment is modeled, and the specific contents are as follows:

[0012] Model the multi-building environment based on the acquired building energy consumption, energy storage and environmental information.

[0013] Furthermore, in step 2, information related to uncertainty in natural agent modeling is introduced, and its specific content is:

[0014] Calculate the predicted residuals of building energy demand using the trained GBDT model :

[0015]

[0016] in The actual energy demand of the building; For the building environment data input to the GBDT model, due to the complex residual distribution, a mixed Gaussian model is used to capture the multimodal characteristics: Where K is the number of Gaussian distributions, is the weight of each Gaussian distribution, is the mean of each Gaussian distribution, is the variance of each Gaussian distribution.

[0017] Furthermore, in step 3, a dynamic environment containing uncertainty is constructed, and its specific contents are as follows:

[0018] From the modeled residual distribution Medium sampling disturbance: Dynamically adjust the prediction value of GBDT:

[0019] Add the GBDT prediction results to the sampled disturbance to generate a dynamic environment with randomness: in, is the energy demand at time t in a dynamic environment, including the deterministic prediction of GBDT and the uncertain disturbance.

[0020] Furthermore, in step 4, a low-carbon energy dispatch model based on the SAC algorithm is defined and initialized, and its specific contents are as follows:

[0021] Define the state, action, and reward function of the reinforcement learning problem, where the state set S includes the current environment and equipment status of the building: , Forecasting building energy demand, is the energy storage state characteristic, For weather characteristics, is the time feature; action set A: includes controllable variables: , is the charging and discharging power of the energy storage system, is the heating power, is the cooling power;

[0022] Reward function r: Comprehensive consideration of energy cost , carbon emissions , load smoothing and robustness :

[0023]

[0024] in, , , , is the weight coefficient;

[0025] The following five neural networks are needed to build SAC: a policy network : Input status , output action ; A state-value network : Input status , output action To Strategy The action-value function under The mapping, is a mapping strategy; a target state value network : Used for stable training; two state action value networks and :enter , output action value;

[0026] Memory initialization: Create an experience replay buffer D to store interaction data , to support the gradient update of the network, set the memory capacity ;

[0027] Initialize the policy network , state value network , target state value network and state-action-value networks and .

[0028] Furthermore, the training update after model initialization in step 4 is as follows:

[0029] First get the current state: Get the current state from the reinforcement learning environment , including building energy demand, energy storage status, weather conditions and time characteristics;

[0030] Secondly, the action is selected through the policy network : Input the current state into the policy network , generate actions ,The actions output by the strategy network include energy storage charging and discharging ,power, heating and cooling power;

[0031] Next action Applied to a dynamic environment, the environment returns to the state of the next moment and instant rewards ;

[0032] Finally, store the interaction data into the memory: store the experience data of the current interaction into the memory Store in memory bank D; when the memory bank capacity exceeds When , remove the earliest data point;

[0033] Every several steps, sample experience data from the memory bank D to update the network;

[0034] Next, update the state action value network and and state-value networks , and minimize the error of the state network :

[0035]

[0036] Where B is the batch size, and are the states and actions of the i-th batch respectively;

[0037] Update policy network : Maximize the expected return of the strategy , and at the same time introduce the entropy regularization term to improve the exploration ability of the strategy:

[0038]

[0039] in is the entropy coefficient, which is used to balance exploration and exploitation;

[0040] Then use the target network update formula: ; is the target network update coefficient;

[0041] At the end of each round of training, determine whether the strategy has converged: when the objective function value tends to be stable or the test performance index reaches the preset threshold, stop training; save model parameters: save the strategy network , state value network , target state value network and state-action-value networks and The final parameter of

[0042] Finally, the trained policy network Deployed into actual systems, it can guide the optimal control of building energy systems in real time.

[0043] In a second aspect, the present invention also provides a multi-building energy management system combining robust Markov and SAC algorithms, the system comprising a building model construction module, an intelligent agent construction and pre-training module, a natural agent introduction module, and a control strategy generation module connected in sequence; wherein,

[0044] The building model building module is used to model the multi-building environment based on the acquired building energy consumption, energy storage and environmental information;

[0045] The agent construction and pre-training module is used to construct a reward function based on individual and collective net electricity, where the reward depends on the building's electricity consumption and the overall energy consumption of the region; and a sequential iterative action selection method based on gradient boosted decision trees (GBDTs) is used to share prediction information between agents to optimize energy consumption decisions;

[0046] The natural agent introduction module is used to introduce natural agents to model uncertainty during the training process. The reinforcement learning agent generates actions based on observations and the policy model, obtains rewards and next states from the environment, stores experience in a replay buffer, and regularly updates the policy.

[0047] The control strategy generation module is used to generate accurate control decisions by performing simulation in a benchmark multi-agent reinforcement learning environment and combining model training with data from a real environment.

[0048] In a third aspect, the present invention also provides a multi-building energy management device combining robust Markov and SAC algorithms, comprising: a processor and a memory, the memory storing machine-readable instructions executable by the processor, the processor being used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the processor executes the steps of the multi-building energy management method combining robust Markov and SAC algorithms.

[0049] In a fourth aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer device, the computer device executes the steps of the multi-building energy management method combining the robust Markov and SAC algorithms.

[0050] In a fifth aspect, the present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the multi-building energy management method combining robust Markov and SAC algorithms. The present invention aims to achieve intelligent scheduling of building energy, particularly considering the complexity and uncertainty of building energy supply systems. By employing a hybrid energy system scheduling method based on the SAC (Soft Actor Critic) algorithm, an optimized scheduling strategy is designed that combines wind power, photovoltaic power, energy storage, thermal storage, and the dynamic changes in building load demand. Due to the volatility and unpredictability of wind and photovoltaic power, and the ever-changing building load, the present invention balances these uncertainties through the rational scheduling of energy storage and thermal storage equipment, while optimizing building energy efficiency and energy utilization. Using a maximum entropy-based reinforcement learning algorithm (SAC algorithm), an intelligent agent learns the optimal scheduling strategy during training to maximize the balance between energy efficiency and building energy demand.

[0051] The building energy scheduling method based on the SAC algorithm of the present invention has the following beneficial effects:

[0052] (1) During periods of low energy demand for buildings, the surplus electricity generated by wind power and photovoltaic power generation can be used for energy storage or heat storage, thus avoiding energy waste, improving the utilization rate of renewable energy, and reducing the burden on the power grid.

[0053] (2) The present invention reduces energy load fluctuations during peak building energy consumption periods by dispatching energy storage and heat storage equipment, thereby reducing the burden on the power grid, while achieving low-carbon operations within the building and reducing carbon emissions from building energy management. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0055] Figure 1 This is a flow chart of a multi-building energy management method that combines robust Markov and SAC algorithms provided by the present invention.

[0056] Figure 2 This is a system structure diagram of a multi-building energy management method that combines robust Markov and SAC algorithms provided by the present invention.

[0057] Figure 3 This is a specific flow chart of a multi-building energy management method that combines robust Markov and SAC algorithms provided by the present invention.

[0058] Figure 4 A comparison chart of a multi-building energy management method combining robust Markov and SAC algorithms provided by the present invention and the other two methods.

[0059] Figure 5 This is a structural diagram of a multi-building energy management device that combines robust Markov and SAC algorithms in the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the examples described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0061] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0062] like Figure 1 As shown, the present invention proposes a multi-building energy management method combining robust Markov and SAC algorithms, comprising the following steps:

[0063] S1: Collect historical data and build a building energy characteristics model;

[0064] S1.1 Collect historical operating data of the building energy management system: weather conditions, including temperature, humidity, and solar radiation, recorded as ; The electricity demand, heating demand and cooling demand of the building are recorded as 、 、 ; The operating status of the energy storage system, including the energy storage status characteristics ; Time characteristics , including hours, days, seasons, etc.

[0065] S1.2 Use Gradient Boosting Decision Tree (GBDT) to build a building energy characteristic model to predict the building’s energy performance in the future. Energy demand at future times is used as the forecast target. , the weather characteristics , time characteristics , building load characteristics and energy storage state characteristics As model input: , select Gradient Boosting Decision Tree (GBDT) as the basic prediction model: ,in is the predicted value of the model, and N is the size of the dataset. The trained GBDT model can achieve short-term prediction of building energy demand:

[0066]

[0067] in, To forecast demand, is the feature input, is the residual.

[0068] S2: Introducing natural agents to model uncertainty;

[0069] S2.1 Calculate the predicted residual of building energy demand using the trained GBDT model:

[0070]

[0071] in The actual energy demand of the building.

[0072] S2.2 Since the residual distribution is relatively complex, a mixed Gaussian model is used to capture multimodal characteristics:

[0073]

[0074] Where K is the number of Gaussian distributions, is the weight of each Gaussian distribution, is the mean of each Gaussian distribution, is the variance of each Gaussian distribution.

[0075] S3 constructs a dynamic environment containing uncertainty based on the modeling results of the residual distribution for the training of reinforcement learning agents. Random sampling perturbation in: Dynamically adjust the prediction value of GBDT:

[0076] Add the GBDT prediction results to the sampled perturbations to generate a dynamic environment with randomness: in, is the energy demand at time t in a dynamic environment, which includes the deterministic prediction of GBDT and the uncertain disturbance.

[0077] S4 builds and trains a low-carbon energy dispatch model based on the SAC algorithm (soft actor-critic algorithm), such as Figure 3 As shown:

[0078] S4.1 defines the state, action, and reward function of the reinforcement learning problem, where the state set S includes the current environment and equipment status of the building: ;Action set A: including controllable variables: ;in, , is the charging and discharging power of the energy storage system, is the heating power, is the cooling power. Reward function r: Comprehensive consideration of energy cost , carbon emissions , load smoothing and robustness :

[0079]

[0080] in, , , , is the weight coefficient.

[0081] S4.2 Construct the following five neural networks required for SAC: a policy network : Input status , output action ; A state-value network : Input status , output action To Strategy The action-value function under The mapping, is a mapping strategy; a target state value network : Used for stable training; two state action value networks and :enter , outputs action values, which are used to improve the stability of training and reduce the over-estimation problem in policy optimization.

[0082] S4.3 Memory Bank Initialization: Create an experience replay buffer D to store interaction data , to support the gradient update of the network. Set the memory capacity .

[0083] S4.4 Initialize the policy network , state value network , target state value network and state-action-value networks and .

[0084] S4.5 The training process for each time step includes the following steps: First, obtain the current state: obtain the current state from the reinforcement learning environment , including building energy demand, energy storage status, weather conditions and time characteristics. Secondly, the action is selected through the strategy network : Input the current state into the policy network , generate actions , the actions output by the strategy network include energy storage charging and discharging power, heating and cooling power, etc. Applied to a dynamic environment, the environment returns to the state of the next moment and instant rewards Finally, store the interaction data into the memory: store the current interaction experience data into the memory Store in memory bank D. When the memory bank capacity exceeds , remove the oldest data point.

[0085] S4.6 Setting a fixed time step interval , every time you pass The next step is to sample experience data from the memory bank D to update the network.

[0086] S4.7 Update the state-action-value network and : Sample a batch of data from the memory bank ; Calculate target value : ;in is the output of the target state value network, is the discount factor; update using gradient descent Parameters:

[0087]

[0088] Where B is the batch size, and are the states and actions of the i-th batch respectively.

[0089] S4.8 Update State Value Network :Calculate the target value:

[0090]

[0091] Minimize the error of the state network:

[0092]

[0093] S4.9 Update Policy Network : Maximize the expected return of the strategy and introduce entropy regularization to improve the exploration ability of the strategy:

[0094]

[0095] in is the entropy coefficient, which is used to balance exploration and exploitation.

[0096] S4.10 soft update target network:

[0097] Use the target network update formula: ; is the target network update coefficient.

[0098] S4.11 At the end of each round of training, determine whether the strategy has converged: stop training when the objective function value tends to be stable or the test performance index reaches the preset threshold.

[0099] S4.12 Save model parameters: Save policy network , state value network , target state value network and state-action-value networks and The final parameter.

[0100] S4.13 will train the policy network Deployed into actual systems, it can guide the optimal control of building energy systems in real time.

[0101] Based on the aforementioned building energy scheduling model, simulations were conducted to optimize the building's energy efficiency and energy costs by rationally scheduling the building's photovoltaic power generation, energy storage, and thermal storage equipment. During off-peak hours, excess electricity was stored, and a scheduling strategy generated by the SAC algorithm was used to optimize both economic efficiency and carbon emissions. Table 1 shows the parameter configurations of the building's energy devices in the simulation environment, Table 2 shows the configuration parameters of the gradient boosting decision tree, and Table 3 lists the parameters used in the SAC algorithm training process.

[0102] Table 1 Parameter configuration of building energy devices

[0103]

[0104] Table 2 Gradient Boosting Decision Tree (GBDT) parameter configuration

[0105]

[0106] Table 3 SAC algorithm parameter configuration

[0107]

[0108] Reference Figure 2 As shown, the present invention also provides a multi-building energy management system combining robust Markov and SAC algorithms, which applies the above-mentioned multi-building energy management method combining robust Markov and SAC algorithms, including a building model construction module, an intelligent agent construction and pre-training module, a natural agent introduction module, and a control strategy generation module connected in sequence; wherein:

[0109] Building model building module, which models the multi-building environment based on the acquired building energy consumption, energy storage and environmental information;

[0110] The agent construction and pre-training module constructs a reward function based on individual and collective net electricity consumption. The reward depends on the building's electricity consumption and the overall energy consumption of the region, thereby encouraging energy self-sufficiency. A sequential iterative action selection method based on gradient boosted decision trees (GBDT) is used, and prediction information is shared between agents to optimize energy consumption decisions.

[0111] The natural agent introduction module introduces natural agents to model uncertainty during training. The reinforcement learning agent generates actions based on observations and the policy model, obtains rewards and next states from the environment, stores experience in a replay buffer, and regularly updates the policy to improve robustness.

[0112] The control strategy generation module generates precise control decisions by simulating in a benchmark multi-agent reinforcement learning environment and training the model with data from the real environment. This optimizes energy management, coordinates agent behaviors, and improves the overall efficiency and adaptability of the system.

[0113] Reference Figure 3 As shown, the present invention discloses a specific flow chart based on a robust SAC building energy scheduling algorithm.

[0114] To further demonstrate the superiority of this method in solving the building energy scheduling problem, this example decided to compare this algorithm with the Proximal Policy Optimization (PPO) algorithm and the Deep Deterministic Policy Gradient (DDPG) algorithm. Figure 4 The following is a training graph comparing two methods. It can be seen that all three algorithms show an upward trend, with their strategies improving. However, DDPG exhibits the most oscillatory and unstable learning behavior, ultimately failing to reach the optimal strategy (the highest reward level in all cases). On the other hand, the PPO algorithm and the algorithm used in the present invention mitigate this instability and demonstrate relatively superior performance. Furthermore, it can be observed that in the constructed building energy environment, the algorithm used in the present invention performs significantly better in terms of both convergence speed and final reward level.

[0115] Corresponding to the aforementioned embodiment of a multi-building energy management method combining robust Markov and SAC algorithms, the present invention also provides an embodiment of a multi-building energy management device combining robust Markov and SAC algorithms.

[0116] See also Figure 5 An embodiment of the present invention provides a multi-building energy management device that combines robust Markov and SAC algorithms, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a multi-building energy management method that combines robust Markov and SAC algorithms in the above embodiment.

[0117] The embodiment of a multi-building energy management device that combines robust Markov and SAC algorithms provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 5 As shown, it is a hardware structure diagram of a multi-building energy management device combining robust Markov and SAC algorithms provided by the present invention, in which any device with data processing capability is located. Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0118] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0119] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0120] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the multi-building energy management method combining the robust Markov and SAC algorithms in the above embodiment is implemented.

[0121] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0122] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the multi-building energy management method combining robust Markov and SAC algorithms.

[0123] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0124] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-building energy management method combining robust Markov and SAC algorithms, characterized in that: The steps include: Step 1: Collect historical operating data of the building energy management system and use the gradient boosting decision tree (GBDT) to build a building energy characteristic model to predict the building energy demand in the future. Step 2: Introduce natural agent modeling uncertainty, the specific contents are: Calculate the predicted residual of building energy demand at time t using the trained GBDT model : ； in The actual building energy demand for the building; The operating data of the building energy management system input to the GBDT model uses a mixed Gaussian model to capture the multimodal characteristics due to the complex residual distribution: Where K is the number of Gaussian distributions, is the weight of each Gaussian distribution, is the mean of each Gaussian distribution, is the variance of each Gaussian distribution, residual distribution; Step 3: According to the residual distribution , construct a dynamic environment containing uncertainty, including: randomly sampling disturbances from the modeled residual distribution, dynamically adjusting the GBDT prediction value, and obtaining a deterministic prediction containing GBDT and uncertainty disturbances The building energy demand at time t in a dynamic environment ; Step 4: Build a low-carbon energy scheduling model based on the SAC algorithm, including: the reinforcement learning agent generates actions based on observations and the policy model, obtains rewards and next states from the environment, stores experience in a replay buffer, and regularly updates the policy; Construct a low-carbon energy dispatch model based on the SAC algorithm, the specific contents of which are: Define the state, action, and reward function of the reinforcement learning problem, where the state set S includes the current environment information and device status of the building: , is the state set at time t, is the building energy demand at time t in a dynamic environment, is the energy storage state characteristic, For weather characteristics, is the time feature; the action set A includes controllable variables: , is the action set at time t, is the charging and discharging power of the energy storage system, is the heating power, is the cooling power; Reward function r: Comprehensive consideration of energy cost , carbon emissions , load smoothing and robustness : ； in, , , , is the weight coefficient, For the state set Select Action Set The reward function of The following five neural networks are needed to build SAC: a policy network : Input status , output action ; A state-value network : Input status ,action , the output is the strategy The action-value function under The mapping, is a mapping strategy; a target state value network : Used for stable training; two state action value networks and :enter , output action value; Memory initialization: Create an experience replay buffer D to store interaction data , The reward function value at time t is used to support the gradient update of the network and set the memory capacity ; Initialize the policy network , state value network , target state value network and state-action-value networks and ; Step 5: Deploy the low-carbon energy scheduling model into the actual system to guide the optimization control of the building energy system in real time.

2. The multi-building energy management method combining robust Markov and SAC algorithms according to claim 1 is characterized in that: The historical operation data of the building energy management system in step 1 includes building energy consumption, energy storage status and environmental information.

3. The multi-building energy management method combining robust Markov and SAC algorithms according to claim 1 is characterized in that: Step 3 constructs a dynamic environment with uncertainty, and its specific contents are as follows: From the modeled residual distribution Medium sampling disturbance: Dynamically adjust the prediction value of GBDT: Add the GBDT prediction results to the sampled disturbance to generate a dynamic environment with randomness: in, is the building energy demand at time t in a dynamic environment, including the deterministic prediction of GBDT and uncertainty disturbances .

4. The multi-building energy management method combining robust Markov and SAC algorithms according to claim 3 is characterized in that: The training update after model initialization in step 4 is as follows: First get the current state: Get the current state from the reinforcement learning environment , including building energy demand, energy storage status, weather conditions and time characteristics; Secondly, the action is selected through the policy network : Input the current state into the policy network , generate actions ,The actions output by the strategy network include energy storage charging and discharging ,power, heating and cooling power; Next action Applied to a dynamic environment, the environment returns to the state of the next moment and instant rewards ; Finally, store the interaction data into the memory: store the experience data of the current interaction into the memory Store in memory bank D; When the memory capacity exceeds When , remove the earliest data point; Every several steps, sample experience data from the memory bank D to update the network; Next, update the state action value network and and state-value networks , and minimize the error of the state network : ； Where B is the batch size, and are the states and actions of the i-th batch respectively; Update policy network : Maximize the expected return of the strategy , and introduce the entropy regularization term to improve the exploration ability of the strategy: ； in is the entropy coefficient; Then use the target network update formula: ; is the target network update coefficient; At the end of each round of training, determine whether the strategy has converged: when the objective function value tends to be stable or the test performance index reaches the preset threshold, stop training; save model parameters: save the strategy network , state value network , target state value network and state-action-value networks and The final parameter of Finally, the trained strategy network Deployed into actual systems, it can guide the optimal control of building energy systems in real time.

5. A system for implementing the multi-building energy management method combining robust Markov and SAC algorithms as described in any one of claims 1 to 4, characterized in that: The system includes a building model construction module, an intelligent agent construction and pre-training module, a natural agent introduction module and a control strategy generation module connected in sequence; The building model construction module is used to construct a building energy characteristic model using a gradient boosted decision tree (GBDT) based on the acquired building energy consumption, energy storage status and environmental information to predict the building energy demand of the building at a future time; The agent construction and pre-training module is used to construct a reward function based on individual and collective net electricity, where the reward depends on the building's electricity consumption and the overall energy consumption of the region; and a sequential iterative action selection method based on gradient boosted decision trees (GBDTs) is used to share prediction information between agents to optimize energy consumption decisions; The natural agent introduction module is used to introduce natural agents to model uncertainty during the training process. The reinforcement learning agent generates actions based on observations and the policy model, obtains rewards and next states from the environment, stores experience in a replay buffer, and regularly updates the policy. The control strategy generation module is used to generate accurate control decisions by performing simulation in a benchmark multi-agent reinforcement learning environment and combining model training with data from a real environment.

6. A multi-building energy management device combining robust Markov and SAC algorithms, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the multi-building energy management method combining robust Markov and SAC algorithms as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that A computer program is stored on a computer-readable storage medium. When the computer program is executed by a computer device, the computer device executes the steps of the multi-building energy management method combining robust Markov and SAC algorithms as described in any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-building energy management method combining robust Markov and SAC algorithms is implemented as described in any one of claims 1 to 4.

Images