Multi-building energy management method and system combining robust Markov and SAC algorithms

Abstract

The invention discloses a multi-building energy management method and system combining robust Markov and SAC algorithms, and the method comprises the steps: obtaining building energy consumption, energy storage conditions and environment information, and forming a multi-building environment simulation system; modeling is carried out on a multi-building environment, and energy management strategy optimization is carried out by combining robust SAC (soft actor commentator algorithm); the data processing module processes observation values, rewards and actions, and inputs the observation values, the rewards and the actions into the reinforcement learning agent module for training; the reinforcement learning agent module comprises an improved neural network model, an experience playback buffer area and an optimized strategy algorithm, and an energy management strategy is optimized through training; the invention provides a method for realizing intelligent dynamic energy management by combining robust Markov and SAC algorithms in a complex and changeable environment, the overall energy efficiency and stability of the system are effectively improved, external data sources such as meteorological data and market electricity price are introduced, and the accuracy and economical efficiency of decision making are improved.

Description

Technical Field

[0001] The present invention belongs to the field of multi-building environment intelligent energy management system, and in particular to a multi-building energy management method and system combining robust Markov and SAC algorithms. Background Art

[0002] In modern society, in order to achieve sustainable development goals and meet the challenges of global climate change, including reducing energy consumption, improving energy efficiency and promoting environmental protection, the development of intelligent buildings has become the choice of many countries in the world. At the same time, with the development of information and communication technology, more and more intelligent devices are realizing remote control functions in building energy management systems, which is especially conducive to the management of distributed energy and energy storage systems. Due to these factors, modern building energy management systems are gradually transforming to distributed, intelligent, new information-physical fusion energy management systems. In the building energy management system of information-physical fusion, the intelligence of energy management equipment is closely related to the overall energy efficiency of the system. Due to the high requirements of real-time performance of physical systems during operation, many devices ignore intelligent optimization at the beginning of design, which also leads to the need for more intelligent management strategies when the energy management system is transformed into a new system of information-physical fusion. And due to the complexity of building energy management systems, it is necessary to consider not only the optimization of traditional energy management strategies, but also new energy management methods that combine the physical characteristics of buildings. Since there are few studies on intelligent optimization of multi-building environment energy management at present, and the corresponding optimization strategies have not been fully studied, it is very important to study the information-physical comprehensive optimization under efficient energy management in order to help energy management system operators better operate and optimize energy use.

[0003] Existing optimization research on building energy management systems mainly consists of traditional control algorithms and simple machine learning algorithms. Although traditional control algorithms such as PID control and rule-based control methods are simple and easy to implement, they are difficult to achieve good performance in complex environments. Although simple machine learning algorithms such as linear regression and decision trees have been improved, they do not perform well in the face of highly nonlinear and dynamically changing multi-building environments.

[0004] Therefore, it is an urgent problem for technicians in this field to propose a SAC algorithm based on Markov game to solve the difficulties existing in the prior art. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a multi-building energy management method and system combining robust Markov and SAC algorithms to optimize strategies and cope with various uncertainties, thereby improving robustness.

[0006] In order to achieve the above object, the present invention provides the following technical solutions: In the first aspect, a multi-building energy management method combining robust Markov and SAC algorithms comprises the following steps:

[0007] Step 1: Model a multi-building environment, collect historical data and construct a building energy characteristic model;

[0008] Step 2: Define the state space, reward value, and action space through the data processing module, train the gradient boosted decision tree GBDT for the building, input the reinforcement learning agent module for training, and introduce the natural agent modeling uncertainty;

[0009] Step 3: Based on the modeling results of the residual distribution, a dynamic environment with uncertainty is constructed for the training of the reinforcement learning agent;

[0010] Step 4: Build and train a low-carbon energy scheduling model based on the SAC algorithm. The reinforcement learning agent generates actions based on the observations and policy model, obtains rewards and next states from the environment, stores the experience in the playback buffer, and regularly updates the policy. Finally, it is applied in an environment constructed based on real data.

[0011] Furthermore, in step 1, the building environment is modeled, and the specific contents are as follows:

[0012] Model the multi-building environment based on the acquired building energy consumption, energy storage and environmental information.

[0013] Furthermore, in step 2, information related to natural agent modeling uncertainty is introduced, and its specific content is:

[0014] Calculate the prediction residual of building energy demand using the trained GBDT model :

[0015]

[0016] in The real energy demand of the building; The building environment data input to the GBDT model uses a mixed Gaussian model to capture multimodal characteristics due to the complex residual distribution: Where K is the number of Gaussian distributions, is the weight of each Gaussian distribution, is the mean of each Gaussian distribution, is the variance of each Gaussian distribution.

[0017] Furthermore, in step 3, a dynamic environment containing uncertainty is constructed, and its specific content is:

[0018] From the modeled residual distribution Medium sampling disturbance: Dynamically adjust GBDT's prediction value:

[0019] Add the prediction results of GBDT to the sampled disturbance to generate a dynamic environment with randomness: in, is the energy demand at time t in a dynamic environment, including the deterministic prediction and uncertain disturbance of GBDT.

[0020] Furthermore, in step 4, a low-carbon energy dispatch model based on the SAC algorithm is defined and initialized, and its specific contents are as follows:

[0021] Define the state, action, and reward function of the reinforcement learning problem, where the state set S includes the current environment and equipment status of the building: , Forecasting demand for building energy, is the energy storage state characteristic, For weather characteristics, is the time feature; action set A: includes controllable variables: , is the charging and discharging power of the energy storage system, is the heating power, is the cooling power;

[0022] Reward function r: Comprehensive consideration of energy cost , Carbon emissions , load smoothness and robustness :

[0023]

[0024] in, , , , is the weight coefficient;

[0025] The following five neural networks are needed to build SAC: a policy network : Input status , output action ; A state value network : Input status , Output Action To Strategy The action value function under The mapping of is a mapping strategy; a target state value network : Used for stable training; two state action value networks and :enter , output action value;

[0026] Memory initialization: Create an experience replay buffer D to store interaction data , to support the gradient update of the network, set the memory capacity ;

[0027] Initialize the policy network , state value network , target state value network And the state action value network and .

[0028] Furthermore, the training update after model initialization in step 4 is as follows:

[0029] First get the current state: Get the current state from the reinforcement learning environment , including building energy demand, energy storage status, weather conditions and time characteristics;

[0030] Secondly, the action is selected through the policy network : Input the current state into the policy network , generate actions ,The actions output by the strategy network include energy storage charging and discharging power, heating and cooling power;

[0031] Next action Applied to a dynamic environment, the environment returns to the state of the next moment and instant rewards ;

[0032] Finally, store the interaction data into the memory: store the experience data of the current interaction Store in memory bank D; when the memory bank capacity exceeds When , remove the earliest data point;

[0033] Every several steps, sample experience data from the memory bank D to update the network;

[0034] Next, update the state action value network and And the state value network , and minimize the error of the state network :

[0035]

[0036] Where B is the batch size, and are the states and actions of the i-th batch respectively;

[0037] Update Policy Network : Maximize the expected return of the strategy , and introduce entropy regularization terms to improve the exploration ability of the strategy:

[0038]

[0039] in is the entropy coefficient, which is used to balance exploration and exploitation;

[0040] Then use the target network update formula: ; is the target network update coefficient;

[0041] At the end of each round of training, determine whether the strategy has converged: when the objective function value tends to be stable or the test performance index reaches the preset threshold, stop training; save model parameters: save the strategy network , state value network , target state value network And the state action value network and The final parameter of

[0042] Finally, the trained strategy network Deployed into actual systems, it can guide the optimal control of building energy systems in real time.

[0043] In a second aspect, the present invention also provides a multi-building energy management system combining robust Markov and SAC algorithms, the system comprising a building model construction module, an intelligent agent construction and pre-training module, a natural agent introduction module and a control strategy generation module connected in sequence; wherein,

[0044] The building model building module is used to model the multi-building environment according to the acquired building energy consumption, energy storage and environmental information;

[0045] The agent construction and pre-training module is used to construct a reward function based on individual and collective net electricity, where the reward depends on the power consumption of the building and the overall energy consumption of the region; and a sequential iterative action selection method based on the gradient boosted decision tree GBDT is used to share prediction information between agents to optimize energy consumption decisions;

[0046] The natural agent introduction module is used to introduce natural agents to model uncertainty during the training process. The reinforcement learning agent generates actions based on the observed values ​​and the policy model, obtains rewards and next states from the environment, stores the experience in the playback buffer, and regularly updates the policy.

[0047] The control strategy generation module is used to generate accurate control decisions by simulating in a benchmark multi-agent reinforcement learning environment and training the model in combination with data from a real environment.

[0048] In the third aspect, the present invention also provides a multi-building energy management device combining robust Markov and SAC algorithms, including: a processor and a memory, the memory storing machine-readable instructions executable by the processor, the processor being used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the processor executes the steps of the multi-building energy management method combining robust Markov and SAC algorithms.

[0049] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer device, the computer device executes the steps of the multi-building energy management method combining the robust Markov and SAC algorithms.

[0050] In a fifth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the multi-building energy management method combining robust Markov and SAC algorithms. The purpose of the present invention is to realize intelligent scheduling of building energy, especially when considering the complexity and uncertainty of the building energy supply system. By adopting a hybrid energy system scheduling method based on the SAC (Soft Actor Critic, SAC) algorithm, combined with the dynamic changes of wind power, photovoltaics, energy storage, heat storage and building load demand, an optimized scheduling strategy is designed. Due to the volatility and unpredictability of wind power and photovoltaics, and the ever-changing building load, the present invention balances these uncertainties by rationally scheduling energy storage and heat storage equipment, while optimizing building energy efficiency and energy utilization. Using a reinforcement learning algorithm based on maximum entropy (SAC algorithm), the optimal scheduling strategy is learned by the intelligent agent during the training process to maximize the balance between energy utilization efficiency and building energy demand.

[0051] The building energy scheduling method based on the SAC algorithm of the present invention has the following beneficial effects:

[0052] (1) During periods of low energy demand in buildings, the surplus electricity generated by wind power and photovoltaic power generation can be used for energy storage or heat storage, thus avoiding energy waste, improving the utilization rate of renewable energy, and reducing the burden on the power grid.

[0053] (2) The present invention reduces energy load fluctuations during peak periods of building energy consumption by dispatching energy storage and heat storage equipment, reduces the burden on the power grid, and simultaneously achieves low-carbon operation within the building, thereby reducing carbon emissions from building energy management. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0055] Figure 1 A method flow chart of a multi-building energy management method combining robust Markov and SAC algorithms provided by the present invention.

[0056] Figure 2 A system structure diagram of a multi-building energy management method combining robust Markov and SAC algorithms provided by the present invention.

[0057] Figure 3 A specific flow chart of a multi-building energy management method combining robust Markov and SAC algorithms provided by the present invention.

[0058] Figure 4 A comparison chart of a multi-building energy management method combining robust Markov and SAC algorithms provided by the present invention and the other two methods.

[0059] Figure 5 This is a structural diagram of a multi-building energy management device that combines robust Markov and SAC algorithms in the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the examples described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0062] like Figure 1 As shown, the present invention proposes a multi-building energy management method combining robust Markov and SAC algorithms, comprising the following steps:

[0063] S1: Collect historical data and build a building energy characteristics model;

[0064] S1.1 Collect historical operating data of the building energy management system: weather conditions, including temperature, humidity, and solar radiation, recorded as ; The electricity demand, heating demand and cooling demand of the building are denoted as , , ; The operating status of the energy storage system, including the energy storage status characteristics ; Time characteristics , including hours, days, seasons, etc.

[0065] S1.2 Use the gradient boosted decision tree (GBDT) to build a building energy characteristic model to predict the building's energy performance at future moments Energy demand at a future time is used as a forecast target , the weather characteristics , Time characteristics , Building load characteristics and energy storage state characteristics As model input: , select the gradient boosted decision tree (GBDT) as the basic prediction model: ,in is the predicted value of the model, and N is the size of the data set; the GBDT model obtained through training can realize the short-term prediction of building energy demand:

[0066]

[0067] in, To predict demand, is the feature input, is the residual.

[0068] S2: Introducing natural agents to model uncertainty;

[0069] S2.1 Calculate the predicted residual of building energy demand using the trained GBDT model:

[0070]

[0071] in The actual energy demand of the building.

[0072] S2.2 Since the residual distribution is relatively complex, a mixed Gaussian model is used to capture multimodal characteristics:

[0073]

[0074] Where K is the number of Gaussian distributions, is the weight of each Gaussian distribution, is the mean of each Gaussian distribution, is the variance of each Gaussian distribution.

[0075] S3 constructs a dynamic environment containing uncertainty based on the modeling results of the residual distribution for the training of reinforcement learning agents. Random sampling perturbation in: Dynamically adjust GBDT's prediction value:

[0076] Add the GBDT predictions to the sampled perturbations to generate a dynamic environment with randomness: in, is the energy demand at time t in a dynamic environment, including the deterministic prediction and uncertain disturbance of GBDT.

[0077] S4 builds and trains a low-carbon energy dispatch model based on the SAC algorithm (soft actor critic algorithm), such as Figure 3 As shown:

[0078] S4.1 defines the state, action, and reward function of the reinforcement learning problem, where the state set S includes the current environment and equipment status of the building: ;Action set A: includes controllable variables: ;in, , is the charging and discharging power of the energy storage system, is the heating power, is the cooling power. Reward function r: Comprehensive consideration of energy cost , Carbon emissions , load smoothness and robustness :

[0079]

[0080] in, , , , is the weight coefficient.

[0081] S4.2 Construct the following five neural networks required for SAC: a policy network : Input status , output action ; A state value network : Input status , Output Action To Strategy The action value function under The mapping of is a mapping strategy; a target state value network : Used for stable training; two state action value networks and :enter , outputs action values, which are used to improve the stability of training and reduce the over-estimation problem in policy optimization.

[0082] S4.3 Memory initialization: Create an experience playback buffer D to store interaction data , to support the gradient update of the network. Set the memory capacity .

[0083] S4.4 Initialize the policy network , state value network , target state value network And the state action value network and .

[0084] S4.5 The training process for each time step includes the following steps: First, obtain the current state: obtain the current state from the reinforcement learning environment , including building energy demand, energy storage status, weather conditions and time characteristics. Secondly, the strategy network selects actions : Input the current state into the policy network , generate actions , the actions output by the strategy network include energy storage charging and discharging power, heating and cooling power, etc. Applied to a dynamic environment, the environment returns to the state of the next moment and instant rewards Finally, store the interaction data into the memory: store the current interaction experience data into the memory Store in memory bank D. When the memory bank capacity exceeds , remove the earliest data point.

[0085] S4.6 Setting a fixed time step interval , every time you pass Step 2: Sample experience data from the memory bank D to update the network.

[0086] S4.7 Update state-action-value network and : Sample a batch of data from the memory bank ; Calculate the target value : ;in is the output of the target state value network, is the discount factor; update using gradient descent Parameters:

[0087]

[0088] Where B is the batch size, and are the states and actions of the i-th batch respectively.

[0089] S4.8 Update state value network : Calculate the target value:

[0090]

[0091] Minimize the error of the state network:

[0092]

[0093] S4.9 Update Strategy Network :Maximize the expected return of the strategy and introduce entropy regularization terms to improve the exploration ability of the strategy:

[0094]

[0095] in is the entropy coefficient, which is used to balance exploration and exploitation.

[0096] S4.10 soft update target network:

[0097] Use the target network update formula: ; is the target network update coefficient.

[0098] S4.11 At the end of each round of training, determine whether the strategy has converged: when the objective function value tends to be stable or the test performance index reaches the preset threshold, stop training.

[0099] S4.12 Save model parameters: Save policy network , state value network , target state value network And the state action value network and The final parameter.

[0100] S4.13 The trained policy network Deployed into actual systems, it can guide the optimal control of building energy systems in real time.

[0101] Based on the above building energy scheduling model, simulation is performed to optimize the building's energy efficiency and energy usage cost by reasonably scheduling photovoltaic power generation, energy storage and heat storage equipment in the building. During off-peak hours, the surplus electricity is stored, and the scheduling strategy generated by the SAC algorithm is used to achieve economic optimization and carbon emission optimization. Table 1 shows the parameter configuration of building energy devices in the simulation environment, Table 2 shows the configuration parameters of the gradient enhancement decision tree, and Table 3 lists the relevant parameters of the SAC algorithm training process.

[0102] Table 1 Parameter configuration of building energy devices

[0103]

[0104] Table 2 Gradient Boosting Decision Tree (GBDT) parameter configuration

[0105]

[0106] Table 3 SAC algorithm parameter configuration

[0107]

[0108] Reference Figure 2 As shown, the present invention also provides a multi-building energy management system combining robust Markov and SAC algorithms, which applies the above-mentioned multi-building energy management method combining robust Markov and SAC algorithms, including a building model construction module, an intelligent agent construction and pre-training module, a natural agent introduction module, and a control strategy generation module connected in sequence; wherein:

[0109] Building model building module, which models the multi-building environment based on the acquired building energy consumption, energy storage and environmental information;

[0110] The agent construction and pre-training module constructs a reward function based on individual and collective net electricity, and the reward depends on the power consumption of the building and the overall energy consumption of the region, thereby encouraging energy self-sufficiency. And use a sequential iterative action selection method based on the gradient boosting decision tree (GBDT), and agents share prediction information to optimize energy consumption decisions;

[0111] Natural agent introduction module: During the training process, natural agents are introduced to model uncertainty. The reinforcement learning agent generates actions based on the observed values ​​and policy model, obtains rewards and next states from the environment, stores experience in the playback buffer, and regularly updates the policy to improve robustness.

[0112] The control strategy generation module generates accurate control decisions by simulating in a benchmark multi-agent reinforcement learning environment and training models based on data from the real environment, thereby optimizing energy management, coordinating agent behaviors, and improving the overall efficiency and adaptability of the system.

[0113] Reference Figure 3 As shown, the present invention discloses a specific flow chart based on a robust SAC building energy scheduling algorithm.

[0114] In order to further demonstrate the superiority of this method in solving the building energy scheduling problem, this embodiment decides to compare this algorithm with the proximal policy optimization (PPO) algorithm and the deep deterministic policy gradient (DDPG) algorithm. Figure 4 is a training graph comparing the two methods. It can be seen that all three algorithms show an upward trend and their strategies are improving. However, DDPG exhibits the highest oscillation and unstable learning behavior and ultimately fails to reach the optimal strategy (the highest reward level in all cases). On the other hand, the PPO algorithm and the algorithm used in the present invention can alleviate this instability problem and show relatively excellent performance. In addition, it can be observed that in the constructed building energy environment, the algorithm used in the present invention is superior in terms of both convergence speed and final reward level.

[0115] Corresponding to the aforementioned embodiment of a multi-building energy management method combining robust Markov and SAC algorithms, the present invention also provides an embodiment of a multi-building energy management device combining robust Markov and SAC algorithms.

[0116] See also Figure 5 An embodiment of the present invention provides a multi-building energy management device combining robust Markov and SAC algorithms, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a multi-building energy management method combining robust Markov and SAC algorithms in the above embodiment.

[0117] An embodiment of a multi-building energy management device combining a robust Markov and SAC algorithm provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 5 As shown, it is a hardware structure diagram of a multi-building energy management device combining robust Markov and SAC algorithms provided by the present invention, in which any device with data processing capability is located, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0118] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0119] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0120] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a multi-building energy management method combining a robust Markov algorithm and a SAC algorithm in the above embodiment is implemented.

[0121] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0122] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the multi-building energy management method combining the robust Markov and SAC algorithms.

[0123] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0124] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-building energy management method combining robust Markov and SAC algorithms, characterized in that: The steps include: Step 1: Model a multi-building environment, collect historical data and construct a building energy characteristic model; Step 2: Define the state space, reward value, and action space through the data processing module, train the gradient boosted decision tree GBDT for the building, input the reinforcement learning agent module for training, and introduce the natural agent modeling uncertainty; Step 3: Based on the modeling results of the residual distribution, a dynamic environment with uncertainty is constructed for the training of the reinforcement learning agent; Step 4: Build and train a low-carbon energy scheduling model based on the SAC algorithm. The reinforcement learning agent generates actions based on the observations and policy model, obtains rewards and next states from the environment, stores the experience in the playback buffer, and regularly updates the policy. Finally, it is applied in an environment constructed based on real data.

2. A multi-building energy management method combining robust Markov and SAC algorithms according to claim 1, characterized in that: In step 1, the building environment is modeled, and its specific contents are as follows: Model the multi-building environment based on the acquired building energy consumption, energy storage and environmental information.

3. The multi-building energy management method combining robust Markov and SAC algorithms according to claim 1, characterized in that: In step 2, information related to uncertainty in natural agent modeling is introduced, and its specific content is as follows: Calculate the prediction residual of building energy demand using the trained GBDT model : ； in The real energy demand of the building; The building environment data input to the GBDT model uses a mixed Gaussian model to capture multimodal characteristics due to the complex residual distribution: Where K is the number of Gaussian distributions, is the weight of each Gaussian distribution, is the mean of each Gaussian distribution, is the variance of each Gaussian distribution.

4. The multi-building energy management method combining robust Markov and SAC algorithms according to claim 3 is characterized in that: Step 3 constructs a dynamic environment with uncertainty, and its specific contents are as follows: From the modeled residual distribution Medium sampling disturbance: Dynamically adjust GBDT's prediction value: Add the prediction results of GBDT to the sampled disturbance to generate a dynamic environment with randomness: in, is the energy demand at time t in a dynamic environment, including the deterministic prediction and uncertain disturbance of GBDT.

5. The multi-building energy management method combining robust Markov and SAC algorithms according to claim 3 is characterized in that: In step 4, a low-carbon energy dispatch model based on the SAC algorithm is defined and initialized. The specific contents are as follows: Define the state, action, and reward function of the reinforcement learning problem, where the state set S includes the current environment and equipment status of the building: , Forecasting demand for building energy, is the energy storage state characteristic, For weather characteristics, is the time feature; Action set A: includes controllable variables: , is the charging and discharging power of the energy storage system, is the heating power, is the cooling power; Reward function r: Comprehensive consideration of energy cost , Carbon emissions , load smoothness and robustness : ； in, , , , is the weight coefficient; The following five neural networks are needed to build SAC: a policy network : Input status , output action ; A state value network : Input status , Output Action To Strategy The action value function under The mapping of is a mapping strategy; a target state value network : Used for stable training; two state action value networks and :enter , output action value; Memory initialization: Create an experience replay buffer D to store interaction data , to support the gradient update of the network, set the memory capacity ; Initialize the policy network , state value network , target state value network And the state action value network and .

6. The multi-building energy management method combining robust Markov and SAC algorithms according to claim 4, characterized in that: The training update after model initialization in step 4 is as follows: First get the current state: Get the current state from the reinforcement learning environment , including building energy demand, energy storage status, weather conditions and time characteristics; Secondly, the action is selected through the policy network : Input the current state into the policy network , generate actions ,The actions output by the strategy network include energy storage charging and discharging power, heating and cooling power; Next action Applied to a dynamic environment, the environment returns to the state of the next moment and instant rewards ; Finally, store the interaction data into the memory: store the experience data of the current interaction Store in memory bank D; When the memory capacity exceeds When , remove the earliest data point; Every several steps, sample experience data from the memory bank D to update the network; Next, update the state action value network and And the state value network , and minimize the error of the state network : ； Where B is the batch size, and are the states and actions of the i-th batch respectively; Update Policy Network : Maximize the expected return of the strategy , and introduce entropy regularization terms to improve the exploration ability of the strategy: ； in is the entropy coefficient, which is used to balance exploration and exploitation; Then use the target network update formula: ; is the target network update coefficient; At the end of each round of training, determine whether the strategy has converged: when the objective function value tends to be stable or the test performance index reaches the preset threshold, stop training; save model parameters: save the strategy network , state value network , target state value network And the state action value network and The final parameter of Finally, the trained strategy network Deployed into actual systems, it can guide the optimal control of building energy systems in real time.

7. A system for implementing the multi-building energy management method combining robust Markov and SAC algorithms as described in any one of claims 1 to 6, characterized in that: The system includes a building model construction module, an intelligent agent construction and pre-training module, a natural agent introduction module and a control strategy generation module connected in sequence; The building model building module is used to model the multi-building environment according to the acquired building energy consumption, energy storage and environmental information; The agent construction and pre-training module is used to construct a reward function based on individual and collective net electricity, where the reward depends on the power consumption of the building and the overall energy consumption of the region; and a sequential iterative action selection method based on the gradient boosted decision tree GBDT is used to share prediction information between agents to optimize energy consumption decisions; The natural agent introduction module is used to introduce natural agents to model uncertainty during the training process. The reinforcement learning agent generates actions based on the observed values ​​and the policy model, obtains rewards and next states from the environment, stores the experience in the playback buffer, and regularly updates the policy. The control strategy generation module is used to generate accurate control decisions by simulating in a benchmark multi-agent reinforcement learning environment and training the model in combination with data from a real environment.

8. A multi-building energy management device combining robust Markov and SAC algorithms, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the multi-building energy management method combining robust Markov and SAC algorithms as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored on a computer-readable storage medium. When the computer program is executed by a computer device, the computer device executes the steps of the multi-building energy management method combining robust Markov and SAC algorithms as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-building energy management method combining the robust Markov and SAC algorithms as described in any one of claims 1 to 6 is implemented.

Images