Building multi-energy network collaborative control method and device based on hierarchical reinforcement learning

Abstract

The application discloses a kind of based on layered reinforcement learning's building multi-energy network collaborative control method and device, comprising:1.building the physical model system of building multi-energy network;2.building the double-flow load forecasting model of residual reinforcement learning, based on asymmetric reward calculation output final load prediction value;3.design layered reinforcement learning-based decision framework, including different operating frequency's high-level management network and bottom execution network;4.build multi-dimensional heterogeneous state space and composite action space;5.design to introduce bottom soft protection mechanism and the evaluation system of nonlinear reward, solve multi-objective conflict and eliminate model boundary overrun risk;6.input real-time state to the optimal layered reinforcement learning model trained, output deterministic instruction execution closed-loop online scheduling.The application is applicable to the multi-energy collaborative scheduling scene of complex building, can give the robust regulation and control scheme of economic optimum under the premise of guaranteeing user comfort and system physical safety.

Description

Technical Field

[0001] This invention relates to the fields of building energy management systems and smart grid demand response technology, specifically to a method and apparatus for collaborative control of multi-energy networks in buildings based on hierarchical reinforcement learning. This invention integrates a physical model of equivalent thermal parameters for HVAC systems, a dynamic model of battery energy storage systems, and a stochastic load model for electric vehicles, utilizing a hierarchical deep reinforcement learning architecture to achieve multi-timescale collaborative optimization and robust control of building energy systems. Background Technology

[0002] Modern buildings have evolved from simple energy consumers into complex multi-energy network systems integrating new energy sources (such as photovoltaics), flexible loads (such as HVAC, trams, and charging stations), and energy storage devices. Building energy systems account for a significant proportion of global final energy consumption, and their load regulation potential is crucial for alleviating peak-valley pressure on the power grid. However, the optimal scheduling of building multi-energy networks faces highly complex, nonlinear, and stochastic challenges.

[0003] First, building thermodynamic processes exhibit significant time lag and nonlinear characteristics. The energy efficiency of HVAC systems is influenced by the outdoor environment, indoor occupant activities, and the thermal properties of the building envelope. Traditional rule-based control (RBC) struggles to capture complex time-of-use electricity pricing signals to achieve economic optimization. Second, the integration of energy storage systems and electric vehicles introduces cross-time-period decision coupling. The arrival and departure times of electric vehicles, as well as their remaining energy demand, are highly stochastic. While traditional model predictive control (MPC) can handle constraints, it heavily relies on accurate future information prediction, and its computational complexity increases exponentially in high-dimensional nonlinear systems.

[0004] Currently, deep reinforcement learning shows great potential in handling sequential decision-making problems for the optimal scheduling of multi-energy networks in buildings. However, standard reinforcement learning algorithms face the following technical bottlenecks when dealing with building energy management: First, the curse of dimensionality and optimization failure caused by the coupling of long and short timescales. Building systems encompass both long-timescale devices (such as cross-period energy migration in energy storage systems) and short-timescale devices (such as instantaneous power regulation in HVAC systems). Traditional deep reinforcement learning places all devices in the same decision space, leading to an exponential expansion of the state-action space, making it difficult for the agent to converge and easily getting trapped in local optima. Second, sparse reward signals and inconsistent timescales. Cross-period energy migration in energy storage systems is a long-term decision, while instantaneous power regulation in air conditioning is a short-term decision. Reinforcement learning with a single structure cannot take into account the coordination of both. Third, the separation between prediction and scheduling lacks safety margin. Traditional predictive control methods heavily rely on the accuracy of early load and weather forecasts. When faced with highly random photovoltaic fluctuations or disorderly access of electric vehicles, prediction errors can directly lead to subsequent scheduling commands exceeding limits; moreover, prediction models that minimize absolute errors cannot avoid the operational risks caused by underestimating errors.

[0005] To address the aforementioned problems, this invention proposes a hierarchical reinforcement learning-based method. This method aims to address the dimensionality curse and optimization failure caused by the coupling of long and short timescales through a hierarchical decision architecture, and introduces an extreme failure constraint mechanism to ensure the efficiency and reliability of the system in complex dynamic environments. Summary of the Invention

[0006] To address the problems in existing technologies for energy dispatching in complex buildings, such as the curse of dimensionality caused by the coupling of long and short time scales, the sparsity of reward signals due to long-range decision-making, and the lack of physical safety margin due to the disconnect between prediction and dispatching, this invention provides a collaborative control method and device for multi-energy networks in buildings based on hierarchical reinforcement learning. This method constrains the exploration process of reinforcement learning through a physical model and utilizes a hierarchical architecture to achieve task decomposition and hierarchical regulation with different time steps and control frequencies.

[0007] The technical solution of this invention is as follows: Step 1: Construct a physical model system for a building's multi-energy network. This physical model system uses equivalent thermal parameter equations to describe the evolution of indoor temperature and establishes dynamic load models for battery energy storage systems and electric vehicles. Step 2: Construct a dual-stream load prediction model based on residual reinforcement learning. Generate a baseline prediction value through a long short-term memory network, and output residual compensation coefficients by the proximal policy optimization agent. The baseline prediction value is fused with the residual compensation coefficients to obtain the final load prediction value. Among them, the parameters of the long short-term memory network are updated through prediction error, and the parameters of the policy network and value network of the proximal policy optimization agent are updated through asymmetric reward backpropagation.

[0008] Step 3: Based on the final load forecast sequence, photovoltaic forecast sequence, electricity price information sequence, battery energy storage system state of charge, indoor temperature, electric vehicle access status and remaining charging demand, construct a multi-dimensional heterogeneous state space to characterize the operating status of the building's multi-energy network; and construct a composite action space that includes HVAC power, battery energy storage system charging and discharging power and electric vehicle charging power.

[0009] Step 4: Design a decision framework based on hierarchical reinforcement learning. The framework includes a low-frequency high-level management network and a high-frequency low-level execution network. Both take a multi-dimensional state space as input. The high-level management network outputs the macroscopic target state of charge of the battery energy storage system. Under the constraint of the macroscopic target state of charge, the low-level execution network outputs the power execution commands of the multi-energy devices corresponding to the composite action space.

[0010] Step 5: Execute an evaluation system that includes a low-level soft protection mechanism and nonlinear rewards. When the power execution command causes the expected state to go out of bounds, the battery energy storage system will cut off the action and apply a constant penalty term. At the same time, the nonlinear function of thermal comfort exceeding the limit will be used to construct a penalty term to induce the system to generate a pre-scheduling strategy.

[0011] Step 6: Perform simulation closed-loop performance evaluation and scheduling, inputting the real-time state into the trained optimal hierarchical reinforcement learning model to output online scheduling instructions.

[0012] This invention provides a clear physical evolution benchmark for reinforcement learning through a physical model, effectively solving the drawback of black-box models that struggle to guarantee physical consistency. Simultaneously, the hierarchical architecture allows for the layered control of long-term economic objectives and short-term action execution with different time steps and control frequencies, significantly improving the learning efficiency of the agent in large-scale state spaces.

[0013] Furthermore, in step 1, when constructing the physical model system of the building's multi-energy network, the thermal inertia of the building itself is characterized as virtual flexible energy storage, and the equivalent thermal parameter equation for its indoor temperature evolution is:

[0014] In the formula, and They are respectively The indoor and outdoor temperatures at any given time. For the building's equivalent heat capacity, For the building's equivalent thermal resistance, For air conditioner energy efficiency ratio, This refers to the actual operating power of the air conditioner. is a dimensionless conversion constant.

[0015] The state of energy (SOC) of a battery energy storage system (BESS) satisfies linear dynamic constraints, and its dynamic load model is as follows:

[0016] In the formula, and Represent Time and The state of charge of the battery in the battery energy storage system at any given time. and They are respectively The charging power and discharging power at any given time. Indicates charge / discharge efficiency. The time step for calculation, and This corresponds to the battery's rated total capacity.

[0017] The system must meet the following hard physical constraints to operate: 1. Energy storage safety range: .

[0018] 2. Thermal comfort zone: Indoor temperature ℃.

[0019] 3. EV charging guarantee: The remaining charging capacity of electric vehicles must be met within the deadline before they leave the building.

[0020] Furthermore, the specific operation process of the residual reinforcement learning-based dual-flow load prediction model described in step 2 is as follows: A Markov state matrix is ​​constructed by fusing multiple features, including historical real load sliding sequences, weather forecast features, and time calendar features; this matrix is ​​used as the baseline state input to a long short-term memory network to obtain the baseline prediction value; simultaneously, the time-step difference sequence of this state matrix is ​​extracted as the dynamic observation state input to the proximal policy optimization agent to obtain residual compensation coefficients; the final load prediction value is obtained through joint calculation, and its residual fusion formula is as follows:

[0021] In the formula, for The final load forecast at any given time; It serves as a benchmark model for predicting long short-term memory networks; for Multi-dimensional characteristics at any given time (including historical load and meteorological data); The dynamic perturbation features are extracted after performing first-order time-step differencing on the multivariate features. This is the continuous residual compensation coefficient output by the proximal policy optimization (PPO) agent based on this dynamic perturbation characteristic.

[0022] The asymmetric reward calculation based on actual load refers to the following: Given that the risk of energy shortage caused by underestimating the predicted value in the building energy network is far greater than the cost of redundant capacity caused by overestimating the predicted value, when the final load prediction value calculated by the fusion is less than the actual load, the system is deemed to have an over-limit risk, and a high penalty coefficient is assigned to the agent. When the final load forecast is greater than or equal to the actual load, the system is determined to be within the safety envelope, and the agent is given a low penalty coefficient. The asymmetric penalty formula is as follows:

[0023] in, This represents the asymmetric reward value. This represents the final load forecast value calculated using the fusion method. This represents the actual load value. Indicates a high penalty coefficient. This indicates a low penalty coefficient. This establishes a redundant policy optimization space, guiding the agent to make predictive actions with physical safety margins.

[0024] Furthermore, in step 4, the long-range decision-making for energy migration across time periods in energy storage is coupled and conflicted with the short-range decision-making for instantaneous power allocation in air conditioning. This results in an exponentially expanding joint state-action space and extremely sparse long-term target reward signals, leading to low agent optimization efficiency and a high susceptibility to local optima. Therefore, within the hierarchical reinforcement learning-based decision framework, the high-level management network operates at a low frequency, responsible for outputting the macroscopic target state of charge (SOC) planning for the energy storage system based on future long-term time-of-use electricity prices and load forecasts. The low-level execution network operates at a high frequency, outputting specific power execution instructions for air conditioning, batteries, and electric vehicle charging stations at each time step under the constraints of the macroscopic target planning. The macroscopic energy storage target reward function issued by the high-level management network to the low-level execution network... for:

[0025] In the formula, This indicates the state of charge of macro-level targets issued by the high-level management network based on long-term planning. The preset target tracking weight coefficient is used to adjust the degree to which the underlying execution network obeys macro-level instructions.

[0026] Furthermore, step 4 addresses the high uncertainty caused by high-proportion photovoltaic fluctuations and random electric vehicle access, as well as the lack of forward-looking anti-interference capabilities in traditional state-action space construction, which makes it difficult to unify the dimensions of collaborative computing for heterogeneous devices. A decision framework based on hierarchical reinforcement learning is designed, comprising a low-frequency high-level management network and a high-frequency low-level execution network. The high-level management network takes the final load forecast sequence, photovoltaic forecast sequence, electricity price information sequence, and battery energy storage system state of charge (SOC) sequence from the multi-dimensional state space as input, and outputs the macroscopic target SOC of the battery energy storage system. The low-level execution network takes the indoor temperature, battery energy storage system SOC, electric vehicle access status and remaining charging demand, current electricity price, current photovoltaic output, current load forecast, and the macroscopic target SOC sequence from the multi-dimensional heterogeneous state space as input, and outputs power execution commands for the multi-energy devices corresponding to the composite action space under the constraints of the macroscopic target SOC sequence.

[0027] Furthermore, the evaluation system described in step 5, which includes an underlying soft protection mechanism and nonlinear rewards, is specifically as follows: When the macroscopic target state of charge output by the higher-level management network causes the battery state of charge expected by the lower-level execution network to exceed the limit, the lower-level execution network controls the battery energy storage system to forcibly limit the battery state of charge to within the safety boundary, and in the macroscopic energy storage target reward function... Apply a soft penalty of a fixed constant:

[0028] in, express Penalty for exceeding battery state of charge limits at all times. This represents a fixed soft penalty constant applied when the battery's state of charge exceeds the limit. and These represent the lower and upper limits of the safe state of charge of the battery, respectively.

[0029] This design can avoid reward mutations during training, reduce the cliff effect when the agent learns near the boundary, and ensure that the physical device does not perform dangerous actions.

[0030] Indoor temperature at time t Beyond the comfort range At that time, a nonlinear penalty term is constructed by using the square of the excess difference:

[0031] In this invention, the thermal comfort range constraint is: ℃, which is the upper limit of the set temperature. Set the lower limit of temperature This is to induce intelligent agents to turn on the air conditioning in advance for pre-cooling or pre-heating during periods of low electricity prices.

[0032] System operating costs It can be represented as:

[0033] in, Indicates time-of-use electricity pricing. This indicates the power purchased from the power grid. This represents the time step for the calculation.

[0034] For electric vehicle loads, a penalty is incurred if the target charging demand is not met when the vehicle leaves. :

[0035] in, This indicates the vehicle's target charging demand. This indicates the actual cumulative charging level when the vehicle leaves. This ensures that the intelligent agent reduces electricity costs without excessively delaying EV charging and affecting user travel needs.

[0036] The comprehensive reward function of the underlying execution network It can be represented as:

[0037] in, and These are the weighting coefficients for thermal comfort, electric vehicle demand, and energy storage safety constraints, respectively. Used to minimize operating costs and constraint penalties Used to track target SOCs issued by higher management.

[0038] Rewards for high-level management networks It can be constructed based on long-cycle operating costs, peak shaving effects, and underlying target execution deviations:

[0039] in, This indicates the length of the forecast time domain for high-level decision-making. Indicates time Operating costs Indicates the peak penalty cost. The weight coefficient of the peak penalty term is represented by this reward design. The higher-level network learns long-term economic strategies, while the lower-level network learns short-term safe execution strategies, thereby achieving multi-timescale collaborative optimization.

[0040] Furthermore, the step of performing closed-loop performance evaluation and online scheduling refers to disabling the random exploration mechanism during actual operation, and collecting real-time information on load, photovoltaic output, indoor temperature, battery state of charge, electric vehicle status, and electricity price. This information is then input into the trained and stored dual-flow load prediction model and hierarchical reinforcement learning decision framework. The high-level management network generates the macroscopic target state of charge of the battery energy storage system, and the low-level execution network outputs multidimensional action commands belonging to the composite action space based on the real-time multidimensional heterogeneous state space and the macroscopic target state of charge. After physical boundary truncation, power limiting, and actual power allocation, these multidimensional action commands form the actual control commands for HVAC, the battery energy storage system, and the electric vehicle charging equipment. Closed-loop performance evaluation and online correction are then performed based on actual electricity metering, thermal comfort, battery state of charge, and electric vehicle charging completion status, thereby forming a safe closed-loop real-time scheduling system.

[0041] Furthermore, the present invention also provides a building multi-energy network collaborative control system based on hierarchical reinforcement learning, which is used to implement the building multi-energy network collaborative control method based on hierarchical reinforcement learning described in the present invention.

[0042] Furthermore, the present invention also provides a building multi-energy network collaborative control device based on hierarchical reinforcement learning, wherein the device is equipped with a building multi-energy network collaborative control system based on hierarchical reinforcement learning.

[0043] Compared with the prior art, the present invention has the following beneficial effects: First, to address the issues of state space dimensionality expansion and reward sparsity caused by the coupling of long and short timescales, a two-layer decision-making architecture with high and low frequency decoupling is constructed. Spatiotemporal dimensionality reduction and phased objective constraints bridge the differences in equipment control frequencies, improving algorithm convergence speed and avoiding local optima. Second, to address the deficiency of traditional forecasting in scheduling's lack of operational safety margin, a dual-stream residual load forecasting mechanism is proposed. Through asymmetric evaluation that assigns high penalty weights to underestimating load, it proactively guides the generation of forecast sequences with safety safeguards, mitigating system over-limit risks from the source. Finally, to address the hidden danger of pure data-driven algorithms lacking physical boundary constraints, low-level action truncation protection and quadratic nonlinear penalties are introduced. While eliminating the risk of boundary policy mutations, this drives the system to autonomously generate forward-looking flexible scheduling strategies based on building thermal inertia, achieving a balance between physical safety and comprehensive economic benefits. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the process of the present invention.

[0045] Figure 2 This is a schematic diagram of the physical equivalent modeling of virtual flexible energy storage in an embodiment of the present invention.

[0046] Figure 3This is a schematic diagram of the dual-flow load prediction model based on residual reinforcement learning constructed in this embodiment of the invention.

[0047] Figure 4 This is a comparison chart of predicted load and actual load output by the dual-stream load prediction model based on residual reinforcement learning constructed in this embodiment of the invention.

[0048] Figure 5 This is a schematic diagram of the decision-making framework based on hierarchical reinforcement learning and the multidimensional heterogeneous state and action space constructed in this embodiment of the invention.

[0049] Figure 6 This is a diagram of the hierarchical reinforcement learning multidimensional tensor input-output architecture in an embodiment of the present invention.

[0050] Figure 7 The curves showing the changes in average round reward and loss function value of the hierarchical reinforcement learning model in this embodiment of the invention are shown.

[0051] Figure 8 This is a diagram illustrating the multi-objective collaborative optimization scheduling effect in an embodiment of the present invention.

[0052] Figure 9 This is a comparison chart of the cumulative costs of multi-energy control and scheduling of the microgrid environment of the office building based on the traditional optimization method and the method of the present invention in an embodiment of the present invention. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0054] This embodiment performs multi-energy collaborative optimization on a typical office building microgrid environment. The microgrid system includes a virtual flexible energy storage module consisting of a photovoltaic power generation array, a battery energy storage system (BESS), and a heating, ventilation, and air conditioning (HVAC) system, as well as a cluster of electric vehicle (EV) charging piles subject to deadline constraints.

[0055] like Figure 1 As shown, a collaborative control method and device for building multi-energy networks based on hierarchical reinforcement learning is described, and the specific process is as follows: S01 constructs a physical model system for building multi-energy networks, characterizing the thermal inertia of the building itself as virtual flexible energy storage.

[0056] Modeling a real-world smart building microgrid, such as... Figure 2 The system's physical topology is shown. The devices that facilitate the interaction of energy and information flows in the building microgrid include: a photovoltaic power generation array, HVAC units, a battery energy storage system (BESS), and multiple EV charging terminals.

[0057] When establishing the physical simulation environment, the equivalent thermal parameter (ETP) equation is used to describe the evolution of indoor temperature in buildings. Specifically, indoor temperature... The state update equation is as follows:

[0058] In the formula, For the building's equivalent heat capacity, The equivalent thermal resistance of the building is used as a model. This model equates the building's temperature regulation process to a virtual energy storage unit with "heat charging and discharging" capabilities. Simultaneously, based on employee commuting patterns, the access and departure times of EVs are set, constructing a funnel-shaped dynamic load model with a charging deadline.

[0059] S02, Construct a dual-stream load prediction model based on residual reinforcement learning to provide high-precision look-ahead features for the underlying execution network. The model structure is as follows: Figure 3 As shown.

[0060] The prediction module's operational logic consists of three core parts: multi-source feature input and state space, residual reinforcement learning two-stream prediction mechanism, and dynamic correction and asymmetric reward evaluation. First, the system extracts historical real load sliding sequences, meteorological forecast features (such as temperature and irradiance), and time calendar features (such as weekdays and times), while also receiving the autoregressive error state extracted from the previous prediction loop. These multi-source data are input together and subjected to multidimensional Markov state vector aggregation to form a unified state matrix.

[0061] Secondly, the aggregated state vectors are concurrently input into the residual reinforcement learning dual-stream prediction mechanism. One data stream is input into the traditional Long Short-Term Memory (LSTM) network base prediction model to provide a prediction baseline and generate initial baseline prediction values; the other data stream is concurrently input into the proximal policy optimization decision brain (PPO agent), which outputs the predicted action compensation coefficients.

[0062] Finally, the initial baseline prediction value generated by LSTM is fused with the prediction action compensation coefficient output by the PPO agent to calculate the final load prediction value, which is calculated using the following formula:

[0063] In the formula, for The final load forecast at any given time; It serves as a benchmark model for predicting long short-term memory networks; for Multidimensional state characteristics at any given time (including historical load and meteorological data); For proximal policy optimization (PPO) agents in The compensation coefficient for continuous actions based on the same state output at any given time.

[0064] After the actual environmental load is revealed, the system performs asymmetric reward calculation based on the deviation between the actual load and the predicted value. This involves applying different weights to penalties for underestimating and overestimating the load, and feeding this reward signal back to the proximal policy optimization decision-making engine to update the network parameters. Simultaneously, the autoregressive error state is extracted and fed back to the input, forming a closed-loop self-learning engine. The output load prediction is compared with the actual load as follows: Figure 4 .

[0065] S03, Construct a decision-making framework based on hierarchical reinforcement learning and a multidimensional heterogeneous state and action space.

[0066] To achieve multi-timescale collaboration, construct, such as Figure 5 The diagram illustrates a two-tiered decision-making framework. The high-level management network operates at a low frequency with a 1-hour step, outputting the macro-level target state of charge (SOC) plan for the energy storage system based on long-term time-of-use pricing and load forecasting. The low-level execution network operates at a high frequency with a 15-minute step, outputting power commands for specific devices under the constraint of the macro-level target. The high-level management network then applies the macro-level energy storage target reward function issued by the low-level execution network. for:

[0067] In the formula, This indicates the state of charge of macro-level targets issued by the high-level management network based on long-term planning. These are preset target tracking weight coefficients used to adjust the degree to which the underlying network obeys macroscopic instructions.

[0068] like Figure 6 The diagram shows a hierarchical reinforcement learning multidimensional tensor input-output architecture. The system's state space includes: current time characteristics, current environmental physical state (such as room temperature, SOC, and EV remaining demand), and the electricity price, photovoltaic power generation, and the basic load look-ahead prediction sequence for the next 24 time steps with Gaussian white noise. The action space consists of three symmetric continuous variables normalized to [-1,1], which are mapped to the HVAC setpoint power, energy storage charging and discharging power, and electric vehicle charging power, respectively.

[0069] S04, design the underlying soft protection mechanism and the evaluation system of non-linear rewards.

[0070] To address the boundary crossing and strategy issues in the reinforcement learning exploration process, this embodiment modifies the evaluation mechanism.

[0071] First, a low-level soft protection mechanism is introduced: when the battery charging / discharging command output by the low-level network causes the expected SOC to exceed the safety boundary of [0.1, 0.9], the low-level battery management system forcibly truncates it within the boundary and applies only a fixed soft penalty term in the reward function.

[0072] This design can avoid reward mutations during training, reduce the "cliff effect" when the agent learns near the boundary, and ensure that the physical device does not perform dangerous actions.

[0073] Secondly, a non-linear reward is introduced: when the indoor temperature exceeds the set comfort range... At this time, instead of the traditional linear penalty, a non-linear penalty term is constructed using the square of the excess difference:

[0074] By simultaneously amplifying the weight of real electricity costs in the reward function, the agent is induced to autonomously learn a strategy of turning on the air conditioner in advance for "pre-cooling / pre-heating" during periods of low electricity prices.

[0075] For electric vehicle loads, a penalty is incurred if the target charging demand is not met when the vehicle leaves:

[0076] This ensures that while reducing electricity costs, the intelligent agent will not excessively delay EV charging, thus affecting users' travel needs.

[0077] S05 performs closed-loop performance evaluation and online scheduling analysis.

[0078] The real-time sensor data is normalized and then input into the trained optimal model to output deterministic control commands.

[0079] The curves showing the change in average round reward and loss function value during training of this model are as follows: Figure 7 As shown.

[0080] To verify the effectiveness of the optimization scheme of this invention, an online scheduling simulation comparison analysis was conducted using extreme high-temperature conditions over seven consecutive days in summer. The multi-objective collaborative operation effect is as follows: Figure 8 As shown in the figure, the intelligent agent has successfully achieved low-charge, high-discharge energy storage. Figure 8 ), and the pre-cooling of air conditioners during periods of low electricity prices ( Figure 8 ), and the peak shaving and valley filling effect of the grid load on EV charging load ( Figure 8 The cumulative cost comparison chart of multi-energy control and scheduling for the microgrid environment of this office building based on traditional optimization methods and the method of this invention is shown in the figure below. Figure 9 As shown.

[0081] pass Figure 8 and Figure 9 Comparative analysis shows that, without sacrificing user thermal comfort and charging needs, this invention significantly reduces the operating cost of building microgrids and peak grid load by tapping into the potential of building thermal inertia and multi-device collaboration.

[0082] In summary, this invention effectively solves the multi-timescale coupled decision-making problem in complex building multi-energy networks through a layered architecture. The explicit introduction of physical constraints (ETP model) and failure protection mechanisms not only improves the interpretability of the decisions but also ensures safety and reliability in real-world industrial production environments. This solution, through software algorithm upgrades, can achieve significant energy-saving and carbon-reduction benefits, and has important practical value for the development of smart microgrids and buildings.

[0083] This application also provides an electronic device, including a processor and a memory.

[0084] The memory is used to store computer programs.

[0085] When the processor executes a program stored in the memory, it implements any of the methods described in this application.

[0086] In one possible implementation, the electronic device of this application embodiment further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0087] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0088] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0089] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0090] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0091] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the methods described in this application.

[0092] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in this application.

[0093] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0094] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0095] The various embodiments in this specification are described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0096] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A collaborative control method for multi-energy networks in buildings based on hierarchical reinforcement learning, characterized in that, This method constrains the exploration process of reinforcement learning through a physical model and utilizes a hierarchical architecture to achieve task decomposition and hierarchical control of different time steps and control frequencies; specifically, it includes the following steps: Step 1: Construct a physical model system for a building's multi-energy network. This physical model system uses equivalent thermal parameter equations to describe the evolution of indoor temperature and establishes dynamic load models for battery energy storage systems and electric vehicles. Step 2: Construct a dual-stream load forecasting model based on residual reinforcement learning. A baseline forecast value is generated using a long short-term memory network, and the residual compensation coefficient is output by the proximal policy optimization agent. The baseline forecast value is then fused with the residual compensation coefficient to obtain the final load forecast value. The specific implementation is as follows: A Markov state matrix is ​​constructed by fusing multiple features, including historical real load sliding sequences, weather forecast features, and time calendar features. This matrix is ​​used as the baseline state input to a long short-term memory network to obtain the baseline prediction value. Simultaneously, the time step difference sequence of this state matrix is ​​extracted as the dynamic observation state input to the proximal policy optimization agent to obtain residual compensation coefficients. The final load prediction value is obtained through joint calculation, and the residual fusion formula is as follows: ； In the formula, for The final load forecast at any given time; It serves as a benchmark model for predicting long short-term memory networks; for The multifaceted characteristics of time; The dynamic perturbation features are extracted after performing first-order time-step differencing on the multivariate features. The continuous residual compensation coefficients output by the near-end policy optimization agent based on this dynamic perturbation feature are used to optimize the near-end policy. The policy network and value network parameters of the near-end policy optimization agent are updated via asymmetric reward backpropagation. The asymmetric reward based on the actual load is calculated as follows: When the final load prediction from the fusion calculation is less than the actual load, the system is deemed to have a risk of exceeding limits, and the agent is given a high penalty coefficient. ; When the final load forecast is greater than the actual load, the system is determined to be within the safety envelope, and the agent is given a low penalty coefficient. The asymmetric penalty formula is as follows: ； in, This represents the asymmetric reward value. This represents the final load forecast value calculated using the fusion method. This represents the actual load value. Indicates a high penalty coefficient. Indicates a low penalty coefficient; Step 3: Based on the final load forecast sequence, photovoltaic forecast sequence, electricity price information sequence, battery energy storage system state of charge, indoor temperature, electric vehicle access status and remaining charging demand, construct a multi-dimensional heterogeneous state space to characterize the operating status of the building's multi-energy network; construct a composite action space that includes HVAC power, battery energy storage system charging and discharging power and electric vehicle charging power. Step 4: Design a decision framework based on hierarchical reinforcement learning. The framework includes a high-level management network that operates at low frequency and a low-level execution network that operates at high frequency. Both take a multi-dimensional state space as input. The high-level management network outputs the macroscopic target state of charge of the battery energy storage system. Under the constraint of the macroscopic target state of charge, the low-level execution network outputs the power execution commands of the multi-energy devices corresponding to the composite action space. Step 5: Execute an evaluation system that includes a low-level soft protection mechanism and nonlinear rewards. When the power execution command causes the expected state to go out of bounds, the action is cut off through the battery energy storage system and a constant penalty term is applied. At the same time, the penalty term is constructed using the nonlinear function of thermal comfort exceeding the limit to induce the system to generate a pre-scheduling strategy. Step 6: Perform simulation closed-loop performance evaluation and scheduling, inputting the real-time state into the trained optimal hierarchical reinforcement learning model to output online scheduling instructions.

2. The building multi-energy network cooperative control method based on hierarchical reinforcement learning according to claim 1, characterized in that, The energy state of the battery energy storage system described in step 1 satisfies linear dynamic constraints, and its dynamic load model is as follows: ； In the formula, and Represent Time and The state of charge of the battery in the battery energy storage system at any given time. and They are respectively The charging power and discharging power at any given time. Indicates charge / discharge efficiency. The time step for calculation, and This corresponds to the battery's rated total capacity.

3. The building multi-energy network cooperative control method based on hierarchical reinforcement learning according to claim 1, characterized in that, Step 4, within the hierarchical reinforcement learning-based decision-making framework, involves a high-level management network operating at low frequency, responsible for outputting the macro-level target state of charge (SOC) plan for the energy storage system based on future long-term time-of-use electricity prices and load forecasts. The low-level execution network operates at high frequency, outputting specific power execution commands for air conditioning, batteries, and electric vehicle charging stations at each time step, constrained by the macro-level target plan. The high-level management network then issues the macro-level energy storage target reward function to the low-level execution network. for: ； In the formula, This indicates the state of charge of macro-level targets issued by the high-level management network based on long-term planning. The preset target tracking weight coefficient.

4. The building multi-energy network cooperative control method based on hierarchical reinforcement learning according to claim 1, characterized in that, The high-level management network takes the final load forecast sequence, photovoltaic forecast sequence, electricity price information sequence, and battery energy storage system state of charge (SOC) sequence in the multi-dimensional state space as input, and outputs the macroscopic target SOC sequence of the battery energy storage system. The low-level execution network takes the indoor temperature, battery energy storage system SOC sequence, electric vehicle access status and remaining charging demand, current electricity price, current photovoltaic output, current load forecast, and macroscopic target SOC sequence in the multi-dimensional heterogeneous state space as input, and outputs the power execution commands of the multi-energy devices corresponding to the composite action space under the constraints of the macroscopic target SOC sequence.

5. The building multi-energy network cooperative control method based on hierarchical reinforcement learning according to claim 1, characterized in that, Step 5 is as follows: When the macroscopic target state of charge output by the higher-level management network causes the battery state of charge expected by the lower-level execution network to exceed the limit, the lower-level execution network controls the battery energy storage system to forcibly limit the battery state of charge to within the safety boundary, and in the macroscopic energy storage target reward function... Apply a soft penalty of a fixed constant: ； in, express Penalty for exceeding battery state of charge limits at any time. This represents a fixed soft penalty constant applied when the battery's state of charge exceeds the limit. and These represent the lower and upper safety limits of the battery's state of charge, respectively. Indoor temperature at time t Beyond the comfort range At that time, a nonlinear penalty term is constructed by using the square of the excess difference: ； in, This is the upper limit of temperature. The lower limit of temperature; System operating costs Represented as: ； in, Indicates time-of-use electricity pricing. This indicates the power purchased from the power grid. This refers to the time step for calculation; For electric vehicle loads, a penalty is incurred if the target charging demand is not met when the vehicle leaves. : ； in, This indicates the vehicle's target charging demand. This indicates the actual cumulative charging amount when the vehicle leaves; The comprehensive reward function of the underlying execution network Represented as: ； in, and These are the weighting coefficients for thermal comfort, electric vehicle charging demand, and energy storage safety constraints, respectively. Used to minimize operating costs and constraint penalties Used to track target SOCs issued by higher management; Rewards for the senior management network Constructed based on long-cycle operating costs, peak shaving effects, and underlying target execution deviations: ； in, This indicates the length of the forecast time domain for high-level decision-making. Indicates the operating cost at any given moment. Indicates the peak penalty cost. This represents the weighting coefficient of the peak penalty term.

6. A building multi-energy network collaborative control device based on hierarchical reinforcement learning, characterized in that, The device is equipped with the method as described in any one of claims 1-5.

Images