Energy-saving optimization method for thermal inertial sensing heating system based on deep reinforcement learning

By employing deep reinforcement learning methods, combined with IoT sensor networks and cross-scale feature fusion, the problem of thermal inertia regulation lag in centralized heating systems was solved, achieving high efficiency, energy saving, and adaptive control, thereby improving the comfort and safety of the heating system.

CN123069247APending Publication Date: 2026-11-10JIANGSU HUIREN NEW ENERGY TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611050079.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-15
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Central heating systems suffer from control lag and energy waste due to thermal inertia. Traditional control methods also suffer from severe overshoot and oscillation, which affect comfort.

Method used

A deep reinforcement learning-based approach is adopted. Multi-source data is collected through an Internet of Things sensor network, and data preprocessing and cross-scale feature fusion are performed to construct a deep reinforcement learning network, generate multiple candidate future action flows, perform future thermal response prediction and action flow evaluation, select the optimal target action flow, and achieve adaptive control.

Benefits of technology

It improves the accuracy of indoor temperature prediction, reduces energy consumption, adapts to changes in building thermal performance and user usage patterns, ensures good control performance under uncertain conditions, and guarantees safe system operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069247A_ABST
    Figure CN123069247A_ABST
Patent Text Reader

Abstract

This invention relates to an energy-saving optimization method for heating systems based on deep reinforcement learning and thermal inertial sensing, belonging to the field of heating energy-saving optimization technology. This invention decouples the multi-timescale thermal response process of a centralized heating system into fast response, medium response, and slow response layers, respectively employing RC lumped parameter models, state-space models, and LSTM for independent modeling and online identification, and achieving adaptive fusion through a cross-scale attention mechanism. The generated temporal lattice thermal state vector can comprehensively and accurately represent the complete thermal state of a building at any given time, reducing the mean absolute error of indoor temperature prediction to within a certain range. Compared to the traditional single RC model, this improves prediction accuracy and saves energy consumption under the same outdoor climate conditions and indoor comfort requirements. Through the collaborative online update of the thermal comfort geometric manifold and the Actor-Critic policy network, it can adapt to the time-varying characteristics of building thermal performance and changes in user usage patterns, ensuring that control performance is not affected by time decay.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of energy-saving optimization technology for heating systems, and in particular to an energy-saving optimization method for heating systems based on thermal inertial sensing using deep reinforcement learning. Background Technology

[0002] Central heating systems refer to a heating method that supplies heat energy to users within a certain area through a heating network using steam or hot water generated by a centralized heat source. A typical central heating system consists of three main parts: the heat source, the heating network, and the users. The heat source includes various forms such as combined heat and power plants, district boiler rooms, industrial waste heat, geothermal energy, and solar energy; the heating network transmits heat to each user terminal through primary networks, heat exchange stations, and secondary networks. Central heating has advantages such as significant economies of scale and high energy utilization efficiency, providing a stable and continuous heat source for a large number of users. When the temperature of the heating medium (hot water) changes, the heat needs to overcome the thermal resistance of each structural layer and heat the heat capacity of each layer before it can affect the indoor air temperature. This long lag characteristic of "control action first, temperature response later" makes a central heating system a typical pure lag, high inertia, and nonlinear thermal object.

[0003] Central heating systems have significant thermal inertia (thermal response delay can reach 2 to 6 hours). Traditional control methods adjust based on current temperature feedback, resulting in serious "overshooting" and "oscillation" problems, leading to energy waste and decreased comfort. Summary of the Invention

[0004] This invention overcomes the shortcomings of existing technologies and provides an energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: The first aspect of this invention provides an energy-saving optimization method for a thermal inertial sensing heating system based on deep reinforcement learning, comprising the following steps: An Internet of Things (IoT) sensor network is deployed inside the target building to collect multi-source data and perform data preprocessing on the multi-source data. The building thermal response process is decomposed into three time scale levels by combining the preprocessed multi-source data information, and cross-scale feature fusion is performed on the three time scale levels to generate a time lattice thermal state vector. A deep reinforcement learning network is constructed, and the time lattice hot state vector is fused into the state space of the deep reinforcement learning network. Multiple candidate future action flows are generated using the deep reinforcement learning network. Future thermal response prediction and candidate action flow evaluation are performed based on time lattice thermal state vector and multiple candidate future action flows. The candidate action flow with the highest comprehensive score is selected as the optimal target action flow.

[0006] Furthermore, in the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning, an Internet of Things (IoT) sensor network is deployed inside the target building. This IoT sensor network is used to collect multi-source data, and the multi-source data is preprocessed, specifically including: An Internet of Things (IoT) sensor network is deployed inside the target building to collect indoor environmental parameters, heating system operating parameters, outdoor environmental parameters, building static parameters, and user-authorized personnel activity parameters. An anomaly detection algorithm based on local outlier factors is used to identify and mark data points that exceed a preset physical range or have abnormal rates of change in the multi-source data information. For missing data, linear interpolation and weighted average based on historical data of the same period are used to fill in the missing data. A low-pass filter is used to filter out high-frequency noise in the temperature signal and retain the low-frequency dominant component of the thermal response process. Data sources with different sampling frequencies are aligned to a unified second-level time granularity using timestamps to obtain preprocessed multi-source data information.

[0007] Furthermore, in the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning, the building thermal response process is decomposed into three time-scale levels by combining preprocessed multi-source data information, specifically: The building thermal response process is decomposed into three time scale levels: the first level, the second level, and the third level. For the first level, the equivalent heat capacity-thermal resistance lumped parameter model is used to establish a first-order differential equation with the indoor air temperature state variable to extract the thermal response characteristics of the first level. For the second level, a state-space model is adopted, with the surface temperature of the inner wall and the temperature of the ground structure layer as state variables, to establish a second-order state equation and extract the thermal response characteristics of the second level. For the second level, a long short-term memory network is used. The outdoor temperature sequence, solar radiation sequence and water supply temperature sequence of the past preset hours are input to predict the internal temperature distribution of the outer envelope and extract the third-level thermal response features.

[0008] Furthermore, in the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning, cross-scale feature fusion is performed across three time scale levels to generate a time-lattice thermal state vector, specifically: The first layer of thermal response features, the second layer of thermal response features, and the third layer of thermal response features are encoded respectively to obtain the first feature vector, the second feature vector, and the third feature vector respectively. The cross-scale attention weights are calculated based on the first feature vector, the second feature vector, and the third feature vector, and the fused hot state vector is calculated based on the cross-scale attention weights, and the time lattice hot state vector is generated.

[0009] Furthermore, in the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning, a deep reinforcement learning network is constructed, and the time lattice thermal state vector is fused into the state space of the deep reinforcement learning network. Multiple candidate future action flows are generated using the deep reinforcement learning network, specifically: A deep reinforcement learning network is constructed, and the time lattice thermal state vector is fused into the state space of the deep reinforcement learning network. The water supply temperature adjustment, the circulation pump frequency adjustment, and the start-stop time offset are set as three continuous control variables to construct the action space. The data in the state space is input into the Actor network, which outputs the probability distribution of actions. The data in the state space is input into the Critic network, which outputs the state-action value. The training algorithm uses SAC and introduces an entropy regularization term to encourage exploration. A baseline action flow is generated based on the current state using an Actor network. Then, exploration noise of different magnitudes is added to the baseline action flow to generate multiple candidate future action flows. A matrix-based point process is used to perform diversity screening on the generated multiple candidate action flows to ensure that the similarity between any two action flows does not exceed a preset threshold.

[0010] Furthermore, in the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning, future thermal response prediction and candidate action flow evaluation are performed based on the time lattice thermal state vector and multiple candidate future action flows. The candidate action flow with the highest comprehensive score is selected as the optimal target action flow, specifically: Construct a differentiable prediction model, input the current time lattice thermal state vector and candidate future action flow, and use the differentiable prediction model to predict the indoor temperature at each future time step; Meanwhile, for each candidate future action flow, a baseline scenario, an optimistic scenario, and a pessimistic scenario are generated. In each scenario, the indoor text and image sequence and cumulative energy consumption within a preset time are output to construct a geometric manifold of thermal comfort information. The indoor temperature and temperature change rate at each moment are mapped to a point on the manifold. The point representation of the temperature sequence predicted by each candidate motion flow on the thermal comfort information geometric manifold is combined to obtain the corresponding trajectory point sequence on the manifold. The average geometric distance from the trajectory to the comfort zone is calculated based on the trajectory point sequence. A comprehensive score is calculated for each candidate motion flow based on the average geometric distance, and the candidate motion flow with the highest comprehensive score is selected as the optimal target motion flow.

[0011] Furthermore, the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning also includes: The first action in the obtained optimal target action flow is immediately sent to the actuator of the central heating system. After each action is executed, the actual indoor temperature change and actual energy consumption feedback data are collected. The actual indoor temperature change is compared with the predicted temperature change, the prediction error is calculated, and the parameters of each level of the model are updated by online gradient descent using the prediction error. The actual execution trajectory is stored as an experience sample in the experience replay pool. Every preset step, a batch of samples is randomly sampled from the experience replay pool to update the parameters of the Actor network and the Critic network. The training objective is to maximize the expected cumulative reward.

[0012] This invention addresses the shortcomings of the prior art and has the following beneficial effects: This invention decouples the multi-timescale thermal response process of a centralized heating system into fast-response, medium-response, and slow-response layers. These layers are independently modeled and identified online using an RC lumped-parameter model, a state-space model, and an LSTM, respectively, and adaptively fused through a cross-scale attention mechanism. The generated temporal lattice thermal state vector comprehensively and accurately represents the complete thermal state of a building at any given time, reducing the mean absolute error of indoor temperature prediction to within a certain range, thus improving prediction accuracy compared to traditional single RC models. Based on the continuous self-optimization capability of reinforcement learning, the system can optimize the control strategy daily while ensuring thermal comfort, saving energy under the same outdoor climate conditions and indoor comfort requirements.

[0013] This invention utilizes the collaborative online updating of thermal comfort geometric manifolds and Actor-Critic strategy networks to adapt to the time-varying characteristics of building thermal performance (such as insulation material aging and decreased window sealing) and changes in user usage patterns (such as adjustments to occupants' schedules and changes in indoor thermal disturbances), ensuring that control performance does not decay over time. Furthermore, the multi-scenario prediction mechanism allows the selection of the optimal action flow to consider optimistic, baseline, and pessimistic external conditions, ensuring good control performance even under uncertainties such as weather forecast deviations and sudden changes in solar radiation. The embedded safety protection mechanism further guarantees the system's safe operation, achieving a unity of intelligent decision-making and safe operation. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained from these drawings without creative effort.

[0015] Figure 1 The overall flowchart of the energy-saving optimization method for thermal inertial sensing heating system based on deep reinforcement learning is shown. Detailed Implementation

[0016] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0017] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0018] like Figure 1 As shown, the first aspect of this invention provides an energy-saving optimization method for a thermal inertial sensing heating system based on deep reinforcement learning, comprising the following steps: Deploy an IoT sensor network inside the target building, use the IoT sensor network to collect multi-source data, and perform data preprocessing on the multi-source data. The building thermal response process is decomposed into three time scale levels by combining the preprocessed multi-source data information, and cross-scale feature fusion is performed on the three time scale levels to generate a time lattice thermal state vector. Construct a deep reinforcement learning network, fuse the temporal lattice hot state vector into the state space of the deep reinforcement learning network, and use the deep reinforcement learning network to generate multiple candidate future action flows; Future thermal response prediction and candidate action flow evaluation are performed based on time lattice thermal state vector and multiple candidate future action flows. The candidate action flow with the highest comprehensive score is selected as the optimal target action flow.

[0019] It should be noted that this invention decouples the multi-timescale thermal response process of the centralized heating system into fast response, medium response, and slow response layers, which are independently modeled and identified online using RC lumped parameter models, state-space models, and LSTM, respectively, and adaptively fused through a cross-scale attention mechanism. The generated time-lattice thermal state vector can comprehensively and accurately characterize the complete thermal state of the building at any given time, reducing the mean absolute error of indoor temperature prediction to within a certain range, thus improving prediction accuracy compared to the traditional single RC model. Based on the continuous self-optimization capability of reinforcement learning, the system can optimize the control strategy daily while ensuring thermal comfort, saving energy consumption under the same outdoor climate conditions and indoor comfort requirements. Through the collaborative online update of the thermal comfort geometric manifold and the Actor-Critic policy network, it can adapt to the time-varying characteristics of building thermal performance (such as aging of insulation materials, decreased window sealing, etc.) and changes in user usage patterns (such as adjustments to occupants' work and rest schedules, changes in indoor thermal disturbance, etc.), ensuring that control performance is not affected by time decay. Furthermore, the multi-scenario prediction mechanism ensures that the selection of the optimal action flow takes into account optimistic, baseline, and pessimistic external conditions, maintaining good control performance even under uncertainties such as weather forecast deviations and sudden changes in solar radiation. The embedded safety protection mechanism further guarantees the safe operation of the system, achieving a unity between intelligent decision-making and safe operation.

[0020] Furthermore, in the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning, an IoT sensor network is deployed inside the target building to collect multi-source data. This multi-source data is then preprocessed, specifically including: Deploy an Internet of Things (IoT) sensor network inside the target building to collect indoor environmental parameters, heating system operating parameters, outdoor environmental parameters, building static parameters, and user-authorized personnel activity parameters. An anomaly detection algorithm based on local outlier factors is used to identify and mark data points that exceed a preset physical range or have abnormal rates of change in multi-source data. For missing data, linear interpolation and weighted average based on historical data of the same period are used to fill in the missing data. A low-pass filter is used to filter out high-frequency noise in the temperature signal and retain the low-frequency dominant component of the thermal response process. Data sources with different sampling frequencies are aligned to a unified second-level time granularity using timestamps to obtain preprocessed multi-source data information.

[0021] It should be noted that indoor environmental parameters include the indoor temperature and relative humidity of each room; central heating system operating parameters include supply water temperature, return water temperature, circulating water flow rate, and floor surface temperature; outdoor environmental parameters include outdoor temperature, solar radiation intensity, wind speed, and wind direction, which can be obtained through on-site small weather stations or by accessing third-party weather APIs; building static parameters include building area, floor height, building envelope type and thermal parameters (wall heat transfer coefficient, window type and area, floor structure and thermal resistance), and building orientation; occupancy data refers to the occupancy status of each room authorized by the user, obtained through infrared sensors or WiFi probes to ensure privacy.

[0022] The anomaly detection algorithm based on Local Outlier Factor (LOF) was employed to identify and label data points that exceeded reasonable physical ranges (e.g., indoor temperatures below -10℃ or above 50℃) or exhibited abnormal rates of change (e.g., temperature jumps exceeding 5℃ / min). For short-term missing data (<5 minutes), linear interpolation was used to fill in the gaps; for long-term missing data (≥5 minutes), a weighted average method based on historical data from the same period (data from the previous 7 days) was used. After preprocessing, the data was divided into two parts: one for real-time updates of the thermal inertial resolver (sliding window, containing data from the past 72 hours), and the other for experiential replay training of the reinforcement learning policy network.

[0023] Furthermore, in the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning, the building thermal response process is decomposed into three time-scale levels by combining preprocessed multi-source data information, specifically: The building thermal response process is decomposed into three time scale levels: the first level, the second level, and the third level. For the first level, the equivalent heat capacity-thermal resistance lumped parameter model is used to establish a first-order differential equation with the indoor air temperature state variable to extract the thermal response characteristics of the first level. It should be noted that indoor air temperature is used. For the state variables, establish a first-order differential equation:

[0024] in The equivalent heat capacity of indoor air. Ground temperature, Outdoor temperature and These are the equivalent thermal resistances of the ground and the walls, respectively. Indoor thermal disturbances (personnel, equipment, etc.) are identified online using the recursive least squares method. , , This method is used to obtain the multi-source thermal response characteristics in the first level, including the parameters mentioned above.

[0025] For the second level, a state-space model is adopted, with the surface temperature of the inner wall and the temperature of the ground structure layer as state variables, to establish a second-order state equation and extract the thermal response characteristics of the second level. It should be noted that the second-order state equations in continuous time satisfy the following relationship:

[0026] in, It is a state vector Regarding time The first derivative (rate of change) is expressed in °C / s (degrees per second). It is the state vector corresponding to the surface temperature of the inner wall and the temperature of the ground structural layer; It is the self-coupling and dissipation coefficient within the system, and is generally taken as 1. It is The matrix that determines the external input vector Each component in the equation is related to the rate of change of state. The driving force intensity, with dimensions 1 / s (the reciprocal of a second), is the water supply temperature. The direct influence coefficient on the temperature change rate of the ground structure layer is determined by factors such as the higher the thermal conductivity of the pipe material, the thinner the pipe wall, the smaller the pipe spacing, and the better the thermal conductivity of the filling layer, resulting in a smaller equivalent thermal resistance. The larger. External driving forces for heating systems, such as underfloor heating water supply temperature and indoor air temperature.

[0027] in, The following relationship must be satisfied:

[0028] in, Satisfy the following relation

[0029] in, The equivalent heat capacity of the ground structure layer. Equivalent heat capacity of interior walls and furniture. The equivalent thermal resistance of the underfloor heating pipes and structural layer. The equivalent thermal resistance between the ground and the air. The equivalent thermal resistance between the interior wall and the air.

[0030] For the second level, a long short-term memory network is used. The outdoor temperature sequence, solar radiation sequence and water supply temperature sequence of the past preset hours are input to predict the internal temperature distribution of the outer envelope and extract the third-level thermal response features.

[0031] Furthermore, in the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning, cross-scale feature fusion is performed across three time scale levels to generate a time-lattice thermal state vector, specifically: The first layer of thermal response features, the second layer of thermal response features, and the third layer of thermal response features are encoded respectively to obtain the first feature vector, the second feature vector, and the third feature vector respectively. The cross-scale attention weights are calculated based on the first, second, and third eigenvectors, and the fused hot state vector is calculated based on the cross-scale attention weights, and the time lattice hot state vector is generated.

[0032] Furthermore, in the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning, a deep reinforcement learning network is constructed, and the time lattice thermal state vector is fused into the state space of the deep reinforcement learning network. Multiple candidate future action flows are generated using the deep reinforcement learning network, specifically: A deep reinforcement learning network is constructed, and the time lattice thermal state vector is fused into the state space of the deep reinforcement learning network. The water supply temperature adjustment, the circulation pump frequency adjustment, and the start-stop time offset are set as three continuous control variables to construct the action space. The data in the state space is input into the Actor network, which outputs the probability distribution of actions. The data in the state space is input into the Critic network, which outputs the state-action value. The training algorithm uses SAC and introduces an entropy regularization term to encourage exploration. A baseline action flow is generated based on the current state using an Actor network. Then, exploration noise of different magnitudes is added to the baseline action flow to generate multiple candidate future action flows. A matrix-based point process is used to perform diversity screening on the generated multiple candidate action flows to ensure that the similarity between any two action flows does not exceed a preset threshold.

[0033] Among them, the Actor network (policy network) is responsible for input states. Output action The probability distribution (for a continuous action space, the mean and standard deviation of the output Gaussian distribution). Network structure: Input layer → Fully connected layer (256-dimensional, ReLU) → Fully connected layer (128-dimensional, ReLU) → Output layer (mean and log standard deviation of 3 actions).

[0034] Critic Network (Value Network): Input State and actions Output state-action value Network structure: Input layer → Fully connected layer (256-dimensional, ReLU) → Fully connected layer (128-dimensional, ReLU) → Output layer (1-dimensional Q-value).

[0035] The Actor network first generates a baseline action flow based on the current state (by applying the Actor network 16 times consecutively, each time using the predicted next state as input). Then, exploration noise of varying magnitudes is added to the baseline action flow (using the Ornstein-Uhlenbeck process to generate time-related noise sequences) to generate multiple diverse candidate action flows. A determinant point process (DPP) is used to filter the diversity of the generated candidate action flows, ensuring that the similarity between any two action flows does not exceed a threshold (cosine similarity < 0.7), thus avoiding redundancy in the candidate set.

[0036] It should be noted that, based on the continuous self-optimization capability of reinforcement learning, the system can optimize the control strategy daily while ensuring thermal comfort, saving energy under the same outdoor climate conditions and indoor comfort requirements. Through the collaborative online updates of the thermal comfort geometric manifold and the Actor-Critic policy network, the system can adapt to the time-varying characteristics of building thermal performance (such as aging of insulation materials, decreased window sealing, etc.) and changes in user usage patterns (such as adjustments to occupants' work and rest schedules, changes in indoor thermal disturbance, etc.), ensuring that the control performance does not decay over time.

[0037] Furthermore, in the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning, future thermal response prediction and candidate action flow evaluation are performed based on the time lattice thermal state vector and multiple candidate future action flows. The candidate action flow with the highest comprehensive score is selected as the optimal target action flow, specifically: Construct a differentiable prediction model, input the current time lattice thermal state vector and candidate future action flows, and use the differentiable prediction model to predict the indoor temperature at each future time step; Input the current thermal vector and candidate action flow The indoor temperature at each future time step is predicted through hierarchical recursion. The differentiable prediction model satisfies the following relationship:

[0038] For the first The time-lattice thermal state vector of the step, For the first The time-lattice thermal state vector of the step, For the first The first candidate action in the flow Step control actions, For the first outdoor air temperature, For the first The intensity of solar radiation at each step, The hot state transition function is a nonlinear mapping function. The features updated at the three levels are re-fused through a cross-attention mechanism to output a new hot state vector. The design of this function ensures that the prediction process is simultaneously constrained by both "physical laws (first-level / second-level mechanism model)" and "data-driven (third-level learning model)".

[0039] For the first Step-by-step prediction of indoor air temperature, For the first The time-lattice thermal state vector is given by step, where g represents the mapping function from thermal state to temperature, which is a readout function. In its implementation, it includes a fixed linear projection layer. (in For the weight vector, (For bias), or more precisely, directly take the air temperature state variable from the first-layer RC model. Since the first-layer RC model itself is constructed with indoor air temperature as the primary state, therefore The function physically corresponds directly to the first hidden state. The components are extracted and fine-tuned using the second layer for radiative heat transfer. This ensures that the output indoor temperature incorporates both the rapid response of convective heat transfer and the delayed effect of ground radiation.

[0040] Meanwhile, for each candidate future action flow, a baseline scenario, an optimistic scenario, and a pessimistic scenario are generated. In each scenario, the indoor text and image sequence and cumulative energy consumption within a preset time are output to construct a geometric manifold of thermal comfort information. The indoor temperature and temperature change rate at each moment are mapped to a point on the manifold. The point representation of the temperature sequence predicted by each candidate motion flow on the thermal comfort information geometric manifold is combined to obtain the corresponding trajectory point sequence on the manifold. The average geometric distance from the trajectory to the comfort zone is calculated based on the trajectory point sequence. A comprehensive score is calculated for each candidate motion flow based on the average geometric distance, and the candidate motion flow with the highest comprehensive score is selected as the optimal target motion flow.

[0041] It should be noted that, considering the uncertainties in external conditions such as outdoor temperature and solar radiation, three prediction scenarios are generated for each action flow: a baseline scenario (using the definitive values ​​from weather forecasts), an optimistic scenario (outdoor temperature 2°C higher than forecast), and a pessimistic scenario (outdoor temperature 2°C lower than forecast). Each scenario outputs the indoor temperature sequence and cumulative energy consumption for the next 4 hours. This multi-scenario prediction mechanism ensures that the selection of the optimal action flow takes into account optimistic, baseline, and pessimistic external conditions, maintaining good control performance even under uncertainties such as weather forecast deviations and sudden changes in solar radiation. The embedded safety protection mechanism further guarantees the safe operation of the system, achieving a unity of intelligent decision-making and safe operation.

[0042] Furthermore, the energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning also includes: The first action in the obtained optimal target action flow is immediately sent to the actuator of the central heating system. After each action is executed, the actual indoor temperature change and actual energy consumption feedback data are collected. The actual indoor temperature change is compared with the predicted temperature change, the prediction error is calculated, and the parameters of each level of the model are updated by online gradient descent using the prediction error. The actual execution trajectory is stored as an experience sample in the experience replay pool. At each step, a batch of samples is randomly sampled from the experience replay pool to update the parameters of the Actor network and the Critic network. The training objective is to maximize the expected cumulative reward.

[0043] It should be noted that through the coordinated online updates of the three components—thermal inertial analyzer, thermal comfort geometric manifold, and Actor-Critic strategy network—the system can adapt to the time-varying characteristics of building thermal performance (such as aging of insulation materials and decreased window sealing) and changes in user usage patterns (such as adjustments to people's work and rest schedules and changes in indoor thermal disturbance). Long-term operation tests show that the system saves additional energy after one year of operation compared to the initial deployment, and the control performance does not decay over time.

[0044] In addition, this method also includes: Each room in the building is defined as a graph node, and the node features contain the temporal lattice thermal state vector of that room. The area, orientation, and occupancy status of the rooms are considered, and the interior walls between the rooms are defined as edges of the graph. Edge features include wall area, heat transfer coefficient, and current door / window opening / closing status. The entire building's centralized heating system is modeled as a heterogeneous graph, using a multi-head graph attention network for inter-node information transfer. Through the attention mechanism, the agent in each room can perceive the current thermal state of adjacent rooms and the direction and intensity of heat transfer that may occur through the interior walls. Multi-zone underfloor heating control is modeled as a Markov decision process. The state space includes the actual thermal state of all rooms, and the observation space includes a summary of the neighboring thermal states aggregated from the room's own sensor data through a multi-head graph attention network. The action space includes independent water supply temperature and flow rate adjustments for each room. A multi-agent proximal policy optimization algorithm is used for training, and a counterfactual baseline is introduced. During training, for each agent, the policies of other agents are fixed, and the changes in global reward when the agent takes different actions are evaluated, thereby estimating the marginal contribution of each agent to the global goal. During the inference phase, agents in each room make independent decisions and perceive the state of their neighbors through the attention weights output by the multi-head graph attention network. When a decision in a certain room may lead to excessive heat transfer to its neighbors, a penalty term is added to the action output to encourage cooperation rather than competition.

[0045] It should be noted that by combining graph attention networks with multi-agent reinforcement learning for multi-zone collaborative control of underfloor heating, this invention applies it to building thermal coupling modeling, enabling agents in each room to "see" the thermal status of their neighbors without exposing their privacy data. Furthermore, this invention transforms this into an opportunity for collaborative optimization, utilizing heat transfer between adjacent rooms to achieve "residual heat sharing," thereby reducing overall energy consumption while ensuring comfort in each room.

[0046] The thermal comfort information geometric manifold is constructed based on "comfort labels" from historical data. However, these labels are derived from pre-defined temperature range assumptions and do not truly reflect the personalized thermal preferences of specific users. Studies have shown that different individuals have significantly different comfort perceptions of the same thermal environment, and there is often a discrepancy between assessments based on mathematical models such as PMV and subjective thermal sensations. Current technologies lack a mechanism to integrate personalized physiological feedback into the dynamic updating of the thermal comfort manifold; therefore, this method also includes: A personalized classifier is constructed by collecting skin temperature, heart rate variability, skin conductance level, and behavioral data from smart wearable devices or non-contact sensors. The inputs are environmental parameters and skin temperature, heart rate variability, skin conductance level, and behavioral data, and the output is a personalized thermal comfort probability. The training data sources include user-generated five-level thermal sensation voting ("cold / cool / comfortable / warm / hot") on the mobile app, and user-manual temperature control panel adjustments, with raising the temperature being implicitly labeled as "cold" and lowering the temperature as implicitly labeled as "hot". The classifier adopts an online incremental learning strategy. Each time a new label data is received, it performs incremental learning locally, expanding the geometric manifold of thermal comfort information into a personalized dynamic manifold, based on the initial thermal comfort manifold constructed from the group average thermal comfort probability data. Using the comfort probability output by the personalized classifier, the baseline manifold is subjected to distance-preserving deformation. On the manifold, regions with comfort probabilities higher than the threshold probability value are expanded into comfort regions, while regions with comfort probabilities lower than the threshold probability value are compressed into regions. The geodesic distance on the personalized manifold is calculated and embedded into the reward function of reinforcement learning.

[0047] It should be noted that the deformed personalized manifold distance is directly embedded into the reinforcement learning reward function:

[0048] in It refers to the geodesic distance on the personalized manifold. This means that for users who prefer warmth, the system allows for slightly higher temperatures without penalty; for users who prefer cooler temperatures, the system allows for slightly lower temperatures, maximizing energy savings while ensuring individual comfort. This is a comfort weighting coefficient, with a value ranging from 0.3 to 0.7. This is the energy consumption weighting coefficient, with a typical value range of 0.2-0.6. This is the stability weighting coefficient, with a value ranging from 0.1 to 0.3. To predict cumulative energy consumption; As a stability penalty item, The negative value of the sum of squares of the changes in action between adjacent control moments was calculated. The greater the change in action (such as sudden rises and falls in water supply temperature), the more negative the value (the heavier the penalty); the smaller the change in action (such as a steady and gradual change in water supply temperature), the closer the value is to 0 (the lighter the penalty).

[0049] It should be noted that this invention integrates differential geometric manifolds with personalized physiological feedback, thereby making personalized comfort modeling a "deformation" of the manifold, which maintains the geometric interpretability of thermal comfort assessment and achieves individualized adaptation.

[0050] In practical applications: First, an Internet of Things (IoT) sensor network is deployed within the target building. This sensor network is the foundation of this invention, and its deployment principle is to acquire the maximum amount of information at the lowest cost. In this embodiment, the target building is a residential building with a typical building envelope and a floor area of ​​120 square meters. Specifically, the following sensors are deployed in the core areas of the building (such as the living room and master bedroom): Indoor environmental monitoring unit: includes a digital temperature sensor with an accuracy of ±0.2℃ and a humidity sensor with an accuracy of ±3%RH, used to collect indoor temperature (T_in) and relative humidity (RH_in), with a sampling frequency set to 1Hz.

[0051] Heating system monitoring unit: Insertion temperature sensors and ultrasonic flow meters are installed at the manifold to collect supply water temperature (T_supply), return water temperature (T_return), and circulating water flow (Q_flow), with the sampling frequency set to 0.1Hz.

[0052] Outdoor environmental monitoring unit: A small weather station is set up in the north-facing shaded area of ​​the building, or connected to a third-party meteorological API via a network interface to obtain outdoor temperature (T_out), solar radiation intensity (I_solar), wind speed (WS), and wind direction (WD), with a sampling frequency of 0.05Hz.

[0053] During the data preprocessing stage, perform the following operations: Anomaly detection and removal: The Local Outlier Factor (LOF) algorithm is used for anomaly detection. The k-distance is set to 20, meaning each data point is compared with its 20 nearest neighbors. If the indoor temperature data at a certain moment is 55℃, its LOF value is calculated. If the LOF value is greater than a preset threshold of 2.0, it is identified as an anomaly and marked.

[0054] Missing value handling: For short-term (<5 minutes) data loss from the sensor, a linear interpolation method is used. For example, if the temperature data at time t and t+1 are normal, but the data at time t+0.5 is missing, the average of the two values ​​is used to fill the missing data. For long-term (≥5 minutes) data loss, a historical data imputation method is used, taking the weighted average of data from the same time point over the past 7 days, with the weighting coefficients decaying exponentially with time (e.g., decay factor λ=0.8).

[0055] Data filtering and alignment: A second-order low-pass Butterworth filter with a cutoff frequency of 0.01Hz is used to filter out high-frequency noise caused by device switching in the temperature signal. All sensor data are resampled according to timestamps to unify the time granularity to 1 second.

[0056] After preprocessing, the data is copied into two copies: one data stream (sliding window, capacity of the past 72 hours) is input into the thermal inertia analysis module; the other data stream is used to build the experience replay buffer for training reinforcement learning agents.

[0057] Multi-scale thermal inertia analysis and feature fusion stage This stage is the core of the invention, which uses a "time lattice" structure to characterize the thermal dynamics of buildings in multiple dimensions.

[0058] First level: Fast response layer modeling. This layer focuses on indoor air temperature, with a thermal response time constant on the order of minutes. A first-order RC (heat capacity-thermal resistance) lumped parameter model is used.

[0059] in, Indoor thermal disturbances include heat dissipation from personnel and equipment. In this embodiment, recursive least squares (RLS) is used to identify parameters online. The forgetting factor λ of the RLS algorithm is set to 0.99 to balance the dependence on historical and recent data. By solving, the equivalent heat capacity at the current moment can be obtained in real time. and thermal resistance , These parameters constitute the first feature vector. .

[0060] Second level: Modeling the intermediate response layer This layer targets the temperature of the ground structure and interior wall surfaces, with a thermal response time constant on the order of hours. A second-order state-space model is adopted, with state vectors... Input vector System matrix and input matrix The expression for has been given in the previous technical solution.

[0061] In this embodiment, the heat capacity parameter in the matrix (Equivalent heat capacity of ground structural layer) and (Equivalent heat capacity of interior walls) Initial values ​​are calculated using the material layer thickness, density, and specific heat capacity from architectural drawings, and then corrected online using a Kalman filter algorithm. By discretizing this second-order equation (sampling period Ts = 60 seconds) and solving it, the predicted temperature of the ground structure layer can be obtained. The second feature vector extracted in this prediction process... It contains delayed thermal response information from the ground-based thermal storage body.

[0062] Third level: Slow response layer modeling This layer addresses the internal temperature distribution of the building envelope (such as exterior walls and roof), where the thermal response time constant can exceed ten hours. A Long Short-Term Memory (LSTM) network is used for data-driven modeling. The LSTM network structure is designed as follows: Input layer: contains outdoor temperature sequence, solar radiation sequence and water supply temperature sequence for the past 24 hours, with a time step of 1 hour.

[0063] Hidden layer: Contains two LSTM units, each with 64 neurons, used to extract temporal features.

[0064] Output layer: Outputs the equivalent temperature inside the outer envelope structure at the current moment. This equivalent temperature serves as the third feature vector. It is used to correct the deviations in the boundary conditions of the first and second layer models.

[0065] Cross-scale feature fusion: In order to combine the feature vectors of the above three scales For effective fusion, this embodiment introduces a cross-scale attention mechanism. Specifically, the three feature vectors are first mapped to the same dimension (e.g., 128-dimensional) through a linear transformation. The dot product of the query vector and the key vector is calculated to obtain the attention score. The score is then normalized using the Softmax function to obtain the attention weights, resulting in the final fused feature vectors. Through this mechanism, the model can adaptively adjust to current conditions (e.g., if solar radiation changes drastically). weight This will correspondingly increase the dependence on features at different scales. Ultimately, the features will be fused. By performing a residual concatenation with the thermal state vector from the previous time step, the time-lattice thermal state vector at the current time step is generated. This vector comprehensively depicts the complete thermal state of the building at the current moment.

[0066] Reinforcement Learning Network Construction: This invention employs the SAC (SoftActor-Critic) algorithm as the core training framework. In each decision cycle (e.g., every 15 minutes), the agent receives the current state. The state space It consists of two parts: one is the time-lattice hot state vector generated in step S200. Secondly, time information (such as the current time and day of the week). The agent's action space. Defined as a three-dimensional continuous control variable: water supply temperature setpoint adjustment variable Circulating pump frequency adjustment amount Hz, and start / stop time offset minute.

[0067] Both the Actor and Critic networks employ fully connected feedforward neural networks. Specifically, the Actor network parameters are as follows: input layer dimension (state space dimension), first hidden layer with 256 neurons, second hidden layer with 128 neurons, both using the ReLU activation function; output layer with 6 neurons, outputting the mean values ​​of the three action variables. and logarithmic standard deviation The Critic network has similar parameters, but its output layer consists of a single neuron, outputting a state-action value. .

[0068] Diverse candidate action stream generation To explore the optimal control strategy and avoid getting trapped in local optima, the following mechanism is used to generate candidate action flows: Baseline action generation: This involves generating the current state. Input the Actor network to obtain the baseline action. .Will The next state predicted after interacting with the environment Input the Actor network again, and get This process continues to generate a baseline action flow for future control time domains (e.g., 4 hours, or 16 time steps).

[0069] Diversity Screening: To reduce redundant computation, a Deterministic Point Process (DPP) is used to screen the 20 action flows generated in the previous step. The DPP sampling algorithm selects a subset that maintains both diversity and high quality. Finally, a maximum of 5 dissimilar candidate action flows (cosine similarity < 0.7) are selected for the next evaluation step.

[0070] Prediction, evaluation and optimal decision-making stages Differentiable prediction model For each selected candidate action flow A differentiable thermal state transition model is used to predict future states. This model is not a simple black box model, but a Physical Information Neural Network (PINN) structure.

[0071] Given the current thermal state and candidate actions The model is based on the following relationship

[0072] Wherein, the transfer function It is a neural network, but its loss function incorporates residual terms from the first and second layers of the physical model. This subjectes the prediction process to both physical laws and data-driven constraints, ensuring the reasonableness of the predictions.

[0073] Subsequently, by reading the function Output predicted indoor temperature:

[0074] The readout function physically corresponds directly to extracting the air temperature state variable from the first-layer RC model and fine-tuning it in conjunction with the radiative heat transfer of the second layer.

[0075] Multi-scenario assessment Considering the uncertainty of weather forecasts, for each action flow Three scenarios are generated: Baseline scenario (P50): Using the definitive forecast value provided by the meteorological API.

[0076] Optimistic scenario (P10): Outdoor temperature forecast +2°C, solar radiation forecast +20%.

[0077] Pessimistic scenario (P90): Outdoor temperature forecast -2°C, solar radiation forecast -20%.

[0078] Comprehensive scoring and decision The comprehensive scoring function takes into account thermal comfort, energy consumption, and control stability. The definition is as follows:

[0079] in, The average geodesic distance of the trajectory on the geometric manifold of thermal comfort information (the closer to the comfort zone, the smaller the value). To predict cumulative energy consumption, This is a penalty term used to control the rate of change of the action. The weighting coefficient is set to [value] in this embodiment. , , Select Highest candidate action flow As the optimal target action flow.

[0080] Online learning and adaptive update mechanism Another key feature of this invention is its ability to continuously learn and evolve during system operation, executing the optimal action flow. The first action Then, the system collects actual indoor temperature changes. and actual energy consumption .

[0081] Model update: Calculate prediction error The error is used to update the parameters of the three-level model (RC model parameters, state-space model parameters, and LSTM network weights) using online gradient descent.

[0082] Experience replay: The actual execution trajectory is stored in the experience replay pool. The reward function is designed in the same way as the scoring function mentioned above, but with a greater emphasis on immediate feedback.

[0083] Policy Update: Every N steps (N=100), a batch of samples (BatchSize=256) is randomly sampled from the experience replay pool, and the parameters of the Actor network and Critic network are updated using the SAC algorithm. The training objective is to maximize the expected cumulative reward.

[0084] Through the coordinated online updates of the above three components, the system can adapt to the time-varying characteristics of building thermal performance and changes in user usage patterns.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0086] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0087] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0088] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0090] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An energy-saving optimization method for thermal inertial sensing heating systems based on deep reinforcement learning, characterized in that, Includes the following steps: An Internet of Things (IoT) sensor network is deployed inside the target building to collect multi-source data and perform data preprocessing on the multi-source data. By combining the preprocessed multi-source data, the building thermal response process is decomposed into three time scale levels, and cross-scale feature fusion is performed on the three time scale levels to generate a time lattice thermal state vector. A deep reinforcement learning network is constructed, and the time lattice hot state vector is fused into the state space of the deep reinforcement learning network. Multiple candidate future action flows are generated using the deep reinforcement learning network. Future thermal response prediction and candidate action flow evaluation are performed based on time lattice thermal state vector and multiple candidate future action flows. The candidate action flow with the highest comprehensive score is selected as the optimal target action flow.

2. The energy-saving optimization method for a thermal inertial sensing heating system based on deep reinforcement learning according to claim 1, characterized in that, An Internet of Things (IoT) sensor network is deployed inside the target building to collect multi-source data. This multi-source data is then preprocessed, specifically including: An Internet of Things (IoT) sensor network is deployed inside the target building to collect indoor environmental parameters, heating system operating parameters, outdoor environmental parameters, building static parameters, and user-authorized personnel activity parameters. An anomaly detection algorithm based on local outlier factors is used to identify and mark data points that exceed a preset physical range or have abnormal rates of change in the multi-source data information. For missing data, linear interpolation and weighted average based on historical data of the same period are used to fill in the missing data. A low-pass filter is used to filter out high-frequency noise in the temperature signal and retain the low-frequency dominant component of the thermal response process. Data sources with different sampling frequencies are aligned to a unified second-level time granularity using timestamps to obtain preprocessed multi-source data information.

3. The energy-saving optimization method for a thermal inertial sensing heating system based on deep reinforcement learning according to claim 1, characterized in that, The building thermal response process is decomposed into three time scale levels based on the preprocessed multi-source data, specifically: The building thermal response process is decomposed into three time scale levels: the first level, the second level, and the third level. For the first level, the equivalent heat capacity-thermal resistance lumped parameter model is used to establish a first-order differential equation with the indoor air temperature state variable to extract the thermal response characteristics of the first level. For the second level, a state-space model is adopted, with the surface temperature of the inner wall and the temperature of the ground structure layer as state variables, to establish a second-order state equation and extract the thermal response characteristics of the second level. For the second level, a long short-term memory network is used. The outdoor temperature sequence, solar radiation sequence and water supply temperature sequence of the past preset hours are input to predict the internal temperature distribution of the outer envelope and extract the third-level thermal response features.

4. The energy-saving optimization method for a thermal inertial sensing heating system based on deep reinforcement learning according to claim 3, characterized in that, Cross-scale feature fusion is performed across three time scale levels to generate a time-lattice hot state vector, specifically: The first layer of thermal response features, the second layer of thermal response features, and the third layer of thermal response features are encoded respectively to obtain the first feature vector, the second feature vector, and the third feature vector respectively. The cross-scale attention weights are calculated based on the first feature vector, the second feature vector, and the third feature vector, and the fused hot state vector is calculated based on the cross-scale attention weights, and the time lattice hot state vector is generated.

5. The energy-saving optimization method for a thermal inertial sensing heating system based on deep reinforcement learning according to claim 1, characterized in that, A deep reinforcement learning network is constructed, and the temporal lattice hot state vector is fused into the state space of the deep reinforcement learning network. Multiple candidate future action flows are then generated using the deep reinforcement learning network, specifically: A deep reinforcement learning network is constructed, and the time lattice thermal state vector is fused into the state space of the deep reinforcement learning network. The water supply temperature adjustment, the circulation pump frequency adjustment, and the start-stop time offset are set as three continuous control variables to construct the action space. The data in the state space is input into the Actor network, which outputs the probability distribution of actions. The data in the state space is input into the Critic network, which outputs the state-action value. The training algorithm uses SAC and introduces an entropy regularization term to encourage exploration. A baseline action flow is generated based on the current state using an Actor network. Then, exploration noise of different magnitudes is added to the baseline action flow to generate multiple candidate future action flows. A matrix-based point process is used to perform diversity screening on the generated multiple candidate action flows to ensure that the similarity between any two action flows does not exceed a preset threshold.

6. The energy-saving optimization method for a thermal inertial sensing heating system based on deep reinforcement learning according to claim 1, characterized in that, Future thermal response prediction and candidate action flow evaluation are performed based on the time lattice thermal state vector and multiple candidate future action flows. The candidate action flow with the highest comprehensive score is selected as the optimal target action flow. Specifically: Construct a differentiable prediction model, input the current time lattice thermal state vector and candidate future action flow, and use the differentiable prediction model to predict the indoor temperature at each future time step; Meanwhile, for each candidate future action flow, a baseline scenario, an optimistic scenario, and a pessimistic scenario are generated. In each scenario, the indoor text and image sequence and cumulative energy consumption within a preset time are output to construct a geometric manifold of thermal comfort information. The indoor temperature and temperature change rate at each moment are mapped to a point on the manifold. The point representation of the temperature sequence predicted by each candidate motion flow on the thermal comfort information geometric manifold is combined to obtain the corresponding trajectory point sequence on the manifold. The average geometric distance from the trajectory to the comfort zone is calculated based on the trajectory point sequence. A comprehensive score is calculated for each candidate motion flow based on the average geometric distance, and the candidate motion flow with the highest comprehensive score is selected as the optimal target motion flow.

7. The energy-saving optimization method for a thermal inertial sensing heating system based on deep reinforcement learning according to claim 1, characterized in that, Also includes: The first action in the obtained optimal target action flow is immediately sent to the actuator of the central heating system. After each action is executed, the actual indoor temperature change and actual energy consumption feedback data are collected. The actual indoor temperature change is compared with the predicted temperature change, the prediction error is calculated, and the parameters of each level of the model are updated by online gradient descent using the prediction error. The actual execution trajectory is stored as an experience sample in the experience replay pool. Every preset step, a batch of samples is randomly sampled from the experience replay pool to update the parameters of the Actor network and the Critic network. The training objective is to maximize the expected cumulative reward.