Thermodynamic system smart energy management method and platform based on reinforcement learning

Abstract

The invention provides a thermodynamic system smart energy management method and platform based on reinforcement learning. The method comprises the steps that structural state vectors reflecting the thermodynamic coupling relation among a solar heat collector, an air source heat pump, a gas water heater and a water tank are constructed; generating a risk-driven hot water demand prediction sequence in combination with the historical water consumption behavior of the user and the heat supply capability of the current system; based on the prediction sequence and the structure state vector, scheduling actions including energy path selection and water tank target heating temperature are generated through a reinforcement learning strategy model, and energy consumption, temperature difference requirements, switching cost and path impedance are comprehensively considered in the scheduling process; and the scheduling action is converted into equipment control instruction execution, and the state is updated to form a closed loop. According to the invention, the prospective and adaptive scheduling of multi-energy cooperation is realized, and the economy, comfort and reliability of the system are improved.

Description

Technical Field

[0001] This invention belongs to the field of data management, and in particular relates to a smart energy management method and platform for thermal systems based on reinforcement learning. Background Technology

[0002] With the diversification of household energy consumption structures and the increasing demands for comfort, real-time performance, and energy efficiency, traditional single-energy hot water systems are struggling to meet the complex and fluctuating water needs of modern households. Currently, common solar water heaters, air-source heat pump water heaters, and gas water heaters each have their advantages, but each has significant shortcomings when used individually. Solar energy is significantly affected by fluctuations in sunlight, providing almost no effective heating during cloudy, rainy, and nighttime conditions, often relying on electric auxiliary heating, leading to soaring energy consumption. Air-source heat pumps experience severe efficiency degradation in low-temperature environments, making it difficult to guarantee stable output; coupled with slow equipment response, they may not be able to compensate in time when users suddenly use water. Gas water heaters, while providing a fast response, rely on fossil fuels, making their operating costs highly susceptible to price fluctuations, and their long-term use results in high carbon emissions. Although systems combining solar energy, heat pumps, and gas exist, they generally rely on simple start-stop rules or fixed priority logic, failing to make forward-looking judgments and scheduling based on user behavior, the system's current heat storage level, future renewable energy supply capacity, and the coupling characteristics between devices. These systems are mostly driven by single-point temperature signals, lacking structured thermal information, making it impossible to accurately understand the overall heating capacity of the system; unable to respond in advance based on predicted behavioral demands; and unable to comprehensively optimize the real-time constraints and dynamic switching costs of multi-energy coordination. Given the suddenness of water usage, the uncertainty of energy supply, and the time-of-use fluctuations in electricity prices, existing methods have significant shortcomings in areas such as insufficient perception of system structure, lack of user demand prediction, rigid energy path selection, delayed preheating decisions, and lack of modeling for scheduling costs among multiple energy sources. Summary of the Invention

[0003] The purpose of this invention is to design a smart energy management method and platform for thermal systems based on reinforcement learning, which can make globally optimal scheduling decisions under the constraints of equipment response characteristics and user load trends.

[0004] To achieve the above objectives, a first aspect of the present invention provides a method for intelligent energy management of thermal systems based on reinforcement learning, the method comprising: A structural state model of the thermal system is constructed. Based on the physical connection relationship between the solar collector, air source heat pump, gas water heater and water tank, and combined with the real-time temperature, flow rate and operating frequency data collected from each equipment node, a structural state vector representing the overall thermodynamic coupling state of the system is generated. Based on the structural state vector and users' historical water usage data, the probability of hot water demand in future periods is calculated, and the probability of hot water demand is adjusted for risk enhancement according to the current heating capacity of the system to generate a risk-driven hot water demand prediction sequence. Based on the structural state vector and the risk-driven hot water demand prediction sequence, a scheduling action is generated through a reinforcement learning strategy model. The scheduling action includes energy path selection and target heating temperature of the water tank. The reinforcement learning strategy model comprehensively considers energy consumption, temperature difference demand, energy switching cost and path impedance when generating the scheduling action. The scheduling actions are converted into control commands for the corresponding energy devices, and the device start-up, shutdown, parameter setting, and path switching operations are executed. The structure state vector is updated after the scheduling cycle ends, forming a closed-loop control.

[0005] Furthermore, the step of constructing the structural state model of the thermodynamic system includes: Model each device node in the thermal system and its connection relationships as a directed graph; Configure status parameters for each device node, including temperature, flow rate, and operating frequency; The thermal coupling strength between the solar collector and the water tank is calculated as the coupling edge parameter.

[0006] Furthermore, the step of generating a risk-driven hot water demand forecast sequence includes: The original water usage probability distribution was obtained based on historical water flow data. The original water usage probability distribution is dynamically adjusted based on the current system's heating capacity to reflect the risk of hot water shortage.

[0007] Furthermore, the system's heat supply capacity is determined by a linear combination of the available heat in the water tank, the sustainable output capacity of the heat pump, and the actual heat transfer capacity of the solar energy.

[0008] Furthermore, prior to the step of generating scheduling actions through a reinforcement learning policy model, the method further includes: Calculate the overall risk score for future periods; When the comprehensive risk score exceeds a preset threshold, a scheduling response is triggered.

[0009] Furthermore, the energy path selection includes a solar energy path, a heat pump path, or a gas path, with each path corresponding to a unique combination of equipment control logic and valve switching.

[0010] Furthermore, the step of converting the scheduling action into control commands for the corresponding energy equipment includes: When the energy path is selected as an air source heat pump path, the operating frequency of the air source heat pump compressor is set; When the energy path is selected as a gas path, the current of the gas proportional valve is set; When the energy path is selected as the solar energy path, the start and stop of the solar circulation pump are controlled.

[0011] Furthermore, the compressor operating frequency of the air source heat pump is dynamically adjusted according to the difference between the current temperature of the water tank and the target heating temperature of the water tank, and the temperature rise rate of the previous scheduling cycle is introduced as a damping suppression term to limit the frequency fluctuation amplitude.

[0012] Furthermore, the step of generating a risk-driven hot water demand forecast sequence includes: A time smoothing regularization term is introduced to suppress drastic changes in prediction results between adjacent time periods, ensuring the continuity of the scheduling strategy.

[0013] A second aspect of the present invention provides a smart energy management platform for thermal systems based on reinforcement learning, the platform comprising: The structural state modeling module is used to generate a structural state vector that characterizes the overall thermo-coupling state of the system based on the physical connection relationship between the solar collector, air source heat pump, gas water heater and water tank, combined with the real-time temperature, flow rate and operating frequency data collected from each equipment node. The demand forecasting module is used to calculate the probability of hot water demand in future periods based on the structure state vector and users' historical water use behavior data, and to perform risk enhancement adjustment on the probability of hot water demand according to the current heating capacity of the system, so as to generate a risk-driven hot water demand forecasting sequence. The scheduling strategy generation module is used to generate scheduling actions based on the structure state vector and the risk-driven hot water demand prediction sequence through a reinforcement learning strategy model. The scheduling actions include energy path selection and target heating temperature of the water tank. The reinforcement learning strategy model comprehensively considers energy consumption, temperature difference demand, energy switching cost and path impedance when generating scheduling actions. The control execution module is used to convert the scheduling actions into control commands for the corresponding energy devices, execute device start-up and shutdown, parameter setting and path switching operations, and update the structure state vector after the scheduling cycle ends to form closed-loop control.

[0014] The beneficial technical effects of the present invention are at least as follows: To address the aforementioned issues, this invention provides a smart energy management method and platform for thermal systems based on reinforcement learning. By structurally modeling the thermal connections between solar collectors, heat pumps, water tanks, and gas-fired devices, a system state vector accurately reflects the coupling characteristics of multiple energy sources. This allows the control system to move beyond relying on single-point temperature start-stop logic and acquire system-wide energy perception capabilities. Secondly, this invention proposes a risk enhancement prediction method based on the difference between future water demand probability and the current system's heating capacity. This method deeply integrates user behavior characteristics with equipment status, assessing potential hot water shortage risks before the system reaches a critical state, thus enabling proactive rather than passive preheating strategy triggering. Furthermore, this invention constructs a multi-objective scheduling strategy model that comprehensively considers heating energy consumption, temperature difference demand, energy switching costs, and path impedance. This allows the system to dynamically, continuously, and interpretably select paths between different energy sources, ensuring optimal energy consumption performance while meeting comfort requirements. Finally, this invention establishes an execution model that directly maps scheduling actions to equipment-level control signals and introduces an equipment response speed suppression mechanism to ensure the stability of energy equipment output and hardware security in actual control, forming a complete closed loop from state perception and risk reasoning to strategy generation and action execution. Through multi-dimensional innovations in structural state modeling, behavior prediction enhancement, multi-energy path planning, and execution stability, this invention achieves forward-looking, collaborative, and efficient control of multi-energy hot water systems. Attached Figure Description

[0015] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0016] Figure 1 This is a flowchart of the intelligent energy management method for thermal systems based on reinforcement learning, as described in this invention.

[0017] Figure 2 This is a framework diagram of the intelligent energy management platform for thermal systems based on reinforcement learning, as described in this invention. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0019] In one or more embodiments, such as Figure 1As shown, a smart energy management method for thermal systems based on reinforcement learning is disclosed, the method comprising the following: S1: Construct a structural state model of the thermal system. Based on the physical connection relationship between the solar collector, air source heat pump, gas water heater and water tank, and combined with the real-time temperature, flow rate and operating frequency data collected from each equipment node, generate a structural state vector that characterizes the overall thermal coupling state of the system. Specifically, this step aims to construct a structural state representation of the hot water system within the current scheduling cycle, serving as the core input for subsequent user hot water demand assessment and energy scheduling strategies. The specific objective is to uniformly construct a structured mathematical representation of the operating status of each energy module, such as solar collectors, air source heat pumps, gas water heaters, and water tanks, according to their actual physical connections. This representation should reflect not only the current operating status (temperature, flow rate, operating frequency, etc.) but also the strength of energy coupling between different devices. The inputs for this step mainly include: a system structure diagram (static) and real-time data collected from each node (dynamic). The system structure diagram is preset by engineers before platform deployment based on the equipment piping connections, including nodes such as collectors, water tank layers, heat pumps, and gas water heaters, as well as the direction of thermal flow between them. The real-time data includes: water tank temperature measurement points (upper, middle, and lower...). , , (), obtained through an immersion platinum resistance sensor; heat pump exhaust temperature The data is collected by the thermocouple at the outlet of the heat pump refrigerant circuit; the compressor operating frequency. The solar collector outlet temperature is read by the frequency converter driver. The water temperature is collected by the NTC thermistor at the return water end; the outlet water temperature of the gas water heater. The temperature is obtained from the temperature probe; the water inlet flow rate of the water tank is... With water flow rate The data is collected by an ultrasonic flow meter on the pipeline. All signals are connected to the controller via an RS485 or 4~20mA analog interface, and the controller performs periodic data readings every 5 minutes.

[0020] First, based on the static structure diagram, a directed graph is constructed for each node and its connections. This graph structure remains fixed after initialization. Then, a thermodynamic state representation is established for each node. For example, the operating state of an air source heat pump during the scheduling cycle... The following vectors can be used to represent the vector: ; in, This indicates the current temperature at the heat pump exhaust end; This indicates the temperature of the middle layer of the water tank, representing the current state of the heat pump's heat exchange target side; This refers to the current operating frequency of the compressor. The compressor's maximum rated frequency is used as a normalization reference. The first component represents the percentage of the temperature difference between the heat pump outlet and the heat exchange target, reflecting the heat exchange driving force; the second component represents the current load intensity.

[0021] To quantify the thermal coupling strength between the solar energy system and the water tank, the following coupling degree expression is introduced: ; in, This indicates the outlet temperature of the solar collector. This is the temperature of the lower layer of the water tank, reflecting the current state of the coldest hot water layer in the system; This indicates the current water flow velocity in the inlet pipe; A reference flow rate value set for experience, used for normalization; It is a small positive number, used to prevent the denominator from being zero; It is an exponential function. This expression outputs a value within a range of... The value within represents the effective heating capacity of the solar energy for the water tank under the current temperature difference and flow rate conditions.

[0022] Finally, the system controller combines the state vectors of all nodes with the coupling strength of the main coupling edges to form a complete structural state vector, denoted as . And cache the scheduling input for the next step.

[0023] S2: Based on the structure state vector and the user's historical water use behavior data, calculate the probability of hot water demand in the future period, and adjust the probability of hot water demand according to the current heating capacity of the system to generate a risk-driven hot water demand prediction sequence. Specifically, this step, based on the hot water system structural state vector constructed in the previous step and combined with users' historical water usage behavior sequences, infers whether there is a potential risk of hot water shortage under the current structural thermal state. Based on this, a weighted probability distribution sequence of hot water demand for the next 24 hours is constructed. Compared to traditional time series forecasting or simple behavioral statistics, this step introduces a "behavioral demand risk adjustment mechanism," integrating user behavior prediction with the current system structural capabilities to construct a risk-driven hot water demand forecasting method that simultaneously possesses behavioral predictability and equipment capability responsiveness.

[0024] The input for this step is the output structure state vector from the previous step. This state vector contains the thermal state representation of the system's main equipment nodes (such as...). ) and the expression of the coupling capability of key energy coupling edges (such as These values ​​are directly measured by sensors and calculated through a set of structured mappings, remaining constant within the current period. In addition, this step incorporates historical behavioral data sequences, specifically the hot water output of users at various times of day over the past 30 days, expressed as... The data comes from the electronic flow meter on the main water supply pipe, is sampled every minute, and is stored as a two-dimensional array by the controller in the form of a time index.

[0025] This step first constructs a raw water usage probability distribution at the purely behavioral level based on historical water output. The calculation method is as follows: for the past 30 days, at time... Traffic value Non-zero frequency statistics are performed, and the probability distribution is obtained by normalizing by day. This process is completed through time alignment, without relying on model inference, ensuring clear logic and controllable results. Then, based on the device state variables in the structural state vector, the future... Heat supply capacity at any time This value is obtained by combining the current available heat in the water tank, the sustainable output capacity of the heat pump, and the actual heat transfer capacity of the solar energy. The available heat in the water tank is determined based on... The heat percentage is calculated by proportionally converting the current temperature difference range between the two values. The sustainable output capacity of the heat pump is determined by... The first component (the proportion of exhaust temperature difference) is normalized and estimated. The actual heat transfer capacity of solar energy is estimated through the coupling degree. A normalized mapping is performed. The three are formed using a linear combination method. This combination method maintains a fixed coefficient during operation, ensuring that the capability value can be determined by... It is derived directly.

[0026] Based on this, a "risk-adjustable hot water demand probability enhancement model" is proposed. Its core idea is: although current user behavior may indicate a probability of water usage, if the system's heating capacity is sufficient, there is no need for hasty scheduling; if the system's heating capacity cannot guarantee water usage, a heating warning should be issued in advance. To this end, the following formula is proposed: ; in, The probability of increased hot water demand at the current forecast time. Based solely on historical usage probability of user behavior, The historical average heat usage at that moment (obtained by converting flow rate into heat). Heat can be supplied to the current structural state. It is an adjustable modulation factor (reflecting the system's sensitivity to insufficiency). To prevent extremely small positive numbers with a denominator of zero, This is the risk enhancement index (adjusting the system's response speed to changes in risk). When the system's heating capacity is lower than the average demand, the left-hand side of this function is closer to 1, increasing the overall probability; conversely, it decreases, reflecting the suppressive effect of equipment availability on risk.

[0027] More importantly, to enhance the temporal consistency of the prediction curve and avoid frequent warm-up caused by short-term fluctuations, we introduce a time smoothing regularization term. As a priori term for the modulation structure state input in the loss function, its form is as follows: ; This regularization term penalizes the difference between the prediction result at a single moment and the average prediction at adjacent moments, suppressing local jitter, thereby ensuring that the scheduling system can maintain the continuity of response and the stability of equipment start-up and shutdown when making energy decisions based on the prediction results. The constant for adjusting the intensity is set by the platform configuration, and is generally set between 0.1 and 0.3.

[0028] For example, when Historical behavior probability Average demand MJ, currently available under device status MJ, set , ,but ; Finally If the difference from adjacent time points is significant, then it will be... Adjustments and compression are performed to avoid "local outliers" interfering with the overall strategy. The output is the enhanced demand forecast sequence. and the reference series of heat demand Both have the ability to adjust their structural state.

[0029] S3: Based on the structural state vector and the risk-driven hot water demand prediction sequence, a scheduling action is generated through a reinforcement learning strategy model. The scheduling action includes energy path selection and target heating temperature of the water tank. The reinforcement learning strategy model comprehensively considers energy consumption, temperature difference demand, energy switching cost and path impedance when generating the scheduling action. Specifically, this step aims to determine the structure state vector. and future hot water usage demand sequence This paper proposes a multi-energy path intelligent scheduling strategy that simultaneously considers hot water supply capacity, energy cost, user comfort, and energy switching stability. Unlike traditional hot water systems that rely on temperature thresholds or simple rules for start-stop control, this step proposes a strategy structure that integrates equipment current heat output capacity assessment, behavior-driven incentive integral mechanism, and energy switching inhibition term. The scheduling strategy is generated using reinforcement learning, fully reflecting the overall goal of this scheme: "system structure perception—behavioral demand integration—economic and comfort balance—energy coordinated control." The input variables include two key parts. The first part is the prediction sequence generated in the previous step. ,in This indicates the probability of a user using water in different time periods in the future. This represents the user's hot water load demand for various time periods in the future. These two quantities have been generated by fusing the deviation between historical behavior frequency and current available heating capacity, reflecting the user's current expectations for future hot water and the supply and demand risks. The second part is the previous step. Including the current heat pump heat exchange status Solar coupling degree Temperature of the upper and lower layers of the water tank Heat pump frequency These variables comprehensively reflect whether the current system has the ability to provide instant heating, whether there is heat storage space, and the heating position of each device in the thermal path.

[0030] To translate the above information into a control strategy, we introduce a scheduling function based on joint modeling of supply and demand risk integrals and energy path impedance. First, we construct a behavior-driven preheating excitation integral function to evaluate whether the system needs to actively perform a preheating operation in the current cycle: ; in Indicates scheduling time The overall risk score, This is the length of the rolling forecast window (usually set to 1248, corresponding to a 14-hour forecast interval). It is a time-weighted factor for each future time period (which can be set to a decreasing type). This is a confidence index for hot water behavior, used to strengthen the incentive weight of high-confidence user behavior; This represents the heat supply capacity of the current system structure at future moments, determined by... The equipment status is calculated jointly. It is a very small positive number. The larger this integral value is, the more prominent the future hot water supply and demand contradiction will be, and the more necessary it is to schedule in advance.

[0031] exist Exceeding the set threshold Afterwards, the system enters the scheduling response state. At this point, three output decisions need to be determined: whether to perform preheating, which energy path to use, and what target temperature to heat the water tank. To this end, the following scheduling objective function is constructed to generate the strategy output: ; in: Indicates the energy path (value can be one of "solar", "heat pump" or "gas"); Indicates the target heating temperature; Indicates energy pathway Heat the water tank to The required energy is estimated by combining energy efficiency and the current state of the equipment. The difference between the current water temperature and the target temperature indicates the amount of heating work required. The energy switching cost indicator is the path variable from the previous cycle recorded by the controller at the end of each scheduling cycle. With the current path The comparison yields the following: After executing the device control actions of the previous cycle, the controller will determine the energy path used in this cycle. Persistently save to local cache, read the record in the current period and compare it with... Compare them; if they are different, then... Select 1 if the value is 1, otherwise select 0. This is the path impedance penalty term, a constraint designed for the three-energy synergistic system in this scheme. It reflects whether a certain path has good thermodynamic response capability based on the current state of the system. For example, if... If the impedance is too low, even if the energy consumption of the solar path is low, it will not have a sufficient heat exchange effect, and its path impedance should be increased.

[0032] This function is a structured scheduling policy generator that considers multiple objectives. arrive This is a weighting coefficient that can be dynamically adjusted during operation. The system uses the path and target temperature corresponding to the minimum value of this function as the current scheduling output.

[0033] For example: In a certain period, the system state , , The gas proportional valve is open, but This indicates limited compensation capacity. Upon entering the scheduling activation state, the controller calculates the objective function and finds that although solar energy consumption is low, its impedance value is high due to insufficient coupling. Therefore, the heat pump path is ultimately selected and the target temperature is set. Generate scheduling actions The output of this step is the scheduling action. ,in For energy route selection, To reach the target temperature, the result will be converted into actual control commands and sent to the equipment control terminal in the next step.

[0034] S4: Convert the scheduling action into control commands for the corresponding energy equipment, execute equipment start-up and shutdown, parameter setting and path switching operations, and update the structural state vector after the scheduling cycle ends to form closed-loop control; Specifically, the goal of this step is to process the scheduling and control actions generated in the previous step. The system actually operates on specific energy devices in the hot water system, completing device startup, parameter setting, and path switching. After execution, it collects the latest device operating status and updates the structure state vector. This forms a complete control closed loop. This process not only requires precise scheduling of energy equipment but also addresses issues such as the response speed, operational inertia, and differences in coupling channels among different equipment in a multi-energy parallel system. Therefore, based on the execution of standard control actions, this step designs a response compensation mechanism and an energy path protection mechanism specifically for the characteristics of the three-energy collaborative structure of this scheme. This ensures the accuracy of equipment response and the stability of control behavior, further improving the reliability of the system's scheduling execution under varying water loads.

[0035] The input for this step is the control action output from the scheduling strategy module. ,in This indicates the energy path that should be activated in the current cycle. Indicates the temperature in the middle of the water tank The target temperature to be reached during this cycle. The controller first determines... The value determines the current energy path: if it's solar energy, the solar circulation pump needs to be activated and the inlet electric valve connected to the water tank needs to be opened; if it's a heat pump, the compressor needs to be turned on, and the solar circuit and gas inlet need to be closed; if it's a gas water heater, the main gas solenoid valve needs to be opened, the proportional valve current needs to be controlled, and other paths need to be closed. Energy path switching is achieved through electric valves and relay control between devices. The controller controls the power-on, on / off, and valve switching of each device through relay output modules (such as relay boards or PLC switch modules). All electric valves are connected to the controller's digital output terminal and are uniformly managed by number. For example, DO1 controls the heat pump start / stop, DO2 controls the gas solenoid valve, and DO3 controls the solar pump. Specific switching commands are implemented by writing low or high levels according to the device number.

[0036] Taking the activation of the heat pump path as an example, the controller receives... After receiving the command, the heat pump compressor is first started by outputting a high level through DO1, and then the target frequency is set through the analog output AO1. And shut down the digital output interfaces corresponding to the solar circulation pump and the gas proportional valve. The target frequency is set by the target temperature. With the current temperature of the middle layer of the water tank The difference determines the frequency setting strategy. To achieve a safe and stable heating process, the controller employs a frequency setting strategy with a dynamic damping suppression term, as shown below: ; In this formula, This is the target frequency setting value for the heat pump compressor in this cycle; To control the target temperature during the operation; The current temperature of the middle layer of the water tank is acquired by the middle layer temperature probe (PT100 platinum resistance thermometer) through the analog input interface AI1, and is read periodically by the controller. This is the frequency adjustment factor, set by the platform configuration file, for example, taking... Hz / ℃; The temperature rise rate of the previous cycle is calculated from the historical temperature difference of the middle layer of the water tank, i.e. Used to sense the current heating rate As an inhibitory factor, if set as This is used to prevent excessive frequency fluctuations.

[0037] For example: Current , , , Time step minutes, then the rate of temperature rise minutes, set , Then the controller calculates: ; The controller sends the frequency setpoint to the heat pump inverter driver via an analog voltage signal or PWM pulse signal output from AO1, thereby realizing the adjustment and control of the heating equipment.

[0038] For gas path control, the controller operates in the same manner according to... and The difference between them sets the target current of the proportional valve. The current is output to the proportional valve drive module via AO2, and the current range is typically set to 0.5~1.5A. The solar path uses a switch control mode, and if the coupling degree... Higher than 0.5 and If DO3 outputs a high level, the circulating pump will start.

[0039] During equipment operation, the controller continuously collects data. , , , Temperature sensor values ​​(all PT100 sensors) are collected once per minute via the AI1~AI4 analog input interfaces and cached locally. At the end of the scheduling cycle, the controller summarizes the states of all nodes (including temperature, current, frequency, etc.) into a structure state vector. It is written to the control platform cache and passed to the next round of scheduling control module as input.

[0040] In one or more embodiments, such as Figure 2 As shown, a smart energy management platform for thermal systems based on reinforcement learning is disclosed, the platform comprising: The structural state modeling module is used to generate a structural state vector that characterizes the overall thermo-coupling state of the system based on the physical connection relationship between the solar collector, air source heat pump, gas water heater and water tank, combined with the real-time temperature, flow rate and operating frequency data collected from each equipment node. The demand forecasting module is used to calculate the probability of hot water demand in future periods based on the structure state vector and users' historical water use behavior data, and to perform risk enhancement adjustment on the probability of hot water demand according to the current heating capacity of the system, so as to generate a risk-driven hot water demand forecasting sequence. The scheduling strategy generation module is used to generate scheduling actions based on the structure state vector and the risk-driven hot water demand prediction sequence through a reinforcement learning strategy model. The scheduling actions include energy path selection and target heating temperature of the water tank. The reinforcement learning strategy model comprehensively considers energy consumption, temperature difference demand, energy switching cost and path impedance when generating scheduling actions. The control execution module is used to convert the scheduling actions into control commands for the corresponding energy devices, execute device start-up and shutdown, parameter setting and path switching operations, and update the structure state vector after the scheduling cycle ends to form closed-loop control.

[0041] It is worth noting that the specific workflow of the intelligent energy management platform for thermal systems based on reinforcement learning provided in this embodiment of the invention is the same as that of the intelligent energy management method for thermal systems based on reinforcement learning described in the above embodiments, and will not be repeated here.

[0042] This invention also provides a reinforcement learning-based intelligent energy management device for thermal systems, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps described in the reinforcement learning-based intelligent energy management method embodiments for thermal systems, for example... Figure 1 The steps S1 to S4 described above; or, when the processor executes the computer program, it implements the functions of each module in the above system embodiments.

[0043] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the reinforcement learning-based intelligent energy management device for thermal systems.

[0044] The reinforcement learning-based intelligent energy management device for thermal systems can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the reinforcement learning-based intelligent energy management device for thermal systems may also include input / output devices, network access devices, buses, etc.

[0045] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASACs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the reinforcement learning-based intelligent energy management device for thermal systems, connecting all parts of the device via various interfaces and lines.

[0046] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the reinforcement learning-based intelligent energy management device for thermal systems by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the operation of the air conditioner controller, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0047] The modules integrated into the reinforcement learning-based intelligent energy management device for thermal systems, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0048] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0049] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A smart energy management method for thermal systems based on reinforcement learning, characterized in that, The method includes: A structural state model of the thermal system is constructed. Based on the physical connection relationship between the solar collector, air source heat pump, gas water heater and water tank, and combined with the real-time temperature, flow rate and operating frequency data collected from each equipment node, a structural state vector representing the overall thermodynamic coupling state of the system is generated. Based on the structural state vector and users' historical water usage data, the probability of hot water demand in future periods is calculated, and the probability of hot water demand is adjusted for risk enhancement according to the current heating capacity of the system to generate a risk-driven hot water demand prediction sequence. Based on the structural state vector and the risk-driven hot water demand prediction sequence, a scheduling action is generated through a reinforcement learning strategy model. The scheduling action includes energy path selection and target heating temperature of the water tank. The reinforcement learning strategy model comprehensively considers energy consumption, temperature difference demand, energy switching cost and path impedance when generating the scheduling action. The scheduling actions are converted into control commands for the corresponding energy devices, and the device start-up, shutdown, parameter setting, and path switching operations are executed. The structure state vector is updated after the scheduling cycle ends, forming a closed-loop control.

2. The intelligent energy management method for thermal systems based on reinforcement learning according to claim 1, characterized in that, The steps for constructing the structural state model of the thermodynamic system include: Model each device node in the thermal system and its connection relationships as a directed graph; Configure status parameters for each device node, including temperature, flow rate, and operating frequency; The thermal coupling strength between the solar collector and the water tank is calculated as the coupling edge parameter.

3. The intelligent energy management method for thermal systems based on reinforcement learning according to claim 1, characterized in that, The steps for generating a risk-driven hot water demand forecast sequence include: The original water usage probability distribution was obtained based on historical water flow data. The original water usage probability distribution is dynamically adjusted based on the current system's heating capacity to reflect the risk of hot water shortage.

4. The intelligent energy management method for thermal systems based on reinforcement learning according to claim 1, characterized in that, The system's heat supply capacity is determined by a linear combination of the available heat in the water tank, the sustainable output capacity of the heat pump, and the actual heat transfer capacity of the solar energy.

5. The intelligent energy management method for thermal systems based on reinforcement learning according to claim 1, characterized in that, Prior to the step of generating scheduling actions through a reinforcement learning policy model, the method further includes: Calculate the overall risk score for future periods; When the comprehensive risk score exceeds a preset threshold, a scheduling response is triggered.

6. The intelligent energy management method for thermal systems based on reinforcement learning according to claim 1, characterized in that, The energy path selection includes a solar energy path, a heat pump path, or a gas path, with each path corresponding to a unique combination of equipment control logic and valve switching.

7. The intelligent energy management method for thermal systems based on reinforcement learning according to claim 1, characterized in that, The step of converting the scheduling action into control commands for the corresponding energy equipment includes: When the energy path is selected as an air source heat pump path, the compressor operating frequency of the air source heat pump is set; When the energy path is selected as a gas path, the current of the gas proportional valve is set; When the energy path is selected as the solar energy path, the start and stop of the solar circulation pump are controlled.

8. The intelligent energy management method for thermal systems based on reinforcement learning according to claim 7, characterized in that, The compressor operating frequency of the air source heat pump is dynamically adjusted according to the difference between the current temperature of the water tank and the target heating temperature of the water tank, and the temperature rise rate of the previous scheduling cycle is introduced as a damping suppression term to limit the frequency fluctuation amplitude.

9. The intelligent energy management method for thermal systems based on reinforcement learning according to claim 1, characterized in that, The steps for generating a risk-driven hot water demand forecast sequence include: A time smoothing regularization term is introduced to suppress drastic changes in prediction results between adjacent time periods, ensuring the continuity of the scheduling strategy.

10. A smart energy management platform for thermal systems based on reinforcement learning, characterized in that: The platform includes: The structural state modeling module is used to generate a structural state vector that characterizes the overall thermo-coupling state of the system based on the physical connection relationship between the solar collector, air source heat pump, gas water heater and water tank, combined with the real-time temperature, flow rate and operating frequency data collected from each equipment node. The demand forecasting module is used to calculate the probability of hot water demand in future periods based on the structure state vector and users' historical water use behavior data, and to perform risk enhancement adjustment on the probability of hot water demand according to the current heating capacity of the system, so as to generate a risk-driven hot water demand forecasting sequence. The scheduling strategy generation module is used to generate scheduling actions based on the structure state vector and the risk-driven hot water demand prediction sequence through a reinforcement learning strategy model. The scheduling actions include energy path selection and target heating temperature of the water tank. The reinforcement learning strategy model comprehensively considers energy consumption, temperature difference demand, energy switching cost and path impedance when generating scheduling actions. The control execution module is used to convert the scheduling actions into control commands for the corresponding energy devices, execute device start-up and shutdown, parameter setting and path switching operations, and update the structure state vector after the scheduling cycle ends to form closed-loop control.

Images