Adaptive energy saving optimization method based on reinforcement learning

Abstract

This invention discloses an adaptive energy-saving optimization method based on reinforcement learning, belonging to the field of energy management technology. First, a physical information digital twin model is constructed to generate a high-fidelity offline experience dataset, avoiding the risks of online trial and error. Then, a conservative offline reinforcement learning algorithm with start-stop penalties is used to pre-train the basic policy network. During the deployment phase, a rolling optimization architecture integrating a time-series fusion transformer with embedded dynamic delay masks and model predictive control is introduced to compensate for control lag caused by building thermal inertia. Simultaneously, adaptive techniques during online testing are used to dynamically track performance drift caused by equipment aging and correct the model prediction baseline. Finally, interpretable decision logs are generated based on integral gradient attribution and a large language model, and the strategy is fine-tuned based on feedback from operations and maintenance personnel. This invention achieves a safe, adaptive, and interpretable energy-saving optimization process, applicable to intelligent control of HVAC systems in data centers and large public buildings.

Description

Technical Field

[0001] This invention relates to the field of energy management technology, and in particular to an adaptive energy-saving optimization method based on reinforcement learning. Background Technology

[0002] With the continued growth of global energy demand and increasingly stringent carbon emission constraints, building energy consumption (including data centers, large public buildings, and communication base stations) has become a significant segment of energy consumption. Heating, ventilation, and air conditioning (HVAC) systems account for nearly half of total building energy consumption, making them crucial. However, building thermal environment systems possess complex characteristics such as high inertia, strong nonlinearity, multivariate coupling, and significant uncertainties. Traditional rule-based control strategies and local optimization methods struggle to achieve a globally optimal balance between energy consumption and comfort under dynamic operating conditions.

[0003] Current mainstream energy-saving optimization technologies include physical mechanism-based modeling optimization methods (such as model predictive control), data-driven predictive optimization methods (such as deep learning), and reinforcement learning-based adaptive control methods. However, the accuracy of physical models depends on accurate identification of thermal parameters and is difficult to adapt to performance drift caused by equipment aging; purely data-driven methods lack physical constraints and are prone to outputting control commands that violate energy conservation under extreme conditions, posing safety hazards; existing reinforcement learning methods rely on online trial and error to explore optimal strategies, but online exploration in HVAC systems can easily disrupt the thermal comfort environment, cause equipment overload, or even system crashes, making them difficult to deploy in practical engineering. Furthermore, deep reinforcement learning generally suffers from the "black box" characteristic of decision-making logic, severely weakening the trust of maintenance personnel in intelligent systems, leading to a large number of intelligent devices being switched back to manual mode in actual operation, making it difficult to achieve energy-saving effects. These shortcomings systematically restrict the effective application of energy-saving optimization methods in engineering. Summary of the Invention

[0004] The purpose of this invention is to provide an adaptive energy-saving optimization method based on reinforcement learning, which solves the above-mentioned problems existing in the prior art.

[0005] To achieve the above objectives, this invention provides an adaptive energy-saving optimization method based on reinforcement learning, comprising the following steps: S1. Combining the thermal characteristics of the target physical space with real-time sensor data, construct a digital twin model of physical information, and generate a high-fidelity offline experience dataset based on the model. S2. Based on a high-fidelity offline experience dataset, a basic policy network is generated by using a reinforcement learning algorithm and introducing a start-stop penalty term. S3. Based on the physical information digital twin model and basic policy network, a temporal fusion transformer (TFT) with embedded dynamic delay mask is used for long-term state representation, and rolling time-domain optimization is performed based on the model predictive control (MPC) framework to output the fine control command at the current moment. S4. Based on the actual feedback after the execution of fine control instructions, dynamically update the global statistics of the physical information digital twin model, and determine whether performance drift has occurred based on the real-time prediction residuals. S5. Based on fine-grained control instructions and contribution scores, a decision log is generated through a lightweight large language model. Combined with the adjustment instructions from operations and maintenance personnel, offline policy gradient adjustments are performed to complete the iteration of the policy network and generate an updated policy network.

[0006] Therefore, this invention employs a reinforcement learning-based adaptive energy-saving optimization method. By constructing a physical information digital twin model embedded with physical partial differential equation constraints, a high-fidelity offline experience dataset is generated, avoiding the risks of online trial and error. Simultaneously, a conservative offline reinforcement learning framework is adopted, and start-stop penalty terms are introduced, extending equipment lifespan while ensuring energy efficiency. During deployment, a dynamic delay mask matrix based on real-time identification of equivalent lag step size is introduced into the time-series fusion transformer, and rolling optimization is performed using a model predictive control framework, solving the control oscillation problem caused by the large lag characteristics of buildings. Furthermore, adaptive technology during online testing dynamically tracks performance drift caused by equipment aging, correcting the terminal cost weight matrix and maintaining the energy-saving stability of the system throughout its entire lifecycle. Finally, interpretable decision logs are generated based on integral gradient attribution and a large language model, and offline strategy fine-tuning is performed based on feedback from maintenance personnel, reducing the rate of human error in takeover and ensuring the effective implementation of the intelligent control system.

[0007] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0008] Figure 1 This is an overall flowchart of an adaptive energy-saving optimization method based on reinforcement learning according to the present invention. Detailed Implementation

[0009] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0010] Please see Figure 1 An adaptive energy-saving optimization method based on reinforcement learning includes the following steps: S1. Combining the thermal properties of the target physical space with real-time sensor data, construct a digital twin model of physical information, and generate a high-fidelity offline experience dataset based on the model; thermal properties are all the macroscopic physical attributes and their dynamic response laws inherent in the target physical space (such as data center computer room, large building or communication base station) and related to heat generation, transfer, storage and dissipation.

[0011] S11. Obtain the geometric and physical parameters of the target physical space, and construct the initial topology of the equivalent thermal resistance-thermal capacity network based on the lumped parameter method; use the physical partial differential equations of the equivalent thermal resistance-thermal capacity network as soft constraints, embed them into the loss function of the deep neural network, construct a physical information digital twin model, and train the physical information digital twin model through the total loss function; the input of the physical information digital twin model is the outdoor dry-bulb temperature. Solar radiation intensity Internal heat source power (Calculated based on IT equipment power consumption) and historical control values (Such as chilled water valve opening, fan speed), the output is the predicted indoor hot spot temperature. ; The total loss function is a weighted average of the data-driven loss and the physical residual loss, expressed as follows: ; In the formula, The total loss value of the physical information digital twin model; These are the trainable parameters for a deep neural network. This represents the number of samples in the batch data. For the first Model predicts temperature; For the sensor to measure the first The return air temperature; This is the physical constraint weighting coefficient, with a value range of 0.1 to 10; Equivalent heat capacity; Equivalent thermal resistance; For continuous time variables; For the first The corresponding air conditioning system is currently supplying cooling / heating. To predict the partial derivative of temperature with respect to time; For the first The outdoor dry-bulb temperature corresponding to each sample; For the first The internal heat source power corresponding to each sample; The partial derivative of temperature with respect to time is calculated using automatic differentiation techniques. The network is forced to learn mappings that satisfy the law of conservation of energy. S12. Virtual Sensing and Reconstruction of Unmeasurable States: Targeting Core Variables That Are Difficult to Measure Directly—Temperature of the Inner Wall of the Building Envelope Using the encoder-decoder structure of a pre-trained physical information digital twin model, along with the effective radiation temperature, easily measurable variables, including return air temperature, supply air temperature, and chilled water flow rate, are mapped to a high-dimensional latent space. The wall temperature is then reconstructed from the latent space by decoding. To obtain the current state vector ,in, This represents the current indoor hotspot temperature. S13. Generation of High-Fidelity Offline Experience Dataset: Using the parameters of the trained physical information digital twin model as a simulator, this simulator replaces the real physical environment. Within a safe boundary, the control variables are randomly disturbed and historical meteorological data are superimposed to perform Monte Carlo simulation, recording each step. The generated tuples are stored in a high-fidelity offline experience dataset. ,in The control action to be performed at the current moment; This is the state vector for the next time step; The immediate reward obtained at the current moment is the sum of the energy consumption penalty and the temperature over-limit penalty, expressed as: ; In the formula, This refers to the real-time power of the chiller unit. This represents the total power of the wind turbine group. This is the reference power under design conditions; This is the penalty coefficient for exceeding temperature limits; The upper limit of the safe operating temperature of the target physical space.

[0012] S2. Based on a high-fidelity offline experience dataset, a basic policy network is generated by using a reinforcement learning algorithm and introducing a start / stop penalty term.

[0013] S21. Definition of Conservative Action Value Function: Adopting the Conservative Q-Learning (CQL) framework, based on reinforcement learning algorithms, and in addition to minimizing the standard Bellman error, it introduces the maximization of the current policy distribution. The difference between the expected value of the dataset and the expected value of the high-fidelity offline empirical dataset is used as a regularization term to define a conservative Q-learning loss function. The expression is: In the formula, This is a conservative coefficient; To control the action vector; Let the expected value be the value under the current policy distribution. This represents the mathematical expectation of the current control action vector under the distribution of a high-fidelity offline experience dataset. Output to the current action value network; The mathematical expectation under the offline experience tuple distribution; This is the control action vector for the next moment; Value network for target actions; Output of the target action value network; Discount factor; S22. Reward Function Reshaping for Equipment Physical Losses: Based on the immediate reward in S13, an additional start-stop penalty term is added for reinforcement learning training to obtain the reshaped immediate reward. The expression is: ; In the formula, For start / stop indication functions, if the current action causes the compressor state to switch from "off" to "on" (or the frequency to jump from 0 to >0), then Otherwise, it is 0; The penalty coefficient is set based on the equivalent power consumption of the starting current surge provided by the equipment manufacturer. S23. Basic Policy Network Pre-training and Output: Utilizing data from a high-fidelity offline empirical dataset, minimize the algorithm using gradient descent. Update the current action value network and update the base policy network using the deterministic policy gradient until the temporal difference error converges. Then, update the parameters of the converged base policy network. Solidification yields the converged basic policy network. ; The input is a full state space vector, and the output is a normalized continuous action. (Mapped to chilled water set temperature or fan frequency percentage); each time... Step-by-step gradient update, using a soft update method to synchronize the current action value network parameters to the target action value network.

[0014] S3. Based on the physical information digital twin model and basic policy network, a time-series fusion transformer with embedded dynamic delay mask is used to represent the long-term state, and rolling time-domain optimization is performed based on the model predictive control framework to output the fine control command at the current moment.

[0015] S31. Feature extraction of the time-series fusion transformer for dynamic delay masks: A time-series fusion transformer is used to process past... Historical state sequence at time 1 , The length of the historical state sequence. For time The state vector at time is used to introduce a dynamic delay mask matrix in its multi-head attention computation. The attention weight allocation is corrected, expressed as follows: ; In the formula, For query matrix; The key matrix; It is a value matrix; This is the matrix transpose. The normalized attention weight matrix and value matrix Multiply by the product to obtain the final weighted aggregated features; It is a normalized exponential function; This is the scaling factor; The dimension of the key vector; Dynamic Delay Mask Matrix It is An upper triangular matrix, whose elements The definition of is: ; In the formula, This represents the equivalent lag step size of the current operating condition as identified in real time by the digital twin model of physical information; , These represent the row index and column index in the dynamic delay mask matrix, respectively. S32. Rolling Time-Domain Optimization Based on Model Predictive Control Framework: The actions output by the converged basic policy network are used as the initial guesses and terminal cost constraints for model predictive control. The optimization objective function of model predictive control is... Set to predict time domain Within this context, the weighted sum of total energy consumption and thermal inequality is minimized, expressed as: ; In the formula, This is the current control moment; For the intermediate iteration index in the prediction time domain; Let be the control action vector to be solved; Basic policy network for state The given reference action vector; For the first The predicted state of the step; The penalty weight matrix is ​​used to control incremental / action deviation; Set point reference state; This is the penalty weight matrix for state deviation; The terminal cost weight matrix is ​​obtained by solving the Riccati equation after linearization of the physical information digital twin model; To predict the predicted state of the time-domain terminal; The physical information digital twin model in S1 serves as the internal predictive model for model predictive control. It is used to predict future states based on the current state and candidate actions. ; S33. Solving and Outputting the Optimal Control Command Sequence: In each rolling optimization step, the objective function is solved using a sequential quadratic programming algorithm. The optimal control command sequence that satisfies the physical constraints is obtained. , For a moment A multidimensional continuous action vector, taking only the first element of the sequence. As a precise control command for the current moment, it is sent to the fieldbus. This includes the chiller unit outlet water temperature setpoint, cooling tower fan frequency setpoint, and terminal air valve opening, outputting this... Simultaneously with the execution, the execution time state vector at the moment the instruction is issued is recorded for online adaptive updates in S4. S4. Based on the actual feedback after the execution of fine control instructions, dynamically update the global statistics of the physical information digital twin model, and determine whether performance drift has occurred based on the real-time prediction residuals.

[0016] S41. Adaptive Time-to-Time (TTT) Statistics Update: As the equipment operates for a long time, condenser scaling or refrigerant leakage can lead to changes in the physical parameters of the physical information digital twin model (such as...). , This leads to implicit drift (conceptual drift). To avoid interrupting online control and costly retraining, this invention freezes the trainable parameters of the deep neural network in the trained physical information digital twin model in S1. It performs online sliding updates only on the mean and variance of the batch normalization layer within the network; and acquires the new batch of sensor data fed back in real time after executing fine control commands. Calculate the actual mean of the current batch of data. With variance The global statistics, including the mean and variance of the batch normalized layer, are updated using an exponential moving average with a forgetting factor. The expression is as follows: ; ; In the formula, Forgetting factor; This represents the mean of the updated normalized layer. This represents the variance of the updated normalized layer. This is the mean of the normalized layer before the last update; The variance of the normalized layer before the last update; S42. Based on the actual return air temperature in the execution time state vector at the time the instruction was issued. Calculate and predict indoor hotspot temperature With actual return air temperature Real-time prediction residuals between , If residuals are predicted in real time Exceeding the preset residual threshold for 3 consecutive minutes If a significant performance drift is detected in the physical information digital twin model, then the updated model will be used. and Immediately inject the physical information digital twin model to complete the unsupervised tracking of the digital twin's aging of the physical entity; simultaneously, using the updated physical information digital twin model with global statistics, re-derive the current optimal control command sequence. The corresponding predicted state trajectory is used, and the terminal cost weight matrix is ​​recalculated based on the new trajectory. This provides a corrected terminal safety anchor point for model predictive control optimization in the next control cycle.

[0017] S5. Based on fine-grained control instructions and contribution scores, a decision log is generated through a lightweight large language model. Combined with the adjustment instructions from operations and maintenance personnel, offline policy gradient adjustments are performed to complete the safe iteration of the policy network and generate an updated policy network.

[0018] S51. Gradient Feature Attribution Calculation for Action Decisions: Addressing the lack of trust among operations and maintenance personnel in the "black box" decision-making of deep networks, this method is applied to policy networks. While deploying inference, the current state vector is computed in parallel. Based on the improved integral gradient method, the current control action vector is calculated with respect to the current state vector. The contribution score of each dimension of the feature is expressed as: ; In the formula, For the first Contribution score of each dimension feature; The current state vector is the first... Features in one dimension; The first of the baseline state vectors Features in one dimension; Index for integration steps; This represents the number of integration steps. For the current policy network; The baseline state vector; S52. Natural Language Decision Log Generation Based on Large Language Model (LLM): The contribution scores calculated in S51 are sorted in descending order. The top 3 contribution scores are used as key features (e.g., "increased outdoor temperature leads to contribution +0.75"). A structured prompt template is constructed, containing the current time period, key features, interference sources, and... The structured prompt template is input into a lightweight large language model (such as Phi-3 or Qwen2.5-1.5B) mounted on a local edge server. After minor adjustments to the model using a few building terminology instructions, the model translates the structured prompt template into a natural language text decision log: "At the current time [14:30], due to the outdoor temperature rising by 2℃ from the baseline (contribution 0.75) and the server room heat load remaining high, the algorithm predicts that maintaining the original settings will trigger a temperature over-limit. Therefore, the chilled water outlet temperature is adjusted from 7℃ to 6.5℃ 15 minutes in advance to reserve cooling capacity." This decision log is then pushed to the operation and maintenance management terminal. S53. Offline strategy fine-tuning and iteration of human-machine collaboration: Operation and maintenance personnel can use the operation and maintenance management terminal to adjust the decision logs generated by LLM by "approving" or "rejecting and correcting the suggested actions", which are recorded as interactive data with human preference labels. And store it in the human interaction dataset. , For the action labels corrected by operations and maintenance personnel, when the amount of interaction data in the human interaction dataset reaches a preset threshold, this interaction data is used to combine behavior cloning and near-end policy optimization to optimize the basic policy network obtained in S23. As the current strategy network The initial parameters, for Perform offline policy gradient adjustment to generate the updated policy network, expressed as: ; In the formula, To fine-tune the total loss offline; The mathematical expectation on the human-interactive dataset; This is the balance coefficient; The mathematical expectation of the current state vector on a high-fidelity offline empirical dataset; For the previous generation of policy networks; The dominant function; This is the clipping function; This is for pruning hyperparameters.

[0019] Therefore, this invention employs a reinforcement learning-based adaptive energy-saving optimization method. By constructing a physical information digital twin model embedded with physical partial differential equation constraints, a high-fidelity offline experience dataset is generated, avoiding the risks of online trial and error. Simultaneously, a conservative offline reinforcement learning framework is adopted, and start-stop penalty terms are introduced, extending equipment lifespan while ensuring energy efficiency. During deployment, a dynamic delay mask matrix based on real-time identification of equivalent lag step size is introduced into the time-series fusion transformer, and rolling optimization is performed using a model predictive control framework, solving the control oscillation problem caused by the large lag characteristics of buildings. Furthermore, adaptive technology during online testing dynamically tracks performance drift caused by equipment aging, correcting the terminal cost weight matrix and maintaining the energy-saving stability of the system throughout its entire lifecycle. Finally, interpretable decision logs are generated based on integral gradient attribution and a large language model, and offline strategy fine-tuning is performed based on feedback from maintenance personnel, reducing the rate of human error in takeover and ensuring the effective implementation of the intelligent control system.

[0020] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An adaptive energy-saving optimization method based on reinforcement learning, characterized in that, Includes the following steps: S1. Combining the thermal characteristics of the target physical space with real-time sensor data, construct a digital twin model of physical information, and generate a high-fidelity offline experience dataset based on the model. S2. Based on a high-fidelity offline experience dataset, a basic policy network is generated by using a reinforcement learning algorithm and introducing a start-stop penalty term. S3. Based on the physical information digital twin model and basic policy network, a time-series fusion transformer with embedded dynamic delay mask is used to represent the long-term state, and rolling time-domain optimization is performed based on the model predictive control framework to output the fine control command at the current moment. S4. Based on the actual feedback after the execution of fine control instructions, dynamically update the global statistics of the physical information digital twin model, and determine whether performance drift has occurred based on the real-time prediction residuals. S5. Based on fine-grained control instructions and contribution scores, a decision log is generated through a lightweight large language model. Combined with the adjustment instructions from operations and maintenance personnel, offline policy gradient adjustments are performed to complete the iteration of the policy network and generate an updated policy network.

2. The adaptive energy-saving optimization method based on reinforcement learning according to claim 1, characterized in that, The specific process of S1 is as follows: S11. Obtain the geometric and physical parameters of the target physical space, and construct the initial topology of the equivalent thermal resistance-thermal capacity network based on the lumped parameter method; use the physical partial differential equations of the equivalent thermal resistance-thermal capacity network as soft constraints, embed them into the loss function of the deep neural network, construct a physical information digital twin model, and train the physical information digital twin model through the total loss function; the input of the physical information digital twin model is the outdoor dry-bulb temperature. Solar radiation intensity Internal heat source power and historical control volume The output is the predicted indoor hotspot temperature. ; The total loss function is a weighted average of the data-driven loss and the physical residual loss, expressed as follows: ； In the formula, The total loss value of the physical information digital twin model; These are the trainable parameters for a deep neural network. This represents the number of samples in the batch data. For the first The model predicts the temperature for each sample. For the sensor to measure the first The return air temperature corresponding to each sample; This is the physical constraint weighting coefficient, with a value range of 0.1 to 10; Equivalent heat capacity; Equivalent thermal resistance; For continuous time variables; For the first The current cooling / heating supply of the air conditioning system corresponding to each sample; To predict the partial derivative of temperature with respect to time; For the first The outdoor dry-bulb temperature corresponding to each sample; For the first The internal heat source power corresponding to each sample; S12, Regarding the temperature of the inner wall surface of the building envelope Using the encoder-decoder structure of a pre-trained physical information digital twin model, along with the effective radiation temperature, easily measurable variables, including return air temperature, supply air temperature, and chilled water flow rate, are mapped to a high-dimensional latent space. The wall temperature is then reconstructed from the latent space by decoding. To obtain the current state vector ,in, This represents the current indoor hotspot temperature. S13. Generation of High-Fidelity Offline Experience Dataset: Using the parameters of the trained physical information digital twin model as a simulator, this simulator replaces the real physical environment. Within a safe boundary, the control variables are randomly disturbed and historical meteorological data are superimposed to perform Monte Carlo simulation, recording each step. The generated tuples are stored in a high-fidelity offline experience dataset. ,in The control action to be performed at the current moment; This is the state vector for the next time step; The immediate reward obtained at the current moment is the sum of the energy consumption penalty and the temperature over-limit penalty.

3. The adaptive energy-saving optimization method based on reinforcement learning according to claim 2, characterized in that, The specific process in S2 is as follows: S21. Employing a conservative Q-learning framework based on reinforcement learning algorithms, and introducing the maximization of the current policy distribution in addition to minimizing the standard Bellman error. The difference between the expected value of the dataset and the expected value of the high-fidelity offline empirical dataset is used as a regularization term to define a conservative Q-learning loss function. The expression is: In the formula, This is a conservative coefficient; To control the action vector; Let the expected value be the value under the current policy distribution. This represents the mathematical expectation of the current control action vector under the distribution of a high-fidelity offline experience dataset. Output to the current action value network; The mathematical expectation under the offline experience tuple distribution; This is the control action vector for the next moment; Value network for target actions; Output of the target action value network; Discount factor; S22. Reward Function Reshaping for Equipment Physical Losses: Based on the immediate reward in S13, a start-stop penalty term is added for reinforcement learning training to obtain the reshaped immediate reward. ; S23. Using data from a high-fidelity offline empirical dataset, minimize the value using gradient descent. Update the current action value network and update the base policy network using the deterministic policy gradient until the temporal difference error converges. Then, update the parameters of the converged base policy network. Solidification yields the converged basic policy network. ; The input is a full state space vector, and the output is a normalized continuous action. ; every time Step-by-step gradient update, using a soft update method to synchronize the current action value network parameters to the target action value network.

4. The adaptive energy-saving optimization method based on reinforcement learning according to claim 3, characterized in that, The specific process in S3 is as follows: S31. Use a time-series fusion converter to process past data. Historical state sequence at time 1 , The length of the historical state sequence. For time The state vector at time is used to introduce a dynamic delay mask matrix in multi-head attention computation. The attention weight allocation is corrected, expressed as follows: ； In the formula, For query matrix; The key matrix; It is a value matrix; This is the matrix transpose. The normalized attention weight matrix and value matrix Multiply by the product to obtain the final weighted aggregated features; It is a normalized exponential function; This is the scaling factor; The dimension of the key vector; S32. Rolling Time-Domain Optimization Based on Model Predictive Control Framework: The actions output by the converged basic policy network are used as the initial guesses and terminal cost constraints for model predictive control. The optimization objective function of model predictive control is... Set to predict time domain Within this context, the weighted sum of total energy consumption and thermal inequality is minimized, expressed as: ； In the formula, This is the current control moment; For the intermediate iteration index in the prediction time domain; Let be the control action vector to be solved; Basic policy network for state The given reference action vector; For the first The predicted state of the step; The penalty weight matrix is ​​used to control incremental / action deviation; Set point reference state; This is the penalty weight matrix for state deviation; This is the terminal cost weight matrix; To predict the predicted state of the time-domain terminal; Physical information digital twin model as an internal predictive model for model predictive control It is used to predict future states based on the current state and candidate actions. ; S33. Solving and Outputting the Optimal Control Command Sequence: In each rolling optimization step, the objective function is solved using a sequential quadratic programming algorithm. The optimal control command sequence that satisfies the physical constraints is obtained. , For a moment The multidimensional continuous action vector, which is the first element in the optimal control command sequence. As a precise control command for the current moment, it is sent to the fieldbus. This includes the chiller unit outlet water temperature setpoint, cooling tower fan frequency setpoint, and terminal air valve opening, outputting this... Simultaneously with the execution, the execution time state vector at the moment the instruction is issued is recorded for online adaptive updates in S4.

5. The adaptive energy-saving optimization method based on reinforcement learning according to claim 4, characterized in that, The specific process in S4 is as follows: S41. Acquire a new batch of sensor data that is fed back in real time after the execution of the fine control command. Calculate the actual mean of the current batch of data. With variance The global statistics, including the mean and variance of the batch normalized layer, are updated using an exponential moving average with a forgetting factor. The expression is as follows: ； ； In the formula, Forgetting factor; This represents the mean of the updated normalized layer. This represents the variance of the updated normalized layer. This is the mean of the normalized layer before the last update; The variance of the normalized layer before the last update; S42. Based on the actual return air temperature in the execution time state vector at the time the instruction was issued. Calculate and predict indoor hotspot temperature With actual return air temperature Real-time prediction residuals between , If residuals are predicted in real time Exceeding the preset residual threshold for 3 consecutive minutes If the performance of the physical information digital twin model is determined to be drifting, then the updated version will be used. and A physical information digital twin model is injected; simultaneously, using the updated global statistics of the physical information digital twin model, the current optimal control command sequence is re-derived. The corresponding predicted state trajectory is used, and the terminal cost weight matrix is ​​recalculated based on the new trajectory. .

6. The adaptive energy-saving optimization method based on reinforcement learning according to claim 5, characterized in that, The specific process in S5 is as follows: S51, in the policy network While deploying inference, the current state vector is computed in parallel. Based on the improved integral gradient method, the current control action vector is calculated with respect to the current state vector. Contribution scores of each dimension of features; S52. Sort the contribution scores in descending order, and use the top 3 contribution scores as key features. Construct a structured prompt word template, including the current time period, key features, and interference sources. ; The structured prompt word template is input into a lightweight large language model mounted on a local edge server. The structured prompt word template is translated into a decision log of natural language text and then pushed to the operation and maintenance management terminal. S53. Operations personnel, through the operations and maintenance management terminal, issue adjustment commands to the decision log for "approval" or "rejection and correction of suggested actions," which are recorded as interactive data with human preference tags. And store it in the human interaction dataset. , For the action labels corrected by operations and maintenance personnel, when the amount of interaction data in the human interaction dataset reaches a preset threshold, a combination of behavior cloning and near-end policy optimization is used to integrate the basic policy network obtained in S23. As the current strategy network The initial parameters, for Perform offline policy gradient adjustment to generate the updated policy network, expressed as: ； In the formula, To fine-tune the total loss offline; The mathematical expectation on the human-interactive dataset; This is the balance coefficient; The mathematical expectation of the current state vector on a high-fidelity offline empirical dataset; For the previous generation of policy networks; The dominant function; This is the clipping function; This is for pruning hyperparameters.

7. The adaptive energy-saving optimization method based on reinforcement learning according to claim 2, characterized in that, The expression for the immediate reward obtained at the current moment in S13 is: ； In the formula, This refers to the real-time power of the chiller unit. This represents the total power of the wind turbine group. This is the reference power under design conditions; This is the penalty coefficient for exceeding temperature limits; The upper limit of the safe operating temperature of the target physical space.

8. The adaptive energy-saving optimization method based on reinforcement learning according to claim 3, characterized in that, The redesigned instant rewards in S22 The expression is: ； In the formula, This is the start / stop indicator function; This is the penalty coefficient.

9. The adaptive energy-saving optimization method based on reinforcement learning according to claim 4, characterized in that, In S31, the dynamic delay mask matrix It is An upper triangular matrix, whose elements The definition of is: ； In the formula, This represents the equivalent lag step size of the current operating condition as identified in real time by the digital twin model of physical information; , These represent the row index and column index in the dynamic delay mask matrix, respectively.

10. The adaptive energy-saving optimization method based on reinforcement learning according to claim 6, characterized in that, The expression for calculating contribution score in S51 is: ； In the formula, For the first Contribution score of each dimension feature; The current state vector is the first... Features in one dimension; The first of the baseline state vectors Features in one dimension; Index for integration steps; This represents the number of integration steps. For the current policy network; This is the baseline state vector.

Images