Adaptive control method for heating, ventilation and air conditioning system based on multi-objective offline reinforcement learning

Abstract

This invention discloses an adaptive control method for HVAC systems based on multi-objective offline reinforcement learning, belonging to the field of intelligent control technology for HVAC systems. The method includes: pre-generating a control strategy network trained by reconstructing the temporal state of historical operating data based on the optimal time series length determined by autocorrelation analysis; collecting state variables at the current moment and the past N-1 moments, where N is the optimal time series length determined by autocorrelation analysis; constructing a two-dimensional temporal state matrix by arranging the collected state variables in chronological order based on the optimal time series length N; inputting the two-dimensional temporal state matrix into the control strategy network, extracting temporal features through a temporal convolutional layer, calculating and outputting the control action at the current moment through a fully connected layer, and sending it to the HVAC actuator. By accurately reflecting the temporal lag effect caused by building thermal inertia through the temporal state matrix, the pseudo-Markov problem is overcome, significantly improving the control stability of the HVAC system.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for HVAC systems, specifically an adaptive control method for HVAC systems based on multi-objective offline reinforcement learning. Background Technology

[0002] With the increasing global requirements for building energy conservation and emission reduction, the intelligent and energy-saving operation and control of central air conditioning (HVAC) systems in large commercial buildings and industrial parks, as major energy consumers, has become a key technical bottleneck.

[0003] Existing HVAC control methods, including traditional PID control and data-driven reinforcement learning, all assume a single-step state assumption, meaning the system's state at the next moment depends only on the current state and the currently executed action. However, HVAC systems exhibit strong time lag and building thermal inertia; the effect of a control action often takes several minutes or even tens of minutes to materialize in state variables such as indoor temperature. When the single-step state assumption is applied to HVAC systems with strong time lag, the model cannot establish long-term state relationships, resulting in a pseudo-Markov problem and causing the control strategy to fail in real-world systems. Summary of the Invention

[0004] The purpose of this invention is to provide an adaptive control method for HVAC systems based on multi-objective offline reinforcement learning to solve the above-mentioned problems.

[0005] The technical solution of this invention is: An adaptive control method for HVAC systems based on multi-objective offline reinforcement learning includes the following steps: A pre-generated control strategy network is obtained, which is trained by reconstructing the time-series state of historical operating data based on the optimal time-series length determined by autocorrelation analysis.

[0006] The system collects state variables of the HVAC system at the current moment and at the past N-1 moments. The state variables include variables such as indoor temperature, equipment operating power, and equipment operating frequency. N is the optimal time series length determined by the autocorrelation analysis, which is used to characterize the time series lag effect caused by building thermal inertia.

[0007] Based on the optimal time series length, the collected state variables at the current time and the past N-1 time series are arranged in time series to construct a two-dimensional time series state matrix.

[0008] The two-dimensional temporal state matrix is ​​input into the control policy network. The control policy network extracts temporal features through a temporal convolutional layer and then calculates and outputs the control action at the current moment through a fully connected layer.

[0009] The control actions are sent to the HVAC actuators. These control actions include the chilled water supply temperature of the main unit, the equipment operating frequency, and the number of operating units.

[0010] Furthermore, the training method for the control policy network includes the following steps: The historical operating dataset of the heating, ventilation, and air conditioning system is used to perform autocorrelation analysis on the state variables in the historical operating dataset, and a curve of the autocorrelation coefficient changing with the lag step is plotted. The optimal time series length N is determined based on the computational cost of different time series lengths and the decay trend of the autocorrelation coefficient.

[0011] Based on the optimal time series length N, the single-step state vector in the historical running data is reconstructed into a time series state matrix containing N consecutive time steps.

[0012] Using the aforementioned temporal state matrix as training samples, a multi-objective offline reinforcement learning algorithm is employed to iteratively update and train the policy network, thereby obtaining the control policy network.

[0013] Furthermore, a temporal convolutional network is used to extract the temporal features of the temporal state matrix; three independent dual-evaluation networks are used to evaluate the Q-values ​​of energy consumption, comfort, and equipment aging, respectively, to guide the updating of the policy network.

[0014] The policy network summarizes the weighted Q-values ​​of the three objectives: ,in, It is the weighted total Q value, , , These are the weighting coefficients for energy consumption, comfort, and equipment aging targets, respectively. , , These are evaluations of network output regarding energy consumption, comfort, and equipment aging. Q value.

[0015] Furthermore, to address the issue that features of different dimensions may cause gradient instability and imbalance in the distribution of time-series vectors during the extraction of time-series features, the time-series convolutional network includes a batch normalization layer to smooth the internal covariance shift of features of different dimensions; after the time-series features are flattened, a layer normalization layer is introduced to stabilize the distribution of time-series vectors.

[0016] Furthermore, existing offline reinforcement learning methods lack strict constraints on historical safety boundaries, and are prone to outputting dangerous actions that could damage devices when faced with out-of-distribution states. To address this, a behavior cloning constraint term is introduced during policy network updates. This term calculates the mean squared error between the action output by the policy network and the action in the corresponding state in historical data. This error is added as a penalty to the total loss function of the policy network to limit the output action of the policy network to a safe range supported by historical data.

[0017] The total loss function of the policy network is: ;in, These are the adaptive weighting coefficients for the behavioral cloning error term. This is the calculated total Q value, and MSE is the mean squared error. This is the action predicted by the current Actor network. These are safety actions recorded in historical data.

[0018] To fully exploit the optimal potential of historical data while ensuring that actions do not deviate significantly from the historical safety range, a fixed behavior cloning weight α is introduced, and the adaptive adjustment ratio λ is calculated using the following formula: .

[0019] in, These are the adaptive weighting coefficients for the behavioral cloning error term. These are hyperparameter coefficients. This refers to the batch size.

[0020] Furthermore, sensor data such as indoor temperature collected in real engineering environments often contain a large amount of error and high-frequency jump noise caused by control dead zones, which disrupts the influence of building thermal inertia on the data, making it impossible for the control model to capture the real thermodynamic evolution. Before performing autocorrelation analysis on the state variables in the historical operation dataset, a moving average filter is used to smooth the state variables to identify state variables containing high-frequency quantization noise. The filtering formula is as follows: .

[0021] in, The filtered value. These are the raw sensor values ​​acquired at time step ti. t For the current time step, W The set sliding window size, This is the smaller of the actual available data length and the window size.

[0022] Furthermore, since the magnitude of rewards varies greatly across dimensions in multi-objective reinforcement learning, which can easily lead to Q-value divergence in the evaluation network, training also includes: traversing the historical running dataset to obtain the global maximum values ​​of rewards for the three dimensions of energy consumption, comfort, and equipment aging. and minimum value Mapping the rewards for each dimension to the interval [-1, 0], the normalization formula is: ;in, To prevent the elimination of zero terms, This refers to the original penalty value calculated for a certain dimension at time step t. , The global minimum and maximum values ​​for the reward in this dimension.

[0023] Furthermore, since historical data may contain abnormal peak values ​​that could interfere with the stability of network training, Huber Loss is used as the loss function for evaluating the network during training. The formula is as follows: .

[0024] .

[0025] in, Let i be the target Q value at time step t. The three objectives to be evaluated are: energy consumption, comfort, and equipment aging. To calculate the normalized reward for this objective, As a discount factor for rewards, The next state Q value predicted by the evaluation network for the j-th target is... The state at the next time step. To incorporate smoothed noise into the action of the next time step, This represents the total loss value of the Critic network. This is the Q-value prediction given by the current Critic network for the current state and action.

[0026] Furthermore, since the learning rate is difficult to dynamically adapt to the model convergence requirements during training, a cosine annealing learning rate scheduler is used to dynamically adjust the learning rate of the policy network during training.

[0027] Compared with the prior art, the beneficial effects of the present invention are: This invention adaptively determines the optimal time series length through autocorrelation analysis and reconstructs the single-step state into a time series state matrix that includes historical evolution trends. This enables the control model to accurately reflect the time series lag effect caused by building thermal inertia, avoiding the pseudo-Markov problem caused by the default single-step state assumption in existing HVAC control methods. This allows the control strategy network to make decisions based on a full understanding of the system's historical evolution, thus avoiding the loss of state correlation and control strategy oscillations caused by physical lag effects. Furthermore, by inputting the time series state matrix containing the current and N-1 past times into the strategy network for forward computation, an end-to-end mapping from historical data to current control actions is achieved, ensuring consistency between control decisions and the actual thermodynamic characteristics of the building. This significantly improves the control stability and operational safety of HVAC systems in multi-objective optimization scenarios. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0029] Figure 2 This is a diagram of the existing Temporal Convolutional Network (TCN) topology.

[0030] Figure 3 This is a diagram of the existing TD3 network topology.

[0031] Figure 4 This invention provides an improved TD3 network topology diagram.

[0032] Figure 5 The diagram shows the autocorrelation analysis of the state variables in this invention, where (a) is the outdoor temperature, (b) is the chilled water supply temperature, (c) is the system cooling load, (d) is the cooling water flow rate, (e) is the wet-bulb temperature, and (f) is the chilled water flow rate.

[0033] Figure 6 This is the autocorrelation diagram of the state variables before the smoothing process of this invention.

[0034] Figure 7 This is the autocorrelation diagram of the state variables after smoothing processing according to the present invention.

[0035] Figure 8 The figure shows the verification results of different timing lengths (SEQ_LEN) of the present invention. (a) shows the trend of BC error and average Q value with SEQ_LEN, and (b) shows the comparison of the calculation time of different SEQ_LEN.

[0036] Figure 9 This is a graph showing the change in the estimated Q-value of the multi-objective evaluation network of this invention.

[0037] Figure 10 This is a graph showing the convergence curve of the Huber loss in the evaluation network of this invention.

[0038] Figure 11 This is a graph showing the change in the total loss function of the strategy network in this invention.

[0039] Figure 12 This is a convergence curve of the behavior cloning and imitation error of the present invention. Detailed Implementation

[0040] The following is combined Figures 1 to 12 The specific embodiments of the present invention will be described in detail below.

[0041] Example like Figure 1 As shown, an adaptive control method for HVAC systems based on multi-objective offline reinforcement learning includes the following steps: A pre-generated control strategy network is obtained. The control strategy network is trained by reconstructing the time-series state of historical operating data based on the optimal time-series length determined by autocorrelation analysis. In this embodiment, the control strategy network is an Actor network, whose network structure includes an input layer, multiple fully connected layers and an output layer. The activation function is the tanh function. The output layer is multiplied by the maximum boundary value allowed by the physical action of the system to output the control action that conforms to the physical constraints of the device.

[0042] The system collects state variables of the HVAC system at the current moment and for the past N-1 moments. The state variables include indoor temperature, equipment operating power, equipment operating frequency, etc., where N is the optimal time series length determined by autocorrelation analysis, used to characterize the time series lag effect caused by building thermal inertia. In this embodiment, N=24, that is, the system collects state variables at the current moment and for the past 23 moments, for a total of 24 consecutive time steps of state data.

[0043] Based on the optimal time series length, the collected state variables at the current time and the past N-1 time series are arranged in time series to construct a two-dimensional temporal state matrix; specifically, the state matrix at any time t is represented as a two-dimensional tensor containing N consecutive time steps: .

[0044] in S t Let be the single-step state vector at time t. R Let N be the set of real numbers and N be the time step. Dstate It is the number of state variables.

[0045] A two-dimensional temporal state matrix is ​​input into the control policy network. The control policy network extracts temporal features through temporal convolutional layers, and then calculates and outputs the control action at the current moment through a fully connected layer; for example... Figure 2As shown, the temporal convolutional network includes a batch normalization layer. The input matrix is ​​first transposed and then fed into a TCN layer containing one-dimensional convolution and batch normalization to extract temporal features. Finally, the feature vector is output through a flattening layer and a LayerNorm layer. This feature vector is then input into a fully connected layer to output the control action at the current time step.

[0046] The control actions are sent to the HVAC actuators. These control actions include the chilled water supply temperature of the main unit, the operating frequency of the equipment, and the number of units in operation.

[0047] In some embodiments, the method for training a control policy network includes the following steps: Historical operating datasets of HVAC systems are used to perform autocorrelation analysis on the state variables in these datasets. A curve showing the autocorrelation coefficient changing with the lag step size is plotted. The optimal time series length N is determined based on the computational cost of different time series lengths and the decay trend of the autocorrelation coefficient. Figure 5 As shown, this embodiment performs autocorrelation analysis on state variables such as outdoor temperature, chilled water supply temperature, system cooling load, cooling water flow rate, wet-bulb temperature, and chilled water flow rate, obtaining the curves of the autocorrelation coefficients of each variable as a function of the lag step. Figure 6 and Figure 7 As shown, after smoothing with a moving average filter, the noise in the indoor temperature autocorrelation plot is significantly reduced, and the thermal inertia trend is clearer. Figure 8 As shown, by combining the BC error, average Q value and computation time of different SEQ_-LENs, the optimal timing length N=24 is determined.

[0048] Specifically, the formula for calculating the autocorrelation coefficient is: .

[0049] in, The autocorrelation coefficient is the value of the autocorrelation coefficient when the lag number is k. The number of lag steps, For the sequence in The value at time, The average value of the sequence. The total number of samples used to calculate correlation. T This represents the total length of the data.

[0050] Based on the optimal time series length N, the single-step state vector in the historical running data is reconstructed into a time series state matrix containing N consecutive time steps. In this embodiment, for each time t in the historical dataset, the state data of the previous N-1 times is extracted and combined with the state data of the current time to form a time series state matrix. Data that cannot form a complete time series state matrix is ​​skipped directly.

[0051] Using the temporal state matrix as training samples, a multi-objective offline reinforcement learning algorithm is employed to iteratively update and train the policy network, resulting in a control policy network. In this embodiment, an offline reinforcement learning algorithm is used, utilizing historical datasets for batch training without online interaction with the environment to avoid the risk of device damage caused by online trial and error. During training, an optimal model saving mechanism is implemented, saving the Actor network weights with the best performance as the final control policy network.

[0052] In some embodiments, a temporal convolutional network is used to extract temporal features of the temporal state matrix; three independent dual-evaluation networks are used to evaluate the Q-values ​​of energy consumption, comfort, and equipment aging, respectively, to guide the updating of the policy network. Figure 3 and Figure 4 As shown, specifically, this embodiment constructs three independent Twin-Critic networks. Compared with the existing TD3 network, this embodiment introduces TCN temporal feature extraction and multi-objective evaluation network architecture. Each group contains two evaluation networks with the same structure but initialized parameters. The smaller value of the two outputs is taken as the Q-value estimate for that dimension to suppress the problem of Q-value overestimation.

[0053] The policy network summarizes the weighted Q-values ​​of the three objectives: ;in, It is the weighted total Q value, , , These are the weighting coefficients for energy consumption, comfort, and equipment aging targets, respectively. , , These are evaluations of network output regarding energy consumption, comfort, and equipment aging. Q The values ​​and weighting coefficients can be adjusted according to the actual engineering needs.

[0054] In some embodiments, to address the issues of gradient instability and potential imbalance in temporal vector distribution caused by features of different dimensions during temporal feature extraction, the temporal convolutional network includes a batch normalization layer to smooth the internal covariance shift of features of different dimensions; after flattening the temporal features, a layer normalization layer is introduced to stabilize the distribution of the temporal vector, such as... Figure 2 As shown, the batch normalization layer (BatchNorm1d) is applied after each convolutional layer of the TCN to normalize the input for each batch, accelerating network convergence and reducing sensitivity to parameter initialization. After the temporal features pass through the flatten layer, a layer normalization layer (LayerNorm) is introduced to normalize each sample of the feature vector independently, preventing gradient vanishing or weight imbalance during multi-objective weighted optimization.

[0055] In some embodiments, existing offline reinforcement learning methods lack strict constraints on historical safety boundaries, and are prone to outputting dangerous actions that could damage devices when faced with out-of-distribution states. When updating the policy network, a behavior cloning constraint term is introduced. This term calculates the mean squared error between the action output by the policy network and the action in the corresponding state in historical data. This error is added as a penalty term to the total loss function of the policy network to limit the output action of the policy network to a safe range supported by historical data. Figure 4 As shown, when the policy network is updated, a behavior cloning constraint term is introduced. The behavior cloning constraint term calculates the mean square error between the action output by the policy network and the action in the corresponding state in the historical data. This error is added as a penalty term to the total loss function of the policy network to limit the output action of the policy network to a safe range supported by the historical data.

[0056] The total loss function of the policy network is: ,in These are the adaptive weighting coefficients for the behavioral cloning error term. It is the calculated total Q The value, MSE is the mean squared error. This is the action predicted by the current Actor network. These are safety actions recorded in historical data.

[0057] To fully exploit the optimal potential of historical data while ensuring that actions do not deviate significantly from the historical safety range, a fixed behavior cloning weight α is introduced, and the adaptive adjustment ratio λ is calculated using the following formula: .

[0058] in, These are the adaptive weighting coefficients for the behavioral cloning error term. These are hyperparameter coefficients. This refers to the batch size.

[0059] like Figure 11 As shown, the total loss function of the policy network in this embodiment gradually converges during training. Figure 12 As shown, the behavioral cloning and imitation error decreases with the increase of training rounds, indicating that the actions output by the policy network are similar to the safe actions in historical data, but also take into account the three dimensions of energy saving, comfort and aging.

[0060] In some embodiments, sensor data such as indoor temperature collected in real engineering environments often contain a large amount of error and high-frequency jump noise caused by control dead zones, which disrupts the influence of building thermal inertia on the data, making it impossible for the control model to capture the real thermodynamic evolution. Before performing autocorrelation analysis on the state variables in the historical operating dataset, a moving average filter is used to smooth the state variables to identify state variables containing high-frequency quantization noise. The filtering formula is as follows: .

[0061] in, The filtered value These are the raw sensor values ​​acquired at time step ti. t For the current time step, W The set sliding window size, This is the smaller of the actual available data length and the window size.

[0062] like Figure 6 and Figure 7 As shown, after smoothing by the moving average filter, the high-frequency noise in the autocorrelation plot is significantly reduced, and the physical thermal inertia trend is clearer.

[0063] In some embodiments, since the magnitude difference of rewards in each dimension in multi-objective reinforcement learning is huge and can easily lead to divergence of the Q-value of the evaluation network, the training process also includes: traversing the historical running dataset to obtain the global maximum value of the rewards in the three dimensions of energy consumption, comfort, and equipment aging. and minimum value Mapping the rewards for each dimension to the interval [-1, 0], the normalization formula is: ;in, To prevent the elimination of zero terms, This refers to the original penalty value calculated for a certain dimension at time step t. , The global minimum and maximum values ​​for the reward in this dimension.

[0064] Among them, the energy consumption reward is Total system operating power.

[0065] Comfort bonus is Penalty for indoor temperature deviating from the set value. The calculation formula is: ,in, As a comfort penalty item, Let be the actual indoor temperature at time t. The set target temperature value.

[0066] Equipment aging bonus This includes start-up and shutdown penalties for equipment such as water pumps and cooling towers, as well as penalties for high-load operation of equipment. The calculation formula is: .

[0067] in, This is the difference between the number of devices started and stopped. For operating frequency, This is a penalty item for equipment aging. The coefficient weight for equipment start-up and shutdown penalties. The weighting of the penalty coefficient for high-load operation of equipment. This is a preset high load threshold.

[0068] like Figure 9 As shown in the figure, the Q-value variation curves estimated by the multi-objective evaluation network in this embodiment show that the Q-values ​​in all three dimensions converge stably after normalization. Figure 10 As shown, the Huber loss of the evaluation network gradually decreases with the increase of training rounds, indicating that the prediction accuracy of the evaluation network is continuously improving.

[0069] In some embodiments, since historical data may contain abnormal peak values ​​that interfere with the stable training of the evaluation network, Huber Loss is used as the loss function for evaluating the network during training, and the formula is as follows: .

[0070] .

[0071] in, Let i be the target Q value at time step t. The three objectives to be evaluated are: energy consumption, comfort, and equipment aging. To calculate the normalized reward for this objective, As a discount factor for rewards, The next state Q value predicted by the evaluation network (out of 2) for the j-th target is... The state at the next time step. To incorporate smoothed noise into the action of the next time step, This represents the total loss value of the Critic network. This is the Q-value prediction given by the current Critic network for the current state and action.

[0072] The actor's total loss function consists of the RL maximization objective term and the BC action error term: .

[0073] like Figure 10 As shown, after adopting Huber Loss, the loss curve of the evaluation network is smoother, and the interference of abnormal peak samples on gradient updates is effectively suppressed.

[0074] In some embodiments, because the learning rate is difficult to dynamically adapt to the model convergence requirements during training, a cosine annealing learning rate scheduler is used to dynamically adjust the learning rate of the policy network during training.

[0075] The above-disclosed embodiments are merely preferred embodiments of the present invention. However, the embodiments of the present invention are not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. An adaptive control method for HVAC systems based on multi-objective offline reinforcement learning, characterized in that, Includes the following steps: A pre-generated control strategy network is obtained, which is trained by reconstructing the time-series state of historical operating data based on the optimal time-series length determined by autocorrelation analysis. The state variables of the HVAC system at the current moment and the past N-1 moments are collected, where N is the optimal time series length determined by the autocorrelation analysis, which is used to characterize the time series lag effect caused by building thermal inertia; Based on the optimal time series length, the collected state variables at the current time and the past N-1 times are arranged in time series to construct a two-dimensional time series state matrix; The two-dimensional temporal state matrix is ​​input into the control policy network. The control policy network extracts temporal features through a temporal convolutional layer and then calculates and outputs the control action at the current moment through a fully connected layer. The control action is sent to the HVAC actuator.

2. The adaptive control method for HVAC systems based on multi-objective offline reinforcement learning according to claim 1, characterized in that, The training method for the control policy network includes the following steps: The historical operation dataset of the heating, ventilation, and air conditioning system is used to perform autocorrelation analysis on the state variables in the historical operation dataset, and a curve of the autocorrelation coefficient changing with the lag step is plotted. The optimal time series length N is determined based on the computational cost of different time series lengths and the decay trend of the autocorrelation coefficient. Based on the optimal time series length N, the single-step state vector in the historical running data is reconstructed into a time series state matrix containing N consecutive time steps; Using the aforementioned temporal state matrix as training samples, a multi-objective offline reinforcement learning algorithm is employed to iteratively update and train the policy network, thereby obtaining the control policy network.

3. The adaptive control method for HVAC systems based on multi-objective offline reinforcement learning according to claim 2, characterized in that, Temporal features of the temporal state matrix are extracted using a temporal convolutional network; three independent dual-evaluation networks are used to evaluate the Q-values ​​of energy consumption, comfort, and equipment aging to guide the updating of the policy network.

4. The adaptive control method for HVAC systems based on multi-objective offline reinforcement learning according to claim 3, characterized in that, The temporal convolutional network includes a batch normalization layer to smooth the internal covariance shift of features with different dimensions; After flattening the temporal features, a normalization layer is introduced to stabilize the distribution of the temporal vectors.

5. The adaptive control method for HVAC systems based on multi-objective offline reinforcement learning according to claim 3, characterized in that, When the policy network is updated, a behavior cloning constraint term is introduced. The behavior cloning constraint term calculates the mean square error between the action output by the policy network and the action in the corresponding state in the historical data. This error is added as a penalty term to the total loss function of the policy network to limit the output action of the policy network to a safe range supported by the historical data.

6. The adaptive control method for HVAC systems based on multi-objective offline reinforcement learning according to claim 2, characterized in that, Before performing autocorrelation analysis on the state variables in the historical running dataset, a moving average filter is used to smooth the state variables in order to identify state variables containing high-frequency quantization noise.

7. The adaptive control method for HVAC systems based on multi-objective offline reinforcement learning according to claim 3, characterized in that, Before training, the process also includes: traversing the historical running dataset to obtain the global maximum value of the rewards across three dimensions: energy consumption, comfort, and device aging. and minimum value Mapping the rewards for each dimension to the interval [-1, 0], the normalization formula is: ； in, To prevent the elimination of zero terms, This refers to the original penalty value calculated for a certain dimension at time step t. , The global minimum and maximum values ​​for the reward in this dimension.

8. The adaptive control method for HVAC systems based on multi-objective offline reinforcement learning according to claim 2, characterized in that, During training, Huber Loss is used as the loss function to evaluate the network, and the formula is: ； ； in, Let i be the target Q value at time step t. The three objectives to be evaluated are: energy consumption, comfort, and equipment aging. To calculate the normalized reward for this objective, As a discount factor for rewards, The next state Q value predicted by the evaluation network for the j-th target is... The state at the next time step. To incorporate smoothed noise into the action of the next time step, This represents the total loss value of the Critic network. This is the Q-value prediction given by the current Critic network for the current state and action.

9. The adaptive control method for HVAC systems based on multi-objective offline reinforcement learning according to claim 2, characterized in that, During training, a cosine annealing learning rate scheduler is used to dynamically adjust the learning rate of the policy network.

Images