Model-free energy-saving control method for heating, ventilation and air conditioning based on maximum likelihood state estimation

Abstract

The present application relates to a kind of model-free heating ventilation air conditioning energy-saving control method based on maximum likelihood state estimation, comprising the following steps: obtaining historical state parameter set and current state parameter set, and based on the historical state parameter set and current state parameter set, the state parameter set of current time to next time is predicted, obtains predicted state parameter set;Based on the predicted state parameter set, the probability distribution is estimated using frequency method, and the maximum likelihood corresponding state is taken as the environment state;Current time control command is obtained as the action space of intelligent agent;The time cumulative heating system performance parameter from current time to next time is obtained, and the reward function is constructed;Based on the environment state, action space and reward function, using model-free reinforcement learning algorithm constructs intelligent agent and trains, finally obtains control action and outputs action value.Compared with prior art, the present application has the advantages of improving the energy-saving effect of heating system control.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of HVAC energy-saving control, and particularly relates to a model-free HVAC energy-saving control method based on maximum likelihood state estimation. BACKGROUND

[0002] In the current model-free Q-Learning control of the HVAC system, the collection of current state data, the acquisition and issuance of control commands, and the collection of next reward data are performed in one control and training cycle. The performance instantaneous parameters of the HVAC system at the next data collection time are taken as the state at the current time and the reward function of the issued control command. Through the training of the agent, the control command that is most beneficial to the next state instantaneous HVAC system performance under a certain state is obtained, so as to improve the overall HVAC system performance. In this control logic, the optimization degree of the content of the control command is limited to the current instantaneous state and the next state instantaneous HVAC system performance.

[0003] The control issuance, state change and performance feedback of the HVAC system have a process nature. The control issuance has a certain delay and asymptotic, which affects the performance of the HVAC system in the whole process between two state points. However, in the model-free Q-Learning control without state estimation, due to the continuity of state change, the delay of control content completion and the globality of system performance in the actual situation, the current environmental state is not the environmental state when the control content is actually completed. The reward function lacks the feedback of the influence of the control content on the whole process HVAC system performance, which leads to the deviation of the state value used for training and acquiring the control command, the lack of globality of the reward, and the inaccuracy of the training model, which cannot achieve higher performance effect. Therefore, an estimated state representing the control completion process of each state and an optimization and control method considering the whole process performance improvement are needed. SUMMARY

[0004] The purpose of the present application is to provide a model-free HVAC energy-saving control method based on maximum likelihood state estimation, which improves the energy-saving effect of HVAC system control.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] A model-free HVAC energy-saving control method based on maximum likelihood state estimation, comprising the following steps:

[0007] Obtain a historical state parameter set and a current state parameter set, and predict the state parameter set from the current time to the next time based on the historical state parameter set and the current state parameter set to obtain a predicted state parameter set;

[0008] Based on the predicted state parameter set, the probability distribution is estimated using the frequency method, and the state corresponding to the maximum likelihood rate is taken as the environmental state.

[0009] Obtain the control commands at the current moment, which serve as the action space for the agent;

[0010] Obtain the cumulative HVAC system performance parameters from the current moment to the next moment, and construct a reward function;

[0011] Based on the environmental state, action space, and reward function, a model-free reinforcement learning algorithm is used to construct and train an agent, which ultimately obtains control actions and outputs action values.

[0012] Furthermore, a long short-term memory network is used for prediction.

[0013] Furthermore, the specific steps for obtaining the environmental state include:

[0014] The predicted state parameter set is preprocessed;

[0015] Based on the preprocessed predicted state parameter set, the probability distribution is estimated using the frequency method, and the estimated parameter values ​​are calculated.

[0016] The likelihood function is calculated based on the estimated parameter values ​​to obtain the maximum likelihood rate, and the corresponding state is taken as the environmental state.

[0017] Furthermore, the preprocessing operation is nearest neighbor multiple rounding, expressed as:

[0018] S = round(S′ / b) × b

[0019] Where S represents the processed data, S′ represents a certain type of state parameter in the predicted state parameter set, round() represents the rounding function, and b represents the value of the nearest neighbor multiple.

[0020] Furthermore, the expression for estimating the probability distribution using the frequency method is as follows:

[0021] P(X=x i Y = y i ,...)=f(x i ,y i ,...) / q

[0022] Where variables X, Y, etc. represent different types of state parameters in the predicted state parameter set, x i y i Equals represents the value of the i-th prediction parameter set after nearest neighbor rounding; P(X=x i Y = y i ,...) indicates that when the predicted state parameters X, Y, etc. take the value xi i i i i i

[0023] Further, the expression of the reward function is:

[0024] R = u E ×E + u C ×C

[0025] Wherein, R represents the reward function, u E , u C represent the weight of each type of parameter participating in the reward calculation, E represents the energy consumption parameter score participating in the reward calculation, and C represents the comfort parameter score participating in the reward calculation.

[0026] Further, the expression of the energy consumption parameter score is:

[0027]

[0028] Wherein, E represents the energy consumption parameter score, e represents the energy consumption parameter per unit time, represents the time cumulative value of the parameter value from the current time t0 to the next data collection time t n .

[0029] Further, the expression of the comfort parameter score is:

[0030]

[0031] Wherein, C represents the comfort parameter score, c i represents the comfort parameter value at a single moment, c i,set represents the standard value of the comfort parameter at the moment, i represents the number of data collection times, n represents the number of data collection times, and abs() represents the absolute value function.

[0032] Further, the model-free reinforcement learning algorithm is a Q-learning algorithm.

[0033] Further, the Q-learning algorithm updates the Q value by continuous iteration to make the intelligent agent take control actions, and the expression of the updated Q value is:

[0034] Q(s,a)←Q(s,a)+α[r+γmaxQ(s′,a′)-Q(s,a)]

[0035] ​​​​​​Wherein, the arrow indicates that the left variable is updated by the right calculation value, Q(s,a) indicates the action-state Q value of taking action a in environment state s, s' indicates the next time environment state, maxQ(s',a') indicates the known maximum Q value in state s', r is the reward brought by the last round of action-state combination, a is the learning rate, and gamma indicates the discount factor.

[0036] Compared with the prior art, the present application has the following beneficial effects:

[0037] (1) The present application estimates the next moment by collecting the historical and current parameter sets, considers the continuity of the state change of the heating and ventilation system, the delay of the control content completion and the globality of the system performance, estimates the state representing the control completion process by comprehensively estimating the current value and the predicted value, constructs a time cumulative reward function, considers the whole process performance improvement, makes the training model more accurate and the performance effect higher, and thus significantly improves the energy saving effect of the heating and ventilation system control.

[0038] (2) The present application adopts the Q-learning model-free learning method, and the Q-learning has model independence and does not need to model the environment in advance. It learns the optimal strategy by interacting with the environment, avoids the assumption of the dynamic characteristics of the environment and the complexity of model construction, and improves the calculation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The present application is a flowchart. DETAILED DESCRIPTION

[0040] The present application will be described in detail below in combination with the drawings and specific embodiments. The present embodiment is implemented on the premise of the technical scheme of the present application, and gives a detailed implementation manner and specific operation process, but the protection scope of the present application is not limited to the following embodiments.

[0041] The present embodiment provides a model-free heating and ventilation air conditioning energy saving control method based on maximum likelihood state estimation, as shown in the figure, the method comprises the following steps: Figure 1

[0042] S1, obtain a historical state parameter set and a current state parameter set, and predict the state parameter set from the current moment to the next moment based on the historical state parameter set and the current state parameter set, to obtain a predicted state parameter set.

[0043] Collect historical data and current data affecting the energy consumption of the heating and ventilation system, and construct a historical state parameter set and a current state parameter set respectively. In the present embodiment, the current moment and the next moment are the current moment and the next moment of collecting data respectively.

[0044] ​The predicted state parameter set can be trained using historical data by an algorithm such as a long short-term memory (LSTM) network, and the historical state parameter set and the current state parameter set are used as inputs to the trained prediction model to predict the next state parameter set, and a prediction model with a mean absolute percentage error (MAPE) of less than 5% is selected.

[0045] S2, based on the predicted state parameter set, a frequency method is used to estimate the probability distribution, and the state corresponding to the maximum likelihood is taken as the environment state.

[0046] Q-learning is a table-based reinforcement learning method, and the state quantity parameters need to be discretized, so the formula for rounding the nearest neighbor multiple of the predicted state parameter set between the current and next data collection is

[0047] S = round(S' / b) * b

[0048] where S represents the processed data, S' represents a state parameter of a certain type in the predicted state parameter set, round() represents a rounding function, and b represents the value of the nearest neighbor multiple. The value range of different types of states can be freely defined, for example, the nearest neighbor multiple of the cooling load is 20 times, and the nearest neighbor multiple of the wet bulb temperature is 1 time.

[0049] In this embodiment, the basic state parameters and the estimated state parameters are taken as the basic state parameter group, where the basic state parameter group is a state group composed of the predicted state parameter set between the current and next data collection, and is used to calculate the estimated state, which is the environment state used for agent training.

[0050] First, the predicted state parameter set between the current and next data collection, i.e., the basic state parameter group, is subjected to nearest neighbor multiple rounding processing to discretize the basic state parameter group and reduce its complexity to a discrete state space. By arranging and combining the values of various types of states to form state tuple parameters, for example, the state quantity of the cooling load being 1203 kW and the wet bulb temperature being 26°C is (1200, 26).

[0051] The frequency method is used to estimate the probability distribution of each state parameter tuple in the discretized basic state group, and the calculation formula is

[0052] P(X=x i ,Y=y i ,...) = f(x i ,y i ,...) / q

[0053] where variables X, Y, etc. represent different types of state parameters in the state parameter set, xi , y i ,...), where P(X=x i , Y=y i ,...) represents the probability that the state variables X, Y, etc. take the values x i , y i , etc., f(x i , y i ,...) represents the frequency of observing the values x i , y i , etc. in the sample, and q represents the total number of samples.

[0054] In the embodiment, when the state is the group [(1200, 26), (1220, 26)], P(X=1200, Y=26)=1 / 2, and P(X=1220, Y=26)=1 / 2.

[0055] When the state group is [(1200, 26), (1220, 26), (1200, 26)], P(X=1200, Y=26)=2 / 3, and P(X=1220, Y=26)=1 / 3.

[0056] After using the nearest neighbor rounding and the frequency method to estimate the probability distribution process of the basic state parameter group, the estimated state, i.e. the environment state in the agent training, is obtained by taking the maximum likelihood value of the probability distribution. For a number of discrete state points, the maximum likelihood value is the maximum probability point. When the current and predicted states are uniformly distributed, the estimated state takes the mean value of the states in the group. For example, when the state is the group [(1200, 26), (1220, 26)], the estimated state is (1220, 26), and when the state is the group [(1200, 26), (1220, 26), (1200, 26)], the estimated state is (1200, 26). The estimated state also needs to be subjected to the same nearest neighbor rounding process. The state space is composed of the collection of all estimated state quantities.

[0057] S3, obtaining the control command currently required to be issued in the heating, ventilation and air conditioning system as the action space of the agent.

[0058] The content of the control command currently required to be issued in the heating, ventilation and air conditioning system is taken as the action space.

[0059] Action space typically refers to the set of all possible actions an agent can choose. In HVAC energy-saving control, common actions include water pump frequency and number of pumps. The action range is set based on control precision and equipment safety limits. Common cooling water pump frequency ranges are 30-50Hz. Q-learning is a table-based reinforcement learning method; the action quantities need to be discretized, the action interval can be set to 2Hz, and the number of pumps interval is precisely 1 pump.

[0060] S4. Obtain the cumulative HVAC system performance parameters from the current moment to the next data acquisition moment, and use them as the reward function.

[0061] The reward function is the optimization objective of the agent, which can be used to evaluate the quality of each action of the agent in the environment. In HVAC energy-saving control, the reward function variables include, but are not limited to, two major categories of HVAC system performance parameters, such as total power consumption and room temperature.

[0062] The parameters in the reward function are all taken as the cumulative time value from the current time to the next data acquisition time, and are calculated by weighting the scores of multiple types of HVAC system performance parameters. The reward function formula is:

[0063] R = u E ×E+u C ×C

[0064] Where R represents the reward function, u E u C This represents the weight of each type of variable involved in the reward calculation. E and C represent the scores of energy consumption and comfort variables involved in the reward calculation, respectively. The u value can be set as a constant or a variable correlation function depending on the optimization focus. The scoring formula for energy consumption parameters is:

[0065]

[0066] Where E represents the energy consumption parameter score, and e represents the energy consumption parameter per unit time. This indicates that the parameter value is from the current time t0 to the next data acquisition time t. n The cumulative time value within the period; the formula for scoring comfort parameters is:

[0067]

[0068] Where C represents the comfort category parameter score, c i c represents the value of comfort parameters at a single moment. i,set This represents the standard value of comfort parameters at that moment, i represents the number of data collection sessions, n represents the number of data collection sessions, and abs() represents the absolute value function. If there are multiple room locations, the results are recalculated based on the relative size of the room areas.

[0069] The lower the system energy consumption, the lower the indoor comfort deviation from the standard, and the greater the reward function value, and the optimization control target is the comprehensive reward after weighting processing.

[0070] S5, based on the environment state, action space and reward function, a model-free reinforcement learning algorithm is used to construct an agent and train it, and finally obtain a control action and output an action value.

[0071] This embodiment is based on Q-learning to construct an agent according to the environment state, action space and reward function; the parameter set estimated by the state estimation is input into the agent to obtain the current control instruction to be issued, and the agent is continuously trained based on the state update of Q-learning.

[0072] The agent is an abstract of the actual problem, and the determination method of the above-mentioned environment state, action space and reward function is referred to, including a discretization processing mode, a training model is built based on a model-free reinforcement learning Q-learning algorithm framework. Q-learning is based on a time difference Markov decision process, in the training process, the Q value of action-state is updated to update the Q table iteratively, so as to obtain the best action strategy in a certain state, the initial value of the Q table is all 0, and the Q table sample is as shown in Table 1.

[0073] Table 1 Q table sample

[0074]

[0075] Wherein, A is the action space, a represents the action, m is the total number of actions contained in the action space, S is the state space, s represents the state, and n is the total number of actions contained in the state space. a can be a discrete action, or can be a combination of multiple control command arrangements, for example, the number of water pumps is 2, and the frequency of the water pump is 45Hz corresponding to the action a m It can be written as (2, 45), or it can be divided into two independent Q tables containing only frequency or quantity.

[0076] Each Q value is represented as Q(s, a), and Q(s, a) represents the action-state Q value obtained by taking action a when the environment state is s, and the standard update method formula is,

[0077] Q(s, a)←Q(s, a)+α[r+γmaxQ(s′,a′)-Q(s,a)]

[0078] Wherein, the arrow indicates that the left variable is updated by the right calculation value, Q(s, a) indicates the action-state Q value of taking action a in environment state s, s' indicates the next time environment state, maxQ(s', a') indicates the known maximum Q value in state s', r is the reward brought by the last round of action-state combination, a is the learning rate, and g indicates the discount factor, that is, the influence degree of future reward on current action decision, which can be set to 0.01.

[0079] The action of updating the Q table is the action decision selected according to the current state, and the commonly used strategy is the standard epsilon-greedy decision strategy, which is represented by pi(a|s) to represent the probability of selecting action a in state s. The strategy balances the balance between experience and exploration of the agent by specifying the exploration rate epsilon of the agent.

[0080] The greater the exploration rate is, the stronger the exploration of the agent is, and the smaller the exploration rate is, the higher the utilization rate of the experience is. The exploration rate determines the probability of directly selecting the action with the highest Q value in the Q table of the current state and the probability of randomly selecting an action when the agent selects an action. Generally, the agent will appropriately increase the exploration rate after each optimization period.

[0081] In the practical application of the agent, the optimized model can be obtained by training the historical data first, and then the current optimized control action can be obtained by inputting the estimated state into the trained model, a certain exploration rate is reserved, and the reward value is recorded to continuously update the Q table.

[0082] The final Q table is used to obtain the control action of the agent, such as the frequency of the water pump and the outlet water temperature of the cooling machine.

[0083] If the above functions are realized in the form of a software function unit and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage medium that can store program codes.

[0084] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, the methods can be tangibly embodied in a machine-readable storage medium having stored thereon instructions that can be used to program a computer to perform any of the methods. The software implementation can be initialized by loading and executing a set of instructions arranged to perform one of the methods into the computer's memory. Alternatively, hard-wired circuitry can be used in place of, or in combination with, software instructions. Thus, the

[0085] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams. Figure 1 means for performing each of the functions specified in the flowchart illustrations and / or block diagrams.

[0086] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams. Figure 1 means for performing each of the functions specified in the flowchart illustrations and / or block diagrams.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams. Figure 1 means for performing each of the functions specified in the flowchart illustrations and / or block diagrams.

[0088] While preferred embodiments of the application have been described, modifications and variations can be apparent to those skilled in the art once aware of the general underlying concepts. Accordingly, the appended claims are intended to embrace all such modifications and variations as fall within the scope of the application.

[0089] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A model-free energy-saving control method for HVAC systems based on maximum likelihood state estimation, characterized in that, Includes the following steps: Obtain the historical state parameter set and the current state parameter set, and predict the state parameter set from the current time to the next time based on the historical state parameter set and the current state parameter set to obtain the predicted state parameter set; Based on the predicted state parameter set, the probability distribution is estimated using the frequency method, and the state corresponding to the maximum likelihood rate is taken as the environmental state. The specific steps to obtain the environmental state include: The predicted state parameter set is preprocessed; Based on the preprocessed predicted state parameter set, the probability distribution is estimated using the frequency method, and the estimated parameter values ​​are calculated. The likelihood function is calculated based on the estimated parameter values ​​to obtain the maximum likelihood rate, and the corresponding state is taken as the environmental state. The expression for estimating the probability distribution using the frequency method is as follows: Among them, variables X , Y This represents different types of state parameters in the predicted state parameter set. x i , y i Indicates the first i The predicted parameter set is rounded down to the nearest neighbor multiple. P ( X = x i , Y = y i , ...) indicates that when predicting state parameters X, Y Values x i 、 y i The probability, f ( x i , y i , ...) indicates that the sample was observed X, Y Values x i ,、y i frequency, q Indicates the total number of samples; The control commands at the current moment are obtained as the action space of the agent. The action space of the agent is the set of all actions that can be selected, including the pump frequency and the number of pumps. Obtain the cumulative HVAC system performance parameters from the current moment to the next moment, and construct a reward function; Based on the aforementioned environmental state, action space, and reward function, a model-free reinforcement learning algorithm is used to construct and train an agent, ultimately obtaining control actions and outputting action values. The model-free reinforcement learning algorithm is a Q-learning algorithm, which iteratively updates the Q-value to enable the agent to perform control actions. The expression for updating the Q-value is: The arrow indicates that the left-hand variable is updated with the calculated value on the right. Q ( s , a ) indicates taking an action under environmental state s. a Action-State Q value, s’ Indicates the state of the environment at the next moment. maxQ ( s’ , a’ ) indicates state s’ The known maximum below Q value, r The reward for the previous action-state combination. α For learning rate, γ This represents the discount factor.

2. The model-free HVAC energy-saving control method based on maximum likelihood state estimation according to claim 1, characterized in that, Long Short-Term Memory (LSTM) networks are used for prediction.

3. The model-free HVAC energy-saving control method based on maximum likelihood state estimation according to claim 1, characterized in that, The preprocessing operation is nearest neighbor multiple rounding, expressed as: in, S This indicates the processed data. This represents a certain type of state parameter in the set of predicted state parameters. round () represents the rounding function. b This indicates the value of the nearest neighbor multiple.

4. The model-free HVAC energy-saving control method based on maximum likelihood state estimation according to claim 1, characterized in that, The expression for the reward function is: in, R Represents the reward function. , This represents the weight of each type of parameter involved in the reward calculation. E This indicates the score of energy consumption parameters used in the reward calculation. C This represents the comfort category parameter score used in the reward calculation.

5. The model-free HVAC energy-saving control method based on maximum likelihood state estimation according to claim 4, characterized in that, The expression for the energy consumption parameter scoring is: in, E This indicates the score for energy consumption parameters. e Parameters representing energy consumption per unit time Indicates the parameter value from the current time t 0 until the next data collection time t n The cumulative time within the period.

6. The model-free HVAC energy-saving control method based on maximum likelihood state estimation according to claim 4, characterized in that, The expression for the comfort category parameter score is: in, C This indicates the rating of comfort-related parameters. c i This represents the value of comfort parameters at a single moment. c i,set This indicates the standard value of the comfort parameters at that moment. i This indicates the number of times data was collected. n Indicates the number of times data was collected. abs () represents the absolute value function.

Images