HVAC temperature control model generation method, HVAC temperature control method and related devices

Abstract

The present disclosure provides a method for generating a HVAC temperature control model, a HVAC temperature control method and related devices, and relates to the fields of artificial intelligence technology such as deep learning, neural networks, HVAC temperature control, and strategy compression. The method includes: based on preset observable parameters in the field of HVAC temperature control, initializing an intelligent agent for outputting corresponding control actions under a reinforcement learning framework to obtain an initialized intelligent agent; according to different control stages in the HVAC temperature control process, training corresponding initialized intelligent agents according to different control stages to obtain trained intelligent agents; using a preset recurrent neural network as a decision network, and generating a target HVAC temperature control model based on the trained intelligent agent and the decision network. The target HVAC temperature control model provided by this embodiment can obtain control actions that are as accurate as possible under incomplete Markov decision-making, thereby improving the accuracy of HVAC temperature control.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, specifically to the fields of artificial intelligence technologies such as deep learning, neural networks, HVAC temperature control, and strategy compression, and in particular to a HVAC temperature control model generation method and a HVAC temperature control method, as well as corresponding devices, electronic devices, computer-readable storage media, and computer program products. Background Art

[0002] HVAC control is a key tool for achieving on-demand HVAC and improving HVAC quality. Machine learning-based HVAC control can help improve the intelligence of the traditional HVAC industry.

[0003] However, in some scenarios, when HVAC control is performed based on machine learning methods, the HVAC control effect is affected and the HVAC control cost is high due to the high cost of HVAC data collection, limited collection time, and lack of data volume. Summary of the Invention

[0004] The embodiments of the present disclosure provide a HVAC temperature control model generation method and a HVAC temperature control method, as well as a matching device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] In the first aspect, an embodiment of the present disclosure proposes a method for generating a HVAC temperature control model, including: initializing an intelligent agent for outputting corresponding control actions under a reinforcement learning framework based on preset observable parameters in the HVAC temperature control field to obtain an initialized intelligent agent; according to different control stages existing in the HVAC temperature control process, training corresponding initialized intelligent agents according to different control stages to obtain trained intelligent agents; using a preset recurrent neural network as a decision network, and generating a target HVAC temperature control model based on the trained intelligent agent and the decision network.

[0006] In the second aspect, an embodiment of the present disclosure proposes a HVAC temperature control model generation device, including: an intelligent agent initialization unit, configured to initialize the intelligent agent for outputting corresponding control actions under the reinforcement learning framework based on preset observable parameters in the HVAC temperature control field, to obtain an initialized intelligent agent; an intelligent agent training unit, configured to train corresponding initialized intelligent agents according to different control stages existing in the HVAC temperature control process, to obtain a trained intelligent agent; a target HVAC temperature control model generation unit, configured to use a preset recurrent neural network as a decision network, and generate a target HVAC temperature control model based on the trained intelligent agent and the decision network.

[0007] In a third aspect, an embodiment of the present disclosure proposes a HVAC temperature control method, including: obtaining preset observable parameters under a target HVAC temperature control scenario; calling a preset target HVAC temperature control model to process the observable parameters and obtain an output control action, and the target HVAC temperature control model is obtained according to the HVAC temperature control model generation method described in the first aspect; controlling the temperature control device to perform the control action.

[0008] In the fourth aspect, an embodiment of the present disclosure proposes a HVAC temperature control device, comprising: an observable parameter acquisition unit, configured to acquire observable parameters preset in a target HVAC temperature control scenario; a model call processing unit, configured to call a preset target HVAC temperature control model to process the observable parameters and obtain an output control action, the target HVAC temperature control model is obtained according to the HVAC temperature control model generation device described in the second aspect; a control action execution unit, configured to control the temperature control device to execute a control action.

[0009] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it can implement the HVAC temperature control model generation method described in the first aspect or the HVAC temperature control method described in the third aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the HVAC temperature control model generation method described in the first aspect or the HVAC temperature control method described in the third aspect when executed.

[0011] In a seventh aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the HVAC temperature control model generation method as described in the first aspect or the HVAC temperature control method as described in the third aspect.

[0012] The HVAC temperature control model generation method and HVAC temperature control method provided by the embodiments of the present disclosure are aimed at the situation where only Markov decision-making with incomplete observation is possible in the field of HVAC temperature control. By using a recurrent neural network as a decision network under a reinforcement learning framework, the sensitivity of the recurrent neural network to features in the time dimension is used to eliminate as much as possible the impact of Markov decision-making with incomplete observation on the accuracy of the prediction results, so that the target HVAC temperature control model generated thereby can achieve better temperature control effects.

[0013] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following drawings:

[0015] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0016] Figure 2 A flow chart of a method for generating a HVAC temperature control model provided in an embodiment of the present disclosure;

[0017] Figure 3 A flowchart of a method for training different agents according to different control stages provided by an embodiment of the present disclosure;

[0018] Figure 4 for Figure 3 A schematic flow chart of the method shown;

[0019] Figure 5 A flow chart of a method for constructing a target HVAC temperature control model provided by an embodiment of the present disclosure;

[0020] Figure 6 A flowchart of a method for guiding a compression decision model based on an optimal state machine according to an embodiment of the present disclosure;

[0021] Figure 7 A flowchart of a decision compression process provided by an embodiment of the present disclosure;

[0022] Figure 8 A flow chart of a method for constructing a target HVAC temperature control model provided by an embodiment of the present disclosure;

[0023] Figure 9 A schematic diagram showing a comparison of the effects of applying a recurrent neural network to solve an incompletely observed Markov decision process according to an embodiment of the present disclosure;

[0024] Figure 10 A flow chart of a HVAC temperature control method provided in an embodiment of the present disclosure;

[0025] Figure 11 A structural block diagram of a HVAC temperature control model generation device provided in an embodiment of the present disclosure;

[0026] Figure 12 A structural block diagram of a HVAC temperature control device provided in an embodiment of the present disclosure;

[0027] Figure 13 A schematic structural diagram of an electronic device suitable for executing a HVAC temperature control model generation method and / or a HVAC temperature control method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other unless there is a conflict.

[0029] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0030] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the HVAC temperature control model generation method, HVAC temperature control method, apparatus, electronic device, and computer-readable storage medium of the present application can be applied.

[0031] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Terminal devices 101, 102, 103 may include sensors installed near various cooling devices in HVAC temperature control scenarios, such as temperature sensors and flow sensors, and may also include analysis equipment for collecting statistics on the load of servers and data centers that are heat generation sources. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0032] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed, such as model generation applications, HVAC temperature control applications, and data transmission applications.

[0033] Terminal devices 101, 102, 103 and server 105 can be either hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here.

[0034] The server 105 can provide various services through various built-in applications. Taking the HVAC temperature control application that can provide HVAC temperature control services for the target HVAC temperature control scenario as an example, the server 105 can achieve the following effects when running the HVAC temperature control application: first, the preset observable parameters under the target HVAC temperature control scenario directly collected or indirectly analyzed by each terminal device are received through the network 104; then, the preset target HVAC temperature control model is called to process the observable parameters and obtain the output control action; finally, the temperature control device is controlled to execute the control action.

[0035] Among them, the target HVAC temperature control model can be trained by the built-in model generation application on the server 105 according to the following steps: first, based on the preset observable parameters in the HVAC temperature control field, the intelligent agent used to output the corresponding control action under the reinforcement learning framework is initialized to obtain the initialized intelligent agent; then, according to the different control stages existing in the HVAC temperature control process, the corresponding initialized intelligent agent is trained according to the different control stages to obtain the trained intelligent agent; finally, the preset recurrent neural network is used as the decision network, and the target HVAC temperature control model is generated based on the trained intelligent agent and the decision network.

[0036] Since training the target HVAC temperature control model requires a large amount of computing resources and strong computing power, the HVAC temperature control model generation method provided in the subsequent embodiments of this application is generally executed by a server 105 with strong computing power and more computing resources. Correspondingly, the HVAC temperature control model generation device is generally also set in the server 105.

[0037] Of course, the server used to train the target HVAC temperature control model can be different from the server used to call the trained target HVAC temperature control model. In particular, the target HVAC temperature control model trained by the server 105 can also be converted into a lightweight target HVAC temperature control model suitable for being embedded in the terminal devices 101, 102, and 103 through model distillation. That is, based on the actual recognition accuracy required, it can be flexibly selected to use the lightweight target HVAC temperature control model in the terminal devices 101, 102, and 103 or to use the more complex target HVAC temperature control model in the server 105.

[0038] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0039] Please refer to Figure 2 , Figure 2 This is a flow chart of a method for generating a HVAC temperature control model provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0040] Step 201: Initialize an agent for outputting corresponding control actions in a reinforcement learning framework based on preset observable parameters in the HVAC temperature control field to obtain an initialized agent;

[0041] This step is intended to be performed by the execution subject of the HVAC temperature control model generation method (e.g. Figure 1 The server 105 shown initializes an agent within a reinforcement learning (RL) framework, which is used to output corresponding control actions, based on pre-defined observable parameters in the HVAC temperature control domain. Specifically, the agent is configured to output corresponding control actions based on the input observable parameters. The observable parameters are derived from the observable operating parameters of the heat generating equipment and the refrigeration equipment in the target HVAC temperature control scenario. The control actions are adjustments to the operating parameters of the refrigeration equipment, so that the adjusted refrigeration equipment can better control the temperature of the target HVAC temperature control scenario.

[0042] The target HVAC temperature control scenario may involve multiple cooling units working together to achieve heat control. Each cooling unit monitors the temperature of the corresponding computer room in real time and adjusts the temperature through precision air conditioning. A single cooling unit may include sub-devices such as primary and secondary chilled water pumps, cold storage tanks, chillers, plate heat exchangers, cooling pumps, and cooling towers. These multiple units operate to enable adaptive switching and operation between automatic cooling, pre-cooling, and cooling modes. Switching between different modes is determined by the activation of different sub-devices and monitoring of ambient temperature thresholds.

[0043] Temperature control strategies in HVAC temperature control scenarios typically require adjusting primary pump frequency, cooling pump frequency, cooling tower power, and chiller chilled water outlet temperature to achieve temperature control transformations. In some embodiments, only the four control variables mentioned above can be used as the actions that the agent can control, and these can be discretized. For example, the agent is only allowed to output the actions of "maintain," "increase," and "decrease." Therefore, under this task definition, the agent's action space is discrete, and the amplitude of "increase" and "decrease" can be adjusted according to actual conditions. For example, [-2, 0, 2] can be specifically selected as the selectable values ​​of the four variables mentioned above.

[0044] For HVAC scenarios, there are many state variables that can be monitored and directly affect control decisions. Taking a single refrigeration unit as an example, there are variables within the system that need to be considered: such as the flow rate of the cold storage tank, which is related to the system status of safe operation, and the aforementioned primary pump frequency, cooling pump frequency, cooling tower power, and chilled water outlet temperature. These state variables are directly related to the actual temperature control strategy. At the same time, there are also some external environmental conditions: such as the current load of the computer room, wet-bulb temperature, computer room temperature, and other factors outside the refrigeration unit that can affect the refrigeration operation.

[0045] In summary, the preset observable parameters may include at least one of the following:

[0046] Computer room load, computer room temperature, cold storage tank flow rate, wet bulb temperature, primary pump frequency, cooling pump frequency, cooling tower power, chiller chilled water outlet temperature.

[0047] Of course, in addition to the items mentioned above, the preset observable parameters can also include other related state quantities. When the above state quantities are not easy to observe or the observation results are inaccurate, other state quantities that can achieve similar effects can be used instead. No specific limitations are made here.

[0048] Furthermore, due to the characteristics of the intelligent agent under the reinforcement learning framework, if high-quality data information and expert operation experience are not available in the initialization stage of this step, the intelligent agent can interact with the environment autonomously and randomly under constraints.

[0049] The initial strategy can be generated based on random noise, for example, Ornstein-Uhlenbeck noise, to ensure that the agent's actions tend to return to the center when they are closer to the edge values. If the noise mean is assumed to be μ, then:

[0050] x t =μ+(x0-μ)e -θt .

[0051] Among them, x0 and x twhere θ represents the state at the initial moment and at time t, respectively; θ indicates the speed of approach to the mean μ; larger values ​​result in exponentially faster approach. Once sufficient interaction experience is accumulated, policy gradients combined with low-probability exploration methods enable the agent to update its policy in a bootstrap-like manner, achieving a cold start.

[0052] If prior expert experience data exists, it is stored in the memory bank. If not, the agent is first randomly controlled under designed constraints. For example, initial interaction experience is generated for the agent based on random noise. This interaction experience refers to the experience of the agent returning a second random noise as output data to the first random noise as input data. When the amount of interaction experience generated meets the preset data volume requirement, the current agent's interaction experience is updated using a combination of policy gradient and small probability exploration methods, allowing the agent to achieve policy updates in a similar way to bootstrapping, achieving a cold start. This ensures that the system obtains interaction data under the premise of stable operation and stores the interaction data in the memory bank.

[0053] Step 202: According to different control stages in the HVAC temperature control process, corresponding initialization agents are trained according to different control stages to obtain trained agents;

[0054] Building on step 201, this step aims to train the corresponding initialized agents according to the different control stages present in the HVAC temperature control process, generating trained agents. Different from operating modes such as automatic cooling, pre-cooling, and refrigeration, the different control stages mentioned in this step refer more to the different levels of screening and selection of output control actions based on actual needs. For example, a safety control stage involves evaluating and screening a large number of output alternative control actions for safety scores, an energy reduction stage involves evaluating and screening a large number of output alternative control actions for energy consumption, and so on.

[0055] Step 203: Use the preset recurrent neural network as the decision network, and generate a target HVAC temperature control model based on the trained agent and the decision network.

[0056] Based on step 202, this step aims to have the above-mentioned execution subject use the preset recurrent neural network determined based on the recurrent neural network architecture as the decision network, and generate a target HVAC temperature control model for temperature control of the HVAC scene based on the trained intelligent agent and the decision network.

[0057] Among them, recurrent neural networks are a type of neural network specially designed to process sequence data of indefinite length. Therefore, they are suitable for analyzing and making decisions on the control of temperature control equipment over long periods in HVAC temperature control scenarios.

[0058] The analysis of state variables in the HVAC temperature control scenario in step 201 shows that temperature control decisions are influenced by system states from multiple different sources. Obtaining complete environmental information is a crucial prerequisite for the MDP (Markov Decision Process). However, in the actual control of HVAC temperature control scenarios, the information available to the agent is not comprehensive. Typically, only critical locations are equipped with sensors, and only rough, local information is collected. Therefore, the POMDP (Partially Observable Markov Decision Process) is the most common scenario. In building HVAC scenarios, two major sources of POMDP are: a large amount of uncollected location information and the lag of location information. Consequently, only partially accurate information from sensors can be provided to the agent. Furthermore, the continuity of the data is largely dependent on the reliability of the equipment, and in reality, equipment does not always operate stably. From this point of view, the temperature control environment does not meet the standard Markov process (that is, Markov decision under complete observation, that is, the reinforcement learning process is essentially a Markov decision process under complete observation). Therefore, in this embodiment, when only part of the state quantity (observable parameters) can be transmitted to the intelligent agent, a HVAC temperature control decision-making intelligent agent framework based on a preset recurrent neural network is proposed to use the characteristics of the recurrent neural network to eliminate the impact of incomplete observation on the accuracy of the prediction results as much as possible.

[0059] The HVAC temperature control model generation method provided by the embodiment of the present disclosure is aimed at the situation where only incomplete observation of Markov decision can be performed in the field of HVAC temperature control. By using a recurrent neural network as the decision network under the reinforcement learning framework, the sensitivity of the recurrent neural network to features in the time dimension is used to eliminate as much as possible the impact of Markov decision-making that can only be incompletely observed on the accuracy of the prediction results, so that the target HVAC temperature control model generated thereby can achieve better temperature control effect.

[0060] Please refer to Figure 3 , Figure 3 A method of training different agents according to different control stages is shown. Figure 2 Step 202 in the process 200 shown provides a specific implementation method. The other steps in the process 200 are not adjusted. The specific implementation method provided in this embodiment is used to replace step 202 to obtain a new complete embodiment.

[0061] The process 300 includes the following steps:

[0062] Step 301: construct a first training set with observable parameters as input and corresponding control actions as output;

[0063] Step 302: Using the first training set to train the agent corresponding to the learning stage, to obtain a first agent;

[0064] The aforementioned steps aim to construct a first training set of sample pairs of "observable parameter-control action" by the aforementioned execution agent, and use this first training set to train the agent corresponding to the learning phase, thereby obtaining a trained first agent capable of outputting candidate control actions corresponding to the input observable parameters. Typically, for a single input observable parameter, the number of output candidate control actions can be numerous due to the complexity of the actual situation, and this set can also be referred to as a candidate control action set.

[0065] A specific training program for this stage could be:

[0066] After accumulating enough experience, randomly sample from the memory bank and use the Bellman formula to calculate the update error of the network:

[0067]

[0068] Among them, s j is the state value observed by the agent, a j is the control action output by the agent, Q(s j ,a j ) represents the state-action value function, r j represents the reward and penalty value. γ is a hyperparameter that balances the reward and penalty value with the state-action value function. The subscript j represents the jth trajectory.

[0069] The agent updates its estimate of the observed value by using gradient descent and propagating back along the network. The strategy adopted by the agent during the training phase is a mapping network from observations to action values. The decision network μ(s) calculates the objective function gradient using the following empirical form and updates along the gradient direction:

[0070]

[0071] in, is a differential operator, Q(s,a) represents the value function of the state-action (s,a), and μ(s|a) is the distribution of state s given by the policy network μ(·) under a given action a.

[0072] Step 303: construct a second training set with control actions as input and corresponding safety assessment scores as output;

[0073] Step 304: Using the second training set to train the agent corresponding to the security control stage, to obtain a second agent;

[0074] The above steps are intended to construct a second training set formed by the sample pairs constructed by the "control action-safety assessment score" by the above-mentioned execution subject, and use the second training set to train the intelligent agent corresponding to the learning stage, thereby obtaining the trained safety assessment score that can be used to determine the corresponding alternative control action, and screen out the second intelligent agent of the screened control action that meets the preset safety requirements based on the safety assessment score.

[0075] The safety assessment score refers to the safety of the corresponding control action, including whether the adjustment parameter items and adjustment range corresponding to the control action are abnormal, and whether the adjustment according to the control action will have a relatively large impact on the current temperature control system, such as a large number of adjustment items, a long adjustment time, excessive system volatility, etc.; the preset safety requirements are used to confirm whether the current safety requirements are met based on the actual safety assessment score of the control action, and only control actions that meet the safety requirements are screened out for future use. Specifically, when the preset safety requirements only require the screening of the safest control actions, it can be specifically manifested as: screening out the control actions with the highest safety assessment scores. Of course, in some other scenarios, the control actions with the top K safety ratings (where K can be: 3, 5, 10, etc.) can also be output.

[0076] Step 305: constructing a third training set with the control action as input and the corresponding power consumption as output;

[0077] Step 306: Use the third training set to train the agent corresponding to the function reduction stage to obtain a third agent.

[0078] The above steps are intended to construct a third training set formed by the sample pairs constructed by "control action-power consumption" by the above-mentioned execution subject, and use the third training set to train the intelligent agent corresponding to the learning stage, and then obtain a trained third intelligent agent that can be used to determine the power consumption corresponding to the screened control action, and screen out the target control action that meets the preset power consumption requirements based on the power consumption.

[0079] The power consumption refers to the energy consumption of the temperature control device after executing the corresponding control action, which is used to evaluate the energy consumption. The preset power consumption requirement is used to confirm whether the current power consumption requirement is met based on the actual power consumption of the control action. Under normal circumstances, the preset power consumption requirement can be specifically expressed as requiring the actual power consumption to be lower than the preset power consumption value, or requiring the lowest power consumption, etc.

[0080] To better understand the solution provided by this embodiment, please refer to Figure 4The flow chart shown, from left to right, respectively: the learning stage of the intelligent agent, the safety control stage, and the energy consumption reduction stage. That is, the value estimation network operating in the safety control stage (that is, the second intelligent agent) learns values ​​guided by the reward signal that focuses on the system safety score, that is, before each decision is made, a working condition analysis is required to determine whether the system is operating within a safe range. The value estimation network operating in the energy consumption reduction stage (that is, the third intelligent agent) learns values ​​that focus on reducing the overall energy consumption of the system to improve efficiency.

[0081] The reward signal consists of two major components: one considers system safety, using conditions such as the flow rate of the cold storage tank as reward indicators, and the other considers system efficiency and energy consumption, aiming to improve the efficiency of the temperature control strategy while ensuring normal system operation. Furthermore, given the multiple modes of temperature control, inconsistent agents are used to learn temperature control strategies guided by different rewards, switching between these modes by determining thresholds.

[0082] This embodiment, through steps 301 to 306, specifically provides a solution for training the intelligent agent in the learning stage, safety control stage, and energy consumption reduction stage respectively, so that in actual application, the control actions finally output are all preferred control actions with sufficient safety and low energy consumption.

[0083] Considering that step 203 may include multiple specific implementations with different focuses, for example, the following three:

[0084] Implementation method 1: A recurrent neural network embedded in an autoencoder network is used as the first decision network, and a target HVAC temperature control model is generated based on the trained intelligent agent and the first decision network.

[0085] That is, the autoencoder network is further embedded in the standard recurrent neural network framework to use the encoder and decoder characteristics of the autoencoder network to compress the decision expression of the decision network in the action / state space, so as to reduce the scale of the decision network and improve decision efficiency.

[0086] Autoencoding is an unsupervised learning algorithm, distinguished from manual encoding, which relies on human experience to encode data. Autoencoding, on the other hand, does not require human intervention. To determine whether the encoder's encoding method is effective, the decoder analyzes the recovered data after decoding. If the decoder can recover data similar to the data compressed by the encoder, then the current encoding is likely effective. Simply put, an autoencoder network is an artificial neural network trained in an unsupervised manner and consists of two components: an encoder and a decoder. Its goal is to learn a data encoding method that compresses input data X into H and can be decoded back into X' (which is as close to the input data X as possible). A key application of autoencoders is in compressed data representation. If the dimensionality of the hidden features is lower than that of the input data X, the encoder can reduce the dimensionality of X to a lower-dimensional feature H. The decoder can then decompress this lower-dimensional feature into X'. X' retains the important information in the input data X but has a lower dimensionality, thus achieving compressed data representation. This embodiment embeds the autoencoder network into a recurrent neural network, which not only compresses the decision expression of the decision network, but also uses the recurrent neural network to minimize the impact of the Markov decision made with partial observations on the accuracy of the output control action.

[0087] Implementation method 2: Based on the trained intelligent agent and the second decision network composed of a recurrent neural network, the target HVAC temperature control model is generated in combination with prior expert experience data under the imitation learning framework.

[0088] This approach uses a standard recurrent neural network as the decision-making network. Furthermore, since HVAC temperature control systems are typically regulated by HVAC experts, the accumulated expert operating trajectories serve as empirical data for the model's Imitation Learning (IL) process. This is achieved by integrating the IL framework to automatically generate appropriate reward functions and hyperparameters, making the model more likely to converge.

[0089] Imitation learning is to train machines to copy continuous human actions, thereby achieving the purpose of imitation. IL is very practical. Suppose there is a training scenario today, but you don’t know how to set the reward value. However, you can collect expert demonstration data, then you can consider using the IL method. This embodiment takes into account the characteristics of IL, and thanks to the fact that HVAC temperature control systems are usually regulated by HVAC experts, the expert working condition trajectories accumulated over a long period of time can be used as experience data for model imitation learning. That is, IL essentially guides the model to continuously imitate the expert trajectory to obtain a higher reward value, that is, a higher discriminator score, thereby accumulating expert decision-making patterns. This embodiment mainly uses imitation learning to automatically generate more appropriate reward functions and hyperparameters to make the model easier to converge.

[0090] Implementation method three: Use the recurrent neural network embedded in the autoencoder network as the third decision network; based on the trained intelligent agent and the third decision network, combine the prior expert experience data under the imitation learning framework to generate the target HVAC temperature control model.

[0091] That is, by additionally introducing an autoencoder network to compress the decision expression and network scale of the decision network and improve decision efficiency, and by combining it with the imitation learning framework to fully learn from prior expert experience how to automatically generate more appropriate reward functions and hyperparameters, so that the model is easier to converge.

[0092] The following will provide specific implementation steps for the different implementation solutions mentioned above in conjunction with the corresponding figures:

[0093] Figure 5 A specific implementation method for generating a target HVAC temperature control model based on the concept of the first embodiment is shown. The process 500 includes the following steps:

[0094] Step 501: Generate an operational experience trajectory of the temperature control device based on the input and output of the trained agent;

[0095] This step is based on the trained intelligent agent (i.e., the intelligent agent capable of realizing temperature control) obtained by the intelligent agent training method provided in the above embodiment, and is intended to generate an operational experience trajectory of the temperature control device according to the input and output of the trained intelligent agent by the above-mentioned execution subject. The operational experience trajectory is obtained by staggering the input state (i.e., the observable parameters at the corresponding moment) and the output (control) action in a time sequence. The operational experience trajectory can be simply expressed as: "state → action → next state → next action → ... final state → final action".

[0096] Step 502: inserting an autoencoder network into the first layer and hidden state layer of the standard recurrent neural network to obtain a first decision network;

[0097] In the first layer and hidden state layer of RNN, insert the autoencoder network, use tanh (hyperbolic tangent function) as the activation function, and discretize the encoding in the autoencoder network so that each encoding value of the autoencoder network is only -1, 0 or 1:

[0098]

[0099] Step 503: Input the state in the operation experience trajectory into the first decision network to obtain the predicted action output by the first decision network;

[0100] Step 504: Using supervised learning to control the predicted action to be closer to the actual action, and determining the first decision network whose degree of closeness is less than a preset degree as the target first decision network;

[0101] The actual action is the action in the operation experience trajectory that matches the state currently input into the first decision network.

[0102] The above steps are intended to enable the execution subject to use the operation experience trajectory obtained in step 501, input the state therein into the first decision network, and obtain the specific action value of the output, and use supervised learning to make the output action close to the action in the historical operation trajectory; if the error cannot converge, the encoding dimension in the autoencoder is expanded, and the process is repeated until the difference between the action output by the inserted network and the action output by the original first decision network is less than a certain threshold, that is, in response to the actual degree of closeness cannot be less than the preset degree, the encoding dimension of the autoencoder network is expanded until the degree of closeness after the expansion of the encoding dimension is less than the preset degree.

[0103] Step 505: Generate a target HVAC temperature control model based on the trained intelligent agent and the target first decision network.

[0104] Based on step 504, this step aims to generate a target HVAC temperature control model based on the trained intelligent agent and the target first decision network by the above-mentioned execution subject.

[0105] In steps 501 to 505 provided in this embodiment, in response to the situation where Markov decision-making cannot be fully observed in the field of HVAC and temperature control, a recurrent neural network is first used as the first decision network under the reinforcement learning framework, so as to minimize the prediction accuracy caused by the Markov decision-making that can only be partially observed by leveraging the sensitivity of the recurrent neural network to features in the time dimension. At the same time, by embedding the autoencoder network into the recurrent neural network, the first decision network is further compressed in the action / state space, thereby increasing the model interpretability and improving the decision efficiency while ensuring the decision accuracy. In addition, by training different intelligent agents for each control stage of HVAC and temperature control, it is possible to better realize the switching of different modes in the HVAC and temperature control process through multiple intelligent agents, thereby improving the temperature control effect.

[0106] Further, in Figure 5 Based on the embodiment shown, Figure 6 A method for guiding a compression decision model based on an optimal state machine is also shown, wherein the process 600 includes the following steps:

[0107] Step 601: extracting a preset number of discrete autoencoders from a target first decision network;

[0108] Step 602: Form a finite state machine based on the encoding status of the discrete autoencoder during the entire temperature control process, the total number of observation parameters and the total number of hidden states obtained by the first decision network during the decision-making process, and the transition process between different hidden states;

[0109] Step 603: compressing the finite state machine using finite state machine compression technology to obtain an optimal state machine;

[0110] Step 604: Using the number of hidden states and the number of observation parameters of the optimal state machine, guide the compression target first decision network.

[0111] like Figure 7 As shown in the flowchart of the decision compression process, in this embodiment, after the training of the inserted discrete autoencoder network is completed, the new first decision network can be used to control in the HVAC scenario. After fully experiencing most possible situations in the HVAC scenario, a preset number of discrete autoencoders (in Figure 7 2 in the figure), and observe the possible encoding situations of the autoencoder in the whole process, analyze the total number of observations and the total number of hidden states obtained by the first decision network in the process of executing the strategy, and record the process of transition between different hidden states to form a finite state machine. Then, the compression technology of the finite state machine is used to obtain the optimal state machine. Finally, with the help of the number of hidden states and the number of observations under the optimal state machine, the compression of the original first decision network is guided, so that the first decision network can be better compressed.

[0112] Specifically, Figure 7 It contains three autoencoders and three discrete encoders (i.e. decoders). Autoencoder I is used to reduce the dimensionality of the features obtained by the feature according to the custom encoding method, autoencoder II is used to insert the hidden state, and autoencoder III is used to reduce the dimensionality of the hidden state. The three decoders correspond to the three autoencoders respectively.

[0113] Specifically, the following process can be used to extract the distribution of decision actions:

[0114] 1) Without inserting the two autoencoders mentioned in step 502, insert a discrete encoder in the part of the model where decisions are made from the hidden state. The encoder dimension is set to the dimension of the action, and the output amplitude value is the value in the discrete action space. Use the control trajectory obtained in step 502 to train so that the output of the policy network is close to the performance of the policy network before the insertion;

[0115] 2) In actual HVAC scenarios, the actions output by the policy network after inserting the discrete autoencoder are counted, and the actions taken under all corresponding states are analyzed. Based on the statistics, the action distribution of the policy network output is estimated, and the specific meaning of the hidden state is explained using the action distribution.

[0116] Figure 8 A specific implementation method for generating a target HVAC temperature control model based on the concept of the first embodiment is shown. The process 800 includes the following steps:

[0117] Step 801: extracting the operating experience trajectory of the temperature control device from the prior expert experience data;

[0118] This step is based on the trained intelligent agent (i.e., the intelligent agent capable of realizing temperature control) obtained by the intelligent agent training method provided in the above embodiment, and aims to extract the operational experience trajectory of the temperature control device from the prior expert experience data by the above-mentioned execution subject. The operational experience trajectory is obtained by staggering the input state (i.e., the observable parameters at the corresponding moment) and the output (control) action in a time sequence. The operational experience trajectory can be simply expressed as: "state → action → next state → next action → ... final state → final action".

[0119] Step 802: Input the state in the operation experience trajectory into the second decision network which is a recurrent neural network, and obtain the predictive control action output by the second decision network;

[0120] Based on 801, this step aims to have the above-mentioned execution subject input the state in the operation experience trajectory into the second decision network played by the recurrent neural network to obtain the predictive control action output by the second decision network.

[0121] Step 803: Under the framework of generative adversarial imitation learning, the generator and discriminator are trained using prior expert experience to obtain hyperparameters and reward functions generated by the target generator that can be discriminated by the target discriminator;

[0122] On the basis of step 802, this step is intended to be performed by the above-mentioned execution subject under the imitation learning framework of the generative adversarial network, using the prior expert experience to train the generator (also called the generative network) and the discriminator (also called the discriminative network) constituting the generative adversarial network, and obtain the target generator and target discriminator after iterative training. The iterative training direction is controlled as follows: the hyperparameters and reward function generated by the control generator are used to make the predicted control action output by the second decision network close to the actual control action, and the hyperparameters and reward function generated by the target generator will be able to pass the discrimination of the target discriminator. The actual control action is the control action in the operating experience trajectory that matches the state currently input to the second decision network.

[0123] Step 804: Generate a target HVAC temperature control model using the trained agent and the second decision network under the settings of hyperparameters and reward functions.

[0124] Based on step 803, this step aims to generate a target HVAC temperature control model based on the trained intelligent agent and the target second decision network by the above-mentioned execution subject.

[0125] Steps 501 to 504 provided in this embodiment are aimed at the case where only incomplete observation of Markov decision-making is possible in the field of HVAC temperature control. First, a recurrent neural network is used as the second decision network under the reinforcement learning framework. The sensitivity of the recurrent neural network to features in the time dimension is used to minimize the impact of the Markov decision-making that can only be incompletely observed on the accuracy of the prediction results. At the same time, by combining imitation learning technology, the problem of difficulty in convergence of reinforcement learning is solved by fully imitating the expert working condition trajectory, and the problem of difficulty in convergence caused by unreasonable reward function and hyperparameter settings is solved, so that the target HVAC temperature control model generated by the application can achieve better temperature control effect.

[0126] like Figure 9 As shown, in an ideal situation, an MDP observes the complete environment state, and the agent's action is based on this observation. However, in real-world situations, like a POMDP, the actor obtains partial observations rather than the state. While the action can still be made, the decision is affected by information asymmetry, and reliability is reduced. To address the POMDP problem, this paper introduces an RNN to provide memory, using hidden states to help the agent obtain more complete environmental information.

[0127] It should be noted that variant models of RNN are also applicable to the solutions provided in the present disclosure. When introducing RNN into the reinforcement learning algorithm, the biggest change is to maintain the hidden state of the RNN. In terms of the choice of reinforcement learning algorithm, in addition to the On-Policy (online policy) model, other types of models including but not limited to AC (Actor-Critic), A2C, A3C, TRPO (Trust Region Policy Optimization, trust region policy optimization algorithm), PPO (Proximal Policy Optimization, deep reinforcement learning) can also be used. The behavior strategy and target strategy of this type of model are the same, so the hidden state under the same experience is also the same, so the hidden state at the time of decision-making can be saved for training. However, for the Off-Policy (offline policy) model, the training and decision-making models are different, and the collected hidden states are not universal.

[0128] See Figure 9 As can be seen in the rightmost sub-figure, the improved algorithm presented in this disclosure integrates an RNN into the Actor. In addition to outputting the policy action, the model also outputs a hidden state, Hidden. The Hidden state from the previous step is used to compensate for the partial observation. Both are fed into the model to make the next decision inference. It should be noted that when the Critic network is fed into Q-value estimation, Hidden is also one of the inputs. This ensures that the states "observed" by the Actor-Critic network are consistent, namely, the partially observed and hidden states.

[0129] To deepen the understanding of the solution provided by the above second embodiment, the following will also provide an overall algorithm flow:

[0130] First, a reinforcement learning model that integrates RNN is given. RL uses PPO as an example. The following is the algorithm flow:

[0131] Step 1: If there is a trained model, load the model structure and parameters; if not, initialize the parameters (strategy function parameter θ, value function parameter φ) and hyperparameters (reward and punishment factor γ, KL divergence coefficient λ, adaptability parameters a, b);

[0132] Step 2: Get the current environment status s t , by the policy function π θ Give the best action a t , the system issues action strategies and executes them;

[0133] Step 3: Get the new environment status s t+1 , according to the reward function definition, calculate the current reward value rt , judge whether the current working condition is the expert trajectory F, and (s t , a t , r t , s t+1 , h t , h out , F) put into the experience cache;

[0134] Step 4: Calculate the advantage function value (when taking a single step, t' refers to t+1), where V(·) is the state value function:

[0135]

[0136] Step 5: Get the historical trajectory of t = {1, 2, ..., T} from the experience cache and update the policy function parameter θ using the following gradient formula i Update to θ i+1 :

[0137]

[0138] Step 6: Update the value function parameters using the following gradient formula Updated to

[0139]

[0140] Step 7: Calculate the KL divergence between the new and old policy functions, and convert the truncation divergence coefficient λ through the adaptability parameter:

[0141]

[0142] When d<d target / a,λ←λ / b; when d>d target ×a, λ←λ×b.

[0143] Step 8: Transform the new policy function π θ Assign to π old , and also save the RNN hidden state Hidden. Repeat steps 2 to 7, continuously iteratively updating the policy function and value function. When a certain number of iterations is met or the specified convergence condition is met, stop the iteration and export the model parameters and hyperparameters for future use.

[0144] An example of defining a policy function and value function: Input fully connected layer - ReLU activation layer - RNN layer - activation layer - output layer. The activation layer before the output layer is selected based on whether the action space is continuous or discrete. The output layer of the policy function is the size of the action space, while the output dimension of the value function is typically 1.

[0145] The above demonstrates that the convergence of reinforcement learning methods depends on the definition of the reward function and the tuning of hyperparameters. This example further integrates the RL+RNN algorithm into imitation learning, proposing a recurrent imitation learning algorithm. This is also due to the fact that HVAC systems are typically regulated by HVAC experts, and the accumulated expert operating trajectories serve as empirical data for the model's imitation learning. IL essentially guides the agent to continuously imitate the expert's trajectories to obtain higher rewards, that is, higher discriminator scores, thereby accumulating expert decision-making patterns and policies.

[0146] Take Generative adversarial imitation learning (GAIL) as an example:

[0147] 1) First, define GAIL’s generator network G(·) and discriminator network D(·), as well as the agent’s policy network Policyπ(·), where θ and w are hyperparameters of π(·) and D(·);

[0148] 2) The expert trajectory τ E and G(·) to construct a batch of random trajectories τ R are marked as 1 and 0 respectively for the training and iteration of D(·);

[0149] 3) Based on the current state State, the policy network Policyπ(·) gives the next action Action, and the discriminator D(·) gives a score based on the state-action pair (State, Action);

[0150] 4) and converted to DRL reward function Put it into step 3 of the above algorithm and iterate the agent network.

[0151] Repeat the above steps and ensure that the fictitious strategy does not deviate from the expert strategy during training. Then, after reaching the convergence condition, the model will output an expert-like decision-making agent Agentπ(·) and a working condition scorer D(·).

[0152] Based on the description of how to implement implementation mode one and implementation mode two respectively in the above embodiments, implementation mode three can be obtained by fully combining implementation mode one and implementation mode two. When the execution steps do not conflict, it can include multiple combinations. For example, it is still based on the generative adversarial imitation learning framework, and it is only necessary to embed the autoencoder network into the recurrent neural network. The details will not be elaborated one by one here.

[0153] The above embodiments illustrate how to train the target HVAC temperature control model from various aspects. In order to highlight the effect of the trained target HVAC temperature control model from the actual usage scenario as much as possible, the present disclosure also provides a method for Figure 10 A solution for using a trained target HVAC temperature control model to solve practical problems is shown. A HVAC temperature control method includes the following steps:

[0154] Step 1001: Obtaining preset observable parameters in a target HVAC temperature control scenario;

[0155] Step 1002: calling a preset target HVAC temperature control model to process observable parameters and obtain output control actions;

[0156] Step 1003: Control the temperature control device to perform a control action.

[0157] That is, in actual application scenarios, the execution subject first obtains the preset observable parameters under the target HVAC temperature control scenario, then inputs them as input data into the preset target HVAC temperature control model, and receives the control action output by the model, and finally controls the corresponding temperature control equipment to execute the control action.

[0158] Further references Figure 11 and Figure 12 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a HVAC temperature control model generation device and an embodiment of a HVAC temperature control device. Figure 2 The HVAC temperature control model generation method embodiment shown corresponds to the HVAC temperature control device embodiment. Figure 10 The above device can be applied to various electronic devices.

[0159] like Figure 11 As shown, the HVAC temperature control model generation device 1100 of this embodiment may include: an agent initialization unit 1101, an agent training unit 1102, and a target HVAC temperature control model generation unit 1103. The agent initialization unit 1101 is configured to initialize an agent for outputting corresponding control actions under a reinforcement learning framework based on preset observable parameters in the HVAC temperature control field, thereby obtaining an initialized agent; the agent training unit 1102 is configured to train corresponding initialized agents according to different control stages in the HVAC temperature control process, thereby obtaining trained agents; and the target HVAC temperature control model generation unit 1103 is configured to use a preset recurrent neural network as a decision network and generate a target HVAC temperature control model based on the trained agent and the decision network.

[0160] In this embodiment, the HVAC temperature control model generation device 1100 includes the agent initialization unit 1101, the agent training unit 1102, and the target HVAC temperature control model generation unit 1103. The specific processing and technical effects thereof can be referred to in the respective Figure 2 The relevant descriptions of steps 201-203 in the corresponding embodiment are not repeated here.

[0161] In some optional implementations of this embodiment, the agent training unit 1102 may be further configured to:

[0162] Constructing a first training set with observable parameters as input and corresponding control actions as output;

[0163] Using the first training set to train an agent corresponding to the learning phase, to obtain a trained first agent, the first agent being configured to output an alternative control action corresponding to the input observable parameter;

[0164] Construct a second training set with control actions as input and corresponding safety assessment scores as output;

[0165] Using the second training set to train an agent corresponding to the safety control stage, a trained second agent is obtained. The second agent is used to determine the safety assessment scores corresponding to the alternative control actions, and to select the screened control actions that meet the preset safety requirements based on the safety assessment scores;

[0166] Constructing a third training set with control actions as input and corresponding power consumption as output;

[0167] The third training set is used to train the intelligent agent corresponding to the function reduction stage to obtain a trained third intelligent agent. The third intelligent agent is used to determine the power consumption corresponding to the control action after screening, and to screen out the target control action that meets the preset power consumption requirements based on the power consumption.

[0168] In some optional implementations of this embodiment, the preset security requirement includes: the highest security assessment score; and the preset power consumption requirement includes: the lowest power consumption.

[0169] In some optional implementations of this embodiment, the observable parameter includes at least one of the following:

[0170] Computer room load, computer room temperature, cold storage tank flow rate, wet bulb temperature, primary pump frequency, cooling pump frequency, cooling tower power, chiller chilled water outlet temperature.

[0171] In some optional implementations of this embodiment, the agent initialization unit 1101 may be further configured to:

[0172] In response to the absence of prior expert experience data stored in the memory bank, generating initial interaction experience for the intelligent agent based on random noise, wherein the interaction experience refers to experience of the intelligent agent returning second random noise serving as output data to the first random noise serving as input data;

[0173] In response to the fact that the amount of interaction experience data meets the preset data volume requirement, the current agent's interaction experience is updated by combining policy gradient with small probability exploration method.

[0174] In some optional implementations of this embodiment, the target HVAC temperature control model generation unit 1103 may include:

[0175] The first model generation subunit is configured to use the recurrent neural network embedded in the autoencoder network as the first decision network, and generate a target HVAC temperature control model based on the trained intelligent agent and the first decision network.

[0176] In some optional implementations of this embodiment, the first model generation subunit may be further configured to:

[0177] Based on the input and output of the trained agent, the operating experience trajectory of the temperature control device is generated. The operating experience trajectory is obtained by interleaving the input state and output action in time sequence.

[0178] Insert the autoencoder network into the first layer and hidden state layer of the standard recurrent neural network to obtain the first decision network;

[0179] Inputting the state in the operation experience trajectory into the first decision network to obtain the predicted action output by the first decision network;

[0180] Using supervised learning, the predicted action is controlled to be closer to the actual action, and the first decision network whose degree of closeness is less than a preset degree is determined as the target first decision network. The actual action is the action in the operation experience trajectory that matches the state currently input to the first decision network.

[0181] The target HVAC temperature control model is generated based on the trained intelligent agent and the target first decision network.

[0182] In some optional implementations of this embodiment, the HVAC temperature control model generating device 1100 may further include:

[0183] The coding dimension expansion unit is configured to expand the coding dimension of the self-encoding network in response to the actual proximity degree being unable to be less than the preset degree until the proximity degree after the coding dimension is expanded is less than the preset degree.

[0184] In some optional implementations of this embodiment, the HVAC temperature control model generating device 1100 may further include:

[0185] a discrete encoder extraction unit configured to extract a preset number of discrete autoencoders from the target first decision network;

[0186] a finite state machine forming unit configured to form a finite state machine according to an encoding condition of the discrete autoencoder during the entire temperature control process, a total number of observation parameters and a total number of hidden states obtained by the first decision network during the decision-making process, and a transition process between different hidden states;

[0187] an optimal state machine acquisition unit, configured to compress the finite state machine using a finite state machine compression technique to obtain an optimal state machine;

[0188] The compression guidance unit is configured to guide the compression target first decision network using the number of hidden states and the number of observation parameters of the optimal state machine.

[0189] In some optional implementations of this embodiment, the target HVAC temperature control model generation unit 1103 includes:

[0190] The second model generation subunit is configured to generate a target HVAC temperature control model based on the trained intelligent agent and the second decision network composed of a recurrent neural network and combined with prior expert experience data under the imitation learning framework.

[0191] In some optional implementations of this embodiment, the second model generation subunit may be further configured to:

[0192] The operating experience trajectory of the temperature control equipment is extracted from the prior expert experience data. The operating experience trajectory is obtained by interleaving the input state and the output control action in a time sequence. The state is determined by the observable parameters.

[0193] The state in the operation experience trajectory is input into the second decision network which is served by the recurrent neural network, and the predictive control action output by the second decision network is obtained;

[0194] Within the framework of generative adversarial imitation learning, the generator and discriminator are trained using prior expert experience to obtain the hyperparameters and reward function generated by the target generator that can be discriminated by the target discriminator. The target generator and target discriminator are obtained after iterative training. The iterative training direction is controlled to control the hyperparameters and reward function generated by the generator to make the predicted control action output by the second decision network close to the actual control action. The actual control action is the control action in the operating experience trajectory that matches the state currently input to the second decision network.

[0195] The trained agent and the second decision network are used to generate the target HVAC temperature control model under the settings of hyperparameters and reward functions.

[0196] In some optional implementations of this embodiment, the hidden state of the previous step output by the second decision network will also be used as additional observation information for the next step to participate in the subsequent control action prediction.

[0197] In some optional implementations of this embodiment, the target HVAC temperature control model generation unit 1103 may include:

[0198] A third decision network determination subunit is configured to use a recurrent neural network embedded in an autoencoder network as a third decision network;

[0199] The third model generation subunit is configured to generate a target HVAC temperature control model based on the trained intelligent agent and the third decision network in combination with prior expert experience data under the imitation learning framework.

[0200] like Figure 12 As shown, the HVAC temperature control device 1200 of this embodiment may include: an observable parameter acquisition unit 1201, a model call processing unit 1202, and a control action execution unit 1203. The observable parameter acquisition unit 1201 is configured to acquire preset observable parameters for a target HVAC temperature control scenario; the model call processing unit 1202 is configured to call a preset target HVAC temperature control model to process the observable parameters and obtain output control actions; and the control action execution unit 1203 is configured to control the temperature control device to execute control actions.

[0201] In this embodiment, in the HVAC temperature control device 1200, the specific processing of the observable parameter acquisition unit 1201, the model call processing unit 1202, and the control action execution unit 1203 and the technical effects thereof may correspond to Figure 9 The relevant descriptions in the method embodiment shown are not repeated here.

[0202] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The HVAC temperature control model generation device and HVAC temperature control device provided by this embodiment are aimed at the situation where only Markov decision-making with incomplete observation is possible in the field of HVAC temperature control. By using a recurrent neural network as a decision network under a reinforcement learning framework, the sensitivity of the recurrent neural network to features in the time dimension is used to eliminate as much as possible the impact of Markov decision-making with incomplete observation on the accuracy of the prediction results, so that the target HVAC temperature control model generated thereby can achieve better temperature control effects.

[0203] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the HVAC temperature control model generation method and / or HVAC temperature control method described in any of the above embodiments when executing.

[0204] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the HVAC temperature control model generation method and / or HVAC temperature control method described in any of the above embodiments when executed.

[0205] An embodiment of the present disclosure provides a computer program product, which, when executed by a processor, can implement the HVAC temperature control model generation method and / or HVAC temperature control method described in any of the above embodiments.

[0206] Figure 13 A schematic block diagram of an example electronic device 13000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0207] like Figure 13 As shown, the device 13000 includes a computing unit 13001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 13002 or a computer program loaded from a storage unit 13008 into a random access memory (RAM) 13003. Various programs and data required for the operation of the device 13000 can also be stored in the RAM 13003. The computing unit 13001, the ROM 13002, and the RAM 13003 are connected to each other via a bus 13004. An input / output (I / O) interface 13005 is also connected to the bus 13004.

[0208] Various components in device 13000 are connected to I / O interface 13005, including: input unit 13006, such as a keyboard, mouse, etc.; output unit 13007, such as various types of displays, speakers, etc.; storage unit 13008, such as a magnetic disk, optical disk, etc.; and communication unit 13009, such as a network card, modem, wireless communication transceiver, etc. Communication unit 13009 allows device 13000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0209] The computing unit 13001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 13001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 13001 performs the various methods and processes described above, such as the HVAC temperature control model generation method and / or the HVAC temperature control method. For example, in some embodiments, the HVAC temperature control model generation method and / or the HVAC temperature control method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 13008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 13000 via the ROM 13002 and / or the communication unit 13009. When the computer program is loaded into RAM 13003 and executed by the computing unit 13001, one or more steps of the HVAC temperature control model generation method and / or HVAC temperature control method described above may be performed. Alternatively, in other embodiments, the computing unit 13001 may be configured to execute the HVAC temperature control model generation method and / or HVAC temperature control method in any other appropriate manner (e.g., via firmware).

[0210] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0211] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0212] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0213] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0214] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0215] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship arises through computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within a cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and virtual private server (VPS) services.

[0216] The technical solution of the disclosed embodiment is aimed at the situation in the field of HVAC temperature control where only incomplete observation of Markov decision can be performed. By using a recurrent neural network as the decision network under the reinforcement learning framework, the sensitivity of the recurrent neural network to features in the time dimension is used to eliminate as much as possible the impact of the Markov decision that can only be incompletely observed on the accuracy of the prediction results, so that the target HVAC temperature control model generated thereby can achieve better temperature control effect.

[0217] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0218] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for generating a HVAC temperature control model, comprising: Based on the preset observable parameters in the HVAC temperature control field, the intelligent agent used to output corresponding control actions in the reinforcement learning framework is initialized to obtain the initialized intelligent agent; Constructing a first training set with the observable parameters as input and corresponding control actions as output; using the first training set to train an agent corresponding to the learning phase to obtain a first agent, the first agent being configured to output candidate control actions corresponding to the input observable parameters; and constructing a second training set with the control actions as input and corresponding safety assessment scores as output; Using the second training set to train an agent corresponding to the safety control stage, a second agent is obtained, wherein the second agent is used to determine the safety assessment scores corresponding to the candidate control actions and select the screened control actions that meet the preset safety requirements based on the safety assessment scores; and constructing a third training set with the control actions as input and the corresponding power consumption as output; Using the third training set to train an agent corresponding to the energy consumption reduction stage to obtain a third agent, the third agent being used to determine the power consumption corresponding to the filtered control actions, and to filter out target control actions that meet preset power consumption requirements based on the power consumption; A preset recurrent neural network is used as a decision network, and a target HVAC temperature control model is generated based on the trained intelligent agent and the decision network.

2. The method according to claim 1, wherein The preset security requirement includes: the security assessment score is the highest; the preset power consumption requirement includes: the power consumption is the lowest.

3. The method according to claim 1, wherein The observable parameters include at least one of the following: Computer room load, computer room temperature, cold storage tank flow rate, wet bulb temperature, primary pump frequency, cooling pump frequency, cooling tower power, chiller chilled water outlet temperature.

4. The method according to claim 1, wherein Initializing the intelligent agent for outputting corresponding control actions in the reinforcement learning framework based on the preset observation parameters in the HVAC temperature control field includes: In response to the absence of prior expert experience data stored in the memory bank, generating initial interaction experience for the agent based on random noise, wherein the interaction experience refers to experience of the agent returning second random noise serving as output data to first random noise serving as input data; In response to the data volume of the interaction experience meeting the preset data volume requirement, the interaction experience of the current intelligent agent is updated by combining the policy gradient and the small probability exploration method.

5. The method according to any one of claims 1 to 4, wherein: The method uses a preset recurrent neural network as a decision network and generates a target HVAC temperature control model based on the trained agent and the decision network, including: A recurrent neural network embedded in an autoencoder network is used as a first decision network, and a target HVAC temperature control model is generated based on the trained intelligent agent and the first decision network.

6. The method according to claim 5, wherein: The method of using the recurrent neural network embedded in the autoencoder network as the first decision network and generating a target HVAC temperature control model based on the trained intelligent agent and the first decision network includes: generating an operational experience trajectory of the temperature control device based on the input and output of the trained agent, wherein the operational experience trajectory is obtained by staggering the input states and output actions in a time sequence; Inserting the autoencoder network into the first layer and hidden state layer of the standard recurrent neural network to obtain the first decision network; Inputting the state in the operation experience trajectory into the first decision network to obtain a predicted action output by the first decision network; Using supervised learning, the predicted action is controlled to approach the actual action, and a first decision network whose degree of approach is less than a preset degree is determined as a target first decision network, wherein the actual action is an action in the operation experience trajectory that matches a state currently input to the first decision network; The target HVAC temperature control model is generated based on the trained intelligent agent and the target first decision network.

7. The method according to claim 6, further comprising: In response to the actual proximity degree being unable to be less than the preset degree, the encoding dimension of the autoencoding network is expanded until the proximity degree after the encoding dimension is expanded is less than the preset degree.

8. The method according to claim 6, further comprising: Extracting a preset number of discrete autoencoders from the target first decision network; forming a finite state machine based on the encoding conditions of the discrete autoencoder during the entire temperature control process, the total number of observation parameters and the total number of hidden states obtained by the first decision network during the decision-making process, and the transition process between different hidden states; Compressing the finite state machine using a finite state machine compression technique to obtain an optimal state machine; The number of hidden states and the number of observation parameters of the optimal state machine are used to guide the compression of the target first decision network.

9. The method according to any one of claims 1 to 4, wherein: The method uses a preset recurrent neural network as a decision network and generates a target HVAC temperature control model based on the trained agent and the decision network, including: According to the trained intelligent agent and the second decision network composed of a recurrent neural network, a target HVAC temperature control model is generated in combination with prior expert experience data under an imitation learning framework.

10. The method according to claim 9, wherein: The method generates a target HVAC temperature control model based on the trained intelligent agent and the second decision network served by the recurrent neural network in an imitation learning framework in combination with prior expert experience data, including: Extracting an operational experience trajectory of the temperature control device from the prior expert experience data, wherein the operational experience trajectory is obtained by staggering input states and output control actions in a time sequence, wherein the states are determined according to the observable parameters; Inputting the state in the operation experience trajectory into a second decision network played by the recurrent neural network to obtain a predictive control action output by the second decision network; In a generative adversarial imitation learning framework, the prior expert experience is used to train a generator and a discriminator to obtain hyperparameters and a reward function generated by a target generator that can be discriminated by a target discriminator. The target generator and the target discriminator are obtained after iterative training. The iterative training direction is controlled to control the hyperparameters and reward function generated by the generator to make the predicted control action output by the second decision network close to the actual control action, where the actual control action is the control action in the operating experience trajectory that matches the state currently input to the second decision network. The trained intelligent agent and the second decision network are used to generate the target HVAC temperature control model under the settings of the hyperparameters and the reward function.

11. The method according to claim 10, wherein: The hidden state of the previous step output by the second decision network will also be used as additional observation information for the next step to participate in the subsequent control action prediction.

12. The method according to any one of claims 1 to 4, wherein: The method uses a preset recurrent neural network as a decision network and generates a target HVAC temperature control model based on the trained agent and the decision network, including: The recurrent neural network embedded in the autoencoder network is used as the third decision network; Based on the trained intelligent agent and the third decision network, a target HVAC temperature control model is generated in combination with prior expert experience data under an imitation learning framework.

13. A HVAC temperature control method, comprising: Obtain the preset observable parameters under the target HVAC temperature control scenario; calling a preset target HVAC temperature control model to process the observable parameters and obtain an output control action, wherein the target HVAC temperature control model is obtained according to the HVAC temperature control model generation method according to any one of claims 1 to 12; Control the temperature control device to perform the control action.

14. A heating and ventilation temperature control model generation device, comprising: An agent initialization unit is configured to initialize an agent for outputting corresponding control actions in a reinforcement learning framework based on preset observable parameters in the HVAC temperature control field, thereby obtaining an initialized agent; an agent training unit configured to construct a first training set having the observable parameters as input and corresponding control actions as output; Using the first training set to train an agent corresponding to the learning phase, a first agent is obtained, wherein the first agent is configured to output an alternative control action corresponding to an input observable parameter; and constructing a second training set that uses the control action as input and outputs a corresponding safety assessment score. Using the second training set to train an agent corresponding to the safety control stage, a second agent is obtained, wherein the second agent is used to determine the safety assessment scores corresponding to the candidate control actions and select the screened control actions that meet the preset safety requirements based on the safety assessment scores; and constructing a third training set with the control actions as input and the corresponding power consumption as output; Using the third training set to train an agent corresponding to the energy consumption reduction stage to obtain a third agent, the third agent being used to determine the power consumption corresponding to the filtered control actions, and to filter out target control actions that meet preset power consumption requirements based on the power consumption; The target HVAC temperature control model generation unit is configured to use a preset recurrent neural network as a decision network and generate a target HVAC temperature control model based on the trained intelligent agent and the decision network.

15. The device according to claim 14, wherein The preset security requirement includes: the security assessment score is the highest; the preset power consumption requirement includes: the power consumption is the lowest.

16. The device according to claim 14, wherein The observable parameters include at least one of the following: Computer room load, computer room temperature, cold storage tank flow rate, wet bulb temperature, primary pump frequency, cooling pump frequency, cooling tower power, chiller chilled water outlet temperature.

17. The device according to claim 14, wherein The agent initialization unit is further configured to: In response to the absence of prior expert experience data stored in the memory bank, generating initial interaction experience for the agent based on random noise, wherein the interaction experience refers to experience of the agent returning second random noise serving as output data to first random noise serving as input data; In response to the data volume of the interaction experience meeting the preset data volume requirement, the interaction experience of the current intelligent agent is updated by combining the policy gradient and the small probability exploration method.

18. The device according to any one of claims 14 to 17, wherein: The target HVAC temperature control model generation unit includes: The first model generation subunit is configured to use the recurrent neural network embedded in the autoencoder network as the first decision network, and generate a target HVAC temperature control model based on the trained intelligent agent and the first decision network.

19. The device according to claim 18, wherein The first model generation subunit is further configured to: generating an operational experience trajectory of the temperature control device based on the input and output of the trained agent, wherein the operational experience trajectory is obtained by staggering the input states and output actions in a time sequence; Inserting the autoencoder network into the first layer and hidden state layer of the standard recurrent neural network to obtain the first decision network; Inputting the state in the operation experience trajectory into the first decision network to obtain a predicted action output by the first decision network; Using supervised learning, the predicted action is controlled to approach the actual action, and a first decision network whose degree of approach is less than a preset degree is determined as a target first decision network, wherein the actual action is an action in the operation experience trajectory that matches a state currently input to the first decision network; The target HVAC temperature control model is generated based on the trained intelligent agent and the target first decision network.

20. The apparatus according to claim 19, further comprising: The coding dimension expansion unit is configured to expand the coding dimension of the autoencoding network until the close degree after the coding dimension is expanded is less than the preset degree in response to the actual close degree being unable to be less than the preset degree.

21. The apparatus according to claim 19, further comprising: a discrete encoder extraction unit, configured to extract a preset number of discrete autoencoders from the target first decision network; a finite state machine forming unit configured to form a finite state machine based on an encoding condition of the discrete autoencoder during the entire temperature control process, a total number of observation parameters and a total number of hidden states obtained by the first decision network during the decision-making process, and a transition process between different hidden states; an optimal state machine acquisition unit, configured to compress the finite state machine using a finite state machine compression technique to obtain an optimal state machine; The compression guidance unit is configured to guide the compression of the target first decision network by utilizing the number of hidden states and the number of observation parameters of the optimal state machine.

22. The device according to any one of claims 14 to 17, wherein: The target HVAC temperature control model generation unit includes: The second model generation subunit is configured to generate a target HVAC temperature control model based on the trained intelligent agent and the second decision network composed of a recurrent neural network in combination with prior expert experience data under the imitation learning framework.

23. The device according to claim 22, wherein The second model generating subunit is further configured to: Extracting an operational experience trajectory of the temperature control device from the prior expert experience data, wherein the operational experience trajectory is obtained by staggering input states and output control actions in a time sequence, wherein the states are determined according to the observable parameters; Inputting the state in the operation experience trajectory into a second decision network played by the recurrent neural network to obtain a predictive control action output by the second decision network; In a generative adversarial imitation learning framework, the prior expert experience is used to train a generator and a discriminator to obtain hyperparameters and a reward function generated by a target generator that can be discriminated by a target discriminator. The target generator and the target discriminator are obtained after iterative training. The iterative training direction is controlled to control the hyperparameters and reward function generated by the generator to make the predicted control action output by the second decision network close to the actual control action, where the actual control action is the control action in the operating experience trajectory that matches the state currently input to the second decision network. The trained intelligent agent and the second decision network are used to generate the target HVAC temperature control model under the settings of the hyperparameters and the reward function.

24. The device according to claim 23, wherein the hidden state of the previous step output by the second decision network will also be used as additional observation information for the next step to participate in the subsequent control action prediction.

25. The device according to any one of claims 14 to 17, wherein the target HVAC temperature control model generation unit comprises: A third decision network determination subunit is configured to use a recurrent neural network embedded in an autoencoder network as the third decision network; The third model generation subunit is configured to generate a target HVAC temperature control model based on the trained intelligent agent and the third decision network in an imitation learning framework in combination with prior expert experience data.

26. A HVAC temperature control device comprising: an observable parameter acquisition unit, configured to acquire preset observable parameters in a target HVAC temperature control scenario; a model calling processing unit configured to call a preset target HVAC temperature control model to process the observable parameters and obtain an output control action, wherein the target HVAC temperature control model is obtained according to the HVAC temperature control model generating device according to any one of claims 14 to 25; The control action execution unit is configured to control the temperature control device to execute the control action.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the HVAC temperature control model generation method described in any one of claims 1-12 and / or the HVAC temperature control method described in claim 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the HVAC temperature control model generation method according to any one of claims 1 to 12 and / or the HVAC temperature control method according to claim 13.

29. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the HVAC temperature control model generation method according to any one of claims 1 to 12 and / or the steps of the HVAC temperature control method according to claim 13.

Images