A control method, device, electronic equipment, and storage medium for an air conditioning room.

Abstract

This disclosure relates to a control method, device, electronic equipment, and storage medium for an air conditioning room. The method includes: constructing a reinforcement learning model and an action value prediction model for the air conditioning room; determining four-tuple data based on the reinforcement learning model of the air conditioning room; training the action value prediction model using the operating state parameters and action parameters of the air conditioning room as inputs and action value parameters as outputs; inputting the known operating state parameters and action parameters of the air conditioning room into the trained action value prediction model, outputting the action values ​​corresponding to different action parameters under the same operating state parameters, and determining the control parameters of the air conditioning room based on the action values. This solves the problem that multiple operating parameters and multiple action parameters of the air conditioning room lead to too many elements in the action value matrix of the reinforcement learning model in the prior art, making it difficult to determine the optimal action.

Description

Technical Field

[0001] This disclosure relates to the field of air conditioning room technology, and in particular to a control method and device, electronic equipment and storage medium for an air conditioning room. Background Technology

[0002] Air conditioning rooms, as relatively complex control systems, contain multiple intricate subsystems such as chillers, pump units, and cooling towers. Traditional modeling methods struggle to handle such complexity. Therefore, reinforcement learning is used to model the system, learning the optimal action that maximizes the state value under different system states. First, based on Markov decision processes, the definitions of the system's states, actions, rewards, and action values ​​need to be established, transforming the problem into a reinforcement learning problem, which can then be solved using reinforcement learning methods.

[0003] Traditional reinforcement learning methods maintain a state-action value matrix Q. By continuously updating the action value matrix through reinforcement learning, the action value of taking different actions in each state of the system can be obtained. By taking actions with higher action value, the entire system can be controlled, resulting in an optimized system control strategy.

[0004] Currently, many control strategies for air conditioning computer rooms adopt this approach. This approach relies on discretizing the continuous system states and possible actions to allow them to be described by a finite state-action value matrix. However, air conditioning systems are complex. As mentioned earlier, air conditioning computer room systems have numerous state variables, including temperature, speed, power, and more than a dozen others. Furthermore, each variable is not a simple discrete value but a precise continuous value. Discretizing these variables while maintaining sufficient model accuracy can result in thousands or even tens of thousands of different discrete states, leading to an extremely large state-action value matrix from which the model cannot learn. For example, considering the 11 state variables mentioned above, simply taking five different discrete values ​​for each state variable results in a number of different states.

[0005] numberofstates = 11 5 =161051

[0006] At this point, it is clearly not feasible to use the method of maintaining the state action value matrix. Summary of the Invention

[0007] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, embodiments of this disclosure provide a control method and apparatus for an air conditioning room, electronic equipment and storage medium.

[0008] In a first aspect, embodiments of this disclosure provide a control method for an air conditioning room, comprising:

[0009] Construct a reinforcement learning model and an action value prediction model for the air conditioning room;

[0010] The four-tuple data is determined based on the reinforcement learning model of the air conditioning room, wherein the four-tuple data includes the operating status parameters of the air conditioning room, the action parameters taken, the operating status parameters after the action is taken, and the immediate feedback.

[0011] The action value prediction model is trained by using the operating status parameters of the air conditioning room and the action parameters taken as inputs, and the action value parameters as outputs. The loss function is constructed using the immediate reward and the action value parameters after taking the action.

[0012] The known operating status parameters and action parameters of the air conditioning room are input into the trained action value prediction model, and the action value corresponding to different action parameters under the same operating status parameters is output. Based on the action value, the action parameters used as control parameters for the air conditioning room are determined.

[0013] In one possible implementation, the construction of the reinforcement learning model for the air conditioning room includes:

[0014] A reinforcement learning model for the air conditioning room is constructed based on the operating status parameters and action parameters of the air conditioning room. When the operating status parameters and action parameters of the air conditioning room are input into the reinforcement learning model, the model outputs the operating status parameters after the action is taken and an immediate reward.

[0015] In one possible implementation, the step of determining four-tuple data based on the reinforcement learning model of the air conditioning room, wherein the four-tuple data includes operating state parameters of the air conditioning room, action parameters taken, operating state parameters after taking the action, and immediate feedback, including:

[0016] For the current operating status parameters of the air conditioning room, an ε-greedy strategy is adopted to select the action parameters. An action parameter is randomly selected with a probability of ε, and the action parameter with the highest action value under the current operating status parameters is selected with a probability of 1-.

[0017] The current operating status parameters and action parameters of the air conditioning room are input into the reinforcement learning model of the air conditioning room to obtain the operating status parameters and immediate reward at the next moment, and a set of training data is obtained. The set of training data is stored in the training pool, and new training data is generated in the same way.

[0018] A portion of the data is sampled from the training pool as a training set for training the action value prediction model.

[0019] In one possible implementation, the action value prediction model includes a first action value prediction model and a second action value prediction model, wherein the first action value prediction model and the second action value prediction model have the same structure and initial network parameters, take the operating status parameters of the air conditioning room and the action parameters taken as inputs, and take action value parameters as outputs. The action value prediction model is trained using a loss function constructed from the immediate reward and the action value parameters after taking the action, including:

[0020] The first step involves training the first action value prediction model using the operating status parameters of the air conditioning room and the action parameters taken as inputs, and the action value parameters as outputs. When the number of training iterations reaches a preset threshold, the network parameters of the second action value prediction model are updated to match the network parameters of the first action value prediction model.

[0021] The second step involves using the immediate rewards after taking action, the output of the first action value prediction model corresponding to the current operating state parameters, and the output of the second action value prediction model corresponding to the operating state parameters at the next moment to construct a loss function to adjust the network parameters of the first action value prediction model.

[0022] The third step is to use the first action value prediction model trained in the first and second steps as the model for predicting action value.

[0023] Fourth, repeat steps one through three to obtain multiple first action value prediction models.

[0024] In one possible implementation, the step of inputting the known operating status parameters of the air conditioning room and the action parameters taken into a trained action value prediction model, and outputting the action value corresponding to different action parameters under the same operating status parameters, includes:

[0025] The first step is to input the known operating status parameters of the air conditioning room and the action parameters taken into the trained action value prediction model to obtain multiple action values ​​corresponding to the same action parameter under the current operating status parameters.

[0026] The second step is to calculate the average value of multiple action values ​​as the action value corresponding to the same action parameter under the current operating state parameters;

[0027] The third step involves repeating the first and second steps while keeping the running state parameters unchanged and changing the action parameters, to obtain multiple action values ​​corresponding to different action parameters under the same running state parameters.

[0028] In one possible implementation, determining the action parameters for use as control parameters of the air conditioning room based on the action value includes:

[0029] Based on the magnitude of the action value, sort the multiple action values ​​corresponding to different action parameters under the same operating state parameter to obtain the first preset number of action values;

[0030] Calculate the probability that the action parameter corresponding to each action value in the pre-preset number of action values ​​will be used as a control parameter;

[0031] The action parameter with the highest probability of being used as the control parameter is used as the action parameter of the air conditioning room.

[0032] In one possible implementation, the operating status parameters of the air conditioning room include any of the following: chilled water inlet temperature, chilled water outlet temperature, cooling water inlet temperature, cooling water outlet temperature, chilled water flow rate, cooling water flow rate, cooling water pump speed, chilled water pump speed, cooling tower fan speed, indoor ambient temperature, outdoor ambient temperature, ambient pedestrian traffic, weather conditions, total system operating power, and total system operating efficiency; each set of action parameters includes any of the following: increasing the chilled water outlet temperature setpoint, decreasing the chilled water outlet temperature setpoint, setting different numbers of operating chiller units, increasing the pump speed setpoint, decreasing the pump speed setpoint, setting different numbers of operating pump units, increasing the cooling tower fan speed setpoint, decreasing the cooling tower fan speed setpoint, increasing the cooling water temperature setpoint, and decreasing the cooling water temperature setpoint.

[0033] Secondly, embodiments of this disclosure provide a control device for an air conditioning room, comprising:

[0034] The building module is used to build reinforcement learning models and action value prediction models for air conditioning rooms;

[0035] The determination module is used to determine the four-tuple data based on the reinforcement learning model of the air conditioning room, wherein the four-tuple data includes the operating status parameters of the air conditioning room, the action parameters taken, the operating status parameters after the action is taken, and the immediate feedback.

[0036] The training module is used to train the action value prediction model by taking the operating status parameters of the air conditioning room and the action parameters as inputs, and the action value parameters as outputs, and using the loss function constructed by the instant reward and the action value parameters after taking the action.

[0037] The output module is used to input the known operating status parameters and action parameters of the air conditioning room into the trained action value prediction model, output the action value corresponding to different action parameters under the same operating status parameters, and use the action parameter corresponding to the maximum action value as the control parameter of the air conditioning room.

[0038] Thirdly, embodiments of this disclosure provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0039] Memory, used to store computer programs;

[0040] The processor, when executing the program stored in the memory, implements the aforementioned control method for the air conditioning room.

[0041] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned control method for an air-conditioning room.

[0042] Compared with the prior art, the technical solutions provided in this disclosure have at least some or all of the following advantages:

[0043] The air conditioning room control method disclosed in this embodiment constructs a reinforcement learning model and an action value prediction model for the air conditioning room. Based on the reinforcement learning model, a four-tuple data set is determined, comprising the air conditioning room's operating state parameters, action parameters taken, operating state parameters after the action is taken, and immediate feedback. Using the air conditioning room's operating state parameters and action parameters taken as inputs, and action value parameters as outputs, a loss function constructed using the immediate feedback and the action value parameters after the action is taken is used to train the action value prediction model. The known operating state parameters and action parameters of the air conditioning room are input into the trained action value prediction model, which outputs the action values ​​corresponding to different action parameters under the same operating state parameters. Based on the action values, action parameters used as control parameters for the air conditioning room are determined. This method solves the problem that multiple operating parameters and multiple action parameters in the air conditioning room lead to an excessive number of action value matrix elements in the reinforcement learning model, making it difficult to determine the optimal action. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0045] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0046] Figure 1This illustration schematically shows a flow chart of a control method for an air conditioning room according to an embodiment of the present disclosure;

[0047] Figure 2 A schematic diagram illustrating the structure of a Q-network according to an embodiment of the present disclosure is shown.

[0048] Figure 3 A simplified schematic diagram illustrating the solution of action values ​​via a Q-network according to an embodiment of the present disclosure is shown.

[0049] Figure 4 A schematic diagram illustrating the structure of a control device for an air conditioning room according to an embodiment of the present disclosure; and

[0050] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0052] See Figure 1 The embodiments of this disclosure provide a control method for an air conditioning room, including:

[0053] S1, Construct a reinforcement learning model and an action value prediction model for the air conditioning room;

[0054] S2, determine the quadruple data according to the reinforcement learning model of the air conditioning room, wherein the quadruple data includes the operating status parameters of the air conditioning room, the action parameters taken, the operating status parameters after taking the action, and the immediate feedback.

[0055] In practical applications, the operating status parameters of the air conditioning room include any of the following: chilled water inlet temperature, chilled water outlet temperature, cooling water inlet temperature, cooling water outlet temperature, chilled water flow rate, cooling water flow rate, cooling water pump speed, chilled water pump speed, cooling tower fan speed, indoor ambient temperature, outdoor ambient temperature, ambient pedestrian traffic, weather conditions, total system operating power, and total system operating efficiency. Each set of action parameters includes any of the following: increasing the chilled water outlet temperature setpoint, decreasing the chilled water outlet temperature setpoint, setting different numbers of operating chiller units, increasing the pump speed setpoint, decreasing the pump speed setpoint, setting different numbers of operating pump units, increasing the cooling tower fan speed setpoint, decreasing the cooling tower fan speed setpoint, increasing the cooling water temperature setpoint, and decreasing the cooling water temperature setpoint.

[0056] S3, using the operating status parameters of the air conditioning room and the action parameters taken as inputs, and the action value parameters as outputs, the action value prediction model is trained using the loss function constructed from the instantaneous reward and the action value parameters after taking the action.

[0057] S4. Input the known operating status parameters and action parameters of the air conditioning room into the trained action value prediction model, output the action value corresponding to different action parameters under the same operating status parameters, and determine the action parameters used as control parameters of the air conditioning room based on the action value.

[0058] In this embodiment, step S1, constructing the reinforcement learning model for the air conditioning room, includes:

[0059] A reinforcement learning model for the air conditioning room is constructed based on the operating status parameters and action parameters of the air conditioning room. When the operating status parameters and action parameters of the air conditioning room are input into the reinforcement learning model, the model outputs the operating status parameters after the action is taken and an immediate reward.

[0060] In practical applications, air conditioning rooms, as relatively complex control systems, contain multiple intricate subsystems such as chillers, pump units, and cooling towers. Traditional modeling methods struggle to handle such complex problems. Therefore, reinforcement learning is used to model the system and learn the optimal action that maximizes the state value under different system states. First, based on Markov decision processes, the definitions of the system's states, actions, rewards, and action values ​​need to be established, transforming the problem into a reinforcement learning problem, which can then be solved using reinforcement learning methods.

[0061] The definitions of system state, action, reward, and action value include:

[0062] State: The state space is S = {s1, s2, ..., s} nThe current state of the system is quite complex, including multiple quantities related to the system: inlet and outlet temperatures and flow rates of chilled water and cooling water, current ambient and indoor temperatures, cooling water pump speed, chilled water pump speed, cooling tower fan speed, and current system power and energy consumption. These quantities can comprehensively characterize the overall state of the system at the current moment and facilitate model learning. The state can be represented by a high-dimensional vector: s = (s1, s2, ..., s...). k ) T .

[0063] Actions: The action space is A = {a1, ..., a...} m The actions include adjusting the settings of the control systems in chillers, pump units, and cooling towers. This includes adjusting the setpoints for the outlet and return temperatures of chilled and cooling water, as well as setting the fan speeds of pump units and cooling towers.

[0064] Return: The system needs to reduce power consumption while ensuring a comfortable air conditioning temperature. Therefore, the return value needs to consider the difference between the current indoor temperature and the suitable temperature, as well as the difference between the current power and the optimal power. A bias term γ is added to ensure a reasonable return value.

[0065] r t =-α·r|T in -T p |-β·(W t -W p )+γ

[0066] Where α and β are proportionality coefficients, which can be adjusted according to the actual operation of the model, and γ is the discount factor for the return.

[0067] Action Value: Action Value Q(S) t A t ) represents taking action A in the state at time t. t The goal of the entire model is to learn a better action value matrix through reinforcement learning. Based on the action value, we can determine which action is optimal in the current system state.

[0068] In this embodiment, in step S2, the four-tuple data is determined based on the reinforcement learning model of the air conditioning room. The four-tuple data includes the operating status parameters of the air conditioning room, the action parameters taken, the operating status parameters after the action is taken, and the immediate feedback, including:

[0069] For the current operating status parameters of the air conditioning room, an ε-greedy strategy is adopted to select the action parameters. An action parameter is randomly selected with a probability of ε, and the action parameter with the highest action value under the current operating status parameters is selected with a probability of 1-ε.

[0070] The current operating status parameters and action parameters of the air conditioning room are input into the reinforcement learning model of the air conditioning room to obtain the operating status parameters and immediate reward at the next moment, and a set of training data is obtained. The set of training data is stored in the training pool, and new training data is generated in the same way.

[0071] A portion of the data is sampled from the training pool as a training set for training the action value prediction model.

[0072] In this embodiment, step S3 includes a first action value prediction model and a second action value prediction model. The first and second action value prediction models have the same structure and initial network parameters. They use the operating status parameters of the air conditioning room and the action parameters taken as inputs, and the action value parameters as outputs. The action value prediction model is trained using a loss function constructed from the immediate reward and the action value parameters after the action is taken. The training includes:

[0073] The first step involves training the first action value prediction model using the operating status parameters of the air conditioning room and the action parameters taken as inputs, and the action value parameters as outputs. When the number of training iterations reaches a preset threshold, the network parameters of the second action value prediction model are updated to match the network parameters of the first action value prediction model.

[0074] The second step involves using the immediate rewards after taking action, the output of the first action value prediction model corresponding to the current operating state parameters, and the output of the second action value prediction model corresponding to the operating state parameters at the next moment to construct a loss function to adjust the network parameters of the first action value prediction model.

[0075] The third step is to use the first action value prediction model trained in the first and second steps as the model for predicting action value.

[0076] Fourth, repeat steps one through three to obtain multiple first action value prediction models.

[0077] In practical applications, taking the first action value prediction model as the Q-network and the second action value prediction model as the target network as an example, the training of the action value prediction model is further explained:

[0078] The number of actions taken can be discretized relatively simply, given a current state S. t The model yields the action value of taking different actions 'a'. This can be represented by a value function:

[0079] Q = f(s) t )

[0080] Where Q = (Q(s) t ,a1),Q(s t ,a2),…,Q(s t ,a m )) T Then we can get the current state of s. t The value of each action is calculated, allowing for different actions to be taken based on these values, thus yielding a better control strategy for the entire system. Therefore, learning the f-function is crucial in the model. A deep learning network (DNN) is used to model this value function; this deep learning network is called a Q-network.

[0081] Q network such as Figure 2 As shown:

[0082] First, Q-networks have the same functionality as traditional state-valued networks, for a given state s t and action a t It can output the corresponding state value, using N. Q This represents the network. Therefore, the input data of the network can be defined as x = (s t ,a t ) T S here t As introduced earlier, it is a k-dimensional vector, where k represents different state variables. The output is the corresponding state value y = Q(S t ,a t ) = N N (A t ,a t |θ), where θ represents the network parameters. The network's training label should be the actual state value, but this is an unknown value; therefore, the state value Q(s) at the next time step is needed. t+1 a) to fit the actual state value. Drawing on the ideas of traditional Q-learning reinforcement learning, the model's loss function can be defined as follows:

[0083]

[0084]

[0085] That is, in s t Take a at all times t Action, obtain the state s of the next moment. t+1 and instant rewards t+1 The neural network is trained using the sum of squares of the differences between the immediate reward and the maximum state value at time t+1, and the state value at time t, as the loss.

[0086] Training data collection:

[0087] Based on the definitions of network input, output, and loss function above, a neural network is built. Training the network requires a large amount of training data, including the current state, the action taken, the corresponding next state, and the immediate reward. D = (s t ,a t ,s t+1 ,r t+1 ) T .

[0088] Regarding the current state s t An ε-greedy strategy can be adopted to select the action to be taken, that is, randomly select an action with a probability of ε, and select the action with a probability of 1-ε that maximizes the value of the current state. Inputting this action into the system will yield the state s for the next time step. t+1 and instant rewards t+1 This yields a set of training data, which is then stored in a data pool. New training data can then be generated continuously using the same method. It's important to note that to ensure data independence, a subset of data is sampled from the training pool as the training set.

[0089] Target network:

[0090] In the network above, the Q-network model is continuously trained using data from the training pool, and the model parameters are constantly updated. However, we can see that N in the loss function... Q (s t+1 The loss function (a|θ) is also constantly being updated, which may cause the Q network to fail to converge. Therefore, a network with the exact same structure as the original Q network is added, with its parameters denoted by θ′. The loss function above is then modified as follows:

[0091]

[0092] The loss function is differentiated with respect to the parameter θ, and the loss is backpropagated using stochastic gradient descent to update the parameters of the Q network.

[0093] Updating the Q-network parameters using the parameters of the target network allows you to set the target network to be updated with the updated Q-parameter values ​​after a certain number of training epochs. This ensures that the target value will not be updated for a certain period during training, guaranteeing network convergence.

[0094] Finally, a well-trained Q-network is obtained, which can obtain the state action value of taking different actions in any state of the system within a reasonable range. Based on this network, the control strategy of the system can be adjusted to ensure that the usage requirements are met and low power consumption is guaranteed as much as possible.

[0095] like Figure 3As shown, in practical applications, although random sampling from the data pool ensures data randomness, a well-performing Q-network is ultimately trained. However, due to the complexity of air conditioning room systems, random initialization and sampling may prevent the Q-network model from encountering some complex situations. Consequently, the trained Q-network might produce results that do not meet the optimization objectives for certain input states. To avoid this, an ensemble learning strategy is considered, which involves multiple initializations and training cycles of the Q-network, retaining the trained network model. This results in multiple trained, differentiated Q-networks {Q1, Q2, ..., Q...}. n These Q-networks will give different action values ​​for the same state. Averaging these action values ​​yields the integrated action value output.

[0096]

[0097] This avoids the problem of a single network performing poorly on certain special input cases.

[0098] Therefore, in this embodiment, step S4, which involves inputting the known operating status parameters of the air conditioning room and the action parameters taken into the trained action value prediction model, and outputting the action value corresponding to different action parameters under the same operating status parameters, includes:

[0099] The first step is to input the known operating status parameters of the air conditioning room and the action parameters taken into the trained action value prediction model to obtain multiple action values ​​corresponding to the same action parameter under the current operating status parameters.

[0100] The second step is to calculate the average value of multiple action values ​​as the action value corresponding to the same action parameter under the current operating state parameters;

[0101] The third step involves repeating the first and second steps while keeping the running state parameters unchanged and changing the action parameters, to obtain multiple action values ​​corresponding to different action parameters under the same running state parameters.

[0102] In practical applications, the Q-network has already been trained using a reinforcement learning strategy. Considering the instability of the network output, directly selecting the strategy based on the maximum output value in some states may increase energy consumption. Therefore, the control strategy needs to be adjusted. In actual operation in the computer room, at the current moment, the collected state information s is input into the Q-network to obtain the predicted action values ​​Q(s,a1), Q(s,a2), ..., Q(s,a3) for all actions in the current state. mThese actions are ranked by value; for example, the top three actions with higher values ​​q1, q2, and q3 are selected. The proportion of these three action values ​​is then considered as a probability, and the action the system needs to take at this moment is chosen based on this probability.

[0103]

[0104] Among them, a qi Represents the value of action q i The corresponding actions. In a real-world environment, these numbers can be adjusted.

[0105] Therefore, in this embodiment, step S4, determining the action parameters used as control parameters for the air conditioning room based on the action value, includes:

[0106] Based on the magnitude of the action value, sort the multiple action values ​​corresponding to different action parameters under the same operating state parameter to obtain the first preset number of action values;

[0107] Calculate the probability that the action parameter corresponding to each action value in the pre-preset number of action values ​​will be used as a control parameter;

[0108] The action parameter with the highest probability of being used as the control parameter is used as the action parameter of the air conditioning room.

[0109] In practical applications, due to the complexity of the air conditioning room system itself, it is impossible to collect all possible data during the training of the Q-network. Therefore, the Q-network model does not perform well for some uncollected data. As a result, in the actual control process of the air conditioning room system, the model cannot effectively optimize control for all situations. Therefore, by using online fine-tuning, that is, by supplementing the data pool used for training with new data points during the actual operation of the system, and sampling some of the newly added data and some of the original data after a preset period of time, the Q-network is fine-tuned. This makes the Q-network more generalizable and robust, and the system can better handle various complex situations.

[0110] See Figure 4 The present disclosure provides a control device for an air conditioning room, comprising:

[0111] Module 11 is used to build a reinforcement learning model and an action value prediction model for the air conditioning room.

[0112] The determination module 12 is used to determine the four-tuple data according to the reinforcement learning model of the air conditioning room, wherein the four-tuple data includes the operating status parameters of the air conditioning room, the action parameters taken, the operating status parameters after the action is taken, and the immediate feedback.

[0113] Training module 13 is used to train the action value prediction model by taking the operating status parameters of the air conditioning room and the action parameters taken as inputs, and the action value parameters as outputs, and using the loss function constructed by the instant reward and the action value parameters after taking the action.

[0114] The output module 14 is used to input the known operating status parameters and action parameters of the air conditioning room into the trained action value prediction model, output the action value corresponding to different action parameters under the same operating status parameters, and take the action parameter corresponding to the maximum action value as the control parameter of the air conditioning room.

[0115] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0116] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0117] In the second embodiment described above, any plurality of the construction module 11, determination module 12, training module 13, and output module 14 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. At least one of the construction module 11, determination module 12, training module 13, and output module 14 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the construction module 11, determination module 12, training module 13, and output module 14 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0118] Reference Figure 5As shown, the electronic device provided in the embodiments of this disclosure includes a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140.

[0119] Memory 1130 is used to store computer programs;

[0120] When processor 1110 executes the program stored in memory 1130, it implements the following control method for the air conditioning room:

[0121] Construct a reinforcement learning model and an action value prediction model for the air conditioning room;

[0122] The four-tuple data is determined based on the reinforcement learning model of the air conditioning room, wherein the four-tuple data includes the operating status parameters of the air conditioning room, the action parameters taken, the operating status parameters after the action is taken, and the immediate feedback.

[0123] The action value prediction model is trained by using the operating status parameters of the air conditioning room and the action parameters taken as inputs, and the action value parameters as outputs. The loss function is constructed using the immediate reward and the action value parameters after taking the action.

[0124] The known operating status parameters and action parameters of the air conditioning room are input into the trained action value prediction model, and the action value corresponding to different action parameters under the same operating status parameters is output. Based on the action value, the action parameters used as control parameters for the air conditioning room are determined.

[0125] The aforementioned communication bus 1140 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0126] The communication interface 1120 is used for communication between the above-mentioned electronic device and other devices.

[0127] The memory 1130 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1130 may also be at least one storage device located remotely from the aforementioned processor 1110.

[0128] The processor 1110 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0129] Based on the same inventive concept, embodiments of this disclosure also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the control method for the air conditioning room as described above.

[0130] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement the control method for an air conditioning room according to embodiments of this disclosure.

[0131] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0132] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0133] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A control method of an air conditioning machine room characterized by comprising: The method comprises: building a reinforcement learning model and an action value prediction model of an air-conditioning machine room; determining a quadruple data according to the reinforcement learning model of the air-conditioning machine room, wherein the quadruple data comprises an operating state parameter of the air-conditioning machine room, an action parameter taken, an operating state parameter after the action is taken, and an immediate reward; training the action value prediction model by taking the operating state parameter of the air-conditioning machine room and the action parameter taken as inputs, taking an action value parameter as an output, and using a loss function constructed by the immediate reward and the action value parameter after the action is taken; inputting the known operating state parameter of the air-conditioning machine room and the action parameter taken into the trained action value prediction model, outputting the action value corresponding to different action parameters under the same operating state parameter, and determining the action parameter used as a control parameter of the air-conditioning machine room according to the action value; the determining of the action parameter used as the control parameter of the air-conditioning machine room according to the action value comprises: sorting a plurality of action values corresponding to different action parameters under the same operating state parameter according to the size of the action value to obtain a preset number of action values; calculating the probability of the action parameter corresponding to each of the preset number of action values as the control parameter; taking the action parameter corresponding to the maximum probability of the action parameter as the control parameter as the action parameter of the air-conditioning machine room.

2. The method of claim 1, wherein, the building of the reinforcement learning model of the air-conditioning machine room comprises: building the reinforcement learning model of the air-conditioning machine room according to the operating state parameter of the air-conditioning machine room and the action parameter taken, so that when the operating state parameter of the air-conditioning machine room and the action parameter taken are input into the reinforcement learning model, the operating state parameter after the action is taken and the immediate reward are output.

3. The method of claim 1, wherein, the determining of the quadruple data according to the reinforcement learning model of the air-conditioning machine room, wherein the quadruple data comprises the operating state parameter of the air-conditioning machine room, the action parameter taken, the operating state parameter after the action is taken, and the immediate reward, comprises: for the current operating state parameter of the air-conditioning machine room, an action parameter taken is selected by an epsilon-greedy strategy, an action parameter is randomly selected with a probability of epsilon, and an action parameter with the maximum action value under the current operating state parameter is selected with a probability of 1-epsilon; inputting the current operating state parameter of the air-conditioning machine room and the action parameter taken into the reinforcement learning model of the air-conditioning machine room to obtain the operating state parameter at the next time and the immediate reward, obtaining a set of training data, storing the set of training data into a training pool, and generating new training data in the same way; sampling a part of the data from the training pool as a training set for training the action value prediction model.

4. The method of claim 1, wherein, the action value prediction model comprises a first action value prediction model and a second action value prediction model, wherein the first action value prediction model and the second action value prediction model are the same in structure and initial network parameters, the action value prediction model is trained by taking the operating state parameter of the air-conditioning machine room and the action parameter taken as inputs, taking an action value parameter as an output, and using a loss function constructed by the immediate reward and the action value parameter after the action is taken, and the training of the action value prediction model comprises: The first step involves training the first action value prediction model using the operating status parameters of the air conditioning room and the action parameters taken as inputs, and the action value parameters as outputs. When the number of training iterations reaches a preset threshold, the network parameters of the second action value prediction model are updated to match the network parameters of the first action value prediction model. The second step involves using the immediate rewards after taking action, the output of the first action value prediction model corresponding to the current operating state parameters, and the output of the second action value prediction model corresponding to the operating state parameters at the next moment to construct a loss function to adjust the network parameters of the first action value prediction model. The third step is to use the first action value prediction model trained in the first and second steps as the model for predicting action value. Fourth, repeat steps one through three to obtain multiple first action value prediction models.

5. The method of claim 4, wherein, The process of inputting known operating status parameters of the air conditioning room and action parameters into a trained action value prediction model, and outputting the action value corresponding to different action parameters under the same operating status parameters, includes: The first step is to input the known operating status parameters of the air conditioning room and the action parameters taken into the trained action value prediction model to obtain multiple action values ​​corresponding to the same action parameter under the current operating status parameters. The second step is to calculate the average value of multiple action values ​​as the action value corresponding to the same action parameter under the current operating state parameters; The third step involves repeating the first and second steps while keeping the running state parameters unchanged and changing the action parameters, to obtain multiple action values ​​corresponding to different action parameters under the same running state parameters.

6. The method of claim 5, wherein, The operating status parameters of the air conditioning room include any of the following: chilled water inlet temperature, chilled water outlet temperature, cooling water inlet temperature, cooling water outlet temperature, chilled water flow rate, cooling water flow rate, cooling water pump speed, chilled water pump speed, cooling tower fan speed, indoor ambient temperature, outdoor ambient temperature, ambient pedestrian traffic, weather conditions, total system operating power, and total system operating efficiency. Each set of action parameters includes any of the following: increasing the chilled water outlet temperature setpoint, decreasing the chilled water outlet temperature setpoint, setting different numbers of operating chiller units, increasing the pump speed setpoint, decreasing the pump speed setpoint, setting different numbers of operating pump units, increasing the cooling tower fan speed setpoint, decreasing the cooling tower fan speed setpoint, increasing the cooling water temperature setpoint, and decreasing the cooling water temperature setpoint.

7. An air conditioning plant room control device, characterized by comprising: include: The building module is used to build reinforcement learning models and action value prediction models for air conditioning rooms; The determination module is used to determine the four-tuple data based on the reinforcement learning model of the air conditioning room, wherein the four-tuple data includes the operating status parameters of the air conditioning room, the action parameters taken, the operating status parameters after the action is taken, and the immediate feedback. The training module is used to train the action value prediction model by taking the operating status parameters of the air conditioning room and the action parameters as inputs, and the action value parameters as outputs, and using the loss function constructed by the instant reward and the action value parameters after taking the action. An output module is configured to input the known operation state parameters and action parameters of the air conditioning machine room into the trained action value prediction model, output action values corresponding to different action parameters under the same operation state parameter, and take the action parameter corresponding to the maximum action value as the control parameter of the air conditioning machine room. The action parameter determined as the control parameter of the air conditioning machine room according to the action value comprises: sorting a plurality of action values corresponding to different action parameters under the same operation state parameter according to the sizes of the action values to obtain a preset number of action values in front; calculating a probability of the action parameter corresponding to each of the preset number of action values as the control parameter; taking the action parameter corresponding to the maximum probability of the action parameter as the control parameter as the control parameter of the air conditioning machine room.

8. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored on the memory to implement the control method of the air conditioning machine room in any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the control method of the air conditioning machine room in any one of claims 1-6.

Images