Air conditioning machine room system control method and device, electronic equipment and storage medium
By constructing a reinforcement learning model and utilizing policy learning networks and value evaluation networks, the complexity of modeling the air conditioning room system was solved, enabling collaborative control and optimization of multi-device data and improving the flexibility and adaptability of system control.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING JIXIN TAIFU MECHANICAL & ELECTRICAL TECH CO LTD
- Filing Date
- 2022-09-30
- Publication Date
- 2026-07-07
AI Technical Summary
Existing technologies model air conditioning room systems as multi-objective optimization problems, which are complex and lack generalization, making it difficult to adjust and improve system control strategies.
By constructing a reinforcement learning model and utilizing a policy learning network and a value evaluation network, the target state variable is selected based on the coupling relationship and correlation coefficient of the state variables of the air conditioning room system. The model is then trained to output the current control parameters, thereby achieving collaborative control of multi-device data.
It simplifies the modeling process, improves the flexibility and adaptability of system control, and enables collaborative optimization of data from multiple devices.
Smart Images
Figure CN115585541B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer room control technology, and in particular to a control method and device, electronic equipment and storage medium for an air-conditioned computer room system. Background Technology
[0002] The air conditioning room system consists of multiple subsystems such as chiller units, water pump units, and cooling towers. The control of the system involves the collaborative processing and optimization of data from multiple devices. In existing technologies, the problem is modeled as a multi-objective optimization problem and then solved.
[0003] However, modeling the problem as a multi-objective optimization problem and then solving it is complex, lacks generalization, and is not conducive to the adjustment and improvement of subsequent system control strategies. Summary of the Invention
[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, embodiments of this disclosure provide a control method and apparatus, electronic equipment and storage medium for an air conditioning room system.
[0005] In a first aspect, embodiments of this disclosure provide a control method for an air conditioning room system, the method comprising:
[0006] Based on the coupling relationship and correlation coefficient between different state variables of multiple devices in the air conditioning room system, a preset number of state variables are selected from all state variables of the air conditioning room system as target state variables.
[0007] A reinforcement learning model is constructed based on the target state variables, wherein the reinforcement learning model includes a policy learning network and a value evaluation network;
[0008] The reinforcement learning model is trained by taking the historical values of the target state variable as input and the historical control parameters of the air conditioning room system corresponding to the historical values of the target state variable as output.
[0009] Obtain the current value of the target state variable of the air conditioning room system, input it into the trained reinforcement learning model, and output the current control parameters of the air conditioning room system.
[0010] In one possible implementation, the preset quantity is determined by principal component analysis.
[0011] In one possible implementation, the correlation coefficient between the different state variables is calculated using the following expression:
[0012]
[0013] Where, r xyLet σ be the correlation coefficient between two state variables x and y, and let cov(x, y) be the covariance between the two state variables x and y. x Let σ be the standard deviation of the state variable x. y Let y be the standard deviation of the state variable y.
[0014] In one possible implementation, training the reinforcement learning model by taking the historical value of the target state variable as input and the historical control parameters of the air conditioning room system corresponding to the historical value of the target state variable as output includes:
[0015] Obtain the historical value of the target state variable at time t, use a greedy strategy to determine the corresponding action information, use the action information as the operating parameters of the air conditioning room system, obtain the target state variable data and real-time report data at time t+1, and collect them into the data pool.
[0016] The reinforcement learning model is trained by taking the historical value of the target state variable at time t, the target state variable data at time t+1, and the real-time report data from the data pool as inputs, and the action information used as the control parameters of the air conditioning room system as outputs.
[0017] In one possible implementation, the step of training the reinforcement learning model by taking the historical value of the target state variable at time t, the target state variable data at time t+1, and the real-time report data from the data pool as input, and the action information used as control parameters for the air conditioning room system as output, includes:
[0018] Input the historical value of the target state variable at time t in the data pool into the policy learning network, output the action information of the action taken at time t, and obtain the target state variable data at time t+1 and the real-time reward data at time t+1 corresponding to the historical value of the target state variable at time t and the action information of the action taken from the data pool.
[0019] Input the action information taken at time t into the value evaluation network to obtain the action value at time t and the gradient change value of the policy learning network parameters.
[0020] Input the target state variable data at time t+1 into the policy learning network, output the action information of the action to be taken at time t+1, input the action information at time t+1 into the value evaluation network, and obtain the action value at time t+1.
[0021] The loss value of the value assessment network is calculated based on the action value at time t, the action value at time t+1, and the immediate reward at time t+1. When the loss value is greater than a preset threshold, the parameters of the value assessment network are adjusted, and the parameters of the policy learning network are updated according to the gradient change value of the policy learning network parameters until the loss value is less than the preset threshold.
[0022] In one possible implementation, the gradient changes of the policy learning network parameters are obtained using the following expression:
[0023]
[0024] in, This represents the gradient changes of the policy learning network parameters when the number of training samples is N, where N is the number of training samples. For the target state variable s i The action information is a i In this case, the derivative of the value assessment network model with respect to the action parameters, For the target state variable s i In the case of i, the gradient change value of the policy learning network parameters output by the value evaluation network, where i takes the value of 1, 2, ..., N.
[0025] In one possible implementation, the loss value of the value assessment network is calculated based on the action value at time t, the action value at time t+1, and the immediate reward at time t+1 using the following expression:
[0026]
[0027] Where L is the loss value of the value evaluation network, N is the number of training samples, and r t+1 For immediate feedback at time t+1, Q′(s) t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ Let Q(s) be the action value at time t+1. t a t |θ Q Let t be the value of the action at time t, and γ be the discount factor.
[0028] Secondly, embodiments of this disclosure provide a control device for an air conditioning room system, comprising:
[0029] The selection module is used to select a preset number of state variables as target state variables from all state variables of the air conditioning room system based on the coupling relationship and correlation coefficient between different state variables of multiple devices in the air conditioning room system.
[0030] A building module is used to construct a reinforcement learning model based on target state variables, wherein the reinforcement learning model includes a policy learning network and a value evaluation network;
[0031] The training module is used to train the reinforcement learning model by taking the historical values of the target state variable as input and the historical control parameters of the air conditioning room system corresponding to the historical values of the target state variable as output.
[0032] The output module is used to obtain the current value of the target state variable of the air conditioning room system, input it into the trained reinforcement learning model, and output the current control parameters of the air conditioning room system.
[0033] Thirdly, embodiments of this disclosure provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;
[0034] Memory, used to store computer programs;
[0035] The processor, when executing the program stored in the memory, implements the control method of the air conditioning room system described above.
[0036] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method for the air conditioning room system described above.
[0037] Compared with the prior art, the technical solutions provided in this disclosure have at least some or all of the following advantages:
[0038] The control method for an air conditioning room system described in this embodiment selects a preset number of state variables as target state variables from all state variables of the air conditioning room system based on the coupling relationship and correlation coefficient between different state variables of multiple devices in the air conditioning room system; constructs a reinforcement learning model based on the target state variables, wherein the reinforcement learning model includes a policy learning network and a value evaluation network; trains the reinforcement learning model using the historical values of the target state variables as input and the historical control parameters of the air conditioning room system corresponding to the historical values of the target state variables as output; obtains the current value of the target state variables of the air conditioning room system and inputs it into the trained reinforcement learning model, outputting the current control parameters of the air conditioning room system; and utilizes the coupling relationship between data variables of multiple devices in the air conditioning room system to perform reinforcement learning modeling, thereby achieving the goal of data collaborative control of multiple devices. Attached Figure Description
[0039] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0040] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0041] Figure 1 A schematic flowchart of a control method for an air conditioning room system according to an embodiment of the present disclosure is shown.
[0042] Figure 2 A schematic flowchart of a control method for another air conditioning room system according to an embodiment of the present disclosure is shown.
[0043] Figure 3 A schematic block diagram of a control device for an air conditioning room system according to an embodiment of the present disclosure is shown; and
[0044] Figure 4 A schematic block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0046] See Figure 1 The present disclosure provides a control method for an air conditioning room system, the method comprising:
[0047] S1. Based on the coupling relationship and correlation coefficient between different state variables of multiple devices in the air conditioning room system, select a preset number of state variables as target state variables from all state variables of the air conditioning room system.
[0048] S2, Construct a reinforcement learning model based on the target state variables, wherein the reinforcement learning model includes a policy learning network and a value evaluation network;
[0049] S3, using the historical value of the target state variable as input and the historical control parameters of the air conditioning room system corresponding to the historical value of the target state variable as output, train the reinforcement learning model;
[0050] S4: Obtain the current value of the target state variable of the air conditioning room system, input it into the trained reinforcement learning model, and output the current control parameters of the air conditioning room system.
[0051] In this embodiment, in step S1, the preset quantity is determined by principal component analysis.
[0052] In this embodiment, the correlation coefficient between the different state variables is calculated using the following expression:
[0053]
[0054] Where, r xy Let σ be the correlation coefficient between two state variables x and y, and let cov(x, y) be the covariance between the two state variables x and y. x Let σ be the standard deviation of the state variable x. y Let y be the standard deviation of the state variable y.
[0055] In some embodiments, the air conditioning room system involves a large number of data variables. The calculation and estimation of air conditioning operating energy consumption, load, etc. are closely related to variables such as the inlet and outlet temperatures of chilled water and cooling water, and the flow rates of chilled water and cooling water. In addition, variables such as the pump speed of cooling water and chilled water, the fan speed of cooling tower, and the number of operating units are easier to operate and adjust. The energy consumption of the air conditioning system itself and the ambient temperature are also important variables.
[0056] To improve the learning efficiency of reinforcement learning models, the relationships between data variables from multiple devices are analyzed. Principal component analysis (PCA) is used to simplify the variables actually used in the model by considering the coupling relationships and correlation coefficients between different state variables. The coupling relationships include computational relationships. During PCA, the number of principal components k with a total variance contribution rate greater than 99% is selected as the number of principal variables.
[0057] The parameters of the water pump unit and cooling tower control the temperature changes and flow rates of cooling water and chilled water. Therefore, these variables are strongly correlated. In the control of the computer room system, once the outlet temperatures of cooling water and chilled water are set, the rotational speed of the water pump and cooling tower can be obtained based on the physical transformation relationship between the variables. Therefore, in the modeling, only the inlet and outlet temperatures of cooling water and chilled water need to be considered.
[0058] The inlet and outlet temperatures of cooling water are strongly correlated, as are the inlet and outlet temperatures of chilled water. This can be explained using the heat transfer formula: Q 冷冻 =CmΔt 冷冻 =Cm(t) out -t inAs can be seen, the specific heat capacity of water is fixed. When the heat change and flow rate are basically fixed, the inlet and outlet temperatures of chilled water have a very strong linear relationship. Therefore, when modeling, only the outlet temperature or the inlet temperature can be selected.
[0059] In reinforcement learning modeling, the selected state variable can be: chilled water outlet temperature t eo Cooling water inlet temperature t ci chilled water flow rate q e Cooling water flow rate q c Air conditioning system power P, ambient temperature T o and room temperature T i The state vector s = (t) consists of these 7 variables. eo , t ci q e q c P, T o T i ) T .
[0060] Based on the selected state variables, the corresponding action information variables can be determined, including the adjustment of the setpoints for chilled water outlet temperature and cooling water inlet temperature.
[0061] See Figure 2 In this embodiment, step S3, which involves training the reinforcement learning model using historical values of the target state variable as input and historical control parameters of the air conditioning room system corresponding to the historical values of the target state variable as output, includes:
[0062] Obtain the historical value of the target state variable at time t, use a greedy strategy to determine the corresponding action information, use the action information as the operating parameters of the air conditioning room system, obtain the target state variable data and real-time report data at time t+1, and collect them into the data pool.
[0063] The reinforcement learning model is trained by taking the historical value of the target state variable at time t, the target state variable data at time t+1, and the real-time report data from the data pool as inputs, and the action information used as the control parameters of the air conditioning room system as outputs.
[0064] In this embodiment, training the reinforcement learning model by taking the historical value of the target state variable at time t, the target state variable data at time t+1, and the real-time report data from the data pool as input, and the action information used as control parameters for the air conditioning room system as output, includes:
[0065] Input the historical value of the target state variable at time t in the data pool into the policy learning network, output the action information of the action taken at time t, and obtain the target state variable data at time t+1 and the real-time reward data at time t+1 corresponding to the historical value of the target state variable at time t and the action information of the action taken from the data pool.
[0066] Input the action information taken at time t into the value evaluation network to obtain the action value at time t and the gradient change value of the policy learning network parameters.
[0067] Input the target state variable data at time t+1 into the policy learning network, output the action information of the action to be taken at time t+1, input the action information at time t+1 into the value evaluation network, and obtain the action value at time t+1.
[0068] The loss value of the value assessment network is calculated based on the action value at time t, the action value at time t+1, and the immediate reward at time t+1. When the loss value is greater than a preset threshold, the parameters of the value assessment network are adjusted, and the parameters of the policy learning network are updated according to the gradient change value of the policy learning network parameters until the loss value is less than the preset threshold.
[0069] In this embodiment, the instant reward is calculated using the following expression:
[0070] r t =r T +r ci +r eo -α·P t +β
[0071] Where α is the coefficient of the power return value, β is the bias term, and r ci r eo Let r be the reported values of the cooling water inlet temperature and the chilled water outlet temperature at time t, respectively. T Let P be the return value of the room temperature at time t. t This is the power return value.
[0072] in,
[0073]
[0074]
[0075] [a ci b ci [a] represents the suitable range for the cooling water inlet temperature. eo b eo [a] represents the suitable range for chilled water outlet temperature. Similarly, [a] T b T [ ] represents the suitable range of room temperature constraints.
[0076] In this embodiment, the gradient change values of the policy learning network parameters are obtained using the following expression:
[0077]
[0078] in, This represents the gradient changes of the policy learning network parameters when the number of training samples is N, where N is the number of training samples. For the target state variable s i The action information is a i In this case, the derivative of the value assessment network model with respect to the action parameters, For the target state variable s i In the case of i, the gradient change value of the policy learning network parameters output by the value evaluation network, where i takes the value of 1, 2, ..., N.
[0079] In this embodiment, the loss value of the value assessment network is calculated based on the action value at time t, the action value at time t+1, and the immediate reward at time t+1 using the following expression:
[0080]
[0081] Where L is the loss value of the value evaluation network, N is the number of training samples, and r t+1 For immediate feedback at time t+1, Q′(s) t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ Let Q(s) be the action value at time t+1. t a t |θ Q Let t be the value of the action at time t, and γ be the discount factor.
[0082] In some embodiments, the method further includes:
[0083] After training the reinforcement learning model, the data in the data pool is updated periodically, and the reinforcement learning model is fine-tuned using the new data.
[0084] In some embodiments, before obtaining the target state variables of the air conditioning room system, the method further includes:
[0085] The target state variables are collected within a preset time period. For each target state variable, the state variables measured at all times within the preset time period are smoothed and fitted to obtain the smoothed target state variables, which are then input into the policy learning network to obtain the action information at time t.
[0086] In some embodiments, obtaining the target state variables of the air conditioning room system, inputting them into a trained reinforcement learning model, and outputting the control parameters of the air conditioning room system includes:
[0087] The acquired target state variables are input into the policy learning network, which outputs the action information taken at time t, the target state variables at time t+1, and the immediate reward at time t+1. The action information taken at time t is then input into the value evaluation network to obtain the action value at time t.
[0088] The loss value of the value assessment network is calculated based on the action value at time t, the action value at time t+1, and the immediate return at time t+1.
[0089] If the loss value is less than the preset threshold, the action information taken at time t will be used as the control parameters of the air conditioning room system.
[0090] If the loss value exceeds a preset threshold, an early warning message will be issued.
[0091] See Figure 3 The embodiments of this disclosure also provide a control device for an air conditioning room system, comprising:
[0092] Module 11 is selected to select a preset number of state variables as target state variables from all state variables of the air conditioning room system based on the coupling relationship and correlation coefficient between different state variables of multiple devices in the air conditioning room system.
[0093] Module 12 is used to construct a reinforcement learning model based on the target state variables, wherein the reinforcement learning model includes a policy learning network and a value evaluation network;
[0094] Training module 13 is used to train the reinforcement learning model by taking the historical value of the target state variable as input and the historical control parameters of the air conditioning room system corresponding to the historical value of the target state variable as output.
[0095] Output module 14 is used to obtain the current value of the target state variable of the air conditioning room system, input it into the trained reinforcement learning model, and output the current control parameters of the air conditioning room system.
[0096] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0097] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0098] In the above embodiments, any and more of the selection module 11, construction module 12, training module 13, and output module 14 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. At least one of the selection module 11, construction module 12, training module 13, and output module 14 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the selection module 11, construction module 12, training module 13, and output module 14 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.
[0099] See Figure 4 The electronic device provided in the embodiments of this disclosure includes a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140.
[0100] Memory 1130 is used to store computer programs;
[0101] When processor 1110 executes the program stored in memory 1130, it implements the following method for controlling the air conditioning room system:
[0102] Select a preset number of target state variables from all state variables of the air conditioning room system;
[0103] A reinforcement learning model is constructed based on the target state variables, wherein the reinforcement learning model includes a policy learning network and a value evaluation network;
[0104] The reinforcement learning model is trained by taking the known target state variable as input and the control parameters of the air conditioning room system corresponding to the target state variable as output.
[0105] Obtain the target state variables of the air conditioning room system, input them into the trained reinforcement learning model, and output the control parameters of the air conditioning room system.
[0106] The aforementioned communication bus 1140 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.
[0107] The communication interface 1120 is used for communication between the above-mentioned electronic device and other devices.
[0108] The memory 1130 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1130 may also be at least one storage device located remotely from the aforementioned processor 1110.
[0109] The processor 1110 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0110] Embodiments of this disclosure also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for controlling the air conditioning room system as described above.
[0111] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement a method for controlling an air conditioning room system according to embodiments of the present disclosure.
[0112] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0113] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0114] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A control method for an air conditioning room system, characterized in that, The air conditioning room system includes multiple subsystems such as chiller units, water pump units, and cooling towers. The method includes: Based on the coupling relationships and correlation coefficients among different state variables of multiple devices in the air conditioning room system, a preset number of state variables are selected as target state variables from all state variables of the air conditioning room system; the preset number is determined by principal component analysis; the target state variable is: chilled water outlet temperature. Cooling water inlet temperature chilled water flow rate Cooling water flow rate Air conditioning system power Ambient temperature and room temperature ; A reinforcement learning model is constructed based on the target state variables, wherein the reinforcement learning model includes a policy learning network and a value evaluation network; The reinforcement learning model is trained by taking the historical values of the target state variable as input and the historical control parameters of the air conditioning room system corresponding to the historical values of the target state variable as output. Obtain the current value of the target state variable of the air conditioning room system, input it into the trained reinforcement learning model, and output the current control parameters of the air conditioning room system. The step of training the reinforcement learning model by taking the historical values of the target state variable as input and the historical control parameters of the air conditioning room system corresponding to the historical values of the target state variable as output includes: The historical values of the target state variables at time t are obtained, and the corresponding action information is determined using a greedy strategy. The action information is used as the operating parameters of the air conditioning room system to obtain the target state variable data and real-time report data at time t+1, which are then collected into the data pool. The action information includes the adjustment of the set values of the chilled water outlet temperature and the cooling water inlet temperature. The reinforcement learning model is trained by taking the historical value of the target state variable at time t, the target state variable data at time t+1, and the real-time report data in the data pool as inputs, and the action information used as the control parameters of the air conditioning room system as outputs. The process of training the reinforcement learning model by taking the historical value of the target state variable at time t, the target state variable data at time t+1, and the real-time report data from the data pool as input, and the action information used as control parameters for the air conditioning room system as output, includes: Input the historical value of the target state variable at time t in the data pool into the policy learning network, output the action information of the action taken at time t, and obtain the target state variable data at time t+1 and the real-time reward data at time t+1 corresponding to the historical value of the target state variable at time t and the action information of the action taken from the data pool. Input the action information taken at time t into the value evaluation network to obtain the action value at time t and the gradient change value of the policy learning network parameters. Input the target state variable data at time t+1 into the policy learning network, output the action information of the action to be taken at time t+1, input the action information at time t+1 into the value evaluation network, and obtain the action value at time t+1. The loss value of the value evaluation network is calculated based on the action value at time t, the action value at time t+1, and the immediate reward at time t+1. When the loss value is greater than a preset threshold, the parameters of the value evaluation network are adjusted, and the parameters of the policy learning network are updated based on the gradient change value of the policy learning network parameters until the loss value is less than the preset threshold. Instant returns are calculated using the following expression: in, The coefficient for power return value, For bias terms, These are the reported values of the cooling water inlet temperature and the chilled water outlet temperature at time t, respectively. Let be the return value of the room temperature at time t. This is the power return value. This is the suitable range for the cooling water inlet temperature. This is the suitable range for chilled water outlet temperature. This refers to the suitable range of room temperature.
2. The method according to claim 1, characterized in that, The correlation coefficient between the different state variables is calculated using the following expression: in, Two state variables The correlation coefficient between them Two state variables Covariance between State variables standard deviation State variables The standard deviation.
3. The method according to claim 1, characterized in that, The gradient changes of the network parameters for the policy learning process can be obtained using the following expression: in, This represents the gradient changes of the policy learning network parameters when the number of training samples is N. The number of training samples, For the target state variable is Action information is In this case, the derivative of the value assessment network model with respect to the action parameters, For the target state variable is In the case of i, the gradient change value of the policy learning network parameters output by the value evaluation network, where i takes the value of 1, 2, ..., N.
4. The method according to claim 1, characterized in that, The loss value of the value assessment network is calculated using the following expression, based on the action value at time t, the action value at time t+1, and the immediate reward at time t+1: in, The loss value of the valuation network. The number of training samples, Provide an immediate return at time t+1. The value of the action at time t+1. The value of the action at time t. This is the discount factor.
5. A control device for an air conditioning room system, characterized in that, The air conditioning room system includes multiple subsystems such as chiller units, water pump units, and cooling towers. The device includes: The selection module is used to select a preset number of state variables as target state variables from all state variables of the air conditioning room system based on the coupling relationships and correlation coefficients between different state variables of multiple devices in the air conditioning room system; the preset number is determined by principal component analysis; the target state variable is: chilled water outlet temperature. Cooling water inlet temperature chilled water flow rate Cooling water flow rate Air conditioning system power Ambient temperature and room temperature ; A building module is used to construct a reinforcement learning model based on target state variables, wherein the reinforcement learning model includes a policy learning network and a value evaluation network; The training module is used to train the reinforcement learning model by taking the historical values of the target state variable as input and the historical control parameters of the air conditioning room system corresponding to the historical values of the target state variable as output. The output module is used to obtain the current value of the target state variable of the air conditioning room system, input it into the trained reinforcement learning model, and output the current control parameters of the air conditioning room system. The step of training the reinforcement learning model by taking the historical values of the target state variable as input and the historical control parameters of the air conditioning room system corresponding to the historical values of the target state variable as output includes: The historical values of the target state variables at time t are obtained, and the corresponding action information is determined using a greedy strategy. The action information is used as the operating parameters of the air conditioning room system to obtain the target state variable data and real-time report data at time t+1, which are then collected into the data pool. The action information includes the adjustment of the set values of the chilled water outlet temperature and the cooling water inlet temperature. The reinforcement learning model is trained by taking the historical value of the target state variable at time t, the target state variable data at time t+1, and the real-time report data in the data pool as inputs, and the action information used as the control parameters of the air conditioning room system as outputs. The process of training the reinforcement learning model by taking the historical value of the target state variable at time t, the target state variable data at time t+1, and the real-time report data from the data pool as input, and the action information used as control parameters for the air conditioning room system as output, includes: Input the historical value of the target state variable at time t in the data pool into the policy learning network, output the action information of the action taken at time t, and obtain the target state variable data at time t+1 and the real-time reward data at time t+1 corresponding to the historical value of the target state variable at time t and the action information of the action taken from the data pool. Input the action information taken at time t into the value evaluation network to obtain the action value at time t and the gradient change value of the policy learning network parameters. Input the target state variable data at time t+1 into the policy learning network, output the action information of the action to be taken at time t+1, input the action information at time t+1 into the value evaluation network, and obtain the action value at time t+1. The loss value of the value evaluation network is calculated based on the action value at time t, the action value at time t+1, and the immediate reward at time t+1. When the loss value is greater than a preset threshold, the parameters of the value evaluation network are adjusted, and the parameters of the policy learning network are updated based on the gradient change value of the policy learning network parameters until the loss value is less than the preset threshold. Instant returns are calculated using the following expression: in, The coefficient for power return value, For bias terms, These are the reported values of the cooling water inlet temperature and the chilled water outlet temperature at time t, respectively. Let be the return value of the room temperature at time t. This is the power return value. This is the suitable range for the cooling water inlet temperature. This is the suitable range for chilled water outlet temperature. This refers to the suitable range of room temperature.
6. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the control method of the air conditioning room system according to any one of claims 1-4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the control method of the air conditioning room system according to any one of claims 1-4.