Optimal control method for cooling system of water-cooled refrigeration station based on reinforcement learning

Abstract

A method for optimizing and controlling a cooling system of a water-cooled refrigeration station based on reinforcement learning, the method comprising: establishing a mathematical model of the cooling system of the water-cooled refrigeration station, and obtaining predefined system states and equipment actions. The mathematical model is called to generate an initial control strategy according to the current system state, and initial control parameters are determined to determine the initial equipment actions. A reward function is used to calculate a reward based on the energy consumption and cooling effect fed back by the cooling system of the water-cooled refrigeration station to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the cooling system of the water-cooled refrigeration station. The estimated maximum value of each action is calculated by the Q-learning algorithm in reinforcement learning to optimize the initial control strategy based on the estimated maximum value, thereby obtaining an optimized control strategy for optimizing the cooling system of the water-cooled refrigeration station, providing stable initial operating conditions, and improving the feasibility of subsequent adjustment of the control strategy.

Description

Technical Field

[0001] The present invention belongs to the cooling technology field of a water-cooled refrigeration station, and more specifically, relates to a cooling system optimization control method for a water-cooled refrigeration station based on reinforcement learning. Background Art

[0002] The cooling system of a water-cooled refrigeration station is usually used in large buildings, industrial facilities and other places to provide stable low-temperature water for air conditioning systems or other equipment that needs cooling. Its internal structure is relatively complex, but the core components usually include the following main parts:

[0003] 1. Chiller

[0004] The chiller is the heart of the system, used to cool the circulating water to a set low temperature. The water-cooled chiller works through a vapor compression refrigeration cycle and includes the following key components:

[0005] Compressor: compresses the refrigerant gas, increasing its pressure and temperature.

[0006] Evaporator: The refrigerant evaporates here, absorbing heat from the circulating water and lowering the water's temperature.

[0007] Condenser: Usually water-cooled, the cooling water removes the heat from the refrigerant and condenses it into liquid.

[0008] Throttling device: regulates the flow and pressure of refrigerant.

[0009] 2. Cooling tower

[0010] The cooling tower is responsible for dissipating the heat taken away from the condenser into the air by evaporation. After the cooling water absorbs heat in the condenser, it flows back to the cooling tower, exchanges heat with the air in the tower, evaporates part of the water to take away the heat, and then cools and flows back to the condenser.

[0011] 3. Cooling water pump

[0012] The cooling water pump is used to maintain the circulation of cooling water between the cooling tower and the condenser. The water pump ensures the stability and speed of water flow, thereby ensuring the heat dissipation effect of the condenser.

[0013] 4. Chilled water pump

[0014] The chilled water pump is responsible for delivering the chilled water cooled by the chiller evaporator to the terminal equipment that needs refrigeration, such as the fan coil unit and air handling unit of the air conditioning system. After absorbing heat in the terminal equipment, the chilled water flows back to the chiller evaporator for cooling again, forming a closed cycle.

[0015] 5. Pipe and valve system

[0016] There is a complex network of pipes inside the cooling system that connects the various parts of the equipment. Valves are used to regulate and control the flow of water, including:

[0017] Butterfly valve, stop valve: control flow and isolate different pipe sections.

[0018] Check valve: prevents water from flowing back.

[0019] Balancing valve: ensures flow balance in each part and ensures stable operation of the system.

[0020] 6. Temperature and pressure sensors

[0021] These sensors monitor the operating status of the system in real time, ensure that the temperature and pressure of cooling water and chilled water are maintained within the set range, and adjust the operating parameters through the control system.

[0022] 7. Control system

[0023] Cooling systems are usually equipped with automated control systems to manage the start and stop of each device, the adjustment of operating status and safety protection measures. Through controllers, sensors and actuators, the control system can adjust the operating parameters of the equipment in real time to improve efficiency and reduce energy consumption.

[0024] 8. Water treatment system

[0025] In order to prevent the cooling water and chilled water from accumulating impurities, causing corrosion or scaling in the system, water treatment equipment is usually configured, including filters, softening devices or chemical treatment equipment to ensure the cleanliness and stability of the water quality.

[0026] However, the existing water-cooled refrigeration station cooling system often lacks relatively stable initial operating conditions in the initial stage of operation, resulting in a weak data basis for the subsequent control strategy adjustment process, which in turn leads to poor feasibility of the adjusted control strategy. Summary of the invention

[0027] In order to solve the deficiencies in the prior art, the purpose of the present invention is to solve the above-mentioned defects and further propose a cooling system optimization control method for a water-cooled refrigeration station based on reinforcement learning.

[0028] The present invention adopts the following technical solution.

[0029] The first aspect of the present invention discloses a method for optimizing and controlling a cooling system of a water-cooled refrigeration station based on reinforcement learning, the method comprising:

[0030] Establishing a mathematical model of the cooling system of the water-cooled refrigeration station and obtaining predefined system states and equipment actions, wherein the mathematical model takes physical parameters of the cooling system of the water-cooled refrigeration station and real-time data collected by sensors as input and takes a system state vector constructed based on the real-time data as output;

[0031] Calling the mathematical model to generate an initial control strategy according to the current system state, and determining initial control parameters to determine the initial device action;

[0032] A reward function is used to calculate a reward based on the energy consumption and cooling effect fed back by the cooling system of the water-cooled refrigeration station, so as to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the cooling system of the water-cooled refrigeration station;

[0033] Calculating the estimated maximum value of each action by using a Q-learning algorithm in reinforcement learning, optimizing the initial control strategy based on the estimated maximum value, and obtaining an optimized control strategy for optimizing the cooling system of the water-cooled refrigeration station;

[0034] Among them, the equipment action includes at least the water pump speed, the cooling tower fan speed, the compressor load and the valve opening; the system state is a set of variables when the cooling system of the water-cooled refrigeration station is in operation, including at least the temperature, flow, humidity, load and energy consumption of each component in the cooling system of the water-cooled refrigeration station.

[0035] Furthermore, the water pump speed is used to adjust the flow of chilled water and cooling water, the cooling tower fan speed is used to control the speed of air flowing through the cooling tower, the compressor load is used to control the cooling capacity by adjusting the operating frequency or the number of openings of the compressor, and the valve opening is used to adjust the resistance of the fluid flowing through the pipeline;

[0036] The expressions of the system state and device action are:

[0037] ,

[0038] ;

[0039] In the formula, and Respectively represent the quantity, Indicates the system status. Indicates device action;

[0040] The calling of the mathematical model generates an initial control strategy according to the current system state, and determines initial control parameters to determine the initial device action, including:

[0041] In the initial stage of operation of the cooling system of the water-cooled refrigeration station, based on the system state vector, the initial water pump speed, cooling tower fan speed, compressor load and valve opening are determined by the mathematical model;

[0042] Among them, the expression of the initial water pump speed is:

[0043]

[0044] In the formula, is the initial pump speed, is the cooling load of the water pump, is the flow coefficient of the water pump, and is the inherent coefficient of the water pump, Indicates the temperature difference between the inlet and outlet of chilled water.

[0045] Furthermore, the initial expression of the cooling tower fan speed is:

[0046]

[0047] In the formula, is the initial cooling tower fan speed, is the heat released by the condenser, is the heat dissipation coefficient of the cooling tower, The temperature difference between the cooling water entering and leaving the cooling tower;

[0048] The initial compressor load expression is:

[0049]

[0050] In the formula, is the initial compressor load, is the maximum cooling capacity of the compressor, is the cooling load of the water pump;

[0051] The initial expression of the valve opening is:

[0052]

[0053] In the formula, is the initial valve opening, is the expected flow rate, is the flow coefficient of the valve, is the pressure difference across the valve.

[0054] Furthermore, the reward function is used to calculate the reward according to the energy consumption and cooling effect fed back by the cooling system of the water-cooled refrigeration station to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the cooling system of the water-cooled refrigeration station, including:

[0055] The reward function of the cooling system of the water-cooled refrigeration station is constructed by introducing a penalty term. The expression is:

[0056] ;

[0057] In the formula, is the total power consumption, is the weight coefficient, is the permissible temperature deviation threshold, is the penalty coefficient.

[0058] Furthermore, the estimated maximum value of the value of each action is calculated by the Q-learning algorithm in reinforcement learning, so as to optimize the initial control strategy based on the estimated maximum value, and obtain an optimized control strategy for optimizing the cooling system of the water-cooled refrigeration station, including:

[0059] Initializing the value estimation table in the Q-learning algorithm, selecting a device action based on the initialized value estimation value through a greedy strategy, and executing the selected device action;

[0060] Among them, the expression of the device action selected by the greedy strategy is:

[0061] ;

[0062] In the formula, represents the cooling system of the water-cooled refrigeration station at time step The system state vector at time , represents any device action selected from the entire action space, Represents the time step The system state vector when executing action The value of is estimated to be the maximum when , All are device actions The elements in .

[0063] Further, the Q-learning algorithm in reinforcement learning is used to calculate the estimated maximum value of each action, so as to optimize the initial control strategy based on the estimated maximum value to obtain an optimized control strategy for optimizing the cooling system of the water-cooled refrigeration station, and further includes:

[0064] The estimated value is updated based on the reward function, and the expression for updating the estimated value is:

[0065] ,

[0066] ;

[0067] In the formula, is the maximum value in the value estimation table, To estimate the maximum value, and Both represent discount factors, between (0,1), represents the cooling system of the water-cooled refrigeration station at time step The system state vector, Represents the time step The reward function is For the replacement formula;

[0068] Determining whether the number of iterative updates of the value estimate reaches a maximum number of iterations preset in the cooling system of the water-cooled refrigeration station, and outputting the updated value estimate when the number of iterative updates reaches the maximum number of iterations; and

[0069] When the number of iterative updates does not reach the maximum number of iterations, a device action is selected based on the estimated value after the previous update through the greedy strategy, and the selected device action is executed.

[0070] Furthermore, the method further comprises:

[0071] The upper limit of the confidence interval of the device action corresponding to the initialized value estimate is calculated. The expression of the upper limit of the confidence interval is:

[0072]

[0073] In the formula, is the upper limit of the confidence interval, Representation cycle Internal device action The number of times it is set, For Action The estimated maximum value of Action for device Elements in

[0074] Taking the upper limit of the confidence interval as the capacity, judging whether the selected device action is selected according to the maximum bipartite matching, so as to obtain a set of elements of the bipartite matching, and recalculating the estimated maximum value;

[0075] The recalculated estimated maximum value is used to replace the maximum value in the value estimation table, and the expression is:

[0076]

[0077] In the formula, Represents the discount factor, between (0,1), is an indicator function. If the event in the brackets If established, ,otherwise , is the set of elements of the binary matching;

[0078] When the recalculated estimated maximum value replaces the maximum value in the value estimation table, the replacement formula K is obtained, and the expression of the replacement formula K is:

[0079] ;

[0080] In the formula, To estimate the maximum value, and Both represent discount factors, between (0,1), represents the cooling system of the water-cooled refrigeration station at time step The system state vector, Represents the time step The reward function of .

[0081] The second aspect of the present invention discloses a cooling system optimization control device for a water-cooled refrigeration station based on reinforcement learning, the device comprising:

[0082] A model building module, used to build a mathematical model of the cooling system of the water-cooled refrigeration station and obtain predefined system states and equipment actions, wherein the mathematical model takes physical parameters of the cooling system of the water-cooled refrigeration station and real-time data collected by sensors as input and takes a system state vector constructed based on the real-time data as output;

[0083] An action determination module, used to call the mathematical model to generate an initial control strategy according to the current system state, and determine initial control parameters to determine the initial device action;

[0084] An action evaluation module, used to calculate a reward according to the energy consumption and cooling effect fed back by the cooling system of the water-cooled refrigeration station through a reward function, so as to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the cooling system of the water-cooled refrigeration station;

[0085] An optimization control module, used for calculating the estimated maximum value of the value of each action by using a Q-learning algorithm in reinforcement learning, so as to optimize the initial control strategy based on the estimated maximum value, and obtain an optimization control strategy for optimizing the cooling system of the water-cooled refrigeration station;

[0086] Among them, the equipment action includes at least the water pump speed, the cooling tower fan speed, the compressor load and the valve opening; the system state is a set of variables when the cooling system of the water-cooled refrigeration station is in operation, including at least the temperature, flow, humidity, load and energy consumption of each component in the cooling system of the water-cooled refrigeration station.

[0087] The third aspect of the present invention discloses a terminal, including a processor and a storage medium; the characteristics are:

[0088] The storage medium is used to store instructions;

[0089] The processor is used to operate according to the instructions to execute the steps of the method described in the first aspect.

[0090] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0091] The beneficial effects of the present invention are that, compared with the prior art, the present invention has the following advantages:

[0092] By establishing a mathematical model of the cooling system of a water-cooled refrigeration station and obtaining the predefined system state and equipment action, the mathematical model is called to generate an initial control strategy according to the current system state, and the initial control parameters are determined to determine the initial equipment action. Then, the reward function is used to calculate the reward according to the energy consumption and cooling effect fed back by the cooling system of the water-cooled refrigeration station to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the cooling system of the water-cooled refrigeration station. Finally, the estimated maximum value of each equipment action is calculated by the Q-learning algorithm in reinforcement learning, so as to optimize the initial control strategy based on the estimated maximum value and obtain the optimal control strategy for optimizing the cooling system of the water-cooled refrigeration station. In the initial stage of system operation, this method uses experience or simple models to determine the control action according to the current system state to meet the basic operation requirements, provide a stable initial operation condition for the system, and provide a data basis for the reinforcement learning algorithm, which improves the feasibility of the subsequent adjustment of the control strategy to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 It is a flow chart of the method for optimizing control of a cooling system of a water-cooled refrigeration station based on reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0094] The present application is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present application.

[0095] like Figure 1 As shown, in one embodiment, a method for optimizing and controlling a cooling system of a water-cooled refrigeration station based on reinforcement learning comprises the following steps:

[0096] Step S110, establish a mathematical model of the cooling system of the water-cooled refrigeration station, and obtain predefined system states and equipment actions. The mathematical model takes the physical parameters of the cooling system of the water-cooled refrigeration station and the real-time data collected by sensors as input, and takes the system state vector constructed based on the real-time data as output.

[0097] Among them, the equipment action includes at least the water pump speed, cooling tower fan speed, compressor load and valve opening; the system state is a set of variables when the cooling system of the water-cooled refrigeration station is in operation, including at least the temperature, flow, humidity, load and energy consumption of each component in the cooling system of the water-cooled refrigeration station.

[0098] Step S120, calling the mathematical model to generate an initial control strategy according to the current system state, and determining initial control parameters to determine initial device actions.

[0099] In some embodiments, the present invention provides a water-cooled refrigeration station cooling system optimization control method based on reinforcement learning, the water pump speed is used to adjust the flow rate of chilled water and cooling water, the cooling tower fan speed is used to control the speed of air flowing through the cooling tower, the compressor load is used to control the cooling capacity by adjusting the operating frequency or opening number of the compressor, and the valve opening is used to adjust the resistance of the fluid flowing through the pipeline.

[0100] The expressions of system status and device action are:

[0101] ,

[0102] .

[0103] In the formula, and Respectively represent the quantity, Indicates the system status. Indicates device action.

[0104] The mathematical model is called to generate an initial control strategy according to the current system state, and the initial control parameters are determined to determine the initial device action, which specifically includes the following steps:

[0105] Step S121, in the initial stage of operation of the cooling system of the water-cooled refrigeration station, based on the system state vector, the initial water pump speed, cooling tower fan speed, compressor load and valve opening are determined through a mathematical model.

[0106] Among them, the expression of the initial pump speed is:

[0107]

[0108] In the formula, is the initial pump speed, is the cooling load of the water pump, is the flow coefficient of the water pump, and is the inherent coefficient of the water pump, Indicates the temperature difference between the inlet and outlet of chilled water.

[0109] In some embodiments, in the water-cooled refrigeration station cooling system optimization control method based on reinforcement learning provided by the present invention, the initial cooling tower fan speed is expressed as:

[0110]

[0111] In the formula, is the initial cooling tower fan speed, is the heat released by the condenser, is the heat dissipation coefficient of the cooling tower, It is the temperature difference between cooling water entering and leaving the cooling tower.

[0112] The initial compressor load is expressed as:

[0113]

[0114] In the formula, is the initial compressor load, is the maximum cooling capacity of the compressor, is the cooling load of the water pump.

[0115] The expression of the initial valve opening is:

[0116]

[0117] In the formula, is the initial valve opening, is the expected flow rate, is the flow coefficient of the valve, is the pressure difference across the valve.

[0118] Step S130, a reward is calculated based on the energy consumption and cooling effect fed back by the cooling system of the water-cooled refrigeration station through a reward function to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the cooling system of the water-cooled refrigeration station.

[0119] In some embodiments, the water-cooled refrigeration station cooling system optimization control method based on reinforcement learning provided by the present invention calculates rewards based on the energy consumption and cooling effect fed back by the water-cooled refrigeration station cooling system through a reward function to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the water-cooled refrigeration station cooling system, specifically including the following steps:

[0120] Step S131, constructing a reward function for the cooling system of the water-cooled refrigeration station by introducing a penalty term, the reward function The expression is:

[0121] .

[0122] In the formula, is the total power consumption, is the weight coefficient, is the permissible temperature deviation threshold, is the penalty coefficient.

[0123] Step S140, calculating the estimated maximum value of each action by using the Q-learning algorithm in reinforcement learning, optimizing the initial control strategy based on the estimated maximum value, and obtaining an optimized control strategy for optimizing the cooling system of the water-cooled refrigeration station.

[0124] In some embodiments, the water-cooling refrigeration station cooling system optimization control method based on reinforcement learning provided by the present invention calculates the estimated maximum value of each action through the Q-learning algorithm in reinforcement learning, optimizes the initial control strategy based on the estimated maximum value, and obtains the optimization control strategy for optimizing the cooling system of the water-cooling refrigeration station, which specifically includes the following steps:

[0125] Step S141, initializing the value estimation table in the Q-learning algorithm, selecting a device action based on the initialized value estimation value through a greedy strategy, and executing the selected device action.

[0126] Among them, the expression of the device action selected by the greedy strategy is:

[0127] .

[0128] In the formula, represents the cooling system of the water-cooled refrigeration station at time step The system state vector at time , represents any device action selected from the entire action space, Represents the time step The system state vector when executing action The value of is estimated to be the maximum when , All are device actions The elements in .

[0129] In some embodiments, the water-cooling refrigeration station cooling system optimization control method based on reinforcement learning provided by the present invention calculates the estimated maximum value of each action through the Q-learning algorithm in reinforcement learning, optimizes the initial control strategy based on the estimated maximum value, and obtains the optimization control strategy for optimizing the cooling system of the water-cooling refrigeration station, which specifically also includes the following steps:

[0130] Step S142, updating the estimated value based on the reward function, the expression for updating the estimated value is:

[0131] ,

[0132] .

[0133] In the formula, is the maximum value in the value estimate table, To estimate the maximum value, and Both represent discount factors, between (0,1), represents the cooling system of the water-cooled refrigeration station at time step The system state vector, Represents the time step The reward function is To replace the formula.

[0134] Step S143, determining whether the number of iterations of the value estimation value has reached the maximum number of iterations preset in the cooling system of the water-cooled refrigeration station, and outputting the updated value estimation value when the number of iterations of the value estimation value has reached the maximum number of iterations.

[0135] Step S144, when the number of iterative updates does not reach the maximum number of iterations, a device action is selected based on the estimated value after the previous update through a greedy strategy, and the selected device action is executed.

[0136] In some embodiments, the water-cooling refrigeration station cooling system optimization control method based on reinforcement learning provided by the present invention further includes the following steps:

[0137] Step S210, calculating the upper limit of the confidence interval of the device action corresponding to the initialized value estimate, the expression of the upper limit of the confidence interval is:

[0138]

[0139] In the formula, is the upper limit of the confidence interval, Representation cycle Internal device action The number of times it is set, For Action The estimated maximum value of Action for device The elements in .

[0140] Step S220 , taking the upper limit of the confidence interval as the capacity, judging whether the selected device action is selected according to the maximum bipartite matching, so as to obtain a set of elements of the bipartite matching, and recalculating the estimated maximum value.

[0141] The recalculated estimated maximum value is used to replace the maximum value in the value estimation table. The expression is:

[0142]

[0143] In the formula, Represents the discount factor, between (0,1), is an indicator function. If the event in the brackets If established, ,otherwise , A set of elements that are bipartite matched.

[0144] Step S230, when the recalculated estimated maximum value replaces the maximum value in the value estimation table, a replacement formula K is obtained, and the expression of the replacement formula K is:

[0145] .

[0146] In the formula, To estimate the maximum value, and Both represent discount factors, between (0,1), represents the cooling system of the water-cooled refrigeration station at time step The system state vector, Represents the time step The reward function of .

[0147] The above-mentioned optimization control method for the cooling system of a water-cooled refrigeration station based on reinforcement learning establishes a mathematical model of the cooling system of the water-cooled refrigeration station, obtains the predefined system state and device action, calls the mathematical model to generate an initial control strategy according to the current system state, and determines the initial control parameters to determine the initial device action. Then, the reward function is used to calculate the reward according to the energy consumption and cooling effect fed back by the cooling system of the water-cooled refrigeration station to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the cooling system of the water-cooled refrigeration station. Finally, the estimated maximum value of each device action is calculated by the Q-learning algorithm in reinforcement learning, so as to optimize the initial control strategy based on the estimated maximum value and obtain the optimized control strategy for optimizing the cooling system of the water-cooled refrigeration station. In the initial stage of system operation, this method uses experience or simple models to determine the control action according to the current system state to meet the basic operation requirements, provides a stable initial operation condition for the system, and provides a data basis for the reinforcement learning algorithm, which improves the feasibility of the control strategy adjusted subsequently to a certain extent.

[0148] In a specific embodiment, the present invention provides a method for optimizing and controlling a cooling system of a water-cooled refrigeration station based on reinforcement learning, comprising steps 1 to 3:

[0149] Step 1: Establish a mathematical model of the cooling system of the water-cooled refrigeration station and define the system state and equipment actions. Construct the system state vector through real-time data collection.

[0150] The input of the mathematical model is the physical parameters of the cooling system and the real-time data collected by the sensors, and the output is the system state vector, which is used to describe the current operating status of the system.

[0151] In this embodiment, the system state is a set of variables used to describe the current operating status of the system. These state variables need to be able to fully reflect the dynamic characteristics of the system, mainly including: temperature, flow, humidity, load, energy consumption, etc. of each component. Equipment action refers to the input variable that can be adjusted by the control system, which is used to affect the system state to achieve the control purpose. Mainly including: water pump speed: adjust the flow of chilled water and cooling water; cooling tower fan speed: control the speed of air flowing through the cooling tower, affecting the heat dissipation effect; compressor load: control the cooling capacity by adjusting the operating frequency or opening number of the compressor; valve opening: adjust the resistance of the fluid flowing through the pipeline; etc.

[0152] Among them, the system status Action with device They can be expressed by the following formulas:

[0153] ,

[0154] .

[0155] In the formula, and Respectively indicate quantity.

[0156] Step 2: Generate an initial control strategy and determine the initial control parameters based on the current system state.

[0157] Among them, the initial control strategy refers to the use of experience or simple models (mathematical models) to determine the control actions in the initial stage of system operation based on the current system state (i.e., system state vector) to meet the basic operating requirements of the system. The purpose is to provide a stable initial operating condition for the system and provide initial data for the reinforcement learning algorithm.

[0158] In this embodiment, the initial control strategy includes: determining the initial water pump speed, cooling tower fan speed, compressor load, valve opening, etc.

[0159] Initial pump speed The expression is:

[0160]

[0161] In the formula, is the cooling load of the water pump, is the flow coefficient of the water pump, is the inherent coefficient of the water pump, Indicates the temperature difference between the inlet and outlet of chilled water.

[0162] Cooling tower fan speed The expression is:

[0163] In the formula, is the heat released by the condenser, is the heat dissipation coefficient of the cooling tower, is the temperature difference between the cooling water entering and leaving the cooling tower,

[0164] Compressor load The expression is:

[0165]

[0166] In the formula, is the maximum cooling capacity of the compressor.

[0167] Valve opening The expression is:

[0168]

[0169] In the formula, is the expected flow rate, is the flow coefficient of the valve, is the pressure difference across the valve.

[0170] Step 3: Evaluate the value of each action and calculate the reward based on the energy consumption and cooling effect fed back by the system. The reward function aims to balance energy consumption and system performance.

[0171] Specifically, in reinforcement learning, the action-value function Indicates in status Take action After that, the cumulative expected reward that can be obtained measures the pros and cons of taking a certain action in a specific state.

[0172] Among them, the reward function mainly includes the following two key factors:

[0173] Energy consumption: including the total power consumption of equipment such as water pumps, fans, and compressors.

[0174] Cooling effect: such as whether the chilled water temperature reaches the set value, the cooling capacity of the system, etc. The cooling effect can be measured by the deviation between the chilled water outlet temperature and the set temperature. As a measure.

[0175] For the refrigeration system, the goal is usually to save energy as much as possible while meeting the temperature requirements. By introducing the penalty term, the reward function expression is:

[0176] .

[0177] In the formula, is the total power consumption, is the weight coefficient, is the permissible temperature deviation threshold, is the penalty coefficient.

[0178] Step 3: Apply the Q-learning algorithm in reinforcement learning to calculate the maximum estimated value of each action .

[0179] Specifically, it includes steps 3.1~3.4:

[0180] Step 3.1, initialize the Q value table, you can set all is 0.

[0181] Step 3.2, select and execute actions, and a greedy strategy can be used to select device actions.

[0182] .

[0183] In the formula, represents the cooling system of the water-cooled refrigeration station at time step The system state vector, It represents a specific device action selected from the entire action space. Represents the time step The system state vector is used to perform actions The value of is estimated to be the maximum value when, , Both said An element in .

[0184] Step 3.3, based on the reward function, update the Q value.

[0185] Specifically, the expression for updating the Q value is:

[0186] ,

[0187] .

[0188] In the formula, and Both represent discount factors, between (0,1), represents the cooling system of the water-cooled refrigeration station at time step The system state vector, Represents the time step The reward function is To replace the formula.

[0189] Step 3.4, based on the preset maximum number of iterations, if the maximum number of iterations has been reached, stop and output, otherwise return to step 3.2 and execute again.

[0190] The maximum value estimate obtained by calculation utilizes the existing system historical data information. However, some equipment actions with fewer settings in the historical data information should also be fully considered, so that the calculation of the action value estimate is more accurate.

[0191] Therefore, in this embodiment, steps S1 to S3 are also included:

[0192] Step S1 (performed after step 3.1) calculates the upper limit of the confidence interval of the device action.

[0193] Specifically, the upper limit of the confidence interval of the action The expression is:

[0194]

[0195] In the formula, Representation cycle Internal device action The number of times it is set, For Action The value of is estimated at maximum.

[0196] In this embodiment, by introducing a confidence interval to adjust the selection of the control action, the system can still achieve relatively efficient regulation in an uncertain environment.

[0197] Step S2, using the upper limit of the confidence interval of the action as the capacity, based on the maximum bipartite matching, determines whether the device action is selected.

[0198] Specifically, construct an action set (i.e. ) and the periodic set (i.e. ), for the elements in the action set To the elements in the periodic set The capacity is , you can choose the Edmonds-Karp algorithm to determine whether it satisfies the bipartite matching. The set of elements that meet the bipartite matching is collectively referred to as .

[0199] Step S3, recalculate to replace .

[0200] Among them, recalculate and replace the above The expression is:

[0201]

[0202] In the formula, Represents the discount factor, between (0,1), is an indicator function. If the event in the brackets If established, ,otherwise .

[0203] At this point, replace the formula , the expression is:

[0204] .

[0205] In the formula, and Both represent discount factors, between (0,1), represents the cooling system of the water-cooled refrigeration station at time step The system state vector, Represents the time step The reward function of .

[0206] The following describes the water-cooled refrigeration station cooling system optimization control device based on reinforcement learning provided by the present invention. The water-cooled refrigeration station cooling system optimization control device based on reinforcement learning described below and the water-cooled refrigeration station cooling system optimization control method based on reinforcement learning described above can refer to each other.

[0207] In one embodiment, a cooling system optimization control device for a water-cooled refrigeration station based on reinforcement learning includes a model building module, an action determination module, an action evaluation module, and an optimization control module.

[0208] The model building module is used to establish a mathematical model of the cooling system of the water-cooled refrigeration station and obtain predefined system states and equipment actions. The mathematical model takes the physical parameters of the cooling system of the water-cooled refrigeration station and the real-time data collected by sensors as input, and takes the system state vector built based on real-time data as output.

[0209] The action determination module is used to call the mathematical model to generate an initial control strategy according to the current system state and determine the initial control parameters to determine the initial device action.

[0210] The action evaluation module is used to calculate rewards based on the energy consumption and cooling effect fed back by the cooling system of the water-cooled refrigeration station through a reward function, so as to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the cooling system of the water-cooled refrigeration station.

[0211] The optimization control module is used to calculate the estimated maximum value of each action through the Q-learning algorithm in reinforcement learning, so as to optimize the initial control strategy based on the estimated maximum value and obtain the optimized control strategy for optimizing the cooling system of the water-cooled refrigeration station.

[0212] Among them, the equipment action includes at least the water pump speed, cooling tower fan speed, compressor load and valve opening; the system state is a set of variables when the cooling system of the water-cooled refrigeration station is in operation, including at least the temperature, flow, humidity, load and energy consumption of each component in the cooling system of the water-cooled refrigeration station.

[0213] In this embodiment, the present invention provides a water-cooled refrigeration station cooling system optimization control device based on reinforcement learning, the water pump speed is used to adjust the flow rate of chilled water and cooling water, the cooling tower fan speed is used to control the speed of air flowing through the cooling tower, the compressor load is used to control the cooling capacity by adjusting the operating frequency or opening number of the compressor, and the valve opening is used to adjust the resistance of the fluid flowing through the pipeline.

[0214] The expressions of system status and device action are:

[0215] ,

[0216] .

[0217] In the formula, and Respectively represent the quantity, Indicates the system status. Indicates device action.

[0218] The action determination module is specifically used for:

[0219] In the initial stage of operation of the cooling system of the water-cooled refrigeration station, the initial water pump speed, cooling tower fan speed, compressor load and valve opening are determined through a mathematical model based on the system state vector.

[0220] Among them, the expression of the initial pump speed is:

[0221]

[0222] In the formula, is the initial pump speed, is the cooling load of the water pump, is the flow coefficient of the water pump, and is the inherent coefficient of the water pump, Indicates the temperature difference between the inlet and outlet of chilled water.

[0223] In this embodiment, the water-cooled refrigeration station cooling system optimization control device based on reinforcement learning provided by the present invention has an initial cooling tower fan speed expression as follows:

[0224]

[0225] In the formula, is the initial cooling tower fan speed, is the heat released by the condenser, is the heat dissipation coefficient of the cooling tower, It is the temperature difference between cooling water entering and leaving the cooling tower.

[0226] The initial compressor load is expressed as:

[0227]

[0228] In the formula, is the initial compressor load, is the maximum cooling capacity of the compressor, is the cooling load of the water pump.

[0229] The expression of the initial valve opening is:

[0230]

[0231] In the formula, is the initial valve opening, is the expected flow rate, is the flow coefficient of the valve, is the pressure difference across the valve.

[0232] In this embodiment, the water-cooling refrigeration station cooling system optimization control device based on reinforcement learning provided by the present invention, the action evaluation module is specifically used for:

[0233] By introducing the penalty term, the reward function of the cooling system of the water-cooled refrigeration station is constructed. The expression is:

[0234] .

[0235] In the formula, is the total power consumption, is the weight coefficient, is the permissible temperature deviation threshold, is the penalty coefficient.

[0236] In this embodiment, the optimization control module of the cooling system optimization control device of a water-cooled refrigeration station based on reinforcement learning provided by the present invention is specifically used for:

[0237] The value estimation table in the Q-learning algorithm is initialized, and the device action is selected based on the initialized value estimation through a greedy strategy, and the selected device action is executed.

[0238] Among them, the expression of the device action selected by the greedy strategy is:

[0239] .

[0240] In the formula, represents the cooling system of the water-cooled refrigeration station at time step The system state vector at time , represents any device action selected from the entire action space, Represents the time step The system state vector when executing action The value of is estimated to be the maximum when , All are device actions The elements in .

[0241] In this embodiment, the optimization control module of the water-cooled refrigeration station cooling system optimization control device based on reinforcement learning provided by the present invention is specifically used for:

[0242] The estimated value is updated based on the reward function. The expression for updating the estimated value is:

[0243] ,

[0244] .

[0245] In the formula, is the maximum value in the value estimate table, To estimate the maximum value, and Both represent discount factors, between (0,1), represents the cooling system of the water-cooled refrigeration station at time step The system state vector, Represents the time step The reward function is To replace the formula.

[0246] Determine whether the number of iterations of the value estimate reaches the maximum number of iterations preset in the cooling system of the water-cooled refrigeration station, and output the updated value estimate when the number of iterations reaches the maximum number of iterations.

[0247] When the number of iterative updates does not reach the maximum number of iterations, a greedy strategy is used to select a device action based on the estimated value after the previous update, and the selected device action is executed.

[0248] In this embodiment, the water-cooling refrigeration station cooling system optimization control device based on reinforcement learning provided by the present invention further includes a formula replacement module for:

[0249] Calculate the upper limit of the confidence interval of the device action corresponding to the initialized value estimate. The expression of the upper limit of the confidence interval is:

[0250]

[0251] In the formula, is the upper limit of the confidence interval, Representation cycle Internal device action The number of times it is set, For Action The estimated maximum value of Action for device The elements in .

[0252] Taking the upper limit of the confidence interval as the capacity, determine whether the selected device action is selected according to the maximum bipartite matching to obtain the set of elements of the bipartite matching and recalculate the estimated maximum value.

[0253] The recalculated estimated maximum value is used to replace the maximum value in the value estimation table. The expression is:

[0254]

[0255] In the formula, Represents the discount factor, between (0,1), is an indicator function. If the event in the brackets If established, ,otherwise , A set of elements that are bipartite matched.

[0256] When the recalculated estimated maximum value replaces the maximum value in the value estimation table, the replacement formula K is obtained, and the expression of the replacement formula K is:

[0257] .

[0258] In the formula, To estimate the maximum value, and Both represent discount factors, between (0,1), represents the cooling system of the water-cooled refrigeration station at time step The system state vector, Represents the time step The reward function of .

[0259] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0260] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the above. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0261] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0262] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of the computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0263] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0264] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0265] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0266] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0267] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for optimizing the cooling system of a water-cooled refrigeration station based on reinforcement learning, characterized in that: The method comprises: Establishing a mathematical model of the cooling system of the water-cooled refrigeration station and obtaining predefined system states and equipment actions, wherein the mathematical model takes physical parameters of the cooling system of the water-cooled refrigeration station and real-time data collected by sensors as input and takes a system state vector constructed based on the real-time data as output; Calling the mathematical model to generate an initial control strategy according to the current system state, and determining initial control parameters to determine the initial device action; A reward function is used to calculate a reward based on the energy consumption and cooling effect fed back by the cooling system of the water-cooled refrigeration station, so as to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the cooling system of the water-cooled refrigeration station; Calculating the estimated maximum value of each action by using a Q-learning algorithm in reinforcement learning, optimizing the initial control strategy based on the estimated maximum value, and obtaining an optimized control strategy for optimizing the cooling system of the water-cooled refrigeration station; Among them, the equipment action includes at least the water pump speed, the cooling tower fan speed, the compressor load and the valve opening; the system state is a set of variables when the cooling system of the water-cooled refrigeration station is in operation, including at least the temperature, flow, humidity, load and energy consumption of each component in the cooling system of the water-cooled refrigeration station.

2. The method for optimizing the cooling system of a water-cooled refrigeration station based on reinforcement learning according to claim 1 is characterized in that: The water pump speed is used to adjust the flow of chilled water and cooling water, the cooling tower fan speed is used to control the speed of air flowing through the cooling tower, the compressor load is used to control the cooling capacity by adjusting the operating frequency or the number of openings of the compressor, and the valve opening is used to adjust the resistance of the fluid flowing through the pipeline; The expressions of the system state and device action are: ， ； In the formula, and Respectively represent the quantity, Indicates the system status. Indicates device action; The calling of the mathematical model generates an initial control strategy according to the current system state, and determines initial control parameters to determine the initial device action, including: In the initial stage of operation of the cooling system of the water-cooled refrigeration station, based on the system state vector, the initial water pump speed, cooling tower fan speed, compressor load and valve opening are determined by the mathematical model; Among them, the expression of the initial water pump speed is: In the formula, is the initial pump speed, is the cooling load of the water pump, is the flow coefficient of the water pump, and is the inherent coefficient of the water pump, Indicates the temperature difference between the inlet and outlet of chilled water.

3. The method for optimizing the cooling system of a water-cooled refrigeration station based on reinforcement learning according to claim 2, characterized in that: The initial expression of the cooling tower fan speed is: In the formula, is the initial cooling tower fan speed, is the heat released by the condenser, is the heat dissipation coefficient of the cooling tower, The temperature difference between the cooling water entering and leaving the cooling tower; The initial compressor load expression is: In the formula, is the initial compressor load, is the maximum cooling capacity of the compressor, is the cooling load of the water pump; The initial expression of the valve opening is: In the formula, is the initial valve opening, is the expected flow rate, is the flow coefficient of the valve, is the pressure difference across the valve.

4. The method for optimizing the cooling system of a water-cooled refrigeration station based on reinforcement learning according to claim 3 is characterized in that: The reward function is used to calculate the reward according to the energy consumption and cooling effect fed back by the cooling system of the water-cooled refrigeration station to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the cooling system of the water-cooled refrigeration station, including: The reward function of the cooling system of the water-cooled refrigeration station is constructed by introducing a penalty term. The expression is: ； In the formula, is the total power consumption, is the weight coefficient, is the permissible temperature deviation threshold, is the penalty coefficient.

5. The method for optimizing the cooling system of a water-cooled refrigeration station based on reinforcement learning according to claim 4 is characterized in that: The method of calculating the estimated maximum value of each action by using the Q-learning algorithm in reinforcement learning, optimizing the initial control strategy based on the estimated maximum value, and obtaining an optimized control strategy for optimizing the cooling system of the water-cooled refrigeration station, includes: Initializing the value estimation table in the Q-learning algorithm, selecting a device action based on the initialized value estimation value through a greedy strategy, and executing the selected device action; Among them, the expression of the device action selected by the greedy strategy is: ； In the formula, represents the cooling system of the water-cooled refrigeration station at time step The system state vector at time , represents any device action selected from the entire action space, Represents the time step The system state vector when executing action The value of is estimated to be the maximum when , All are device actions The elements in .

6. The method for optimizing and controlling the cooling system of a water-cooled refrigeration station based on reinforcement learning according to claim 5, characterized in that: The method further comprises: calculating the estimated maximum value of the value of each action by a Q-learning algorithm in reinforcement learning, optimizing the initial control strategy based on the estimated maximum value, and obtaining an optimized control strategy for optimizing the cooling system of the water-cooled refrigeration station. The estimated value is updated based on the reward function, and the expression for updating the estimated value is: ， ； In the formula, is the maximum value in the value estimate table, To estimate the maximum value, and Both represent discount factors, between (0,1), represents the cooling system of the water-cooled refrigeration station at time step The system state vector, Represents the time step The reward function is For the replacement formula; Determining whether the number of iterative updates of the value estimate reaches a maximum number of iterations preset in the cooling system of the water-cooled refrigeration station, and outputting the updated value estimate when the number of iterative updates reaches the maximum number of iterations; and When the number of iterative updates does not reach the maximum number of iterations, a device action is selected based on the estimated value after the previous update through the greedy strategy, and the selected device action is executed.

7. The method for optimizing the cooling system of a water-cooled refrigeration station based on reinforcement learning according to claim 6, characterized in that: The method further comprises: The upper limit of the confidence interval of the device action corresponding to the initialized value estimate is calculated. The expression of the upper limit of the confidence interval is: In the formula, is the upper limit of the confidence interval, Representation cycle Internal device action The number of times it is set, For Action The estimated maximum value of Action for device Elements in Taking the upper limit of the confidence interval as the capacity, judging whether the selected device action is selected according to the maximum bipartite matching, so as to obtain a set of elements of the bipartite matching, and recalculating the estimated maximum value; The recalculated estimated maximum value is used to replace the maximum value in the value estimation table, and the expression is: In the formula, Represents the discount factor, between (0,1), is an indicator function. If the event in the brackets If established, ,otherwise , is the set of elements of the binary matching; When the recalculated estimated maximum value replaces the maximum value in the value estimation table, the replacement formula K is obtained, and the expression of the replacement formula K is: ； In the formula, To estimate the maximum value, and Both represent discount factors, between (0,1), represents the cooling system of the water-cooled refrigeration station at time step The system state vector, Represents the time step The reward function of .

8. A cooling system optimization control device for a water-cooled refrigeration station based on reinforcement learning, characterized in that: The device comprises: A model building module, used to build a mathematical model of the cooling system of the water-cooled refrigeration station and obtain predefined system states and equipment actions, wherein the mathematical model takes physical parameters of the cooling system of the water-cooled refrigeration station and real-time data collected by sensors as input and takes a system state vector constructed based on the real-time data as output; An action determination module, used to call the mathematical model to generate an initial control strategy according to the current system state, and determine initial control parameters to determine the initial device action; An action evaluation module, used to calculate a reward according to the energy consumption and cooling effect fed back by the cooling system of the water-cooled refrigeration station through a reward function, so as to evaluate the value of each action in the initial control strategy and balance the energy consumption and cooling performance of the cooling system of the water-cooled refrigeration station; An optimization control module, used for calculating the estimated maximum value of the value of each action by using a Q-learning algorithm in reinforcement learning, so as to optimize the initial control strategy based on the estimated maximum value, and obtain an optimization control strategy for optimizing the cooling system of the water-cooled refrigeration station; Among them, the equipment action includes at least the water pump speed, the cooling tower fan speed, the compressor load and the valve opening; the system state is a set of variables when the cooling system of the water-cooled refrigeration station is in operation, including at least the temperature, flow, humidity, load and energy consumption of each component in the cooling system of the water-cooled refrigeration station.

9. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images