Automatic intelligent control method, device and equipment for liquid cooling system

Abstract

The invention discloses an automatic intelligent control method, device and equipment for a liquid cooling system, and the method comprises the steps: circularly inputting the state data of the liquid cooling system collected at a t moment to a reinforcement learning model, obtaining the optimal control parameters of the liquid cooling system, and controlling the operation of the liquid cooling system according to the optimal control parameters, when the state data of the liquid cooling system are collected at the t + 1 moment, determining a reward value of the reinforcement learning model according to the state data of the liquid cooling system at the t + 1 moment, the state data at the t moment, the optimal control parameters at the t moment, the reward value and the state data at the t + 1 moment are stored in an experience playback pool for training and updating the reinforcement learning model, and the state and the energy consumption of the liquid cooling system are synchronously optimized and adjusted in the circulating interaction process. And the reinforcement learning model is trained and updated by using empirical data, so that the liquid cooling system has an evolution type automatic intelligent control capability.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and automated control technology, and specifically to an automated intelligent control method, device and equipment for a liquid cooling system. Background Art

[0002] Liquid cooling systems are widely used to dissipate heat in high-energy-consuming and high-heat-generating equipment, such as computing centers. To ensure the heat dissipation needs of the equipment that needs heat dissipation, the water temperature entering and exiting the liquid cooling system needs to be maintained within the specified range; to ensure the normal operation of the liquid cooling system itself, the temperature and pressure of all components must also be within the specified range; to ensure the energy-saving effect of the liquid cooling system itself, the total energy consumption (power) of all components must be minimized while ensuring the first two requirements. The stable operation of the liquid cooling system is crucial to the safety of the equipment. In addition, the liquid cooling system itself consumes a lot of electricity. Therefore, ensuring the stable operation and energy-saving effect of the liquid cooling system at the same time has great social and economic benefits.

[0003] In the related technologies, control strategies such as PID control or logic control are often used to control the liquid cooling system. Although these control strategies have certain control capabilities, they do not have the learning ability to evolve synchronously with the heat generation of the equipment that needs to dissipate heat and the environment. As a result, when the heat generation of the equipment that needs to dissipate heat or the abnormal change pattern of the ambient temperature exceeds the capability of the control and adjustment strategy, these control strategies are difficult to optimize the liquid cooling system status and energy-saving effect at the same time. Summary of the invention

[0004] The present application provides a method, device and equipment for automatic intelligent control of a liquid cooling system, which can solve the problem in the related art that when the liquid cooling system is controlled and adjusted through strategies such as PID control or logic control, when the heat generation of the equipment to be dissipated or the abnormal change pattern of the ambient temperature exceeds the capability of the control and adjustment strategy, the liquid cooling system state and energy-saving effect cannot be adjusted to the optimal state at the same time.

[0005] In a first aspect, an embodiment of the present application provides an automated intelligent control method for a liquid cooling system, the automated intelligent control method for a liquid cooling system comprising:

[0006] Circularly inputting the state data of the liquid cooling system collected at time t into the reinforcement learning model to obtain the optimal control parameters of the liquid cooling system, and controlling the operation of the liquid cooling system according to the optimal control parameters;

[0007] When the state data of the liquid cooling system is collected at time t+1, the reward value of the reinforcement learning model is determined according to the state data of the liquid cooling system at time t+1, and the state data at time t, the optimal control parameters at time t, the reward value and the state data at time t+1 are stored in an experience replay pool for training and updating the reinforcement learning model.

[0008] In conjunction with the first aspect, in one embodiment:

[0009] The status data includes the measured values ​​of the temperature, water pressure, water flow and power of each component of the liquid cooling system, as well as the ambient wet bulb temperature and the ambient dry bulb temperature;

[0010] The optimal control parameters include the mean value of the optimal control parameter distribution of the heat dissipation components in the liquid cooling system, the optimal speed of the water pump and the optimal opening of the valve.

[0011] In one embodiment, controlling the operation of the liquid cooling system according to the optimal control parameters includes:

[0012] Determine, according to the optimal control parameter distribution mean and a preset distribution function relationship, the target control parameter distribution interval corresponding to when the total power consumption and total heat dissipation of the heat dissipation components in the liquid cooling system are optimal;

[0013] Determining target control parameters of each heat dissipation component in the liquid cooling system according to the optimal control parameter distribution mean and the target control parameter distribution interval;

[0014] The operation of each heat dissipation component is controlled according to the target control parameters of each heat dissipation component.

[0015] In one implementation, generating the distribution function relationship includes:

[0016] Setting a plurality of control parameter distribution means within a control parameter interval of the heat dissipation component;

[0017] For each control parameter distribution mean, adjusting different control parameter distribution intervals to control the heat dissipation component to operate according to the corresponding target control parameter, and obtaining the total power consumption and total heat dissipation of the heat dissipation component at different control parameter distribution intervals, and generating a comprehensive evaluation index, wherein the comprehensive evaluation index decreases with an increase in the total power consumption and increases with an increase in the total heat dissipation;

[0018] Taking the maximum comprehensive evaluation index as the goal, selecting a corresponding target control parameter distribution interval for each control parameter distribution mean;

[0019] The distribution function relationship is obtained by fitting the distribution means of multiple control parameters and the corresponding target control parameter distribution intervals.

[0020] In one embodiment, the state data at time t, the optimal control parameters at time t, the reward value, and the state data at time t+1 are stored in an experience replay pool for training and updating the reinforcement learning model, including:

[0021] The state data at time t, the optimal control parameters at time t, the reward value, and the state data at time t+1 are stored as one piece of experience data in a preset experience replay pool;

[0022] Experience data is randomly selected from the experience replay pool and input into the reinforcement learning model to train and update the reinforcement learning model.

[0023] In one embodiment, storing the experience data in a preset experience replay pool further includes:

[0024] Determine whether the fluctuation degree of the state data and the optimal control parameters corresponding to the current moment and N moments before the current moment is within a preset fluctuation range;

[0025] If so, the latest piece of experience data generated at the current moment is stored in the experience replay pool; otherwise, it is not stored in the experience replay pool.

[0026] In one embodiment, storing the experience data into a preset experience replay pool further includes:

[0027] If the amount of experience data in the experience replay pool is greater than a preset amount threshold, the earliest piece of experience data stored in the experience replay pool is deleted, and the latest piece of experience data is stored in the pool.

[0028] In one embodiment, randomly selecting experience data from the experience replay pool and inputting it into the reinforcement learning model to train and update the reinforcement learning model further includes:

[0029] Determining the probability of each piece of experience data being selected according to the reward value of each piece of experience data in the experience replay pool, wherein the greater the reward value, the higher the probability of the corresponding experience data being selected;

[0030] Through a preset random sampling function, based on the reward value and selection probability of each piece of experience data in the experience replay pool, as well as the number of experience data selected, experience data is selected from the experience replay pool to train and update the reinforcement learning model.

[0031] In a second aspect, an embodiment of the present application provides an automatic intelligent control device for a liquid cooling system, the automatic intelligent control device for a liquid cooling system comprising:

[0032] A calculation module, which is used to cyclically input the state data of the liquid cooling system collected at time t into the reinforcement learning model, obtain the optimal control parameters of the liquid cooling system, and control the operation of the liquid cooling system according to the optimal control parameters;

[0033] An update module is used to determine the reward value of the reinforcement learning model according to the state data of the liquid cooling system at time t+1 when the state data of the liquid cooling system is collected at time t+1, and store the state data at time t, the optimal control parameters at time t, the reward value and the state data at time t+1 into an experience replay pool for training and updating the reinforcement learning model.

[0034] In a third aspect, an embodiment of the present application provides an automated intelligent control device for a liquid cooling system, characterized in that the automated intelligent control device for a liquid cooling system comprises a processor, a memory, and an automated intelligent control program for a liquid cooling system stored in the memory and executable by the processor, wherein when the automated intelligent control program for a liquid cooling system is executed by the processor, the steps of the automated intelligent control method for a liquid cooling system as described in any one of the above items are implemented.

[0035] The embodiments of the present application provide a method, device and equipment for automated intelligent control of a liquid cooling system. The optimal control parameters of the liquid cooling system are obtained by cyclically inputting the state data of the liquid cooling system collected at time t into a reinforcement learning model, and the operation of the liquid cooling system is controlled according to the optimal control parameters. When the state data of the liquid cooling system is collected at time t+1, the reward value of the reinforcement learning model is determined according to the state data of the liquid cooling system at time t+1, and the state data at time t, the optimal control parameters at time t, the reward value and the state data at time t+1 are stored in an experience replay pool for training and updating the reinforcement learning model, thereby achieving synchronous optimization and adjustment of the state and energy consumption of the liquid cooling system, and using experience data to train and update the reinforcement learning model, so that the liquid cooling system has evolutionary automated intelligent control capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of an embodiment of the automatic intelligent control method for a liquid cooling system of the present application;

[0037] Figure 2 This is a control flow diagram of the automatic intelligent control method for the liquid cooling system of this application;

[0038] Figure 3 Schematic diagram of the cycle process of the automatic intelligent control method of the liquid cooling system in this application

[0039] Figure 4 This is a schematic diagram of the structure of the automatic intelligent control system of the liquid cooling system of this application;

[0040] Figure 5 This is a functional module diagram of an embodiment of an automatic intelligent control device for a liquid cooling system of the present application;

[0041] Figure 6 This is a schematic diagram of the hardware structure of the liquid cooling system automatic intelligent control device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0044] In a first aspect, an embodiment of the present application provides an automated intelligent control method for a liquid cooling system.

[0045] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the automatic intelligent control method for the liquid cooling system of this application. Figure 1 As shown, the automatic intelligent control method of the liquid cooling system includes:

[0046] Step S101, cyclically inputting the state data of the liquid cooling system collected at time t into the reinforcement learning model, obtaining the optimal control parameters of the liquid cooling system, and controlling the operation of the liquid cooling system according to the optimal control parameters.

[0047] Step S102: When the status data of the liquid cooling system is collected at time t+1, the reward value of the reinforcement learning model is determined according to the status data of the liquid cooling system at time t+1, and the status data at time t, the optimal control parameters at time t, the reward value and the status data at time t+1 are stored in the experience replay pool for training and updating the reinforcement learning model.

[0048] In one embodiment, the reinforcement learning model deep neural network can adopt a double deep Q network (Double Deep Q-Network or DDQN).

[0049] In one embodiment, the state data of the liquid cooling data includes the measured values ​​of the temperature, water pressure, water flow and power of each component of the liquid cooling system, as well as the ambient wet bulb temperature and the ambient dry bulb temperature; the optimal control parameters of the liquid cooling system include the optimal control parameter distribution mean of the heat dissipation component in the liquid cooling system, the optimal values ​​of the control quantities such as the optimal speed of the water pump and the optimal opening of the valve. If the structure composition or model type of the liquid cooling system is different, different state data and different control parameters can be set according to needs.

[0050] Specifically, controlling the operation of the liquid cooling system according to the optimal control parameters includes: determining the corresponding target control parameter distribution interval when the total power consumption and total heat dissipation of the heat dissipation components in the liquid cooling system are optimal according to the optimal control parameter distribution mean and a preset distribution function relationship; determining the target control parameters of each heat dissipation component in the liquid cooling system according to the optimal control parameter distribution mean and the target control parameter distribution interval; and controlling the operation of each heat dissipation component according to the target control parameters of each heat dissipation component.

[0051] For ease of understanding, the control parameter distribution mean, target control parameter distribution interval, and distribution function relationship are first explained.

[0052] In one embodiment, generating the distribution function relationship includes: setting multiple control parameter distribution means within the control parameter interval of the heat dissipation component; for each control parameter distribution mean, adjusting different control parameter distribution intervals to control the heat dissipation component to operate according to the corresponding target control parameters, and obtaining the total power consumption and total heat dissipation of the heat dissipation component at different control parameter distribution intervals, and generating a comprehensive evaluation index, wherein the comprehensive evaluation index decreases with the increase of the total power consumption and increases with the increase of the total heat dissipation; with the maximum comprehensive evaluation index as the goal, selecting the corresponding target control parameter distribution interval for each control parameter distribution mean; fitting multiple control parameter distribution means and the corresponding target control parameter distribution intervals to obtain the distribution function relationship.

[0053] Exemplarily, since the core part of the liquid cooling system is the heat dissipation zone, which is also the most power-consuming part, the control and adjustment of the heat dissipation zone is crucial to the state of the entire liquid cooling system and the heat dissipation effect. In order to expand the heat dissipation capacity of the liquid cooling system, the heat dissipation zone of the liquid cooling system can be composed of a group of identical heat dissipation components, and each heat dissipation component has an independent effect on the entire liquid cooling system. For example, the heat dissipation fan group is composed of a plurality of identical fans, and the independent effect of each fan on the liquid cooling system is equivalent. Since the number of control parameters determines the dimension of the reinforcement learning model, for example, when a heat dissipation zone has 5 heat dissipation components, a control parameter is required for each heat dissipation component, then the reinforcement learning model needs to output 5 control parameters for the heat dissipation zone. The more control parameters there are, the more data the model training process requires and the more time-consuming it is. In addition, the increase in the model dimension will increase the instability and overfitting risk of the model. Therefore, in this embodiment, a unified control parameter is designed for a heat dissipation component group in a heat dissipation zone in the reinforcement learning model.

[0054] Specifically, in a liquid cooling system, if a certain heat dissipation zone of the liquid cooling system is composed of k heat dissipation components, and its control parameters are respectively a1, a2, ..., ak, the total power consumption corresponding to the heat dissipation zone is D, and the total heat dissipation is Q. The total power consumption D can be obtained by measuring the relevant electric meters, and the total heat dissipation Q = C × V × Delta_T, where C is the specific heat of the liquid, V is the flow measurement value passing through the heat dissipation zone, and Delta_T is the degree of temperature reduction of the liquid entering and leaving the heat dissipation zone. In this embodiment, the comprehensive evaluation index of the total power consumption D and the total heat dissipation Q is set to E = G (D, Q), where the condition satisfied by the function G is that the comprehensive evaluation index E is monotonically decreasing relative to the total power consumption D, and monotonically increasing relative to the total heat dissipation Q. When the total power consumption is smaller and the total heat dissipation is larger, the comprehensive evaluation index is larger, and when the total power consumption is larger and the total heat dissipation is smaller, the comprehensive evaluation index is smaller.

[0055] In order to find the optimal control parameter combination strategy (or optimal distribution of control parameters) of the heat dissipation components in the heat dissipation zone, this embodiment adopts a simplified method, setting the control parameters of the heat dissipation component group in the heat dissipation zone as a control parameter combination, and each control parameter in the control parameter combination is a discrete distribution, which is determined by two parameters, one is the control parameter distribution mean a_mean, and the other is the control parameter distribution interval da.

[0056] Wherein, the control parameter discrete distribution mode is different, and the control parameter distribution mean a_mean and the control parameter distribution interval da have different meanings. In the present embodiment, the control parameter discrete distribution mode includes arithmetic distribution and geometric distribution. When the control parameter is arithmetic distribution, the control parameter distribution mean a_mean is the arithmetic mean of the control parameter, and the control parameter distribution interval da is the tolerance of adjacent control parameters. When the control parameter is geometric distribution, the control parameter distribution mean a_mean is the geometric mean of the control parameter, and the control parameter distribution interval da is the common ratio of adjacent control parameters.

[0057] This embodiment generates a distribution function relationship through an offline test, and the specific test method is as follows:

[0058] Taking the control parameter discrete distribution mode as an arithmetic distribution as an example, when the liquid cooling system is in a steady state, multiple control parameter distribution means are set in the control parameter interval of the heat dissipation component. Taking the heat dissipation zone including 5 heat dissipation components, the control parameter is voltage, and the voltage interval is 2.0-3.0V as an example, this embodiment can select m voltages in the interval of 2.0-3.0V as the control parameter distribution mean a_mean, recorded as (a_mean_1, a_mean2, ..., a_mean_m).

[0059] The first group of experiments is implemented, and the control parameter distribution mean a_mean is set to a predetermined value a_mean_1. In the first group of experiments, a_mean_1 is a constant value. In each experiment in the first group of experiments, the different control parameter distribution intervals da are adjusted randomly or in a custom manner, so as to control the operation of the heat dissipation component according to different arithmetic distribution target control parameters, and calculate the comprehensive evaluation index E after the time interval dt. For example, in the first group of experiments, a_mean_1 = 2.5V, the control parameter distribution interval da1 = 0.2V selected in the first experiment, then the target control parameters corresponding to the 5 heat dissipation components of the arithmetic distribution are 2.1V, 2.3V, 2.5V, 2.7V, 2.9V respectively, and the heat dissipation component is controlled to operate according to the corresponding target control parameters to obtain the corresponding comprehensive evaluation index E; the control parameter distribution interval da2 = 0.1V selected in the second experiment, then the target control parameters corresponding to the 5 heat dissipation components of the arithmetic distribution are 2.3V, 2.4V, 2.5V, 2.6V, 2.7V respectively, and the heat dissipation component is controlled to operate according to the corresponding target control parameters to obtain the corresponding comprehensive evaluation index E. The target control parameters of each heat dissipation component are within the adjustable range (2.0-3.0V).

[0060] In this way, n adjustment experiments in the first group of experiments are carried out to find the target control parameter distribution interval da corresponding to the experiment with the largest evaluation index value E, and mark it as da_1. After the first group of experiments is completed, the control parameter distribution mean and the corresponding target control parameter distribution interval array (a_mean_1, da_1) can be output.

[0061] For m control parameter distribution means (a_mean_1, a_mean 2, ..., a_mean_m), a group of experiments are conducted, and a total of m groups of experiments are conducted. For each group of experiments, the corresponding target control parameter distribution interval da_j (j=1, 2, ..., m) can be recorded. The distribution function relationship da=G(a_mean) can be fitted by a function fitting method for the m control parameter distribution means a_mean and the corresponding target control parameter distribution interval da. The fitting method can adopt existing technologies such as linear regression and polynomial fitting.

[0062] The significance of this implementation is that for a certain control parameter distribution mean a_mean, the corresponding optimal target control parameter distribution interval da can be calculated using the fitting function relationship, so that the comprehensive effect of energy saving and heat dissipation generated by the distribution is optimal.

[0063] Therefore, after the reinforcement learning model outputs the optimal control parameter distribution mean, the distribution function relationship can be used to calculate the corresponding target control parameter distribution interval da, so as to determine the target control parameters of each heat dissipation component in the liquid cooling system according to the optimal control parameter distribution mean and the target control parameter distribution interval, and then combined with the control parameters of other components in the liquid cooling system (such as the optimal speed of the water pump and the optimal opening of the valve, etc.), they are sent to the liquid cooling system for control and adjustment, so that the comprehensive effect of energy saving and heat dissipation of the liquid cooling system is optimal.

[0064] It can be understood that the beneficial effect of this method is that, in the reinforcement learning model, one control parameter is used for multiple heat dissipation components in the heat dissipation zone instead of one control parameter for each heat dissipation component, which can effectively reduce the data dimension of reinforcement learning, increase the stability of the reinforcement learning model and reduce the risk of overfitting of the model.

[0065] It is worth noting that the time t and the time t+1 in this embodiment are only used to indicate the order of two adjacent times, and the time interval between the time t and the time t+1 is dt. The time interval dt can be calibrated according to the demand. In this embodiment, the value of dt is greater than or equal to the time required for the liquid cooling system to reach a stable state after the operation of the liquid cooling system is controlled and adjusted according to the optimal control parameters.

[0066] Exemplary, such as Figure 2 and Figure 3As shown, the liquid cooling system automatic intelligent control method includes a cyclic interaction process between a reinforcement learning model and a liquid cooling system. The specific process includes: the state data S of the liquid cooling system obtained at time t is converted into t The optimal control parameters of the liquid cooling system are calculated by the reinforcement learning model, that is, action A t , and the optimal control parameter (action A t ) is sent to the liquid cooling system to regulate and control the liquid cooling system.

[0067] At time t+1, the status data S of the liquid cooling system is collected. t+1 , synchronously calculate the reward value R of the reinforcement learning model t , where the reward value R t is a reward evaluation function for the state data of the liquid cooling system. t , the optimal control parameter A at time t t , the reward value R t and S t+1 The state data at that moment is stored in the experience replay pool for training and updating the reinforcement learning model.

[0068] In one embodiment, the reward value R t According to the reward function R t =w1*F(S t+1 )+w2*L(S t+1 ) is calculated. F is the stability index function of the liquid cooling system. Specifically, the stability index function F can be the Euclidean distance function between the state data of the liquid cooling system and the preset state standard value; L is the energy consumption index function. Specifically, the energy consumption index function L can be the sum of the energy consumption of all components of the liquid cooling system; w1 is the first weight, and w2 is the second weight. In the reward function, the more stable the liquid cooling system is, the greater the reward value is, and the lower the energy consumption is, the greater the reward value is.

[0069] In one embodiment, the state data at time t, the optimal control parameters at time t, the reward value and the state data at time t+1 are stored in an experience replay pool for training and updating the reinforcement learning model, including: storing the state data at time t, the optimal control parameters at time t, the reward value and the state data at time t+1 as one piece of experience data in a preset experience replay pool; randomly selecting experience data from the experience replay pool and inputting it into the reinforcement learning model to train and update the reinforcement learning model.

[0070] For example, the state data S at time t t , the optimal control parameter A at time t t , the reward value R t and the state data S at time t+1t+1 Composed array (S t , R t , A t , S t+1 ), and the array (S t , R t , A t , S t+1 ) as S t+1 A piece of experience data corresponding to a moment is stored in the preset experience replay pool.

[0071] When the status data of the liquid cooling system is collected at each moment, the corresponding experience data can be generated as at moment t+1. t When the state data S at time t-1 is t-1 , the optimal control parameter A at time t-1 t-1 , the corresponding reward value R t-1 and the state data S at time t t Composed array (S t-1 , R t-1 , A t-1 , S t ), and the array (S t-1 , R t-1 , A t-1 , S t ) as S t A piece of experience data corresponding to a moment is stored in the preset experience replay pool.

[0072] Corresponding experience data can also be generated for other moments in turn, and the experience data generated at each moment can be put into the experience replay pool.

[0073] As a preferred implementation, storing the experience data into a preset experience replay pool also includes: determining whether the fluctuation degree of the state data and the optimal control parameters corresponding to the current moment and N moments before the current moment is within a preset fluctuation range; if so, storing the most recently generated experience data at the current moment into the experience replay pool; otherwise, not storing it in the experience replay pool.

[0074] Explanatory, since during the online operation of the liquid cooling system, the data in the fluctuation period accounts for a small proportion, and the data in the steady-state period accounts for a large proportion, the implementation method of this embodiment can make all the data in the fluctuation period be stored in the experience replay pool, and make a small amount of representative data at the beginning and end of the steady-state period be stored in the experience replay pool, that is, the proportion of experience in the fluctuation period in the experience replay pool is increased, and the proportion of experience in the steady-state period is reduced, thereby effectively preventing the problem of over-sampling of steady-state data causing prediction bias of the reinforcement learning model when selecting training data in the experience replay pool for the reinforcement learning model training update process.

[0075] In one embodiment, storing the experience data into a preset experience replay pool also includes: if the amount of experience data in the experience replay pool is greater than a preset amount threshold, deleting the earliest experience data stored in the experience replay pool and storing the latest generated experience data.

[0076] It is worth mentioning that the space of the experience replay pool is limited. When the experience replay pool can store a maximum of M experiences, if the number of experience data in the experience replay pool reaches M, a first-in-first-out storage method is adopted, that is, the earliest experience stored is removed from the pool and the latest experience is put into the pool to store experience data, thereby ensuring that the number of experiences in the experience replay pool is maintained at a maximum of M.

[0077] As a preferred implementation, experience data is randomly selected from the experience replay pool and input into the reinforcement learning model to train and update the reinforcement learning model, and also includes: determining the probability of selection of each piece of experience data according to the reward value in each piece of experience data in the experience replay pool, wherein the larger the reward value, the higher the probability of selection of the corresponding experience data; through a preset random sampling function, based on the reward value and selection probability of each piece of experience data in the experience replay pool, and the number of experience data selected, experience data is selected from the experience replay pool to train and update the reinforcement learning model.

[0078] Exemplarily, data sampling is performed in the current experience replay pool for training and updating the deep neural network of the reinforcement learning model. This embodiment uses a neural network mini-batch random sampling method to sample data from the experience replay pool as the current training data for the reinforcement learning model. In order to enable the reinforcement learning model to eventually have the ability to output actions with high reward values ​​with a higher probability, during each training, each experience in the experience replay pool is calculated according to its reward value R. The probability P of being sampled makes the experience with a larger reward value more likely to be sampled.

[0079] This embodiment uses the softmax(Rs) function to calculate the probability Ps, where Rs is an array of reward values ​​for all experience data in the resource replay pool, and Ps is an array of sampling probabilities corresponding to the reward value array Rs. During the sampling process at each time step, the sampling function sample(Rs,n,Ps) is used to obtain the mini-batch sampling set required for this training of the reinforcement learning model, where n is the number of samples in the mini-batch, and the sampling function can use a random sampling function such as numpy.random.choice. It is worth noting that the experience replay pool will be updated as the experience data is put in, so the reward value R and sampling probability P corresponding to all experiences in the pool will also be updated synchronously.

[0080] Preferably, experience data can be randomly selected from the experience replay pool at each moment and input into the reinforcement learning model to train and update the reinforcement learning model; or experience data can be randomly selected from the experience replay pool after every preset K moments and input into the reinforcement learning model to train and update the reinforcement learning model. Therefore, when the value of K is larger, the time interval for updating the reinforcement learning model is longer.

[0081] It is worth noting that after the status data of the liquid cooling system is collected at time t+1 and the reinforcement learning model is trained and updated based on the selected empirical data, the status data of the liquid cooling system collected at time t+1 will continue to be input into the reinforcement learning model to obtain the optimal control parameters corresponding to time t+1, and the operation of the liquid cooling system will be controlled according to the corresponding optimal control parameters at time t+1.

[0082] The above process is executed in a loop to continuously train and update the reinforcement learning model DDQN deep neural network, so that the model can continuously adapt to the liquid cooling system status, load and environmental changes, and keep the liquid cooling system status and energy saving effect in the optimal state at all times.

[0083] It is worth noting that before the model is put into operation, the initial model of the reinforcement learning model DDQN deep neural network can use a randomly initialized model or a model obtained by offline training. This embodiment uses the model obtained by offline training as the initial model. The offline training method is: adjust the control parameters of the system by multiple random or predetermined methods (such as using the control variable method and linear adjustment), and record the corresponding state of the system and calculate the corresponding reward. In addition, the collection interval is set to dt, and the (S t , R t , A t , S t+1 ) are collected as offline training datasets and used for offline training of the reinforcement learning model DDQN deep neural network.

[0084] In a specific embodiment, a reinforcement learning intelligent module can be integrated into the intelligent liquid cooling system based on the conventional liquid cooling system. Figure 4 As shown. The liquid cooling system automatic intelligent control method based on deep reinforcement learning involved in the present invention can be encapsulated into a reinforcement learning intelligent module. The reinforcement learning intelligent module forms a cyclic interactive relationship with the liquid cooling system. The data acquisition interval is set to dt, and the liquid cooling system state data acquisition module sends the collected state data (state S) of the liquid cooling system to the reinforcement learning intelligent module every dt time. The reinforcement learning model in the reinforcement learning intelligent module calculates the optimal control parameters (action A) in real time and sends it to the liquid cooling system control parameter configuration module. The configuration module immediately passes all control parameters to the liquid cooling system to realize control and regulation. Subsequently, the state of the liquid cooling system changes or fluctuates (stable within dt time), and after dt time, the data acquisition module sends state data to the intelligent module again, and so on. The cycle enables the intelligent module to continuously adapt to changes in system state, load and environment, and keeps the system state and energy-saving effect in the optimal state at all times.

[0085] The liquid cooling system automatic intelligent control method provided in the embodiment of the present application uses artificial intelligence reinforcement learning technology, without manual 24-hour monitoring and frequent adjustments, to automatically and intelligently realize the liquid cooling system's self-adaptation to the load and environment, ensure the stability and functionality of the liquid cooling system, and at the same time achieve the lowest energy consumption, so that the system state and energy-saving effect are always maintained in the optimal state, and the overall effect is better than manual adjustment, with social and economic benefits of safety and energy saving and emission reduction. And the reinforcement learning model is trained and updated using empirical data, so that the liquid cooling system has an evolutionary automatic intelligent control capability.

[0086] In a second aspect, an embodiment of the present application also provides an automated intelligent control device for a liquid cooling system.

[0087] In one embodiment, referring to Figure 5 , Figure 5 This is a functional module diagram of an embodiment of the liquid cooling system automatic intelligent control device of the present application. Figure 5 As shown, the liquid cooling system automatic intelligent control device includes:

[0088] A calculation module, which is used to cyclically input the state data of the liquid cooling system collected at time t into the reinforcement learning model, obtain the optimal control parameters of the liquid cooling system, and control the operation of the liquid cooling system according to the optimal control parameters;

[0089] An update module is used to determine the reward value of the reinforcement learning model according to the state data of the liquid cooling system at time t+1 when the state data of the liquid cooling system is collected at time t+1, and store the state data at time t, the optimal control parameters at time t, the reward value and the state data at time t+1 into an experience replay pool for training and updating the reinforcement learning model.

[0090] Further, in one embodiment, the status data includes measured values ​​of temperature, water pressure, water flow and power of various components of the liquid cooling system, as well as ambient wet-bulb temperature and ambient dry-bulb temperature;

[0091] The optimal control parameters include the mean value of the optimal control parameter distribution of the heat dissipation components in the liquid cooling system, the optimal speed of the water pump and the optimal opening of the valve.

[0092] Furthermore, in one embodiment, the device is also used for:

[0093] Determine, according to the optimal control parameter distribution mean and a preset distribution function relationship, the target control parameter distribution interval corresponding to when the total power consumption and total heat dissipation of the heat dissipation components in the liquid cooling system are optimal;

[0094] Determining target control parameters of each heat dissipation component in the liquid cooling system according to the optimal control parameter distribution mean and the target control parameter distribution interval;

[0095] The operation of each heat dissipation component is controlled according to the target control parameters of each heat dissipation component.

[0096] Furthermore, in one embodiment, the device is also used for:

[0097] Setting a plurality of control parameter distribution means within a control parameter interval of the heat dissipation component;

[0098] For each control parameter distribution mean, adjusting different control parameter distribution intervals to control the heat dissipation component to operate according to the corresponding target control parameter, and obtaining the total power consumption and total heat dissipation of the heat dissipation component at different control parameter distribution intervals, and generating a comprehensive evaluation index, wherein the comprehensive evaluation index decreases with an increase in the total power consumption and increases with an increase in the total heat dissipation;

[0099] Taking the maximum comprehensive evaluation index as the goal, selecting a corresponding target control parameter distribution interval for each control parameter distribution mean;

[0100] The distribution function relationship is obtained by fitting the distribution means of multiple control parameters and the corresponding target control parameter distribution intervals.

[0101] Furthermore, in one embodiment, the device also uses the state data at time t, the optimal control parameter at time t, the reward value and the state data at time t+1 as one piece of experience data and stores them in a preset experience replay pool;

[0102] Experience data is randomly selected from the experience replay pool and input into the reinforcement learning model to train and update the reinforcement learning model.

[0103] Furthermore, in one embodiment, the device is also used for:

[0104] Determine whether the fluctuation degree of the state data and the optimal control parameters corresponding to the current moment and N moments before the current moment is within a preset fluctuation range;

[0105] If so, the latest piece of experience data generated at the current moment is stored in the experience replay pool; otherwise, it is not stored in the experience replay pool.

[0106] Furthermore, in one embodiment, the device is also used for:

[0107] If the amount of experience data in the experience replay pool is greater than a preset amount threshold, the earliest piece of experience data stored in the experience replay pool is deleted, and the latest piece of experience data is stored in the pool.

[0108] Furthermore, in one embodiment, the device is also used for:

[0109] Determining the probability of each piece of experience data being selected according to the reward value of each piece of experience data in the experience replay pool, wherein the greater the reward value, the higher the probability of the corresponding experience data being selected;

[0110] Through a preset random sampling function, based on the reward value and selection probability of each piece of experience data in the experience replay pool, as well as the number of experience data selected, experience data is selected from the experience replay pool to train and update the reinforcement learning model.

[0111] Among them, the functional implementation of each module in the above-mentioned liquid cooling system automatic intelligent control device corresponds to the various steps in the above-mentioned liquid cooling system automatic intelligent control method embodiment, and its functions and implementation processes will not be repeated here one by one.

[0112] On the third aspect, an embodiment of the present application provides an automated intelligent control device for a liquid cooling system. The automated intelligent control device for a liquid cooling system may be a device with data processing capabilities, such as an intelligent control device for a liquid cooling system.

[0113] Reference Figure 6 , Figure 6The hardware structure diagram of the liquid cooling system automation intelligent control device involved in the embodiment of the present application is shown in FIG. In the embodiment of the present application, the liquid cooling system automation intelligent control device may include a processor, a memory, a communication interface, and a communication bus.

[0114] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0115] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the liquid cooling system automation intelligent control device, and the interface used to realize the interconnection between the liquid cooling system automation intelligent control device and other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.

[0116] The memory can be various types of storage media, such as random-access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0117] The processor may be a general-purpose processor, and the general-purpose processor may call the liquid cooling system automatic intelligent control program stored in the memory, and execute the liquid cooling system automatic intelligent control method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the liquid cooling system automatic intelligent control program is called may refer to the various embodiments of the liquid cooling system automatic intelligent control method of the present application, and will not be repeated here.

[0118] Those skilled in the art will understand that Figure 6 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0119] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0120] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0121] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0122] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0123] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0125] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A liquid cooling system automatic intelligent control method, characterized in that: The liquid cooling system automatic intelligent control method comprises: Circularly inputting the state data of the liquid cooling system collected at time t into the reinforcement learning model to obtain the optimal control parameters of the liquid cooling system, and controlling the operation of the liquid cooling system according to the optimal control parameters; When the state data of the liquid cooling system is collected at time t+1, the reward value of the reinforcement learning model is determined according to the state data of the liquid cooling system at time t+1, and the state data at time t, the optimal control parameters at time t, the reward value and the state data at time t+1 are stored in an experience replay pool for training and updating the reinforcement learning model.

2. The liquid cooling system automatic intelligent control method according to claim 1, characterized in that: The status data includes the measured values ​​of the temperature, water pressure, water flow and power of each component of the liquid cooling system, as well as the ambient wet bulb temperature and the ambient dry bulb temperature; The optimal control parameters include the mean value of the optimal control parameter distribution of the heat dissipation components in the liquid cooling system, the optimal speed of the water pump and the optimal opening of the valve.

3. The liquid cooling system automatic intelligent control method according to claim 2, characterized in that: Controlling the operation of the liquid cooling system according to the optimal control parameters includes: Determine, according to the optimal control parameter distribution mean and a preset distribution function relationship, the target control parameter distribution interval corresponding to when the total power consumption and total heat dissipation of the heat dissipation components in the liquid cooling system are optimal; Determining target control parameters of each heat dissipation component in the liquid cooling system according to the optimal control parameter distribution mean and the target control parameter distribution interval; The operation of each heat dissipation component is controlled according to the target control parameters of each heat dissipation component.

4. The liquid cooling system automatic intelligent control method according to claim 3, characterized in that: Generating the distribution function relationship includes: Setting a plurality of control parameter distribution means within a control parameter interval of the heat dissipation component; For each control parameter distribution mean, adjusting different control parameter distribution intervals to control the heat dissipation component to operate according to the corresponding target control parameter, and obtaining the total power consumption and total heat dissipation of the heat dissipation component at different control parameter distribution intervals, and generating a comprehensive evaluation index, wherein the comprehensive evaluation index decreases with an increase in the total power consumption and increases with an increase in the total heat dissipation; Taking the maximum comprehensive evaluation index as the goal, selecting a corresponding target control parameter distribution interval for each control parameter distribution mean; The distribution function relationship is obtained by fitting the distribution means of multiple control parameters and the corresponding target control parameter distribution intervals.

5. The liquid cooling system automatic intelligent control method according to claim 1, characterized in that: The state data at time t, the optimal control parameter at time t, the reward value and the state data at time t+1 are stored in an experience replay pool for training and updating the reinforcement learning model, including: The state data at time t, the optimal control parameters at time t, the reward value, and the state data at time t+1 are stored as one piece of experience data in a preset experience replay pool; Experience data is randomly selected from the experience replay pool and input into the reinforcement learning model to train and update the reinforcement learning model.

6. The liquid cooling system automatic intelligent control method according to claim 5, characterized in that: Storing experience data into a preset experience replay pool also includes: Determine whether the fluctuation degree of the state data and the optimal control parameters corresponding to the current moment and N moments before the current moment is within a preset fluctuation range; If yes, then the latest piece of experience data generated at the current moment is stored in the experience replay pool; Otherwise, it will not be stored in the experience replay pool.

7. The liquid cooling system automatic intelligent control method according to any one of claims 5 or 6, characterized in that: Storing experience data into a preset experience replay pool also includes: If the amount of experience data in the experience replay pool is greater than a preset amount threshold, the earliest piece of experience data stored in the experience replay pool is deleted, and the latest piece of experience data is stored in the pool.

8. The liquid cooling system automatic intelligent control method according to claim 7, characterized in that: Randomly selecting experience data from the experience replay pool and inputting it into the reinforcement learning model to train and update the reinforcement learning model, further comprising: Determining the probability of each piece of experience data being selected according to the reward value of each piece of experience data in the experience replay pool, wherein the greater the reward value, the higher the probability of the corresponding experience data being selected; Through a preset random sampling function, based on the reward value and selection probability of each piece of experience data in the experience replay pool, as well as the number of experience data selected, experience data is selected from the experience replay pool to train and update the reinforcement learning model.

9. An automatic intelligent control device for a liquid cooling system, characterized in that: The liquid cooling system automatic intelligent control device comprises: A calculation module, which is used to cyclically input the state data of the liquid cooling system collected at time t into the reinforcement learning model, obtain the optimal control parameters of the liquid cooling system, and control the operation of the liquid cooling system according to the optimal control parameters; An update module is used to determine the reward value of the reinforcement learning model according to the state data of the liquid cooling system at time t+1 when the state data of the liquid cooling system is collected at time t+1, and store the state data at time t, the optimal control parameters at time t, the reward value and the state data at time t+1 into an experience replay pool for training and updating the reinforcement learning model.

10. An automatic intelligent control device for a liquid cooling system, characterized in that: The liquid cooling system automatic intelligent control device includes a processor, a memory, and a liquid cooling system automatic intelligent control program stored in the memory and executable by the processor, wherein when the liquid cooling system automatic intelligent control program is executed by the processor, the steps of the liquid cooling system automatic intelligent control method as described in any one of claims 1 to 8 are implemented.

Images