Inner fan control method, device and equipment, medium and computer program product

Abstract

The invention discloses an inner draught fan control method and device, equipment, a medium and a computer program product, and relates to the technical field of inner draught fans, and the inner draught fan control method comprises the steps that a historical reward value obtained after an inner draught fan executes a first state adjustment action of the last time step is obtained; the first state adjustment action is an action determined based on an empirical value in a previous preset empirical value table; on the basis of the historical reward value and the environment state of the inner fan in the current time step, an empirical value in a previous preset empirical value table is updated; and based on a target empirical value corresponding to the environment state of the current time step in the updated current preset empirical value table, controlling the inner fan to execute a second state adjustment action of the current time step so as to adjust the current state parameter value of the inner fan to a target state parameter value corresponding to the target empirical value. By introducing reinforcement learning, the inner fan can continuously interact with the environment and accumulate experience, so that the optimal control scheme under different working conditions is learned, and the self-adaptive control of the state of the inner fan is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of internal fans, and in particular to an internal fan control method, device, equipment, medium and computer program product. Background Art

[0002] As the scale of data centers continues to expand and the requirements for energy efficiency and thermal environment stability continue to increase, the optimization control of the state of precision air conditioner variable frequency internal fans has attracted much attention. The state of the internal fan refers to the speed, air volume, static pressure, etc. of the internal fan.

[0003] In the process of implementing the present invention, the inventors found that the prior art has at least the following problems:

[0004] Taking the speed control of internal fans as an example, currently, the commonly used speed control methods for variable frequency internal fans include temperature feedback control, pressure difference feedback control, etc. These control methods generally adopt the PID (Proportional Integral Derivative) control principle. Although the existing technology has met the basic cooling needs of data centers to a certain extent, it generally has limitations such as low energy efficiency, large temperature fluctuations, and delayed response, and cannot effectively cope with dynamic load changes and complex environments.

[0005] In summary, how to achieve adaptive control of the internal fan status under different working conditions is a problem that needs to be solved. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide an internal fan control method, device, equipment, medium and computer program product, which can realize adaptive control of the internal fan state under different working conditions. The specific scheme is as follows:

[0007] In a first aspect, the present application discloses a method for controlling an internal fan, comprising:

[0008] Obtaining a historical reward value obtained by the indoor fan after executing the first state adjustment action of the previous time step; the first state adjustment action is an action determined based on the experience value in the previous preset experience value table;

[0009] Based on the historical reward value and the environmental state of the internal fan at the current time step, the experience value in the previous preset experience value table is updated;

[0010] Based on the updated target experience value corresponding to the environmental state of the current time step in the current preset experience value table, the internal fan is controlled to perform the second state adjustment action of the current time step to adjust the current state parameter value of the internal fan to the target state parameter value corresponding to the target experience value.

[0011] Optionally, the internal fan control method further includes:

[0012] Based on each state parameter value in the target state parameter variation range, determine the state adjustment action that can be executed by the indoor fan at the current time step to construct a target action set; the target state parameter variation range is the state parameter variation range corresponding to the environmental state of the current time step;

[0013] Acquire the experience value corresponding to each action in the target action set from the updated current preset experience value table to obtain the experience value set corresponding to the environment state at the current time step;

[0014] Based on the target exploration rate corresponding to the current time step, a target experience value is selected from the experience value set.

[0015] Optionally, the internal fan control method further includes:

[0016] The target exploration rate is obtained by linearly decaying the initial exploration rate based on the number of steps in the current time step;

[0017] Alternatively, the target exploration rate can be obtained by exponentially decaying the initial exploration rate based on the number of steps in the current time step.

[0018] Optionally, the selecting a target experience value from the experience value set based on the target exploration rate corresponding to the current time step includes:

[0019] Determining the probability of each experience value in the experience value set being selected based on the target exploration rate corresponding to the current time step;

[0020] Performing an experience value selection operation on the experience value set based on the selected probability to obtain a target experience value;

[0021] The probability of selecting the maximum experience value in the experience value set is 1-target exploration rate, and the probability of selecting the remaining experience values ​​except the maximum experience value is the target exploration rate.

[0022] Optionally, the historical reward value is a reward value calculated using a preset reward function; the preset reward function is a reward function constructed based on any one or more of the first indicator, the second indicator and the third indicator;

[0023] Among them, the first indicator is calculated based on the difference between the maximum temperature and the minimum temperature in the cold channel, and is used to characterize the temperature uniformity of the cold channel; the second indicator is calculated based on any one or several parameters of the internal fan energy consumption, the internal fan power and the internal fan cooling capacity, and is used to characterize the energy efficiency of the internal fan; the third indicator is calculated based on the temperature deviation between the actual temperature of the cold channel and the preset optimal temperature, and is used to characterize the temperature deviation of the cold channel.

[0024] Optionally, the process of constructing the preset reward function includes:

[0025] Using a preset weight coefficient to weight each indicator used to construct a reward function to obtain the preset reward function;

[0026] The sum of the weight coefficients is 1.

[0027] Optionally, if the state parameter value is the rotation speed of the internal fan, the process of controlling the internal fan further includes:

[0028] Real-time monitoring of temperature values ​​in different areas of the cold channel;

[0029] If any temperature value exceeds the preset maximum temperature, the current speed of the internal fan is increased to the preset upper limit speed;

[0030] If any temperature value is lower than the preset minimum temperature, the current speed of the internal fan is reduced to the preset lower limit speed;

[0031] Wherein, the preset upper limit speed and the preset lower limit speed are set based on the current load size of the internal fan.

[0032] Optionally, the updating of the experience value in the previous preset experience value table based on the historical reward value and the environmental state of the indoor fan at the current time step includes:

[0033] Select the maximum experience value corresponding to the environmental state of the current time step from the previous preset experience value table;

[0034] Based on the historical reward value, the maximum experience value and the target parameter, the experience value corresponding to the first state adjustment action in the previous preset experience value table is updated to obtain a current preset experience value table;

[0035] The target parameter represents the importance of the current decision to the future reward value.

[0036] In a second aspect, the present application discloses an internal fan control device, comprising:

[0037] A reward acquisition module, used to obtain a historical reward value obtained by the indoor fan after executing a first state adjustment action in the previous time step; the first state adjustment action is an action determined based on the experience value in the previous preset experience value table;

[0038] An experience updating module, used for updating the experience value in the previous preset experience value table based on the historical reward value and the environmental state of the internal fan at the current time step;

[0039] A control module is used to control the internal fan to perform a second state adjustment action at the current time step based on the target experience value corresponding to the environmental state of the current time step in the updated current preset experience value table, so as to adjust the current state parameter value of the internal fan to the target state parameter value corresponding to the target experience value.

[0040] In a third aspect, the present application discloses an electronic device, including:

[0041] Memory, used to store computer programs;

[0042] The processor is used to execute the computer program to implement the steps of the above-disclosed internal fan control method.

[0043] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed internal fan control method are implemented.

[0044] In a fifth aspect, the present application discloses a computer program product, including a computer program / instruction, which implements the steps of the aforementioned disclosed internal fan control method when executed by a processor.

[0045] It can be seen that the present application needs to obtain the historical reward value obtained by the internal fan after executing the first state adjustment action of the previous time step; the first state adjustment action is an action determined based on the experience value in the previous preset experience value table; based on the historical reward value and the environmental state of the internal fan at the current time step, the experience value in the previous preset experience value table is updated; based on the updated target experience value in the current preset experience value table corresponding to the environmental state of the current time step, the internal fan is controlled to execute the second state adjustment action of the current time step to adjust the current state parameter value of the internal fan to the target state parameter value corresponding to the target experience value.

[0046] Beneficial effect: The present application introduces a reinforcement learning algorithm into the control scheme of the internal fan to achieve adaptive control of the state of the internal fan under different working conditions. Specifically, at the beginning of the current time step, it is first necessary to obtain the historical reward value obtained by the internal fan after executing the first state adjustment action of the previous time step, wherein the first state adjustment action is specifically an action determined based on the experience value in the previous preset experience value table, and further based on the historical reward value and the environmental state of the internal fan at the current time step, the experience value in the previous preset experience value table is updated. In other words, after executing the action of the previous time step, the present application needs to correct the experience value according to the feedback value given to the internal fan by the environment, so as to help the internal fan establish its own experience and make decisions more efficiently and directionally. After updating the current preset experience value table, based on the target experience value corresponding to the environmental state of the current time step in the current preset experience value table, the internal fan is controlled to execute the second state adjustment action of the current time step to adjust the current state parameter value of the internal fan to the target state parameter value corresponding to the target experience value. That is, when executing the state adjustment action at each time step, the application selects the target experience value corresponding to the environmental state of the current time step from the updated experience value table, and then learns the latest state adjustment action to achieve adaptive adjustment of the state of the internal fan. By introducing reinforcement learning, the above scheme allows the internal fan to continuously interact with the environment, accumulate experience, and thus learn the optimal control scheme under different working conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 This is a flow chart of an internal fan control method disclosed in this application;

[0049] Figure 2 A flow chart of a specific internal fan control method disclosed in this application;

[0050] Figure 3 This is a schematic diagram of the structure of an internal fan control device disclosed in this application;

[0051] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] Taking the speed control of the internal fan as an example, currently, the commonly used control methods for the speed of the variable frequency internal fan include temperature feedback control, pressure difference feedback control, etc. These control methods generally adopt the PID control principle. Although the existing technology has met the basic cooling needs of the data center to a certain extent, it generally has limitations such as low energy efficiency, large temperature fluctuations, and delayed response, and cannot effectively cope with dynamic load changes and complex environments. To this end, the embodiments of the present application disclose an internal fan control method, device, equipment, medium and computer program product, which can realize adaptive control of the state of the internal fan under different working conditions.

[0054] See also Figure 1 As shown, the embodiment of the present application discloses a method for controlling an internal fan, the method comprising:

[0055] Step S11: obtaining a historical reward value obtained by the indoor fan after executing the first state adjustment action in the previous time step; the first state adjustment action is an action determined based on the experience value in the previous preset experience value table.

[0056] In this embodiment, at the beginning of the current time step, it is first necessary to obtain the historical reward value obtained by the internal fan after executing the first state adjustment action of the previous time step, wherein the first state adjustment action is specifically an action determined based on the experience value in the previous preset experience value table. It should be pointed out that the present application is pre-set with a preset experience value table, and the previous preset experience value table here refers to the experience value table corresponding to the previous time step. In the present application, when executing an action at each time step, it is necessary to determine the current action to be executed based on the experience value in the current latest preset experience value table. It can be understood that the internal fan determines the first state adjustment action according to the experience value in the previous preset experience value table in the previous time step, and after executing the first state adjustment action, the environment will give the internal fan certain feedback, so that the corresponding historical reward value can be calculated.

[0057] In a specific implementation, the above-mentioned state adjustment action may specifically be controlling the internal fan to change its own rotation speed.

[0058] Step S12: Based on the historical reward value and the environmental state of the indoor fan at the current time step, the experience value in the previous preset experience value table is updated.

[0059] In this embodiment, the concepts of reward value, environment, action, and experience value mentioned above are first explained:

[0060] The reward value is a quantitative feedback on the quality of the environment after the agent performs an action. It is a numerical value used to measure the contribution of the agent's control action (such as adjusting the speed of the internal fan) to achieving system goals (such as ensuring thermal environment safety, reducing energy consumption, etc.); the reward value is the guide for the agent's learning. The agent continuously tries different actions and obtains corresponding reward values, thereby learning which actions can bring higher rewards under different environmental conditions, and then adjusts its control strategy to pursue higher cumulative rewards.

[0061] The environment is the external system in which the agent is located. In this scenario, it mainly includes factors related to the thermal environment of the data center. The environment provides status information for the agent. The agent selects actions based on the environmental status and affects the environment through actions. At the same time, the environment will feedback the corresponding reward value to the agent. The change of environmental status is the basis for the agent to learn and adjust the control strategy.

[0062] An action is an operation that the agent can take in the environment. In the precision air conditioner internal fan control scenario, the action is to change the speed of the internal fan. Action is a means for the agent to influence the environment and obtain feedback. By constantly trying different actions, the agent can explore the dynamic characteristics of the environment and learn the optimal action strategy under different environmental conditions, thereby achieving effective control of the precision air conditioner internal fan.

[0063] Experience value is an indicator used to measure the value of taking a specific action under a specific environmental state. In this scenario, the experience value is recorded in the preset experience value table. Experience value is an important basis for the agent's decision-making. When choosing an action, the agent tends to choose an action with a higher experience value, because it means that taking the action under this state may obtain a higher cumulative reward. By continuously updating the experience value, the agent can gradually learn the optimal action strategy under different environmental states, thereby achieving optimal control of the fan speed in the precision air conditioner.

[0064] Therefore, in this embodiment, the experience value in the previous preset experience value table needs to be updated based on the historical reward value and the environmental state of the internal fan at the current time step. That is to say, after executing the action of the previous time step, the application needs to correct the experience value according to the feedback value given to the internal fan by the environment of the current time step, thereby updating the experience value table corresponding to the previous time step, and obtaining the current preset experience value table corresponding to the current time step, so as to help the internal fan build its own experience and make decisions more efficiently and directionally.

[0065] It should be pointed out that the present application can obtain an initial preset experience value table through historical data training in advance. By utilizing the information in the historical data, the internal fan can understand in advance the possible rewards of various actions under different environmental conditions, which helps the intelligent agent to more quickly identify higher-profit control actions in the actual operating environment and reduce unnecessary exploration time, thereby accelerating the learning process of the entire reinforcement learning controller, enabling it to adapt to the actual operating conditions of the data center more quickly and improve the practicality and effectiveness of the control method.

[0066] Specifically, historical data consists of multiple steady-state data. Therefore, when constructing the initial preset experience value table based on historical data, it is necessary to traverse each steady-state data one by one, obtain the comprehensive reward value according to the feedback / reward calculation method, and discretize the corresponding state and action, so as to update the experience value table of the reinforcement learning agent, and provide an initial experience value table with a certain basis for subsequent real-time optimization control. This can avoid the agent from making completely random action selections in the initial stage, so that the agent has a certain amount of "experience" accumulation before starting actual control, so that it can converge to a better control strategy more quickly in the subsequent learning process. In this way, in the actual control process, when encountering working conditions similar to historical data, the agent can make relatively reasonable decisions more quickly, enhance the reliability and stability of the control strategy, reduce problems such as equipment working condition oscillation that may be caused by initial random exploration, and better ensure the cooling efficiency and thermal environment stability of the data center.

[0067] In addition, it should be pointed out that the optimization cycle interval of this method can be set to 10 minutes to 30 minutes, that is, the time interval between the previous time step and the current time step is 10 minutes to 30 minutes. At the same time, the system stability judgment can be added, and the next optimization can only be performed when the system is stable. The reasons are: (1) The optimization actions of the untrained reinforcement learning optimization controller are relatively random, and a short optimization cycle may cause equipment operating conditions to fluctuate; (2) The reinforcement learning optimization controller needs to receive environmental feedback to evaluate the pros and cons of the previous round of decisions, and the computer room air conditioner needs a certain amount of time to stabilize after a round of optimization. Specifically, the consistency of the feedback values ​​of the two samples can be used to judge whether the working condition is stable. Only after it is stable can accurate feedback be given.

[0068] Step S13: Based on the updated target experience value corresponding to the environmental state of the current time step in the current preset experience value table, control the indoor fan to perform the second state adjustment action of the current time step to adjust the current state parameter value of the indoor fan to the target state parameter value corresponding to the target experience value.

[0069] In this embodiment, after the current preset experience value table is updated, based on the target experience value corresponding to the environmental state of the current time step in the current preset experience value table, the internal fan is controlled to perform the second state adjustment action of the current time step, so as to adjust the current state parameter value of the internal fan to the target state parameter value corresponding to the target experience value. That is, when the application performs the state adjustment action of each time step, the target experience value corresponding to the environmental state of the current time step is selected from the updated experience value table, and then the latest state adjustment action is learned to achieve the adaptive adjustment of the state of the internal fan.

[0070] By introducing reinforcement learning, the above scheme enables the internal fan to continuously interact with the environment and accumulate experience, thereby learning the optimal control scheme under different working conditions, and can also reduce operating energy consumption while ensuring the safety of the thermal environment.

[0071] In a specific implementation, the above method also includes: based on the value of each state parameter in the target state parameter variation range, determining the state adjustment action that can be executed by the internal fan at the current time step to construct a target action set; the target state parameter variation range is the state parameter variation range corresponding to the environmental state of the current time step; obtaining the experience value corresponding to each action in the target action set from the updated current preset experience value table to obtain the experience value set corresponding to the environmental state of the current time step; based on the target exploration rate corresponding to the current time step, selecting the target experience value from the experience value set.

[0072] First of all, it should be pointed out that the environmental state in the embodiment of the present application may specifically be the system cooling load and the supply air temperature, the state adjustment action may specifically refer to the speed adjustment action, the state parameter value refers to the specific speed value, and executing the state adjustment action is to adjust the current speed of the indoor fan to the corresponding speed value. Among them, the system cooling load can be specifically reflected by parameter indicators such as the power of the cabinet, the cooling capacity of the air conditioner, and the return air temperature. The reasons for selecting the system cooling load and the supply air temperature as the environmental state are as follows: (1) The system cooling load and the supply air temperature are not affected by the system operating state and can be regarded as environmental variables; (2) When the system cooling load and the supply air temperature remain unchanged, there is an optimal speed for the indoor fan.

[0073] In this embodiment, different state parameter variation ranges can be set for different environmental states. Taking the environmental state as cooling load and the state parameter as rotation speed as an example, when the cooling load is relatively high, the rotation speed variation range of the internal fan is set to 70 to 100, and when the cooling load is relatively low, the rotation speed variation range of the internal fan is set to 60 to 80. It should be noted that the above values ​​are only used for illustration, and in actual applications, the ranges corresponding to different cooling loads can be formulated according to certain rules.

[0074] Therefore, the present application needs to first determine the target state parameter variation range corresponding to the environmental state of the current time step, and then determine the state adjustment action that can be executed by the internal fan at the current time step based on each state parameter value in the target state parameter variation range to construct a target action set. It can be understood that after obtaining the target state parameter variation range, it is necessary to discretize with a certain step size, for example, it can be discretized with a step size of 0.5 to obtain each discrete state parameter value, and then determine the state adjustment action that can be executed by the internal fan at the current time step. Then, the experience value (i.e., Q value) corresponding to each action in the target action set is obtained from the updated current preset experience value table to obtain the experience value set corresponding to the environmental state of the current time step, and finally, based on the target exploration rate corresponding to the current time step, the target experience value is selected from the experience value set. See Table 1, which is a specific example of a preset experience value table (referred to as Q table) disclosed in the present application.

[0075] Table 1

[0076]

[0077] Among them, T represents the supply air temperature, P represents the cooling load, the combination of T and P constitutes the environmental state s, n represents the speed of the indoor fan, and a represents the action corresponding to n; in addition, the subscripts m, n, y, and x all represent the total number of values ​​of each parameter.

[0078] Further, the exploration rate usually represents the probability of the agent randomly selecting an action in the learning process selection in order to explore the unknown state space. Its main purpose is to balance the trade-off between "trying new actions" and "selecting the best known actions" in the learning process. A reasonable exploration strategy can help the agent explore the environment more effectively and find the optimal strategy. In this embodiment, the exploration rate can be set to be negatively correlated with the number of time steps; accordingly, the determination process of the target exploration rate corresponding to the current time step includes: linearly decaying the initial exploration rate based on the number of steps in the current time step to obtain the target exploration rate; or, exponentially decaying the initial exploration rate based on the number of steps in the current time step to obtain the target exploration rate. It can be understood that in a specific implementation, the present application adopts a larger exploration rate in the early stage of training and a smaller exploration rate in the later stage of training, and the exploration rate decreases linearly or exponentially with the number of learning times (that is, the number of time steps). Therefore, the present application linearly decays or exponentially decays the initial exploration rate based on the number of steps in the current time step to obtain the target exploration rate corresponding to the current time step.

[0079] The linear attenuation formula is as follows:

[0080] ;

[0081] Where T is the total number of decay steps, ϵ minis the preset minimum exploration rate, t is the current time step, ϵ0 is the initial exploration rate, and ϵ t is the target exploration rate corresponding to the current time step.

[0082] The formula for exponential decay is as follows:

[0083]

[0084] Where λ is the decay rate, ϵ min is the preset minimum exploration rate, t is the current time step, ϵ0 is the initial exploration rate, and ϵ t is the target exploration rate corresponding to the current time step.

[0085] In addition, it is also possible to maintain a fixed exploration rate throughout the learning process, and dynamically adjust the exploration rate based on the agent's performance in the environment. A common strategy is to increase or decrease the exploration rate based on the agent's performance:

[0086]

[0087] in, Is a minimum value used to adjust the exploration rate.

[0088] In a specific implementation, the target experience value is selected from the experience value set based on the target exploration rate corresponding to the current time step, including: determining the selection probability of each experience value in the experience value set based on the target exploration rate corresponding to the current time step; performing an experience value selection operation on the experience value set based on the selection probability to obtain the target experience value; wherein the selection probability of the maximum experience value in the experience value set is 1-target exploration rate, and the selection probability of the remaining experience values ​​except the maximum experience value is the target exploration rate.

[0089] That is, this application needs to determine the probability of each experience value in the experience value set being selected based on the target exploration rate corresponding to the current time step. Specifically, the target exploration rate is between 0 and 1, so the probability of the maximum experience value in the experience value set being selected is 1-target exploration rate, and the probability of the remaining experience values ​​except the maximum experience value being selected is the target exploration rate. In other words, if the target exploration rate corresponding to the current time step is 0, the maximum experience value in the experience value set is directly selected as the target experience value. If the target exploration rate corresponding to the current time step is 1, the remaining experience values ​​in the experience value set except the maximum experience value are randomly selected as the target experience value. It should be pointed out that when the exploration rate is 0, the intelligent agent will always move in the direction of the maximum Q value in the Q table. The disadvantage of this solution is the lack of exploration and may fall into a local optimum.

[0090] It can be seen that the present application introduces a reinforcement learning algorithm into the control scheme of the internal fan to achieve adaptive control of the state of the internal fan under different working conditions. Specifically, at the beginning of the current time step, it is first necessary to obtain the historical reward value obtained by the internal fan after executing the first state adjustment action of the previous time step, wherein the first state adjustment action is specifically an action determined based on the experience value in the previous preset experience value table, and further based on the historical reward value and the environmental state of the internal fan at the current time step, the experience value in the previous preset experience value table is updated. In other words, after executing the action of the previous time step, the present application needs to correct the experience value according to the feedback value given to the internal fan by the environment, so as to help the internal fan establish its own experience and make decisions more efficiently and directionally. After updating the current preset experience value table, based on the target experience value corresponding to the environmental state of the current time step in the current preset experience value table, the internal fan is controlled to execute the second state adjustment action of the current time step to adjust the current state parameter value of the internal fan to the target state parameter value corresponding to the target experience value. That is, when executing the state adjustment action at each time step, the application selects the target experience value corresponding to the environmental state of the current time step from the updated experience value table, and then learns the latest state adjustment action to achieve adaptive adjustment of the state of the internal fan. By introducing reinforcement learning, the above scheme allows the internal fan to continuously interact with the environment, accumulate experience, and thus learn the optimal control scheme under different working conditions.

[0091] See also Figure 2 As shown, the embodiment of the present application discloses a specific method for controlling an internal fan. Compared with the previous embodiment, this embodiment further illustrates and optimizes the technical solution. Specifically, it includes:

[0092] Step S21: obtaining a historical reward value obtained by the indoor fan after executing the first state adjustment action in the previous time step; the first state adjustment action is an action determined based on the experience value in the previous preset experience value table.

[0093] In this embodiment, the above-mentioned historical reward value is a reward value calculated using a preset reward function; the preset reward function is a reward function constructed based on any one or several indicators among the first indicator, the second indicator and the third indicator; wherein the first indicator is calculated based on the difference between the maximum temperature and the minimum temperature in the cold channel, and is used to characterize the temperature uniformity of the cold channel; the second indicator is calculated based on any one or several parameters among the energy consumption of the internal fan, the power of the internal fan and the cooling capacity of the internal fan, and is used to characterize the energy efficiency of the internal fan; the third indicator is calculated based on the temperature deviation between the actual temperature of the cold channel and the preset optimal temperature, and is used to characterize the temperature deviation of the cold channel.

[0094] It is understandable that this embodiment can use a single-objective reward function or a multi-objective reward function when calculating the reward value. In a specific implementation, the preset reward function is constructed based on any one or more of the first indicator, the second indicator and the third indicator.

[0095] The first indicator is calculated based on the difference between the maximum temperature and the minimum temperature in the cold channel, and is used to characterize the temperature uniformity of the cold channel, that is, the temperature extreme difference of the cold channel = temperature extreme - temperature extreme. The smaller the extreme difference, the greater the feedback value (reward value).

[0096] The second indicator is calculated based on any one or more parameters of the internal fan energy consumption, internal fan power and internal fan cooling capacity, and is used to characterize the energy efficiency of the internal fan. The higher the energy efficiency, the lower the energy consumption and the greater the feedback value (reward value).

[0097] The third indicator is calculated based on the temperature deviation between the actual temperature of the cold channel and the preset optimal temperature, which is used to characterize the temperature deviation of the cold channel. The smaller the deviation, the greater the feedback value (reward value). Since temperature sensors are arranged at multiple key locations of the cold channel to detect the temperature of each location area, in order to reduce the amount of calculation, only the temperature deviation between the maximum and minimum temperature values ​​and the corresponding preset optimal temperature can be calculated.

[0098] Specifically, the process of constructing the preset reward function includes: weighting each indicator used to construct the reward function using a preset weight coefficient to obtain the preset reward function; wherein the sum of each weight coefficient is 1.

[0099] In this way, taking the preset reward function based on the above three indicators as an example, the reward value is calculated as follows:

[0100] Reward value = first weight coefficient × first indicator + second weight coefficient × second indicator + third weight coefficient × third indicator;

[0101] Among them, the first weight coefficient + the second weight coefficient + the third weight coefficient = 1

[0102] Step S22: Select the maximum experience value corresponding to the environmental state of the current time step from the previous preset experience value table.

[0103] In this embodiment, when updating the experience value in the previous preset experience value table based on the historical reward value and the environmental state of the internal fan at the current time step, it is necessary to select the maximum experience value corresponding to the environmental state of the current time step from the previous preset experience value table.

[0104] Step S23: Based on the historical reward value, the maximum experience value and the target parameter, the experience value corresponding to the first state adjustment action in the previous preset experience value table is updated to obtain the current preset experience value table; wherein the target parameter represents the degree of importance of the current decision to the future reward value.

[0105] In this embodiment, the experience value corresponding to the first state adjustment action in the previous preset experience value table is updated based on the historical reward value, the maximum experience value and the target parameter. The specific experience value update expression is:

[0106] ;

[0107] Where s is the environmental state of the previous time step, a is the first state adjustment action of the previous time step, is the experience value corresponding to the first state adjustment action in the previous preset experience value table, is the environmental state at the current time step, The state adjustment action corresponding to the maximum experience value corresponding to the environmental state of the current time step in the previous preset experience value table, is the maximum experience value corresponding to the environmental state of the current time step in the previous preset experience value table, is the learning rate, r is the historical reward value, The target parameter is used to reflect the importance of the current decision to the future reward value.

[0108] This scheme will make the learning rate of the agent Set to 0.9 to accelerate the learning process of the reinforcement learning controller. Since this method is a real-time optimization control method, the optimization target is always only for the current working conditions, and future rewards are not given much attention. Set to 0.01. In addition, the fan frequency conversion time is very short. When the current frequency is not suitable for the real-time working conditions of the system, the fan frequency can be adjusted quickly. Therefore, the controller only needs to optimize the control action with the current target as the optimal direction, without considering the future impact. A value of 0.01 is reasonable.

[0109] It should also be pointed out that the learning rate mentioned above refers to the size of the step size each time the model parameters are updated, which determines the importance the agent attaches to new information each time it learns. A higher learning rate can make the model converge quickly, but may cause instability; while a lower learning rate may lead to slow convergence. In addition to the fixed value of 0.9 mentioned above, the learning rate can also be adjusted dynamically, such as gradual decay, exponential decay, etc.

[0110] In a specific implementation, the strategy for gradually decaying the learning rate is as follows:

[0111] ;

[0112] in, is the learning rate corresponding to the current time step, is the initial learning rate, drop_factor is the ratio of each decay, such as 0.5, and step_size is the number of steps for decay. The appropriate step_size can be determined according to the optimization cycle and time. For example, if the optimization cycle is 1 hour and you want to decay once a week, then step_size=24×30=720. The strategy of gradual decay of learning rate is suitable for scenarios with relatively stable loads in data centers.

[0113] In a specific implementation, if the performance does not improve within a certain number of learning times, the learning rate can be reduced using the following formula:

[0114] ;

[0115] The decay_factor is a number between 0 and 1. This situation is suitable for scenarios where the load in the data center changes greatly.

[0116] In a specific implementation, the learning rate may also decay exponentially as follows:

[0117] ;

[0118] in, It is the decay rate, which is used to control the speed of decay.

[0119] Step S24: Based on the updated target experience value corresponding to the environmental state of the current time step in the current preset experience value table, control the internal fan to perform the second state adjustment action of the current time step to adjust the current state parameter value of the internal fan to the target state parameter value corresponding to the target experience value.

[0120] In this embodiment, it should also be pointed out that if the state parameter value is the speed of the internal fan, then in the process of controlling the internal fan, the above method also includes: real-time monitoring of the temperature values ​​of different areas in the cold channel; if any temperature value exceeds the preset maximum temperature, the current speed of the internal fan is increased to the preset upper limit speed; if any temperature value is lower than the preset minimum temperature, the current speed of the internal fan is reduced to the preset lower limit speed; wherein the preset upper limit speed and the preset lower limit speed are set based on the current load size of the internal fan. That is, in the process of controlling the speed of the internal fan, the temperature values ​​of different areas in the cold channel will be monitored in real time. When a certain cold channel temperature sensor exceeds the set range, for example, greater than the preset maximum temperature of 27 or the preset minimum temperature is less than 18°C, the optimization control is stopped, and the speed of the internal fan is increased to the preset upper limit speed or reduced to the preset lower limit speed, thereby protecting the thermal environment safety of the server. Among them, it is necessary to reasonably set the preset upper limit speed and the preset lower limit speed of the internal fan based on the current load size of the internal fan to prevent the possibility of potential risks in the intelligent agent during the optimization control process. The higher the load, the higher the lower limit of the internal fan speed should be set.

[0121] Specifically, the lower limit speed of the internal fan should ensure that the minimum supply and return air temperature difference set by the air conditioner is met under the load P. Generally, the minimum supply and return air temperature difference of the precision air conditioner in the data center is set to 8 to 12.

[0122] ;

[0123] Among them, M is the fan air supply mass flow rate, P is the load, c is the air specific heat capacity, and Δt is the minimum air supply and return temperature difference set by the precision air conditioner. The lower limit speed of the internal fan corresponding to the load P can be determined based on the above fan air supply mass flow rate M, and the upper limit speed can simply be the lower limit speed of the internal fan plus a fixed value (such as 20), or directly use 100% speed as the upper limit speed.

[0124] For more specific processing procedures of the above steps S21 and S24, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.

[0125] It can be seen that in order to improve the cooling efficiency and thermal environment stability of the data center, this application introduces a reinforcement learning algorithm in the internal fan state control, analyzes and learns the operating conditions of the data center in real time, and realizes adaptive internal fan speed control. This method can dynamically adjust the speed of the internal fan according to load changes and environmental conditions, thereby reducing operating energy consumption while ensuring thermal environment safety.

[0126] See also Figure 3 As shown, the embodiment of the present application discloses an internal fan control device, the device comprising:

[0127] The reward acquisition module 11 is used to obtain the historical reward value obtained by the indoor fan after executing the first state adjustment action in the previous time step; the first state adjustment action is an action determined based on the experience value in the previous preset experience value table;

[0128] An experience updating module 12 is used to update the experience value in the previous preset experience value table based on the historical reward value and the environmental state of the internal fan at the current time step;

[0129] The control module 13 is used to control the indoor fan to perform the second state adjustment action of the current time step based on the target experience value corresponding to the environmental state of the current time step in the updated current preset experience value table, so as to adjust the current state parameter value of the indoor fan to the target state parameter value corresponding to the target experience value.

[0130] It can be seen that the present application introduces a reinforcement learning algorithm into the control scheme of the internal fan to achieve adaptive control of the state of the internal fan under different working conditions. Specifically, at the beginning of the current time step, it is first necessary to obtain the historical reward value obtained by the internal fan after executing the first state adjustment action of the previous time step, wherein the first state adjustment action is specifically an action determined based on the experience value in the previous preset experience value table, and further based on the historical reward value and the environmental state of the internal fan at the current time step, the experience value in the previous preset experience value table is updated. In other words, after executing the action of the previous time step, the present application needs to correct the experience value according to the feedback value given to the internal fan by the environment, so as to help the internal fan establish its own experience and make decisions more efficiently and directionally. After updating the current preset experience value table, based on the target experience value corresponding to the environmental state of the current time step in the current preset experience value table, the internal fan is controlled to execute the second state adjustment action of the current time step to adjust the current state parameter value of the internal fan to the target state parameter value corresponding to the target experience value. That is, when executing the state adjustment action at each time step, the application selects the target experience value corresponding to the environmental state of the current time step from the updated experience value table, and then learns the latest state adjustment action to achieve adaptive adjustment of the state of the internal fan. By introducing reinforcement learning, the above scheme allows the internal fan to continuously interact with the environment, accumulate experience, and thus learn the optimal control scheme under different working conditions.

[0131] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part refer to the description of the embodiments of the method part, which will not be described here. Moreover, the invention has the same beneficial effects as the above-mentioned internal fan control method.

[0132] Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the internal fan control method executed by the electronic device disclosed in any of the aforementioned embodiments.

[0133] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0134] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0135] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0136] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the internal fan control method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to data received by the electronic device and transmitted from an external device, the data 223 can also include data collected by its own input and output interface 25.

[0137] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the internal fan control method disclosed in any of the aforementioned embodiments are implemented.

[0138] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the internal fan control method disclosed in any of the aforementioned embodiments.

[0139] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0140] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0141] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0142] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0143] The above is a detailed introduction to the internal fan control method, device, equipment, medium and computer program product provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for controlling an internal fan, characterized in that: include: Obtain the historical reward value obtained by the internal fan after executing the first state adjustment action of the previous time step; The first state adjustment action is an action determined based on the experience value in the previous preset experience value table; Based on the historical reward value and the environmental state of the internal fan at the current time step, the experience value in the previous preset experience value table is updated; Based on the updated target experience value corresponding to the environmental state of the current time step in the current preset experience value table, the internal fan is controlled to perform the second state adjustment action of the current time step to adjust the current state parameter value of the internal fan to the target state parameter value corresponding to the target experience value.

2. The internal fan control method according to claim 1, characterized in that: Also includes: Based on each state parameter value in the target state parameter variation range, determine the state adjustment action that can be executed by the indoor fan at the current time step to construct a target action set; the target state parameter variation range is the state parameter variation range corresponding to the environmental state of the current time step; Acquire the experience value corresponding to each action in the target action set from the updated current preset experience value table to obtain the experience value set corresponding to the environment state at the current time step; Based on the target exploration rate corresponding to the current time step, a target experience value is selected from the experience value set.

3. The internal fan control method according to claim 2, characterized in that: Also includes: The target exploration rate is obtained by linearly decaying the initial exploration rate based on the number of steps in the current time step; Alternatively, the target exploration rate can be obtained by exponentially decaying the initial exploration rate based on the number of steps in the current time step.

4. The internal fan control method according to claim 2, characterized in that: The selecting a target experience value from the experience value set based on the target exploration rate corresponding to the current time step includes: Determining the probability of each experience value in the experience value set being selected based on the target exploration rate corresponding to the current time step; Performing an experience value selection operation on the experience value set based on the selected probability to obtain a target experience value; The probability of selecting the maximum experience value in the experience value set is 1-target exploration rate, and the probability of selecting the remaining experience values ​​except the maximum experience value is the target exploration rate.

5. The internal fan control method according to claim 1, characterized in that: The historical reward value is a reward value calculated using a preset reward function; the preset reward function is a reward function constructed based on any one or more of the first indicator, the second indicator and the third indicator; Among them, the first indicator is calculated based on the difference between the maximum temperature and the minimum temperature in the cold channel, and is used to characterize the temperature uniformity of the cold channel; the second indicator is calculated based on any one or several parameters of the internal fan energy consumption, the internal fan power and the internal fan cooling capacity, and is used to characterize the energy efficiency of the internal fan; the third indicator is calculated based on the temperature deviation between the actual temperature of the cold channel and the preset optimal temperature, and is used to characterize the temperature deviation of the cold channel.

6. The internal fan control method according to claim 5, characterized in that: The process of constructing the preset reward function includes: Using a preset weight coefficient to weight each indicator used to construct a reward function to obtain the preset reward function; The sum of the weight coefficients is 1.

7. The internal fan control method according to claim 5, characterized in that: If the state parameter value is the rotation speed of the internal fan, the process of controlling the internal fan further includes: Real-time monitoring of temperature values ​​in different areas of the cold channel; If any temperature value exceeds the preset maximum temperature, the current speed of the internal fan is increased to the preset upper limit speed; If any temperature value is lower than the preset minimum temperature, the current speed of the internal fan is reduced to the preset lower limit speed; Wherein, the preset upper limit speed and the preset lower limit speed are set based on the current load size of the internal fan.

8. The method for controlling an internal fan according to any one of claims 1 to 7, characterized in that: The updating of the experience value in the previous preset experience value table based on the historical reward value and the environmental state of the internal fan at the current time step includes: Select the maximum experience value corresponding to the environmental state of the current time step from the previous preset experience value table; Based on the historical reward value, the maximum experience value and the target parameter, the experience value corresponding to the first state adjustment action in the previous preset experience value table is updated to obtain a current preset experience value table; The target parameter represents the importance of the current decision to the future reward value.

9. An internal fan control device, characterized in that: include: A reward acquisition module, used to obtain the historical reward value obtained by the internal fan after executing the first state adjustment action of the previous time step; The first state adjustment action is an action determined based on the experience value in the previous preset experience value table; An experience updating module, used for updating the experience value in the previous preset experience value table based on the historical reward value and the environmental state of the internal fan at the current time step; A control module is used to control the internal fan to perform a second state adjustment action at the current time step based on the target experience value corresponding to the environmental state of the current time step in the updated current preset experience value table, so as to adjust the current state parameter value of the internal fan to the target state parameter value corresponding to the target experience value.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the internal fan control method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the internal fan control method as described in any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the internal fan control method described in any one of claims 1 to 8 are implemented.

Images