Equipment fan control method based on reinforcement learning, equipment and medium

Abstract

The invention discloses an equipment fan control method and equipment based on reinforcement learning and a medium, and relates to the technical field of fan control. The method comprises the steps that an action space is constructed according to operation data of a fan, a state space is constructed according to operation data of target equipment, and the operation data of the target equipment comprises working temperature; obtaining fan power, calculating rewards according to the fan power and the working temperature, and constructing a strategy network and a value network according to the state space; obtaining the current sensor reading and the rotating speed of the fan, and carrying out online training on the strategy network and the value network; after training is completed, sensor readings are collected, and fan actions with the maximum probability are executed. By means of the method, the total power consumption of the fan meeting the temperature constraint condition is minimum while it is guaranteed that the equipment temperature is controlled within the ideal range.

Description

Technical Field

[0001] The present application relates to the field of fan control technology, and in particular to a device fan control method, device and medium based on reinforcement learning. Background Art

[0002] There are two main types of fan speed control at present: for example, CN202211065196 provides an intelligent fan speed control method, device and storage medium for solving fuzzy temperature detection, divides the temperature interval according to experience, sets different fan speeds in different intervals, and finally stabilizes the temperature in the target interval; CN202410076337 provides a fan control method, system and medium based on fuzzy PID algorithm, introduces PID control, obtains one of the temperature sensor values, and uses PID fuzzy control to realize automatic adjustment of fan speed, and automatically controls the fan speed according to temperature negative feedback. Since the PID algorithm requires three parameters: conditional proportion, differential, and integral, there is also a modified parameter self-tuning PID algorithm.

[0003] In addition, the method of matching fan speeds according to temperature ranges requires a large number of experiments; PID control with self-tuning parameters can avoid a large number of thermal experiments. There are multiple sensors in the equipment to measure the temperature of different areas, and different working components require different working temperatures. Therefore, it is inappropriate to use a single temperature sensor value as the global control input.

[0004] Under the "dual carbon" requirements, the requirements for equipment power consumption are indeed getting higher and higher. On the premise of meeting the operating temperature requirements of each component in the equipment, pursuing the lowest total fan power consumption is an important energy-saving direction.

[0005] Through the above analysis, the problems and defects of the existing technology are: the fan power consumption is not considered according to the fan speed corresponding to the temperature range, the temperature control requirements of each area on the equipment are different, multiple areas are not controlled separately, and the total power consumption of the fan is not considered. Summary of the invention

[0006] The embodiments of the present application provide a device fan control method, device and medium based on reinforcement learning, which can solve the problems in the prior art that the fan power consumption is not considered according to the fan speed corresponding to the temperature range, the temperature control requirements of each area on the device are different, multiple areas are not controlled separately, and the total power consumption of the fan is not considered.

[0007] In the first aspect, an embodiment of the present application provides a device fan control method based on reinforcement learning, the method comprising: constructing an action space according to the operating data of the fan, and constructing a state space according to the operating data of the target device, wherein the operating data of the target device includes the operating temperature; obtaining the fan power, calculating the reward according to the fan power and the operating temperature, and constructing a policy network and a value network according to the state space; obtaining the current sensor reading and the fan speed, and performing online training on the policy network and the value network; after the training is completed, collecting the sensor readings and executing the fan action with the highest probability.

[0008] In one implementation of the present application, an action space is constructed based on the operating data of the fan, and a state space is constructed based on the operating data of the target device, specifically including: obtaining the adjustment type of the fan, the area where it is located, and the number of fans in each area, and constructing the action space, where the adjustment types of the fan include remaining unchanged, increasing one gear, and decreasing one gear; obtaining the operating temperature and temperature threshold of the target device, as well as the adjustment type and number of fans, and constructing the state space.

[0009] In one implementation of the present application, the fan power is obtained, the reward is calculated based on the fan power consumption and operating temperature, and the policy network and the value network are constructed based on the state space, specifically including: normalizing the operating temperature and the temperature threshold of the target device to obtain a temperature reward; normalizing the current power and the power threshold of the fan to obtain a power reward; obtaining a reward function based on the temperature reward and the power reward; obtaining a value network based on the state space; and obtaining a policy network based on the state space and the action space.

[0010] In one implementation of the present application, the current sensor reading and the fan speed are obtained, and the policy network and the value network are trained online, specifically including: obtaining the current sensor reading and the fan speed, and determining whether the trigger conditions for starting training are met, the trigger conditions including the temperature reward and the power reward being reduced to a first preset threshold; when the trigger conditions for starting training are met, the reading and the speed are input into the value network and the policy network to obtain the optimal value network and the policy network.

[0011] In one implementation of the present application, the reading and the rotation speed are input into the value network and the strategy network to obtain the optimal value network and the strategy network, specifically including: inputting the reading and the rotation speed into the strategy network to obtain the probability distribution of the fan action; performing sampling execution according to the probability distribution to generate a quintuple, the quintuple including the fan action and the corresponding probability, the sensor reading, the power reward and the temperature reward after the fan action is executed for the first preset time; adding the quintuple to the head of the experience data queue, and judging whether the experience data in the experience data queue reaches the first preset number; if the experience data does not reach the first preset number, continuing to generate quintuples; if the experience data reaches the first preset number, automatically overflowing the experience data at the tail, and extracting the second preset number of experience data, training the strategy network and the value network to obtain the optimal value network and the strategy network.

[0012] In one implementation of the present application, after training is completed, sensor readings are collected at a preset period and the fan action with the highest probability is executed, specifically including: after training is completed, an optimal strategy network is obtained, sensor readings and fan speeds are collected at a preset period; the sensor readings and fan speeds are input into the optimal strategy network to obtain the fan action probability distribution, and the fan action with the highest probability is executed.

[0013] In one implementation of the present application, before determining whether the trigger conditions for starting training are met, the method also includes: determining whether it is during the peak load period of the equipment, and determining whether there is a high temperature alarm based on the reading; if it is not during the peak load period of the equipment and there is no high temperature alarm, determining whether the trigger conditions for starting training are met; if there is a high temperature alarm, running the fan at full speed for a second preset time.

[0014] In one implementation of the present application, after training the policy network and the value network based on empirical data and taking the action with the highest probability for execution, the method also includes: regularly evaluating and verifying the policy network and the value network, and checking the reward function if the temperature reward and the power reward do not reach the second preset threshold; real-time monitoring and feedback of the fan action, and immediately issuing an early warning if the sensor reading exceeds the temperature threshold, or the current power consumption of the fan exceeds the power threshold.

[0015] In the second aspect, an embodiment of the present application also provides a device fan control device based on reinforcement learning, the device comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: construct an action space based on the operating data of the fan, and construct a state space based on the operating data of the target device, wherein the operating data of the target device includes the operating temperature; obtain the fan power, calculate the reward based on the fan power and the operating temperature, and construct a policy network and a value network based on the state space; obtain the current sensor readings and the fan speed, and perform online training on the policy network and the value network; after the training is completed, collect the sensor readings and execute the fan action with the highest probability.

[0016] On the third aspect, the embodiment of the present application also provides a non-volatile computer storage medium for device fan control based on reinforcement learning, which stores computer executable instructions, and the computer executable instructions are set to: construct an action space based on the operating data of the fan, and construct a state space based on the operating data of the target device, wherein the operating data of the target device includes the operating temperature; obtain the fan power, calculate the reward based on the fan power and the operating temperature, and construct a policy network and a value network based on the state space; obtain the current sensor reading and the fan speed, and perform online training on the policy network and the value network; after the training is completed, collect the sensor readings and execute the fan action with the highest probability.

[0017] The device fan control method, device and medium based on reinforcement learning provided by the embodiment of the present application not only consider temperature control, but also introduce power consumption as an optimization target. By constructing a reward function to combine the two, the power consumption of the fan can be reduced as much as possible while ensuring that the device temperature is controlled within the ideal range; by constructing a fine state space and strategy network, the temperature of different areas on the device can be controlled separately, which is more in line with the complex needs in practical applications; the control method based on the PPO algorithm has strong learning ability and adaptability. By continuously training the strategy network and the value network, the system can automatically adjust the control strategy of the fan according to the actual operation data to adapt to the changes in different working conditions and load conditions. Since this method can achieve fine control of the fan power consumption and temperature, it can extend the service life of the fan and reduce maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1A flowchart of a device fan control method based on reinforcement learning provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of fan placement for a device fan control method based on reinforcement learning provided in an embodiment of the present application;

[0021] Figure 3 A value network architecture diagram of a device fan control method based on reinforcement learning provided in an embodiment of the present application;

[0022] Figure 4 A strategy network architecture diagram of a device fan control method based on reinforcement learning provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of a process flow of online training of a device fan control method based on reinforcement learning provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the internal structure of a device fan control device based on reinforcement learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0026] The embodiments of the present application provide a device fan control method, device and medium based on reinforcement learning, which solves the problems in the prior art that the fan power consumption is not considered according to the fan speed corresponding to the temperature range, the temperature control requirements of each area on the device are different, multiple areas are not controlled separately, and the total power consumption of the fan is not considered.

[0027] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0028] Figure 1 A flow chart of a device fan control method based on reinforcement learning provided in an embodiment of the present application. Figure 1 As shown, the device fan control method based on reinforcement learning provided in the embodiment of the present application specifically includes the following steps:

[0029] Step 10: constructing an action space according to the operating data of the fan, and constructing a state space according to the operating data of the target device, wherein the operating data of the target device includes an operating temperature;

[0030] In the embodiment of the present application, the scheme can be divided into two steps, the first step is to construct the elements of the reinforcement learning model, and the second step is to use the reinforcement learning algorithm for training and reasoning.

[0031] As an optional embodiment, an action space is constructed according to the operation data of the fan, and a state space is constructed according to the operation data of the target device, which may specifically include: Step 101: obtaining the adjustment type, the area where the fan is located, and the number of each area, and constructing the action space, wherein the adjustment types of the fan include keeping unchanged, increasing one gear, and decreasing one gear;

[0032] In this step, for example, there are three types of fan gear adjustment: keep unchanged, increase one gear, and decrease one gear. If there are 4 fans in the target device, 3 will appear. 4 = 81 actions, and the action space grows exponentially with the number of fans.

[0033] If there are many fans, in order to reduce the action space, the fans are grouped according to the distribution of the heat generating components of the target equipment. Figure 2 For example, the target device's heat generating components are mainly on the left side, so fans 1 and 2 are divided into group 1, and fans 3 and 4 are divided into group 2. If the heat generating components in the device are evenly distributed, the odd-numbered fans can be divided into one group, and the even-numbered fans can be divided into another group. The fans in the group use the same action, so the action becomes 3. 2 =9 types, so that the fans in the equipment are controlled by partitions, reducing the action space of reinforcement learning.

[0034] Step 102: Obtain the operating temperature, temperature threshold, and fan adjustment type and number of the target device, and construct a state space.

[0035] In this step, the state space can be represented by the state vector

[0036] S=[T1,T2,..,T m ,Th1,Th2,…,Th m ,F1,F2,…,F n ], where m is the number of temperature sensors, n is the number of fans, T i Represents the reading of the ith temperature sensor, Th i Indicates the maximum operating temperature of the device where the i-th sensor is located. If it is a chip, it is given by the chip manual; if it is a temperature sensor on the motherboard, it is given by the hardware design; F i Indicates the speed of the i-th fan.

[0037] Step 20: Get the fan power, calculate the reward based on the fan power and operating temperature, and build the policy network and value network based on the state space;

[0038] In this step, we start to build the PPO (Proximal Policy Optimization) algorithm for optimization to achieve the second step, online learning and reasoning.

[0039] As an optional embodiment, the fan power is obtained, the reward is calculated according to the fan power consumption and the operating temperature, and the policy network and the value network are constructed according to the state space, which may specifically include: Step 201: normalizing the operating temperature and the temperature threshold of the target device to obtain the temperature reward;

[0040] In this step, the device temperature reward is the minimum value of the difference between all temperature sensor thresholds (temperature thresholds of the target device) and the readings, and normalized:

[0041]

[0042] Step 202: normalize the current power and the power threshold of the fan to obtain a power reward;

[0043] In this step, the power of all fans is summed and normalized:

[0044]

[0045] Among them, p i Indicates the current power reading of the i-th fan, is the maximum speed power of the i-th fan, which can be given in the fan product manual.

[0046] Step 203: Obtain a reward function according to the temperature reward and the power reward;

[0047] In this step, the reward is equal to the current device operating temperature reward R t Subtract the normalized fan power P to get the reward function: R = R t -P.

[0048] It is understandable that this calculation process reflects the trade-off between pursuing the suitability of the equipment operating temperature and considering the fan energy consumption. If the equipment operating temperature is very low (i.e. R t is very high), but the fan power is also high (i.e. P is large), then the final reward may be reduced because high fan power may mean unnecessary energy consumption and noise.

[0049] Step 204: Obtain a value network according to the state space;

[0050] like Figure 3 As shown, the input state S uses a two-layer fully connected neural network to output the state value V, which is responsible for evaluating the long-term value that these actions may bring. The value network uses Indicates that is the network parameter to be learned, initialized to a random value in the [0,1] interval.

[0051] Step 205: Obtain a policy network according to the state space and the action space.

[0052] In this step, the policy network is a deep neural network used to model the agent's strategy, that is, the probability distribution of the actions that can be output in a given state. Figure 4 As shown, the input is state S. According to the state space construction method, a two-layer fully connected neural network is used to output the probability distribution of the action, for example, the 9 actions mentioned in step 101, each action has a corresponding probability.

[0053] Step 30: Get the current sensor reading and fan speed, and perform online training on the policy network and value network;

[0054] As an optional embodiment, obtaining the current sensor reading and the speed of the fan, and performing online training on the policy network and the value network may specifically include: Step 301: obtaining the current sensor reading and the speed of the fan, and determining whether a trigger condition for starting training is met, the trigger condition including that the temperature reward and the power reward are reduced to a first preset threshold;

[0055] That is to say, based on the current state space and action space, we get an initial value network and policy network, and then train and update them, and finally achieve the action of automatically adjusting the fan according to the current temperature. Figure 5 .

[0056] In this step, the target device can be configured to turn on or off training, and manually determine whether to stop; it can also be automatically turned on or off according to "turn off training if the reward does not increase significantly, or turn on training rules if the reward decreases significantly".

[0057] Step 302: When the triggering conditions for starting training are met, the reading and the rotation speed are input into the value network and the strategy network to obtain the optimal value network and strategy network.

[0058] As an optional embodiment, inputting the reading and the speed into the value network and the strategy network to obtain the optimal value network and the strategy network may specifically include: Step 3021: inputting the reading and the speed into the strategy network to obtain the probability distribution of the fan action;

[0059] Step 3022: Perform sampling according to the probability distribution to generate a quintuple, the quintuple including the fan action and the corresponding probability, the sensor reading, the power reward and the temperature reward after the fan action is performed for the first preset time;

[0060] In this step, the policy network Input the current state, output the action probability distribution, sample the action execution according to the probability size, and record the five-tuple: current state, sampled action, action probability, reward after executing the action for 2 minutes, and state after executing the action for 2 minutes.

[0061] Step 3023: adding the quintuple to the head of the experience data queue, and determining whether the experience data in the experience data queue reaches a first preset number;

[0062] In this step, the five-tuple is added to the head of the data queue. For example, if the queue length is 20, the queue automatically overflows the tail data.

[0063] Step 3024: If the amount of experience data does not reach the first preset number, continue to generate quintuples;

[0064] In this step, if the queue data is not enough for a round of training, continue sampling to generate quintuples.

[0065] Step 3025: When the experience data reaches a first preset amount, the experience data at the tail is automatically overflowed, and a second preset amount of experience data is extracted to train the strategy network and the value network to obtain the optimal value network and strategy network.

[0066] In this step, k quintuples are randomly drawn from the queue. The recommended number of quintuples is the queue length. For example, when the queue length is 20, 5 quintuples are extracted.

[0067] In the specific implementation, the extracted five-tuple is used to optimize the policy network and value network parameters once, and each five-tuple is calculated Advantages of action: Policy network loss function definition: Here, the proximal strategy is used to optimize clipping. Clip is a truncation function. Restricted to (1-ε, 1+ε), the hyperparameter ε takes the value of 0.1 or 0.2. θ (a i |S i ) is the policy network π θ The output action probability, θ is the network parameter to be optimized, is the action probability recorded in the five-tuple, does not participate in the gradient calculation, calculates the loss function gradient and optimizes the policy network parameters: Where α is a hyperparameter, the recommended value is 1e-3, and the value network loss function is defined as: is a scalar and does not participate in gradient calculation. It is a value network Status The value output of It is to optimize network parameters, calculate the loss function gradient and optimize the value network parameters: Where β is a training hyperparameter, and the recommended value is 1e-3.

[0068] Step 40: After the training is completed, the sensor readings are collected and the fan action with the highest probability is executed. As an optional embodiment, after the training is completed, the sensor readings are collected according to a preset period and the fan action with the highest probability is executed, which may specifically include: Step 401: After the training is completed, the optimal strategy network is obtained, and the sensor readings and fan speed are collected according to a preset period;

[0069] Step 402: Input the sensor reading and the fan speed into the optimal strategy network to obtain the fan action probability distribution, and execute the fan action with the highest probability.

[0070] In this step, the current policy network and value network parameters are saved, and the network parameters marked with old do not participate in the network parameter optimization; the temperature and fan speed can be periodically collected to form the state data S, input into the policy network, output the action probability distribution, and take the action with the maximum probability to execute, that is:

[0071]

[0072] As an optional embodiment, before determining whether the triggering condition for starting training is met, the method may further include: determining whether it is during the peak load period of the equipment, and determining whether there is a high temperature alarm based on the reading;

[0073] In this step, when determining whether the triggering conditions for starting training are met, system stability factors are also considered, such as avoiding starting training during peak equipment load or during critical task execution.

[0074] If the equipment is not at its peak load period and there is no high temperature alarm, determine whether the triggering conditions for starting training are met; if there is a high temperature alarm, make the fan run at full speed for the second preset time.

[0075] In this step, in order to prevent the device from being damaged by high temperature, a high temperature alarm is determined before each sampling action is executed. If a high temperature alarm exists, the fan runs at full speed for 2 minutes before attempting to execute the sampling action.

[0076] As an optional embodiment, after training the policy network and the value network based on empirical data and taking the action with the highest probability for execution, the method may also include: Step 50: Regularly evaluate and verify the policy network and the value network, and if the temperature reward and the power reward do not reach the second preset threshold, check the reward function; Step 60: Real-time monitoring and feedback of the fan action, if the sensor reading exceeds the temperature threshold, or the current power consumption of the fan exceeds the power threshold, immediately issue an early warning.

[0077] It can be understood that the specific numbers in the "9 actions", "fan running at full speed for 2 minutes", "queue length 20", "execute action for 2 minutes" and the hyperparameters k, N, ε, α, and β in the embodiments of the present application can be adjusted according to actual conditions.

[0078] In addition, multiple elements such as humidity and fan noise can be added to the reward function, all of which can be determined based on actual operations.

[0079] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, the embodiment of this application also provides a device fan control device based on reinforcement learning, whose structure is as follows: Figure 6 shown.

[0080] Figure 6 A schematic diagram of the internal structure of a device fan control device based on reinforcement learning provided in an embodiment of the present application. Figure 6 As shown, the device includes:

[0081] at least one processor 201;

[0082] and, a memory 202 communicatively connected to the at least one processor;

[0083] Among them, the memory 202 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 201 to enable at least one processor 201 to: construct an action space based on the operating data of the fan, and construct a state space based on the operating data of the target device, wherein the operating data of the target device includes the operating temperature; obtain the fan power, calculate the reward based on the fan power and the operating temperature, and construct a policy network and a value network based on the state space; obtain the current sensor reading and the fan speed, and perform online training on the policy network and the value network; after the training is completed, collect the sensor readings and execute the fan action with the highest probability.

[0084] Some embodiments of the present application provide corresponding Figure 1 A non-volatile computer storage medium for device fan control based on reinforcement learning stores computer executable instructions, wherein the computer executable instructions are configured to: construct an action space according to the operating data of the fan, and construct a state space according to the operating data of the target device, wherein the operating data of the target device includes the operating temperature; obtain the fan power, calculate the reward according to the fan power and the operating temperature, and construct a policy network and a value network according to the state space; obtain the current sensor reading and the fan speed, and perform online training on the policy network and the value network; after the training is completed, collect the sensor reading and execute the fan action with the highest probability.

[0085] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the IoT device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0086] The system and medium provided in the embodiments of the present application correspond one-to-one to the method. Therefore, the system and medium also have similar beneficial technical effects to the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the system and medium will not be repeated here.

[0087] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0088] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0089] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0090] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0091] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0092] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0093] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0094] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0095] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A device fan control method based on reinforcement learning, characterized in that: The method comprises: constructing an action space according to the operating data of the fan, and constructing a state space according to the operating data of the target device, wherein the operating data of the target device includes an operating temperature; Obtain fan power, calculate a reward based on the fan power and operating temperature, and construct a policy network and a value network based on the state space; Obtain current sensor readings and fan speed, and perform online training on the policy network and the value network; After training is complete, sensor readings are collected and the fan action with the highest probability is executed.

2. According to the device fan control method based on reinforcement learning in claim 1, it is characterized in that: The action space is constructed based on the fan's operating data, and the state space is constructed based on the target device's operating data, including: Acquire the adjustment type, the area where the fan is located, and the number of each area to construct the action space, wherein the adjustment types of the fan include keeping unchanged, increasing one gear, and decreasing one gear; The operating temperature and temperature threshold of the target device, as well as the adjustment type and number of fans, are obtained to construct the state space.

3. The device fan control method based on reinforcement learning according to claim 1, characterized in that: Obtain fan power, calculate rewards based on the fan power consumption and operating temperature, and construct a policy network and a value network based on the state space, specifically including: Normalizing and calculating the operating temperature and the temperature threshold of the target device to obtain a temperature reward; Normalizing and calculating the current power and the power threshold of the fan to obtain a power reward; Obtaining a reward function according to the temperature reward and the power reward; Obtaining a value network according to the state space; A policy network is obtained according to the state space and the action space.

4. The device fan control method based on reinforcement learning according to claim 1, characterized in that: Obtain the current sensor reading and fan speed, and perform online training on the policy network and value network, including: Obtaining the current sensor reading and the fan speed, and determining whether a trigger condition for starting training is met, wherein the trigger condition includes the temperature reward and the power reward being reduced to a first preset threshold; When the triggering condition for starting training is met, the reading and the rotation speed are input into the value network and the strategy network to obtain the optimal value network and the strategy network.

5. The device fan control method based on reinforcement learning according to claim 4 is characterized in that: The inputting the reading and the speed into the value network and the strategy network to obtain the optimal value network and strategy network specifically includes: Inputting the reading and the rotation speed into the strategy network to obtain the probability distribution of the fan action; Perform sampling execution according to the probability distribution to generate a quintuple, the quintuple including a fan action and a corresponding probability, a sensor reading, a power reward and a temperature reward after executing the fan action for a first preset time; Adding the five-tuple to the head of an experience data queue, and determining whether the experience data in the experience data queue reaches a first preset number; When the empirical data does not reach a first preset number, continue to generate the quintuple; When the experience data reaches a first preset amount, the experience data at the tail is automatically overflowed, and a second preset amount of experience data is extracted to train the strategy network and the value network to obtain the optimal value network and strategy network.

6. The device fan control method based on reinforcement learning according to claim 5, characterized in that: After the training is completed, the sensor readings are collected at a preset period and the fan action with the highest probability is executed, including: After the training is completed, the optimal strategy network is obtained, and the sensor readings and fan speed are collected according to the preset period; The sensor reading and fan speed are input into the optimal strategy network to obtain the fan action probability distribution, and the fan action with the maximum probability is executed.

7. The device fan control method based on reinforcement learning according to claim 4, characterized in that: Before determining whether the triggering condition for starting training is met, the method further includes: Determine whether it is the peak load period of the equipment, and determine whether there is a high temperature alarm based on the reading; If the equipment is not at its peak load period and there is no high temperature alarm, determine whether the triggering conditions for starting training are met; If a high temperature alarm is present, the fan is operated at full speed for a second preset time.

8. The device fan control method based on reinforcement learning according to claim 5, characterized in that: After training the strategy network and the value network according to the empirical data and taking the action with the highest probability for execution, the method further includes: Regularly evaluating and verifying the policy network and the value network, and if the temperature reward and the power reward do not reach a second preset threshold, checking the reward function; The fan action is monitored and fed back in real time, and if the sensor reading exceeds a temperature threshold, or the current power consumption of the fan exceeds a power threshold, an immediate warning is issued.

9. A device fan control device based on reinforcement learning, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: constructing an action space according to the operating data of the fan, and constructing a state space according to the operating data of the target device, wherein the operating data of the target device includes an operating temperature; Obtain fan power, calculate a reward based on the fan power and operating temperature, and construct a policy network and a value network based on the state space; Obtain current sensor readings and fan speed, and perform online training on the policy network and the value network; After training is complete, sensor readings are collected and the fan action with the highest probability is executed.

10. A non-volatile computer storage medium for device fan control based on reinforcement learning, storing computer executable instructions, characterized in that: The computer executable instructions are configured to: constructing an action space according to the operating data of the fan, and constructing a state space according to the operating data of the target device, wherein the operating data of the target device includes an operating temperature; Obtain fan power, calculate a reward based on the fan power and operating temperature, and construct a policy network and a value network based on the state space; Obtain current sensor readings and fan speed, and perform online training on the policy network and the value network; After training is complete, sensor readings are collected and the fan action with the highest probability is executed.

Images