Cooling fan unit control method and system based on deep reinforcement learning algorithm

Abstract

The invention relates to a control method and system of a cooling fan unit based on a deep reinforcement learning algorithm. The method comprises the following steps: initializing parameters of an Actor network and parameters of a Critic network; the current state of the cooling fan unit is obtained, the current state comprises temperature information, current time and a date type, and the temperature information comprises the temperature of heating equipment and the ambient temperature around a fan; obtaining an action based on the current state and the Actor network; according to the action, the cooling fan unit is controlled to operate, a new state of the cooling fan unit and a return obtained after the action is executed are obtained, an experience sample is obtained, and the experience sample is added into an experience playback pool; and updating network parameters of the Critic network based on the experience samples in the experience playback pool, and updating network parameters of the Actor network based on the current state and the Critic network until a target value is reached. According to the method, the optimal control of the cooling fan unit is realized by constructing the optimal control model of the cooling fan unit.

Description

Technical Field

[0001] The present invention relates to the field of HVAC engineering technology for electric power equipment, and in particular to a control method and system for a heat dissipation fan unit based on a deep reinforcement learning algorithm. Background Art

[0002] Large power equipment such as power transformers and dry-type iron core reactors generate a lot of heat during operation, and must be installed indoors for protection from rain, small animals, and noise. How to improve the efficiency of ventilation and heat dissipation is an urgent problem to be solved in the power industry.

[0003] Most existing technologies model the current state of the system, without considering the impact of the fan's action on the current state of the system, and without fully modeling the state of the system based on all actions and the transfer situation at the next moment. Obvious methods, such as designing the operation mode of the fan according to empirical formulas, cannot achieve adaptive adjustment and do not take into account special circumstances such as the site environment, equipment operating status, and daily load curves. Moreover, most existing technologies can only adjust the wind force, but cannot change the wind direction and position of the air outlet, which is prone to problems such as wind path short circuit. Summary of the invention

[0004] To this end, the present invention provides a control method and system for a cooling fan unit based on a deep reinforcement learning algorithm. By constructing an optimization control model of the cooling fan unit and solving it with a deep reinforcement learning algorithm, optimization control of the cooling fan unit of electrical equipment is achieved. The method and system can be applied to the heat dissipation of electrical equipment in substations, thereby improving the heat dissipation effect, saving energy consumption, reducing infrastructure investment costs, ensuring the safe operation of equipment, and improving the reliability of stable power supply.

[0005] In order to solve the above technical problems, the present invention provides a control method for a heat dissipation fan unit based on a deep reinforcement learning algorithm, comprising:

[0006] Initialize the parameters of the Actor network and the Critic network;

[0007] Obtain the current state of the cooling fan unit, the current state including temperature information, current time and date type, wherein the temperature information includes the temperature of the heating device and the ambient temperature of the fan;

[0008] Based on the current state and the Actor network, obtain an action;

[0009] According to the action, the cooling fan unit is controlled to operate, a new state of the cooling fan unit and a reward obtained after executing the action are obtained, an experience sample consisting of the current state, the action, the new state and the reward are obtained, the experience sample is repeatedly obtained, and the experience sample is added to the experience playback pool;

[0010] Based on the experience samples in the experience replay pool, the network parameters of the Critic network are updated, and based on the current state and the Critic network, the network parameters of the Actor network are updated until a target value is reached.

[0011] In one embodiment of the present invention, there are two Critic networks, including a first Critic network Q φ1 and the second critic network Q φ2 Both use a multilayer perceptron model with 11 input layer nodes, 256 first hidden layer nodes, and a rectified linear unit activation function. The second hidden layer has 128 nodes, and a leaky rectified linear unit activation function. The output layer has 1 node, and a hyperbolic tangent function activation function.

[0012] In one embodiment of the present invention, the Actor network adopts a multilayer perceptron model, the number of nodes in the input layer is 6; the number of nodes in the first hidden layer is 256, and the activation function is a rectified linear unit; the number of nodes in the second hidden layer is 128, and the activation function is a leaky rectified linear unit; the number of nodes in the output layer is 5, and the activation function is a hyperbolic tangent function; the Actor network μ is randomly initialized. θ The parameter θ is subject to

[0013] In one embodiment of the present invention, it also includes:

[0014] Acquire multiple temperature measurement points as the temperature information, including respectively collecting the first temperature s at the surface of the transformer and located at the gap between the heat dissipation blades and the bottom position. t1 and the second temperature s t2 , and collect the third temperature s of the air temperature in the environment at the bottom of the fan t3 and the fourth temperature s t4 .

[0015] In one implementation of the present invention, the current time is the total number of seconds that have passed in a day divided by 86400; the date type is divided into weekdays, weekends, Spring Festival and other holidays, and the values ​​are -1, -0.5, 0.5 and 1 respectively.

[0016] In one embodiment of the present invention, based on the current state and the Actor network, an action is obtained; comprising:

[0017] The fan guide plate rotation angle, fan displacement, and fan wind force are used as actions. The fan guide plate rotation angle value range is normalized to [-1, 1], where -1 means no rotation and 1 means 360-degree rotation. The fan displacement is normalized to [-1, 1], where -1 means no movement and 1 means movement of a predetermined distance. The fan wind force is normalized to [-1, 1], where -1 means no wind output and 1 means wind output at rated power.

[0018] The temperature information, current time s t , Date Types d As the current state s, the action a is selected as:

[0019] a=clip(μ θ (s)+∈,a Low ,a High )

[0020] in The mean is 0 and the variance is σ 2 Gaussian noise distribution, standard deviation σ = 0.1; μ θ is the Actor network; the lower limit of the action amount is a Low =(-1,-1,-1,-1,-1); action limit a High =(1,1,1,1,1).

[0021] In one embodiment of the present invention, according to the action, the cooling fan unit is controlled to operate, a new state of the cooling fan unit and a reward obtained after executing the action are obtained, an experience sample consisting of the current state, the action, the new state and the reward is obtained, the experience sample step is repeatedly obtained, and the experience sample is added to the experience playback pool; comprising:

[0022] The temperature information, current time, and date type after executing action a are taken as the new state s′, and the reward r is obtained:

[0023]

[0024] where T1, T2, ..., T M are the temperatures of M temperature measurement points, P1, P2, v, P N is the power consumption of N fans, k1=0.2, k2=0.1 are weight coefficients;

[0025] Initialize an empty experience replay pool Take (s,a,r,s′) as a sample and store it in the experience replay pool middle.

[0026] In one embodiment of the present invention, based on the experience samples in the experience replay pool, the network parameters of the Critic network are updated, and based on the current state and the Critic network, the network parameters of the Actor network are updated until the optimal goal is achieved; including:

[0027] Initialize the first target Critic network And the second target Critic network The parameters correspond to the first Critic network And the second critic network The parameters are the same, that is, φ targ,1 ←φ1、φ targ,2 ←φ2; Initialize the target Actor network Parameters to make it consistent with the Actor network μ θ The parameters are the same, that is, θ targ ←θ;

[0028] When the experience replay pool When the number of samples in is greater than the preset number threshold, the experience replay pool Randomly select a batch of data B = {(s, a, r, s′)};

[0029] The estimated target value is:

[0030]

[0031] in:

[0032]

[0033] is the discount factor; c is the random noise clipping parameter;

[0034] According to the following formula, the first critic network is descented in the direction of gradient And the second critic network Update the parameters φ1 and φ2;

[0035]

[0036] According to the following formula, the Actor network μ is descented in the direction of the following gradient θ The parameters θ are updated:

[0037]

[0038] Update the first target Critic network according to the following formula Second target critic network and the target Actor network Parameters:

[0039] φ targ,i ←ρφ targ,i +(1-ρ)φ i ,i=1,2

[0040] θ targ ←ρθ targ +(1-ρ)θ;

[0041] Among them, ρ is the soft update learning rate.

[0042] In one embodiment of the present invention, the objective function is:

[0043]

[0044] Where T 1t ,T 2t ,v,T Mt is the temperature of M temperature measurement points at time t, P 1t ,P 2t ,...,P Nt is the power consumption of N wind turbines at time t, r t is the return at time t, k1=0.2, k2=0.1 are weight coefficients; γ is the discount factor, and the optimization target R t is the discounted return at time t.

[0045] The present invention also provides a system of a heat dissipation fan unit based on a deep reinforcement learning algorithm, comprising:

[0046] A control unit, wherein the control unit is configured to execute the control method of the heat dissipation fan unit based on the deep reinforcement learning algorithm according to any one of claims 1 to 9;

[0047] A bracket, fixedly or slidably connected to the mounting surface;

[0048] A plurality of fans arranged beside the heat generating device, at least one of which can be moved by the bracket, the fan is used to provide wind force for cooling the heat generating device, and the control unit can control the wind force;

[0049] A guide plate is mounted on the surface of the fan outlet, the guide plate can rotate, and its rotation axis is parallel to or coincides with the rotation axis of the fan blades;

[0050] A temperature sensing unit is used to obtain temperature information.

[0051] The above technical solution of the present invention has the following advantages compared with the prior art:

[0052] The control method and system of the heat dissipation fan unit based on the deep reinforcement learning algorithm described in the present invention directly learn the control strategy through the interaction between the deep reinforcement learning algorithm and the environment. There is no need to accurately model the system model of the heat dissipation fan unit. It is suitable for complex heat dissipation environments and can automatically optimize the action of the fan according to the real-time changes of the on-site environment and the operating status of the electrical equipment, thereby realizing intelligent adjustment of the wind speed, wind direction and fan position.

[0053] The present invention takes the temperature of key points on the surface of the heating equipment, the ambient temperature, and various status information such as time and date as input, comprehensively considers the balance between heat dissipation effect and energy consumption, and finds the optimal heat dissipation solution under different conditions through the exploration mechanism of reinforcement learning, avoiding problems such as traditional air path short circuit and heat dissipation blind spots.

[0054] The present invention adopts a deep reinforcement learning algorithm to optimize energy consumption through long-term planning, reduce unnecessary fan operation, and achieve energy-saving effects. The fan operation strategy is optimized to achieve lower energy consumption, while improving the heat dissipation efficiency of the equipment, reducing dependence on external energy, and complying with the concept of green environmental protection. By accurately controlling the temperature of electrical equipment, equipment failure or performance degradation due to high temperature is avoided, ensuring the safe operation of the equipment. Through efficient heat dissipation, the stable operation of electrical equipment during peak load periods is guaranteed, thereby improving the reliability and stability of the power supply system.

[0055] The present invention adopts a double-delayed deep deterministic policy gradient algorithm to enhance the stability and convergence speed of the optimization strategy, and solves the limitations of traditional algorithms in multi-step decision-making scenarios. It is not only suitable for indoor electrical equipment heat dissipation, but can also be extended to systems that require dynamic temperature control in other industrial scenarios, such as server rooms and industrial machinery heat dissipation. Through optimized control, the service life of the equipment can be extended, maintenance costs can be reduced, and the return on investment can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0057] Figure 1 It is a method flow chart of a heat dissipation fan unit based on a deep reinforcement learning algorithm of the present invention.

[0058] Figure 2 It is a system diagram of the heat dissipation fan unit based on the deep reinforcement learning algorithm of the present invention.

[0059] Figure 3 It is a structural diagram of the guide plate of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0061] In the present invention, if directions (up, down, left, right, front and back) are described, it is only for the convenience of describing the technical solution of the present invention, and does not indicate or imply that the technical features referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention.

[0062] In the present invention, "several" means one or more, "multiple" means more than two, "greater than", "less than", "exceed" and the like are understood to exclude the number itself; "above", "below", "within" and the like are understood to include the number itself. In the description of the present invention, if there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0063] In the present invention, unless otherwise clearly defined, the words "set", "install", "connect" and the like should be understood in a broad sense, for example, they can be directly connected or indirectly connected through an intermediate medium; they can be fixedly connected or detachably connected or integrally formed; they can be mechanically connected or electrically connected or able to communicate with each other; they can be the internal connection of two elements or the interaction relationship between two elements. Those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0064] Reference Figure 2 , Figure 3 As shown, a system of a heat dissipation fan unit based on a deep reinforcement learning algorithm includes:

[0065] Control unit;

[0066] The bracket is fixedly or slidably connected to the mounting surface. For example, the bracket can be fixed to the ground, wall or ceiling through a slide rail, so that the bracket can slide as a whole on the slide rail to change the spatial position;

[0067] A plurality of fans arranged beside a heat generating device (such as a power transformer, a dry core reactor, etc.), at least one of which can be moved by the bracket, the fan is used to provide wind for heat dissipation of the heat generating device, and the control unit can control the wind force; the fan converts electrical energy into the rotation of the fan, generates a strong airflow, and blows directly on the surface of the heat generating device, thereby accelerating the convection heat transfer process, taking away the heat on the surface of the device, and promoting the diffusion of heat inside the device, effectively reducing the temperature of the device, and the wind force can be adjusted according to the control signal;

[0068] A guide plate is installed on the surface of the fan outlet, and the guide plate can rotate, and its rotation axis is parallel to or coincides with the rotation axis of the fan blades, so as to change the wind direction of the air outlet;

[0069] The temperature sensing unit is used to obtain temperature information, measure the temperature of key locations on the surface of the equipment, or the air temperature in the environment, and send the temperature information to the control unit through communication technology.

[0070] The control unit is used to collect the temperature information obtained by each temperature sensing unit, obtain the current time and date type through the internal clock and calendar, output the deflector rotation angle, sliding distance (displacement) and wind force of each fan as the action amount, and can collect the ambient temperature, date type and time at the next moment as feedback. The goal is to minimize the average temperature collected by each temperature sensing unit and the total energy consumption of the fan. The objective function is

[0071]

[0072] Where T 1t ,T 2t ,...,T Mt is the temperature of M temperature measurement points at time t, P 1t ,P 2t ,...,P Nt is the power consumption of N wind turbines at time t, r t is the return at time t, k1=0.2, k2=0.1 are weight coefficients; γ=0.9 is the discount factor, and the optimization target R t is the discounted return at time t; the control strategy is iteratively optimized using the double-delayed deep deterministic policy gradient algorithm to achieve the goal of optimal control.

[0073] Specifically, in this embodiment, the system of the heat dissipation fan unit includes a fan fixed on the ground and a fan installed on the slide rail, which are used to convert electrical energy into the rotation of the fan, generate strong airflow, and blow directly on the surface of the transformer equipment, thereby accelerating the convection heat transfer process, taking away the heat on the surface of the equipment, and promoting the diffusion of heat inside the equipment, effectively reducing the temperature of the equipment. The wind force is respectively a p1 、a p2 It can be adjusted according to the control signal, and the value range is normalized to the continuous range of [-1,1], where -1 means no wind power is output, and 1 means wind power is output at rated power.

[0074] In addition, the bracket is connected to the fan to fix the fan. The guide plate is installed at the air outlet of the two fans and can rotate according to the set rotation angle. Its rotation axis is parallel or coincident with the rotation axis of the fan blade. Its rotation angle a g1、a g2 The value range is normalized to the continuous range of [-1,1], where -1 means no rotation and 1 means 360-degree rotation. The guide vane has a fixed angle of attack and can change the wind direction of the air outlet when rotated to different positions.

[0075] The slide rail is fixed on the ceiling, and the bracket and fan can slide on the slide rail as a whole to change the spatial position. The total length of the slide rail is about 10 meters, and the position of the bracket and fan on the slide rail is a x Normalized to [-1, 1], -1 means no sliding, 1 means sliding 10 meters. Fans in different positions can effectively eliminate or reduce blind spots in the system heat dissipation area and avoid air path short circuits.

[0076] The control unit may adopt a single chip microcomputer.

[0077] Reference Figure 1 As shown, based on the above system, a control method for a cooling fan unit based on a deep reinforcement learning algorithm is provided, specifically using a double-delayed deep deterministic policy gradient algorithm as a specific algorithm for deep reinforcement learning, including the following steps:

[0078] S1. Initialize the parameters of the Actor network and the Critic network.

[0079] There are two Critic networks, including the first Critic network And the second critic network Both use a multi-layer perceptron model, with 11 nodes in the input layer; 256 nodes in the first hidden layer, and the activation function is the rectified linear unit; 128 nodes in the second hidden layer, and the activation function is the leaky rectified linear unit; the number of nodes in the output layer is 1, and the activation function is the hyperbolic tangent function.

[0080] The Actor network adopts a multi-layer perceptron model, with 6 input layer nodes; 256 first hidden layer nodes, and the activation function is a rectified linear unit; 128 second hidden layer nodes, and the activation function is a leaky rectified linear unit; 5 output layer nodes, and the activation function is a hyperbolic tangent function; the Actor network μ is randomly initialized. θ The parameter θ follows the standard normal distribution, that is,

[0081] S2. Obtain the current state of the cooling fan unit, where the current state includes temperature information, current time and date type.

[0082] Among them, two of the temperature sensing units are arranged on the transformer surface, respectively at the gap between the heat dissipation blades and the bottom position, to measure the heating conditions at key positions, and the reading is recorded as s t1 、st2 ; Two other measuring points are arranged at the bottom of the fan to measure the air temperature s in the environment t3 、s t4 , the temperature data is sent to the control unit through communication technology.

[0083] The temperature information includes the temperature of the heating device and the ambient temperature of the fan.

[0084] The current time is the total number of seconds that have passed in a day divided by 86400. The date types are weekdays, weekends, Spring Festival, and other holidays, and the values ​​are -1, -0.5, 0.5, and 1 respectively.

[0085] S3. Based on the current state and the Actor network, an action is obtained, wherein the wind deflector rotation angle of the fan, the displacement of the fan, and the wind force of the fan are used as actions. The value range of the wind deflector rotation angle of the fan is normalized to [-1, 1], where -1 means no rotation and 1 means rotation of 360 degrees; the displacement of the fan is normalized to [-1, 1], where -1 means no movement and 1 means movement of a predetermined distance. The wind force of the fan is normalized to [-1, 1], where -1 means no wind output and 1 means wind output at rated power.

[0086] Specifically include:

[0087] The temperature information (i.e. s t1 、s t2 、s t3 、s t4 ), current time s t , Date Types d As the current state s, the action a is selected as:

[0088] a=clip(μ θ (s)+∈,a Low ,a High )

[0089] in The mean is 0 and the variance is σ 2 Gaussian noise distribution, standard deviation σ = 0.1; μ θ is the Actor network; the lower limit of the action amount is a Low =(-1,-1,-1,-1,-1); action limit a High =(1,1,1,1,1).

[0090] S4. According to the action a, control the cooling fan unit to operate, obtain the new state of the cooling fan unit, and the reward obtained after executing the action, obtain an experience sample consisting of the current state, the action, the new state and the reward, repeat the steps of obtaining the experience sample (i.e., S2 to S3), and add the experience sample to the experience playback pool. Specifically:

[0091] Deconstruct and assign the selected action a, and get a=(a g1 ,a g2 ,a x ,a p1 ,a p2 ), where a g1 、a g2 is the corresponding rotation angle of the guide plate of each fan, a x is the corresponding distance (displacement) that the fan slides on the slide rail, a p1 、a p2 is the wind force of each fan. By controlling the system according to the above action amount, the velocity field of the indoor airflow can be changed, the flow state of the airflow can be optimized, and the heat dissipation effect can be improved.

[0092] After executing action a, after a period of time, the temperature information, current time, and date type are taken as the new state s′, and the reward r is obtained:

[0093]

[0094] Initialize an empty experience replay pool Take (s,a,r,s′) as a sample and store it in the experience replay pool Initialize the loop number i=0, and loop through steps S2 to S4.

[0095] S5. Based on the experience samples in the experience replay pool, the network parameters of the Critic network are updated by soft update or gradient descent method, and based on the current state and the Critic network, the network parameters of the Actor network are updated by soft update or gradient descent method until the specified number of iterations is reached. The update period of the Actor network and the target Critic network is d=5.

[0096] Specifically:

[0097] Initialize the first target Critic network And the second target Critic network The parameters correspond to the first Critic network And the second critic network The parameters are the same, that is, φ targ,1 ←φ1、φtarg,2 ←φ2; Initialize the target Actor network Parameters to make it consistent with the Actor network μ θ The parameters are the same, that is, θ targ ←θ;

[0098] When the experience replay pool When the number of samples in is greater than a preset number threshold (such as 10000), the experience replay pool Randomly select a batch of data B = {(s, a, r, s′)};

[0099] The estimated target value is:

[0100]

[0101] in:

[0102] γ is the discount factor; c is the random noise clipping parameter;

[0103] According to the following formula, the first critic network is descented in the direction of gradient And the second critic network Update the parameters φ1 and φ2;

[0104]

[0105] If imodd = 0, the Actor network μ is descented in the following direction according to the following formula θ The parameters θ are updated:

[0106]

[0107] Update the first target Critic network according to the following formula Second target critic network and the target Actor network Parameters:

[0108] φ targ,i ←ρφ targ,i +(1-ρ)φ i ,i=1,2

[0109] θ targ ←ρθ targ +(1-ρ)θ;

[0110] Among them, ρ = 0.01 is the soft update learning rate.

[0111] The objective function is:

[0112]

[0113] Where T 1t ,T 2t ,...,T Mt is the temperature of M temperature measurement points at time t, P 1t ,P 2t ,...,P Nt is the power consumption of N wind turbines at time t, r t is the return at time t, k1=0.2, k2=0.1 are weight coefficients; γ is the discount factor, and the optimization target R t is the discounted return at time t.

[0114] Through the above method, the cooling fan unit based on the deep reinforcement learning algorithm also completes the cooling work in the process of learning cooling. As time goes by, the cooling fan unit based on the deep reinforcement learning algorithm will continue to iterate and optimize, and gradually adaptively learn efficient cooling techniques without human intervention or assistance.

[0115] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0119] Finally, it should be noted that the above specific implementation methods are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A control method for a heat dissipation fan unit based on a deep reinforcement learning algorithm, characterized in that: include: Initialize the parameters of the Actor network and the Critic network; Obtain the current state of the cooling fan unit, the current state including temperature information, current time and date type, wherein the temperature information includes the temperature of the heating device and the ambient temperature of the fan; Based on the current state and the Actor network, obtain an action; According to the action, the cooling fan unit is controlled to operate, a new state of the cooling fan unit and a reward obtained after executing the action are obtained, an experience sample consisting of the current state, the action, the new state and the reward are obtained, the experience sample is repeatedly obtained, and the experience sample is added to the experience playback pool; Based on the experience samples in the experience replay pool, the network parameters of the Critic network are updated, and based on the current state and the Critic network, the network parameters of the Actor network are updated until the optimal goal is achieved.

2. The control method of the heat dissipation fan unit based on the deep reinforcement learning algorithm according to claim 1 is characterized in that: There are two Critic networks, including the first Critic network And the second critic network Both use a multi-layer perceptron model, with 11 nodes in the input layer; 256 nodes in the first hidden layer, and the activation function is the rectified linear unit; 128 nodes in the second hidden layer, and the activation function is the leaky rectified linear unit; the number of nodes in the output layer is 1, and the activation function is the hyperbolic tangent function.

3. The control method of the heat dissipation fan unit based on the deep reinforcement learning algorithm according to claim 1 is characterized in that: The Actor network adopts a multi-layer perceptron model, with 6 input layer nodes; 256 first hidden layer nodes, and the activation function is a rectified linear unit; 128 second hidden layer nodes, and the activation function is a leaky rectified linear unit; 5 output layer nodes, and the activation function is a hyperbolic tangent function; the Actor network μ is randomly initialized. θ The parameter θ is subject to 4. The control method of the heat dissipation fan unit based on the deep reinforcement learning algorithm according to claim 1 is characterized in that: Also includes: Acquire multiple temperature measurement points as the temperature information, including respectively collecting the first temperature s at the surface of the transformer and located at the gap between the heat dissipation blades and the bottom position. t1 and the second temperature s t2 , and collect the third temperature s of the air temperature in the environment at the bottom of the fan t3 and the fourth temperature s t4 .

5. The control method of the heat dissipation fan unit based on the deep reinforcement learning algorithm according to claim 1 is characterized in that: The current time is the total number of seconds that have passed in a day divided by 86400; the date type is divided into weekdays, weekends, Spring Festival and other holidays, and the values ​​are -1, -0.5, 0.5 and 1 respectively.

6. The control method of the heat dissipation fan unit based on the deep reinforcement learning algorithm according to claim 2 is characterized in that: Based on the current state and the Actor network, an action is obtained; including: The fan guide plate rotation angle, fan displacement, and fan wind force are used as actions. The fan guide plate rotation angle value range is normalized to [-1, 1], where -1 means no rotation and 1 means 360-degree rotation. The fan displacement is normalized to [-1, 1], where -1 means no movement and 1 means movement of a predetermined distance. The fan wind force is normalized to [-1, 1], where -1 means no wind output and 1 means wind output at rated power. The temperature information, current time s t , Date Types d As the current state s, choose action a as: a=clip(μ θ (s)+∈,a Low ,a High ) in The mean is 0 and the variance is σ 2 Gaussian noise distribution, standard deviation σ = 0.1; μ θ is the Actor network; the lower limit of the action amount is a Low =(-1,-1,-1,-1,-1); action limit a High =(1,1,1,1,1).

7. The control method of the heat dissipation fan unit based on the deep reinforcement learning algorithm according to claim 6 is characterized in that: According to the action, the cooling fan unit is controlled to run, a new state of the cooling fan unit and a reward obtained after executing the action are obtained, an experience sample consisting of the current state, the action, the new state and the reward are obtained, the experience sample step is repeatedly obtained, and the experience sample is added to the experience playback pool; including: The temperature information, current time, and date type after executing action a are taken as the new state s′, and the reward r is obtained: where T1, T2, ..., T M are the temperatures of M temperature measurement points, P1, P2, ..., P N is the power consumption of N fans, k1=0.2, k2=0.1 are weight coefficients; Initialize an empty experience replay pool Take (s,a,r,s′) as a sample and store it in the experience replay pool middle.

8. The control method of the heat dissipation fan unit based on the deep reinforcement learning algorithm according to claim 7 is characterized in that: Based on the experience samples in the experience replay pool, the network parameters of the Critic network are updated, and based on the current state and the Critic network, the network parameters of the Actor network are updated until the optimal goal is achieved; including: Initialize the first target Critic network And the second target Critic network The parameters correspond to the first Critic network And the second critic network The parameters are the same, that is, φ targ,1 ←φ1、φ targ,2 ←φ2; Initialize the target Actor network Parameters to make it consistent with the Actor network μ θ The parameters are the same, that is, θ targ ←θ; When the experience replay pool When the number of samples in is greater than the preset number threshold, the experience replay pool Randomly select a batch of data B = {(s, a, r, s′)}; The estimated target value is: in: γ is the discount factor; c is the random noise clipping parameter; According to the following formula, the first critic network is descented in the direction of gradient And the second critic network Update the parameters φ1 and φ2; According to the following formula, the Actor network μ is descented in the direction of the following gradient θ The parameters θ are updated: Update the first target Critic network according to the following formula Second target critic network and the target Actor network Parameters: f targ,i ←rf targ,i +(1-r)φ i ,i=1.2 i targ ←ρθ targ +(1-ρ)θ; Among them, ρ is the soft update learning rate.

9. The control method of the heat dissipation fan unit based on the deep reinforcement learning algorithm according to claim 8 is characterized in that: The objective function is: Where T 1t ,T 2t ,...,T Mt is the temperature of M temperature measurement points at time t, P 1t ,P 2t ,…,P Nt is the power consumption of N wind turbines at time t, r t is the return at time t, k1=0.2, k2=0.1 are weight coefficients; γ is the discount factor, and the optimization target R t is the discounted return at time t.

10. A system for a heat dissipation fan unit based on a deep reinforcement learning algorithm, characterized in that: include: A control unit, wherein the control unit is configured to execute the control method of the heat dissipation fan unit based on the deep reinforcement learning algorithm according to any one of claims 1 to 9; A bracket, fixedly or slidably connected to the mounting surface; A plurality of fans arranged beside the heat generating device, at least one of which can be moved by the bracket, the fan is used to provide wind force for cooling the heat generating device, and the control unit can control the wind force; A guide plate is mounted on the surface of the fan outlet, the guide plate can rotate, and its rotation axis is parallel to or coincides with the rotation axis of the fan blades; A temperature sensing unit is used to obtain temperature information.

Images