Intelligent and adaptive liquid cooling of electronic devices

Abstract

A system configured for cooling electronic devices implements flow control devices that vary a flow of a cooling liquid past the electronic devices through at least one cooling loop or conduit, or through an immersion tank. A controller detects and / or predicts overall cooling requirements and / or localized hotspots and adjusts the flow control devices to direct proportionately more cooling liquid to the hotspots. The flow control devices can be any combination of variable speed pumps and / or adjustable valves, vents, and / or baffles. Active manifolds can be placed on two opposing sides of the devices, on four opposing sides, and / or above and / or below the devices. Temperatures proximate the devices can be measured by separate sensors and / or sensors integral to components of the electronic devices. Hotspots can be predicted by monitoring current flows, power flows, and / or voltages of the electronic devices, and / or inferred from network activity and / or from a workload queue.

Description

INTELLIGENT AND ADAPTIVE LIQUID COOLING OF ELECTRONICDEVICESInventors:Wendell Wong Shun Yin Neil HavrillaMichael Albert MancusoRELATED APPLICATIONS

[0001] This application claims the priority of US application 19 / 097,075 filed on is a April 1, 2025, and of US Application 18 / 767,367, filed on July 9, 2024. Both of these applications are included herein by reference in their entirety for all purposes.FIELD OF THE INVENTION

[0001] The invention relates to cooling of electronic devices, and more particularly to liquid cooling of electronic devices.BACKGROUND OF THE INVENTION

[0002] Many electronic devices, including most computers, comprise some sort of cooling mechanism designed to moderate the temperatures of the integrated circuits (ICs) included in the device, as well as the overall temperature within the device housing. For example, most laptop and desktop computers include at least one fan that circulates ambient air throughout the housing interior. In addition, circuits with high power consumption, such as central processing units (CPUs), often include dedicated fans to ensure that they do not overheat.

[0003] This “forced air convection” approach is effective for many individual electronic devices having housings that are surrounded by ambient air. However, forced air convection may be insufficient for cooling high power consumption devices that generate excessive amounts of heat, and / or large “clusters” or “arrays” of components or devices that are closely spaced together, and fill most of the volume within a housing, a room, or building. In particular, cooling of so-called “data centers,” “server farms,” artificial intelligence (Al) systems, and “super-computers” (referred to herein collectively as “data centers”) can be challenging. In such cases, cooling of the electronic devices by a circulated liquid coolant can be necessary.

[0004] Data centers are one of the most energy-intensive building usage types, consuming from 10 to 50 times as much energy per square foot as a typical commercial office building. Collectively, data centers account for approximately 2% of the total U.S. electricity usage, and this figure is expected to increase as data centers become more numerous, due in part to the rapid increase in artificial intelligence systems.

[0005] While much of the energy that is consumed by a data center is due to the power consumed and dissipated by the electronics, a significant amount of energy is also consumed by the cooling systems that are required for removing the dissipated energy from the electronics. Accordingly, there is a need to optimize the energy efficiency of data center cooling liquid systems.

[0006] With reference to Fig. 1A, a liquid coolant 102, such as water, oil, ethylene glycol, or some other coolant, is sometimes used to cool one or more high power consumption devices 100. According to this approach, the coolant 102 is delivered as a cooling liquid to the electronic devices 100 by a liquid coolant pump 106, Heat from the electronic devices 100 is thereby absorbed by the coolant 102 and delivered to a heat exchanger or other heat transfer apparatus 108, from which the heat is transferred to a radiator, cooling tower, or other heat dissipation device 110, while the cooled liquid coolant is recycled to the electronic devices 100. The cyclic path that is followed by the coolant, together with any pumps 106, 136, valves, 122, gauges 126, 132 etc. is sometimes referred to herein as a cooling “loop.”

[0007] In some applications, the coolant remains a liquid throughout the cooling loop, while in the example of Fig. 1A the coolant 102 is vaporized when absorbing heat from the electronic devices 100, and the resulting vaporized coolant 112 isdrawn by a vapor suction pump 136 to the heat transfer apparatus 108, where it is “condensed” back to a liquid state. In Fig. 1A, the electronic device 100 comprises a printed circuit board 128 on which a plurality of integrated circuit “dies” 130 are mounted. The cooling liquid 102 is circulated through a device cooling manifold 134 that is in direct thermal contact with the dies 130, where it absorbs heat and is vaporized.

[0008] A coolant liquid flow control valve 122 can be used to adjust a degree of cooling of the electronic devices 100 by the cooling loop. In some cases, the valve 122 is operated manually, while in the illustrated example the valve 122 is operated by a valve controller 124, which can actuate the flow control valve 124 according to commands remotely input by a user, or automatically according to temperature measurements or other criteria. The example of Fig. 1A further includes a flow measuring device 132, and a plurality of temperature and pressure gauges or sensors 126.

[0009] With reference to the top view of Fig. IB, a similar approach can be used to cool an array of electronic devices 100. According to this approach, a Coolant Distribution Unit (CDU) comprises inlet 114 and outlet 116 manifolds which circulate the cooling liquid 102 through cooling conduits 118, such as copper pipes, which extend between and / or through the device housings 120 in thermal communication with the electronic devices 100, causing the cooling liquid 102 to absorb heat from the electronic devices 100, and to conduct the heat to an external heat transfer apparatus 108, which in the illustrated example transfers the heat to a cooling tower 110 from which it is dissipated into the atmosphere.

[0010] The cooling conduits 118 may include fins (not shown) which increase ambient heat absorption into the pipes. In some applications, the electronic devices 100 that generate the most heat, such as CPUs, are placed in direct thermal contact with the cooling conduits 118, so that they are cooled with the greater efficiency. In similar applications, the device housings 120 of Fig. IB are omitted, and the cooling liquid conduits 118 are directed to or through thehousings of the electronic devices 100, or directly to electronic components of the electronic devices 100, as shown in Fig. 1A.

[0011] In the example of Fig. IB, the coolant remains a cooling liquid after absorbing the heat that is dissipated by the electronic devices 100. Accordingly, a single cooling liquid pump 106 is used to circulate the cooling liquid 102 through the cooling conduits 118, and then through the heat transfer apparatus 108. As in Fig. 1A, a flow control valve 122 can be used to adjust the flow of the cooling liquid 102, and thereby adjust a degree of cooling of the electronic devices 100.

[0012] With reference to the top view of Fig. 1C, another approach is to locate the electronic devices 100 in a liquid-tight immersion tank 104, and to immerse the electronic devices 100 in a dielectric cooling liquid 102. This approach places each electronic component of each of the electronic devices 100 in direct physical and thermal contact with the cooling liquid 102.

[0013] It will be understood, that the heat exchangers 108 and cooling towers 110 that are included in some of the present drawings are intended to generically represent any external heat transfer and dissipation apparatus, unless otherwise stated or required by context, and are not meant to limit the invention to a specific heat dissipation apparatus design.

[0014] Figs. IB and 1C illustrate a plurality of electronic devices 100 enclosed in separate housings and arranged within a device cabinet 120 or immersion tank 104 in a horizontal column from bottom to top in the figures. In similar applications, the electronic devices 100 are arranged as a grid of devices in perpendicular horizontal rows and columns. According to this approach, each “server” 100 in Figs. IB and 1C would be replaced by a row, or “bank,” of individual electronic device housings arranged in a horizontal row from left to right. In still other applications, a three-dimensional “matrix” of electronic devices 100 extend horizontally in rows and columns, and vertically in “tiers.” Still other embodiments are applied to single electronic devices having a plurality of electronic components, for example a large “motherboard” comprising aplurality of integrated circuits, whether or not each electronic device is enclosed within a separate housing.

[0015] It will be noted that, unless otherwise stated or required by context, terms such as “electronic device,” “electronic component,” and “server” are used generically and interchangeably herein to refer to any electronic element of a system that requires cooling, whether or not the electronic device is within a housing, and whether or not the device is a network server, unless otherwise required by context. It will be further noted that terms such as “array” and “device array” are used herein generically to refer to any group of electronic devices and / or components that are to be cooled by the present invention, regardless of how they are physically arranged.

[0016] In the examples of Figs. IB and 1C, the cooling liquid is uniformly distributed among all of the electronic devices 100 by the inlet 114 and outlet 116 manifolds. This approach essentially assumes that each of the electronic devices 100 will have substantially the same heat output. However, in general, the electronic devices 100 may not all be identical to each other. And even if they are identical electronically, the heat that is generated by each of the electronic devices 100 may vary according to the workload that is placed upon it. This can lead to temperature variations, and to “hotspots,” within the array of electronic devices 100. In such cases, if insufficient cooling is applied to the hotspots, the electronic devices 100 in the hotspots will consume excess power and operate less efficiently.

[0017] If the locations of the hotspots do not vary with time, it may be possible to cool them by pre-configuring the manifolds 114, 116 to direct more of the cooling liquid flow to the hotspot(s). However, if the hotspots are transitory, for example due to shifting workloads among the servers, then it may be necessary to compensate for hotspots by increasing the overall flow of the cooling liquid 102 through the device cabinet 120 or immersion tank 104, thereby providing the required amount of cooling to the hotspots, while providing an excess of coolingto other regions of the device cabinet 120 or immersion tank 104. While effective, this approach can be wasteful of energy, and can require an expensive “overdesign” of the cooling system, whereby the total cooling capacity significantly exceeds the total amount of heat that will ever be generated within the device array. Also, when the circulation of the cooling liquid is increased until the hotspot(s) is / are sufficiently cooled, thereby applying more cooling to other regions than is needed, the temperature of the effluent cooling liquid from the device array will be reduced, thereby decreasing the efficiency of the heat dissipation apparatus 108, 110.

[0018] What is needed, therefore, is an electronic device liquid cooling system that can provide adequate liquid cooling to transitory electronic device hotspots, while reducing the required cooling capacity of the overall system and while optimizing the overall energy efficiency of the cooling system.SUMMARY OF THE INVENTION

[0019] The present invention is an electronic device liquid cooling system that can provide adequate liquid cooling to transitory electronic device hotspots, while reducing the required cooling capacity of the overall system and while optimizing the overall energy efficiency of the cooling system.

[0020] According to embodiments of the present invention, rather than operating one or more coolant pumps at fixed operating rates, while possibly adjusting the flow rate of the coolant using one or more flow control valves, the present invention implements one or more variable speed coolant pumps, and adjusts the flow rate of the coolant by varying the operating speeds of the one or more variable speed coolant pumps, without reliance on flow control valves. Accordingly, cooling of the electronic devices is optimized by adjusting the coolant pumping rate according to variable cooling demands, while the energy efficiency of the cooling system is optimized by reducing the operating rate of the variable speed coolant pumps, and thereby reducing the energy consumption of thecoolant pumps, when maximum cooling of the electronic devices is not required. In various embodiments, the variable speed coolant pumps are variable frequency pumps.

[0021] In some embodiment where the coolant remains a liquid throughout its cooling cycle, the cooling loop comprises only a single variable speed liquid coolant pump. In other embodiments where cooling of the electronic devices causes the coolant to vaporize, the cooling loop comprises at least one variable speed liquid coolant pump and at least one separate variable speed vapor suction pump. In embodiments, an isolation valve is included in the cooling loop, for example to meet safety requirements.

[0022] In embodiments, all of the electronic devices are cooled by a single cooling loop, while in other embodiments a plurality of cooling loops are implemented, each having at least one dedicated variable speed coolant pump. By separately adjusting the coolant flow rates in a plurality of cooling loops, specific racks, set of racks, set of chips in a rack, and / or specific chipsets can be targeted with the amount of cooling that is needed, rather than increasing the cooling of all of the electronic devices due to the needs of one sub population that may require maximum cooling.

[0023] Similarly, in embodiments, at least one of the cooling loops is branched, having a plurality of flow paths, and in some of these embodiments, isolation valves included in the flow paths are opened and closed so as to concentrate the cooling where it is needed among the electronic devices.

[0024] In embodiments, at least one of the cooling loops comprises redundant variable speed pumps configured such that, in case of a pump failure, the operating speeds of the remaining pumps can be increased in compensation.

[0025] In various embodiments, adjustment of the operating speeds of the variable speed pumps is reactive, predictive, or both. In some reactive embodiments, the controller receives temperature measurements from one or morelocations within an array of electronic devices. In some of these embodiments, at least one of the temperature measurements is provided by a sensor that is integral to an integrated circuit (IC), such as a central processing unit (CPU), and is configured to report a temperature of the IC. Other embodiments include at least one temperature sensor configured to measure a local temperature of the coolant or ambient air.

[0026] In some predictive embodiments, the controller is able to adjust the operating speeds of the coolant pumps in advance of any actual changes in electronic device temperature, and in some embodiments also in advance of any actual changes in power consumption by the electronic devices, thereby minimizing or avoiding temperature fluctuations of the electronic devices, and providing increased longevity and improved operational stability of the devices. In some of these embodiments, changes in the heat output of the electronic devices are anticipated by monitoring the amount of current that is drawn by at least one of the electronic devices, such as by one or more servers. This approach is predictive, in that an increase in current usage, and a consequent increase in heat dissipation, generally precedes the resultant rise in device temperature.

[0027] In other embodiments where predictive control is implemented, local variations in heat generation within the device array are predicted according to anticipated changes in the workloads that each electronic device will be subjected to. For example, an anticipated workload can be inferred from network activity, and / or on from an internal server scheduler that queues tasks to be performed by the electronic devices. Both are normally precursors to an incoming request for processing of data that will result in a spike in the activity of a server or other electronic device, and hence an increase in heat dissipation.

[0028] In various embodiments that implement manifolds to distribute the cooling liquid, “passive” manifolds 114, 116, as illustrated in Figs. IB and IC, which direct cooling liquid through an array of electronic devices 100 in a fixed flow pattern, are replaced by “active” manifolds that can be directed by acontroller to vary the flow pattern of the cooling liquid through the cooling pipes or immersion tank, thereby redirecting cooling liquid, as needed, to increase the cooling of hotspots without applying excessive cooing to other regions. In embodiments, the flow pattern of the cooling liquid can be changed as rapidly as every minute, or even every second, if necessary.

[0029] It will be understood that the term “hotspot,” as used herein, refers to any region in a device array that requires, or is predicted to require, additional cooling, so as to maintain optimal conditions. In some embodiments, any region within a device array that is, or will be, generating more than an average amount of heat per square cm is considered to be a hotspot. In embodiments, the active manifolds of the disclosed invention are adjusted to compensate for changes in the heat being generated by the various electronic devices, and thereby to provide a more uniform temperature within the device array, while control of the overall flow rate of the cooling liquid through the device array controls the average temperature.

[0030] In various embodiments, at least one pair of active manifolds is provided, being an inlet manifold and a corresponding outlet manifold, arranged on horizontally opposing sides of the electronic device cabinet or immersion tank. In other embodiments, active manifolds are placed on all four sides of the cabinet or immersion tank. Some embodiments include at least one active manifold located on the bottom of the cabinet or immersion tank.

[0031] In various embodiments, the adjustment of the active manifolds is reactive, predictive, or both. According to measured and / or predicted variations in temperature throughout the components and / or devices, the controller adjusts the overall flow of the cooling liquid through the device array, and also causes the active manifolds to direct proportionally more of the cooling liquid toward the regions of the array that are, or will soon be, generating excessive heat, while directing proportionally less of the cooling liquid toward other regions.

[0032] In embodiments where reactive manifold control is implemented, the controller receives temperature measurements from a plurality of locations within the device array. In some of these embodiments, at least one of the temperature measurements is provided by a sensor that is integral to an integrated circuit (IC), such as a central processing unit (CPU), and is configured to report a temperature of the IC. Other embodiments include at least one temperature sensor configured to measure a local temperature of the cooling liquid or ambient air.

[0033] Predictive control of the active manifolds, as is implemented in some embodiments, enables the active manifolds to direct additional coolant to locations within the device array where it will be most needed, in advance of any actual changes in temperature, and in some embodiments also in advance of any actual changes in heat generation, thereby minimizing or avoiding temperature fluctuations within the device array, and providing increased IC longevity and improved operational stability. In some embodiments where predictive manifold control is implemented, changes in the heat output of the electronic components are anticipated by monitoring the amount of current that is drawn by at least one of the various components, such as one or more servers or other electronic devices. This approach is predictive, in that an increase in current usage, and a consequent increase in heat dissipation, generally precedes the resultant rise in component temperature.

[0034] In other embodiments where predictive manifold control is implemented, local variations in heat generation within the device array are predicted according to anticipated changes in the workloads that each electronic device or component will be subjected to. For example, an anticipated workload can be inferred from network activity, and / or on from an internal server scheduler that queues tasks to be performed by the device array. Both are normally precursors to an incoming request for processing of data that will result in a spike in the activity of a server or other data processing device, and hence a local increase in heat generation.

[0035] According to the present invention, each of the active manifolds includes an inlet and a plurality of spaced apart manifold outlets, or an outlet and a plurality of spaced apart manifold inlets, and further includes a plurality of adjustable flow control devices that separately control the flow of cooling liquid into or out from each of the plurality of inlets or outlets. In embodiments, at least one of the active manifolds comprises a plurality of variable speed pumps, such as variable frequency pumps, which are separately controlled by the controller and arranged such that each pump is directed to a separate inlet or outlet. In embodiments, at least one of the pumps is an “intelligent” rotary pump that is configured to maintain a specified flow output even under changing flow impedances, such as might occur if a downstream filter becomes partially blocked. In some of these embodiments, the intelligent pump estimates the liquid flow according to a pre-calibrated relationship between impeller speed and impeller torque, and then adjusts the impeller speed accordingly to provide the specified flow output.

[0036] In other embodiments, at least one of the active manifolds comprises a plurality of remotely controlled valves, vents, and / or baffles, which are separately adjusted by the controller and arranged such that each valve, vent, or baffle is directed to a separate inlet or outlet. Depending on the embodiment, the valves, vents, and / or baffles can be electronically adjustable, pneumatically adjustable, hydraulically adjustable, or adjustable by any other means known in the art. In still other embodiments, at least one of the active manifolds includes a combination of variable speed pumps and remotely controlled valves, vents, and / or baffles which, in combination, separately control the flow into or out of each of the plurality of inlets and outlets.

[0037] A first general aspect of the present invention is a cooling system configured to cool a plurality of electronic devices. The cooling system comprises a controller, at least one cooling loop through which a coolant can flow as a liquid into proximity and thermal communication with the electronic devices, thereby absorbing heat from the electronic devices, a heat transfer apparatus configured toremove the absorbed heat from the coolant, and a variable speed liquid coolant pump configured to cause the coolant to flow through the cooling loop. The controller is configured to adjust a liquid coolant pump operating speed of the variable speed liquid coolant pump, thereby varying a flow rate of the coolant through the cooling loop, according changes in cooling requirements of the electronic devices.

[0038] In embodiments, the cooling system is configured to ensure that the coolant remains in a liquid state after absorbing the heat from the electronic devices.

[0039] In any of the above embodiments, the cooling system can be configured to allow the coolant to vaporize due to absorbing the heat from the electronic devices, the heat transfer apparatus can be a condenser configured to accept the vaporized coolant and return the coolant to the liquid state, and the cooling system can further include a variable speed vapor suction pump having a vapor suction pump operating speed that is controlled by the controller, the variable speed vapor suction pump being configured to draw the vaporized coolant from the electronic devices and direct the vaporized coolant to the condenser.

[0040] In any of the above embodiments, the variable speed liquid coolant pump can be a variable frequency pump.

[0041] In any of the above embodiments, the cooling loop can further include an isolation valve configured to stop the flow of the coolant through the cooling loop.

[0042] In any of the above embodiments, the cooling system can include a plurality of cooling loops and a corresponding plurality of variable speed liquid coolant pumps under control of the controller, each of the cooling loops being associated with a corresponding one of the plurality of variable speed liquid coolant pumps, each of the cooling loops being configured to direct the coolant into proximity and thermal communication with a corresponding subset of the electronic devices. In some of these embodiments, the controller is configured toadjust the operating rates of the variable speed liquid coolant pumps according to changes in cooling requirements of each of the subsets of the electronic devices. In any of these embodiment, the cooling system can be configured to allow the coolant to vaporize due to absorbing the heat from the electronic devices, the heat transfer apparatus can be a condenser configured to accept the vaporized coolant and return the coolant to the liquid state, each of the cooling loops can further include a variable speed vapor suction pump under control of the controller, and each of the variable speed vapor suction pumps can be configured to draw the vaporized coolant from a respective one of the subsets of the electronic devices and direct the vaporized coolant to the condenser.

[0043] Any of these embodiments can further include a plurality of isolation valves, each of the isolation valves being configured to stop the flow of the coolant through a corresponding one of the cooling loops. In some of these embodiments, each of the plurality of isolation valves is under separate control of the controller.

[0044] In any of the above embodiments, the cooling loop can include a plurality of branches, each of the branches comprising a flow path configured to direct a corresponding flow of the coolant through the flow path into proximity and thermal communication with a corresponding subset of the electronic devices. Some of these embodiments further include a plurality of isolation valves, each of the isolation valves being configured to stop the flow of the coolant through an associated one of the flow paths. In some of these embodiments, each of the plurality of isolation valves is under separate control of the controller.

[0045] In any of the above embodiments, the controller can be configured to receive a temperature measurement from a first temperature sensor proximate the electronic devices. In some of these embodiments, the first temperature sensor is integral to a first electronic device of the plurality of electronic devices, and is configured to measure an internal temperature of the first electronic device.

[0046] In any of the above embodiments, the controller can be configured to predict a heat dissipation increase of the electronic devices in advance of a temperature increase therein. In some of these embodiments, the controller is configured to predict the local heat dissipation increase of the electronic devices at least in part according to at least one of: an amount of current flowing through the electronic devices, and an amount of electrical power flowing to the electronic devices. In any of these embodiments, the controller can be configured to predict the heat dissipation increase of the electronic devices at least in part according to a workload prediction that is applicable to the electronic devices. And in some of these embodiments, the workload prediction is inferred from information regarding network activity, and / or information derived from an internal server scheduler that queues tasks to be performed by the electronic devices.

[0047] A second general aspect of the present invention is a cooling system configured to cool a plurality of electronic devices. The cooling system includes a controller, an active inlet manifold comprising a plurality of spaced apart manifold outlets through which a cooling liquid can flow into proximity and thermal communication with the electronic devices, and an active outlet manifold comprising a plurality of spaced apart manifold inlets through which the cooling liquid can flow from proximity with the electronic devices into the active outlet manifold.

[0048] The active inlet manifold includes a first plurality of remotely adjustable flow control devices that separately control the flow of the cooling liquid through each of the plurality of manifold outlets, and the active outlet manifold includes a second plurality of remotely adjustable flow control devices that separately control the flow of the cooling liquid through each of the plurality of manifold inlets. The controller is configured to detect and / or predict a local heat dissipation increase in a hotspot region of the electronic devices, and adjust the flow control devices of the active manifolds to cause more of the cooling liquid to flow in thermal communication with the hotspot region as compared to other regions proximate the electronic devices.

[0049] In some embodiments, the cooling liquid is directed through spaced apart pipes that flow in thermal communication with the electronic devices, while in other embodiments the electronic devices are enclosed within an immersion tank, and the cooling liquid fills and flows through the immersion tank in direct physical contact with the electronic devices.

[0050] In any of the above embodiments, at least one of the first and second pluralities of remotely adjustable flow control devices can be a variable speed pump. In some of these embodiments the variable speed pump is an intelligent rotary pump that is configured to maintain a controller specified flow rate of the cooling liquid therethrough.

[0051] In any of the above embodiments, at least one of the first and second pluralities of adjustable flow control devices can be a remotely adjustable valve, vent, or baffle.

[0052] In any of the above embodiments, at least one of the inlet active manifold and the outlet active manifold can include a variable speed pump and a remotely adjustable valve, vent, or baffle.

[0053] In any of the above embodiments, the active inlet manifold can be a first active inlet manifold, the active outlet manifold can be a first active outlet manifold, the first active inlet and outlet manifolds can be arranged on opposing first and second sides of the electronic devices, and configured to direct the cooling liquid in a first horizontal direction past the electronic devices, and the cooling system can further include a second active inlet manifold and a second active outlet manifold, the second active inlet and outlet manifolds being located on opposing third and fourth sides of the electronic devices and configured to direct the cooling liquid in a second horizontal direction past the electronic devices, the second horizontal direction being orthogonal to the first horizontal direction.

[0054] Any of the above embodiments can further include at least one of a third active inlet manifold located below the electronic devices and a third active outlet manifold located below the electronic devices.

[0055] In any of the above embodiments, the controller can be configured to receive a temperature measurement from a first temperature sensor proximate the electronic devices. In some of these embodiments, the first temperature sensor is integral to a first electronic component of the plurality of electronic devices, and is configured to measure an internal temperature of the first electronic component.

[0056] In any of the above embodiments, the controller can be configured to predict the local heat dissipation increase in the hotspot region of the immersion tank in advance of a temperature increase therein. In some of these embodiments, the controller is configured to predict the local heat dissipation increase in the hotspot region of the immersion tank at least in part according to a measurement of an electrical status of a first electronic component of the plurality of electronic devices, the electrical status being at least one of: an amount of current flowing through the first electronic component; an electrical voltage applied to the first electronic component; and an amount of electrical power flowing to the first electronic component. In any of these embodiments, the controller can be configured to predict the local heat dissipation increase in the hotspot region of the immersion tank at least in part according to a workload prediction that is applicable to the first electronic device. In some of these embodiments, the workload prediction is inferred from information regarding network activity, and / or information derived from an internal server scheduler that queues tasks to be performed by the first electronic device.

[0057] In any of the above embodiments, the plurality of electronic devices can be arranged in at least one of a plurality of horizontal rows, a plurality of horizontal columns, and a plurality of vertical tiers.

[0058] A third general aspect of the present invention is a method of cooling a plurality of electronic devices. The method includes providing a cooling systemcomprising: a controller; an active inlet manifold comprising a plurality of spaced apart outlets through which a cooling liquid can flow into proximity and thermal communication with the electronic devices; and an active outlet manifold comprising a plurality of spaced apart inlets through which the cooling liquid can flow from proximity with the electronic devices into the active outlet manifold; and the active inlet manifold includes a first plurality of remotely adjustable flow control devices that separately control the flow of the cooling liquid through each of the plurality of outlets, and the active outlet manifold includes a second plurality of remotely adjustable flow control devices that separately control the flow of the cooling liquid through each of the plurality of inlets.

[0059] The method includes at least one of detecting and predicting a local heat dissipation increase in a hotspot region of the electronic devices, and adjusting by the controller of the flow control devices of the active manifolds to cause more of the cooling liquid to flow in thermal communication with the hotspot region as compared to other regions proximate the electronic devices.

[0060] In embodiments, predicting the local heat dissipation increase in the hotspot region comprises receiving a measurement of an electrical status of a first electronic component of the plurality of electronic devices, the electrical status being at least one of: an amount of current flowing through the first electronic component; an electrical voltage applied to the first electronic component; and an amount of electrical power flowing to the first electronic component.

[0061] Any of the above embodiments, predicting the local temperature increase in the hotspot region of the immersion tank can include receiving a workload prediction applicable to a first electronic component of the plurality of electronic devices. And in some of these embodiments, receiving the workload prediction comprises inferring an anticipated workload applicable to the first electronic component from information regarding network activity, and / or information derived from an internal server scheduler that queues tasks to be performed by the electronic devices.

[0062] The features and advantages described herein are not all-inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and not to limit the scope of the inventive subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Fig. 1A illustrates a single cooling loop of the prior art in which the coolant is vaporized by absorbed heat, and the coolant flow rate is controlled by fixed speed pumps in combination with a flow control valve;

[0064] Fig. IB is top view of a piped cooling liquid system of the prior art;

[0065] Fig. 1C is top view of a liquid immersion cooling system of the prior art;

[0066] Fig. 2A illustrates a single cooling loop in an embodiment of the present invention in which the coolant is vaporized by absorbed heat, and the coolant flow rate is controlled by controlling the operating speeds of a variable speed liquid coolant pump and a variable speed vapor suction pump, without reliance on a flow control valve;

[0067] Fig. 2B is a top view of a piped liquid cooling system in an embodiment of the present invention that implements a plurality of cooling loops similar to Fig. 2A, in which the flow control devices are variable speed pumps included in active manifolds;

[0068] Fig. 2C illustrates an embodiment in which a single cooling loop is branched into a plurality of flow paths, each including a dedicated, remotely controlled isolation valve;

[0069] Fig. 2D is a top view of a liquid immersion cooling system in an embodiment of the present invention in which the flow control devices of the active manifolds are variable speed pumps;

[0070] Fig. 2E is a top view of an embodiment of the present invention that includes four cooling liquid immersion tanks arranged side-by-side;

[0071] Fig.3 is a top view of an embodiment of the present invention in which the flow control devices of the active manifolds are variable valves, vents, and / or baffles;

[0072] Fig. 4 is a top view of an embodiment of the present invention in which the flow control devices of the active manifolds are a combination of variable speed pumps with variable valves, vents, and / or baffles;

[0073] Fig. 5 is a top view of an embodiment of the present invention that comprises two opposing pairs of active inlet and outlet manifolds arranged on all four sides of the immersion tank;

[0074] Fig. 6 is a bottom view of an embodiment of the present invention that includes active manifolds on the bottom of the immersion tank; and

[0075] Fig. 7 is a flow diagram that illustrates a method embodiment of the present invention.DETAILED DESCRIPTION

[0076] The present invention is an electronic device liquid cooling system that can provide adequate liquid cooling to transitory electronic device hotspots, while reducing the required cooling capacity of the overall system and while optimizing the overall energy efficiency of the cooling system.

[0077] With reference to Fig. 2A, in embodiments, rather than operating one or more coolant pumps 106, 136 at fixed operating rates, while possibly adjusting thecoolant flow rate(s) using one or more flow control valves 122, the present invention implements one or more variable speed coolant pumps 214, 216, and adjusts the coolant flow through the cooling conduits 118 by varying the operating speeds of the one or more coolant pumps 214, 216, without reliance on flow control valves 122. Accordingly, cooling of the electronic devices 100 is optimized by adjusting the coolant pumping rate through a cooling loop according to variable cooling demands, while the energy efficiency of the cooling system is optimized by reducing the pump operating rates, and thereby reducing the energy consumption of the pumps 214, 216, whenever maximum cooling is not required. In the embodiment of Fig. 2A, the variable speed pumps 214, 216 are variable frequency pumps that are controlled by a variable frequency drive (VFD) 236.

[0078] In some embodiment where the coolant remains a liquid throughout its cooling cycle, the cooling loop comprises only a single variable speed liquid coolant pump 214. In the embodiment of Fig. 2A, cooling of the electronic devices 100 causes the coolant 102 to vaporize 112. Accordingly, the illustrated cooling loop comprises at least one variable speed liquid coolant pump 214 and at least one variable speed vapor suction pump 216. The embodiment of Fig. 2 also comprises an isolation valve 238 that is included to meet safety requirements.

[0079] In the embodiment of Fig. 2A, all of the electronic devices 100 are cooled by a single cooling loop. With reference to Fig. 2B, in other embodiments a plurality of cooling loops are implemented, each comprising at least one cooling conduit 118, and each comprising at least one dedicated variable speed coolant pump 214. In the illustrated embodiment, the cooling liquid 102 is vaporized as it cools the electronic devices 100. Accordingly, the embodiment of Fig. 2B comprises a dedicated liquid coolant pump 214 and a dedicated coolant vapor suction pump 216 for each of the cooling conduits 118. By separately adjusting the operating speeds of the variable speed coolant pumps 214, 216, and thereby separately adjusting the coolant flow rates in the cooling conduits 118, specific racks, set of racks, set of chips in a rack, and / or specific chipsets can each be targeted with an optimal amount of cooling, rather than increasing the cooling ofall of the electronic devices 100 due to the needs of one sub-population that may require maximum cooling.

[0080] Embodiments include redundant variable speed pumps 214, 216 configured such that, in case of a pump failure, the operating speeds of the remaining pumps can be increased in compensation. In some of these embodiments, failure of a vapor suction pump 216 included in a cooling loop can be compensated by increasing the operating speed of a liquid coolant pump 214 that is also included in the cooling loop, and vice versa.

[0081] In various embodiments, adjustment of the operating speeds of the variable speed coolant pumps 214, 216 is reactive, predictive, or both. The embodiment of Fig. 2B implements reactive adjustment of the coolant pumps 214, 216. according to temperature data 206 received by the controller 204 from one or more locations within the array of electronic devices 100. In various embodiments, sensors located near or within the electronic devices 100 provide the temperature data. In embodiments, at least some of the temperature data is provided by a sensor that is integral to an integrated circuit (IC), such as a central processing unit (CPU), and is configured to report a temperature of the IC. Other embodiments include at least one temperature sensor configured to measure a local temperature of the coolant or ambient air.

[0082] Predictive control of the variable speed pumps, as is implemented in some embodiments, enables the speeds of the coolant pumps 214, 216 to be adjusted in advance of any actual changes in temperature, and in some embodiments also in advance of any actual changes in power consumption by the electronic devices 100, thereby minimizing or avoiding temperature fluctuations of the electronic devices 100, and providing increased longevity and improved operational stability of the devices 100. In some of these embodiments, changes in the heat output of the electronic devices 100 are anticipated by monitoring the amount of current that is drawn by at least one of the electronic devices 100, such as one or more servers. This approach is predictive, in that an increase in currentusage, and a consequent increase in heat dissipation, generally precedes the resultant rise in device temperature.

[0083] In other embodiments where predictive control is implemented, local variations in heat generation of the electronic devices 100 are predicted according to anticipated changes in the workloads that each electronic device 100 will be subjected to. For example, an anticipated workload can be inferred from network activity, and / or on from an internal server scheduler that queues tasks to be performed by the electronic devices. Both are normally precursors to an incoming request for processing of data that will result in a spike in the activity of a server or other electronic device 100, and hence an increase in heat dissipation.

[0084] With reference to Fig. 2C, in embodiments at least one of the cooling loops is branched, such that each of the conduits 118 is a separate flow path. In some of these embodiments, isolation valves 238 included in the flow paths 118 are opened and closed to cause the coolant to flow only through those flow paths 118 where it is needed. In the embodiment of Fig. 2C, the coolant is not vaporized when cooling the electronic devices 100, such that only one variable speed liquid cooling pump 214 is implemented. The liquid coolant 102 is delivered to an inlet manifold 200 which divides the coolant flow among the plurality of flow paths 118. After cooling the electronic devices 100, the liquid coolant 102 flows into an outlet manifold 202, which combines the flows from all of the flow paths 118 and directs the combined flow to the liquid coolant pump 214, and thence to the heat transfer apparatus 108.

[0085] In embodiments the “passive” inlet 114 and outlet 116 manifolds of Figs. IB and 1C, which direct cooling liquid through the device array in a fixed or manually adjusted flow pattern, are replaced by “active” inlet 200 and outlet 202 manifolds that can be remotely controlled and adjusted via wired or wireless interconnection 208 by a controller 204 to vary the flow pattern of the cooling liquid, thereby redirecting cooling liquid 102, as needed, to increase the cooling of hotspots without applying excessive cooing to other regions within the devicearray. In embodiments, the cooling flow pattern can be adjusted by the controller every minute, or even every second, as needed.

[0086] It will be understood that the term “hotspot,” as used herein, refers to any region within the device array that requires, or is predicted to require, additional cooling, so as to maintain optimal conditions. In some embodiments, any region within the device array that is generating, or is expected to generate, heat that is above an average heat generation of the devices within the device array is considered to be a hotspot. In embodiments, the active manifolds 200, 202 of the disclosed invention are adjusted to compensate for actual and / or anticipated changes in the temperature and / or the heat being generated in a local region within the device array, and thereby to provide a more uniform temperature within the device array, while adjustment of the overall flow rate of the cooling liquid controls the average temperature within the device array.

[0087] In various embodiments, the adjustments to the active manifolds 200, 202 are reactive, predictive, or both. According to measured and / or predicted variations in heat generation within the device array, the controller 204 adjusts the overall flow of the cooling liquid, and also causes the active manifolds 200, 202 to direct proportionally more of the cooling liquid toward the regions that are, or will soon be, generating excessive heat, while directing proportionally less of the cooling liquid toward other regions within the device array.

[0088] In embodiments where reactive manifold control is implemented, the controller 204 receives temperature measurements 206 from a plurality of locations within the device array. In some of these embodiments, at least one of the temperature measurements is provided by a sensor that is integral to an integrated circuit (IC), such as a central processing unit (CPU), and is configured to report a temperature of the IC. Other embodiments include at least one temperature sensor configured to measure a local temperature of the device array. According to measured temperature differences, the controller 204 adjusts the overall flow of the cooling liquid through the device array, and also causes theactive manifolds 200, 202 to direct proportionally more of the cooling liquid 102 toward hotter regions, and less of the cooling liquid 102 toward the cooler regions.

[0089] In embodiments where predictive control of the active manifolds is implemented, the controller 204 is able to cause the active manifolds 200, 202 to direct additional coolant to locations within the device array where it will be most needed in advance of actual changes in temperature, thereby minimizing or avoiding temperature fluctuations within the device array, and providing increased IC longevity and improved operational stability. In some embodiments where predictive manifold control is implemented, changes in the temperatures of the electronic components 100 are anticipated by monitoring the amount of current that is drawn by at least one of the components 100, such as one or more servers 100 or other electronic devices. Based on a known or measured voltage that is applied to the component 100, the power consumption of the component 100 can be determined. This approach is predictive, in that an increase in current usage, and a consequent increase in heat dissipation, generally precedes the resultant rise in component temperature.

[0090] In other embodiments where predictive manifold control is implemented, local variations in temperature within the device array are predicted according to anticipated changes in the workloads that each device or component 100 will be subjected to. For example, an anticipated workload can be inferred from network activity, and / or on from an internal server scheduler that queues tasks for the electronic devices 100. Both are normally precursors to an incoming request for processing of data that will result in a spike in the activity of a server or other data processing device 100, and hence a local increase in heat generation.

[0091] According to embodiments of the present invention, each of the active manifolds is either an inlet manifold 200, which includes an inlet 218 and a plurality of spaced apart active manifold outlets 210, or an outlet manifold 202, which includes an outlet 220 and a plurality of spaced apart manifold inlets 212. In the piped liquid cooling system of Figs. 2B and 2C, the manifold outlets 210and inlets 212 are connect to liquid cooling conduits 118 that convey the cooling liquid 102 through the array of electronic devices 100. In the liquid immersion cooling system of Fig. 2D, the active manifold outlets 210 connect to inlet openings 228 of the immersion tank 104, and the active manifold inlets 212 connect to outlet openings 230 of the immersion tank 104.

[0092] In these embodiments, each active manifold 200, 202 further includes a plurality of-remotely controlled adjustable flow control devices that separately control the flow of the cooling liquid 102 into or out of each of the plurality of active manifold outlets 210 or inlets 212.

[0093] In the embodiments of Figs. 2B and 2C, each of the active manifolds 200, 202 comprises a plurality of variable speed pumps 214, 216, such as variable frequency pumps, which are separately controlled by the controller 204 and arranged such that each pump 214, 216 is directed to a separate active manifold outlet 210 or inlet 212. In embodiments, at least one of the pumps 214, 216 is an “intelligent” rotary pump that is configured to maintain a specified flow output even under changing flow impedances, such as might occur if a downstream filter becomes partially blocked. In some of these embodiments, the intelligent pump estimates the liquid flow according to a pre-calibrated relationship between impeller speed and impeller torque, and then adjusts the impeller speed accordingly to provide the specified flow output

[0094] By actively controlling the speeds of the pumps 214, 216, the illustrated embodiments mitigate and / or prevent the occurrence of hotspots within the array of electronic devices 100. As an example, if, at a certain time, the “uppermost” server 100 in the device array is generating more heat than the other servers 100, or is expected to do so, the controller 204 adjusts the “uppermost” pumps 214, 216 to operate at higher speeds compared to the other pumps 214, 216. Note that the term “uppermost” is used herein to refer to an uppermost location in the figure. Because Figs. 2B and 2C are top views, the “uppermost” and “lowermost” regions of the device arrays in Figs. 2B and 2C are actually horizontally opposed sideregions of the cabinet 120 or immersion tank 104 that contains the array of electronic devices 100, and not a vertical top or bottom of the cabinet 120 or immersion tank 104.

[0095] The embodiments of Figs. 2B - 2D include only one pair of active inlet 200 and outlet 202 manifolds located on opposite sides of the cabinet 120 or immersion tank 104. This arrangement can be desirable, due to its simplicity and to minimize cost. Also, as illustrated in Fig. 2E, this approach can be preferable in embodiments where a plurality of cabinets 120 or immersion tanks 104 are arranged side-by-side in a row. In the embodiment of Fig. 2E, in which four immersion tanks 104 are implemented, placement of the active manifolds 200, 202 along only one pair of opposing sides of each immersion tank 104 is advantageous because it allows the immersion tanks 104 to be placed close together side-by- side.

[0096] The inlet 222 and outlet 224 external pipes that deliver cooling liquid to the inlets 218 of the inlet active manifolds 200 and withdraw cooling liquid from the outlets 220 of the outlet active manifolds 202 are also visible in Fig. 2E. In addition, it can be seen that, in the embodiment of Fig. 2E, each immersion tank 104 contains 15 electronic devices 100 arranged as an array of three horizontal rows and five horizontal columns.

[0097] With reference to Fig. 3, in other embodiments at least one of the active manifolds 300, 302 comprises a plurality of adjustable valves, vents, or baffles 304, 306, which are separately adjusted by the controller 204 and arranged such that each valve, vent, or baffle 304, 306 is directed to a separate active manifold outlet 210 or inlet 212. In the illustrated embodiment, the cooling liquid is circulated through a liquid immersion tank 104 by a single, external pump 308, which controls the overall flow rate of the cooling liquid, and hence the average temperature within the immersion tank 104, while the valves, vents, and / or baffles 304, 306 control the distribution of the cooling liquid flow within the immersion tank 104. Similar embodiments employ remotely adjustable valves, vents, orbaffles 304, 306 to direct cooling liquid through cooling pipes 118. Depending on the embodiment, the valves, vents, and / or baffles 304, 306 can be electronically adjustable, pneumatically adjustable, hydraulically adjustable, or adjustable by any other means known in the art.

[0098] With reference to Fig. 4, in still other embodiments at least one of the active manifolds 400, 402 includes a combination of variable speed pumps 404, 406 with associated valves, vents, and / or baffles 408, 410 which, in combination, separately control the flow into and out of each of the active manifold outlets 210 and inlets 212. Fig. 4 illustrates this approach for a liquid immersion cooling embodiment. Similar embodiments employ a combination of variable speed pumps 404, 406 with associated valves, vents, and / or baffles 408, 410 to direct cooling liquid through cooling conduits 118.

[0099] Fig. 5 is a top view of an immersion cooling embodiment in which a single immersion tank 104 is provided with active manifolds 500, 502, 504, 506 on all four sides, arranged in two orthogonal opposing pairs of active inlet manifolds 500, 504 and active outlet manifolds 502, 506. The controller 204 has been omitted from the figure so that other features can be more easily seen. In the illustrated embodiment, the immersion tank 104 contains 20 electronic devices 100, arranged in an array of four horizontal rows and five horizontal columns. Similar embodiments provide an electronic device cabinet 120 with active manifolds 500, 502, 504, 506 on all four sides arranged in two orthogonal opposing pairs of active inlet manifolds 500, 504 and active outlet manifolds 502, 506.

[0100] By providing two orthogonal pairs of opposing active manifolds 500, 502, 504, 506, the controller 204 is able to maximize the flow of the cooling liquid to any of the electronic devices 100 that is operating as a hotspot. In the illustrated embodiment, the electronic device 100 in the second row and fourth column is shown with a heavy outline, indicating that it is currently a hotspot. In response, the controller 204 will cause the active manifolds 500, 502 to direct ahigher percentage of cooling liquid through the fourth column, and will cause the orthogonal pair of active manifolds 504, 506 to direct a higher percentage of cooling liquid through the second row.

[0101] With reference to the bottom view of Fig. 6, in various embodiments, for example when the electronic devices 100 are arranged in vertically offset tiers, at least one active manifold 600 is located at the bottom of the electronic device cabinet 120 or immersion tank 104. In the illustrated embodiment, three active manifolds 600 are located at the bottom of an immersion tank 104.

[0102] Some embodiments provide an opposing active manifold at the top of the device array cabinet 120 or immersion tank 104. However, this can be problematic in the case of an immersion tank 104, in that the level of the cooling liquid at the top of the tank may vary, and the margin of cooling liquid above the electronic devices 100 may be small. Instead, in the embodiment of Fig. 6, the cooling liquid that is introduced by the bottom active manifolds 600 is drawn out of the immersion tank 104 through the active outlet manifold 202 that is located on the side of the immersion tank 104, together with the cooling liquid that is introduced from the opposing side horizontal active inlet manifold 200.According to this approach, in embodiments, the active outlet manifold 202 has a sufficient liquid flow capacity to receive all of the cooling liquid that is introduced into the immersion tank 104 by both the side inlet active manifold 200 and the bottom active manifolds 600.

[0103] Fig. 7 illustrates a method embodiment of the present invention that implements predictive control of active manifolds 200, 202. According to the disclosed embodiment, a future appearance of a hotspot is predicted 700 by the controller according to workloads that are scheduled to be assigned to the electronic devices 100, and / or measurements of current flows to the electronic devices 100, as described above. Based on this prediction, the controller adjusts 702 the active manifolds 200, 202 to increase the flow of the cooling liquid to the anticipated hotspot. In the illustrated embodiment, the controller also receivesinput from a sensor that monitors 704 the temperature at the hotspot location, to ensure that the cooling is adequate, and to adjusts 706 the cooling flow still further if needed.

[0104] The foregoing description of the embodiments of the invention has been presented for the purposes of illustration and description. Each and every page of this submission, and all contents thereon, however characterized, identified, or numbered, is considered a substantive part of this application for all purposes, irrespective of form or placement within the application. This specification is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of this disclosure.

[0105] Although the present application is shown in a limited number of forms, the scope of the disclosure is not limited to just these forms, but is amenable to various changes and modifications. The present application does not explicitly recite all possible combinations of features that fall within the scope of the disclosure. The features disclosed herein for the various embodiments can generally be interchanged and combined into any combinations that are not selfcontradictory without departing from the scope of the disclosure. In particular, the limitations presented in dependent claims below can be combined with their corresponding independent claims in any number and in any order without departing from the scope of this disclosure, unless the dependent claims are logically incompatible with each other.

Claims

CLAIMSWhat is claimed is:

1. A cooling system configured to cool a plurality of electronic devices, the cooling system comprising: a controller; at least one cooling loop through which a coolant can flow as a liquid into proximity and thermal communication with the electronic devices, thereby absorbing heat from the electronic devices; a heat transfer apparatus configured to remove the absorbed heat from the coolant; and a variable speed liquid coolant pump configured to cause the coolant to flow through the cooling loop; wherein the controller is configured to adjust a liquid coolant pump operating speed of the variable speed liquid coolant pump, thereby varying a flow rate of the coolant through the cooling loop, according changes in cooling requirements of the electronic devices.

2. The cooling system of claim 1, wherein the cooling system is configured to ensure that the coolant remains in a liquid state after absorbing the heat from the electronic devices.

3. The cooling system of any preceding claim, wherein: the cooling system is configured to allow the coolant to vaporize due to absorbing the heat from the electronic devices; the heat transfer apparatus is a condenser configured to accept the vaporized coolant and return the coolant to the liquid state; and the cooling system further comprises a variable speed vapor suction pump having a vapor suction pump operating speed that is controlled by the controller, the variable speed vapor suction pump being configured to draw the vaporized coolant from the electronic devices and direct the vaporized coolant to the condenser.

4. The cooling system of any preceding claim, wherein the variable speed liquid coolant pump is a variable frequency pump.

5. The cooling system of any preceding claim, wherein the cooling loop further comprises an isolation valve configured to stop the flow of the coolant through the cooling loop.

6. The cooling system of any preceding claim, wherein the cooling system comprises a plurality of cooling loops and a corresponding plurality of variable speed liquid coolant pumps under control of the controller, each of the cooling loops being associated with a corresponding one of the plurality of variable speed liquid coolant pumps, each of the cooling loops being configured to direct the coolant into proximity and thermal communication with a corresponding subset of the electronic devices.

7. The cooling system of claim 6, wherein the controller is configured to adjust the operating rates of the variable speed liquid coolant pumps according to changes in cooling requirements of each of the subsets of the electronic devices.

8. The cooling system of claim 6 or claim 7, wherein: the cooling system is configured to allow the coolant to vaporize due to absorbing the heat from the electronic devices; the heat transfer apparatus is a condenser configured to accept the vaporized coolant and return the coolant to the liquid state; each of the cooling loops further comprises a variable speed vapor suction pump under control of the controller; and each of the variable speed vapor suction pumps is configured to draw the vaporized coolant from a respective one of the subsets of the electronic devices and direct the vaporized coolant to the condenser.

9. The cooling system of any of claims 6-8, further comprising a plurality of isolation valves, each of the isolation valves being configured to stop the flow of the coolant through a corresponding one of the cooling loops.

10. The cooling system of claim 9, wherein each of the plurality of isolation valves is under separate control of the controller.

11. The cooling system of any preceding claim, wherein the cooling loop comprises a plurality of branches, each of the branches comprising a flow path configured to direct a corresponding flow of the coolant through the flow path into proximity and thermal communication with a corresponding subset of the electronic devices.

12. The cooling system of claim 11, further comprising a plurality of isolation valves, each of the isolation valves being configured to stop the flow of the coolant through an associated one of the flow paths.

13. The cooling system of claim 12, wherein each of the plurality of isolation valves is under separate control of the controller.

14. A cooling system configured to cool a plurality of electronic devices, the cooling system comprising: a controller; an active inlet manifold comprising a plurality of spaced apart manifold outlets through which a cooling liquid can flow into proximity and thermal communication with the electronic devices; and an active outlet manifold comprising a plurality of spaced apart manifold inlets through which the cooling liquid can flow from proximity with the electronic devices into the active outlet manifold; wherein the active inlet manifold includes a first plurality of remotely adjustable flow control devices that separately control the flow of the cooling liquid through each of the plurality of manifold outlets, and the active outlet manifold includes a second plurality of remotely adjustable flow control devices that separately control the flow of the cooling liquid through each of the plurality of manifold inlets; and wherein the controller is configured to:at least one of detect and predict a local heat dissipation increase in a hotspot region of the electronic devices; and adjust the flow control devices of the active manifolds to cause more of the cooling liquid to flow in thermal communication with the hotspot region as compared to other regions proximate the electronic devices.

15. The cooling system of claim 14, wherein the cooling liquid is directed through spaced apart pipes that flow in thermal communication with the electronic devices.

16. The cooling system of claim 14 or claim 15, wherein the electronic devices are enclosed within an immersion tank, and wherein the cooling liquid fills and flows through the immersion tank in direct physical contact with the electronic devices.

17. The cooling system of any of claims 14-16, wherein at least one of the first and second pluralities of remotely adjustable flow control devices is a variable speed pump.

18. The cooling system of claim 17, wherein the variable speed pump is an intelligent rotary pump that is configured to maintain a controller specified flow rate of the cooling liquid therethrough.

19. The cooling system of any of claims 14-18, wherein at least one of the first and second pluralities of adjustable flow control devices is a remotely adjustable valve, vent, or baffle.

20. The cooling system of any of claims 14-19, wherein at least one of the inlet active manifold and the outlet active manifold comprises a variable speed pump and a remotely adjustable valve, vent, or baffle.

21. The cooling system of any of claims 14-20 wherein: the active inlet manifold is a first active inlet manifold; the active outlet manifold is a first active outlet manifold;the first active inlet and outlet manifolds are arranged on opposing first and second sides of the electronic devices, and are configured to direct the cooling liquid in a first horizontal direction past the electronic devices; and the cooling system further comprises a second active inlet manifold and a second active outlet manifold, the second active inlet and outlet manifolds being located on opposing third and fourth sides of the electronic devices and configured to direct the cooling liquid in a second horizontal direction past the electronic devices, the second horizontal direction being orthogonal to the first horizontal direction.

22. The cooling system of any of claims 14-21, further comprising at least one of a third active inlet manifold located below the electronic devices and a third active outlet manifold located below the electronic devices.

23. The cooling system of any preceding claim, wherein the controller is configured to receive a temperature measurement from a first temperature sensor proximate the electronic devices.

24. The cooling system of claim 23, wherein the first temperature sensor is integral to a first electronic component of the plurality of electronic devices, and is configured to measure an internal temperature of the first electronic component.

25. The cooling system of any preceding claim, wherein the controller is configured to predict the local heat dissipation increase in the hotspot region of the immersion tank in advance of a temperature increase therein.

26. The cooling system of claim 25, wherein the controller is configured to predict the local heat dissipation increase in the hotspot region of the immersion tank at least in part according to a measurement of an electrical status of a first electronic component of the plurality of electronic devices, the electrical status being at least one of: an amount of current flowing through the first electronic component; an electrical voltage applied to the first electronic component; andan amount of electrical power flowing to the first electronic component.

27. The cooling system of claim 25 or 26, wherein the controller is configured to predict the local heat dissipation increase in the hotspot region of the immersion tank at least in part according to a workload prediction that is applicable to the first electronic device.

28. The cooling system of claim 27, wherein the workload prediction is inferred from information regarding network activity, and / or information derived from an internal server scheduler that queues tasks to be performed by the first electronic device.

29. The cooling system of any preceding claim, wherein the plurality of electronic devices are arranged in at least one of a plurality of horizontal rows, a plurality of horizontal columns, and a plurality of vertical tiers.

30. A method of cooling a plurality of electronic devices, the method comprising: providing a cooling system comprising: a controller; an active inlet manifold comprising a plurality of spaced apart outlets through which a cooling liquid can flow into proximity and thermal communication with the electronic devices; and an active outlet manifold comprising a plurality of spaced apart inlets through which the cooling liquid can flow from proximity with the electronic devices into the active outlet manifold; and wherein the active inlet manifold includes a first plurality of remotely adjustable flow control devices that separately control the flow of the cooling liquid through each of the plurality of outlets, and the active outlet manifold includes a second plurality of remotely adjustable flow control devices that separately control the flow of the cooling liquid through each of the plurality of inlets;at least one of detecting and predicting a local heat dissipation increase in a hotspot region of the electronic devices; and adjusting by the controller of the flow control devices of the active manifolds to cause more of the cooling liquid to flow in thermal communication with the hotspot region as compared to other regions proximate the electronic devices.

31. The method of claim 30, wherein predicting the local heat dissipation increase in the hotspot region comprises receiving a measurement of an electrical status of a first electronic component of the plurality of electronic devices, the electrical status being at least one of: an amount of current flowing through the first electronic component; an electrical voltage applied to the first electronic component; and an amount of electrical power flowing to the first electronic component.

32. The method of claim 30 or 31, wherein predicting the local temperature increase in the hotspot region of the immersion tank comprises receiving a workload prediction applicable to a first electronic component of the plurality of electronic devices.

33. The method of claim 32, wherein receiving the workload prediction comprises inferring an anticipated workload applicable to the first electronic component from information regarding network activity, and / or information derived from an internal server scheduler that queues tasks to be performed by the electronic devices.

Images