Multi-phase multi-system architecture

Abstract

A multiphase cooling system for electronics racks, server farms, and data centers is described. An inlet of a three-way flow control valve (FCV) is coupled to a primary coolant source. A first outlet of the FCV is coupled to a single-phase cooling system, and a second outlet of the FCV is coupled to a two-phase cooling system. The FCV is configured to adjust the amount of coolant between the single-phase cooling system and the two-phase cooling system. Upon detecting a rise in vapor pressure in a return line of the two-phase cooling system, the FCV can be adjusted to direct more coolant to the two-phase cooling system and less coolant to the single-phase system. The FCV can continuously monitor the vapor pressure and adjust the amount of coolant to each cooling system accordingly.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to cooling of data centers. More specifically, embodiments of the present invention relate to a multi-phase, multi-cooling system architecture for cooling servers in a rack, cluster, or data center. Background Art

[0002] Cooling is a prominent element in the design of computer systems and data centers. The number of high-performance electronic components, such as high-performance processors, housed within servers has steadily increased, increasing the amount of heat generated and dissipated during normal server operation. If the environment in which servers used within a data center operate is allowed to rise in temperature over time, the reliability of the servers will decline. Maintaining an appropriate thermal environment is crucial to the proper operation of these servers within the data center, as well as their performance and longevity. This creates a need for more effective and efficient cooling solutions, particularly when cooling high-performance servers.

[0003] Server racks in a data center may contain servers with different types of electronic components, which generate heat that needs to be removed by the cooling system. Different electronic components can have substantially different heat loads. Typically, single-phase cooling systems are used to remove heat generated within a server system or a rack of server systems. Single-phase cooling systems are cooling systems in which the coolant, or working fluid, remains in a liquid state. Rack-level coolant distribution for heat removal is a single point of failure.

[0004] In the prior art, rack-level cooling is performed by a single-phase system that pumps coolant to the server racks and transfers heat to the coolant. The coolant then returns to the cooling system to remove heat from the coolant. The speed of the pump that pumps coolant to the server racks may need to be varied to remove different amounts of heat generated by the components within the server racks. As the heat load increases, the speed of the pump may not be able to be increased further. Furthermore, increasing the speed of the pump beyond a certain design point becomes an inefficient way to remove heat. In addition, the pump may fail, resulting in the single-phase system being unable to remove heat from the heat-generating components. Therefore, whether it is an operational failure of the pump or the pump is unable to operate quickly enough to remove the generated heat, the single-phase system may include a single point of failure in cooling the heat-generating components within the server racks in the existing cooling system. Summary of the Invention

[0005] The present invention aims to solve one of the above-mentioned technical problems in the prior art.

[0006] An embodiment of the present invention provides a data center cooling system, comprising: a flow control device, the flow control device having an inlet port connected to a facility cooling fluid supply line; a first cooling system, the first cooling system being connected to the facility cooling fluid supply line via a first outlet port of the flow control device, wherein the first cooling system provides a single-phase cooling fluid to at least one of a plurality of server chassis in an electronics rack; a second cooling system, the second cooling system being connected to the facility cooling fluid supply line via a second outlet port of the flow control device, wherein the second cooling system provides a two-phase cooling fluid to at least one of the plurality of server chassis in the electronics rack; a sensor, the sensor being connected to a steam return line of the second cooling system; and a controller for controlling the flow control device to adjust the facility cooling fluid supply line between the first cooling system and the second cooling system in response to a sensor signal received from the sensor.

[0007] In some embodiments, the controller controls the flow control device to adjust an amount of facility cooling fluid from the facility cooling fluid supply line between the first cooling system and the second cooling system in response to a sensor value obtained from the sensor signal, wherein the facility cooling fluid is used to cool the single-phase cooling fluid and condense vapor evaporated from the two-phase cooling fluid.

[0008] In some embodiments, in response to the sensor value exceeding a predetermined threshold, the controller controls the flow control device to increase an amount of the facility cooling fluid to the second cooling system.

[0009] In some embodiments, the flow control device includes a three-way control valve, which is configured to: provide all of the facility cooling fluid to the first cooling system or the second cooling system; or provide a first portion of the facility cooling fluid to the first cooling system and a second portion of the facility cooling fluid to the second cooling system based on one or more sensor values ​​obtained from the sensor.

[0010] In some embodiments, the first cooling system is configured to remove heat from the single-phase cooling fluid circulating in a first cooling loop of the first cooling system, the first cooling loop cooling a first plurality of heat-generating components in at least one of the server chassis in the electronics rack.

[0011] In some embodiments, the second cooling system includes a two-phase condenser system configured to remove heat from the two-phase cooling fluid in a second two-phase cooling loop of the second cooling system, the second two-phase cooling loop cooling a second plurality of heat-generating components of at least one of the server chassis in the electronics rack, wherein at least one heat-generating component is in both the first plurality of heat-generating components and the second plurality of heat-generating components, and the at least one heat-generating component is mounted to a cold plate that is cooled by both the first cooling loop and the second cooling loop.

[0012] In some embodiments, the sensor comprises a pressure sensing device that measures steam pressure in the second two-phase cooling circuit in the second cooling system.

[0013] In some embodiments, the first cooling system and the second cooling system are integrated into a single cooling system.

[0014] In some embodiments, the first cooling system is a primary cooling system and the second cooling system is a secondary cooling system.

[0015] In some embodiments, the data center cooling system further includes: a first liquid supply line and a second liquid return line coupled between the first cooling system and the single-phase manifold of the electronics rack; and a third liquid supply line and a vapor return line coupled between the second cooling system and the two-phase manifold of the electronics rack.

[0016] In some embodiments, the electronics rack is one of a plurality of electronics racks coupled to the first cooling system and the second cooling system.

[0017] In some embodiments, each electronics rack is coupled to the first cooling system via a pair of liquid supply and liquid return lines, and is coupled to the second cooling system via a pair of liquid supply and vapor return lines.

[0018] An embodiment of the present invention also provides a data center system, comprising: a facility cooling fluid supply line, the facility cooling fluid supply line being used to receive cooling fluid from a cooling fluid source; a plurality of electronic racks, each electronic rack comprising a single-phase rack manifold for distributing single-phase cooling fluid, and a two-phase rack manifold for distributing two-phase cooling fluid to one or more server chassis contained in the corresponding electronic rack; and a data center cooling system according to the above embodiment, the data center cooling system being connected to the electronic racks.

[0019] An embodiment of the present invention also provides an electronic rack, comprising: a plurality of stacked server chassis, each server chassis containing one or more servers, each server including one or more processors; a single-phase manifold, the single-phase manifold being connected to at least a portion of the server chassis to distribute single-phase cooling fluid to the server chassis; and a two-phase manifold, the two-phase manifold being connected to at least a portion of the server chassis to distribute two-phase cooling fluid to the server chassis, wherein each of the single-phase manifold and the two-phase manifold is connected to a cooling system to receive and return the single-phase cooling fluid and the two-phase cooling fluid, respectively, wherein the cooling system is the data center cooling system according to the above-mentioned embodiment, the first cooling system is connected to the single-phase manifold to provide the single-phase cooling fluid to the server chassis, and the second cooling system is connected to the two-phase manifold to provide the two-phase cooling fluid to the server chassis. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Embodiments of the invention are illustrated by way of example and not limitation in the figures of the accompanying drawings and in which like references indicate similar elements.

[0021] Figure 1A is a block diagram illustrating an example of an electronics rack according to one embodiment.

[0022] Figure 1B is a block diagram illustrating an example of a server cluster in an electronics rack according to one embodiment.

[0023] Figure 1C is a block diagram illustrating an example of a data center facility according to one embodiment.

[0024] Figure 2 is a block diagram illustrating an example of a side view of an electronics rack according to one embodiment.

[0025] Figure 3 is a block diagram illustrating an example of a top view of an electronics rack according to one embodiment.

[0026] Figure 4 is a block diagram illustrating an example of a cold plate configuration according to one embodiment.

[0027] Figure 5 A flow chart illustrating a method of cooling using a multi-phase multi-cooling system architecture according to one embodiment is shown. DETAILED DESCRIPTION

[0028] Various embodiments and aspects of the present invention will be described with reference to the details discussed below, and the accompanying drawings will illustrate various embodiments. The following description and drawings are illustrative of the present invention and should not be construed as limiting the present invention. Numerous specific details are described to provide a thorough understanding of the various embodiments of the present invention. However, in some cases, known or conventional details are not described in order to provide a concise discussion of the embodiments of the present invention.

[0029] References in the specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. The phrase "in one embodiment" appearing in different places in the specification does not necessarily refer to the same embodiment.

[0030] In a first aspect, an electronics rack cooling system is configured to cool heat-generating components in a plurality of servers mounted in an electronics rack. The cooling system includes a flow control device, a first cooling system, and a second cooling system. An inlet port of the flow control device is coupled to a primary cooling fluid source (e.g., a facility cooling fluid source). The first cooling system is coupled to the facility cooling fluid source via a first outlet port of the flow control device. The first cooling system may be a single-phase cooling system. The second cooling system is coupled to the facility cooling fluid source via a second outlet port of the flow control device. The second cooling system may be a two-phase cooling system.

[0031] A two-phase or "phase change" cooling system is a cooling system in which the coolant or working fluid of the cooling system is selected so that the coolant is intended to evaporate in the process of removing heat from heat-generating components. A sensor is coupled to a flow control device via a controller and to a vapor return line of the second cooling system. The flow control device is configured to adjust the flow of facility cooling liquid received from a facility cooling liquid source to the first cooling system and / or the second cooling system in response to a control command or signal received from the controller based on a sensor signal received from the sensor. The sensor continuously reads the vapor pressure of the vapor return line of the two-phase cooling circuit of the second cooling system.

[0032] In response to the changing sensor readings, the controller sends a control signal or command to instruct the flow control device to adjust the amount of facility cooling fluid between the first cooling system and the second cooling system. In one embodiment, the flow control device may be a three-way control valve. The three-way control valve may adjust the facility cooling fluid flow rate based on one or more sensor values ​​obtained from the sensor so that all of the facility cooling fluid delivered by the facility cooling fluid source enters the first cooling system or the second cooling system, or a variable portion of the facility cooling fluid received from the facility cooling fluid source enters the first cooling system or the second cooling system.

[0033] In one embodiment, the flow control device can be two two-way control valves, each having an inlet port coupled to a source of facility cooling fluid. The first two-way control valve can have an outlet port coupled to a coolant inlet of the first cooling system, and the second two-way control valve can have an outlet port coupled to a coolant inlet of the second cooling system. The sensor can be a pressure sensing device that measures vapor pressure in a two-phase cooling circuit in the second cooling system. In one embodiment, the first cooling system comprises a single-phase cooling system configured to remove heat from the single-phase cooling fluid circulating in the first cooling circuit of the first cooling system.

[0034] The first cooling circuit removes heat from a first plurality of heat-generating components in at least one of the plurality of electronics racks. In one embodiment, the second cooling system is configured to remove heat from the second plurality of heat-generating components and / or the first plurality of heat-generating components within the server rack. The second cooling system may include a two-phase condenser system configured to remove heat from vapor of a two-phase coolant in a second cooling circuit of the second cooling system. The first cooling system and the second cooling system may be integrated into a single cooling system combining the two cooling systems.

[0035] In a second aspect, a multiphase cooling system is configured to cool heat-generating components in a server cluster within one or more electronics racks. The server cluster cooling system may have the architecture and functionality of a multiphase, multi-cooling system for a single electronics rack generally as described in the first aspect, with the following features. In the server cluster cooling system comprising one or more electronics racks, a first cooling circuit of a first cooling system provides single-phase cooling fluid to a first single-phase cooling fluid supply manifold in each electronics rack. Any server within the electronics rack may have heat-generating components within the electronics rack to be cooled by the first cooling system.

[0036] The servers can be coupled to a single-phase cooling fluid supply manifold for the racks. Heat generated by these components is removed through heat transfer within the servers using one or more cold plates coupled to the single-phase cooling fluid supply and / or a two-phase coolant supply of a second cooling circuit of a secondary cooling system. Heat is removed from the electronics racks via a single-phase cooling fluid return line of the single-phase cooling system and / or a steam return line of the two-phase cooling system. The single-phase cooling fluid return line from each rack is coupled to a single single-phase cooling fluid return line that returns to the primary cooling system.

[0037] Similarly, the second cooling circuit of the second cooling system provides a two-phase coolant supply to a second two-phase coolant supply manifold in each electronics rack. Any server within the electronics rack may have heat-generating components within the electronics rack to be cooled by the second cooling system. The servers may be coupled to the second two-phase coolant supply manifold for the electronics rack. Heat generated by these components is removed via heat transfer from components coupled to the cold plates within the servers to the second two-phase coolant supply, and heat is removed from the electronics rack via a vapor return line.

[0038] The steam return line from each rack is connected to a single steam return line that returns to the second cooling system. A sensor is connected to the single steam return line for all electronics racks in the server cluster. The sensor measures the steam pressure within the single steam return line for the server cluster. An increase in steam pressure indicates an increase in the heat load within the second cooling circuit of the second cooling system. As in the first aspect described above, changes in the steam pressure in the steam return line for the server cluster cause the flow control device to adjust the amount of facility cooling fluid supplied to each of the first cooling system and the second cooling system.

[0039] In a third aspect, a data center cooling system for cooling heat-generating components within a plurality of servers and a plurality of electronics racks can be substantially as described in the first and second aspects above, with the following features. In the data center cooling system, the number of servers and electronics racks can significantly exceed the number of server clusters, such that each cooling system can have one or more cooling circuits. In one embodiment, the plurality of cooling circuits of a first cooling system can be connected at a central location, such as at the first cooling system. The plurality of cooling circuits of a second cooling system can be connected at a central location, such as at the second cooling system. A steam pressure sensor can be coupled to a single steam return line at the location where the plurality of two-phase cooling circuits are connected.

[0040] Figure 1AFIG2 is a block diagram illustrating an example of a cooling system 100A for an electronics rack 200 according to one embodiment. Cooling system 100A for electronics rack 200 includes a first cooling system 120 and a second cooling system 121. In one embodiment, cooling systems 120 and 121 can be integrated into a single cooling system. A primary cooling liquid source 137 can be a facility cooling liquid source, such as from a cooling tower, chiller, or other large-scale cooling source. A facility cooling fluid return line 136 returns warm cooling fluid that has been removed from one or both of the first cooling circuit of first cooling system 120 and / or the second cooling circuit of second cooling system 121. The inlet of a three-way flow control device 140 is connected to facility cooling liquid supply 137. A first outlet of three-way flow control device 140 is coupled to a single-phase cooling fluid supply connection on first cooling system 120. A second outlet of three-way flow control device 140 is coupled to a two-phase cooling fluid supply connection on second cooling system 121. Three-way flow control device 140 regulates the flow of cooling fluid from facility cooling fluid supply line 137 to first cooling system 120 and second cooling system 121. In one embodiment, three-way flow control device 140 is a three-way control valve. In one embodiment, three-way flow control device 140 can be two two-way flow control valves that operate independently or in conjunction with each other, each having an inlet coupled to facility cooling fluid supply line 137 and an outlet coupled to a facility cooling fluid supply connection of one of the two cooling systems 120 and 121.

[0041] The first cooling system 120 may be a single-phase cooling system including a first cooling loop. The first cooling loop provides a single-phase cooling fluid supply 132 to a single-phase cooling fluid distribution manifold 142 in the electronics rack 200. The electronics rack 200 may include one or more servers, such as servers 203A...203D (collectively referred to as "servers 203" unless otherwise specified). Each server 203 may include one or more heat-generating components, such as a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence accelerator, etc. The first plurality of heat-generating components within servers 203A...203D may be coupled to one or more cold plates (not shown) within servers 203A...203D. Each cold plate may be coupled to the first cooling loop single-phase cooling fluid supply 132 via the single-phase cooling fluid supply distribution manifold 142. Each server 203 may be connected to the single-phase cooling fluid supply 132 via a connection 150 on the single-phase cooling fluid supply distribution manifold 142. The server 203 can deliver the single-phase cooling fluid from the single-phase cooling fluid supply line 132 to one or more heat-generating components within the server 203. A cold plate (not shown) transfers heat from the one or more heat-generating components within the server 203 to the coolant supplied by the cooling fluid supply 132. The cold plate (not shown) transfers the heat to the cooling fluid return line 131 that returns to the cooling system 120. In one embodiment, the single-phase cooling fluid in the first cooling circuit of the first cooling system 120 is a first working fluid having a vaporization temperature that is higher than a design temperature of the cooling circuit, such that the single-phase cooling fluid in the first cooling circuit is designed not to vaporize during normal design heat loads.

[0042] Second cooling system 121 may be a two-phase cooling system. In one embodiment, the working fluid in the second cooling circuit of second cooling system 121 is selected so that the working fluid vaporizes within the design operating range of a second plurality of heat-generating components within one or more servers 203 in electronics rack 200. The design temperature range of the second plurality of heat-generating components may differ from the design temperature range of the first plurality of heat-generating components. The second plurality of heat-generating components within servers 203A...203D may be coupled to one or more cold plates (not shown) within servers 203A...203D.

[0043] Each cold plate can be coupled to the second cooling circuit two-phase coolant supply 134 via a two-phase coolant distribution manifold 144. The server 203 can deliver the two-phase coolant to one or more cold plates coupled to one or more heat-generating components. One or more cold plates (not shown) within the server 203 transfer heat from one or more heat-generating components to the coolant two-phase coolant supply line 134, which then turns into steam in the return line 133. Each server 203 can have a connection 150 that couples the steam return line in the server 203 to the steam return manifold 143 that delivers the steam 133 back to the two-phase cooling system 121.

[0044] A pressure sensor 145 is installed in the steam return line 133. As the second working fluid evaporates, the steam pressure in the steam return line 133 increases. The change in steam pressure in the steam return line 133 is detected by the pressure sensor 145. The pressure sensor 145 can send a signal to a controller (not shown), which in turn sends a control signal or command to instruct the three-way flow control device 140 to adjust the amount of facility cooling liquid from the facility cooling fluid supply line 137 between the first cooling system and the second cooling system in response to the change in steam pressure in the steam return line 133.

[0045] In one embodiment, as the steam pressure in the steam return line 133 increases, the three-way control device can increase the flow rate from the facility cooling fluid supply line 137 to the second cooling system 121. When the pressure sensor 145 detects a drop in steam pressure, the three-way flow control device can direct less facility cooling fluid supply to the second cooling system 121. The second cooling system 121 can include a condenser so that increasing the amount of facility cooling fluid from the facility cooling fluid supply line 137 to the second cooling system 121 can quench or condense and thereby cool the second working fluid or steam within the second cooling circuit of the second cooling system 121. The cooled second working fluid can be recycled to the two-phase coolant supply line 134 for additional heat removal within the electronics rack 200.

[0046] In one embodiment, the two cooling systems 120 and 121 can be operated in parallel in a combined manner. In such operation, pressure can be used to adjust the optimal combined operation of the systems 120 and 121. The facility cooling fluid supply line 137 and the facility cooling fluid return line 136 are managed and distributed between the cooling systems 120 and 121 for redundant design and optimized performance.

[0047] In one embodiment, the first plurality of heat-generating components cooled by the first cooling circuit may be the same plurality of heat-generating components as the second plurality of heat-generating components that the second cooling circuit may additionally or alternatively cool. The second cooling system 121 may operate as a cooling system that is fully redundant with the first cooling system 120, such that the same plurality of heat-generating components in one or more servers are cooled by the first cooling system, the second cooling system, or a controllably variable combination of the first cooling system and the second cooling system. In one embodiment, the electronics rack 200 may also include a rack management unit (RMU) 202. Figure 2 The RMU 202 is described in detail.

[0048] Figure 1B is a block diagram illustrating an example of a cooling system having multiple phases and multiple cooling systems for cooling a server cluster 100B according to one embodiment. The cooling system for the server cluster 100B is substantially the same as that described above with reference to FIG. Figure 1A The cooling system for the electronic rack 100A is the same as described above, with the main difference being that in the server cluster, the cooling system is Figure 1A A single electronics rack 100A in FIG. 1 has more electronics racks and servers to cool.

[0049] A cooling system 100B for a server cluster includes a plurality of electronics racks, such as electronics racks 200A, 200B, and 200C (collectively referred to as "electronics racks 200" unless otherwise specified). A first cooling system 120 may be a single-phase cooling system having a first cooling circuit including a single-phase cooling fluid supply line 132 and a single-phase cooling fluid return line 131. The first cooling circuit removes heat from a first plurality of heat-generating components within one or more servers in the plurality of electronics racks 200 in the server cluster.

[0050] A single-phase cooling fluid supply line 132 supplies cooling fluid to each electronics rack 200 in the server cluster that has a server 203 with at least one heat-generating component from the first plurality of heat-generating components. For each electronics rack 200, the single-phase cooling fluid supply line 132 is coupled to a single-phase cooling fluid supply distribution manifold 142 within the electronics rack 200. Each electronics rack 200 that has a server 203 with at least one heat-generating component from the first plurality of heat-generating components also has a single-phase cooling fluid return manifold 141 within the electronics rack 200.

[0051] Each single-phase cooling fluid return manifold 141 in the electronics racks 200A-200C of the server cluster is coupled to a single single-phase cooling fluid return line 131 that returns the single-phase cooling fluid 131 to the first cooling system 120. The first cooling system 120 removes the heat received from the single-phase cooling fluid return line 131 and recycles the now cooled cooling fluid as a single-phase cooling fluid supply 132.

[0052] The second cooling system 121 can be a two-phase cooling system having a second cooling loop including a two-phase cooling fluid supply line 134 and a vapor return line 133. The second working fluid in the two-phase cooling system 121 can be different from the first working fluid in the first cooling system 120. The second working fluid can be selected to evaporate within the design operating temperature range of the second plurality of heat-generating components within the server cluster. The second cooling loop removes heat from the second plurality of heat-generating components within one or more servers 203 in the plurality of electronics racks 200 in the server cluster.

[0053] A two-phase coolant supply line 134 supplies coolant to each electronics rack 200 in the server cluster that has a server 203 with at least one heat-generating component from the second plurality of heat-generating components. For each electronics rack 200, the two-phase coolant supply line 134 is coupled to a two-phase coolant supply distribution manifold 144 in the electronics rack 200. Each electronics rack 200 that has a server 203 with at least one heat-generating component from the second plurality of heat-generating components also has a steam return manifold 143 in the electronics rack 200. Each steam return manifold 143 in the electronics racks 200 of the server cluster is coupled to a single steam return line 133, which is coupled to the second cooling system 121. The second cooling system 121 removes heat from the steam return line 133 and recycles the now-cooled steam as the two-phase coolant supply 134.

[0054] The pressure sensor 145 can be coupled to the single steam return line 133 at a point downstream from where all of the steam return manifolds 143 in the rack 200 join to the single steam return line 133. The sensor 145 can measure the steam pressure within the steam return line 133. As described above, with reference to Figure 1A Flow control device 140 can adjust the amount of facility cooling fluid supplied from facility cooling fluid supply line 137 to each of cooling systems 120 and 121 in response to changes in vapor pressure detected by sensor 145. Facility cooling fluid return line 136 returns warm cooling fluid that has had heat removed from one or both of the first cooling circuit of first cooling system 120 and / or the second cooling circuit of second cooling system 121.

[0055] In one embodiment, each steam circuit of each electronics rack includes a pressure sensor 145 before converging into a single steam return circuit 133. If the racks are all very similar or identically configured and positioned, the steam circuit pressure for a single server cluster can be mathematically derived from the pressure sensors 145 in each steam circuit of each electronics rack. The mathematically derived pressure can be, for example, an average, a weighted average, the highest value among the pressure sensors 145, etc. The derived pressure value can then be used to actuate the three-way flow control device 140 to adjust the facility cooling flow to each of the first cooling system and the second cooling system. In one embodiment, each rack steam return 133 can include a pressure sensor 145.

[0056] As mentioned above about Figure 1A As described for a single server 203, Figure 1B In one embodiment, the first plurality of heat-generating components cooled by the first cooling circuit can be the same plurality of heat-generating components as the second plurality of heat-generating components that can additionally or alternatively be cooled by the second cooling circuit. The second cooling system 121 can operate as a fully redundant cooling system with the first cooling system 120, such that the same plurality of heat-generating components in one or more servers are cooled by the first cooling system, the second cooling system, or a controllably variable combination of the first cooling system and the second cooling system.

[0057] Figure 1C is a block diagram illustrating an embodiment of a data center 100C according to one embodiment. In this embodiment, Figure 1C A top view of at least a portion of a data center 100C is shown. Figure 1C According to one embodiment, a data center system 100A includes one or more rows of electronic racks of information technology (IT) components, devices, or instruments 101-102 (e.g., computer servers or computing nodes) that provide data services to various clients over a network (e.g., the Internet).

[0058] In this embodiment, each row includes an array of electronics racks, such as electronics racks 110A-110N. However, more or fewer rows of electronics racks may be implemented. Typically, rows 101-102 are arranged in parallel, with their front ends facing each other and their rear ends facing away from each other, with an aisle 103 formed in the middle to allow management personnel to walk through it. Cool / cold air 139 enters aisle 103 from air supply system 135. However, other configurations or arrangements may also be used. For example, two rows of electronics racks may be arranged back to back facing each other without forming an aisle in the middle, with their front ends facing away from each other. The rear ends of the electronics racks may be connected to the room cooling liquid manifold.

[0059] In one embodiment, each electronics rack (e.g., electronics racks 110A-110N) includes a housing to house a number of IT components arranged and operated in a stacked manner therein. The electronics rack may include a plurality of cooling liquid manifolds (not shown), a number of server slots (e.g., standard racks or chassis configured with the same or similar form factors), and a number of server chassis (also referred to as server blades or server racks) that can be inserted into and removed from the server slots.

[0060] Each server chassis represents a compute node having one or more processors, memory, and / or persistent storage (e.g., a hard disk), wherein a compute node may include one or more servers operating therein. At least one processor is attached to a liquid cold plate (also referred to as a cold plate assembly) to receive cooling liquid. In addition, one or more optional cooling fans are associated with the server chassis to provide air cooling to the compute nodes contained therein. Note that the cooling system 120 and the condenser or two-phase ("condenser") cooling system 121 can be coupled to multiple data center systems (e.g., data center system 100C).

[0061] In one embodiment, cooling systems 120 and 121 include an external liquid loop connected to a cooling tower or dry cooler external to the building / building container, a facility cooling fluid return line 136, and a facility cooling fluid supply line 137. Cooling system 120 may include, but is not limited to, countercurrent heat exchangers, evaporative cooling, free air, rejection of large thermal masses, and waste heat recovery designs. Cooling system 121 may be an evaporative cooling system with a condenser or quench system. Cooling system 120 and cooling system 121 (or "condenser system 121") may include or be coupled to a facility cooling liquid source, such as facility cooling liquid, that provides facility cooling fluid to cooling system 120 and cooling system 121. The amount of facility cooling fluid 137 provided to each of cooling system 120 and cooling system 121 may be adjusted using a three-way flow control device 140 to direct some facility cooling fluid 137 to cooling system 120 and some facility cooling fluid 137 to cooling system 121.

[0062] Second cooling system 121 may be a two-phase cooling system. Two-phase cooling system 121 may use a different working fluid than single-phase cooling system 120. The working fluid in second cooling system 121 may be selected so that the working fluid evaporates at a design temperature threshold. Two-phase cooling system 121 may include a pressure sensor 145 that can measure changes in steam pressure in steam return line 133. Pressure sensor 145 may be located at various locations along steam return line 133, as may be appropriate for sensor 145 to detect changes in steam pressure within steam return line 133.

[0063] In one embodiment, each electronics rack may include one or more cooling liquid manifolds (not shown) for the cooling circuits of each cooling system 120 and 121. Each server within the electronics rack may be coupled to an available fluid connector on the cooling liquid manifold using, for example, a connecting hose assembly 128. In one embodiment, each server chassis is modularly coupled to the cooling liquid manifold such that a server chassis may be removed from the electronics rack without affecting the operation of the remaining server chassis and cooling liquid manifolds in the electronics rack.

[0064] In another embodiment, each server chassis is coupled to a cooling liquid manifold via a quick-release coupling assembly having a server liquid inlet connector and a server liquid outlet connector. The server liquid outlet connector is coupled to a flexible hose to distribute cooling liquid to the cold plate to which the processor is coupled. The server liquid inlet connector receives cooling liquid from a cooling liquid manifold mounted at the rear end of the electronics rack via a rack liquid inlet connector. The server liquid outlet connector discharges single-phase cooling fluid or two-phase vapor, carrying heat exchanged from the processor, to the cold plate and then to the cooling liquid manifold via a rack liquid outlet connector.

[0065] In one embodiment, a cooling liquid manifold disposed on the rear end of each electronics rack is coupled to a single-phase cooling fluid supply line 132 (also referred to as a room supply manifold) to receive cooling liquid from the cooling system 120. Another cooling liquid manifold disposed on the rear end of each electronics rack can be coupled to a two-phase coolant supply line 134 of the second cooling system 121. The cooling liquids 131 and 132 of the first cooling system 120 are distributed through a first cooling circuit of the first cooling system attached to a cold plate assembly on which the processors are mounted to remove heat from the processors.

[0066] The cold plate is similar to a radiator configuration, with liquid distribution pipes attached or embedded in it. The resulting warmer or hotter liquid, carrying heat exchanged from the processor, is delivered back to the cooling system 120 via a single-phase cooling fluid return line 131 (also known as a room return manifold). Similarly, the two-phase coolant supply 134 and vapor return 133 of the second cooling system 121 are distributed through the second cooling circuit of the second cooling system, attached to the cold plate assembly on which the processor is mounted, to remove heat from the processor.

[0067] As in about Figure 1A The server cooling system described and the Figure 1BIn the described server cluster cooling system, in one embodiment, the first plurality of heat-generating components cooled by the first cooling circuit can be the same plurality of heat-generating components as the second plurality of heat-generating components that can be additionally or alternatively cooled by the second cooling circuit. The second cooling system 121 can operate as a fully redundant cooling system with the first cooling system 120, such that the same plurality of heat-generating components in one or more servers are cooled by the first cooling system, the second cooling system, or a controllable and variable combination of the first cooling system and the second cooling system.

[0068] The facility cooling fluid supply line 137 and the facility cooling fluid return line 136 are referred to as data center or facility liquid supply / return lines (e.g., overall liquid supply / return lines) and supply cooling liquid to cooling system 120 and cooling system 121. Cooling systems 120 and 121 each have a cooling circuit that extends to all of the electronics racks in rows 101-102. Cooling system 120 has a cooling circuit that includes a single-phase cooling fluid supply 132 and a single-phase cooling fluid return 131, coupled to all of the electronics racks in rows 101-102. Cooling system 121 has a cooling circuit that includes a two-phase coolant supply 134 and a vapor return line 133, which can extend to all of the electronics racks in rows 101-102.

[0069] In one embodiment, data center system 100C further includes an optional airflow delivery system ("air supply system") 135 to generate a cool airflow 139 that is passed through the air spaces of the server chassis of the electronics racks to remove heat generated by the computing nodes (e.g., servers) due to their operation, and to remove the heat removed by the airflow to the external environment or a cooling system (e.g., an air-to-liquid heat exchanger) to reduce the temperature of the airflow. For example, air supply system 135 generates an airflow of cool / cold air 139 that circulates from aisle 103 through electronics racks 110A-110N, thereby removing the removed heat as warm / hot air 138.

[0070] Cool airflow 139 enters the electronics rack through the front of the electronics rack, and warm / hot airflow 138 exits the electronics rack through the rear of the electronics rack. Warm / hot air 138, carrying with it the exchanged heat, is exhausted from the room / building or cooled using a separate cooling system (e.g., an air-to-liquid heat exchanger). Thus, this cooling system is a hybrid liquid-air cooling system, in which a portion of the heat generated by the processors is removed by cooling liquid via corresponding cold plates, while the remaining heat generated by the processors (or other electronic devices or processing equipment) is removed by air cooling.

[0071] Figure 2 is a block diagram illustrating an electronic rack 200 according to one embodiment. The electronic rack 200 may represent Figures 1A to 1C Any of the electronic racks shown in , such as electronic racks 110A-110N, 200 or 200A-200C. Figure 2 According to one embodiment, the electronics rack 200 includes, but is not limited to, a rack management unit (RMU) 202 and one or more server chassis 203A-203F (collectively, server chassis 203). The server chassis 203 can be inserted into a server slot array (e.g., a standard shelf) from either the front 204 or rear 205 of the electronics rack 200. Note that while six server chassis 203A-203F are shown, more or fewer server chassis can be maintained within the electronics rack 200. In one embodiment, the electronics rack 200 can be open to the environment or partially contained within a rack container, as long as cooling fans can generate airflow from the front to the rear.

[0072] In addition, for at least some of the server chassis 203, an optional fan module (not shown) is associated with the server chassis. Each fan module includes one or more cooling fans. The fan modules can be mounted on the rear end of the server chassis 203 or on the electronics rack to generate airflow. The airflow flows from the front end 204, travels through the air space of the server chassis 203, and exits at the rear end 205 of the electronics rack 200.

[0073] Each server chassis 203 may include one or more IT components (e.g., a central processing unit or CPU, a general-purpose / graphics processing unit (GPU), memory, and / or storage). Each IT component may perform data processing tasks, wherein the IT component may include software installed in a storage device, loaded into the memory, and executed by one or more processors to perform the data processing tasks. The server chassis 203 may include a host server (referred to as a host node) coupled to one or more computing servers (also referred to as computing nodes, such as CPU servers and GPU servers).

[0074] A host server (with one or more CPUs) typically interfaces with clients over a network (e.g., the Internet) to receive requests for specific services, such as storage services (e.g., cloud-based storage services such as backup and / or recovery), executing applications to perform certain operations (e.g., image processing, deep data learning algorithms or modeling, etc., as part of a Software as a Service or SaaS platform). In response to the request, the host server assigns the task to one or more of the compute nodes or compute servers (with one or more GPUs) managed by the host server. The computer server performs the actual task, which may generate heat during operation.

[0075] Electronics rack 200 further includes an optional RMU 202 and CDU 201. RMU 202 is configured to provide and manage power to servers 203. RMU 202 can be coupled to a power supply unit (not shown) to manage the power consumption of the power supply unit. The power supply unit can include the necessary circuitry (e.g., an alternating current (AC) to direct current (DC) or direct current (DC) to direct current (DC) power converter, a battery, a transformer, or a voltage regulator, etc.) to provide power to the remaining components of electronics rack 200.

[0076] In one embodiment, the RMU 202 includes an optimization module 221 (not shown) and a rack management controller (RMC) 222. The RMC 222 may include a monitor for monitoring the operating status of various components (e.g., computing nodes 203 and fan modules) within the electronic rack 200. Specifically, the RMC 222 receives operational data representing the operating environment of the electronic rack 200 from various sensors. For example, the monitor may receive operational data representing the temperature of the processor, cooling liquid, and airflow, which may be captured and collected via various temperature sensors. The monitor may also receive data representing the fan power and pump power generated by the fan module 231 and the liquid pump 212, which may be proportional to the speed of each of the fan module 231 and the liquid pump 212. These operational data are referred to as real-time operational data. Please note that the monitor may be implemented as a separate module within the RMU 202.

[0077] Notice, Figure 2 The rack configuration shown in is shown and described for illustrative purposes only; other configurations or arrangements may also be applicable. The cold plates 400 of the server chassis 203 can be coupled to rack manifolds, such as 141-144, which can be directly coupled to the first coolant loops 131-132 and the second coolant loops 133-134. Each server 203 can be coupled to a manifold 141-143. Each server 203 can deliver single-phase cooling fluid supply and return 132 / 131 ("single-phase cooling loop") to one or more cold plates, or can deliver two-phase coolant supply and vapor return 134 / 133 ("two-phase cooling loop") to one or more cold plates in the server 203. In one embodiment, one or more or all of the cold plates can be coupled to both the single-phase cooling loop and the two-phase cooling loop.

[0078] Although not shown, a power supply unit can be disposed within the electronics rack 200. The power supply unit can be implemented as a standard chassis that is the same as or similar to the server chassis, wherein the power supply chassis can be inserted into any standard rack, replacing any server chassis 203. In addition, the power supply chassis can further include a battery backup unit (BBU) to provide battery power to the server chassis 203 when the main power source is unavailable. The BBU can include one or more battery packs, each battery pack including one or more battery cells, and the necessary charging and discharging circuits for charging and discharging the battery cells.

[0079] Figure 3 is a block diagram illustrating a top view of a server 203A in an electronics rack 200 according to one embodiment. The front of the electronics rack 200 is labeled 204, while the rear of the electronics rack is labeled 205. As described above, the first cooling system 120 may include a first cooling circuit having a single-phase cooling fluid supply line 132 and a single-phase cooling fluid return line 131. The single-phase cooling fluid supply line 132 may be coupled to a manifold 142 mounted in the rear 205 of the electronics rack 200 using, for example, a hose connection 128. The single-phase cooling fluid return line 131 may be coupled to the manifold 141 using, for example, a hose connection 128.

[0080] Similarly, the second cooling system can have a second cooling circuit with a two-phase cooling fluid supply 134 that can be coupled to manifold 144. A vapor return line 133 can be coupled to manifold 143 using, for example, hose connection 128. Any server 203 having a heat generating device to be coupled to the first cooling system can be coupled to manifolds 141 and 142 in electronics rack 200 using hose connection 128. Similarly, any server having a heat generating device to be coupled to the second cooling circuit can be coupled to manifolds 133 and 134 in electronics rack 200 using hose connection 128. The servers are cooled using two cooling circuits, each of which includes two manifold-to-server connections: one inlet ("supply") and one outlet ("return"). The actual cooling components in the server, such as cold plates, are also designed to operate with two cooling circuits. The design of the cold plates for operating with two cooling circuits is not subject of this disclosure.

[0081] Figure 4 is a block diagram illustrating a processor cold plate configuration according to one embodiment. The processor / cold plate assembly 400 may represent Figure 2 Any processor / cold plate configuration of the server chassis 203 shown in FIG. Figure 4Processor 401 is plugged into a processor socket mounted on a printed circuit board (PCB) or motherboard 402, which is connected to other electrical components or circuits of the data processing system or server. Processor 401 also includes a cold plate 403 attached thereto. Cold plate 403 is connected to multiple rack manifolds (not shown), which are connected to single-phase cooling fluid supply lines 132 and single-phase cooling fluid return lines 131, as well as two-phase coolant supply lines 134 and steam return lines 133. Some of the heat generated by processor 401 is removed by the cooling fluid via cold plate 403. The remaining heat enters the air space below or above, where it can be removed by airflow 139 generated by cooling fans 404. Cool air 139 passes through the air space, which heats the air to become warm air 138, which is then exhausted out the back (not shown) of electronics rack 200, where servers 203 are mounted.

[0082] Figure 5 A flowchart 500 is shown of a method for cooling heat-generating components within one or more servers in one or more electronics racks. In operation 501, the method is performed on a cooling system including a flow control device coupled to a facility cooling liquid. The flow control device is configured to output the facility cooling liquid to a first, single-phase cooling system and / or a second, two-phase cooling system. A sensor 145 measures a vapor pressure of a vapor return 133 in a circulation loop of the second, two-phase cooling system. The characteristics of the two-phase coolant are selected by design so that the two-phase coolant will vaporize at a design temperature that is within the range at which the second plurality of heat-generating devices are designed to operate.

[0083] In operation 502, a flow control device (140) is configured to adjust an amount of facility cooling liquid supplied between a first cooling system (120) and a second cooling system (121).

[0084] In operation 503, in response to a reading of a change in a value of a sensor (145), the flow control device (140) is configured to adjust an amount of a facility cooling fluid supply line 137 between the single-phase cooling system (120) and the two-phase cooling system (121). The adjustment includes: (1) the flow control device (140) providing all of the facility cooling fluid supply line 137 received from the facility cooling liquid source to the single-phase cooling system (120) or the two-phase cooling system (121), or (2) the flow control device (140) providing a variable portion of the facility cooling fluid supply line 137 received from the facility cooling liquid source to the single-phase cooling system (120) or the two-phase cooling system (121) based on the sensor value obtained from the sensor (145).

[0085] In the foregoing description, embodiments of the present invention have been described with reference to specific exemplary embodiments thereof. Although the rack orientation is shown as horizontal, this is not intended to be limiting. Different server rack orientations (e.g., vertical, or upward / downward) may be implemented using the present disclosure. It will be apparent that various modifications may be made to the present invention without departing from the broader spirit and scope of the invention as set forth in the following claims. Accordingly, the present description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A data center cooling system, comprising: a flow control device having an inlet port coupled to a facility cooling fluid supply line; a first cooling system coupled to the facility cooling fluid supply line via a first outlet port of the flow control device, wherein the first cooling system provides single-phase cooling fluid to at least one of a plurality of server chassis in an electronics rack; a second cooling system coupled to the facility cooling fluid supply line via a second outlet port of the flow control device, wherein the second cooling system provides two-phase cooling fluid to at least one of the plurality of server chassis in the electronics rack; a sensor coupled to a steam return line of the second cooling system; as well as a controller for controlling the flow control device to adjust the facility cooling fluid supply line between the first cooling system and the second cooling system in response to a sensor signal received from the sensor; wherein the controller controls the flow control device to adjust an amount of facility cooling fluid from the facility cooling fluid supply line between the first cooling system and the second cooling system in response to a sensor value obtained from the sensor signal, wherein the facility cooling fluid is used to cool the single-phase cooling fluid and condense vapor evaporated from the two-phase cooling fluid.

2. The data center cooling system according to claim 1, wherein: In response to the sensor value exceeding a predetermined threshold, the controller controls the flow control device to increase an amount of the facility cooling fluid to the second cooling system.

3. The data center cooling system according to claim 1, wherein: The flow control device includes a three-way control valve, which is configured as follows: providing all of the facility cooling fluid to the first cooling system or the second cooling system; or A first portion of the facility cooling fluid is provided to the first cooling system and a second portion of the facility cooling fluid is provided to the second cooling system based on one or more sensor values ​​obtained from the sensors.

4. The data center cooling system according to claim 1, wherein: The first cooling system is configured to remove heat from the single-phase cooling fluid circulating in a first cooling loop of the first cooling system, the first cooling loop cooling a first plurality of heat-generating components in at least one of the server chassis in the electronics rack.

5. The data center cooling system according to claim 4, wherein: The second cooling system includes a two-phase condenser system configured to remove heat from the two-phase cooling fluid in a second two-phase cooling loop of the second cooling system, the second two-phase cooling loop cooling a second plurality of heat-generating components of at least one of the server chassis in the electronics rack, wherein at least one heat-generating component is in both the first plurality of heat-generating components and the second plurality of heat-generating components, and the at least one heat-generating component is mounted to a cold plate that is cooled by both the first cooling loop and the second cooling loop.

6. The data center cooling system according to claim 5, wherein: The sensor comprises a pressure sensing device that measures steam pressure in the second two-phase cooling circuit in the second cooling system.

7. The data center cooling system according to claim 1, wherein: The first cooling system and the second cooling system are integrated into a single cooling system.

8. The data center cooling system according to claim 1, wherein: The first cooling system is a primary cooling system and the second cooling system is a secondary cooling system.

9. The data center cooling system of claim 1 , further comprising: a first liquid supply line and a second liquid return line coupled between the first cooling system and a single-phase manifold of the electronics rack; and A third liquid supply line and a vapor return line are coupled between the second cooling system and the two-phase manifold of the electronics rack.

10. The data center cooling system of claim 1, wherein: The electronics rack is one of a plurality of electronics racks coupled to the first cooling system and the second cooling system.

11. The data center cooling system according to claim 10, wherein: Each electronics rack is coupled to the first cooling system via a pair of liquid supply and liquid return lines, and is coupled to the second cooling system via a pair of liquid supply and vapor return lines.

12. A data center system, comprising: a facility cooling fluid supply line for receiving cooling fluid from a cooling fluid source; a plurality of electronics racks, each electronics rack including a single-phase rack manifold for distributing a single-phase cooling fluid and a two-phase rack manifold for distributing a two-phase cooling fluid to one or more server chassis contained within the respective electronics rack; and The data center cooling system of any one of claims 1-11, coupled to the electronics rack.

13. An electronic rack comprising: A plurality of server chassis arranged in a stack, each server chassis containing one or more servers, each server including one or more processors; a single-phase manifold coupled to at least a portion of the server chassis to distribute single-phase cooling fluid to the server chassis; as well as a two-phase manifold coupled to at least a portion of the server chassis to distribute a two-phase cooling fluid to the server chassis, wherein each of the single-phase manifold and the two-phase manifold is coupled to a cooling system to receive and return the single-phase cooling fluid and the two-phase cooling fluid, respectively, wherein the cooling system is a data center cooling system according to any one of claims 1-11, the first cooling system is coupled to the single-phase manifold to provide the single-phase cooling fluid to the server chassis, and the second cooling system is coupled to the two-phase manifold to provide the two-phase cooling fluid to the server chassis.

Images