Highly integrated liquid cooling system design for data center IT rooms

Abstract

Disclosed are row cooling units and connection units having a network of integrated fluid distribution conduits that can be interconnected to form a liquid cooling system to remove heat generated by servers housed within server racks used in a data center. Components of the row cooling units and connection units can be connected to supply and return loops of a data center facility to distribute cooling liquid to electronic components of the servers and return heated liquid to remove heat. The network of fluid distribution conduits integrated into the row cooling units and connection units enables the configuration of the liquid cooling system to be independent of the fixed infrastructure of the facility, thereby providing scalability, serviceability, ease of maintenance, while increasing the efficiency, flexibility, availability, and reliability of the liquid cooling system that are critical to the operation and performance of the data center.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to data center cooling. More specifically, embodiments of the present invention relate to designs for liquid cooling of electronics racks in data centers. Background Art

[0002] Heat dissipation is a prominent factor in computer system and data center design. The number of high-performance electronic components, such as high-performance processors, housed within servers has steadily increased, increasing the amount of heat generated and dissipated during normal server operation. If the environment in which servers operate is allowed to heat up over time, the reliability of servers used within a data center decreases. Maintaining a proper thermal environment is crucial to the proper operation of these servers within the data center, as well as to server performance and longevity. This, especially when cooling these high-performance servers, requires more effective and efficient heat dissipation solutions.

[0003] Thermal management of high-performance servers in increasingly dense data centers can utilize a combination of liquid and air cooling systems. To improve liquid cooling efficiency, liquid cooling systems position the cooling coils closer to the heat load, necessitating the design of liquid cooling piping and fluid distribution networks at the room level. In most data centers without raised floors, the fluid piping and distribution networks may be located atop the server racks, increasing the installation and maintenance costs of operating and maintaining the liquid cooling system and increasing design complexity due to the limited space atop the server racks. Flowing fluid over the tops of the servers can also pose reliability issues. Furthermore, the system may not be flexible enough to adapt and scale to meet the stringent thermal management requirements of cloud computing data centers, which demand rapid performance and power density. This is exemplified by the dynamic nature of IT hardware configuration and deployment scenarios, which necessitate upgrades and updates to IT equipment at a rate significantly faster than the lifecycle of the equipment housing the data center. Furthermore, the mismatch between the shortened lifecycle of system hardware due to rapidly changing business workloads and the relatively long lifecycle of data centers necessitates a resilient hardware solution. Summary of the Invention

[0004] According to one aspect of the present application, a liquid cooling device for a data center is provided, which may include:

[0005] one or more row cooling units, each of the row cooling units having a first fluid distribution conduit to distribute supply liquid received from a cooling liquid source to a heat load and to distribute return liquid received from the heat load back to the cooling liquid source; and

[0006] an end unit having a second fluid distribution conduit to distribute the supply liquid received from the supply line of the data center to the one or more row cooling units and to distribute the return liquid received from the one or more row cooling units to the return line of the data center, the row cooling units and the end unit being configured to connect the first fluid distribution conduit and the second fluid distribution conduit to recirculate the supply liquid and remove heat from the varying combined heat load of the data center without changing the supply line of the data center.

[0007] According to another aspect of the present application, a data center system is provided, which may include:

[0008] a plurality of electronics racks, each of the electronics racks including a plurality of blade servers, and each of the blade servers corresponding to one or more servers, wherein each of the electronics racks includes a rack manifold having a rack liquid supply line for receiving a supply of cooling liquid from a cooling liquid source and a rack liquid return line for returning warm liquid including heat exchanged from the blade servers to the cooling liquid source;

[0009] a plurality of row cooling units, each of the row cooling units having a first fluid distribution conduit for distributing supply liquid received from the cooling liquid source to the rack liquid supply line of a corresponding one of the electronics racks and distributing return liquid received from the rack liquid return line of the corresponding one of the electronics racks back to the cooling liquid source; and

[0010] an end unit having a second fluid distribution conduit for distributing supply liquid received from the supply lines of the data center system to the plurality of row cooling units and distributing return liquid received from the plurality of row cooling units to the return lines of the data center system, the row cooling units and the end unit being configured to connect the first fluid distribution conduit and the second fluid distribution conduit to recirculate the supply liquid and remove heat from the varying combined heat loads of the plurality of electronics racks without changing the supply lines of the data center system. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like references indicate similar elements.

[0012] Figure 1 is a block diagram illustrating an example of a data center facility according to one embodiment.

[0013] Figure 2 is a block diagram illustrating an example of an electronics rack according to one embodiment.

[0014] Figure 3 is a block diagram illustrating an example of a cold plate configuration according to one embodiment.

[0015] Figure 4 is a side view of a row cooling unit with an integrated liquid cooling system according to one embodiment.

[0016] Figure 5 is a side view of components of several rows of cooling units with end units forming a coolant supply circuit and a return circuit according to one embodiment.

[0017] Figure 6 is a side view of an end unit of a liquid cooling system according to one embodiment.

[0018] Figure 7 is a top view of components consisting of two of the row cooling units having end units and connected to IT room facilities to form a cool liquid supply loop and a warm liquid return loop of a liquid cooling system, according to one embodiment.

[0019] Figure 8 is a side view of components of two rows of row cooling units having end units connected to a facility configured with a liquid cooling system according to one embodiment.

[0020] Figure 9 is a flow chart illustrating an example of a method for assembling row cooling units and end units to configure a liquid cooling system according to one embodiment. DETAILED DESCRIPTION

[0021] Various embodiments and aspects of the present invention will be described with reference to the details discussed below, and the accompanying drawings will illustrate various embodiments. The following description and drawings are illustrative of the present invention and do not constitute a limitation of the present invention. Many specific details are described to provide a thorough understanding of the various embodiments of the present invention. However, in some cases, in order to provide a concise discussion of embodiments of the present invention, well-known or conventional details are not described.

[0022] Reference in the specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase "in one embodiment" in various places in the specification does not necessarily refer to the same embodiment.

[0023] The present invention discloses a row cooling unit and a connection unit having a network of integrated fluid distribution pipes. The row cooling units and the connection units can be interconnected to configure a liquid cooling system to remove heat generated by servers housed in server racks used in a data center. Components of the row cooling units and the connection units can be connected to supply and return circuits of a data center facility to distribute cooling liquid to electronic components of the servers and return heated liquid to dissipate heat. The network of fluid distribution pipes integrated into the cooling units and the connection units enables the configuration of the liquid cooling system independent of the facility's fixed infrastructure, thereby providing scalability, maintainability, and ease of maintenance, while increasing the efficiency, flexibility, availability, and reliability of the liquid cooling system, which are critical to the operation and performance of the data center.

[0024] In one embodiment, row cooling units are designed with integrated fluid distribution piping to provide cooling capacity and deliver the cooling fluid contained within the row cooling units (cooling capacity can be understood as the coolant stored and circulated within the row cooling unit's distribution piping) to the servers. The upper portion of the cooling unit may include cooling coils, and the lower portion may include connectors and hoses for distributing and returning the cooling fluid. By interconnecting the connectors and hoses on the distribution piping of adjacent row cooling units, multiple row cooling units can be assembled to deliver coolant to a row of server racks.

[0025] In one embodiment, the end unit is designed with integrated fluid distribution piping for connection to one or both row cooling units to form a fluid distribution and return loop. The end unit may include a connector located at the bottom for connecting to a complementary connector located at the bottom of the row cooling unit. Piping may extend between the bottom and top of the end unit for connection to the data center's facilities infrastructure to complete the fluid distribution system. Locating the fluid distribution system's connectors and hoses at the bottom of the end unit and row cooling units simplifies the design, maintenance, and repair of the liquid cooling system while increasing its reliability.

[0026] In one embodiment, the connector on one side of a row cooling unit or end unit is a male connector, and the connector on the opposite side is a female connector. The male connector can be connected to a rigid tube, and the female connector can be connected to a flexible hose. A row cooling unit can be connected to an adjacent row cooling unit or end unit by connecting the male connector of the row cooling unit to the female connector of the adjacent cooling unit or end unit on one side, and then connecting the female connector of the row cooling unit to the male connector of the adjacent cooling unit on the opposite side. Within the overall system loop, each row cooling unit can be individually removed, repaired, or replaced without disrupting the operation of other row cooling units or interrupting the recirculation of fluid within the overall cooling system.

[0027] Figure 1 is a block diagram illustrating an example of a data center or data center unit according to one embodiment. In this example, Figure 1 A top view of at least a portion of a data center is shown. Figure 1 According to one embodiment, a data center system 100 includes one or more rows of information technology (IT) racks 101-102 of information technology components, equipment, or tools, such as computer servers or computing nodes that provide data services to various clients via a network (e.g., the Internet). In this embodiment, each row includes an array of electronics racks, such as electronics racks 110A-110N, arranged in rows 101 and 102. However, more or fewer rows of electronics racks may be implemented. Typically, rows 101-102 are arranged in parallel, with their front ends facing each other and their back ends facing away from each other, thereby forming aisles 103 between the rows to allow personnel to walk through and perform services. However, other configurations or arrangements may also be used.

[0028] In one embodiment, each of the electronics racks (e.g., electronics racks 110A-110N) includes an enclosure to house a plurality of electronics racks of IT components operating therein. The electronics racks may include a cooling liquid manifold, a plurality of server slots, and a plurality of blade servers that can be inserted into and removed from the server slots. Each blade server represents a compute node having one or more processors, memory, and / or permanent storage devices (e.g., hard drives). At least one of the processors may be attached to a liquid cold plate (also referred to as a cold plate assembly) to receive cooling liquid. Additionally, one or more optional cooling fans are associated with the blade servers to provide air cooling to the compute nodes housed therein. Note that the cooling system 120 may be coupled to a plurality of data center systems, such as data center system 100.

[0029] In one embodiment, heat dissipation system 120 includes an external liquid circuit connected to a cooling tower or dry cooler outside the building / containment vessel. Heat dissipation system 120 may include, but is not limited to, evaporative cooling, free air, rejection of large thermal masses, waste heat recovery designs, or a chiller system with an active refrigeration cycle. Heat dissipation system 120 may include or be coupled to a coolant source that provides coolant.

[0030] In one embodiment, each blade server is modularly coupled to the de-heating liquid manifold so that the blade server can be removed from the electronics rack without affecting the operation of the electronics rack and the remaining blade servers on the de-heating liquid manifold. In another embodiment, each blade server is coupled to the de-heating liquid manifold (also known as the cooling liquid manifold) via a quick-release coupling assembly having a first liquid outlet connector coupled to a flexible hose for distributing de-heating liquid to the processors and a first liquid inlet connector. The first liquid inlet connector is configured to receive cooling liquid from the cooling liquid manifold mounted on the rear end of the electronics rack via a second liquid inlet connector. The first liquid outlet connector is configured to transmit warmer or hotter liquid, transferred from the processors to the de-heating liquid manifold via the second liquid outlet connector, back to a coolant distribution unit (CDU) within the electronics rack.

[0031] In one embodiment, a cooling liquid manifold is located at the rear end of each electronics rack and is coupled to a liquid supply line 132 to receive cooling liquid (also known as coolant) from the cooling system 120. The cooling liquid is distributed via a liquid distribution circuit attached to a cold plate assembly on which the processors are mounted to remove heat from the processors. The cold plate is configured similarly to a radiator with liquid distribution pipes connected to or embedded therein. The resulting warmer or hotter liquid, carrying heat removed from the processors, is returned to the cooling system 120 via a liquid return line 131. The liquid supply / return lines 131-132 are referred to as data center or room liquid supply / return lines (e.g., global liquid supply / return lines) and supply heat-removing liquid to all electronics racks in rows 101-102. In one embodiment, the liquid supply / return lines 131-132 may be integrated within the electronics racks 110A-110N. Electronics racks 110A-110N can be assembled to form rows 101 or 102 by interconnecting the liquid supply / return lines 131-132 of adjacent electronics racks 110A-110N and the liquid supply / return lines 131-132 of the end units with the supply / return lines of the data center facility to complete the cooling liquid supply and return loops. Liquid supply line 132 and liquid return line 131 are coupled to the heat exchanger of the CDU located within each of the electronics racks, forming the primary loop. The secondary loop of the heat exchanger is coupled to each of the blade servers in the electronics rack to deliver cooling liquid to the cold plates of the processors.

[0032] In one embodiment, data center system 100 further includes an optional air delivery system 135 to generate airflow through the air spaces of the blade servers in the electronics racks to remove heat generated by the compute nodes (e.g., servers) due to their operation, and to discharge the heat removed by the airflow to the external environment 108 outside the enclosure / room / building. For example, air supply system 135 generates a cool / cold air flow that circulates from duct 103 through electronics racks 110A-110N, removing the removed heat. The cool air flow enters the electronics racks through the front ends of the racks, and the warm / hot air flow exits the racks through the rear ends of the racks. The warm / hot air, carrying the removed heat, is exhausted from the room / building. Thus, the cooling system is a hybrid liquid-air cooling system, in which a portion of the heat generated by the processors is removed by coolant via corresponding cold plates, while the remaining heat generated by other electronic equipment or processing devices is removed by air cooling.

[0033] Figure 2 2 is a block diagram illustrating an electronic rack according to one embodiment. Figure 1 Any of the electronics racks shown, such as electronics racks 110A-110N. Figure 2 According to one embodiment, electronics rack 200 includes, but is not limited to, a CDU 201, an optional rack management unit (RMU) 202, and one or more blade servers 203A-203E (collectively, blade servers 203). Blade servers 203 can be inserted into the server slot array from either the front 204 or rear 205 of electronics rack 200. Note that while five blade servers 203A-203E are shown, more or fewer blade servers can be maintained within electronics rack 200. It should also be noted that the specific locations of CDU 201, RMU 202, and blade servers 203 are shown for illustrative purposes only; other arrangements or configurations of CDU 201, RMU 202, and blade servers 203 can also be implemented. In one embodiment, electronics rack 200 can be open to the environment or partially contained by a rack container, as long as cooling fans can generate airflow from the front to the rear.

[0034] Additionally, for at least some of blade servers 203, an optional fan module (not shown) is associated with the blade server. Each of the fan modules includes one or more cooling fans. The fan modules can be mounted at the rear end of blade server 203 or on the electronics rack to generate airflow that flows from front end 204, travels through the air space of blade server 203, and exits at rear end 205 of electronics rack 200.

[0035] In one embodiment, the CDU 201 primarily includes a heat exchanger 211, a liquid pump 212, and a pump controller (not shown), as well as other components such as a liquid reservoir, a power supply, and monitoring sensors. The heat exchanger 211 can be a liquid-to-liquid heat exchanger or a cooling coil. The heat exchanger 211 includes a first loop 223 having an inlet and an outlet, each having a first pair of liquid connectors coupled to liquid supply / return lines 131-132. The liquid supply / return lines 131-132 are integrated with the electronics rack 200 to form a primary loop. The integrated liquid supply / return lines 131-132 can be located at the bottom of the rear end 205 of the electronics rack 200. The liquid supply / return lines 131-132, also known as room liquid supply / return lines, are coupled to the cooling system 120 via the supply / return lines 131-132 of one or more adjacent electronics racks 200 and / or end units, as will be explained below. In addition, heat exchanger 211 also includes a second loop having two ports with a second pair of liquid connectors coupled to liquid manifold 225 to form a second loop. The second loop may include a supply manifold (also referred to as a rack liquid supply line) that supplies cool liquid to blade servers 203 and a return manifold (also referred to as a rack liquid return line) that returns warmer liquid to CDU 201. Note that CDU 201 can be any type of commercially available or customizable CDU. In one embodiment, heat exchanger 211 of CDU 201 can be an air-to-liquid heat exchanger or a cooling coil. The details of CDU 201 will not be described here.

[0036] Each of the blade servers 203 may include one or more IT components (e.g., a central processing unit or CPU, a graphics processing unit (GPU), memory, and / or storage devices). Each IT component may perform data processing tasks, wherein the IT component may include software that is installed in a storage device, loaded into the memory, and executed by one or more processors to perform the data processing tasks. The blade servers 203 may include a host server (referred to as a host node) coupled to one or more compute servers (also referred to as compute nodes, such as CPU servers and GPU servers, and servers with ASIC and FPGA units). The host server (having one or more CPUs) typically interfaces with clients over a network (e.g., the Internet) to receive requests for specific services, such as storage services (e.g., cloud-based storage services such as backup and / or recovery), and to execute applications to perform certain operations (e.g., image processing, video streaming, deep learning algorithms or modeling, etc., as part of a software-as-a-service or SaaS platform). In response to the request, the host server assigns the task to one or more of the compute nodes or compute servers (e.g., having one or more GPUs or ASICs) managed by the host server. The compute servers perform the actual tasks, which may generate heat during operation.

[0037] Electronics rack 200 also includes an optional RMU 202 and a power supply unit (PSU) (not shown) configured to provide and manage power to servers 203 and CDU 201. RMU 202 may be coupled to a battery backup unit (also not shown) to provide backup energy to electronics rack 200 and servers 203. The PSU may include necessary circuitry (e.g., an alternating current (AC) to direct current (DC) or DC to DC power converter, a battery, a transformer, or a regulator) to regulate power to the remaining components of electronics rack 200.

[0038] In one embodiment, the RMU 202 includes an optimization module 221 and a rack management controller (RMC) 222. The RMC 222 may include a monitor to monitor the operating status of various components within the electronics rack 200, such as the blade servers 203, the CDU 201, and the fan modules. Specifically, the monitor receives operational data from various sensors representing the operating environment of the electronics rack 200. For example, the monitor may receive operational data representing the temperatures of the processors, coolant, and airflow, which may be captured and collected by various temperature sensors. The monitor may also receive data representing the fan power and pump power generated by the fan modules and the liquid pump 212. The fan power and pump power data may be proportional to the speeds of the fan modules and the liquid pump, respectively. This operational data is referred to as real-time operational data. Note that the monitor may be implemented as a separate module within the RMU 202. Based on the operational data, the optimization module 221 performs optimization using a predetermined optimization function or optimization model to derive a set of optimal fan speeds for the fan modules and an optimal pump speed for the liquid pump 212.

[0039] Figure 3 is a block diagram illustrating a processor cold plate configuration according to one embodiment. The processor / cold plate structure 400 may be represented as follows. Figure 2 Any processor / cold plate configuration of the blade server 203 shown. Figure 3 , processor 301 is inserted into a processor socket, which is mounted on a printed circuit board (PCB) or motherboard 302, which is connected to other electronic components or circuits of the data processing system or server. Processor 301 also includes a cold plate 303 attached to processor 301, which is connected to a liquid supply line and a liquid return line. Part of the heat generated by processor 301 is removed by a coolant passing through cold plate 303. The remaining heat enters open air space 305, where it can be removed by airflow generated by cooling fan 304.

[0040] Back to Figure 2 According to one embodiment, the electronics rack 200 may also optionally include one or more liquid distribution units (LDUs), such as LDUs 250A-250E (collectively, LDUs 250), located between the blade servers 203 and the rack manifold 225. Each LDU operates as a local liquid distribution manifold and a cooling device for the blade servers 203. Figure 2 , LDU 250 is shown as being external to blade server 203, but LDU 250 may be designed into blade server 203. In this example, there is an LDU corresponding to one of blade servers 203. However, in other embodiments, although not shown, an LDU may be associated with multiple blade servers 203.

[0041] Similar to the CDU, a primary loop is formed between the LDU and the rack manifold 225, while a secondary loop is formed between the LDU and the blade servers. As a result, the liquid distribution loop is significantly shorter than in conventional systems. In conventional systems, the coolant received from the supply line 132 or from the CDU 201 must pass through each of the cold plates in the blade servers 203. As a result, the liquid distribution loop is much longer, and the power required to pump the liquid is much higher. With the LDU 250, each secondary loop connected to the cold plate of the blade server is a localized, separate loop. In one embodiment, the coolant distributed to the blade servers is a two-phase coolant that converts between liquid and vapor forms based on temperature. In this configuration, a liquid pump for the secondary loop may not be required.

[0042] Figure 4 FIG. 4 is a side view of a row cooling unit 401 with an integrated liquid cooling system according to one embodiment. The row cooling unit 401 may be integrated into a Figure 1 In any electronic rack shown, for example, Figure 2 The row cooling unit 401 may have two main parts—an upper portion 403 and a bottom portion 405 .

[0043] The upper portion 403 may include a cooling coil 407. In one embodiment, the cooling coil 407 may include some of the primary functions of the CDU 201 or the CDU. In one embodiment, the cooling unit may be a liquid-to-air cooling coil or a liquid-to-liquid cooling coil. The lower portion 405 may include connectors, pipes, and hoses for a primary fluid circuit 409 including a supply circuit and a return circuit. The supply circuit serves as Figure 1 1 and delivers the fluid to the blade servers 203. The cooling coils 407 can be connected to the main fluid circuit 409 via one or more two-way valves 423, which can be used to shut off the flow of cool liquid to the cooling coils 407 or the flow of warm liquid back from the cooling coils 407. Thus, the two-way valves can be used to shut off fluid recirculation within the row cooling unit 401.

[0044] In one embodiment, one end of the supply or return loop of the primary fluid circuit 409 is connected to a flexible hose 411. The end of the flexible hose 411 is coupled to a female connector 413. The opposite end of the supply or return loop of the primary fluid circuit 409 is connected to a male connector 415 via a rigid tube. The female and male connectors are complementary, such that the female connector 413 of a row cooling unit 401 can be connected to a corresponding male connector 415 of an adjacent row cooling unit 401 for the supply or return loop of the primary fluid circuit 409. Thus, the supply loop of the row cooling unit 401 can receive coolant from an adjacent row cooling unit 401 of a row of server racks via the flexible hose 411 and the female connector 413 of the supply line of the primary fluid circuit 409 located at the bottom 405, thereby supplying the coolant to the cooling coils 407. The return circuit of the row cooling unit 401 can return the liquid carrying heat exchanged from the blade servers 203 to the adjacent row cooling unit 401 via the pipes, flexible hoses 411, and female connectors 413 of the liquid return line of the main fluid circuit 409. In one embodiment, the cool liquid received from the adjacent row cooling unit 401 or the warm liquid returned to the adjacent row cooling unit 401 can be transported via the male connector 415 of the supply circuit or the return circuit, respectively. In one embodiment, when the flexible hose 411 and the connected female connector 413 are not connected to the adjacent row cooling unit 401, the flexible hose 411 and the connected female connector 413 are stored at the bottom 405.

[0045] The shutoff valve 417 of the supply circuit can be used to shut off the flow of coolant to the downstream cooling unit 401. In one embodiment, the shutoff valve 417 of the return circuit can be used to shut off the flow of warm liquid from the upstream cooling unit 401. Sensors such as temperature sensor 419 and pressure sensor 421 can be connected to the supply circuit or the return circuit to monitor the temperature and pressure of the coolant or warm liquid. The upper portion 403 and the bottom portion 405 are modular and can be removed, replaced, or repaired separately. The piping or fluid connection between the upper portion 403 and the bottom portion 405 can be connected using quick disconnects or flange disconnects.

[0046] Figure 5FIG2 is a side view of components of several row cooling units 401 having end units 501 to form a cool liquid supply circuit and a warm liquid return circuit, according to one embodiment. The row cooling units 401 are assembled adjacent to each other so that their bottoms 405 abut to form connection ports 425 between adjacent pairs of row cooling units 401 for the supply and return circuits. For example, a connection port 425 for the supply circuit can be formed by connecting the female connector 413 of the supply circuit of a row cooling unit 401 to the male connector 415 of the supply circuit of an adjacent upstream row cooling unit 401 supplying cool liquid. Similarly, a connection port 425 for the return circuit can be formed by connecting the female connector 413 of the return circuit of a row cooling unit 401 to the male connector of the return circuit of an adjacent downstream row cooling unit 401 carrying away warm liquid.

[0047] End units 501 connect the components of row cooling units 401 to the supply and return lines of the data center facility, completing the coolant supply and warm liquid return loops. End units 501 may include connectors on the bottom for connecting to complementary connectors on the bottom of the component's end row cooling units 431. Note that any cooling unit 401 can function as an end row cooling unit 431, as the bottom design and connectors follow the same design specifications. Piping may extend between the bottom and top of end units 501 to connect to the facility's supply and return lines from above. End units 501 may be placed adjacent to the component's end row cooling units 431 so that their bottoms abut, forming connection ports 503 for the supply and return loops. For example, connection ports 503 for the supply loop may be formed by connecting the female connector 413 of the supply loop of the end row cooling unit 431 to the male connector of the supply loop of the end unit 501 supplying coolant. Similarly, the connection port 503 for the return circuit can be formed by connecting the female connector 413 of the return circuit of the end row cooling unit 431 with the male connector of the return circuit of the end unit 501 carrying away the warm liquid.

[0048] In one embodiment, the end unit 501 may include valves and fluid pumps to shut off or control the flow of cooling or warming fluids. Advantageously, the network of fluid distribution pipes, hoses, and connectors integrated into the row cooling units 401 and the end unit 501 allows the configuration of the liquid cooling system to be independent of the fixed infrastructure of the facility, thereby providing scalability, maintainability, and ease of maintenance while increasing the efficiency and reliability of the liquid cooling system.

[0049] Figure 6is a side view of an end unit 501 of a liquid cooling system according to one embodiment. The end unit 501 may include an integrated fluid distribution conduit for connecting to the ends of one or two rows of cooling units 401, such as Figure 5 The end row cooling unit 431 is shown to form a cooling fluid supply loop and a warm liquid return loop. The end unit 501 can be symmetrical so that it can be connected to the row cooling unit 401 from either side. For example, Figure 6 The front side (outside the page) of the end unit 501 is shown covered by side panel A 505 and the back side (inside the page) covered by side panel B 507. The end unit 501 is symmetrical in the sense that when side panel A 505 is removed, the end unit 501 can be connected to the end row cooling unit 431 from the front side, or alternatively, when side panel B 507 is removed, the end unit 501 can be connected to the end row cooling unit 431 from the back side.

[0050] In one embodiment, the female side of end unit 501 has a first set of branches for the supply and return circuits, the ends of which are coupled to female connector 413. Female connector 413 can be connected to flexible hoses, as in row cooling unit 401. The opposite male side of end unit 501 has a second set of branches for the supply and return circuits, the ends of which are coupled to male connector 415. Male connector 415 can be connected to rigid tubing, as in row cooling unit 401. The female connector 413 and male connector 415 of end unit 501 can be identical to the female connector and male connector of row cooling unit 401. This means that the female connector 413 and male connector 415 of end unit 501 can mate with the corresponding male connector 415 and female connector 413 of row cooling unit 401 to achieve the full functionality of the liquid cooling system. For example, the female connector 413 or male connector 415 of the end unit 501 can be connected to the corresponding male connector 415 or female connector 413 of the end row cooling unit 431, respectively, to supply cooling fluid via the supply loop and receive hot fluid via the return loop. The first and second sets of branches of the supply and return loops can be connected to each of the two end row cooling units 431 located at the ends of two rows of server racks, respectively. In one embodiment, only one set of branches can be connected to one end row cooling unit 431 at the end of a row of server racks, while the other set of branches can be closed using a valve. The end unit 501 can have a pump 513 for pumping cooling fluid. The pumps 513 can operate in parallel or as redundant units for each other.

[0051] The top 515 of the end unit 501 may include female connectors 519 on both branches of the supply and return circuits for connecting to the facility's supply and return lines from above. The female connectors 519 can be connected to flexible hoses, allowing the tubing of the end unit 501 to be flexibly extended for connection to the facility's male connectors. The female connectors 519 used for connecting to the facility may have a larger diameter than the female connectors 413 on the bottom, allowing for greater flow rates. The two branches of the supply and return circuits are connected by tubing that traverses the top 515. A removable door can be installed below the top 515 between the female and male sides to allow access between the two connected rows of cooling units 401. In one embodiment, the female connector 519 on the top 515 of the end unit 501 can be a male connector that connects to the female connector of the facility. The choice of female or male connector type may depend on the overall system design.

[0052] Figure 7 is a top view of components of two rows of row cooling units 401 according to one embodiment, wherein the row cooling units 401 have end units 501 and are connected to the facilities of the IT room to form a cool liquid supply loop and a warm liquid return loop of the liquid cooling system. The two rows of row cooling units 401 distribute and return coolant to two rows of server racks 601. On one side of the row cooling unit 401, the female connectors of the supply loop and the return loop are connected to corresponding male connectors of an adjacent row cooling unit 401 or end unit 501. On the opposite side of the row cooling unit 401, the male connectors of the supply loop and the return loop are connected to corresponding female connectors of another adjacent row cooling unit 401 or end unit 501. The dotted lines indicate the connection direction of the female-male connectors. Although Figure 7 Only one end unit 501 connected to the utility loop is shown, but in one embodiment the utility loop may be connected to two end units 501 .

[0053] Each row of row cooling units 401 and server racks 601 is terminated by two end units 501, one at each end. Because the end units 501 are symmetrical, the end units 501 can be connected to the row cooling units 401 from either the front or the back. One end unit 501 can be connected to the supply and return lines of the facility to complete the distribution loop. In one embodiment, both end units 501 can be connected to the supply and return lines of the facility. Liquid or warm liquid can flow in either direction. That is, the fluid can flow in a clockwise or counterclockwise direction. The ability to pump fluid in either direction provides redundancy so that the row cooling units 401 can be removed, repaired, or replaced individually without interrupting the operation of the liquid cooling system. For example, in order to repair a row cooling unit 401, the supply and return circuits of two adjacent row cooling units 401 can be closed by their respective shut-off valves 417 to stop the flow of coolant into the row cooling unit 401 being repaired and the return of hot liquid from the row cooling unit 401 being repaired. As described above, both end units can be designed to be connected to the facility. This allows for easy connection to the facility even during repairs. Figure 7 Full system operation is also possible with both of the top two end row cooling units 431 on the top shown shut down simultaneously. In one embodiment, only a subset of the row cooling units 401 have upper portions 403 that include cooling coils 407 to distribute and return cooling liquid to the subset of server racks 601.

[0054] Figure 8 FIG2 is a side view of components of two rows of row cooling units 401 having end units 501 connected to a facility to construct a liquid cooling system, according to one embodiment. The end units 501 can be connected to the supply and return lines of the facility from the top to complete the distribution circuit of the liquid cooling system. The bottom of the end units 501 abuts the row cooling units 401 to form connection ports 503 for connecting the supply and return circuits to the row cooling units 401 on either side of the end units 501. Valves (not shown) and pumps 511 of the end units 501 can be used to control the flow of fluid. As shown, the network of fluid distribution piping integrated into the row cooling units 401 and the end units 501 enables the liquid cooling system to be configured to the server racks 601 independently of the fixed infrastructure of the facility, thereby providing scalability, serviceability, and ease of maintenance while increasing the efficiency and reliability of the liquid cooling system, which are critical to the operation and performance of the data center.

[0055] Figure 9 is a flow chart illustrating an example of a method 900 for assembling row cooling units 401 and end units 501 to construct a liquid cooling system according to one embodiment. The method 900 may be performed by processing logic that may include software, hardware, or a combination thereof.

[0056] At operation 901, method 900 connects a first connector of a liquid supply line of an end unit with integrated fluid distribution piping 501 to a complementary connector of a liquid supply line of a row cooling unit with integrated fluid distribution piping 401. The first connector can be a male connector or a female connector, with its complementary connector being a female connector or a male connector, respectively.

[0057] At operation 903, method 900 connects a first connector of the liquid return line of the end unit with integrated fluid distribution piping 501 to a complementary connector of the liquid return line of the row cooling unit with integrated fluid distribution piping 401. The first connector can be a male connector or a female connector, with its complementary connector being a female connector or a male connector, respectively.

[0058] At operation 905, method 900 connects a second connector of a liquid supply line of a row cooling unit having integrated fluid distribution piping to a complementary connector of a liquid supply line of an additional row cooling unit having integrated fluid distribution piping. The second connector can be a male connector or a female connector, with the complementary connector being a female connector or a male connector, respectively. In one embodiment, the female connector couples with a flexible hose, while the male connector couples with a rigid tube. Preferably, the second connector is a female connector to make it easier for an operator to manually align the female connector with the male connector when making a connection between two row cooling units.

[0059] At operation 907, method 900 connects a second connector of the liquid return line of the row cooling unit with integrated fluid distribution piping to a complementary connector of the liquid return line of the additional row cooling unit with integrated fluid distribution piping. The second connector can be a male connector or a female connector, with its complementary connector being a female connector or a male connector, respectively.

[0060] At operation 909, method 900 connects a second connector of the liquid supply line of end unit 501 with integrated fluid distribution piping to a complementary connector of the liquid supply line of the facility with integrated fluid distribution piping supplying cooling liquid. The second connector can be a male connector or a female connector, and its complementary connector can be a female connector or a male connector, respectively.

[0061] At operation 911, method 900 connects the second connector of the liquid return line of the end unit with integrated fluid distribution piping 501 to a complementary connector of the liquid return line of the facility. The second connector can be a male or female connector, with its complementary connector being a female or male connector, respectively.

[0062] In the foregoing description, embodiments of the present invention have been described with reference to specific exemplary embodiments thereof. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope of the present invention as set forth in the appended claims. The description and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A liquid cooling device for a data center, comprising: a plurality of row cooling units, each of the row cooling units having a first fluid distribution conduit to distribute supply liquid received from a cooling liquid source to a heat load and to distribute return liquid received from the heat load back to the cooling liquid source, the first fluid distribution conduit comprising a first connector, a second connector, a third connector and a fourth connector, the third connector comprising a connector complementary to a connector in the first connector, the fourth connector comprising a connector complementary to a connector in the second connector; wherein the first connector is configured to receive the supply liquid from the cooling liquid source for distribution to the heat load, the second connector is configured to return the return liquid received from the heat load to the cooling liquid source, the third connector is configured to supply the supply liquid to a first adjacent row cooling unit in the row cooling units, and the fourth connector is configured to receive return liquid returned from the first adjacent row cooling unit or a second adjacent row cooling unit in the row cooling units; and An end unit having a second fluid distribution conduit to distribute the supply liquid received from the supply line of the data center to the plurality of row cooling units, and to distribute the return liquid received from the plurality of row cooling units to the return line of the data center, the row cooling units and the end unit being configured to connect the first fluid distribution conduit and the second fluid distribution conduit to recirculate the supply liquid and remove heat from a varying combined heat load of the data center without changing the supply line of the data center.

2. The liquid cooling device according to claim 1, wherein: The first connector, the second connector, the third connector, and the fourth connector are disposed in a bottom portion of each of the row cooling units.

3. The liquid cooling device according to claim 1, wherein: The first connector includes one of a male connector or a female connector, wherein the third connector includes a connector complementary to the male connector or the female connector of the first connector, wherein the second connector includes one of a male connector or a female connector, and wherein the fourth connector includes a connector complementary to the male connector or the female connector of the second connector.

4. The liquid cooling device according to claim 3, wherein: The male connector is coupled to a rigid conduit of the first fluid distribution conduit, and wherein the female connector is coupled to a flexible hose of the first fluid distribution conduit.

5. The liquid cooling device according to claim 1, wherein: The first fluid distribution conduit further includes a shutoff valve configured to shut off supply liquid to the first adjacent row of cooling units in the rows of cooling units.

6. The liquid cooling device according to claim 1, wherein: The upper portion of the row cooling unit includes a cooling coil configured to receive the return liquid from the heat load and return the return liquid to the first fluid distribution conduit.

7. The liquid cooling device according to claim 1, wherein: The second fluid distribution pipeline comprises: a first connector configured to receive a supply of liquid from the supply line of the data center; a second connector configured to return the return liquid to the return line of the data center; a third connector configured to supply the supply liquid to a first end row cooling unit among the plurality of row cooling units; a fourth connector configured to receive return liquid from the first end row cooling unit of the plurality of row cooling units; a fifth connector configured to supply the supply liquid to a second end row cooling unit among the plurality of row cooling units; and A sixth connector is configured to receive return liquid from the second end row cooling unit of the plurality of row cooling units.

8. The liquid cooling device according to claim 7, wherein: The first connector and the second connector are disposed in an upper portion of the end unit, and wherein the third connector, the fourth connector, the fifth connector, and the sixth connector are disposed in a bottom portion of the end unit.

9. The liquid cooling device according to claim 7, wherein: The end unit includes a symmetrical front side and a rear side to allow the second fluid distribution pipe of the end unit to supply the supply liquid to the first end row cooling unit or the second end row cooling unit, and receive the return liquid from the first end row cooling unit or the second end row cooling unit from the front side or the rear side.

10. A data center system, comprising: a plurality of electronic racks, each of the electronic racks including a plurality of blade servers, and each of the blade servers corresponding to one or more servers, wherein each of the electronic racks includes a rack manifold having a rack liquid supply line for receiving a supply of cooling liquid from a cooling liquid source and a rack liquid return line for returning warm liquid including heat exchanged from the blade servers to the cooling liquid source; a plurality of row cooling units, each of the row cooling units having a first fluid distribution conduit for distributing supply liquid received from the cooling liquid source to the rack liquid supply line of a corresponding one of the electronic racks, and distributing return liquid received from the rack liquid return line of the corresponding one of the electronic racks back to the cooling liquid source, the first fluid distribution conduit comprising a first connector, a second connector, a third connector, and a fourth connector, the third connector comprising a connector complementary to a connector in the first connector to fluidly communicate supply circuits of adjacent row cooling units, the fourth connector comprising a connector complementary to a connector in the second connector to fluidly communicate return circuits of adjacent row cooling units; wherein the first connector is configured to receive supply liquid from the cooling liquid source for distribution to the corresponding one of the electronic racks; the second connector is configured to return return liquid received from the corresponding one of the electronic racks to the cooling liquid source; the third connector is configured to supply the supply liquid to a first adjacent row cooling unit in the row cooling units; the fourth connector is configured to receive return liquid returned from the first adjacent row cooling unit or a second adjacent row cooling unit in the row cooling units; and an end unit having a second fluid distribution conduit for distributing supply liquid received from a supply line of the data center system to the plurality of row cooling units and distributing return liquid received from the plurality of row cooling units to a return line of the data center system, the row cooling units and the end unit being configured to connect the first fluid distribution conduit and the second fluid distribution conduit to recirculate the supply liquid and remove heat from a varying combined heat load of the plurality of electronic racks without changing the supply line of the data center system.

11. The data center system according to claim 10, wherein: The first connector, the second connector, the third connector, and the fourth connector are disposed in a bottom portion of each of the row cooling units.

12. The data center system according to claim 10, wherein: The first connector includes one of a male connector or a female connector, wherein the third connector includes a connector complementary to the male connector or the female connector of the first connector, wherein the second connector includes one of a male connector or a female connector, and wherein the fourth connector includes a connector complementary to the male connector or the female connector of the second connector.

13. The data center system according to claim 12, wherein: The male connector is coupled to a rigid conduit of the first fluid distribution conduit, and wherein the female connector is coupled to a flexible hose of the first fluid distribution conduit.

14. The data center system according to claim 10, wherein: The first fluid distribution conduit further includes a shutoff valve configured to shut off supply liquid to the first adjacent row of cooling units in the rows of cooling units.

15. The data center system according to claim 10, wherein: An upper portion of the row cooling unit includes a cooling coil configured to receive the return liquid from a corresponding one of the electronic racks and return the return liquid to the first fluid distribution conduit.

16. The data center system of claim 10, wherein the second fluid distribution conduit comprises: a first connector configured to receive the supply liquid from the supply line of the data center system; a second connector configured to return the return liquid to the return line of the data center system; a third connector configured to supply the supply liquid to a first end row cooling unit among the plurality of row cooling units; a fourth connector configured to receive return liquid from the first end row cooling unit of the plurality of row cooling units; a fifth connector configured to supply the supply liquid to a second end row cooling unit among the plurality of row cooling units; and A sixth connector is configured to receive return liquid from the second end row cooling unit of the plurality of row cooling units.

17. The data center system according to claim 16, wherein: The first connector and the second connector are disposed in an upper portion of the end unit, and wherein the third connector, the fourth connector, the fifth connector, and the sixth connector are disposed in a bottom portion of the end unit.

18. The data center system according to claim 10, wherein: The data center system also includes two of the end units to distribute the supply liquid to the plurality of row cooling units, wherein the first fluid distribution conduit is located at a bottom of the plurality of row cooling units, and wherein only a subset of the plurality of row cooling units includes supplying the supply liquid to an upper portion of a subset of the plurality of electronic racks.

Images