Data center delivery point layout and configuration

Abstract

Disclosed herein is a layout, arrangement, and configuration of room airflow, server racks, and cooling delivery subsystems supporting a data center having a system architecture with separate compute and storage delivery points (PoDs). A cooling row is arranged in the rear side of a rack housing a PoD to supply a cooling delivery subsystem to the PoD, the cooling delivery subsystem comprising a recirculation loop that supplies cooling fluid to the PoD and returns warm fluid to a cooling source. The modular cooling units of the cooling row include cooling coils for cooling airflow leaving the rack. The cooling capacity of the cooling fluid supplied to the cooling delivery subsystem for the PoD can be independently adjusted and controlled in a feedback loop based on the temperature of the air after the PoD is cooled by the cooling coil, where the air temperature is, for example, the outlet air temperature from the cooling row housing the PoD).

Description

Technical Field

[0001] Embodiments of the present invention generally relate to data center configurations. More specifically, embodiments of the present invention relate to the design and layout of thermal management and facility infrastructure to support the deployment and operation of electronics racks in data centers with different compute and storage point of delivery (PoD) configurations. Background Art

[0002] Computing clusters and data centers serve the computing, storage, and other information technology (IT) needs of today's information economy. With the rapid development of technology and the continuous improvement of performance requirements, new applications and workloads such as machine learning, video streaming, high-performance computing, and new computing architectures such as the separation of computing and storage resources or PoDs have been significantly developed. This computing architecture with separated computing and storage PoDs is of great significance to the design and layout of data center facilities and hardware infrastructure required to support them. For example, hardware configurations such as rack layout, system capacity, power rating, cooling method, networking configuration, etc. may be different. Therefore, traditional data center infrastructure may not be able to effectively and cost-effectively support the deployment and operation of new computing architectures. Data center infrastructure may also not be easily adapted to support computing architectures with different computing and storage separation configurations (e.g., computing and storage PoDs of different sizes and ratios).

[0003] Heat dissipation or thermal management is an important consideration in data center design. As the number of high-performance electronic components such as high-performance processors and storage devices in computing and storage containers has steadily increased, the amount of heat generated and dissipated during ordinary operation of the PoD has also increased. If the ambient temperature in which the data center operates is allowed to increase over time, the reliability of the electronic components used in the data center will decrease. Maintaining a proper thermal environment is critical to the proper operation of these PoDs in the data center, as well as their performance and lifespan.

[0004] Thermal management of high-performance PoDs with increasing power density in data centers can use a combination of liquid and air cooling systems. However, it is difficult to flexibly adjust and scale the cooling infrastructure of the data center to meet the changing hardware configurations of the separated computing and storage PoDs. The need to upgrade and update IT equipment and deploy new computing architectures with separated computing and storage PoDs at a much faster rate than the life cycle of the facilities that house the data center makes it challenging to realize the full potential of the new computing architecture. Summary of the invention

[0005] According to one aspect of the present application, a liquid cooling device for a data center is provided, which may include:

[0006] a plurality of interconnected modular cooling units, each of the modular cooling units having a fluid distribution subsystem configured to supply coolant received from a coolant source to a liquid-to-air heat exchanger and to return warm liquid from the liquid-to-air heat exchanger to the coolant source; and

[0007] a plurality of cooling coils of the liquid-to-gas heat exchanger, each of the cooling coils being configured to receive cooling liquid from a corresponding one of the modular cooling units to remove heat from an airflow of a heat load, wherein a cooling capacity of the cooling liquid supplied to the liquid cooling device is controlled based on a temperature of the airflow after the liquid-to-gas heat exchanger.

[0008] According to another aspect of the present application, a data center system is provided, which may include:

[0009] one or more rows of a plurality of electronics racks, each of the electronics racks housing a plurality of electronic components; and

[0010] One or more rows of row cooling units, each row of row cooling units being associated with a corresponding row of electronic racks, wherein each row of cooling units comprises:

[0011] a plurality of interconnected modular cooling units, each of the modular cooling units having a fluid distribution subsystem configured to supply coolant received from a coolant source to a liquid-to-air heat exchanger and to return warm liquid from the liquid-to-air heat exchanger to the coolant source; and

[0012] a plurality of cooling coils of the liquid-to-gas heat exchanger, each of the cooling coils being configured to receive cooling liquid from a corresponding one of the modular cooling units to remove heat from an airflow from one of the electronic racks, wherein after the airflow from the corresponding row of the electronic racks transfers heat to the liquid-to-gas heat exchanger, the cooling capacity of the cooling liquid supplied to the row cooling unit is controlled based on the temperature of the airflow from the row cooling unit, and wherein the layout of the system is configured so that the airflow from the row cooling unit flows into an adjacent row of the electronic racks and its associated row cooling unit for an entire area or portion of the data center in the same direction as the inlet airflow. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments of the invention are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0014] Figure 1 is a block diagram illustrating an example of a data center facility.

[0015] Figure 2is a block diagram illustrating an example of an electronics rack according to one embodiment.

[0016] Figure 3 is a block diagram illustrating an example of a cold plate configuration according to one embodiment.

[0017] Figure 4 is a top view layout diagram of cooling rows and associated rack rows according to one embodiment, showing that the cooling rows are arranged in the same direction throughout the data center.

[0018] Figure 5 is a top view layout of cooling rows and associated racks in a data center, arranged such that each pair of cooling rows faces each other, according to one embodiment.

[0019] Figure 6 is a top view layout of cooling rows and associated racks in a data center, arranged such that upper and lower portions of the cooling rows face in opposite directions, according to one embodiment.

[0020] Figure 7 is a side or rear view of a cooling row showing fluid connections to each cooling coil for liquid supply and return circuits, each cooling coil being used to individually cool each rack, according to one embodiment.

[0021] Figure 8 is a side or rear view of a cooling row showing the cooling coil banks for liquid supply and return circuits to each cooling coil bank that is shared between two adjacent racks according to one embodiment.

[0022] Fig. 9 is a top view layout of cooling rows and associated racks configured such that the temperature of air flow from the cooling rows is used to independently control the cooling capacity of coolant supplied to the cooling rows, according to one embodiment.

[0023] Fig.10 is a top view of a cooling row connected to a facility infrastructure prior to attaching a rack to form a cool liquid supply loop and a warm liquid return loop of a liquid cooling system according to one embodiment. DETAILED DESCRIPTION

[0024] Various embodiments and aspects of the present invention will be described with reference to the details discussed below, and the accompanying drawings will illustrate various embodiments. The following description and the accompanying drawings are illustrative of the present invention and should not be construed as limiting the present invention. Many specific details are described to provide a thorough understanding of various embodiments of the present invention. However, in some cases, well-known or conventional details are not described in order to provide a brief discussion of embodiments of the present invention.

[0025] References in the specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the invention. The phrase "in one embodiment" appearing in various places in the specification does not necessarily all refer to the same embodiment.

[0026] The present invention discloses a layout, arrangement and configuration of room airflow, server racks and cooling delivery subsystems supporting a data center having a system architecture with separated computing and storage delivery points (PoDs). The computing or storage PoDs, which may be referred to as IT PoDs, and the cooling delivery subsystems serving the IT PoDs are arranged so that the thermal management of the IT PoDs can be flexibly supported by the infrastructure of the data center housing the IT PoDs. Since the number, ratio and configuration of computing and storage PoDs may be different to serve different IT applications, the computing and storage PoDs may have different power loads, power densities, etc., thereby requiring different power and cooling delivery subsystems.

[0027] In one embodiment, the cooling delivery subsystem may include liquid cooling, or a combination of liquid cooling and air cooling to flexibly support the thermal management functions of the IT PoD. The cooling capacity of the liquid supplied to the cooling delivery subsystem of the PoD can be independently adjusted and controlled in a feedback loop based on the temperature of the air after the PoD is cooled by the cooling liquid, such as the inlet and / or outlet air temperature from the rack housing the PoD. Changes in cooling capacity may include changes in the temperature or volume of the cooling liquid. Advantageously, the computing and storage PoDs can have different numbers of racks with different power loads and still have independent temperature control, so that IT PoDs with different power densities can be managed with a consistent thermal management solution.

[0028] In one embodiment, a cooling row is arranged in the back side of each row of racks that accommodates the PoD to supply a cooling delivery subsystem to the PoD. The cooling delivery subsystem may include a recirculation loop that supplies cooling fluid to the PoD and returns warm fluid to the cooling source. The design of the cooling row eliminates the traditional hot aisle and cold aisle layout or any containment structure of the data center by effectively treating the entire space of the data center as a cold aisle. The cooling row includes modular cooling units arranged in a row to supply a liquid cooling loop to cool the row of racks. The main unit in each cooling unit may be a cooling coil for cooling the airflow leaving one or more racks. The cooling coil is supplied with cooling liquid and used as a heat exchanger to cool the temperature of the airflow heated by the electronic components housed in the rack.

[0029] In one embodiment, in addition to or in place of the liquid-to-air cooling coils, liquid-to-liquid cooling coils (e.g., rack-level liquid cooling units, rack manifolds, immersion cooling boxes, etc.) may be connected to the cooling distribution subsystem. Different types of IT racks (e.g., computing racks, storage racks, networking equipment racks) may be arranged at any location of the PoD. Any IT space in the data center may be configured as a computing PoD, a storage PoD, or a heterogeneous computing and storage PoD. The cooling row design may be flexibly adapted to meet the cooling requirements of various layouts, arrangements, heat loads, and configurations of the PoD.

[0030] In one embodiment, the cooling rows arranged in the back side of the rows of racks can be arranged in the same direction throughout the data center room. After being cooled by the cooling coils of the cooling rows, the air from the racks flows in approximately the same direction. The cooling capacity of the cooling liquid supplied to each row can be independently controlled based on the temperature of the air flow from the cooling row. Thus, the temperature of the air from different cooling rows can be controlled to be relatively uniform regardless of the power load of the racks associated with the cooling row. Due to the uniformly directed airflow from the racks and the uniform temperature of the airflow, each location in the room can be considered the same in terms of thermal management.

[0031] In one embodiment, even when the cooling rows arranged in the rear side of the rack row face each other so that the air from each of a pair of cooling rows flows toward each other, the air temperature from the two cooling rows can be independently controlled to be relatively uniform. This layout is similar to the alternating hot aisle and cold aisle arrangement of a conventional PoD, but due to the independent control of the air temperature of the cooling rows, the layout does not experience the temperature gradient between the hot aisle and the cold aisle of the conventional PoD.

[0032] In one embodiment, the cooling rows and associated racks can be divided into upper and lower sections. The upper sections of all rows can be in one direction, and the lower sections of all rows can be in the opposite direction. Thus, air from the two sections of the row can flow in opposite directions. This arrangement can also lend itself to thermal management through independent temperature control of the rows, thereby removing any temperature gradients between locations in the room, or keeping the room temperature uniform.

[0033] In one embodiment, one of the key components of a cooling row includes a modular cooling unit with integrated fluid distribution piping that can be interconnected to form a cooling distribution subsystem that supplies cooling liquid to cooling coils to remove heat generated by electronic components housed within racks associated with the cooling row. The components of the modular cooling unit can be connected to the cooling liquid supply and return loops of the data center facility to distribute cooling liquid to the electronic components of the racks and return heated liquid for heat dissipation. The cooling capacity of the cooling rows can be independently controlled. The network of fluid distribution piping integrated into the cooling unit enables the configuration of the cooling distribution subsystem to be independent of the fixed infrastructure of the facility, thereby providing scalability, serviceability, ease of maintenance, while improving the efficiency, resilience, availability and reliability of the liquid cooling system.

[0034] Figure 1 is a block diagram illustrating an example of a data center or data center unit. In this example, Figure 1 A top view of at least a portion of a data center is shown. Figure 1 According to one embodiment, a data center system 100 includes one or more rows of electronic racks of information technology (IT) components, equipment, or tools 101-102, such as computer servers, computing nodes, or storage nodes that provide data services to various clients via a network (e.g., the Internet). In this embodiment, each row includes an array of electronic racks such as electronic racks 110A-110N arranged in rows 101 and 102. However, more or fewer rows of electronic racks may be implemented. Each row may be part of a computing PoD, a storage PoD, or a heterogeneous computing and storage PoD. Typically, rows 101-102 are aligned in parallel, with the front ends facing each other (or facing away from each other; for example, if deployed Figure 1 The multiple systems shown, then, depend on how the racks are grouped, two of the racks can be understood as facing away from each other, and the rear ends facing away from each other, thereby forming a passage 103 between them to allow management personnel to walk and perform services therein. However, other configurations or arrangements may also be applied.

[0035] In one embodiment, each of the electronic racks (e.g., electronic racks 110A-110N) includes a housing to house multiple electronic racks of IT components operating therein. The electronic racks may include a heat dissipation liquid manifold, multiple server slots, and multiple blade servers that can be inserted into and removed from the blade servers or server slots. Each blade server represents a computing node having one or more processors, memory, and / or permanent storage devices (e.g., hard disks). At least one of the processors may be connected to a liquid cold plate (also referred to as a cold plate assembly) to receive a cooling liquid. In addition, one or more optional cooling fans are associated with the blade servers to provide air cooling to the computing nodes housed therein. Note that the heat dissipation system 120 may be coupled to multiple data center systems, such as the data center system 100.

[0036] In one embodiment, the heat dissipation system 120 includes an external liquid loop connected to a cooling tower or dry cooler outside the building / containment vessel. The heat dissipation system 120 may include, but is not limited to, evaporative cooling, free air, rejection of large thermal mass, waste heat recovery design, or a chiller system with an active refrigeration cycle. The heat dissipation system 120 may include or be coupled to a cooling liquid source that provides a cooling liquid.

[0037] In one embodiment, each blade server is modularly coupled to a heat dissipation liquid manifold so that a blade server can be removed from an electronics rack without affecting the operation of the remaining blade servers on the electronics rack and the heat dissipation liquid manifold. In another embodiment, each blade server is coupled to a heat dissipation liquid manifold (also referred to as a cooling liquid manifold) by a quick release coupling assembly having a first liquid inlet connector and a first liquid outlet connector coupled to a flexible hose to distribute heat dissipation liquid to a processor. The first liquid inlet connector is used to receive heat dissipation liquid from a heat dissipation liquid manifold mounted at a rear end of the electronics rack through a second liquid inlet connector. The first liquid outlet connector is used to emit warmer or hotter liquid carrying heat exchanged from the processor to the heat dissipation liquid manifold via the second liquid outlet connector back to a coolant distribution unit (CDU) within the electronics rack.

[0038] In one embodiment, a cooling liquid manifold is disposed at the rear end of each electronic rack and is coupled to a liquid supply line 131 to receive cooling liquid (also referred to as coolant) from the cooling system 120. The cooling liquid is distributed through a liquid distribution loop attached to a cold plate assembly to remove heat from the processors, where the processors are mounted on the cold plate assembly. The cold plate is configured similar to a radiator with a liquid distribution tube attached or embedded therein. The resulting warmer or hotter liquid carrying heat exchanged from the processors is delivered back to the cooling system 120 via a liquid return line 132.

[0039] Liquid supply / return lines 131-132 are referred to as data center or room liquid supply / return lines (e.g., overall liquid supply / return lines) that supply cooling liquid to all of the electronic racks in rows 101-102. As will be discussed, in one embodiment, liquid supply / return lines 131-132 may be integrated within modular cooling units connected to the electronic racks 110A-110N. Electronic racks 110A-110N may be assembled to form rows 101 or 102 by interconnecting the liquid supply / return lines 131-132 of adjacent modular cooling units to form a cooling row, and connecting the liquid supply / return lines 131-132 of the modular cooling units of the cooling row to the cooling liquid manifolds of the electronic racks 110A-110N to complete the cooling liquid supply and return loops with the supply / return lines of the data center facility. Liquid supply lines 131 and liquid return lines 132 are coupled to the heat exchangers of the CDUs located within each of the electronic racks, thereby forming a main loop. A secondary loop of the heat exchanger is coupled to each blade server in the electronics rack to deliver cooling fluid to the cold plate of the processor.

[0040] In one embodiment, the data center system 100 further includes an optional air flow delivery system 135 to generate airflow so that the airflow passes through the air space of the blade servers of the electronic rack to exchange the heat generated by the computing nodes (e.g., servers) due to the operation of the computing nodes, and discharge the heat exchanged by the airflow to the external environment 108 outside the house / room / building. For example, the air supply system 135 generates a cool / cold air flow to circulate through the electronic racks 110A-110N from the channel 103 to carry away the exchanged heat. The cold air flow enters the electronic rack through the front end of the electronic rack, and the warm / hot air flow leaves the electronic rack from the rear end of the electronic rack. The warm / hot air with the exchanged heat is discharged from the room / building. Thus, the cooling system is a hybrid liquid-air cooling system, in which part of the heat generated by the processor is removed by the coolant through the corresponding cold plate, and the remaining part of the heat generated by other electronic equipment or processing devices is removed by air flow cooling.

[0041] In one embodiment, the electronics rack 102 may have its own internal airflow delivery system, such as a fan, to move airflow through the air space of the electronics rack 102 to exchange heat generated by the computing or storage PoD with the cooling coils of the modular cooling units of the cooling row. The cooling coils are supplied with cooling liquid from the liquid supply / return lines 131-132 to cool the airflow. After exchanging heat with the cooling coils, the cooling airflow of the electronics rack 102 may flow out of the cooling row and be used in a feedback loop to regulate and control the cooling capacity of the cooling liquid supplied to the cooling row of the electronics rack 102.

[0042] Figure 22 is a block diagram showing an electronic rack according to one embodiment. The electronic rack 200 may represent Figure 1 Any of the electronic racks shown, for example, electronic racks 110A-110N. Figure 2 According to one embodiment, the electronic rack 200 includes, but is not limited to, a CDU 201, an optional rack management unit (RMU) 202, and one or more blade servers 203A-203E (collectively referred to as blade servers 203). The blade servers 203 can be inserted into the server slot array from the front end 204 or the back end 205 of the electronic rack 200, respectively. Note that although five blade servers 203A-203E are shown here, more or fewer blade servers can be maintained in the electronic rack 200. It should also be noted that the specific locations of the CDU 201, RMU 702, and blade servers 203 are shown only for illustrative purposes; other arrangements or configurations of the CDU 201, RMU 202, and blade servers 203 can also be implemented. In one embodiment, the electronic rack 200 can be open to the environment or partially contained by a rack container, as long as the cooling fan can generate airflow from the front end to the back end. The electronic rack 200 can be part of a computing PoD, a storage PoD, or a heterogeneous computing and storage PoD.

[0043] In addition, for at least some blade servers 203, an optional fan module (not shown) is associated with the blade server. Each of the fan modules includes one or more cooling fans. The fan module can be mounted on the rear end of the blade server 203 or on the electronics rack to generate an airflow that flows out from the front end 204, travels through the air space of the blade server 203, and exits at the rear end 205 of the electronics rack 200.

[0044] In one embodiment, the CDU 201 mainly includes a heat exchanger 211, a liquid pump 212 and a pump controller (not shown), as well as some other components such as a liquid reservoir, a power supply, a monitoring sensor, etc. The heat exchanger 211 can be a liquid-liquid heat exchanger or a cooling coil. The heat exchanger 211 includes a first loop 223 having an inlet and outlet port, which has a first pair of liquid connectors connected to liquid supply / return lines 131-132, and the liquid supply / return lines 131-132 are integrated into the electronic rack 200 to form a main loop. The integrated liquid supply / return lines 131-132 can be set at the bottom of the rear end 205 of the electronic rack 200. The liquid supply / return lines 131-132 are also called room liquid supply / return lines, which are connected to the heat dissipation system 120 through the supply / return lines 131-132 of the modular cooling units of the cooling row, which will be explained below. In addition, the heat exchanger 211 also includes a secondary loop with two ports having a second pair of liquid connectors connected to the liquid manifold 225 to form a secondary loop, which may include a supply manifold (also referred to as a rack liquid supply line) and a return manifold (also referred to as a rack liquid return line), the supply manifold supplies cool liquid to the blade servers 203, and the return manifold returns warmer liquid to the CDU 201. Note that the CDU 201 can be any type of commercially available or customizable CDU. In one embodiment, the heat exchanger 211 of the CDU 201 can be an air-to-liquid heat exchanger or a cooling coil. The details of the CDU 201 will not be described here.

[0045] Each blade server 203 may include one or more IT components (e.g., a central processing unit or CPU, a graphics processing unit (GPU), a memory, and / or a storage device). Each IT component may perform data processing tasks, wherein the IT component may include software installed in a storage device, loaded into a memory, and executed by one or more processors to perform data processing tasks. In one embodiment, the blade server 203 may be part of a computing PoD, and may include a master server (referred to as a master node) connected to one or more computing servers (also referred to as computing nodes, such as CPU servers and GPU servers, and servers with ASIC and FPGA units). The master server (with one or more CPUs) typically interfaces with a client through a network (e.g., the Internet) to receive requests for specific services, such as storage services (e.g., cloud-based storage services, such as backup and / or recovery), executing applications to perform certain operations (e.g., image processing, video streaming, deep learning algorithms or modeling, etc., as part of a service software or SaaS platform). In response to the request, the master server assigns the task to one or more of the computing nodes or computing servers (e.g., with one or more GPUs or ASICs) managed by the master server. The compute servers perform the actual tasks, which may generate heat during operation. In one embodiment, when the IT component is part of a storage PoD that includes an array of storage devices, the IT component may provide storage services such as cloud-based storage services to the client or the compute PoD.

[0046] Electronic rack 200 also includes an optional RMU 202 and a power supply unit (PSU) (not shown) configured to provide and manage power to server 203 and CDU 201. RMU 202 may be coupled to a battery backup unit (also not shown) to provide backup energy to electronic rack 200 and server 203. PSU may include necessary circuitry (e.g., alternating current (AC) to direct current (DC) or DC to DC power converter, battery, transformer or regulator, etc.) to regulate power supplied to the remaining components of electronic rack 200.

[0047] In one embodiment, the RMU 202 includes an optimization module 221 and a rack management controller (RMC) 222. The RMC 222 may include a monitor to monitor the operating status of various components within the electronic rack 200, such as the blade server 203, the CDU 201, and the fan module. Specifically, the monitor receives operating data representing the operating environment of the electronic rack 200 from various sensors. For example, the monitor may receive operating data representing the temperature of the processor, the coolant, and the airflow, which may be captured and collected by various temperature sensors. The monitor may also receive data representing the fan power and pump power generated by the fan module and the liquid pump 212, which may be proportional to the speed of each of the fan module and the liquid pump. These operating data are referred to as real-time operating data. Note that the monitor may be implemented as a separate module within the RMU 202. Based on the operating data, the optimization module 221 performs optimization using a predetermined optimization function or optimization model to derive a set of optimal fan speeds for the fan module and an optimal pump speed for the liquid pump 212. In one embodiment, the RMU 202 may adjust and control the cooling capacity of the cooling liquid supplied to the cooling unit connected to the rack 200 based on the temperature of the air flow after heat exchange through the heat exchanger 211 or the cooling coil of the cooling unit.

[0048] Figure 3 is a block diagram illustrating a processor cold plate configuration according to one embodiment. The processor / cold plate structure 301 / 303 may be represented as follows Figure 2 Any processor / cold plate configuration of the blade server 203 shown. Figure 3 , the processor 301 is inserted into a processor socket, wherein the processor socket is mounted on a printed circuit board (PCB) or motherboard 302, which is connected to other electronic components or circuits of the data processing system or server. The processor 301 also includes a cold plate 303 attached thereto, and the cold plate 303 is connected to a liquid supply line and a liquid return line. Part of the heat generated by the processor 301 is removed by the coolant through the cold plate 303. The remainder of the heat enters the open air space 305, which can be removed by the airflow generated by the cooling fan 304.

[0049] Back to Figure 2 According to one embodiment, the electronic rack 200 also includes one or more liquid distribution units (LDUs), such as LDUs 250A-250E (collectively referred to as LDUs 250), located between the blade servers 203 and the rack manifold 225. Each LDU operates as a local liquid distribution manifold and a cooling device for the blade servers 203. Figure 22, the LDU 250 is shown as being external to the blade server 203, but it may be designed in the blade server 203. In this example, there is an LDU corresponding to one of the blade servers 203. However, in other embodiments, although not shown, the LDU may be associated with multiple blade servers 203.

[0050] Similar to the CDU, a primary loop is formed between the LDU and the rack manifold 225, while a secondary loop is formed between the LDU and the blade server. As a result, the liquid distribution loop is significantly shortened compared to conventional systems. In conventional systems, the cooling liquid received from the supply line 132 or from the CDU 201 must pass through each cold plate in the blade server 203. As a result, the liquid distribution loop is much longer and the power requirement for pumping the liquid is much higher. With the LDU 250, each secondary loop connected to the cold plate of the blade server is a local single loop. In one embodiment, the cooling liquid distributed to the blade server is a two-phase cooling liquid that converts between liquid form and vapor form based on temperature. In this configuration, a liquid pump for the secondary loop may not be required.

[0051] In one embodiment, a cooling row is connected to each rack row, such as Figure 2 The rack 200 is provided to accommodate the PoD to provide a cooling distribution subsystem to the PoD. The cooling distribution subsystem may include a recirculation loop that supplies cooling fluid to the PoD and returns warm fluid to a cooling source of the data center infrastructure. Different types of IT racks (e.g., computing racks, storage racks, networking device racks) may be arranged at any location of the PoD. Any space in the data center may be configured as a computing PoD, a storage PoD, or a heterogeneous computing and storage PoD. The cooling row may be flexibly configured to meet the cooling requirements of various layouts, arrangements, heat loads, and configurations of the PoD.

[0052] Figure 41 is a top view layout of cooling rows 401 and associated rows of racks 200 according to one embodiment, showing that the cooling rows 401 are arranged in the same direction throughout the data center 100. Cooling fans in the racks 200 can draw air heated by electronic components through the air spaces of the racks 200 to be cooled by the cooling coils (not shown) of the cooling rows 401. Cooling liquid is supplied to the cooling coils of the cooling rows 401 from the cooling supply / cooling return lines 131-132. For all racks 200 of the cooling rows 401, as shown by the outlet airflow 403, and for all racks 200 of a various cooling rows 401, the air from the racks 200 flows out of the cooling rows 401 in the same direction after being cooled by the cooling coils of the cooling rows 401. Thus, the outlet airflow 403 from one cooling row 401 can be in the same direction as the inlet airflow into the adjacent cooling row 401. For example, air outlets from one row of racks 200 and associated cooling rows 401 may face air inlets of an adjacent row of racks 200 and associated cooling rows 401 .

[0053] In one embodiment, the temperature of the outlet airflow 403 from the cooling row 401 can be measured to control the cooling capacity, such as temperature or volume, of the cooling liquid supplied to the cooling row 401. The rack 200 can accommodate different arrangements of computing or storage PoDs, regardless of the power load requirements of the PoDs. The number, ratio, and configuration of computing and storage PoDs can be changed to serve different IT applications. One PoD 405 is shown surrounding two rows of racks 200. Because the racks 200 housing the PoDs can have different power loads and power densities, different amounts of heat can be dissipated, and different cooling delivery subsystems may be required. In one embodiment, in addition to or in lieu of providing liquid cooling for airflow, the cooling delivery subsystem may include liquid-to-liquid cooling coils, Figure 3 The cold plate 303 shown, or other liquid-to-liquid cooling devices (eg, rack-level liquid cooling units, rack manifolds, immersion cooling boxes, etc.) are used to flexibly support the thermal management functions of the PoD.

[0054] In one embodiment, based on the temperature of the outlet airflow 403 from the different cooling rows 401, the cooling capacity of the different cooling rows 401 can be independently controlled to address the different power loads of the PoDs. Thus, the air temperature from the different cooling rows 401 can be controlled to be relatively uniform, regardless of the power load of the racks 200 connected to the cooling rows 401. Due to the uniformly directed airflow from the racks 200 and the uniform temperature of the airflow, each location in the data center 100 can be considered to be the same in terms of thermal management. This means that any change in IT racks can be deployed in any IT space in the data center 100 without causing any thermal impact, or only causing negligible thermal impact, on any other space. Thus, all data centers 100 can be considered to use cold aisles of cooling rows 401, which also enables the configuration of the cooling distribution subsystem to be independent of the fixed cooling infrastructure of the data center 100.

[0055] Figure 5 4 is a top view layout of cooling rows 401 and associated racks 200 in a data center 100 according to one embodiment, the cooling rows 401 and associated racks 200 are arranged so that each pair of cooling rows 401 faces each other. This embodiment can be considered to merge the design concepts proposed in the current disclosure with a traditional hot aisle-cold aisle data center. For example, when a data center is modified to support the current disclosed design, instead of a complete reconstruction, some existing facilities such as power facilities or networking facilities can remain intact, and the current thermal design configuration can also be implemented to save costs. The outlet airflows 503 from each cooling row 401 of the pair flow toward each other. In one embodiment, if a pair of racks 200 cooled by the pair of cooling rows 401 have similar power loads based on the outlet airflow 503 from either cooling row 401, the cooling capacity of the pair of cooling rows 401 can be jointly controlled.

[0056] In one embodiment, the cooling capacity of the pair of cooling rows 401 may be independently controlled to account for the different power loads of the pair of racks 200 cooled by the pair of cooling rows 401 based on the temperature of the outlet airflow 503 from the respective cooling rows 401, even though the outlet airflow 503 from the pair of cooling rows 401 may be mixed. Independent control of the pair of cooling rows 401 allows the temperature of the outlet airflow 503 from the two cooling rows 401 to be relatively uniform. This arrangement of racks 200 is similar to the alternating hot aisle cold aisle arrangement of a conventional PoD, but since the cooling capacity of the cooling rows 401 connected to the racks 200 is independently controlled, the arrangement does not experience the temperature gradient between the hot aisle and the cold aisle of a conventional PoD.

[0057] Figure 6is a top view layout of cooling rows 401 and associated racks 200 in a data center 100 according to one embodiment, wherein the cooling rows 401 and associated racks 200 are arranged such that the upper and lower portions of the cooling rows face opposite directions. The upper portions of all cooling rows 401 may be in one direction, and the lower portions of all cooling rows 401 may be in the opposite direction. Thus, the outlet airflow 603 from the two portions of the cooling rows 401 may flow in opposite directions. This layout enables different data center room airflow dynamic designs that can be used for different use cases, such as PoD partitions. The cooling capacity of the cooling rows 401 may also be independently controlled to remove any temperature gradients between locations in the data center 100, or to maintain a uniform room temperature. In one embodiment, optimization and parameter analysis may be used to optimally control the airflow 603. For example, Figure 2 The RMU 202 may monitor the temperature and volume of the airflow through the rack 200. Based on this data, the optimization module 221 of the RMU 202 may perform optimization using a predetermined optimization model to derive a set of optimized fan speeds for the fan modules of the rack 200, thereby controlling the airflow 603.

[0058] Figure 7 is a side or rear view of a cooling row 401 according to one embodiment, showing fluid connections 701 for liquid supply and return loops 131-132 to each cooling coil 703, which is used to individually cool each rack 200. A cooling row may include modular cooling units arranged in a row to supply a liquid cooling loop to the racks 200 cooling the row. The primary unit in each cooling unit may be a cooling coil 703 for cooling an airflow blowing through one or more racks 200. The cooling coils are supplied with cooling liquid from a cooling supply line 131 through a cooling distribution subsystem, and the cooling coils act as liquid-to-air heat exchangers to cool the temperature of the airflow carrying heat from the electronic components housed in the racks 200.

[0059] In one embodiment, in addition to or in lieu of the liquid to air cooling coil 703, Figure 3 The liquid-to-liquid cooling device (e.g., a rack-level liquid cooling unit, a rack manifold, an immersion cooling box, etc.) or the cold plate 303 may be connected to the cooling distribution subsystem of the cooling row 401. In one embodiment, if the liquid-to-liquid cooling device has sufficient cooling capacity to cool the rack 200, it may not be necessary to use the cooling coil 703 to perform liquid-to-air cooling on the airflow. In this case, the cooling coil 703 can be easily removed.

[0060] Different types of IT racks (e.g., computing racks, storage racks, networking device racks) can be arranged at any position of the rack 200. Any portion of the rack 200 can be configured or divided into computing PoDs, storage PoDs, or heterogeneous computing and storage PoDs. The cooling distribution subsystem including the liquid-air cooling coils 703 and the cooling rows 401 of liquid-liquid devices has a cooling capacity that can be flexibly adapted to support the cooling requirements of various layouts, arrangements, heat loads, and configurations of PoDs accommodated in the rack 200.

[0061] In one embodiment, the modular cooling units of the cooling row 401 have integrated fluid distribution pipes that can be interconnected to form a cooling distribution subsystem for distributing cooling liquid from the cooling supply line 131 to the cooling coils 703 and liquid-to-liquid devices of the racks 200, and returning the heated liquid through the return line 132. For example, the fluid connection 701 and the fluid port 705 can supply cooling liquid to the cooling coil 703 and / or a liquid-to-liquid cooling device such as a rack-level liquid cooling unit, a rack manifold, an immersion cooling box, a cold plate, etc., and return the heated liquid from the cooling coil 703 and / or the liquid-to-liquid cooling device. In one embodiment, the fluid port 705 can be used as a redundant port for the primary port (e.g., the fluid connection 701) of the cooling coil 703. In one embodiment, the cooling coil 703 can be located higher than the rack 200 to more effectively exchange heat with the warmed air from the rack 200. The modular cooling units of cooling row 401 are serviceable from the easily accessible rear side of cooling row 401. In one embodiment, cooling row 401 may include internal structure to prevent fluid spillage in the event of any leaks in the cooling distribution subsystem.

[0062] The cooling capacity of the cooling row 401 can be independently controlled by controlling the temperature and / or volume of the cooling liquid from the cooling supply line 131. In one embodiment, the cooling capacity of the modular cooling units of the cooling row 401 can be independently controlled by a pump (not shown) that controls the volume of the cooling liquid supplied to the rack 200. Figure 7 The cooling coils 703 of each modular cooling unit of the cooling row 401 are shown to meet the cooling needs of each rack 200 separately. The integrated fluid distribution piping and modular configuration of the cooling row 401 enable the configuration of the cooling distribution subsystem for the racks 200 to be independent of the fixed infrastructure of the facility, thereby improving the scalability, efficiency, resilience, availability and reliability of the liquid cooling system.

[0063] Figure 8is a side or rear view of a cooling row 401 according to one embodiment, showing fluid connections for liquid supply and return loops 131-132 to each cooling coil group 703 shared between two adjacent racks 200. The two cooling coils of the cooling coil group 703 can also serve as redundant backups for each other. Cooling liquid can be supplied to the cooling coil group 703 from the cooling distribution subsystem of a pair of adjacent modular cooling units through the fluid connection 701. The cooling coil group 703 can function as a liquid-to-air heat exchanger to cool the temperature of the heat-carrying airflow from the electronic components contained in the pair of racks 200. In one embodiment, more than two cooling coils can be arranged in a common area.

[0064] Fig. 9 4 is a top view layout diagram of a cooling row 401 and associated racks 200 according to one embodiment, which are configured such that the temperature T1 of the air flow from the cooling row 401 is used to independently control the cooling capacity of the cooling liquid supplied to the cooling row 401. The rack 200 can accommodate various layouts, arrangements and configurations of computing PoDs, storage PoDs or heterogeneous computing and storage PoDs to serve different IT applications. PoDs can have different power loads, power densities, etc., and thus require different cooling delivery subsystems, such as liquid-air cooling, liquid-liquid cooling, or a combination thereof, for heat exchange.

[0065] For example, the first PoD 901 may include two conventional racks 200, such as racks for homogeneous computing. The second PoD 902 may include two high-density racks 260, which have a higher power density for running machine learning or high-performance computing (HPC) workloads, and thus have more stringent cooling requirements than conventional racks 200. The first PoD 901 and the second PoD 902 may use liquid-air cooling coils (not shown). In addition to the liquid-air cooling coils, the third PoD 903 may also use a liquid-liquid cooling device, such as an immersion cooling tank 905. The fluid flowing in the cooling coils may be the same for the entire room. If certain IT rooms or racks require different types of working fluids, such as immersion cooling fluids, additional units may be added to the cooling coils, racks 200, or immersion tanks 905.

[0066] In one embodiment, the cooling capacity of the liquid supplied to the cooling delivery subsystem of the PoD may be independently adjusted and controlled in a feedback loop based on the temperature T1 of the outlet airflow of the cooling row 401 of the rack connected to the PoD after the airflow has been cooled by the cooling coils of the cooling row 401. For example, the RMU 202 of the rack or the facility may monitor the temperature T1 of the cooling row 401 to adjust the temperature or volume of the cooling liquid supplied from the cooling supply line 131 to the cooling row 401. In one embodiment, the volume of the cooling liquid supplied to the cooling row 401 may be adjusted by a valve or pump (not shown) of the cooling row 401. In one embodiment, the cooling capacity of each PoD or the cooling capacity of each cooling row of the PoD may be independently controlled by the monitored temperature T1 from the PoD or the cooling row of the PoD. Thus, the temperature T1 may be considered to be the cold aisle temperature control of the PoD, enabling the PoD to independently and dynamically adjust the cooling capacity according to the PoD, regardless of the fixed infrastructure of the facility. Control of cooling capacity may be performed at different levels, such as at the cooling coil level, the cooling row 401 level, or at the PoD level, such as PoD 405 , 901 , 902 , or 903 .

[0067] In one embodiment, temperature T2 may be the air temperature within the rack. The airflow in the rack may be controlled and adjusted in a feedback loop based on the temperature T2 of the rack. For example, Figure 2 The RMU 202 in the rack can monitor the temperature T2 and volume of the airflow passing through the rack. Based on this data, the optimization module 221 of the RMU 202 can perform optimization using a predetermined optimization model to derive a set of optimal fan speeds for the fan modules of the rack to control the airflow. Similarly, the PoD can independently and dynamically adjust the airflow of the PoD, regardless of the fixed infrastructure of the facility.

[0068] In one embodiment, if the cooling delivery subsystem uses liquid-liquid cooling, the temperature of the warm liquid returned by the liquid-liquid cooling device of the PoD after heat exchange can be measured. The rack or facility of the RMU 202 can monitor the liquid temperature of the cooling row 401 to adjust the temperature or volume of the cooling liquid supplied from the cooling supply line 131 to the cooling row 401. In one embodiment, different cooling supply and return lines 131-132 can supply cooling liquid to different cooling rows 401 or PoDs, and return warm liquid from different cooling rows 401 or PoDs.

[0069] Fig.10 FIG. 4 is a top view of a cooling row 401 connected to the facility infrastructure to form a cool liquid supply loop 131 and a warm liquid return loop 132 of a liquid cooling system prior to rack attachment, according to one embodiment. Figure 4 , Figure 5 , Figure 6As shown, the cooling rows 401 can be flexibly arranged into various layouts to accommodate different layouts, arrangements and configurations of computing PoDs, storage PoDs or heterogeneous computing and storage PoDs. Fig.10 A data center room setup is shown before any IT equipment is deployed. Individual equipment within cooling row 401 is not shown in this figure. Advantageously, the modular configuration of cooling row 401 and the cooling distribution subsystem can flexibly support rapidly changing IT needs, including changes in power density, form factor, cooling method, power delivery mechanism, rack layout, networking configuration, PoD configuration, etc., without making the data center infrastructure obsolete.

[0070] In the above description, embodiments of the present invention have been described with reference to specific exemplary embodiments of the present invention. It will be apparent that various modifications may be made to the present invention without departing from the broader spirit and scope of the present invention as set forth in the appended claims. Therefore, this specification and the accompanying drawings should be understood in an illustrative rather than a restrictive sense.

Claims

1. A liquid cooling device for a data center, comprising: a plurality of interconnected modular cooling units, each of the modular cooling units having a fluid distribution subsystem configured to supply coolant received from a coolant source to a liquid-to-air heat exchanger and to return warmed liquid from the liquid-to-air heat exchanger to the coolant source; as well as a plurality of cooling coils of the liquid-to-air heat exchanger, each of the cooling coils being configured to receive cooling liquid from a corresponding one of the modular cooling units to remove heat from an airflow of a heat load; wherein the data center comprises a plurality of row cooling units, each of the row cooling units comprising the modular cooling units arranged in a row; Wherein, based on the temperature of the air flow of each of the row cooling units after heat exchange with the liquid-to-gas heat exchanger, the cooling capacity of the cooling liquid supplied to each of the row cooling units is controlled, and the row cooling units are arranged on the rear side of the racks in the same or opposite directions in the data center, and provide air flows of different temperatures to racks with different power loads to achieve consistent thermal management effects at various locations in the data center.

2. The liquid cooling device according to claim 1, wherein: The cooling capacity of the cooling liquid supplied to the liquid cooling device is controlled independently of the cooling capacity of the cooling liquid supplied from the cooling liquid source to the second liquid cooling device.

3. The liquid cooling device according to claim 1, wherein: The heat load includes a point of delivery (PoD) of the data center, wherein the PoD includes one of a computing PoD, a storage PoD, and a combination of the computing PoD and the storage PoD.

4. The liquid cooling device according to claim 1, wherein: The plurality of cooling coils are positioned above a source of the heat load.

5. The liquid cooling device according to claim 1, wherein: Controlling the cooling capacity of the cooling liquid based on the temperature of the gas flow after the liquid-to-gas heat exchanger includes controlling the volume or temperature of the cooling liquid supplied to the liquid cooling device.

6. The liquid cooling device according to claim 1, wherein: A plurality of cooling coils from the liquid-to-air heat exchangers of a plurality of adjacent ones of the modular cooling units are combined to remove heat from an airflow of a plurality of adjacent heat loads.

7. The liquid cooling device according to claim 1, wherein: The fluid distribution subsystem is also configured to supply cooling liquid to a source of the heat load to remove additional heat from the heat load using a liquid-to-liquid heat exchanger, and to return warm liquid from the liquid-to-liquid heat exchanger to the cooling liquid source.

8. The liquid cooling device according to claim 7, wherein: The cooling capacity of the cooling liquid supplied to the liquid cooling device is further controlled based on the temperature of the warm liquid returned from the liquid-liquid heat exchanger.

9. A data center system, comprising: one or more rows of a plurality of electronic racks, each of the electronic racks housing a plurality of electronic components; and One or more rows of row cooling units, each row of row cooling units being associated with a corresponding row of electronic racks, wherein each row of cooling units comprises: a plurality of interconnected modular cooling units, each of the modular cooling units having a fluid distribution subsystem configured to supply coolant received from a coolant source to a liquid-to-air heat exchanger and to return warm liquid from the liquid-to-air heat exchanger to the coolant source; and The liquid-to-gas heat exchanger has multiple cooling coils, each of which is configured to receive cooling liquid from a corresponding one of the modular cooling units to remove heat from the air flow of one of the electronic racks, wherein, after the air flow of the corresponding row of the electronic racks transfers heat to the liquid-to-gas heat exchanger, the cooling capacity of the cooling liquid supplied to the row cooling unit is controlled based on the temperature of the air flow from the row cooling unit, and wherein the layout of the system is configured so that the air flow from the row cooling unit flows into the adjacent row of the electronic racks and its associated row cooling units for the entire area or part of the data center in the same direction as the inlet air flow, and the row cooling units are arranged at the rear side of the racks in the same or opposite directions in the data center, so as to achieve consistent thermal management effects at various locations in the data center by providing air flows of different temperatures to racks with different power loads.

10. The system of claim 9, wherein: The cooling capacity of the cooling liquid supplied to the row cooling unit is controlled independently of the cooling capacity of the cooling liquid supplied from the cooling liquid source to the second row cooling units.

11. The system of claim 9, wherein: The row cooling units and corresponding rows of the electronics racks comprise a point of delivery (PoD) of the data center, wherein the PoD comprises one of a compute PoD, a storage PoD, and a combination of the compute PoD and the storage PoD.

12. The system of claim 9, wherein: The plurality of cooling coils are positioned above corresponding rows of electronic racks.

13. The system of claim 9, wherein: Controlling the cooling capacity of the cooling liquid based on the temperature of the air flow from the row cooling unit after the liquid-to-air heat exchanger includes controlling the volume or temperature of the cooling liquid supplied to the row cooling unit.

14. The system of claim 9, wherein: A plurality of cooling coils from the liquid-to-air heat exchangers of a plurality of adjacent ones of the modular cooling units are combined to remove heat from an airflow of a plurality of adjacent ones of the electronics racks.

15. The system of claim 9, wherein: Each of the electronics racks includes a rack manifold having a rack liquid supply line to a liquid-to-liquid heat exchanger and a rack liquid return line from the liquid-to-liquid heat exchanger, and wherein the fluid distribution subsystem is further configured to supply the cooling liquid to the rack liquid supply line of one of the electronics racks to remove additional heat from the electronics rack using the liquid-to-liquid heat exchanger, and to return warm liquid from the liquid-to-liquid heat exchanger to the cooling liquid source through the rack liquid return line.

16. The system of claim 15, wherein: The cooling capacity of the cooling liquid supplied to the row cooling unit is further controlled based on the temperature of the warm liquid returning from the liquid-to-liquid heat exchanger.

17. The system of claim 9, wherein: One of the electronics racks is configured to control airflow in the electronics rack based on a temperature of the airflow in the electronics rack.

18. The system of claim 9, further comprising a plurality of rows of the electronics racks and a corresponding plurality of rows of row cooling units arranged in the same direction in the data center.

19. The system of claim 9, further comprising a plurality of rows of the electronic racks and a corresponding plurality of rows of the row cooling units, wherein each pair of the plurality of rows of the row cooling units are arranged to face each other.

20. The system of claim 9, further comprising a plurality of rows of the electronic racks and a corresponding plurality of rows of the row cooling units, wherein: Each of the row cooling units includes an upper row cooling unit and a lower row cooling unit, and wherein one or more of the upper row cooling units in the plurality of rows of row cooling units are arranged in one direction, and one or more of the lower row cooling units in the plurality of rows of row cooling units are arranged in an opposite direction.

Images