Deck Construction for Cooling Redundancy in Data Centers

Abstract

The system includes a plurality of modular hot aisle cooling units (MHACUs) (114a-114c) arranged in series in a data hall (110), each MHACU configured to cool a plurality of servers (112) in the data hall, the servers being arranged in a plurality of storage modules (802a-802d) in the data hall, each storage module including a hot aisle (803). The system also includes a plurality of decks (804a-804b), each deck connected to at least two of the hot aisles of the plurality of storage modules and configured to allow heated air to flow between the hot aisles. The system also includes a pump package (120) configured to supply cooling fluid to the plurality of MHACUs. The system also includes at least one computing device (140) configured to control at least one of air throughput, discharge air temperature, and discharge fluid temperature in each of the plurality of MHACUs to customize cooling levels for different storage modules of the plurality of storage modules.

Description

[Technical field]

[0001] TECHNICAL FIELD Embodiments of the present disclosure relate to cooling systems, and more particularly, to vestibule structures for cooling redundancy in data centers. [Background technology]

[0002] Colocation data centers typically require flexibility in space utilization to accommodate diverse customer demands. For example, some colocation data centers require facilities to serve both ASHRAE (American Society of Heating, Refrigerating, and Air-Conditioning Engineers) permissible and ASHRAE recommended customers. Many data center providers prefer closed systems where no direct outside air or liquid is supplied to the data center for cooling. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure provides a vestibule structure for cooling redundancy in a data center.

[0004] In a first aspect, a system includes a plurality of modular hot aisle cooling units (MHACUs) arranged in series in a data hall, each MHACU configured to cool a plurality of servers in the data hall, the servers being arranged in a plurality of storage modules in the data hall, each storage module including a hot aisle (803). The system also includes a plurality of decks, each connected to the hot aisles of at least two of the plurality of storage modules and configured to allow heated air to flow between the hot aisles. The system also includes a pump package configured to supply cooling fluid to the plurality of MHACUs. The system also includes at least one computing device configured to control at least one of air throughput, discharge air temperature, and discharge fluid temperature in each of the plurality of MHACUs to customize cooling levels for different storage modules of the plurality of storage modules.

[0005] In a second embodiment, the method provides cooling fluid from a pump package to a first modular hot aisle cooling unit (MHACU) of a plurality of MHACUs arranged in series in a data hall via a fluid supply line, each MHACU configured to cool a plurality of servers in the data hall, the servers being disposed in a plurality of storage modules in the data hall, each storage module including a hot aisle, at least some of the hot aisles being connected via a plurality of decks such that heated air flows between at least some of the hot aisles. The method also determines that a temperature of the cooling fluid in the first MHACU has risen to a first temperature that is less than a predetermined maximum temperature. The method also provides at least a portion of the cooling fluid to a second MHACU of the plurality of MHACUs in response to determining that the temperature of the cooling fluid in the first MHACU has risen to the first temperature. The method also determines that the temperature of the cooling fluid in the second MHACU has risen to at least a second temperature that is the predetermined maximum temperature. The method also provides the cooling fluid to a fluid return line (304) for return to the pump package in response to determining that the temperature of the cooling fluid in the second MHACU has increased to the second temperature. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 illustrates an exemplary cooling system for cooling a data center according to the present disclosure. [Diagram 2] FIG. 2 shows further details of an example modular hot aisle cooling unit (MHACU) according to the present disclosure. [Figure 3A] FIG. 3A illustrates details for improving the efficiency of heat rejection through increased heat content of the fluid in accordance with the present disclosure. [Figure 3B] FIG. 3B illustrates a floor plan of a data hall in which the efficiency improvement techniques of FIG. 3A are used, in accordance with the present disclosure. [Figure 4A] FIG. 4A illustrates an example data hall having different levels of cooling density in accordance with the present disclosure. [Figure 4B] FIG. 4B illustrates an example data hall having different levels of cooling density in accordance with the present disclosure. [Figure 4C] FIG. 4C illustrates an example data hall having different levels of cooling density in accordance with the present disclosure. [Figure 5A] FIG. 5A illustrates an example installation of cooling coils that may be used as one or more modular hot aisle cooling units (MHACUs) in accordance with the present disclosure. [Figure 5B] FIG. 5B illustrates an example installation of cooling coils that may be used as one or more modular hot aisle cooling units (MHACUs) in accordance with the present disclosure. [Figure 5C] FIG. 5C illustrates an example installation of cooling coils that may be used as one or more modular hot aisle cooling units (MHACUs) in accordance with the present disclosure. [Figure 5D] FIG. 5D illustrates an example installation of cooling coils that may be used as one or more modular hot aisle cooling units (MHACUs) in accordance with the present disclosure. [Figure 5E] FIG. 5E illustrates an example installation of cooling coils that may be used as one or more modular hot aisle cooling units (MHACUs) in accordance with the present disclosure. [Figure 6] FIG. 6 is an exemplary diagram of a computing device for use in a cooling system in accordance with the present invention. [Figure 7] FIG. 7 is a flow chart illustrating an example of a cooling process using the cooling system of FIG. 1 according to the present disclosure. [Figure 8A] 8A-8D illustrate different views of an example data hall including one or more decks in accordance with the present disclosure. [Figure 8B] FIG. 8B illustrates a different view of an example data hall including one or more decks in accordance with the present disclosure. [Figure 8C] FIG. 8C illustrates a different view of an example data hall including one or more decks in accordance with the present disclosure. [Figure 9A]9A-9C illustrate different views of another example data hall including one or more decks of different configurations in accordance with the present disclosure. [Figure 9B] FIG. 9B illustrates different views of another example data hall including one or more decks of different configurations in accordance with the present disclosure. [Figure 9C] FIG. 9C illustrates different views of another example data hall including one or more decks of different configurations in accordance with the present disclosure. [Figure 10] FIG. 10 illustrates another exemplary data hall including multiple decks in a grid configuration in accordance with the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Other technical features may be readily apparent to those skilled in the art from the following drawings, descriptions, and claims.

[0008] 1-10 described below and the various embodiments used to explain the principles of the present invention are merely for illustrative purposes and should not be construed as limiting the scope of the present invention in any way. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged system or device.

[0009] For simplicity and clarity, some features and components are not explicitly shown in all figures, including those shown in relation to other figures. It will be understood that all features shown in the figures may be used in any of the described embodiments. The omission of a feature or component from a particular figure is for purposes of brevity and clarity and is not meant to imply that the feature or component cannot be used in the embodiment described in relation to that figure. It will be understood that an embodiment of the present disclosure may include any one, more than one, or all of the features described herein. Also, an embodiment of the present disclosure may include other features not enumerated herein, in addition or in the alternative.

[0010] As mentioned above, colocation data centers typically require flexibility in space utilization to accommodate diverse customer demands. For example, some colocation data centers must be equipped to provide space for both ASHRAE permissible and ASHRAE recommended customers. Many data center providers prefer closed systems where no direct outside air or liquid is supplied to the data center for cooling.

[0011] To address these and other issues, embodiments of the present disclosure provide an indoor cooling system for use with colocation data centers. The disclosed indoor cooling system is designed to operate over a wide range of fluid temperatures. The disclosed embodiments include cooling coils and immersion devices configured in various shapes and heights to match the power and heat density of the incoming liquid temperature (EFT) for liquid cooling, and the specified supply air temperature (SAT) for air cooling for part or all of a computing device rack, computing device row, computing device room, or computing device system. The system efficiency can be derived through heat collection near the heat load by eliminating the long air flow paths required in traditional colocation data center installations from the computing devices to the air handling devices or direct contact with the cooling fluid during immersion. System efficiency can also be expressed in the amount of heat collected by air-to-fluid and / or fluid-to-fluid transfer through custom configurations, locations, sizes, shapes, and ridges of the cooling coils. Additional efficiencies can be found in the higher discharge fluid temperature (LFT) of the coil available through the fluid-to-fluid heat transfer performed within the immersion cooling system, and further efficiencies due to the higher quality heating fluid available to remote heat recovery users or remote heat rejection plants external to the data hall.

[0012] FIGURE 1 illustrates an exemplary cooling system 100 for cooling a data center according to the present disclosure. The embodiment of the cooling system 100 illustrated in FIGURE 1 is for illustrative purposes only. Other embodiments of the cooling system 100 may be used without departing from the scope of the present disclosure.

[0013] As shown in FIG. 1, the cooling system 100 includes a data hall 110, a pump package 120, a fluid cooler 130, and a computing device 140.

[0014] The data hall 110 represents at least a portion of a colocation data center and is an enclosed space housing a number of servers 112 arranged in server racks. As is known in the art, the servers 112 generate a significant amount of thermal energy that tends to heat up the space within the data hall 110, thereby requiring cooling to maintain the temperature of the data hall 110 at an appropriate level for the proper operation of the servers 112 and for the comfort of any personnel within the data hall 110.

[0015] The data hall 110 includes an indoor cooling system that includes one or more modular hot aisle cooling units (MHACUs) 114. The MHACUs 114 are positioned above, behind, and / or in front of the servers 112 and are operable to cool the servers 112. In particular, each MHACU 114 can be mounted above, behind, and / or in front of a server rack in the data hall 110. The MHACUs 114 can be configured in different shapes and sizes, installed at different heights, and in different arrangements and combinations to partially or fully match the power and heat density of a given supply air temperature for a rack of computing devices, a row of computing devices, a room, or a computing device facility.

[0016] The MHACU 114 receives heated air rising from the servers 112 (e.g., about 130°F to 140°F if ASHRAE acceptable, or about 100°F if ASHRAE recommended), cools the heated air to cool air (e.g., about 95°F if ASHRAE acceptable, or about 80°F if ASHRAE recommended), and outputs the cool air to cool the servers 112. In some embodiments, the amount of airflow delivered by the MHACU 114 to the data hall 110 depends on the amount of power delivered to the data hall 110. For example, the MHACU 114 may deliver at least 80 cubic feet per minute (CFM) of air at a temperature of 80°F (or at least 108 CFM of air at a temperature of 95°F) for every kilowatt (1 kW) of power delivered to the data hall 110.

[0017] Each MHACU 114 is modular and located above, behind, and / or in front of one or more racks of servers 112. While FIG. 1 shows three MHACUs 114, there may be more or fewer, depending on the embodiment. The number of MHACUs 114 is easily scaled for an application and depends on the load density of the servers 112, the cooling capacity of each MHACU 114, etc. In some embodiments, each MHACU 114 can provide between about 150 kW and 700 kW of cooling capacity, while other embodiments can provide other cooling capacities.

[0018] The MHACU 114 improves system efficiency over conventional techniques by handling heat collection closer to the heat load; that is, the MHACU 114 eliminates the long air flow paths required in conventional colocation data center facilities from the computing devices to the air handling equipment. Overall system efficiency can also be addressed in the amount of heat collected by the air-fluid transfer through the custom configuration, location, size, shape, and ridge of the cooling coils. Further efficiency can be seen in the high discharge fluid temperature (LFT) of the coils to a remote heat removal plant outside the data hall 110.

[0019] The MHACUs 114 are designed to operate over a wide range of fluid temperatures. Each MHACU 114 can be individually controlled (including air throughput, discharge air temperature, discharge fluid temperature, etc.) to customize cooling levels in real time in different parts of the data hall 110. For example, if some of the servers 112 generate a greater load and require additional cooling, one or more MHACUs 114 in the vicinity of those servers 112 can be controlled to increase cooling capacity.

[0020] In some embodiments, the MHACUs 114 are connected in series and fluidly coupled to the pump package 120. This may be referred to as a "serial topology," while in other embodiments, the MHACUs 114 may be connected in parallel. The connections to the MHACUs 114 may be formed individually, or in parallel or series, in any combination with each other, or as specific groups, to produce the intended results, e.g., to collect the maximum amount of heat through air-fluid movement. The cooling fluid received from the pump package 120 (e.g., about 90°F for ASHRAE acceptable values, or about 75°F for ASHRAE recommended values) flows into each MHACU 114 and is used to cool the heated air from the servers 112. Once the fluid has cooled the hot air in the data hall 110, the heated fluid returns to the pump package 120. In some embodiments, at least a portion of the heated fluid may be sent to one or more immersion tanks 145, as described in more detail below. In some embodiments, the fluid is water, but other suitable fluids may be used and are within the scope of the present disclosure.

[0021] In some embodiments, the system 100 also includes a heat recovery heat exchanger 150 for use in downstream heat recovery to support the needs of one or more heat recovery users. In some embodiments, the immersion tank 145 can also generate higher quality heat suitable for downstream heat recovery. This higher quality heat is available to the heat recovery heat exchanger 150 to support the needs of the heat recovery users.

[0022] FIG. 2 illustrates further details of one example of the MHACU 114 according to the present disclosure. As shown in FIG. 2, the MHACU 114 includes one or more variable speed fans 202, one or more fluid valves 204, and at least one coil 206 for transferring thermal energy from the heated air to the cooled fluid. The MHACU 114 also includes at least one control system 208 for controlling the operation and speed of the fans 202 and the position of the valves 204. The at least one control system 208 is communicatively coupled to one or more sensors including one or more pressure sensors 210, thermometers or other temperature sensors 212, equipment sensors 214, fluid flow sensors (not shown), and the like. In some embodiments, the temperature sensors 212 can measure, for example, air temperature in the supply and return passages, air temperature in the inlet and return air streams, fluid temperature in the supply and return lines, fluid temperature in and out of the coil 206, and the like. The fluid flow sensors can include direct fluid contact sensors, pipe surface contact sensors, infrared sensors, and the like. The type and number of sensors can be customized to direct a particular fluid flow, air flow, fluid pressure, air pressure, heat content of a given fluid, heat content of a given volume of air, relative humidity, etc. Pressure sensor 210 can measure the pressure difference between the supply and return air passages, fluid pressure at the input and output to coil 206, etc. Other sensors can include one or more anemometers to measure air velocity within the air stream, or one or more ultrasonic fluid flow sensors.

[0023] The valves 204 may include any suitable valves in any suitable combination for controlling the flow of fluid in and around the MHACU 114. Examples of the valves 204 may include (but are not limited to) a two-way control valve, a three-way control valve, a four-way control valve, a six-way control valve, a balancing valve, an actuator control valve, a thermal control valve, a flow control valve, a pressure control valve, and a compensation valve.

[0024] In some embodiments, each blower 202, individually or in combination with one or more attributes supporting the computing device, may be dynamically controlled or set to a specific fixed value to maintain the appropriate air supply volume, air temperature, or static pressure difference between the hot return air aisle and the cold supply air aisle. In general, the data transmitted from the sensors to the control system 208 may be used individually or in any combination to improve the power efficiency, cooling efficiency of the data center, or to reduce total water consumption through real-time response to individual rack, row, room, or field cooling load demands. For example, computing device power may be matched with the cooling supply provided based on the actual heat load calculated from the computing device power demand. Cooling only the heat generation of the computing device may improve cooling efficiency. Total water consumption may be reduced by not over-pumping through cooling towers or adiabatic cooling spray cooling solutions and maintaining water loss from drift and surface evaporation. Effective control of Computing Device Inlet Air Temperature (EAT) and Coil Discharge Fluid Temperature (LFT) control is configured through sensor inputs and programmed calculations to meet the exact cooling requirements of the immediate rack, row, room or site.

[0025] In some embodiments, the equipment sensors 214 are remote sensors used at or around the computing equipment (e.g., servers 112) in the data hall 110 to detect or measure characteristics or parameters of the computing equipment. For example, the equipment sensors 214 can include on-board power sensors embedded in the computing device, server, or network equipment to measure the power used by the computing equipment. As another embodiment, the equipment sensors 214 can include on-board thermal or fan speed sensors embedded in the computing device, server, or network equipment to measure the heat generated by the computing equipment or the current fan speed of the equipment. As yet another example, the equipment sensors 214 can include on-board sensors to measure the CPU or hash rate utilization of the server 112. These measurements can be provided to the control system 208 to control the cooling. In some embodiments, room-level thermal sensors can be used to override local coil controls to meet global (overall data center space) thermal requirements. In some embodiments, room-level static pressure sensors can be used to override local coil controls to meet global positive pressure requirements in the supply air aisle.

[0026] In some embodiments, measurements collected by equipment sensors 214 can be used as a heat load proxy. As described below, through real-time monitoring and collection of power output and known locations, heat load values ​​can be calculated for individual areas such as equipment, racks, rows, rooms, buildings, or sites.

[0027] Below are examples of power measurements that can be used as a proxy for heat loading. ●Device level: A power strip with POC (point of connection) sensing output. ● Rack Level: Power sensors on local power strips supporting devices within a single rack. • Rack Level: Individual power metered or monitored busway electrical taps or circuit breakers that directly support a particular rack. • Row Level: Busway input power sensing meters or individual metering of electrical branch circuits supporting row level power distribution. Room Level: Power sensor input from data center distribution panel, circuit breaker, metering sensor, or panel circuit board metering. ●Building Level: Power sensor inputs from data center power distribution boards and / or distribution circuit breakers supporting the critical loads of data center equipment. ● Field level: Power sensor monitoring of substation output breakers at the field level to the individual buildings where computing devices are supported.

[0028] In some embodiments, control can be facilitated using Data Center Infrastructure Management (DCIM) techniques. As known in the art, DCIM can be used to describe processes, procedures, control inputs, and control outputs for microcontroller and macro-management of power and cooling of computing devices or data center infrastructure. DCIM techniques can take into account individual or collective inputs from computing devices, computing equipment rack-level aggregation of power and / or cooling demand, computing device power or cooling demand aggregated at row level, aggregation of device power or cooling demand at room level, building-level aggregation of device power or cooling demand, field-level demand for device power and cooling, etc.

[0029] Once thermal energy is transferred from the air to the cooled fluid, thereby heating the fluid, the heated fluid (e.g., approximately 120° F. if ASHRAE acceptable, approximately 90.3° F. if ASHRAE recommended) is output from each MHACU 114 back to the pump package 120 and delivered to the fluid cooler 130 for rejecting the heat stored in the fluid. The discharge fluid temperature (LFT) from the coil 206 can be controlled by the position of the valve 204, and / or the air volume created by the speed of the fan 202 and the exit air temperature from the coil 206. In some embodiments, the control system 208 (which may be part of or include the computing device 140) simultaneously controls the temperature of the cooling air (exiting the MHACU 114 and entering the cold aisle) and the temperature of the heated fluid (exiting the MHACU 114) by varying both the fan air volume and the cooling fluid flow rate.

[0030] FIG. 3A illustrates details for improving the efficiency of heat rejection through increasing the heat content of the fluid, according to the present disclosure. Conventional industry practices are inefficient in increasing and / or returning relatively high fluid temperatures to the heat rejection system due to inconsistent heat rejection of computing workloads in data centers, thermal dilution of the heat content of the cooled supply fluid to the heat rejection system (e.g., from a combination of fluid streams of different temperatures), and / or low supply fluid temperatures resulting from mixed supply air temperatures specified by end users or computing equipment manufacturers. Generally, low supply fluid starting temperatures result in relatively low return fluid temperatures. For example, some conventional systems exhibit a heated return fluid at a temperature of about 60°F to 75°F. Significant heat transfer and power efficiency improvements can be realized when the heated fluid exiting the heat rejection system can be returned at the highest fluid temperature the system design can accommodate (e.g., about 120°F for some air-cooled systems). That is, the greater the temperature difference (delta T) between the cooled supply fluid and the heated return fluid, the more efficient the heat rejection plant and equipment. The details shown in FIG. 3A provide at least one solution to these problems.

[0031] As shown in FIG. 3A, multiple MHACUs 114 (here identified as 114a-114c) are fluidly coupled to one another within the data hall 110. FIG. 3A shows three MHACUs 114a-114c, although there may be more or less, depending on the embodiment. Fluid supplied to the MHACUs 114a-114c is received from the pump package 120 via a fluid supply line 302. Heated fluid to be returned to the pump package 120 is conveyed via a fluid return line 304. Each MHACU 114a-114c is associated with a temperature sensor 311a-311c and a fluid control valve 321a-321c.

[0032] Initially, some or all of the cooled supply fluid from the pump package 120 is input to the first MHACU 114a. The fluid travels through one or more coils 206 in the MHACU 114a and absorbs thermal energy from the air in the data hall 110. This causes the temperature of the fluid to rise (measured by the temperature sensor 311a). If so much thermal energy is absorbed by the MHACU 114a that the fluid temperature rises to a predefined maximum value (e.g., 120°F), the control valve 321a is controlled to return all of the fluid to the fluid return line 304. Alternatively, if there is less thermal energy transfer and the fluid temperature rises to a temperature lower than the maximum value (e.g., 75°F), the control valve 321a is controlled to supply at least a portion of the fluid to the second MHACU 114b.

[0033] In the second MHACU 114b, the fluid travels through one or more coils 206 and absorbs thermal energy from the air in the data hall 110. This causes a temperature rise in the fluid (measured by temperature sensor 311b). When the MHACU 114b absorbs enough thermal energy to increase the fluid temperature to a maximum, the control valve 321b is controlled to return all the fluid to the fluid return line 304. Alternatively, if there is less thermal energy transfer and the fluid temperature rises to a lower temperature (e.g., 90°F), the control valve 321b is controlled to provide at least a portion of the fluid to the third MHACU 114c. This serial flow process continues until a maximum fluid temperature is reached or there are no more MHACUs in the series. In some embodiments, the fluid returns through fluid return line 304 and enters or bypasses one or more dip tanks 352 depending on the position of control valve 321d, which can route some or all of the fluid through dip tank 352 or bypass dip tank 352 on the way to pump package 120. Dip tank 352 can represent (or represent) dip tank 145 of FIG.

[0034] Continuous heating of the cooling fluid as shown in Figure 3A can be enabled using input measurements of any or all of the sensors mentioned above. The sensor inputs can be used in conjunction with predefined calculations, algorithms, and design protocols to control various components including: • Fluid flow rates in individual coils, rows, rooms, and / or regions. • Fluid heat release control point rates for individual coils, rows, rooms, and / or sites. ● Fan speeds for individual coils, rows, and / or rooms. • System or individual fluid pressures for individual coils, rows, and / or rooms. ●System or individual air pressure settings at row and / or room level. • Pump speeds at row, room, and / or field level. • Fluid mix ratios at coil, row, and / or room level.

[0035] In some embodiments, continuous heating of the cooling fluid as shown in FIG. 3A can directly support the inlet fluid temperature (EFT), for example, for data center immersion cooling using immersion tank 352, data center direct rack liquid cooling, district heating, heat recovery to one or more heat recovery users using heat recovery heat exchanger 150, etc.

[0036] FIG. 3B illustrates a plan view of a data hole 350 in which the efficiency improvement technique of FIG. 3A is used, according to the present disclosure. The data hole 350 may represent the data hole 110 of FIG. 1. As shown in FIG. 3B, across the data hole 350, there is a series heat gain in the fluid (i.e., the temperature of the fluid increases from cooler to hotter). This corresponds to possible zones within the data hole 350 that may have different cooling requirements. In some embodiments, the data hole 350 includes one or more of the immersion tanks 352. The immersion tanks 352 can accept high inlet fluid temperatures (EFT) of over 120° F. and generate outlet fluid temperatures (LFT) of over 150° F., and may be the entire or part of the components of the fluid return line 304.

[0037] In some embodiments, one or more of the MHACUs 114 do not include any air filters. Instead, the MHACUs 114 may rely on a dedicated outdoor air system (DOAS) pressurized unit to clean the air.

[0038] The use of the MHACU 114 in the data hall 110 offers many advantageous advantages over existing solutions. The MHACU is mounted above and / or behind or in front of the server racks, so little or no floor space is required. It also does not require duct work in the floor, reducing the need for floor raising. This reduces infrastructure costs. The MHACU 114 uses less energy than existing solutions because there are no duct work losses, no under-floor distribution losses, and no filter pressure losses. The MHACU 114 offers a modular design that offers rack and load density flexibility. Local control of each MHACU 114 helps ensure uniform temperature cooled air to the server rack air inlets.

[0039] The fluid cooler 130 receives heated fluid from the MHACU 114 in the data hall 110 via the pump package 120. The fluid cooler 130 cools the heated fluid using a multi-coil heat exchanger system and outputs the cooled fluid to the pump package 120 for delivery to the MHACU 114 in the data hall 110. The fluid cooler 130 may include any suitable heat removal device or feature, such as, but not limited to, an open loop evaporative cooling tower or surface water routed through a heat exchanger to isolate the data hall cooling system from external contaminants, closed circuit cooling towers, closed loop adiabatic cooling, air-cooled chillers, conventional chiller systems, and the like. Although FIG. 1 illustrates the cooling system 100 with one fluid cooler 130, this is by way of example only. In other embodiments, the cooling system 100 may include multiple fluid coolers 130, each with an isolated flow. In further embodiments, the cooling system 100 may include multiple fluid coolers 130 with combined flows for redundancy. In yet other embodiments, the cooling system 100 can include multiple fluid coolers 130 with combined flows for cooling multiple data halls 110, thus providing increased redundancy at a lower cost.

[0040] As described above, cooling system 100 includes one or more computing devices 140 for controlling the operation of cooling system 100. In some embodiments, each computing device 140 may be a service operated by a third party, such as a person or business. Each computing device 140 may be housed and operated at a location separate from where the remainder of cooling system 100 is located. That is, each computing device 140 is not bound to a particular location.

[0041] 4A-4C illustrate exemplary data halls 110 having different levels of cooling density according to the present disclosure. In particular, FIG. 4A illustrates a data hall 110 having low density cooling (e.g., about 3kW-9kW per rack), FIG. 4B illustrates a data hall 110 having medium density cooling (e.g., about 15kW-20kW per rack), and FIG. 4C illustrates a data hall 110 having high density cooling (e.g., about 30kW-50kW per rack). As shown in FIG. 4A-4C, the cooling density can be increased by increasing the number of MHACUs 114 disposed above each data hall. In FIG. 4A-4C, the MHACUs 114 are illustrated as having a shape similar to an upside-down "V". However, this is merely an example, and in other embodiments, the MHACUs 114 can have any other suitable shape. For example, one or more of the MHACUs 114 may have a right-side-up "V" shape, a "U" or "W" shape (either right-side-up or upside-down), a cone shape (either concave or convex to the grade), or a flat coil surface parallel to the grade, or a flat coil perpendicular to the grade. In embodiments having a flat coil shape, the coil is in a position perpendicular to the grade, directly behind or in front of the data center equipment racks, which is an efficient and effective way to collect significant heat through air-to-fluid transfer. Such coils may be used independently or in combination with overhead coils.

[0042] As discussed above, the MHACU 114 can be configured in different shapes and sizes and installed at different heights and in different configurations and combinations to partially or fully accommodate the power and heat density of a given supply air temperature for a rack of computing devices, a row of computing devices, a room, or a computing device installation. For example, the MHACU 114 can be installed overhead parallel to the rear of the rack in a single file configuration down the center of the hot aisle, in a dual path parallel to the rear of the rack, perpendicular to the rear of the rack, penetrating the top of the rack footprint on each side of the aisle, or any combination of these. In some embodiments, the MHACU 114 may be mounted to a surface adjacent to the rear of the rack in a rolling or movable panel configuration. Additionally, the mounting frame for mounting the MHACU114 may include any one or more of the following features: a height adjustable frame, a support frame supported by the floor, support frame hinges, support frame rollers, a support frame suspended from above to any suitable structure, a frame with a mounting toolbar, a frame with plug and play lighting, a frame with plug and play controls and sensors, coolant and power distribution frame mounts, coolant and power distribution plug and play connections, and a frame and enclosure sealed with an air bypass of, for example, 2% or less at 0.33 inches water column (wc).

[0043] 5A-5E show an exemplary installation of a cooling coil 500 that can be used as an MHACU 114 according to the present disclosure. Notably, in the embodiment shown in FIGS. 5A-5E, the coil 500 is placed behind or in front of the equipment racks, rather than overhead. When the coil 500 is behind the equipment racks, as shown in FIG. 5D, the incoming air comes directly from the equipment being cooled. When the coil 500 is in front of the equipment racks, the incoming air is unconditioned and can come from anywhere inside the room or space, or from the ambient air outside the room or building. The coil 500 can be in a bypass arrangement (e.g., see FIG. 5C), hinged, or swinging outward (e.g., see FIG. 5A), or folded (e.g., fold-over or accordion style) behind the equipment rack (e.g., see FIG. 5B). In some embodiments, the coil 500 can use the same supply fluid control and return fluid characteristics as the MHACU 114.

[0044] In some embodiments, the coil 500 is larger in width than one data center equipment rack. In some embodiments, the coil 500 does not require support of any data center equipment racking, but may contact equipment racking if specified by design or user. In some embodiments, the coil 500 may be supported on tracks and / or rollers that contact the floor or floor system. In some embodiments, the coil 500 may be suspended from overhead to a building structure. In some embodiments, the coil 500 may be supported by a custom mounting frame or brackets that are supported from overhead or grade (see, e.g., FIG. 5E). This may be useful in areas where floor space is scarce.

[0045] For access to the equipment rack, the coil 500 can move or slide parallel to the equipment rack, swing like a door when attached to a hinge, or rise into the overhead ceiling or headroom. In some embodiments, the coil 500 can be configured to zigzag or overlap to increase the surface area exposed to the air.

[0046] To accommodate the moving coil 500, the coil fluid lines may be flexible or rigid, or may include composite flex pipes with rigid pipe flex joints. In some embodiments, the cooling coil assembly may have fans and sensors connected to the coil 500, which will also have flexible connections and conductors that allow the coil 500 to move within a predetermined range to meet design requirements or user needs. In some embodiments, one or more of the fluid lines, electrical paths, and sensors may be designed to allow movement of the coil 500 to gain access to the data center equipment.

[0047] In some embodiments, the coil 500 may be designed as a passive coil with no active external blower system, with all airflow generated by the computing device. In some embodiments, the coil 500 may be designed as an active system with a fan 202 attached directly to the coil 500 or external to the computing device at any location in the vicinity of the coil 500 and communicating through a duct or other enclosure or with a bypass system designed to direct the air. The airflow of the external fan system may be constant at a controlled variable speed and variable pressure.

[0048] 1-5E illustrate an example cooling system 100 and associated details, various modifications may be made to FIGS. 1-5E. For example, various components within the cooling system 100 may be combined, further subdivided, duplicated, rearranged, or omitted, and additional components may be added according to specific needs. As a particular example, in a data center having a larger data hall 110, the cooling system 100 may include multiple fluid coolers 130, multiple MHACUs 114, and multiple pump packages 120 connected in parallel for a common fluid connection of all components within the data hall 110. As another example, in some data halls, one or more computer room air handler (CRAH) units may be implemented in addition to or in place of one or more of the MHACUs 114. Additionally, although FIGS. 1-5E illustrate an example cooling system for use with a data center, the described functionality may be used in any other suitable device or system.

[0049] 6 illustrates a computing device 600 for use in a cooling system in accordance with the present invention. The computing device 600 may be the computing device 140 described above in FIG. 1. The computing device 600 may be configured to control operations at various components within the apparatus 100. For example, the computing device 600 may control or monitor operations associated with the MHACU 114, the pump package 120, or the fluid cooler 130.

[0050] 6, computing device 600 includes a bus system 605 supporting communication between a processor 610, a storage device 615, a communication interface 620, and an input / output (I / O) unit 625. The processor 610 executes instructions that may be loaded into a memory 630. The processor 610 may include any suitable number and type of processor or other device in any suitable configuration. Exemplary types of processor 610 include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuits.

[0051] Memory 630 and persistent storage 635 are examples of storage device 615, which represents any structure capable of storing and facilitating retrieval of information (such as data, program code, and / or other suitable information, either temporary or persistent). Memory 630 may represent random access memory, or any other suitable volatile or non-volatile storage device. Persistent storage 635 may include one or more components or devices that support long-term storage of data, such as read-only memory, hard drives, flash memory, or optical disks. For example, persistent storage 635 may store one or more databases of data, standard data, results, data, client applications, and the like.

[0052] The communication interface 620 supports communication with other systems. For example, the communication interface 620 may include a network transceiver or a wireless transceiver that facilitates communication through the system 100. The communication interface 620 may support communication over any suitable physical or wireless communication line. The I / O section 625 inputs and outputs data. For example, the I / O unit 625 may provide a connection for user input through a keyboard, mouse, keypad, touch screen, or other suitable input device. The I / O unit 625 may also send output to a display, printer, or other suitable output device.

[0053] Although Figure 6 illustrates one embodiment of a computing device 600, various modifications may be made to Figure 6. For example, various components of Figure 6 may be combined, further subdivided, or omitted, and additional components may be added according to particular needs. As a particular example, although shown as one system, computing device 600 may include multiple computer systems, which may be remotely located. In another example, different computing systems may provide some or all of the processing, storage, and / or communication resources according to the present disclosure.

[0054] Figure 7 is a flow chart illustrating an example of a cooling process 700 using the cooling system 100 of Figure 1, according to various embodiments of the present disclosure. The embodiment of the cooling process 700 shown in Figure 7 is for illustrative purposes only. Other embodiments of the cooling process 700 may be used without departing from the scope of the present disclosure.

[0055] Referring to FIG. 7, in operation 701, the cooled supply fluid from the pump package 120 is supplied to the MHACUs 114a-114c via the fluid supply line 302. In operation 703, some or all of the cooled supply fluid from the pump package 120 is admitted to the MHACU 114a, the first MHACU in the series. The fluid travels through one or more coils 206 in the MHACU 114a and absorbs thermal energy from the air in the data hall 110. In operation 705, the temperature sensor 311a measures that the fluid temperature in the MHACU 114a rises to a temperature below a predetermined maximum value. In operation 707, in response to the measured temperature, the control valve 321a is controlled to supply at least a portion of the fluid to the second MHACU 114b. The fluid travels through one or more coils 206 in the MHACU 114b and absorbs thermal energy from the air in the data hall 110. In operation 709, the temperature sensor 311b measures that the fluid temperature in the MHACU 114b rises to at least a predetermined maximum temperature. In operation 711, in response to the measured temperature, the control valve 321b is controlled to provide fluid to the fluid return line 304. In operation 713, the heated fluid is returned to the pump package 120 via the fluid return line 304.

[0056] Although the process 700 described above illustrates example operations that may be performed in accordance with the principles of the present disclosure, various modifications may be made to the process 700. For example, although shown as a series of steps, various steps in the process 700 may overlap, occur in parallel, occur in a different order, or occur multiple times. In other examples, steps may be omitted or replaced by other steps.

[0057] Because of their importance, data centers are designed to operate with very high reliability and uptime. Reliability and uptime requirements are key factors in data center design and engineered systems.

[0058] As mentioned above, data center designed cooling solutions range from simple to complex topologies consisting of basic mechanical and electrical components that require routine maintenance and can have catastrophic failures without warning. Therefore, it is common in data center cooling designs to provide solutions that provide some level of redundancy. This redundancy is required to provide the complete cooling requirements of a rack, row, module, space, room, container or building during scheduled maintenance windows or unscheduled outages.

[0059] To achieve the reliability of the designed system, data center designers and engineers may provide redundant critical individual components to meet the reliability and uptime requirements of the system. In some embodiments, alternative means and methods may be provided for data center designers and engineers to provide the reliability and uptime requirements.

[0060] In the simplest design, and with the lowest reliability, a mechanical system has one of each of its critical components. The capacity of the critical components is equal to the workload demand. In Normal Plus One ("N+1"), the most common redundancy scenario, the system includes one additional piece or capacity increment of the critical equipment. Most cloud data center designs include redundant technologies and components as part of the design. Redundancy improves the reliability of the facility in terms of uptime of the computing equipment. In many systems, the uptime expectation is 99.999% of the time for the critical computing load. This is often referred to as "five-nines uptime."

[0061] The most efficient designs in data centers use air separation (also known as containment) and MHACUs to achieve the best results. This air separation requires that the supply and return air streams are isolated from each other through the use of physical barriers that prevent infiltration or mixing. This separation forces the air through engineered cooling paths provided by the MHACUs. In general, to achieve the most cost-effective and energy-efficient design and insulation, longer and fewer containment paths should be used so that fewer redundant components are required.

[0062] Within the data center industry, hyperscalers and colocation service providers have customer physical space requirements that range from a few data center racks of computing equipment to hundreds of racks of computing equipment. A typical requirement is 20-100 racks. The racks are often arranged in the hot and cold aisle configurations described above. Data centers built for hyperscalers and other Internet service clients generally have security requirements to physically isolate one client's racks from another client's racks. In most designs, this is as simple as a large room with cage-type walls that isolate the clients.

[0063] Air separation modules typically contain a hot aisle and include solid end cap walls that prevent personnel from contacting or accessing separate isolated equipment. This security requirement has led data center designers to use storage modules that contain small computing equipment rack rows (e.g., containing less than 10 racks) and small deployments, with the MHACU providing the cooling solution. Each small storage module as a standalone space has the same redundancy requirements for cooling as a much larger module with hundreds of racks. For example, a small storage module uses one extra MHACU unit than is required to meet the "N+1" minimum cooling redundancy design, but in a data center with many short storage modules, this represents an unnecessarily large number of redundant MHACU units.

[0064] To address these and other issues, various embodiments of the present disclosure may include one or more decks that link two or more small storage modules together for cooling redundancy purposes while maintaining the security of each short storage module.

[0065] 8A-8C show different views of an exemplary data hall 110 including one or more decks according to the present disclosure. In particular, FIG. 8A shows a perspective view of the data hall 110, FIG. 8B shows a top view of the data hall 110, and FIG. 8C shows a perspective view of the data hall 110 with certain components removed to facilitate clarity and understanding. The embodiment of the data hall 110 shown in FIG. 8A-8C is for illustrative purposes only. Other embodiments of the data hall 110 may be used without departing from the scope of the present disclosure.

[0066] As shown in FIGS. 8A-8C, the data hall 110 includes multiple racks 801 of data equipment (e.g., servers 112) grouped into small storage modules 802a-802d. In some embodiments, each storage module 802a-802d represents data equipment for one data center customer. The storage modules 802a-802d are "small" in that each storage module 802a-802d includes only a few racks 801 (e.g., less than 10 or 20 racks 801) compared to "large" storage modules that may include tens or hundreds of racks. Each storage module 802a-802d includes a hot aisle 803, and the racks 801 of each storage module 802 are disposed on either side of the hot aisle 803. One or more MHACUs 114 are disposed above the hot aisle 803 of each storage module 802a-802d to provide cooling for the computing equipment within each storage module 802a-802d, as described above.

[0067] In a conventional data hall, each storage module is generally separated in terms of air movement. That is, the hot aisle of each storage module is separated from the hot aisle of other storage modules by a solid wall or a solid door that restricts the air movement between the storage modules. In contrast, the data hall 110 includes multiple decks 804a-804b that enable air movement between two or more storage modules 802a-802d. That is, each deck 804a-804b provides a connected return air common plenum path for hot air flow between two or more storage modules 802a-802d. For example, in FIG. 8B, deck 804a connects the hot aisles 803 of storage modules 802a and 802b for air flow, and deck 804b connects the hot aisles 803 of storage modules 802c and 802d for air flow. Each deck 804a-804b may be formed in any size or shape to provide a common omnidirectional air path in communication with two or more individual hot aisles 803. As described in more detail below, each deck 804a-804b also provides any required or desired physical security using caged or open-airflow doors at specific locations to secure data equipment, while allowing the hot airflow within each hot aisle 803 to travel in any direction throughout the deck connection system.

[0068] The connected air paths between the multiple storage modules 802a-802d, as provided by decks 804a-804b, allow for a reduction in the number of redundant MHACUs 114 in the data hall 110 while still meeting the requirement of "N+1" redundancy. For example, deck 804a connects the hot aisles 803 of storage modules 802a and 802b for airflow, so that each of storage modules 802a and 802b does not need to have its own redundant MHACU 114. Instead, one redundant MHACU 114 can provide sufficient redundancy for both storage modules 802a and 802b. Similarly, rather than each of storage modules 802c and 802d having its own redundant MHACU 114, instead, one redundant MHACU 114 can provide adequate redundancy for both storage modules 802c and 802d.

[0069] Each deck 804a-804b can include both solid walls that are substantially or completely impermeable to airflow and mesh walls that readily permit airflow through the walls. The mesh walls of each deck 804a-804b form a physical barrier between the deck 804a-804b and the adjacent storage module 802a-802d while allowing bidirectional airflow through the mesh walls. For example, hot air can travel from the hot aisle 803 of storage module 802a, through the mesh wall between the hot aisle 803 and deck 804a, through deck 804a, through a second mesh wall between the hot aisle 803 of storage module 802b, and into the hot aisle 803 of storage module 802b. Hot air can also travel in the opposite direction from storage module 802b, through deck 804a (and through its mesh walls) and into storage module 802a. Each mesh wall may be formed of any suitable material, such as steel mesh, insulation, etc., that facilitates the movement of air through the mesh wall while restricting the movement of people.

[0070] Other walls of each deck 804a-804b, which may include ceilings, floors, and walls parallel to the rows of racks 801, may be formed as solid walls. Solid walls may be constructed from a variety of air impermeable materials, including rigid materials, soft woven-type materials, rolled sheet materials, individual strips of materials, and the like. Such materials may be transparent, translucent, opaque, or combinations thereof. The ceilings of each deck 804a-804b may be formed from the same materials as the ceiling materials of the data hall 110, including, but not limited to, plaster, drywall, dropped ceiling panels and frames, structural frames, floor or roof elements of a building or space, steel containers or modules, and the like.

[0071] The materials forming the solid and mesh walls of each deck 804a-804b may be any floor structure (at grade or elevated), any overhead structure (including ceiling grid structures, beams and girders, floor or roof structures as described above), data center sub-structures designed as part of a containment system, data center equipment racking, one or more cables (e.g., steel, wire, rope, etc.), a framing system (formed from metal, plastic, composites, etc.), one or more vertical or horizontal grids or grilles, etc.

[0072] The interior of each deck 804a-804b may be accessed from one or more portals or passageways. For example, each deck 804a-804b may be accessed through one or more solid doors 805 on the supply air passageway. Each solid door 805 is impermeable to airflow to isolate the hot aisle 803 from the supply air passageway. Each solid door 805 may be transparent, translucent, or opaque and may be sealed as needed. Each deck 804a-804b may also be accessed from the hot aisle 803 through solid or mesh cage type doors 805 disposed within mesh or solid walls. Any of the doors 805 may be alarm doors for emergency evacuation. Any of the doors 805 may have a locking device to prevent unauthorized access. Any of the doors 805 may have panic hardware if required by building or fire codes or preferences.

[0073] Although not specifically part of the decks 804a-804b, one or more security doors 806 may be located within the hot aisles 803 of one or more of the storage modules 802a-802d to isolate certain customer equipment within the storage modules 802a-802d.

[0074] 8A-8C, the decks 804a-804b are arranged to fluidly connect storage modules 802a-802d that are linearly aligned with one another, however, the decks can be configured in other arrangements to connect storage modules that are not in a linear arrangement.

[0075] 9A-9C show different views of another exemplary data hall 110 including one or more decks of different configurations according to the present disclosure. Specifically, FIG. 9A is a perspective view of the data hall 110, FIG. 9B is a top view of the data hall 110, and FIG. 9C is a heat map of the data hall 110. The embodiment of the data hall 110 shown in FIG. 9A-9C is for illustrative purposes only. Other embodiments of the data hall 110 may be used without departing from the scope of the present disclosure.

[0076] As shown in FIGS. 9A-9C, the data hall 110 includes multiple small storage modules 902a-902d, each including a hot aisle adjacent to multiple racks of data equipment. The data hall 110 also includes a deck 904 that connects the hot aisles of all four storage modules 902a-902d. Similar to decks 804a-804b, deck 904 can include mesh walls and / or doors that align with the hot aisles of each storage module 902a-902d. Other walls and / or doors of deck 904 can be solid to prevent air movement between the supply air passage of the data hall 110 and deck 904. As shown by the small arrows in FIG. 9B, hot air from one of the storage modules 902a-902d can flow through deck 904 to any other of the storage modules 902a-902d. This is also shown in the heat map of Figure 9C, where H-shaped region 906 represents a single continuous "hot aisle" that branches through all four storage modules 902a-902d. Deck 904 thus connects storage modules 902a-902d that are both linearly and parallel arranged.

[0077] 10 illustrates another exemplary data hall 110 including multiple decks in a grid configuration according to the present disclosure. The embodiment of the data hall 110 illustrated in FIG. 10 is for illustrative purposes only. Other embodiments of the data hall 110 may be used without departing from the scope of the present disclosure.

[0078] As shown in FIG. 10, the data hall 110 includes a number of small storage modules 1002, each including a hot aisle adjacent to a number of racks of data equipment. The data hall 110 also includes a number of decks 1004 disposed at points between adjacent storage modules 1002. The decks 1004 also connect the hot aisles of all the storage modules 1002 into a single continuous "hot aisle" grid. The data hall 110 also includes a number of expansion decks 1005 disposed at the ends of the rows of storage modules 1002, as shown in FIG. 10. To expand the data hall 110, additional storage modules 1002 can be installed on the "open" ends of one or more of the expansion decks 1005. The additional storage modules 1002 are then connected to the "hot aisle" grid for cooling redundancy. The data hall 110 also includes a termination point 1006 disposed at the other end of the rows of storage modules 1002. The termination point 1006 can include a solid door to allow personnel access to the storage modules 1002.

[0079] Although FIG. 10 illustrates the data hall 110 configured with decks 1004 forming a “hot aisle” grid, other embodiments may include decks connecting storage modules to form a circular or ring configuration, a hub and spoke configuration, or any other suitable configuration.

[0080] It may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The term "couple" and its derivatives refer to any direct or indirect communication between two or more elements, whether or not those elements are in physical contact with one another. The terms "transmit," "receive," and "communicate," and their derivatives, encompass both direct and indirect communication. The terms "include" and "comprise," and their derivatives, mean an open-ended inclusion. The term "or" is inclusive and / or. The term "associated" and its derivatives mean having the property of being contained within, contained in, connected to, communicatively cooperating, interleaving, juxtaposing, adjacent to, coupled, having, having a relationship, and the like. When used within a term, the phrase "such as" means that the subsequently listed term is illustrative of the previously listed term, and not limiting. The phrase "at least one of," when used in conjunction with a list of items, means that various combinations of one or more of the listed items may be used, and that only one item in the list may be required. For example, "at least one of A, B, and C" includes any of A, B, C, A and B, A and C, B and C, and A and B and C.

[0081] Furthermore, various functions described herein may be implemented or supported by one or more computer programs, each of which is formed from computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, associated data, or portions thereof adapted for implementation in suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, compact disc (CD), digital video disc (DVD), or any other type of memory. A "non-transitory" computer-readable medium excludes wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer readable media includes media on which data may be permanently stored and media on which data may be stored and later overwritten, such as a rewritable optical disk or erasable memory device.

[0082] Definitions of other specific words and phrases are provided throughout this patent document. Those skilled in the art should understand that in many, if not most, cases, such definitions apply to prior and future uses of such defined words and phrases. Although the present disclosure has been described with exemplary embodiments, various changes and modifications may be suggested to those skilled in the art. The present disclosure is intended to encompass such changes and modifications that fall within the scope of the appended claims. Nothing in this application should be read as implying that any particular element, step, or function is an essential element that must be included in the scope of the claims. The scope of patented subject matter is defined by the claims.

Claims

1. a plurality of modular hot aisle cooling units (MHACUs) (114a-114c) arranged in series in a data hall (110), each MHACU configured to cool a plurality of servers (112) in said data hall, said servers being arranged in a plurality of storage modules (802a-802d) in said data hall, each storage module including a hot aisle (803); a plurality of decks (804a-804b), each connected to the hot aisles of at least two of the plurality of storage modules and configured to allow heated air to flow between the hot aisles; a pump package (120) configured to supply cooling fluid to a plurality of said MHACUs; at least one computing device (140) configured to control at least one of air throughput, discharge air temperature, and discharge fluid temperature in each of the plurality of MHACUs to customize cooling levels for different storage modules of the plurality of storage modules; Including, System (100).

2. each of the plurality of decks includes a plurality of mesh walls; each mesh wall disposed between the deck and one or more of the at least two storage modules; each mesh wall configured to allow the heated air to flow through the mesh wall while restricting personnel movement therethrough; The system of claim 1 .

3. The system of claim 2 , wherein each of the plurality of decks further comprises at least one solid wall and at least one door.

4. a first temperature sensor (311a) configured to measure a temperature of the cooling fluid in a first MHACU of the plurality of MHACUs; a second temperature sensor (311b) configured to measure a temperature of the cooling fluid in a second MHACU of the plurality of MHACUs; The system of claim 1 further comprising:

5. at least one coil (206) disposed in the first MHACU, the cooling fluid being conveyed through the at least one coil, the at least one coil being configured to transfer thermal energy from the heated air to the cooling fluid while the heated air passes through the at least one coil; The system of claim 4.

6. The system of claim 5 , wherein the heated air is heated by a plurality of the servers and flows from the plurality of the servers to the first MHACU.

7. The system of claim 1 , wherein each of the plurality of MHACUs is located above, behind, or in front of a plurality of the servers.

8. one or more equipment sensors (214) disposed proximate to or within at least one of the plurality of servers and communicatively connected to at least one of the computing devices; one or more of the equipment sensors are configured to measure one or more characteristics of a plurality of the servers; the one or more equipment sensors include at least one of a power sensor, a thermal sensor, a fan speed sensor, and a CPU sensor; The system of claim 1 .

9. At least one of the computing devices comprises: determining that a temperature of the cooling fluid in a first MHACU (114a) of the plurality of MHACUs has increased to a first temperature, the first temperature being less than a predetermined maximum temperature; controlling the system to supply at least a portion of the cooling fluid to a second MHACU (114b) of the plurality of MHACUs in response to determining that the temperature of the cooling fluid in the first MHACU has increased to the first temperature; determining that a temperature of the cooling fluid in the second MHACU has increased to at least a second temperature, the second temperature being the predetermined maximum temperature; in response to determining that the temperature of the cooling fluid in the second MHACU has increased to the second temperature, controlling the system to provide the cooling fluid to a fluid return line (304) for return to the pump package. The system of claim 1 further configured to:

10. At least one of the computing devices comprises: calculating a thermal load based on the power demands of the plurality of servers; using the calculated heat loads to determine the customized cooling levels in different parts of the data hall; The system of claim 9 further configured to:

11. The system of claim 9 , wherein the return fluid line includes at least one dip tank (145) fluidly coupled between a plurality of the MHACUs and the pump package.

12. receiving heated fluid from a plurality of said MHACUs via said pump package; cooling the heated fluid to form a cooled fluid; configured to deliver the cooling fluid to the pump package; The system of claim 1 , further comprising a fluid cooler (130).

13. providing (730) cooling fluid from a pump package (120) to a first MHACU (114a) of a plurality of modular hot aisle cooling units (MHACUs) (114a-114c) arranged in series in a data hall (110) via a fluid supply line (302); each MHACU configured to cool a plurality of servers (112) in said data hall; The servers are located in a plurality of storage modules (802a-802d) in the data hall; Each storage module includes a hot aisle (803); at least some of the hot aisles are connected via a plurality of decks (804a-804b) such that heated air flows between at least some of the hot aisles; determining (705) that a temperature of the cooling fluid in the first MHACU has risen to a first temperature that is less than a predetermined maximum temperature; providing at least a portion of the cooling fluid to a second MHACU of the plurality of MHACUs in response to determining that the temperature of the cooling fluid in the first MHACU has increased to the first temperature (707); determining (709) that the temperature of the cooling fluid in the second MHACU has increased to at least a second temperature, the second temperature being the predetermined maximum temperature; in response to determining that the temperature of the cooling fluid in the second MHACU has increased to the second temperature, providing (711) the cooling fluid to a fluid return line (304) for return to the pump package; The method includes:

14. each of the plurality of decks includes a plurality of mesh walls; each mesh wall disposed between the deck and one or more of at least some of the hot aisles; each mesh wall is configured to permit the heated air to flow therethrough while restricting movement of personnel therethrough. The method of claim 13.

15. The method of claim 14 , wherein each of the plurality of decks further comprises at least one solid wall and at least one door.

Images