Hardware immersion cooling system

Abstract

The invention relates to a hardware immersion cooling system (1) comprising a tank (2). The tank (2) is configured to receive a plurality of server modules (10) and contain a cooling fluid. The hardware immersion cooling system (1) comprises a flow distribution system (4) configured to distribute the cooling fluid through the tank (2). The flow distribution system (4) comprises a plurality of flow bars (5) extending along the bottom of the tank (2). The flow distribution system (4) comprises a tank manifold (20) which is configured to distribute cooling fluid over a set of flow bars (5), wherein the tank manifold (20) comprises a plenum (21).

Description

[0001] Hardware immersion cooling system

[0002] TECHNICAL FIELD

[0003] The invention relates to a hardware immersion cooling system.

[0004] BACKGROUND

[0005] Data centres comprise hardware racks in which a plurality of server modules can be placed. Server modules are also known as server cassettes or chassis. A server module may comprise all sorts of heat generating electrical components, such as server memory boards, hard disks, processors, switches, power units etc. Server modules typically are plate shaped, i.e. have a substantially rectangular, flat shape. The hardware racks are configured to receive a plurality of server modules in parallel next to each other.

[0006] When in operation, the electrical components consume electrical energy and produce heat. To prevent overheating and ensure optimal operation, the server modules need to be cooled during operation. Cooling may be achieved by air-cooling.

[0007] Alternatively, cooling may be achieved by immersion cooling. Immersion cooling is a method to cool the server modules by submerging the server modules in a cooling fluid, e.g. a liquid, which absorbs and transfers heat away from the electrical components. The server modules have an open structure, configured to accommodate the cooling fluid to flow along the electrical components. The cooling fluid is a di-electric (electrically non- conductive) liquid, making it safe for submerging electrical components.

[0008] Fig. 1 shows an example of a hardware immersion cooling system 1 , which will be explained in more detail below. The hardware immersion cooling system 1 is configured to be loaded with server modules. Fig. 2 schematically shows a tank 2 of a hardware immersion cooling system 1.

[0009] As will be explained in more detail below with reference to Fig. 2, a flow distribution system 4 may be provided to ensure optimal distribution of the cooling fluid throughout the tank 2 and over the plurality of server modules. The flow distribution system may comprise a plurality of flow bars 5 running along the bottom of the tank 2. Each flow bar 5 may have a plurality of flow distribution outlets 6 to discharge cooling fluid into the tank 2. WO2014 / 109869 describes an appliance immersion tank system comprising: a generally rectangular tank adapted to immerse in a dielectric fluid a plurality of appliances, each in a respective appliance slot distributed vertically along, and extending transverse to, the long axis of the tank; a primary circulation facility adapted to circulate the dielectric fluid through the tank; a secondary fluid circulation facility adapted to extract heat from the dielectric fluid circulating in the primary circulation facility, and to dissipate to the environment the heat so extracted; and a control facility adapted to coordinate the operation of the primary and secondary fluid circulation facilities as a function of the temperature of the dielectric fluid in the tank. WO2014 / 109869 also describes an alternate embodiment comprising a ladder-arrangement of tubular spray bars, each bar of which supplies dielectric fluid to a respective appliance slot.

[0010] Alternative systems are known from W02023 / 084052A1 , US2017 / 311484A1 , US2023 / 302469A1 , US2017 / 265328A1 and US2019 / 281727.

[0011] Known flow distribution systems are relatively complicated in design and do not provide for an optimum distribution of the cooling fluid over the server modules.

[0012] SUMMARY

[0013] The object is to provide a straightforward flow distribution system for a hardware immersion cooling system, which provides for an improved distribution of the cooling fluid over the server modules.

[0014] The object is solved by a hardware immersion cooling system, the hardware immersion cooling system comprising a tank, wherein the tank is configured to receive a plurality of server modules and contain a cooling fluid, wherein the hardware immersion cooling system comprises a flow distribution system configured to distribute the cooling fluid through the tank, the flow distribution system comprising a plurality of flow bars extending along the bottom of the tank, and wherein the flow distribution system comprises a tank manifold which is configured to distribute cooling fluid over a set of flow bars, wherein the tank manifold comprises a plenum.

[0015] The tank may comprise a plurality of receiving slots for receiving respective server modules. The receiving slots may have a vertical orientation and be parallel to each other. The tank is configured to contain and circulate a cooling fluid for cooling the server modules when loaded in the receiving slots. Each receiving slot may have one associated flow bar, where the flow bar is positioned directly below the receiving slot. The tank manifold is configured to receive cooling fluid and distribute the cooling fluid over the set of flow bars. The tank manifold is fluidly connected to the set of flow bars. The tank manifold is configured to distribute the cooling fluid in parallel to the different flow bars comprised by the set.

[0016] A set comprises a number of flow bars. The set of flow bars may comprise the (total) plurality of flow bars comprised by the flow distribution system. However, the set of flow bars may also comprise less than the plurality of flow bars comprised by the flow distribution system. For instance, the plurality of flow bars in the hardware immersion cooling system may be 48, while the set of flow bars associated with the tank manifold may be 24 or 12. One or more further tank manifolds may be provided to provide cooling fluid to the remaining 24 or 36 flow bars, e.g. there may be four tank manifolds, each serving 12 flow bars.

[0017] The tank manifold may be arranged to distribute the cooling fluid evenly over the set of flow bars.

[0018] The term plenum is used to refer to a chamber or space within the tank manifold that is provided to collect and evenly distribute cooling fluid over the flow bars. The relative size of the plenum creates a steady pressure throughout the plenum thereby ensuring even distribution of cooling fluid. The plenum decouples the inflow through the plenum inlet(s) from the outflow through the plenum outlets, thereby evenly distributing cooling fluid over the flow bars. This prevents for instance flow bars being closer to a plenum inlet from receiving relatively more cooling fluid.

[0019] By combining a plenum and a manifold it is possible to create a compact flow distribution system, having a low stack-height to support increasing server module heights and maximize density. A high efficiency and flow optimization is achieved to reduce energy losses and increase cooling power.

[0020] An advantage of the tank manifold comprising a plenum is reduction of the flow rate variation over the flow bars. For instance, a flow rate variation of less than 10% may be achieved over the flow bars. This provides for a more even distribution of cooling fluid over the flow bars, providing for more stable and predictable cooling duties for the respective server modules. Furthermore, an advantage of the proposed design over for instance a shower head design, is that the use of flow-bars reduce residency time of cooling fluid which reduces the pre-heating of the cooling fluid before entering the tank. Furthermore, the use of flow bars over a shower head design also removes recirculations and closed eddies liable to form in the shower head, where debris from the tank, pump and open fluid surface could be deposited and remain and accumulate, which presents operational risk (affecting flow balance but also could dissolve, corrode, contaminate, dislodge and block openings and / or other openings e.g., pump impeller, heatsink channels etc.).

[0021] According to an embodiment the flow distribution system comprises a plurality of tank manifolds, wherein each tank manifold is configured to distribute cooling fluid over a different set of flow bars.

[0022] Each set of flow bars comprises a number of flow bars, e.g. 12 of 24 flow bars. The sets of flow bars are not overlapping, i.e. each flow bar is only part of one set. According to an example, the flow distribution system comprises forty-eight flow bars and four tank manifolds and each tank manifold is configured to distribute cooling fluid over a set of twelve flow bars.

[0023] Providing two or more tank manifolds contributes to a more even flow distribution over the flow bars and throughout the tank.

[0024] According to an embodiment the plenum extends horizontally along the set of flow bars.

[0025] The flow bars extend horizontally in a first direction parallel to each other along the bottom of the tank. The plenum extends horizontally in a second direction, perpendicular to the first direction. The plenum extends along flow bar inlets of the flow bars of the set.

[0026] The plenum may extend along a side wall of the tank.

[0027] Such a plenum can provide substantially the same flow rate to all flow bars associated with the plenum.

[0028] According to an embodiment the plenum is located to the side of the set of flow bars.

[0029] The plenum is located to the side of the flow bars. In particular, the plenum is not located underneath the flow bars, or at least part of the plenum is not located underneath the flow bars. This has the advantage that the plenum doesn’t increase the tank height or only increases the tank height in a limited way, which is important for servicing access. A low stack-height allows for larger server modules, e.g. for height restricted server centres.

[0030] The plenum may be positioned inside the tank.

[0031] According to an embodiment the plenum comprises a plenum inlet and a plurality of plenum outlets, wherein the plenum inlet is located above the level of the plenum outlets.

[0032] The reference to level is to be understood in the vertical direction. The plenum outlets may be at a level below the flow bars. The plenum inlet may be located at or close to the top of the plenum and the plenum outlets may be located at or close to the bottom of the plenum remote from the plenum inlet.

[0033] By having the plenum outlets remote from the plenum inlet, undesired direct flow paths from the plenum inlet to specific plenum outlets is prevented, which could result in uneven flow distribution.

[0034] The plenum inlet may be a horizontal plenum inlet. Alternatively, the plenum inlet may be a vertical plenum inlet, whereby the plenum optionally comprises a flow breaking element (e.g. a baffle) positioned in front of the vertical plenum inlet to quell the vertical inflow of cooling fluid. The baffle may be a substantially horizontal wall, e.g. a straight, curved or shaped wall, to divert the inflow of cooling fluid. The wall may for instance be a cone shape with the cone point being directed to the vertical plenum inlet. This prevents a direct leak stream from the plenum inlet to one or more plenum outlets located closest to the plenum inlet.

[0035] According to an embodiment wherein the plenum is a space configured to act as a buffer zone for cooling fluid. According to a further embodiment, the plenum is fluidly connected to plenum outlet flow paths, wherein a cross-sectional flow area of all plenum outlet flow paths (i.e. the sum of cross-sectional flow areas of all plenum outlet flow paths associated with the plenum) is less than 50%, preferably less than 40% and most preferably less than 25%, of a cross-sectional flow area of the plenum.

[0036] The plenum is a space completely filled or configured to be completely filled with cooling fluid when the hardware immersion cooling system is in use. Motion is possible in a plenum as long as things move simultaneously into each other’s places. The internal design of the plenum is such that the cooling fluid can flow freely through the plenum, i.e. without obstructions (with the exception of the flow breaking element) and that it functions as a buffer zone for equalization of pressure.

[0037] The plenum is fluidly connected to conduits via the plenum inlet and plenum outlets, which conduits constrain the cooling fluid and cause a pressure drop. The conduit fluidly connected to the plenum inlet is referred to as the plenum inlet flow path. The conduits fluidly connected to the plenum outlets are referred to as the plenum outlet flow paths. The plenum outlet flow paths may comprise a bridge and the flow bar, as will be described in more detail below.

[0038] The ratio of the pressure drop of the plenum outlet flow path(s) to the plenum is controlled by the total aperture area of the plenum outlet flow paths relative to the typical cross- sectional flow area of the plenum. According to an embodiment, this total conduit - plenum flow area ratio is less then 0.5 (50%), preferably less than 0.4 and more preferably less than 0.25. With such a ratio a less than 10% standard deviation in flow distribution over the flow distribution outlets within a set may be achieved. So, according to an embodiment, the cross-sectional flow area of all plenum outlet flow paths is less than 50% of the cross-sectional flow area of the plenum as experienced by the cooling fluid. The cross-sectional flow area of the plenum is taken in a direction perpendicular to the nominal flow direction through the plenum.

[0039] A plenum is an open camber with inlet and outlet conduits. Where a plenum is used to control flow distribution there is typically a single plenum inlet and multiple plenum outlets. To achieve flow balance across the plenum outlets, the flow area on the plenum is required to be significantly higher than the sum of the plenum outlet areas, and vice- versa. Hence, if a plenum serves are larger number of plenum outlets the plenum flow area is proportionately large too, resulting in voluminous plenum which make a significant space-claim and constrains design and increases solution density. The relationship between cross-section flow area of the plenum and manifold, and the corresponding volume density is a structural distinction between a manifold-based and plenum-based flow distribution system.

[0040] The use of a row manifold and row tank manifold to divide the flow upstream of the plenum enables less voluminous solutions, where the tank manifold pre-determines the destination of the flow to a particular server module (chassis) location, and again at the chassis manifold level. This advantage in density enables a higher value proposition in terms of floor-area, over-head height clearance, and lower deviation in flowrate which is particularly performant at high flowrates (cooling duty).

[0041] The proposed design is advantageous over for instance a shower head design which requires small holes to create significant pressure losses (referred to as minor losses), whereas the present design in which conduits (flow bars) are provided as plenum outlet flow paths create an additional constraint and so high-pressure loss without area constraint. This management of pressure loss and hence flow balance, in the form of a plenum and manifold is therefore advantageous, as it enables high flow, and larger flow areas to enable higher flow rates to be handled to deliver higher capacity cooling and precision targeting of cooling duty.

[0042] Furthermore, the proposed design is advantageous over for instance a design wherein a plenum or manifold directly distributes flow across a plurality of receiving slots (rows) and simultaneously across a plurality of positions within a receiving slot (columns), as for instance in a shower head design. Such two-directional flow management is not able to be optimized as there may be interference between the flow paths to every row-and- column position. High demand outlets will interfere (occlude, override, entrain) other outlet locally and remotely, with fluid structures (like jets, recirculation, stagnation, separation) free to form in the plenum or manifold. Such fluid structures are liable to be unstable in time and disrupted by changing tank server modules population (removing and replacing a server module), as well as overall flow-rate changes and overall temperature (fluid properties). These effects are amplified by high flowrates and large distance between constraints (large cavities) as defined by Reynolds’ number (see Osbourne Reynolds’ work on turbulence). Having a single path to flow distribution outlets and segregating the flow reduces and largely eliminates this risk and enables high flow rate (cooling capacity) operation.

[0043] According to an embodiment the flow distribution system comprises two or more tank manifolds, and the hardware immersion cooling system comprises a row-tank manifold which is fluidly connected in parallel to the two or more tank manifolds to distribute cooling fluid over the two or more tank manifolds.

[0044] From a flow perspective, the two or more tank manifolds are provided in parallel. The row-tank manifold may be fluidly connected to a row manifold to receive cooling fluid. The row manifold is configured to receive cooling fluid from a coolant distribution unit (CDU). The row manifold is configured to distribute cooling fluid over a plurality of tanks of hardware immersion cooling systems. The tanks may be provided in a row (or other configuration) via a single or dual row manifold which is balanced to provide equal flow to each tank.

[0045] The row-tank manifold may comprise a single sided row-tank manifold inlet or a dual sided row-tank manifold inlet being fluidly connected to the row manifold. A dual sided row-tank inlet allows for redundant cooling supply and / or improved balancing of the flow towards the row-tank manifold.

[0046] The row-tank manifold may comprise a plurality of row-tank manifold outlets which are fluidly connected to respective tank manifolds. The row-tank manifold contributes to an even flow distribution over the two or more tank manifolds.

[0047] According to an embodiment the flow distribution system comprises bridges, wherein each bridge fluidly connects one flow bar of the set to the associated plenum.

[0048] Each flow bar of the set has an associated bridge. The combination of a bridge and associated flow bar may be referred to as the plenum outlet flow path.

[0049] According to an embodiment the bridges are at least partially at a level below the flow bars.

[0050] The bridges are at least partially underneath the flow bars. The bridge may be completely below the level of the flow bars. The bridges define a flow path for the cooling fluid between the plenum and the flow bar which is at least partially below the level of the flow bars. Preferably, the bridges are also at least partially at a level below the plenum. In other words, the bridges may define a curved (e.g. U-shaped) flow path. The bridges may comprise a downstream part closest to the flow bar which defines an upward flow path towards the flow bar. The bridges may comprise a horizontal part defining a substantially horizontal flow path. Optionally, the bridges may comprise an upstream part closest to the plenum which defines a downward flow path from the plenum.

[0051] This shape forces the cooling fluid to follow a curved flow pat, for instance first flow downwards and then upwards when flowing, between the plenum and the flow bar. This contributes to decouple the flow in the flow bar from the plenum, thereby contributing to an even distribution of cooling fluid over the different flow bars and minimizes flow rate variation caused by different flow resistances between different flow bars and associated server modules.

[0052] According to an embodiment each bridge comprises a constriction or throttling zone.

[0053] The throttling zone may be formed as a constriction (e.g. constriction slot), i.e. an area with a reduced cross sectional flow area. The throttling zones create a managed pressure-drop and pressure contribute to decouple the flow bar and the server module from the plenum. The throttling zone creates a Poiseuille flow to pressure-decouple the flow bar and the server module from the plenum. This contributes to an even flow distribution over the different flow bars and associated server modules and minimizes flow rate variations caused by different flow resistances between different flow bars and associated server modules.

[0054] According to an embodiment the flow bars extend horizontally along the bottom of the tank with an upstream end near the tank manifold and with a downstream end away from the tank manifold, wherein the flow distribution system comprises a reflow bar fluidly connecting the downstream ends of two or more flow bars.

[0055] Without the reflow bar the downstream ends of the flow bars may be closed. With the reflow bar, the downstream ends of all or some flow bars are fluidly connected, contributing to even less flow rate variations over the different flow bars. The reflow bar may contribute to reducing flow rate variation to less than 5% over the flow bars.

[0056] The reflow bar may fluidly connect any number of downstream ends of flow bars. According to an embodiment, the reflow bar fluidly connects the downstream ends of all flow bars in a set. According to an alternative embodiment, the reflow bar fluidly connects the downstream ends of all flow bars comprised by the hardware immersion cooling system, i.e. of all sets.

[0057] According to an embodiment the flow bars comprise a plurality of flow distribution outlets, optionally having different outlet areas.

[0058] Each flow bar may comprise a plurality of flow distribution outlets, for instance five. The flow distribution outlets may have different outlet areas to distribute the flow rate over the different flow distribution outlets of a flow bar as required. For instance, the different outlet areas may be chosen such that each flow distribution outlet (being upstream of downstream) receives the same flow rate. This may be obtained by choosing increasingly larger outlet areas towards the downstream end of the flow bar. According to another example, the different outlet areas may be chosen such that specific flow distribution outlet receive a higher flow rate than other flow distribution outlets of the same flow bar. This may be done to target more cooling fluid towards certain parts of the server module than to others, in accordance with required cooling duty of the server module.

[0059] According to an embodiment the flow distribution system comprises flow distribution outlets with an associated closing mechanism that can move between a closed position in which the flow distribution outlet is closed and an open position in which the flow distribution outlet is opened, wherein the closing mechanism is biased towards the closed position, and wherein the closing mechanism can be pushed from the closed position to the open position through the flow distribution outlet.

[0060] The closing mechanism can be pushed from the closed position to the open position through the flow distribution outlet 6 from the outside of the flow bar, in particular by loading a server module in a receiving slot.

[0061] The closing mechanism may be provided by a lid and one or more coil springs that bias the lid to the closed position.

[0062] According to an embodiment the flow bars may comprise upwardly pointing nozzles. The flow distribution outlets are positioned at the upward end of the nozzles. Such engineered nozzles provide for a targeted flow that has momentum, instead of a more diffusive flow. This leverages forced convention potential resulting in more cooling fluid being targeted into the server modules, instead of by-passing the server modules. This contributes to increased cooling, enabling cooling with relatively warm cooling fluid and / or higher CPU / GPU powers. This embodiment also allows for creating a mechanical coupling to the server modules, thereby further preventing by-pass flows and enabling precision cooling towards certain heat sinks, as will be explained in more detail below.

[0063] According to an embodiment the flow distribution system is configured to form a direct, preferably leak free, fluid connection with the server module. The term direct fluid connection is used to indicate that no cooling fluid may leak causing fluid by-pass. A direct fluid connection may also be referred to as a leak free fluid connection.

[0064] This may provide for a tight and good integrity seal between the flow distribution system and the server module, thereby preventing by-pass, i.e. cooling fluid which flows along the outside of the server modules rather than through the server modules.

[0065] The inlet side of the server module preferably makes a direct fluid connection to the flow distribution system, e.g. a flow bar manifold, to prevent bypass.

[0066] Preferably, the fluid connection between the inlet side and the one or more flow distribution outlets is formed by one or more parallel, direct fluid connections, preventing leakage of cooling fluid which could by-pass the server module.

[0067] The server module may have inlet openings aligned with respective flow distribution outlets in the flow distribution system, forming a tight and good integrity seal between the flow distribution system and the server module.

[0068] Alternatively, the nozzles, described above, may advantageously be used for this. The nozzles may be configured to penetrate the server modules to discharge cooling fluid to the internals of the server module using targeted convection.

[0069] The term server module as used in this text also encompasses assemblies which may comprise a server module and a server module extension block of any kind, for instance to add height to server block allowing easier servicing.

[0070] According to an embodiment the hardware immersion cooling system comprises a plurality of server modules, whereby a direct fluid connection is established between the server module and the flow distribution system.

[0071] Such a direct fluid connection prevents by-pass of cooling fluid, i.e. cooling fluid which flows outside of the server modules. The direct fluid connection preferably is a leakproof fluid connection. The direct fluid connection may be established in any suitable manner. The server modules may have receiving holes which are configured to receive the nozzles and form a leak-proof connection. It will be understood that this may also be accomplished by means of an opposite arrangement, wherein the server module or server module extension block comprises extending or protruding inlet conduits configured to enter fluid flow distribution outlets to form a direct fluid connection. The direct fluid connection may be a mechanical connection.

[0072] According to an embodiment the hardware immersion cooling system comprises at least one server module with an associated server module extension block.

[0073] The server module extension block may be connected to the server module, preferably to the base of the server module.

[0074] BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In the figures, the subject-matter of the invention is schematically shown, wherein identical or similarly acting elements are usually provided with the same reference signs.

[0076] Figure 1 schematically shows an example of a hardware immersion cooling system according to the prior art,

[0077] Figure 2 schematically shows the internals of a hardware immersion cooling system according to the prior art,

[0078] Figure 3 schematically shows a server module according to the prior art,

[0079] Fig. 4a schematically shows a flow distribution system according to an embodiment,

[0080] Fig.’s 4b-4c schematically show side views of a flow distribution systems according to different embodiments,

[0081] Figure 5a-b schematically show flow schemes according to different embodiments,

[0082] Figure 6a schematically shows a server module in a hardware immersion cooling system according to the prior art,

[0083] Fig.’s 6b-c schematically show a server module in a hardware immersion cooling system according to different embodiments.

[0084] DESCRIPTION OF EMBODIMENTS

[0085] Fig. 1 shows an example of an immersion hardware rack according to the prior art, also referred to as a hardware immersion cooling system 1. A hardware immersion cooling system 1 typically comprises a liquid tight tank 2 which can be filled with a dielectric cooling fluid. The tank 2 may comprise a plurality of receiving slots 3, each configured to receive a server module 10. An example of a server module 10 is shown in Fig. 3. Each receiving slot 3 is configured to receive a server module 10 in a vertical orientation. The tank 2 comprises an open top side, allowing the server modules 10 to be inserted and removed from the receiving slots 3 in a vertical orientation. The hardware immersion cooling system 1 may comprise one or more tank lids 9 to open and close the open top side of the tank 2. In operation, the hardware immersion cooling system 1 is filled with dielectric cooling fluid such that the server modules 10 are at least partially immersed when positioned in the respective receiving slot 3.

[0086] The hardware immersion cooling system 1 is arranged to circulate the dielectric cooling fluid through the tank 2, along and through the server modules and the electrical components comprised by the server modules. A heat exchanger (not shown) is provided outside the tank 2 to reduce the temperature of the cooling fluid received from the tank 2 and circulate the hence obtained cooled cooling fluid back into the tank 2.

[0087] The warmed cooling fluid is typically removed from the tank 2 at or near the top of the tank 2, for instance with the aid of a weir. The cooled cooling fluid is typically (re-) introduced into the tank 2 at or near the bottom of the tank 2.

[0088] The hardware immersion cooling system 1 is designed to allow server modules to be inserted and removed when needed, for instance for maintenance purposes or when server modules 10 are due to be replaced.

[0089] To distribute the cooling fluid throughout the tank 2 and over the plurality of server modules 10, the hardware immersion cooling system 1 may comprise a flow distribution system 4, an example of which is schematically shown in Fig. 2. The tank comprises side walls 8. The flow distribution system 4 shown comprises a tank distribution conduit 7 and a plurality of flow bars 5 which are fluidly connected to the tank distribution conduit 7 and each run in parallel along the bottom of the tank 2. Each receiving slot 3 may have an associated flow bar 5. Each flow bar 5 may have a plurality of flow distribution outlets 6 to discharge cooling fluid into the tank 2 and into the server.

[0090] Fig. 3 shows a server module 10 comprising one or more lifting connectors 11. The lifting connectors 11 are positioned on a top side 14 of the server module 10 and can be attached to a lifting mechanism of a service trolley for lifting and lowering the server modules out of and into the hardware immersion cooling system 1. The lifting connectors 11 are shown as a D-shape ring, but it will be understood that many alternative embodiments of lifting connectors 11 may be used. The top side 14 is defined as the side of the server module 10 facing upwards when positioned in a receiving slot 3. The electrical components are positioned on the inside of the server module 10, protected by two panels 16 (front panel and back panel, only one of which is shown in Fig. 3) and two side panels 18 (only one of which is shown in Fig. 3). The server module 10 further has a profiled bottom side or base 15, i.e. it may comprise protrusions forming legs 17 which, once inserted, interact with supports provided in the tank 2 to hold the server module 10 in place.

[0091] Embodiments are provided that provide a straightforward flow distribution system that improves the distribution of the cooling fluid over the different flow bars.

[0092] Fig. 4a shows a hardware immersion cooling system 1 comprising a flow distribution system 4 according to an embodiment. The flow distribution system 4 comprises two tank manifolds 20, each comprising a plenum 21. The plenum 21 is provided by a relatively large space configured to receive, contain and discharge cooling fluid.

[0093] Fig. 4a shows two tank manifolds 20 each serving twelve flow bars 5. It will be understood that more tank manifolds 20 and associated flow bars 5 may be present.

[0094] The plenum 21 extends horizontally next to and along the flow bars 5. The plenum 21 is located adjacent to the tank wall 8 laterally so not to increase the tank height (important for servicing access).

[0095] As more clearly shown in Fig. 4b, each plenum 21 comprises a plenum inlet 22 and a plurality of plenum outlets 23 (of which only one is visible in Fig. 4b), one plenum outlet 23 for each flow bar 5 associated with the respective plenum outlet 23. The plenum inlet 22 is located above and remote from the plenum outlets 23. The plenum inlet 22 is located close to the top of the plenum 21 and the plurality of plenum outlets 23 are located close to the bottom of the plenum 21 , preferably below the level of the flow bars 5. The plenum inlet 22 is a horizontal plenum inlet. The inflow of cooling fluid is horizontally. As the plenum outlets 23 are at a lower level, there is no risk of direct leak streams flowing from the plenum inlet 22 to one or a few closest plenum outlets 23, which would disturb the even distribution of cooling fluid over the flow bars 5.

[0096] Fig. 4c shows an alternative embodiment wherein the plenum inlet 22 is a vertical plenum inlet. The inflow of cooling fluid is vertical. Fig. 4c further shows a flow breaking element 25 to break and divert the vertical flow as received from the vertical plenum inlet 22. The flow breaking element 25 serves to break the flow and thereby prevents direct leak streams flowing from the plenum inlet 22 to one or a few closest plenum outlets 23 from occurring.

[0097] Fig. 4a and 4b further show bridges 24 which fluidly connect the plenum with the respective flow bars 5. The bridges 24 are part of the plenum outlet flow path. Each flow bar 5 associated with the plenum 21 has an associated bridge 24. The bridge 24 is at least partially at a level below the plenum 21 and the flow bar 5, thereby creating a curved, e.g. U-shaped, flow path.

[0098] Each bridge 24 comprises a throttling zone, i.e. formed as a narrowing or constriction. The throttling zones are not visible in the Figures.

[0099] Figure 5a schematically shows a flow scheme according to an embodiment, not showing all the details of Fig.’s 4a-c but showing the flow scheme from a more high-level perspective. Fig. 5a shows two adjacent tank manifolds with plenums 21 as described above with reference to Fig.’s 4a-c. It will be understood that more tank manifolds may be present. Fig. 5a further shows a row-tank manifold 30 positioned upstream of the tank manifolds which row-tank manifold 30 is fluidly connected to the plenums 21 , in particular via plenum inlets 22.

[0100] The row-tank manifold 30 is fluidly connected to an upstream row manifold 31 , which row manifold 31 distributes cooling fluid to one or more row-tank manifolds 30 (of which only one is shown in Fig. 5a). The row manifold 31 is fluidly connected to a coolant distribution unit 32 to receive coolant fluid from the coolant distribution unit 32.

[0101] Fig. 5a further shows flow bars 5 which are connected to the plenum 21 (via bridge 24 not shown in Fig. 5a) by an upstream end 51. The flow bars 5 further comprise a downstream end 52 which is a dead-end as shown in Fig. 5a. Fig. 5b shows an alternative embodiment, wherein the downstream ends 52 are all fluidly connected by a reflow bar or reflow conduit 53.

[0102] In use, the coolant distribution unit 32 pumps cooling fluid towards the row manifold 31. The row manifold 31 is depicted as a single row manifold but may also be provided as a dual row manifold which is balanced to provide equal flow to each tank. There are primary branches from the row manifold 31 to the respective row-tank manifolds 30, of which one is shown in Fig.’s 5a / b. The primary branches may be formed as conduits to fluidly connect the row manifold 31 to the respective row-tank manifolds 30. It is noted that the entry to the row-tank manifold 30 may be single sided (as depicted) or dual sided for redundant cooling supply and / or to balance the tank manifold. The second entry may be disabled but remain in place to promote flow balancing.

[0103] The row-tank manifold 30 has a series of secondary branches to further segment the flow into sets of server modules (each one being given a unit count II) where the set contains nominally 12U (or any other number if II). In Fig.’s 5a / b a set of 6U is shown. So, for a tank with 48U there are nominally 4 branches in the row-tank manifold 30 each servicing nominally 12U server module locations. The flow from the row-tank manifold 30 outlets to the respective plenums 21 via respective plenum inlets 22. The flow enters the plenum 21 at the top and remotely from the plenum outlets 23 through which the flow is distributed across the flow bars 5 and corresponding receiving slots and server modules 10 (if present). The secondary branches may be formed as conduits to fluidly connect the row-tank manifold 30 with the respective tank manifolds.

[0104] The flow has a tertiary branch from the plenum 21 vertically downwards through a discrete port (bridge 24) for each II location, and through a throttling zone or constriction slot into a flow-bar, functioning as a secondary plenum. The throttling zone or construction slot creates Poiseuille flow to pressure-decouple the flow from the tank flow.

[0105] Each flow-bar 5 is dedicated to a U-location and presents the flow from the flow bar 5 to the server module 10 at discrete locations (nominally five, six as shown in Fig.’s 5a / b) as engineered ports (flow distribution outlets 6), providing a fourth branch. The distribution may be tuned to suit the cooling requirements of the server module (optionally with server module extension block as explained below).

[0106] Each flow-bar is identical at every U-location, and the delivery pressure to each flow-bar is uniform, thereby providing equal flow potential for each U location across a multitude of U locations and across a multitude of tanks. Any flow variation within a server module 10 may be governed by the server module extension block (as described in more detail below) which is tailored to the flow (cooling) requirements of the server module internal equipment, both in terms of location and magnitude.

[0107] Fig. 6a schematically depicts a cross sectional view of a server module 10 positioned in a hardware immersion cooling system 1. The server module 10 is shown in a cross- sectional view, schematically showing electrical, heat generating components 12 comprised by the server module 10. Between the base 15 of the server module 10 and the flow distribution outlets 6 is a gap through which cooling fluid discharged by the flow distribution system 4 through the flow distribution outlet 6 can by-pass the server module 10. By-pass flows are shown with arrows in Fig. 6a, but it will be understood that by-pass flows may also occur in a direction perpendicular to the plane of the drawing. By-pass flows contribute relatively little to cooling.

[0108] According to an embodiment, the server module 10 may be configured to form a direct, preferably leak-free, fluid connection between the flow distribution system 4 and the inner space of the server module 10.

[0109] The base 15 of the server module 10 may comprise extending or protruding inlet conduits 4T as depicted in Fig. 6d. When such a server module 10 is positioned in a receiving slot 3 of the hardware immersion cooling system 1 the inlet conduits 4T engage with a flow distribution outlet 6 to form a fluid connection between the flow distribution system 4 and the inner space of the server module 10.

[0110] The inlet conduits 41 may be part of the server module 10 or may be directly attached to the base 15 of the server module 10. Alternatively, as shown in Fig. 6b, a server module extension block 20 is provided which is connected to the base 15 of the server module 10, whereby the inlet conduits 41 are part of the server module extension block 20. The server module 10 and the server module extension block 20 may be configured to be mechanically connected.

[0111] Some of the flow distribution outlets 6, which do not receive an inlet conduit 41 , 4T may be closed off by the presence of the server module 10 or server module extension block 20. This allows for targeting cooling fluid to those parts of the server module 10 that require most cooling and prevents by-pass of cooling fluid.

[0112] Alternative configurations are possible as well. For instance, the flow distribution system 4 may comprise upwardly pointing nozzles, extending upwardly from the flow bar 5, which are configured to engage with respective inlet openings of the server module 10 or server module extension block 20 as applicable, when positioned in a receiving slot 3 to form a fluid connection. Fig. 6c show a further embodiment, in which the flow distribution system 4 comprises flow distribution outlets 6 with an associated closing mechanism 47, 48. The closing mechanism is configured to be in a closed position in which the flow distribution outlet 6 is closed or in an open position in which the flow distribution outlet 6 is opened. The closing mechanism is biased towards the closed position and is configured to be pushed open from the outside through the flow distribution outlet 6. The closing mechanism comprises a lid 47 and spring 48.

[0113] The lids 47 can be pushed open from the outside (seen from the flow distribution system 4) by the extending / protruding inlet conduits 41 of the server module 10 or server module extension block 20. The base 15 of the server module 10 may be configured to open one or more lids 47 when being positioned in the receiving slot 3 of the hardware immersion cooling system 1. In the embodiment depicted in Fig. 6c this is done by the inlet conduits 41 which extend from the base 15 of the server module 10 or server module extension block 20 as applicable. The other flow distribution outlets 6 remain closed thanks to the closing mechanism and thereby effectively prevent by-pass of cooling fluid, even if a gap would remain present between the flow bar 5 and the server module 10 or server module extension block 20.

[0114] According to the embodiment depicted in Fig. 6c, the lids 47 are biased to the closed position by coil springs 48. The inlet conduits 41 are configured to enter the flow distribution outlet 6 and push lid 47 to the opened position when an assembly is positioned. When the assembly is removed, the coil springs 48 move the lid 47 back to the closed position. The inlet conduits 41 comprise conduit side openings 43 provided in the wall forming the inlet conduits 41 .

[0115] This embodiment has the advantage that the design of the server module 10 or server module extension block 20 can be used to determine which flow distribution outlets 6 are used and which are not used to be able to better manage the flow and distribution of cooling fluid towards those parts of the server module 10 requiring most cooling duty.

[0116] Such a server module extension block 20 comprises an inner space configured to contain and transmit cooling fluid, an inlet side comprising one or more inlet openings configured to receive cooling fluid into the inner space, a discharge side comprising one or more discharge openings configured to discharge cooling fluid from the inner space. The discharge side is connected to or is configured to be connected to a server module 10. The use of a server module extension block 20 brings several advantages, such as adjusting the level of the top side of the server module 10 when placed in the hardware immersion cooling system, to facilitate servicing. The server module extension block 20 may be used to equalize the levels of the top sides 14 of the server modules and thereby overcome issues caused by different heights of server modules 10. Also, it allows to reduce by-pass of cooling fluid, for instance by facilitating a direct fluid connection between the server module and the flow bar via the server module extension block. Furthermore, a server module extension block 20 allows for the addition flow redistribution means inside the server module extension block to redistribute the cooling fluid over the different parts of the server module in accordance with required cooling duty by those different parts (columns).

[0117] Fig. 6c shows an example of such flow redistribution means, provided by internal conduits 42 inside the server module extension block 20 forming a direct, one on one, fluid connection between a single flow distribution outlet 6 / inlet opening of the server module extension block and a discharge opening of the server module extension block 20. This embodiment allows for an improved management of the flow and flow distribution of cooling fluid towards the server module 10. It will be understood that the presence of internal conduits 42 is optionally and the presence of the inlet conduits 41 is possible without the presence of internal conduits 42. Other flow redistribution means may be provided by the server module extension block, including flow deflectors, (micro- ) pumps, specific designed inlet and outlet openings (position, number, size) of the server module extension block.

[0118] Finally, Fig. 6d shows an embodiment wherein the server module 10 is provided with downwardly protruding inlet conduits 4T which are configured to enter the flow distribution outlets 6 and form a fluid connection between the flow distribution system 4 and the inner space of the server module 10. As shown in Fig. 6d the flow distribution outlets 6 are not provided with a closing mechanism, but it will be understood that the embodiment of Fig. 6d may also be provided with a closing mechanism as described above.

[0119] The invention is not limited to the embodiments shown in the drawings and described hereinbefore, which may be varied in different manners within the scope of the claims and their technical equivalents. REFERENCE LIST

[0120] 1 . Hardware immersion cooling system

[0121] 2. Tank

[0122] 3. Receiving slots

[0123] 4. Flow distribution system

[0124] 5. Flow bar

[0125] 6. Flow distribution outlet

[0126] 8. Wall

[0127] 9. Tank lid

[0128] 10. Server module

[0129] 11 . Lifting connector12. Components

[0130] 14. Top side of the server module

[0131] 15. Bottom side or base of the server module

[0132] 16. (Front / back) Panel

[0133] 17. Leg

[0134] 18. Side panels

[0135] 20. Tank manifold

[0136] 21. Plenum

[0137] 22. Plenum inlet

[0138] 23. Plenum outlet

[0139] 24. Bridge

[0140] 25. Flow breaking element

[0141] 30. Row tank manifold

[0142] 31. Row manifold

[0143] 32. Coolant Distribution Unit

[0144] 41. Inlet conduits

[0145] 42. Internal conduits

[0146] 43. Conduit side openings

[0147] 47. Lid

[0148] 48. Coil spring

[0149] 51. Upstream end of flow bar

[0150] 52. Downstream end of flow bar

[0151] 53. Reflow bar

Claims

CLAIMS1. Hardware immersion cooling system (1) comprising a tank (2), wherein the tank (2) is configured to receive a plurality of server modules (10) and contain a cooling fluid, wherein the hardware immersion cooling system (1) comprises a flow distribution system (4) configured to distribute the cooling fluid through the tank (2), the flow distribution system (4) comprising a plurality of flow bars (5) extending along the bottom of the tank (2), and wherein the flow distribution system (4) comprises a tank manifold (20) which is configured to distribute cooling fluid over a set of flow bars (5), wherein the tank manifold (20) comprises a plenum (21).

2. Hardware immersion cooling system (1) according to claim 1 , wherein the flow distribution system (4) comprises a plurality of tank manifolds (20), wherein each tank manifold (20) is configured to distribute cooling fluid over a different set of flow bars (5).

3. Hardware immersion cooling system (1) according to any one of the preceding claims, wherein the plenum (21) extends horizontally along the set of flow bars (5).

4. Hardware immersion cooling system (1) according to any one of the preceding claims, wherein the plenum (21) is located to the side of the set of flow bars (5).

5. Hardware immersion cooling system (1) according to any one of the preceding claims, wherein the plenum (21) comprises a plenum inlet (22) and a plurality of plenum outlets (23), wherein the plenum inlet (22) is located above the level of the plenum outlets (23).

6. Hardware immersion cooling system (1) according to any one of the preceding claims, wherein the plenum (21) is a space configured to act as a buffer zone for cooling fluid.

7. Hardware immersion cooling system (1) according to any one of the preceding claims, wherein the flow distribution system (4) comprises two or more tank manifolds (20), and wherein the hardware immersion cooling system (1) comprises a row-tank manifold (30) which is fluidly connected in parallel to the two or more tank manifolds (20) to distribute cooling fluid over the two or more tank manifolds (20).

8. Hardware immersion cooling system (1) according to any one of the preceding claims, wherein the flow distribution system (4) comprises bridges (24), wherein each bridge (24) fluidly connects one flow bar (5) of the set to the associated plenum (21).

9. Hardware immersion cooling system (1) according to claim 8, wherein the bridges (24) are at least partially at a level below the flow bars (5).

10. Hardware immersion cooling system (1) according to any one of the claims 8 - 9, wherein each bridge (24) comprises a constriction or throttling zone.

11. Hardware immersion cooling system (1) according to any one of the preceding claims, wherein the flow bars (5) extend horizontally along the bottom of the tank (2) with an upstream end (51) near the tank manifold (20) and with a downstream end (52) away from the tank manifold (20), wherein the flow distribution system (4) comprises a reflow bar (53) fluidly connecting the downstream ends (52) of two or more flow bars (5).

12. Hardware immersion cooling system (1) according to any one of the preceding claims, wherein the flow bars (5) comprise a plurality of flow distribution outlets (6).

13. Hardware immersion cooling system (1) according to claim 12, wherein the flow distribution system (4) comprises flow distribution outlets (6) with an associated closing mechanism (47, 48) that can move between a closed position in which the flow distribution outlet (6) is closed and an open position in which the flow distribution outlet (6) is opened, wherein the closing mechanism is biased towards the closed position, and wherein the closing mechanism can be pushed from the closed position to the open position through the flow distribution outlet (6).

14. Hardware immersion cooling system (1) according to any one of the preceding claims, wherein the flow distribution system (4) is configured to form a direct, preferably leak free, fluid connection with the server module (10).

15. Hardware immersion cooling system (1) according to any one of the preceding claims, wherein the hardware immersion cooling system (1) comprises a plurality of server modules (10), whereby a direct fluid connection is established between the server module (10) and the flow distribution system (4).

16. Hardware immersion cooling system (1) according any one of the preceding claims, wherein the hardware immersion cooling system (1) comprises at least one server module (10) with an associated server module extension block (20).

17. Hardware immersion cooling system (1) according to any of the preceding claims, wherein the plenum is fluidly connected to plenum outlet flow paths, wherein a cross- sectional flow area of all plenum outlet flow paths is less than 50% of a cross-sectional flow area of the plenum.

Images