Immersion cooling system and components thereof

Abstract

Systems for immersion cooling are disclosed. In an aspect, an immersion cooling system includes an immersion cooling tank, a busbar, and one or more semiconductor dies. In an additional aspect, an immersion cooling system includes an immersion cooling tank containing an immersion cooling fluid cavity, one or more sensors, and one or more castors for moving the immersion cooling tank. In an additional aspect, an immersion cooling system includes an immersion cooling tank and a heat exchanger disposed within the immersion cooling tank.

Description

IMMERSION COOLING SYSTEM AND COMPONENTS THEREOFCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Provisional Application No. 63 / 581,439, filed September 8, 2023, which is hereby incorporated by reference in its entirety.BACKGROUND

[0002] As feature sizes and transistor sizes have decreased for integrated circuits (ICs) including chips and semiconductor dies, the amount of heat generated by a single chip, such as a microprocessor, has increased. Chips that once were air cooled have evolved to chips needing more heat dissipation than can be provided by air alone. In some cases, immersion cooling of chips in a tank containing a cooling fluid is employed to maintain IC chips at appropriate operating temperatures.

[0003] One type of immersion cooling is two-phase immersion cooling, in which heat from a semiconductor die boils the cooling fluid. The boiling creates a cooling-fluid vapor in the tank, which is condensed by cooling coils back to liquid form. Heat from the semiconductor dies can then be sunk into the liquid-to-gas and gas-to-liquid phase transitions of the cooling fluid. This allows chips to be run at a faster rate and allows for more computationally intensive processes than were previously possible. Production of immersion cooling tanks and integrated hardware (including computing hardware such as servers) can be very expensive and testing these systems prior to use can tie up precious resources.SUMMARY

[0004] Systems are disclosed herein for immersion cooling tanks that can be used to validate hardware for immersion cooling systems with a minimum of resources. The inventors have recognized the need for a realistic testing environment for components such as servers and similar computing hardware without dedicating a full (and highly valuable) immersion cooling system like a full-scale immersion cooling tank.

[0005] Two-phase immersion cooling systems disclosed herein may include one or more of a sensor, a sensor bar, a heat exchanger, a semiconductor die, a server, a busbar, a pass-through, a castor or similar wheel mechanism, a lid, a pump, a filter and / or filtration media, a controller, and a vent pipe.

[0006] In some aspects, the techniques described herein relate to a system for immersion cooling, the system including an immersion cooling tank, a busbar disposed within the immersion cooling tank and configured to be biased to a voltage, and one or more castors attached to a bottom of the immersion cooling tank for moving the immersion cooling tank.

[0007] In some aspects, the techniques described herein relate to a system for immersion cooling, the system including an immersion cooling tank at least partially filled with immersion cooling fluid, one or more semiconductor dies disposed within the immersion cooling tank and immersed within the immersion cooling fluid, and one or more pass-throughs configured to communicatively couple the one or more semiconductor dies with one or more elements external to the immersion cooling tank.

[0008] All combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are part of the inventive subject matter disclosed herein. The terminology used herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The skilled artisan will understand that the drawings primarily are for illustrative purposes and are not intended to limit the scope of the inventive subject matter described herein. The drawings are not necessarily to scale; in some instances, various aspects of the inventive subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference characters generally refer to like features (e.g., functionally similar and / or structurally similar elements).

[0010] FIG. 1A illustrates an immersion cooling system including an immersion cooling tank in accordance with the invention.

[0011] FIG. IB illustrates an exemplary sensor bar in greater detail.

[0012] FIG. 2A illustrates an alternate perspective view of the immersion cooling system in accordance with the present invention.

[0013] FIG. 2B illustrates an additional perspective of the immersion cooling system.

[0014] FIG. 2C illustrates a plurality of different types of pass-throughs that may be used as part of a reconfigurable pass-through plate.

[0015] FIG. 3 illustrates a side view perspective of the immersion cooling system.

[0016] FIG. 4 illustrates an electrical layout inside an immersion cooling tank in accordance with the invention.

[0017] FIG. 5 depicts an exploded view of a server rack configured to support one or more servers during operation.

[0018] FIG. 6 further illustrates an exploded view of the server rack assembly.

[0019] FIG. 7 depicts aspects of an immersion-cooling system for dissipating heat from one or more semiconductor die packages via immersion cooling.DETAILED DESCRIPTION

[0020] FIG. 1A illustrates an immersion cooling system including an immersion cooling tank in accordance with the invention. The immersion cooling system may be a two-phase immersion cooling system and include a container such as an immersion cooling tank having an immersion cooling fluid cavity. The immersion cooling system may contain elements and / or sub-systems for multiple functions including filtration, power management, sensors, controller, and data management. The immersion cooling tank may have an exemplary footprint of about 1200 mm by about 1000 mm by about 1400 mm (length by width by height). The immersion cooling fluid cavity within the immersion cooling fluid tank may have dimensions of about 550 mm by about 950 mm by about 800 mm.

[0021] The top of the immersion cooling tank may include a lid for sealing the immersion cooling fluid cavity from exposure to an external environment during operation. The lid may include or be in contact with a gasket, which may provide a seal between the contents of the immersion cooling tank and the external environment. A lid may have dimensions of about 850mm by 500 mm. In an embodiment, a lid may be made of aluminum.

[0022] The top of the immersion cooling tank may further include a means to electrically couple one or more elements internal to the immersion cooling tank with one or more elements external to the immersion cooling tank. Such means may be a power pass-through, which may provide a sealed conduit through which power cables may pass through the immersion cooling tank wall. Elements internal to the immersion cooling tank may include one or more busbars, one or more sensor bars, one or more servers, one or more semiconductor dies, one or more computing devices, one or more integrated circuits, or the like. Elements external to the immersion cooling tank may include one or more power sources, one or more batteries, one or more power grids(such as a municipal power grid, a city power grid, a town power grid, a data center power grid, a local power grid, etc.). Power cables may be connected to the computing hardware contained within the immersion cooling tank through the power pass-through while preventing liquid or vapor from escaping from the immersion cooling tank. The power pass-through may also provide power access within the immersion cooling tank while further maintaining a pressure seal such that a positive or negative gauge pressure may be maintained within the immersion cooling tank.

[0023] The top of the immersion cooling tank may further include an inlet and outlet tube for a heat exchanger contained within the immersion cooling tank. The inlet and / or outlet tube may include threaded pipe. The heat exchanger may allow for bi-directional flow of cooling water or other fluid; accordingly, each tube may function as inlet and / or outlet (and switch between functions) as needed. The heat exchanger may include one or more tubes, pipes, conduits, etc., such as one or more copper tubes, which may facilitate a flow of water or other cooling fluid through the heat exchanger to remove excess heat from an immersion cooling vapor or froth and recondense the vapor or froth back into liquid form.

[0024] The heat exchanger may be rated to a capacity of between about 30-40 kW, between about 25-50 kW, between about 10-100 kW, or a suitable heat removal rate capacity. In an embodiment, a heat exchanger may have a heat removal rate capacity of about 36 kW. In an embodiment, the heat exchanger may include two parallel heat exchangers operation simultaneously, each with approximately * of the total heat removal rate capacity of the combined heat exchanger capacities. An inlet temperature of a heat exchanger fluid may be about 20° C. A heat exchanger fluid flow rate may be about 30 gallons per minute (GPM), between about 25-35 GPM, between about 20-50 GPM, between about 10-75 GPM, or any suitable range or subrange. In operation, a pressure differential between an inlet and an outlet of a heat exchanger may be about 100 mbar.

[0025] When the heat exchangers operate with water flowing at a flow rate on the order of hundreds of gallons per minute, the water pressure may be appreciably high, thus increasing the risk of a water leak in the condenser. A large water leak may be readily detected by monitoring the water pressure through the condenser conduits. For smaller leaks, the heat exchanger may include a leak detection system with pressure sensors disposed within one or both of the water boxes of each condenser. Further details and examples of a leak detection system for the condenser and, in particular, the water boxes, may be found in U.S. Provisional Application No. 63 / 519,095, filed on August 11, 2023 and entitled “EXTERNAL LEAK DETECTION SYSTEM,” which is incorporated by reference herein in its entirety.

[0026] The heat exchanger may be at least partially surrounded by an insulated wall designed to minimize heat transfer to an external environment through the insulated walls of the immersion cooling tank.

[0027] A desiccant box may be located above the heat exchanger to capture water vapor in an upper portion of the immersion cooling tank cavity (also referred to as the headspace of the immersion cooling tank) and arrest transient humidity during immersion cooling system startup as well as keeping stable humidity low during steady state operation of the immersion cooling system. Water vapor can react chemically with immersion cooling fluid and form harmful reaction products including pentafluoropropionic acid (PFPA). The desiccant box may contain one or more desiccants such as silica gels, activated charcoal, calcium sulfate, calcium chloride, and the like.

[0028] The inside of the immersion cooling tank may include a sensor bar for mounting one or more sensors such as fluid level sensors, humidity sensors, pressure sensors, as well as leads for these and other sensors. The one or more sensors may be configured to detect or measure a property associated with the immersion cooling system such as a pressure, a temperature, a voltage, a current, a presence or absence of fluid, a humidity, a status of computing hardware, or any appropriate property. A sensor bar may additionally function to provide a conduit or structure for cables to sensors on or near the sensor bar. FIG. IB illustrates an exemplary sensor bar in greater detail. Sensors used in accordance with the invention may include one or more of a temperature sensor measuring a heat exchange fluid temperature at the inlet of the heat exchanger, a temperature sensor measuring a heat exchange fluid temperature at the outlet of the heat exchanger, an immersion cooling fluid temperature sensor, an immersion cooling vapor temperature sensor, a headspace temperature sensor, and / or one or more pressure sensors measuring a pressure of an internal portion of the immersion cooling tank.

[0029] It should be appreciated that the sensors above are non-limiting examples. In some implementations, additional sensors may be included, for example, to measure the liquid level, froth level, boiling point of the coolant liquid, liquid impedance, froth impedance, vapor impedance, and / or dielectric constants of the coolant liquid. Further details and examples of these fluid sensors that may be found in U.S. Provisional Application No. 63 / 512,575, filed on July 7, 2023 and entitled, “Fluid Sensor for Immersion-Cooling Systems,” and U.S. Provisional Application No. 63 / 516,748, filed on July 31, 2023 and entitled, “DI-ELECTRIC MONITORING OF IMMERSION FLUID DURING COOLING OPERATION.” Each of the aforementioned references is incorporated by reference herein in its entirety.

[0030] A sensor bar may include a quick release clip at the bottom of the sensor bar allowing for a secure connection to the bottom of the immersion cooling tank cavity while allowing the sensor bar to be pulled out of the immersion cooling tank without needing to reach into the immersion cooling fluid cavity (eliminating the need for a human operator to reach into immersion cooling fluid to remove the sensor bar). The quick release clip may include a thin metal tab sloping away from a wall of the immersion cooling tank and upwards. As the sensor bar is lowered onto the quick release clip, the bottom of the sensor bar may be guided between the metal tab and the wall of the immersion cooling tank until the sensor bar solidly contacts both. The top of the sensor bar (above the immersion cooling fluid) may be securely attached to a wall of the immersion cooling tank (for example using bolts, screws, or other attachment mechanisms), accordingly securing the sensor bar in place. Sensors connected to or disposed on the sensor bar may include fluid level sensors, temperature sensors, pressure sensors, humidity sensors, voltage sensors, current sensors, dielectric probes, capacitive sensors, and the like.

[0031] The immersion cooling system may further include a tank controller box, which may house one or more controllers responsible for operating one or more elements of the immersion cooling system. The one or more controllers may include a programmable logic controller (PLC), field-programmable gate array (FPGA), a microprocessor (e.g., a central processing unit (CPU)), a digital signal processing (DSP) die, an application-specific integrated circuit (ASIC), and / or other densely patterned semiconductor die. PLCs used in accordance with the present invention may be designed specifically for high reliability, low cost, ability to withstand wider temperature ranges, vibrations, or impacts than typical commercial computers, and resistance to electrical noise. A PLC may be programmed using an ICE 61131-3 standard and associated Structured Text and / or Instruction List programming languages. A PLC may be programmed using the Ladder Diagram, Eunction Block Diagram, or Sequential Eunction Chart graphical programming languages.

[0032] Further details of a PLC for the two-phase immersion cooling systems disclosed herein may be found in U.S. Provisional Application No. 63 / 519,086, filed on July 7, 2023 and entitled, “LOCAL PLC CONTROLS WITH STATE MACHINE FOR OPERATIONAL CLOUD,” which is incorporated by reference herein in its entirety.

[0033] The controller may be a “tank manager” controller configured to control operations of the tank or components inside the tank such as condenser coil flow rate, immersion cooling filter flow rate, server racks or semiconductor dies including throttling, rate limiting, or depowering semiconductor dies within an immersion cooling system, ventilation systems, cooling fans, tankpower, and the like. A PLC may be limited to simple functions as compared to controller. These functions may include monitoring the one or more sensors, disconnecting power to tank, changing to one of a predetermined number of states, and sending notification messages (also referred to as tickets) to an entity such as a human operator or controller. The benefit of this simple functionality is increased robustness, reliability, and resistance to environmental disruptions such as power surges, abnormally high or low temperatures, vibrations (e.g., due to seismic activity) as compared to more general-purpose computers. A PLC may also provide improved resistance to malware, hacking, and similar digital attack vectors designed to disrupt functioning of systems to which a PLC is communicatively coupled when compared to controller or similarly complex control devices.

[0034] The tank controller box may include one or more fans (not shown) to provide air or other cooling to one or more controllers housed within the tank controller box.

[0035] The immersion cooling system may further include an external filtration system configured to remove chemical, particulate, or other contaminants from immersion cooling fluid within the immersion cooling tank during operation. An external filtration system is illustrated in greater detail in FIG. 3. The external filtration system may include on or more housings for filtration media, a filter pump for forcing fluid from the immersion cooling fluid cavity through the filters and back into the tank, and one or more valves for controlling a flow of immersion cooling fluid through the external filtration system.

[0036] The inside of the immersion cooling tank may include one or more datum pins for communicatively coupling computing hardware (such as servers or semiconductor dies) to one or more external elements such as a computer, server, data center, network, phone, tablet, or similar external element. Datum pins may provide one or more connections for storing, communicating, receiving, updating, calculating, or modifying data using computing hardware inside the immersion cooling tank.

[0037] The immersion cooling system may further include one or more castors for easily moving the immersion cooling system from one location to another by rolling the immersion cooling system along a floor or other surface. The one or more castors may include a braking or locking mechanism for preventing unwanted movement once an immersion cooling system is in a desired location.

[0038] FIG. IB illustrates a sensor bar used in accordance with the invention in greater detail. In an embodiment, a sensor bar may include one or more fluid level sensors (such as a float sensor) at one or more levels within an immersion cooling tank. This may provide an indication of fluidlevel states critical for operation of the immersion cooling system. For example, fluid level sensors may be disposed on the sensor bar in redundant pairs at a “low” level, a “medium” level, and a “high” or “normal” level. An immersion cooling system controller may control, initiate, adjust, terminate, or monitor operation of one or more elements of the immersion cooling system based readings from one or more sensors attached or connected to the sensor bar. For example, an immersion cooling system controller may shut off power to computing hardware within the immersion cooling tank in response to a “low” fluid level sensor indicating that a level of immersion cooling liquid has fallen below a bottom pair of fluid level sensors.

[0039] The sensor bar may further include a cable consolidation box, which may consolidate cables communicatively coupling one or more sensors located within the immersion cooling tank. In an embodiment, the cable consolidation box may provide a single lead external to the immersion cooling tank for communicatively coupling sensors and other electronics within the immersion cooling tank to an external device such as a PLC or tank manager controller using only a single external cable.

[0040] The sensor bar may further include one or more connector leads for sensors disposed elsewhere (not directly on the sensor bar) within the immersion cooling tank, such as one or more humidity and / or pressure sensors.

[0041] FIG. 2A illustrates an alternate perspective view of the immersion cooling system in accordance with the present invention. The immersion cooling system may include one or more bidirectional pressure relief valves configured to release pressure from the immersion cooling tank when an internal gauge pressure increases above a threshold or to raise an internal gauge pressure when it falls below a threshold. The immersion cooling tank may operate in a slight overpressure or vacuum state depending on conditions within the immersion cooling tank during startup or steady state operation. The bidirectional pressure relief valves may prevent a pressure differential between the inside of the immersion cooling tank and an external atmosphere from exceeding a threshold and potentially deforming or damaging the immersion cooling tank or other elements of the immersion cooling system.

[0042] Further details and examples of a pressure relief valve and valve assemblies that include the pressure relief valve may be found in International PCT Application No.PCT / US2023 / 067598, filed on May 30, 2023 and entitled, “PRESSURE RELIEF VALVE AND ADSORBENT CHAMBER FOR TWO-PHASE IMMERSION COOLING SYSTEMS AND METHODS FOR USING SAME,” which is incorporated by reference herein in its entirety.

[0043] The immersion cooling system may additionally include a bellow vent pipe for connecting a bellow to the tank. A bellow may be used to manage vapor and pressure during operation of the tank and provide an additional, adjustable volume that may expand in response to increased pressure during operation. This may prevent an internal pressure of the immersion cooling tank from increasing above a pressure threshold and potentially damaging or deforming the immersion cooling tank. The bellow vent pipe may provide a connection and / or conduit for vapor to move back and forth from inside the immersion cooling tank to inside the bellow in response to operational changes in pressure.

[0044] Further details and examples of bellows may be found in International PCT Application No. PCT / US2023 / 067056, filed on May 16, 2023 and entitled, “BELLOWS FOR IMMERSION COOLING,” which is incorporated by reference herein in its entirety.

[0045] The top of the immersion cooling tank may further include a means to electrically couple one or more elements internal to the immersion cooling tank with one or more elements external to the immersion cooling tank. Such means may be a power pass-through, which may include an electrical connection to one or more cables or bars for providing power to the immersion cooling system and computing hardware within the tank. Elements internal to the immersion cooling tank may include one or more busbars, one or more sensor bars, one or more servers, one or more semiconductor dies, one or more computing devices, one or more integrated circuits, or the like. Elements external to the immersion cooling tank may include one or more power sources, one or more batteries, one or more power grids (such as a municipal power grid, a city power grid, a town power grid, a data center power grid, a local power grid, etc.). The power pass- through may provide electrical connectivity to but fluidic and barometric isolation from a component, device, entity, or element outside of the immersion cooling tank. The power pass- through may be capable of transmitting tens or hundreds of kilowatts of power to the immersion cooling system and / or computing hardware within the immersion cooling tank. In an embodiment, the power pass-through may be capable of transmitting greater than 10 kW of power; greater than 100 kW of power, greater than 250 kW of power, greater than 500 kW of power, or greater than 1 MW of power. In an embodiment, a power pass-through may be designed to transmit between 100 kW and 1 MW of power during operation. In an embodiment, a power pass-through may be designed to transmit between 250 kW and 750 kW of power during operation. In an embodiment, a power pass-through may be configured to transmit between 10 kW and 100 kW of power during operation. In an embodiment, a power pass- through may be configured to transmit 10 kW of power or less during operation. A power pass- through may be connected to one or more bus bars located within the immersion cooling tank.

[0046] As illustrated in FIG. 1A, the immersion cooling tank may include heat exchanger inlet and outlet tubes, insulated wall, tank controller box, castors, and a filtration system. FIG 2A more clearly illustrates a filter pump, which is disposed externally to the immersion cooling tank while remaining fluidically coupled to the immersion cooling tank.

[0047] The immersion cooling tank may include a power shelf rack configured to support and hold one or more power racks for providing power to immersion cooling system components including computing hardware.

[0048] FIG. 2B illustrates an additional perspective of the immersion cooling system including one or more mounting strut elements for mounting components to one or more sides of the immersion cooling tank. For example, a tank controller box may by removably attached to the side of an immersion cooling tank by a connection with one or more mounting struts.

[0049] FIGS. 2B and 2C illustrate a pass-through plate, which may be customizable or reconfigurable. These may be plates designed to incorporate a plurality of different types of means to communicatively couple the one or more semiconductor dies with one or more elements external to the immersion cooling tank. Such means may be in the form of one or more pass-throughs including fiber optic pass-through, RJ45 pass-through, DA- 15 pass-through, and / or T / K TC pass-through. FIG. 2C illustrates a plurality of different types of pass-throughs that may be used as part of a reconfigurable pass-through plate. One or more pass-throughs may be used to communicatively couple one or more components inside the immersion cooling tank (such as computing hardware) to one or more external elements such as a computer, server, data center, network, or similar external element while maintaining fluidic and barometric isolation inside the immersion cooling tank.

[0050] FIG. 2C shows an RJ45 pass-through, which may be used to provide an ethemet protocol connection from one or more components inside the immersion cooling tank (such as computing hardware) to one or more external elements such as a computer, server, data center, network, or similar external element while maintaining fluidic and barometric isolation inside the immersion cooling tank. In an embodiment, one or more RJ45 passthroughs may provide ethernet protocol communication for the operation of computing hardware within the immersion cooling tank.

[0051] FIG. 2C additionally illustrates a fiber pass-through, which may provide a fiber optic connection from one or more components inside the immersion cooling tank (such as computing hardware) to one or more external elements such as a computer, server, data center, network, or similar external element while maintaining fluidic and barometric isolation inside the immersion cooling tank. In an embodiment, one or more fiber pass-throughs may provide a high-bandwidth,low-latency data connection to computing hardware within the immersion cooling tank in order to allow for computationally intensive processing such as Al model training.

[0052] FIG. 2C additionally illustrates a DB-25 pass-through, which may provide a communicative connection according to the DB-25 connectivity protocol from one or more components inside the immersion cooling tank (such as computing hardware) to one or more external elements such as a computer, server, data center, network, or similar external element while maintaining fluidic and barometric isolation inside the immersion cooling tank. In an embodiment, one or more DB-25 pass-throughs may provide communicative connections to sensors or other components disposed within the immersion cooling tank.

[0053] FIG. 3 illustrates a side view perspective of the immersion cooling system. FIG. 3 illustrates several components described in previous figures including heat exchanger inlet and outlet tubes, power pass-through, bellow vent pipe, power rack shelf, and castors. The immersion cooling tank may further include a drain for removing immersion cooling fluid or other fluids from the immersion cooling tank.

[0054] FIG. 3 additionally illustrates a detailed view of a filtration system in accordance with the invention. A filtration system may include a fluidically sealed pump disposed externally to the immersion cooling tank. The pump may be a centrifugal pump configured to operate at around 24 V. In an embodiment, the pump may operate between 20-30 V. A filtration system may include a plurality of pipes or tubes (e.g., copper tubes) providing a fluid circuit for immersion cooling fluid to be transported from the immersion cooling fluid cavity within the immersion cooling tank, through the pump, through chemical and particulate filters, and back into the immersion cooling tank.

[0055] The filtration system may include one or more valves for manually controlling a flow and / or flow path of immersion cooling fluid within the filtration system. For example, the one or more valves may start or stop a flow of immersion cooling fluid through the filtration system. A human operator may wish to stop the flow of immersion cooling fluid while a filter is being changed or a pump is being repaired. Once the repair is complete, the valves may be opened and the flow of immersion cooling fluid through the filtration system may be restarted.

[0056] The filtration system may include one or more housings for filters. In an embodiment, the filtration system may include a chemical filter housing and a particulate filter housing. A chemical filter housing may house one or more filtration media for removing chemical contamination from the immersion cooling fluid. Filtration media for removing chemical contamination may include silica gels, metal sulfates, activated alumina, activated carbon, andthe like. In an embodiment, a chemical filter housing may contain a plurality of filtration media, for example, an activated alumina filtration media and an activated carbon filtration media.

[0057] A particulate filter housing may include a mesh or similar particulate filter that removes particulates from the immersion cooling fluid by providing a barrier through which fluid may flow, but particulates above a threshold size cannot. For example, a particulate filter may be embodied as a mesh with openings less than 1 um in diameter.

[0058] Further details and examples of filtration systems and / or systems related to the operation of a filtration system may be found in U.S. Provisional Application No. 63 / 519,090, filed on August 11, 2023 and entitled, “TWO PHASE IMMERSION COOLING FILTRATION MEDIA HEALTH MONITORING,” U.S. Provisional Application No. 63 / 519,080, filed on August 11, 2023 and entitled, “EXTERNAL FILTRATION SYSTEM FOR IMMERSION COOLING WITH THE FLEXIBLE ARRANGEMENT FOR THE MAINTENANCE AND THE APPARATUS THEREOF,” and U.S. Provisional Application No. 63 / 519,088, filed on August 11, 2023 and entitled “FLEXIBLE CAGE-STYLE FILTRATION ASSEMBLY SYSTEM FOR IMMERSION COOLING AND THE COMPONENTS THEREOF.” Each of the aforementioned references is incorporated by reference herein in its entirety.

[0059] The concentration of contaminants in the coolant liquid may also be reduced through careful selection of materials that are compatible with the coolant liquid (e.g., materials that do not readily leach out contaminants) and / or modifying sensitive components that are particularly susceptible to contamination (e.g., heat-generating electronic components / devices).

[0060] For example, underfill or edge bonds may be applied to any heat generating electronic device (e.g., a ball grid array (BGA) device). Further examples of mitigation strategies for reducing contamination using an edge bond, underfill, or a protective coating may be found in U.S. Provisional Application No. 63 / 503,401, filed on May 19, 2023 and entitled, “Edge Bonding for Immersion Systems,” U.S. Provisional Application No. 63 / 503,405, filed on May 19, 2023 and entitled, “Application of Underfilling BGA Devices in Two Phase Immersion Cooling,” U.S. Provisional Application No. 63 / 505,129, filed on May 31, 2023 and entitled, “PROTECTIVE COATING FOR ELECTRONIC DEVICES IN IMMERSION COOLING SYSTEMS,” and U.S. Provisional Application No. 63 / 514,103, filed on July 17, 2023 and entitled, “EDGE SEALED ELECTRONIC DEVICE FOR IMMERSION COOLING ENVIRONMENT AND METHODS FOR MAKING SAME.” Each of the aforementioned references is incorporated by reference herein in its entirety.

[0061] FIG. 4 illustrates an electrical layout inside an immersion cooling tank in accordance with the invention. The immersion cooling tank many include a heat exchanger analogous to the heat exchanger illustrated in FIG. 1A. One or more power cables may connect from the power pass-through to a bus bar located in the bottom of the immersion cooling tank. The one or more power cables may be greater than Uz” in diameter, for example about 1” in diameter.

[0062] The bus bar may provide a power connection to one or more servers disposed within the immersion cooling tank. The bus bar may be kept above a certain voltage during operation, for example above 1 V, above 10 V, above 100 V, or above 1000 V. The bus bar may include one or more sockets for electrically coupling the bus bar to one or more computing hardware elements such as one or more servers. The bus bar may include a high voltage bar and a ground bar.

[0063] The immersion cooling tank may include computational hardware such as one or more servers configured to perform calculations or other computing operations. Computational hardware may include one or more processors, graphics processing units (GPUs), FPGAs, PCBs, integrated circuits, semiconductor dies, or the like. The immersion cooling tank may contain enough immersion cooling liquid to completely immerse the servers in immersion cooling liquid during operation.

[0064] The immersion cooling tank may further include a tray with guides for assisting server installation and supporting one or more elements within the immersion cooling tank, including supporting the servers. The guides may assist a human operator or automated server installation process by aligning one or more power, data, or structural connectors disposed on the servers with corresponding connection points on the tray, bus bar, or immersion cooling tank.

[0065] FIGS. 5 and 6 illustrate exemplary server rack structures for easily installing servers within an immersion cooling tank while allowing for ease of maintenance. FIG. 5 depicts an exploded view of a server rack configured to support one or more servers during operation.

[0066] The server rack may include one or more filler panels (for example a left-side filler panel and a right-side filler panel) for providing support and stiffness. The server rack assembly may further include one or more gates for busbar maintenance. The server rack assembly may include an inner rack body providing structure and containment for the servers and other components disposed within or on the server rack assembly. The upper four corners of the inner rack body may include a lift system, for example one or more L brackets with attachment structures such as eye hooks for lifting and lowering the server assembly into the immersion cooling tank during installation, maintenance, and repair operations.

[0067] Further examples of a lift system to facilitate installation and / or removal of an assembled unit to / from the tank may be found in U.S. Provisional Application No. 63 / 519,075, filed on August 11, 2023 and entitled, “Testing of Immersion-Cooled Servers in a Manufacturing Environment,” which is incorporated by reference herein in its entirety.

[0068] The bottom of the server rack assembly may include a busbar and busbar cover. The busbar may deliver electrical power to the servers or other electronics within the immersion cooling tank during operation. The busbar cover may function to prevent components contacting the busbar in unintended locations and potentially short circuiting or otherwise damaging one or more components housed within the server rack assembly. The bottom four corners of the inner rack body may include one or more guide pins for aligning the server rack in the proper location during installation, maintenance, and repair operations.

[0069] To further reduce or, in some instances, mitigate alignment errors - particularly between the assembled unit and the guide rails, a compliant gasket (also referred to as a sleeve) may be disposed on the sides of the assembled unit to fill any gaps between the assembled unit and each guide rail. Further details and examples of a sleeve to facilitate installation and / or removal of an assembled unit to / from the tank may be found in U.S. Provisional Application No. 63 / 516,996, filed on August 1, 2023 and entitled, “Multi-Purpose Sleeve for Immersion-Cooled Servers,” which is incorporated by reference herein in its entirety.

[0070] Further details and examples of a busbar may be found in U.S. Provisional Application No. 63 / 514,259, filed on July 18, 2023 and entitled, “Busbar for Immersion-Cooling System,” and U.S. Provisional Application No. 63 / 515,423, filed on July 25, 2023 and entitled, “Busbar for Immersion-Cooling System.” Each of the aforementioned references is incorporated by reference herein in its entirety.

[0071] One or more filler panels may include one or more busbar channels for routing a portion of a busbar upwards towards the top of the tank. These one or more busbar channels may be covered by a busbar channel cover, for example to prevent unintended contact with the busbar when it is biased to high voltages during operation.

[0072] FIG. 6 further illustrates an exploded view of the server rack assembly. The server rack assembly may be analogous to the server rack assembly illustrated in FIG. 5 and may include left-side and right-side filler panels, an inner rack door or gate for busbar maintenance, and a rear busbar insulation cover for insulating a portion of the busbar contained in a busbar slot on a right-side filler panel. The server rack assembly may include a busbar base tray for isolating abottom of the busbar from the inner rack body. The server rack assembly may include an inner rack body analogous to the inner rack body illustrated in FIG. 5.

[0073] The server rack assembly of FIG. 6 may further include a busbar and busbar cover analogous to the busbar and busbar cover illustrated in FIG. 5. The inner rack bottom may include bakelite as an insulation for insulating a protecting the busbar during installation and operation. The inner rack body may further include one or more L brackets analogous to those illustrated in FIG. 5 for lifting, installing, and removing the server rack assembly from the immersion cooling tank. The inner rack body may be welded to provide rigidity.

[0074] FIG. 7 depicts aspects of an immersion-cooling system 700 for dissipating heat from one or more semiconductor die packages 705 via immersion cooling. Each package 705 can include one or more semiconductor dies that produce heat when the system is in operation. The immersion-cooling system 700 in the illustrated example of FIG. 7 is a two-phase immersioncooling system, though the invention may also be implemented in a single-phase immersioncooling system.

[0075] The immersion-cooling system 700 includes a container such as tank 720 filled, at least in part, with immersion cooling liquid 764. The immersion-cooling system 700 can further include at least one chiller 780 that flows a heat-transfer fluid through at least one condenser coil 770 that is disposed in the tank 720 and headspace 708. Condenser coil 770 and chiller 780 may be part of a heat exchanger. The packages 705 can be mounted on one or more printed circuit boards (PCBs) 757 that are immersed, at least in part, in the immersion cooling liquid 764. Immersion-cooling system 700 may further include a filter 775 disposed adjacent to the tank 720.

[0076] Filter 775 may include a filtration media, a housing, and a pump configured to force immersion cooling liquid 764 through filter 775 to remove contaminants, particulates, or other impurities that may be added to immersion cooling liquid 764 during use. Filter 775 may be housed outside of tank 720 while being in fluidic communication with immersion cooling liquid 764 in tank 720. Alternatively, filter 775 may be submerged within immersion cooling liquid 764 inside of tank 720.

[0077] Immersion cooling liquid 764 may be a hydrocarbon, a fluoroketone, an oil, or a similar dielectric liquid that will act as an insulator while simultaneously transferring heat from package 705 more efficiently than air. An example of immersion cooling liquid 764 is Novec™ 649 produced by 3M™. An exemplary immersion cooling liquid 764 used in accordance with embodiments of the present invention may have a dielectric constant baseline value of about 1.8-2 at a frequency of about 1 kHz.

[0078] In an embodiment of the invention, immersion cooling liquid 764 may be considered unacceptably contaminated if the dielectric constant and / or dielectric loss tangent of immersion cooling fluid being used in an immersion-cooling system 700 differs by a threshold amount as compared to unused or pure immersion cooling liquid 764. For example, immersion cooling liquid 764 may be considered unacceptably contaminated or degraded if the dielectric constant and / or dielectric loss tangent differs by a threshold of 10% or more as compared to unused or pure immersion cooling liquid 764. In an embodiment, a dielectric constant and / or dielectric loss tangent variation threshold may be 20%, 15%, 5%, 3%, 1%, or any suitable threshold.

[0079] Contamination of the immersion cooling liquid 764 and resulting changes to dielectric constant and / or dielectric loss tangent may alter or negatively impact operation of components within immersion cooling liquid 764 including semiconductor die(s) 750. An altered dielectric constant and / or dielectric loss tangent may result in undesirable cross-talk between components on a PCB, additional noise or reduction in signal strength transmitted along exposed wires of a PCB or semiconductor die(s) 750 submerged in immersion fluid, and / or signal dissipation through the immersion cooling liquid 764. Signal loss may be severe enough that two elements may be effectively represented as being separated by an open circuit despite being physically connected. In an embodiment, a dielectric constant and / or dielectric loss tangent variation threshold may be selected based on an observed or inferred effect on one or more submerged semiconductor die(s) 750. For example, an increase in PCIe bit error rate above an error rate baseline may be correlated with an increase in dielectric constant and / or dielectric loss tangent above a dielectric constant and / or dielectric loss tangent baseline. Accordingly, operation of semiconductor die(s) 750 may be throttled or suspended when a dielectric constant and / or dielectric loss tangent of immersion cooling liquid 764 exceeds a predetermined threshold.

[0080] Changes to dielectric constant and / or dielectric loss tangent may be caused by contaminants within immersion cooling liquid 764. In some cases, changes to dielectric constant and / or dielectric loss tangent may be reversed by filtering the contaminants from immersion cooling liquid 764. In some embodiments, upon detecting an increase in dielectric constant and / or dielectric loss tangent of immersion cooling liquid 764, controller 702 may instruct filter 775 to increase filtration throughput or notify a user that an immersion cooling liquid 764 filtration media may need to be replaced. If a dielectric constant and / or dielectric loss tangent exceeds a predetermined threshold, controller 702 may throttle or shut down one or more semiconductor die(s) 750, generate a notification that immersion cooling liquid 764 should be replaced, trigger an alarm, etc.

[0081] The illustrated example of FIG. 7 is not intended to be to scale. The immersion-cooling system 700 may house and provide immersion cooling liquid 764 to tens, hundreds, or even thousands of packages 705. In some cases, the immersion-cooling system 700 can be small (e.g., the size of a floor unit air conditioner, approximately 1 meter high, 0.5 meter width, 0.5 meter depth or length). In some implementations, the immersion-cooling system can be large (e.g., the size of a van or larger, approximately 2.5 meters high, 2.5 meters width, 4 meters depth or length).

[0082] The immersion-cooling system 700 can also include a controller 702 (e.g., a microcontroller, programmable logic controller (PLC), microprocessor, field-programmable gate array, logic circuitry, memory, or some combination thereof) to manage system operation. Controller 702 can perform various system functions such as monitoring temperatures of system components, cooling fluid level, tank access, chiller operation etc. The controller 702 can further issue commands to control system operation such as executing a start-up sequence, executing a shut-down sequence, assigning workloads among the packages, changing cooling fluid level, changing the temperature of the heat-transfer fluid circulated by the chiller 780, etc. In some implementations, the controller 702 can include (or itself be) a baseboard management controller (BMC) 704. That is, the BMC 704 may monitor and control all aspects of system operation for the immersion-cooling system 700 in addition to monitoring and controlling workloads of the semiconductor dies 750 in the packages 705 cooled by the system. The immersion-cooling system 700 can also include a network interface controller (NIC) 703 to allow the system to communicate over a network, such as a local area network or wide area network. The immersion-cooling system 700 can further include a fluid sensor array 790 having a plurality of fluid sensors 710. Fluid sensors 710 may include one or more leak detection sensors at least partially submerged in immersion cooling liquid 764.

[0083] The semiconductor die(s) 750 and can be mounted on and attached to a printed circuit board (PCB) 755 (sometimes referred to as a substrate) in device package 705. The package 705 can be made commercially available as an off-the-shelf (OTS) product. The package 705 can be used for single-phase or two-phase immersion cooling of at least one semiconductor die 750, such as a microprocessor (e.g., a central processing unit (CPU) and / or graphics processing unit (GPU)), voltage regulator (VR), high bandwidth memory (HBM), a digital signal processing (DSP) die, an application- specific integrated circuit (ASIC), field-programmable gate array (FPGA), and / or other densely patterned semiconductor die.

[0084] In the two-phase immersion-cooling system 700 of FIG. 7, heat flows from the semiconductor die 750 where it is generated into the heat spreader 752. The heat spreader 752 isin thermal contact with an immersion cooling liquid 764 that can flow over and extract heat from the heat spreader 752. The amount of heat delivered by the heat spreader 752 to the immersion cooling liquid 764 is enough to boil the immersion cooling liquid 764 that contacts the heat spreader 752 (creating bubbles 765 and potentially creating froth 767 when bubbles 765 reach the surface of immersion cooling liquid 764). The vapor 766 from the boiled immersion cooling liquid 764 can be cooled and condensed back to liquid droplets 768, for example, by the condenser coil 770. The heat-transfer fluid, such as chilled water, from the chiller 780 can be circulated through the condenser coil 770 to lower the temperature of the condenser coil 770 below the condensation point in the headspace 708 of the tank 720. As a result, vapor 766 condenses on exterior surfaces of the condenser coil 770 and liquid droplets 768 from the condensed vapor can drip and / or flow back to the immersion cooling liquid 764. Although a single condenser coil 770 is depicted in FIG. 7, there can be a plurality of condenser coils 770 in tank 720 to condense the vapor 766 into droplets. Some or all of the condenser coils 770 may or may not be located directly over the PCBs 757. Instead, the condenser coil(s) 770 can be located near one or more walls of the tank 720, such that the condenser coil(s) 770 are not directly over the PCBs 757 on which the packages 705 are mounted.

[0085] To improve thermal performance in two-phase immersion-cooling system 700, the heat spreader 752 can include a boiling enhancement coating (BEC) on at least one surface. The BEC can be formed from copper or a copper alloy and can be porous, for example, though BECs can take various forms. In some cases, the BEC is a micro porous copper coating having a thickness from approximately or exactly 50 microns to 700 microns thick (which may be produced by electroplating and / or etching). In some implementations, the BEC comprises a mesh copper layer bonded (e.g., via resistance heating) to at least an outer surface of the heat spreader 752. In some cases, the BEC is applied as particulates to at least one smooth surface of the heat spreader 752 and then subsequently sintered to adhere to one another and to the heat spreader 752. The BEC provides an improved surface area to contact the immersion cooling liquid 764 and can increase the heat transfer coefficient from the heat spreader 752 to the immersion cooling liquid 764 by up to a factor of 15 versus a smooth surface on the heat spreader 752. Accordingly, BECs can increase thermal conductivity to, and accelerate the boiling of, the immersion cooling liquid 764.

[0086] The immersion cooling systems disclosed herein may meet various safety and environmental standards set forth by various organizations including, but not limited to, the International Electrotechnical Commission (IEC), and the European Union (EU). For example, the immersion cooling systems disclosed herein may meet the requirements defined by the IECstandard IEC 62368-1 entitled, “Audio / video, information and communication technology equipment - Part 1: Safety requirements,” which standard is hereby incorporated herein by reference. In another example, the immersion cooling systems disclosed herein may meet the requirements defined by the EU Restriction on Hazardous Substances (RoHS) directive, which standard is hereby incorporated herein by reference.

[0087] The immersion cooling system may generally include components that may be serviced and / or replaced overtime. These components include, but is not limited to, the filter pump for the filtration system, the carbon filter for the filtration system, bellows, the sensor bar, the desiccant assembly, and the pressure transducers of the filtration system.

[0088] An immersion cooling system including an immersion cooling tank may be moved, positioned, or repositioned using one or more castors. For example, the immersion cooling tank may be positioned to connect one or more elements external to the immersion cooling tank to one or more elements internal to the immersion cooling tank. For example, the immersion cooling tank may be positioned to electrically couple a power source external to the immersion cooling tank to one or more elements within the immersion cooling tank such as computing hardware, one or more busbars, one or more sensors, one or more sensor bars, or other elements. One or more elements external to the immersion cooling tank may be electrically coupled to one or more elements internal to the immersion cooling tank using a means for electrically coupling as described above. One or more operations of the computing hardware may be validated using the immersion cooling system. For example, the immersion cooling system may be used to determine if computing hardware such as a server or semiconductor die is exhibiting operational parameters above a threshold, below a threshold, within a predetermined range, or the like. For example, validation may include determining computing hardware operational parameters are within a predetermined voltage range, within a predetermined current range, within a predetermined computational speed range, within a predetermined power range, within a predetermined temperature range, or the like. Additionally or alternatively, instead of any of the above recited ranges, a validation process may include determining if any of the above parameters are above or below a threshold.Conclusion

[0089] While various inventive embodiments have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is deemed to be within the scope of the inventive embodiments described herein. More generally, those skilled in the artwill readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the inventive teachings is / are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation, many equivalents to the specific inventive embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed. Inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the inventive scope of the present disclosure.

[0090] Also, various inventive concepts may be embodied as one or more methods, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

[0091] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0092] The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.”

[0093] The phrase “and / or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and / or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and / or” clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to “A and / or B”, when used in conjunction with open-ended language such as “comprising” can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionallyincluding elements other than A); in yet another embodiment, to both A and B (optionally including other elements); etc.

[0094] As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of’ or “exactly one of,” or, when used in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e., “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.” “Consisting essentially of,” when used in the claims, shall have its ordinary meaning as used in the field of patent law.

[0095] As used herein in the specification and in the claims, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, “at least one of A and B” (or, equivalently, “at least one of A or B,” or, equivalently “at least one of A and / or B”) can refer, in one embodiment, to at least one, optionally including more than one, A, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in yet another embodiment, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other elements); etc.

[0096] In the claims, as well as in the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases “consisting of’ and “consisting essentially of’ shall be closed or semi-closed transitional phrases, respectively, as set forth in the United States Patent Office Manual of Patent Examining Procedures, Section 2111.03.

Claims

CLAIMS1. A system for immersion cooling, the system comprising: an immersion cooling tank; a busbar disposed within the immersion cooling tank, the busbar configured to be biased to a voltage; one or more castors attached to a bottom of the immersion cooling tank for moving the immersion cooling tank; one or more sensors disposed within the immersion cooling tank, the one or more sensors configured to monitor one or more properties of the system; and computing hardware disposed within the immersion cooling tank and electrically coupled to the busbar, wherein the computing hardware is at least partially immersed in an immersion cooling fluid.

2. A system for immersion cooling, the system comprising: an immersion cooling tank containing immersion cooling fluid; one or more semiconductor dies disposed within the immersion cooling tank and immersed within the immersion cooling fluid; and one or more means to communicatively couple the one or more semiconductor dies with one or more elements external to the immersion cooling tank.

3. The system of claim 2, wherein the one or more means comprise at least one of a fiber optic pass-through, an RJ45 pass-through, a DB-25 pass-through, or a T / K TC pass-through.

4. A method for immersion cooling, the method comprising: positioning, using one or more castors, an immersion cooling tank to connect to a pass- through; electrically coupling, using the pass-through, an external power source to a computing hardware disposed within the immersion cooling tank; validating an operation of the computing hardware using power supplied by the power source; wherein the pass-through is disposed on a wall of the immersion cooling tank and comprises at least one of a fiber optic pass-through, an RJ45 pass-through, a DB-25 pass- through, or a T / K TC pass-through.

Images