Testing of immersion-cooled servers in a manufacturing environment

Abstract

Testing of assembled units such as internet servers in a manufacturing facility is described. Tests are carried out in test tanks of immersion-cooling systems where the assembled units are at least partially submerged while testing is performed. A lifting apparatus to lift and load two or more assembled units at a time is described.

Description

TESTING OF IMMERSION-COOLED SERVERS IN A MANUFACTURING ENVIRONMENTCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Provisional Application No. 63 / 519,075 filed August 11, 2023, which is hereby incorporated by reference in its entirety.BACKGROUND

[0002] As feature sizes and transistor sizes have decreased for integrated circuits (ICs), the amount of heat generated by a single chip, such as a microprocessor, has increased. Chips that once were air cooled have evolved to chips needing more heat dissipation than can be provided by air alone. In some cases, immersion cooling of chips in a tank containing a coolant liquid is employed to maintain IC chips at appropriate operating temperatures.

[0003] One type of immersion cooling is two-phase immersion cooling, in which heat from a semiconductor die is high enough to boil the coolant liquid in the tank. The boiling creates a coolant-liquid vapor in the tank, which is condensed by cooled coils or pipes back to liquid form. Heat from the semiconductor dies can then be sunk into the liquid-to-gas and gas- to-liquid phase transitions of the coolant liquid.SUMMARY

[0004] The present disclosure relates to testing large assembled electrical units (such as servers weighing at least 150 pounds or more) that will be operated in an immersion-cooling system. The testing is carried out in an immersion-cooling test tank in a manufacturing environment. Many server units are to be tested quickly in test tanks so that high manufacturing throughput can be achieved.

[0005] Some implementations relate to a process to test a plurality of assembled units for operation in an immersion-cooling system. The process can comprise acts of: arranging two or more assembled units of the plurality of assembled units in a first aligned configuration beside a test tank, wherein the first aligned configuration resembles a second aligned configuration that the two or more assembled units will have when installed in the test tank at their respective installed locations; installing the two or more assembled units into the test tank and into coolant liquid located in the test tank such that at least a portion of each assembled unit of the two or more assembled units is submerged in the coolant liquid; guiding the two or more assembled units into the second aligned configuration at their installed locations; and automaticallyconnecting, for each of the two or more assembled units, electrical power connections to a power busbar in the tank when the two or more assembled units arrive at their installed locations.

[0006] All combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are part of the subject matter disclosed herein. In particular, all combinations of subject matter appearing in this disclosure are part of the subject matter disclosed herein. The terminology used herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The skilled artisan will understand that the drawings primarily are for illustrative purposes and are not intended to limit the scope of the subject matter described herein. The drawings are not necessarily to scale; in some instances, various aspects of the subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference characters generally refer to like features (e.g., functionally and / or structurally similar elements).

[0008] FIG. 1 depicts an example of a two-phase immersion-cooling system that can be used to cool semiconductor dies.

[0009] FIG. 2 depicts a test tank for testing assembled units and also illustrates an assembled unit installed in the test tank.

[0010] FIG. 3 depicts a group of assembled units, which can be installed in the test tank of FIG. 2, held in an aligned configuration by framework.

[0011] FIG. 4A depicts a lifting head lowering at least one assembled unit into a test tank like that of FIG. 2.

[0012] FIG. 4B depicts an assembled unit partially loaded into a test tank like that of FIG. 2.

[0013] FIG. 4C depicts another implementation of a lifting head lowering at least one assembled unit into a test tank like that of FIG. 2.

[0014] FIG. 5 depicts a lifting apparatus beside a test tank.

[0015] FIG. 6 illustrates acts associated with a testing process for testing assembled units in a tank of an immersion-cooling system.

[0016] FIG. 7A depicts an immersion-cooling system including a test tank for testing assembled units.

[0017] FIG. 7B is a perspective view of the immersion-cooling system of FIG. 7A.

[0018] FIG. 8 illustrates an example floor layout of an immersion-cooling system.

[0019] FIG. 9 illustrates acts associated with a failure analysis process.

[0020] FIG. 10 illustrates acts associated with another failure analysis process.DETAILED DESCRIPTION1. Overview of Immersion- Cooling System

[0021] FIG. 1 depicts aspects of an immersion-cooling system 160 for dissipating heat from one or more semiconductor die packages 105 containing semiconductor dies via immersion cooling. FIG. 1 depicts an instance while the semiconductor dies are operating and generating enough heat to boil coolant liquid 164 in the tank 107. Each package 105 can include one or more semiconductor dies that produce heat when the semiconductor dies are in operation (e.g., in a server system). The immersion-cooling system 160 in the illustrated example of FIG. 1 is a two-phase immersion-cooling system, though the invention can also be implemented for singlephase immersion-cooling systems.

[0022] The immersion-cooling system 160 includes a tank 107 filled, at least in part, with coolant liquid 164. The immersion-cooling system 160 can further include at least one chiller 180 that flows a heat-transfer fluid through at least one condenser coil 170 or pipes located in the head space 109 of the tank 107. The condenser coil(s) 170 or pipes can condense coolant liquid vapor 166 into droplets 168 that return to the coolant liquid 164. The packages 105 can be mounted on one or more printed circuit boards (PCBs) 157 that are immersed, at least in part, in the coolant liquid 164. In some implementations, a foam or froth 167 can form in the tank 107 above the coolant liquid 164 when the system is in operation. The froth 167 can be formed mostly from bubbles 165 that collect across the surface of the coolant liquid 164.

[0023] The illustrated example of FIG. 1 is not intended to be to scale. The immersioncooling system 160 may house and provide coolant liquid 164 to tens, hundreds, or even thousands of packages 105. According to some implementations, the packages 105 can be included in assembled units, such as server units. In some cases, the tank 107 of an immersioncooling system 160 can be small (e.g., the size of a floor unit air conditioner, approximately 1 meter high, 0.5 meter width, 0.5 meter depth or length). In some implementations, the tank 107 of an immersion-cooling system can be large (e.g., the size of an automotive van, approximately 2.5 meters high, 2.5 meters width, 4 meters depth or length). In some cases, the tank 107 of an immersion-cooling system 160 can be larger than an automotive van.

[0024] The immersion-cooling system 160 can also include a controller 102 (e.g., a microcontroller, programmable logic controller, microprocessor, field-programmable gate array, logic circuitry, memory, or some combination thereof) to manage operation of at least theimmersion-cooling system 160. The controller 102 can perform various system functions such as monitoring temperatures of system components, coolant liquid fluid level, technician access to the interior of the tank, chiller operation, etc. The controller 102 can further issue commands to control system operation such as executing a start-up sequence, executing a shut-down sequence, assigning workloads among the packages 105 and / or assembled units, changing coolant liquid fluid level, changing the temperature of the heat-transfer fluid circulated by the chiller 180, etc. In some implementations, the controller 102 can include (or itself be) a baseboard management controller (BMC) 104. That is, the BMC 104 may monitor and control all aspects of system operation for the immersion-cooling system 160 in addition to monitoring and controlling workloads of the semiconductor dies 150 in the packages 105 and / or workloads among the assembled units cooled by the immersion-cooling system 160. The controller 102 can include processing circuitry P and a memory M, which, in operation, control various functions of the controller (e.g., instructions or configuration information stored in the memory can be executed, employed, or combinations thereof, by the processing circuitry to perform various functions of the controller). The immersion-cooling system 160 can also include a network interface controller (NIC) 103 to allow the system to communicate over a network, such as a local area network or wide area network.2. Test Tank for Testing Assembled Units in a Manufacturing Facility

[0025] FIG. 2 depicts a test tank 207 for immersion-cooling of assembled units 210 that can be loaded into and unloaded from the test tank 207. The test tank 207 can be installed in a manufacturing facility so that assembled units 210 can be operated in an immersion-cooling environment to stress test the assembled units 210 and / or emulate real- world operation of the assembled units 210. The assembled units 210 can be servers, dedicated processing units, data- mining units, high bandwidth memory units, or other types of processing and / or memory units.

[0026] In some implementations, the tank 207 can include compartments 205 that are not filled with coolant liquid 164. The compartments 205 can be outside the main volume of the tank that is filled with coolant liquid 164, for example. The compartments 205 can be used to house server components that need not be cooled by liquid (e.g., signal routing electronics and connectors). Condensing pipes 270 can be located above and to the side of the assembled units 210 so that the assembled units can be lowered into and removed from the test tank 207 without contacting the condensing pipes 270. There can be many more condensing pipes than the number shown in FIG. 2.

[0027] In some implementations, the test tank 207 has a same or a similar configuration as tanks deployed to data centers, etc. Process modifications can be employed to facilitate use in testing operations.

[0028] In a manufacturing facility, it can be important to quickly test a large number of assembled units 210 so that high manufacturing throughput can be achieved. Accordingly, a single test tank 207 can be sized to contain 10 or more assembled units 210 (e.g., at least from 10 to 70 assembled units 210). In some cases, a test tank 207 can be sized to contain more than 70 units. The assembled units 210 can be placed in one or more rows (e.g., extending in the ±x direction in the illustration) within the test tank 207. In the manufacturing facility, there can be a plurality of test tanks 207 each containing a plurality of assembled units 210. The test tanks 207 can be operated continuously (with standby periods during exchange of assembled units 210) for days, weeks, or months before substantial servicing of any of the test tanks is needed. A manufacturing facility can have from 5 to 50 test tanks or more. Newly manufactured assembled units 210 can be loaded into a test tank 207, tested at normal or accelerated operating conditions, and unloaded from the test tank to make way for a next batch of newly manufactured assembled units 210. Such loading, testing, and unloading can occur multiple times during a week to as much as one or more times during a day.3. Loading, Alignment, and Unloading of Assembled Units

[0029] Assembled units 210 can be loaded into and unloaded from the test tank 207 through an opening 220 in the test tank 207 that is covered by an access lid or access door 212 when testing of the assembled units is performed. Although the access door 212 is depicted on the top side of the test tank 207, the access door 212 could be located on a side wall of the test tank 207. In an embodiment, the access door 212 is located above the highest level that coolant liquid 164 reaches in the test tank 207 so that the assembled units 210 can be unloaded from and loaded into the test tank 207 without draining the coolant liquid 164 or a portion thereof from the test tank 207 to perform the unloading / loading operation.

[0030] The assembled units 210 can each be mounted in the test tank 207 in a way that establishes physical support of the assembled unit 210 within the tank and establishes electrical connections for power and data communications. For example, there can be one or more busbars 230 located along the base of the test tank or along a sidewall of the test tank 207. An assembled unit 210 can be lowered through the opening 220 carefully such that it does not contact the condensing pipes 270 within the tank. The assembled unit 210 can include male and / or female electrical connectors (shown in FIG. 4A) that engage with mating connectors on the busbar(s) 230. Accordingly, the assembled unit 210 can be aligned as it is lowered to engage the matingconnectors. When an assembled unit 210 arrives at its installed location, it can be physically supported by the tank while at the same time automatically engage the mating connectors to establish power and / or data links to the installed assembled unit 210.

[0031] In some implementations, alignment of the assembled units 210 can be assisted by mechanical registration features (e.g., conical pins and / or conically- shaped receiving holes or detents that force alignment of the assembled unit 210 prior to engagement of electrical connectors). In some implementations, alignment of the assembled units 210 can be assisted by imaging apparatus (e.g., CCD or CMOS cameras) and image processing of acquired images. For example, alignment marks or features on the assembled unit 210 can be continuously imaged and compared to alignment marks or features within the test tank (to which the marks or features on the assembled unit 210 should register) as the assembled unit 210 is lowered into place. Automated image processing can provide feedback to lifting apparatus that can lower and align the assembled unit 210 into the test tank 207. The lifting apparatus can comprise at least one robotic arm capable of lifting and placing at least one assembled unit 210 at a time.

[0032] According to some implementations, two or more assembled units 210 can be lifted and placed in the test tank 207 at the same time. For example, at least 2 to 10 assembled units 210 can first be installed in a metal framework 310 or otherwise precisely located with respect to each other outside the test tank 207, as depicted in FIG. 3 for example. The framework 310 can hold the multiple assembled units 210 in an aligned configuration that will mate or approximately mate electrical connectors on the assembled units 210 to electrical connectors on the busbar(s) 230 within the test tank 207. The framework 310 and registered assembled units 210 can then be lifted, aligned, and lowered into the test tank 207 by lifting apparatus. Mechanical registration features 320 and / or imaging apparatus and image processing as described above can assist with alignment of the framework 310 into the tank. In some cases, the framework 310 can remain in the test tank 207 while the assembled units 210 are being tested. In other implementations, the framework 310 can be removed from the tank while the assembled units 210 are being tested. According to some implementations, the framework 310 can be placed into the test tank 207 prior to installation of the assembled units and provide a guide to one or more assembled units 210 as the units are lowered into the tank 207.

[0033] In some cases, the head on the lifting apparatus is configured to grab two or more assembled units 210 in an aligned or nearly aligned configuration. For example, the lifting head effectively includes a framework that aligns the two or more assembled units prior to their installation in the test tank 207. In some cases, a hydraulic jaw in the lifting head can grasp two or more assembled units 210 into an aligned or nearly aligned configuration. The hydraulic jawcan include guiding features to guide the assembled units into a nearly aligned configuration as the jaw closes and grasps the units for lifting.

[0034] Because the test tank 207 can include coolant liquid 164 that is above the height of the installed assembled units 210, special measures can be taken when using imaging apparatus and image processing to aid in alignment of the assembled units. Since the coolant liquid has a different optical refractive index than air, optical refraction can be accounted for when viewing submerged alignment features from non-normal incidence angles with cameras located above the surface of the coolant liquid 164. Further, as the lifting apparatus moves the assembled units 210 into and within the coolant liquid 164, it can create waves on the surface of the coolant liquid 164. Such waves can optically distort images of alignment features located below the surface of the coolant liquid 164. Consequently, when using imaging apparatus for aligning the assembled units, slow movement and / or delays after movement can be implemented to mitigate optical distortion from waves in the tank.

[0035] To avoid refractive and distortion effects of the coolant liquid 164, imaging apparatus can be located on the lifting head or at a base of the tank such that the imaging apparatus becomes immersed or is immersed when image processing begins to align the assembled units 210 to their installed locations within the test tank 207.

[0036] FIG. 4A depicts one approach to aligning and loading assembled units 210 into the test tank 207 that may not use imaging and image processing to align the assembled units. A testing system 460 includes a test tank 207 and a lifting apparatus 500. The lifting apparatus includes a lifting head 450, which can include at least one pair of claws 454 that engage with lift features 214 that are connected to each assembled unit 210. The lift features 214 can comprise loops, holes, detents, tabs, or voids in metal that can be engaged by the claws 454 to lift the assembled unit 210. The lifted unit 210 can dangle below the lifting head 450 when lifted such that the base of the assembled unit can move in one or more degrees of freedom with respect to the lifting head 450. In some implementations, the lifting head 450 can have multiple pairs of claws 454 spaced apart (in the +r direction in FIG. 4A) to grasp lift features 214 on multiple assembled units 210 spaced apart in the ±x direction. In an embodiment, the spacing apart of the pairs of claws 454 in the +r direction on the lifting head 450 matches the spacing apart that the assembled units 210 will have when mounted in the test tank 207 in their respective installed locations so that the lifting head 450 can place multiple assembled units 210 into the test tank 207 at the same time.

[0037] Guide rails 203 can be installed in the tank (e.g., as part of a framework that is installed permanently or temporarily during loading) to guide the assembled unit(s) 210 into their final installed location(s) while the assembled unit(s) 210 is or are being loaded into the tank207. The guide rails 203 may allow for some free motion (e.g., up to ± 1 mm or even up to ± 5 mm in x and y directions) of the base of the assembled unit 210 such that registration pins or alignment pins 215 can engage with alignment receptacles 216 to pull the base of the assembled unit 210 into alignment (e.g., to within ± 0.5 mm in x and y directions of an installed location), so that the electrical connectors on the assembled unit(s) 210 can mate with electrical connectors on the busbar(s) 230. The electrical connectors on an assembled unit 210 can comprise power pins 217 and data signal pins 213. In some implementations, the signal pins 213 can have a tighter alignment tolerance than the power pins 217. In some implementations, the alignment pins and / or electrical pins can be located on the base of the tank and / or busbar instead of on the assembled unit and the assembled unit 210 can include mating features (e.g., sockets) for the pins. In some implementations, power connections to assembled units 210 are provided via the busbar 230, and I / O connections (e.g., network connectivity) to assembled units 210 are provided via cables, for example, cables coupled to the top of the assembled units 210, and external switches.

[0038] In some implementations, assembled units 210 having different characteristics (e.g., size, weight, coupling configurations, etc.) can be accommodated in the test tanks 207. For example, in some implementations, the test tank 207 is configured to accommodate up to 32 assembled units of a first type, up to 32 assembled units 207 of a second type, or up to a combination of 32 units including units of the first type and units of the second type.

[0039] FIG. 4B shows a perspective view of the test tank 207 and an assembled unit 210 being loaded into position. The test tank 207 includes compartments 205 on either side of the main chamber of the tank into which the assembled units are placed. The main chamber will be filled with coolant liquid 164 to immerse the assembled units 210. The assembled unit 210 is guided by guide rails 203 into its installed location. Alignment pins 215 can engage with receiving holes 216 at the base of the tank to pull the base of the assembled unit 210 into alignment prior to engagement of electrical power pins 217. The alignment pins 215 can be located away from the busbar 230.

[0040] According to some implementations, the assembled unit 210 includes feet 218 at the base of the unit. The feet 218 can be arranged to carry all the weight of the assembled unit 210 and physically support the assembled unit when the unit is in its installed location. As such, there can be negligible (e.g., less than 2 psi) or no weight on portions of the assembled unit around and near (e.g., within 1 cm from) the electrical connectors that mate to connectors on the busbar(s) 230. Reduction of load-bearing weight on or near the electrical connectors can prolong the life of the connectors.

[0041] In another approach depicted in FIG. 4C, laser diodes or LEDs 410 and positionsensitive detectors 420 can be used to align assembled units 210 and / or a framework 310 (see FIG. 3) in which the assembled units are mounted as the units are placed in the test tank 207. If the assembled units 210 or framework 310 have a power source or are connected to a power source when being loaded, either the assembled units 210, the framework 310, or the base of the test tank can include at least two laser diodes or LEDs 410 configured to emit two optical beams 415 that are spatially separated. If the assembled units 210 and framework 310 have no power, then mirrors or retroreflectors 430 can be mounted on the framework 310 or assembled units 210, as depicted in FIG. 4C. The optical beams 415 emitted from laser diodes or LEDs can be reflected from the assembled units 210 or framework 310 and detected by position-sensitive detectors 420 as the lifting head 450 lowers the assembled unit(s) 210 into the test tank 207. The positions of the returned optical beams 415 from the framework 310 or assembled units 210 on the position-sensitive detectors 420 can indicate the degree of alignment of the assembled units with respect to their installed locations and provide feedback signals to the lifting apparatus. The use of at least one planar mirror 435 on the framework 310 or assembled units 210 and arrangement of the laser diode or LED 410 such that the return optical beam 416 is reflected at an angle with respect to the incident optical beam 415 can indicate a distance of the assembled unit 210 from its installed location. An advantage of using laser diodes or LEDs and positionsensitive detectors is that they can be compact, operate immersed in the liquid environment, provide precise alignment (to less than ± 0.3 mm), and avoid the use of potentially more complex image-processing algorithms. In some implementations, the lifting head 450 and optical apparatus can provide sufficient alignment such that a framework 310 is not needed or used in the test tank 207 while multiple assembled units 210 are loaded at the same time.

[0042] In some implementations, the system 460 can include a controller, such as the controller 102 or 104 of FIG. 1. The controller monitors and controls some or all aspects of system operation for the system 460, including monitoring and controlling testing of the assembled units 210.4. Example Lifting Apparatus

[0043] FIG. 5 depicts an example of lifting apparatus 500 that can be used to lift assembled units 210 and place them into the test tank 207 in an aligned configuration such that electrical connectors on the assembled units 210 align and mate to mating electrical connectors in the test tank 207. The lifting apparatus 500 can be a wheeled vehicle that can maneuver and travel around a factory floor, for example. In some implementations, the lifting apparatus can travel on tracks or rails installed in the factory. The lifting apparatus 500 can include a turret 510that can rotate (in an x-y plane in the drawing), an articulating lifting arm 520, and a lifting head 450. The articulating lifting arm 520 and lifting head 450 can operate with hydraulic rams and / or stepper motors and gears. The rotation of the turret 510 can be up to 90 degrees or more such that the lifting arm 520 and lifting head 450 can pick assembled units 210 from the factory floor or cart at a first location and place the units into the test tank 207 at a second location. The lifting apparatus 500 can be operated manually in some cases and / or include robotic / automated operation. For example, at least alignment of the assembled units 210 when loaded into the test tank 207 can be fully automatic using feedback signals.

[0044] In some cases, the lifting head 450 can be configured to provide motion in multiple degrees of freedom without movement of the lifting arm 520. For example, the lifting head 450 can provide fine motion control in at least the x and y directions while the lifting arm is stationary so that the assembled units 210 can be precisely positioned in the x and y directions to align electrical connectors on the assembled units with mating electrical connectors in the test tank 207. In some implementations, the lifting head 450 can further provide fine motion control in the z direction without movement of the lifting arm 520 to finally lower the assembled units 210 into electrical contact with mating electrical connectors in the tank. For example, the lifting apparatus 500 can travel alongside the test tank 207, and the turret 510 and lifting arm 520 can be used to lift and place one or more assembled units 210 into the test tank 207 in a near final location (e.g., within about 3 cm from a final location). Then fine motion control on the lifting head 450 can be used to align and place the assembled unit(s) 210 in the final location where electrical connectors on the assembled unit(s) 210 contact and connect with mating electrical connectors in the test tank 207. In some implementations, the lifting head 450 can further provide fine rotation control about one or more axes without movement of the lifting arm 520. For example, the lifting head may be able to rotate assembled units 210 in an x-y plane, in a z-y plane, and / or in a z-x plane prior to lowering the assembled unit(s) 210 into a final location. The lifting apparatus 500, lifting arm 520, and lifting head 450 can be sized and rated to lift at least 200 pounds. In some cases, the lifting apparatus 500, lifting arm 520, and lifting head 450 can be sized and rated to lift up to 2500 pounds.

[0045] A testing facility can have one or more lifting apparatuses 500. In some cases, there can be one lifting apparatus 500 stationed beside each test tank 207 in the facility, and there can be up to six or more test tanks 207 at the testing facility. In some implementations, the lifting apparatus 500 can be shared by two or more test tanks 207.5. Testing Process

[0046] FIG. 6 is a flow chart of a testing process 600 that can be performed when testing operation of assembled units 210 in an immersion-cooling system 160 in a manufacturing facility. The testing process 600 can include fewer or more acts than those depicted in FIG. 6. As part of an initial set-up, the testing process 600 can include installing a test tank 207 in a manufacturing or test facility. After installation, the test tank 207 can be filled (act 602) with coolant liquid 164 to or below an operating level used during normal operation of assembled units 210 when installed in the tank. The level of the coolant liquid 164 during the initial set-up can be below the normal level to allow for installation of the assembled units 210 and their corresponding volume displacement of the coolant liquid. Without assembled units in the tank 207, the access door 212 on the tank can be closed and the immersion cooling system 160 checked for liquid and vapor leaks (act 604) of the coolant liquid. Liquid leaks can be visibly checked at all tank seals located below the level of the coolant liquid 164. To check for vapor leaks, the headspace in the tank can be pressurized and valved off from the pressurizing source. The pressure in the tank can then be monitored for a quick (e.g., several seconds or less) return to ambient pressure, which would indicate a vapor leak in the system.

[0047] If the tank is well sealed such that there are no or negligible liquid and vapor leaks, then a system test can be performed (act 608) on the immersion cooling system 160. The cooling system test can comprise checking the operational statuses of sensors (e.g., temperature and pressure) deployed in the system. The cooling system test can further comprise checking / replacing fluid filters used to filter coolant liquid in the tank. The cooling system test may further include checking / replacing a desiccant in the tank that is used to remove water vapor from the headspace in the tank. The cooling system test can further include checking operation of the chiller 180 that circulates heat-exchange fluid through the condensing pipes 270. The cooling-system test may further include running a self-diagnostic test of the cooling system’s controller. The result of the cooling system test can verify that the test tank 207 is ready for use to test a plurality of assembled units 210.

[0048] For each run during which assembled units 210 are tested, the testing process 600 can comprise creating and recording a run log (act 612) that stores information collected about each assembled unit 210 during the test process 600. The information collected can include, but is not limited to, each unit’s identifier (e.g., a serial number), results of functionality tests, temperatures of components reached during the testing process 600, processing speed reached, power consumption, etc. Additionally, video data can be recorded of loading, unloading, and internal tank operation during the testing process 600. Monitoring (act 615) of the run can be initiated prior to loading assembled units 210 (act 620) into the test tank 207.

[0049] The assembled units 210 can be moved from a staging area into position beside the test tank prior to loading. According to some implementations, the assembled units 210 can be arranged in groups beside the test tank, where the number of assembled units 210 in a group (e.g., two or more) is the number that will be lifted and installed in the test tank 207 at the same time. The assembled units 210 in a group can be aligned beside the tank in a first aligned configuration with respect to each other that resembles a second aligned configuration that the units in the group will have when installed in their respective installed locations in the test tank 207. The first aligned configuration can resemble the second aligned configuration when all units in the group are within ± 10 mm and ± 2 degrees of their positions and orientations in the second aligned configuration, for example. Such alignment outside the tank can allow the lifting head 450 of the lifting apparatus 500 to grasp all the assembled units 210 in the group in such a way that they are nearly in the aligned configuration of their installed locations. In some implementations, the lifting head 450 can guide the assembled units 210 into an intermediate aligned configuration that is closer to the second aligned configuration.

[0050] When the assembled units 210 are beside the test tank 207, the access door 212 of the tank can be opened (act 618). Opening the access door 212 at the last possible moment can reduce evaporative loss of coolant liquid vapor from the tank 207. Because the coolant liquid 164 can be expensive, measures can be taken to reduce the loss of coolant liquid 164 and coolant liquid vapor from the test tank 207. For example, a collar can be placed around the tank’s opening 220 to make it more difficult for coolant liquid vapor to spill over edges of the tank when loading and unloading assembled units 210.

[0051] The lifting apparatus 500 can then grasp one or a group of assembled units 210, lift the unit(s) over the tank’s opening 220, and load the unit(s) into the tank (act 620). The assembled units 210 can be aligned and guided into their installed locations as described above. When the assembled units arrive at the bottom of the test tank 207 and are aligned to their respective installed locations, electrical power connection to the busbar 230 can be automatically made by power pins 217 that mate with connectors (e.g., sockets) in the busbar 230. Cabling inside the tank (e.g., ethernet and / or fiber-optic cabling) can be connected to each assembled unit. Once all units are installed in the test tank 207, the level of the coolant liquid 164 can be checked (act 625) and topped off or drained if necessary to achieve an appropriate operating level for which at least a portion of each assembled unit is submerged.

[0052] The tank’s access door 212 can then be closed and power cycled on the assembled units 210 (act 630). The cycling of power can verify that all assembled units have valid connections, can be powered on and off, and can be communicated with. Once valid connections and communications are verified for a majority of the installed assembled units, each responsiveassembled unit 210 can undergo functional testing (act 640). A functional test can comprise booting up the assembled unit and performing normal operational functions (such as data communications, data processing, application execution, memory read and store operations, hardware and firmware configuration checks, etc.). After functional tests are complete, each responsive assembled unit 210 can be subjected to further full-capacity testing (act 650) such as run-out testing and stress testing. Full capacity testing can emulate normal and maximumcondition operation of the assembled units 210. During full-capacity testing, components in the assembled unit 210 may reach their highest operating temperatures that they would experience under maximum or extreme operating conditions in a real-world deployment. The functional tests and the full-capacity testing can be performed with open-source test diagnostics, proprietary test diagnostics, or combinations thereof. Test wrappers can be employed and the testing and collection of the test results and log files performed in an automated fashion. In some implementations, functional testing is repeated after the run-out and stress testing.

[0053] After completing tests, the assembled units 210 can be powered down (act 660). The test tank 207 can be placed in a standby status (act 670) and idle time expended to allow coolant liquid 164 within the tank to cool and reduce vapor pressure within the tank. When pressure within the tank reduces to ambient level, the access door 212 can be opened, cabling disconnected, and the assembled units unloaded from the tank with the lifting apparatus 500 (act 680). After or prior to unloading the assembled units 210, the run log for the batch can be completed and saved (act 690) for subsequent review. Information from the run log may be provided with each assembled unit that is marketed as proof of performance.

[0054] The testing process 600 can then return to running a cooling system check (act 608) in preparation for the next batch of assembled units 210. The cooling system check can further include inspection for guide rail, busbar, and electrical connector damage. Such inspection may be done robotically by a video camera and using image processing. In some implementations, the total run time for a batch of assembled units can be from 30 hours to 50 hours.

[0055] In some cases, a portion or all of the testing process 600 can be monitored and / or controlled remotely, via the internet. A remote operator can be located in another room, another state, or another country. Access to the test facility can be restricted to authorized personnel who may be required to authenticate their identity when entering the test facility or implementing any portion of the test process 600.

[0056] FIGS. 7A, and 7B illustrates an example testing system 260 including a testing tank 207. Power is provided to the testing tank 207 via redundant power racks 211 that couple to a busbar configuration (See busbar 230 in FIG. 4B) in the testing tank 207. As illustrated inFIG. 7A, the power racks 211 are positioned on the right and left sides of the testing tank 207. The testing tank 207 has a lid 212 and a bellows structure 219 to reduce fluid loss. The bellows structure 219 includes a bellow 221 and vents 223, and manages vapor. The testing system 260 as illustrated includes couplings 225 to facilitate access facility utilities, such as a facility water source, waste treatment system, etc.

[0057] FIG. 8 illustrates an example floor layout 800 in which a testing system, such as the testing system 260 of FIGS. 7A and 7B, can be employed in a manufacturing or operational facility. The floor layout 800 includes a staging area 832 and an immersion-cooling system area 834. Assembled units 210 are staged in the staging area 832 for loading into immersion-cooling systems 260 positioned in the immersion-cooling system area 834. The lifting apparatus 500 lifts assembled units 210 in the staging area 832 and positions and loads the assembled units into immersion-cooling systems 260 positioned in the immersion-cooling system area 834. As illustrated, the lifting apparatus 500 includes one or more robotic units 536 and a system of rollers 538. The robotic units 536 can carry two assembled units 210 at a time. The roller system 538 can be used together with the robotic units 536 to move assembled units 210 from the staging area 832 to respective positions in immersion-cooling systems 260 positioned in the immersion-cooling system area 834.

[0058] FIG. 9 is a flow chart of a failure analysis process 900 that can be performed in response to detecting a failure associated with one or more assembled units 210 in an operational system. The failure analysis process 900 can include fewer or more acts than those depicted in FIG. 9, and may employ all or portions of process 600 of FIG. 6. An indication of a failure is received (act 902) and a failure type associated with the indication is determined (act 904). When it is not determined that the failure type is associated with a board failure, the process 900 can perform other failure processing (act 906). When it is determined that the failure type is associated with a board failure, the process 900 unloads the server s / assembled units 210 from the operational system (act 908). The servers / assembled units are moved to a testing tank (act 910) and tests are run on the servers / assembled units (act 912). For example, the servers / assembled units 210 may be moved to a tank 207 and tested using the process 600 of FIG. 6.

[0059] Based on the results of the tests, one or more component or commodity failures are identified (act 914) and remedial action taken (act 916), such as replacing or reseating a failed component or commodity, this may include unloading the servers / assembled units from the tank 207 before taking remedial action and and reloading the servers (act 918) after taking the remedial action. Tests on the servers / assembled units are performed again (act 920). If the tests results are consistent with proper operation and within specifications, the servers / assembledunits are unloaded from the test tank 207 (act 922) and moved to a production tank (act 924), such as a tank 107 of FIG. 1.

[0060] FIG. 10 is a flow chart of a failure analysis process 1000 that can be performed in response to detecting a failure associated with one or more boards in an operational system. The failure analysis process 1000 can include fewer or more acts than those depicted in FIG. 10, and may employ all or portions of process 600 of FIG. 6, or all or portions of process 900 of FIG. 9. An indication of a failure is received (act 1002) and a failure type associated with the indication is determined (act 1004). When it is not determined that the failure type is associated with a board failure, the process 1000 can perform other failure processing (act 1006). When it is determined that the failure type is associated with a board failure, the process 1000 unloads the servers / assembled units 210 from the operation system (act 1008).

[0061] The servers / assembled units are moved to a testing tank (act 1010) and failure analysis profile testing conducted (act 1032). In response to the testing indicating a processor or board failure (act 1034), the server / assembled units are moved to a board failure station (act 1036), which can include disassembly of the assembled units. A board failure stress test is performed (act 1038). Based on the results of the board failure stress test, one or more component or commodity failures are identified and remedial action taken (act 1040), such as replacing or reseating a failed component or commodity. The boards of the server / assembled units are configured (act 1042). In some implementations, the boards are reassembling (act 1044) into assembled units. Tests on the servers / assembled units are performed again (act 1046), which can include performing the process 600 of FIG. 6. If the tests results are consistent with proper operation and within specifications, the servers / assembled units are unloaded from the test tank 207 and moved to a production tank (act 1048), such as a tank 107 of FIG. 1.

[0062] In one aspect, a process to test a plurality of assembled units for operation in an immersion-cooling system is provided. Two or more assembled units of the plurality of assembled units are arranged in a first aligned configuration beside a test tank. The first aligned configuration resembles a second aligned configuration that the two or more assembled units will have when installed in the test tank at respective installed locations. The two or more assembled units are loaded into the test tank and into coolant liquid located in the test tank such that at least a portion of each assembled unit of the two or more assembled units is submerged in the coolant liquid. The two or more assembled units are guided into the second aligned configuration at the respective installed locations. For each of the two or more assembled units, electrical power connections are automatically connected to a power busbar in the tank when the two or more assembled units are installed at the respective locations.

[0063] In some implementations, the guiding comprises guiding the two or more assembled units with guide rails to the respective installed locations. In some implementations, the guide rails are oriented vertically in a framework that is installed permanently or temporarily in the test tank.

[0064] In some implementations, the guiding comprises guiding the two or more assembled units with alignment pins and alignment receptacles to the respective installed locations. In some implementations, the alignment pins extend from a bottom of each assembled unit of the two or more assembled units, and the alignment receptacles are located at a base of the test tank.

[0065] In some implementations, the guiding comprises detecting a position of at least one optical beam reflected from a reflector mounted on an assembled unit of the two or more assembled units.

[0066] In some implementation, the process comprises placing, with a lifting head of a lifting apparatus, the two or more assembled units over the busbar such that the two or more units are aligned to within ± 5 mm of the respective installed locations.

[0067] In some implementations, the two or more assembled units each comprise an internet server weighing at least 150 pounds.

[0068] In some implementations, the process comprises: closing a lid on the test tank; cycling power to at least one of the assembled units of the two or more assembled units; and establishing communications with at least one of the assembled units of the two or more assembled units. In some implementations, the process comprises running functional tests for at least one of the assembled units of the two or more assembled units.

[0069] In one aspect, a non-transitory computer-readable medium’s contents cause a processing system to control a process. In the process, two or more assembled units of a plurality of assembled units are arranged in a first aligned configuration beside a test tank. The first aligned configuration resembles a second aligned configuration that the two or more assembled units will have when installed in the test tank at respective installed locations. The two or more assembled units are loaded into the test tank and into coolant liquid located in the test tank such that at least a portion of each assembled unit of the two or more assembled units is submerged in the coolant liquid. The two or more assembled units are guided into the second aligned configuration at the respective installed locations. For each of the two or more assembled units, electrical power connections are automatically connected to a power busbar in the tank when the two or more assembled units are installed at the respective locations.

[0070] In some implementations, the guiding comprises detecting a position of at least one optical beam reflected from a reflector mounted on an assembled unit of the two or more assembled units.

[0071] In some implementations, the process comprises placing, with a lifting head of a lifting apparatus, the two or more assembled units over the busbar such that the two or more units are aligned to within ± 5 mm of the respective installed locations.

[0072] In some implementations, the process comprises closing a lid on the test tank; cycling power to at least one of the assembled units of the two or more assembled units; and establishing communications with at least one of the assembled units of the two or more assembled units. In some implementations, the process comprises running functional tests for at least one of the assembled units of the two or more assembled units.

[0073] In some implementations, the contents comprise instructions executed by processing circuitry of the processing system to control the process.

[0074] In one aspect, a system comprises an immersion-cooling tank, a lifting apparatus, and control circuitry coupled to the immersion-cooling tank and to the lifting apparatus. The immersion-cooling tank, in operation, contains coolant liquid. The control circuitry, in operation, executes instructions to: load two or more assembled units of a plurality of assembled units into coolant liquid in the immersion-cooling tank, wherein the two or more assembled units are arranged in a first aligned configuration beside the immersion-cooling tank which resembles a second aligned configuration that the two or more assembled units will have when installed in the immersion-cooling tank at respective installed locations; guide the two or more assembled units into the second aligned configuration at the respective installed locations; and automatically connect, for each of the two or more assembled units, electrical power connections to a power busbar in the immersion-coolant tank when the two or more assembled units are installed at the respective locations.

[0075] In some implementations, the guiding comprises guiding the two or more assembled units with guide rails to the respective installed locations. In some implementations, the guide rails are oriented vertically in a framework that is installed permanently or temporarily in the immersion-cooling tank.

[0076] In some implementations, the control circuitry, in operation, executes instructions to run functional tests on at least one of the assembled units of the two or more assembled units.

[0077] While various embodiments have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is deemed to be within thescope of the embodiments described herein. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the teachings is / are used. Those skilled in the art will recognize or be able to ascertain, using no more than routine experimentation, many equivalents to the specific embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that embodiments may be practiced otherwise than as specifically described. Embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

[0078] Also, various disclosed concepts may be embodied as one or more methods, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

[0079] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0080] Unless stated otherwise, the terms “approximately” and “about” are used to mean within ± 20% of a target (e.g., dimension or orientation) in some embodiments, within ± 10% of a target in some embodiments, within ± 5% of a target in some embodiments, and yet within ± 2% of a target in some embodiments. The terms “approximately” and “about” can include the target. The term “essentially” is used to mean within ± 3% of a target.

[0081] The indefinite articles “a” and “an,” as used herein, unless clearly indicated to the contrary, should be understood to mean “at least one.”

[0082] The phrase “and / or,” as used herein, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and / or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and / or” clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to “A and / or B,” when used in conjunction with open-ended language suchas “comprising” can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); etc.

[0083] As used herein, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of’ or “exactly one of’ or “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives i.e., “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.” “Consisting essentially of,” shall have its ordinary meaning as used in the field of patent law.

[0084] As used herein, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, “at least one of A and B” (or, equivalently, “at least one of A or B,” or, equivalently “at least one of A and / or B”) can refer, in one embodiment, to at least one, optionally including more than one, A, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in yet another embodiment, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other elements); etc.

[0085] In the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases “consisting of’ and “consisting essentially of’ shall be closed or semi-closed transitional phrases, respectively.

[0086] The various embodiments described above can be combined to provide further embodiments. Aspects of the embodiments can be modified, if necessary to employ concepts of the various patents, applications and publications to provide yet further embodiments.

[0087] These and other changes can be made to the embodiments in light of the abovedetailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.

Claims

CLAIMS1. A process to test a plurality of assembled units for operation in an immersioncooling system, the process comprising: arranging two or more assembled units of the plurality of assembled units in a first aligned configuration beside a test tank, wherein the first aligned configuration resembles a second aligned configuration that the two or more assembled units will have when installed in the test tank at respective installed locations; loading the two or more assembled units into the test tank and into coolant liquid located in the test tank such that at least a portion of each assembled unit of the two or more assembled units is submerged in the coolant liquid; guiding the two or more assembled units into the second aligned configuration at the respective installed locations; and automatically connecting, for each of the two or more assembled units, electrical power connections to a power busbar in the tank when the two or more assembled units are installed at the respective locations.

2. The process of claim 1, wherein the guiding comprises guiding the two or more assembled units with guide rails to the respective installed locations.

3. The process of claim 2, wherein the guide rails are oriented vertically in a framework that is installed permanently or temporarily in the test tank.

4. The process of claim 1, wherein the guiding comprises guiding the two or more assembled units with alignment pins and alignment receptacles to the respective installed locations.

5. The process of claim 4, wherein: the alignment pins extend from a bottom of each assembled unit of the two or more assembled units, and the alignment receptacles are located at a base of the test tank.

6. The process of claim 1, wherein the guiding comprises detecting a position of at least one optical beam reflected from a reflector mounted on an assembled unit of the two or more assembled units.

7. The process of claim 1, comprising: placing, with a lifting head of a lifting apparatus, the two or more assembled units over the busbar such that the two or more units are aligned to within ± 5 mm of the respective installed locations.

8. The process of claim 1, wherein the two or more assembled units each comprise an internet server weighing at least 150 pounds.

9. The process of claim 1, comprising: closing a lid on the test tank; cycling power to at least one of the assembled units of the two or more assembled units; and establishing communications with at least one of the assembled units of the two or more assembled units.

10. The process of claim 9, comprising running functional tests for at least one of the assembled units of the two or more assembled units.

11. A non-transitory computer-readable medium having contents which cause a processing system to control a process, the process comprising: arranging two or more assembled units of a plurality of assembled units in a first aligned configuration beside a test tank, wherein the first aligned configuration resembles a second aligned configuration that the two or more assembled units will have when installed in the test tank at respective installed locations; loading the two or more assembled units into the test tank and into coolant liquid located in the test tank such that at least a portion of each assembled unit of the two or more assembled units is submerged in the coolant liquid; guiding the two or more assembled units into the second aligned configuration at the respective installed locations; and automatically connecting, for each of the two or more assembled units, electrical power connections to a power busbar in the tank when the two or more assembled units are installed at the respective locations.

12. The non-transitory computer-readable medium of claim 11, wherein the guiding comprises detecting a position of at least one optical beam reflected from a reflector mounted on an assembled unit of the two or more assembled units.

13. The non-transitory computer-readable medium of claim 11, wherein the process comprises: placing, with a lifting head of a lifting apparatus, the two or more assembled units over the busbar such that the two or more units are aligned to within ± 5 mm of the respective installed locations.

14. The non-transitory computer-readable medium of claim 11, wherein the process comprises: closing a lid on the test tank; cycling power to at least one of the assembled units of the two or more assembled units; and establishing communications with at least one of the assembled units of the two or more assembled units.

15. The non-transitory computer-readable medium of claim 14, wherein the process comprises: running functional tests for at least one of the assembled units of the two or more assembled units.

16. The non-transitory computer-readable medium of claim 11 wherein the contents comprise instructions executed by processing circuitry of the processing system to control the process.

17. A system, comprising; an immersion-cooling tank, which, in operation, contains coolant liquid; a lifting apparatus; and control circuitry coupled to the immersion-cooling tank and to the lifting apparatus, wherein the control circuitry, in operation, executes instructions to: load two or more assembled units of a plurality of assembled units into coolant liquid in the immersion-cooling tank, wherein the two or more assembled units are arranged in a first aligned configuration beside the immersion-cooling tank which resembles a second alignedconfiguration that the two or more assembled units will have when installed in the immersioncooling tank at respective installed locations; guide the two or more assembled units into the second aligned configuration at the respective installed locations; and automatically connect, for each of the two or more assembled units, electrical power connections to a power busbar in the immersion-coolant tank when the two or more assembled units are installed at the respective locations.

18. The system of claim 17, wherein the guiding comprises guiding the two or more assembled units with guide rails to the respective installed locations.

19. The system of claim 18, wherein the guide rails are oriented vertically in a framework that is installed permanently or temporarily in the immersion-cooling tank.

20. The system of claim 17, wherein the control circuitry, in operation, executes instructions to: run functional tests on at least one of the assembled units of the two or more assembled units.

Images