DYNAMIC PCIe LINK RETRAINING IN IMMERSION FLUID

Abstract

An immersion cooling environment is described herein, including but not limited to immersion-cooled racks in a tank containing a cooling fluid to cool computing systems including high-density, computation intensive, and systems implementing artificial intelligence (AI) computations. These systems can employ a plurality of computing components using PCIe interface for establishing communication channels for transmitting data between the plurality of components. PCIe link monitoring and retraining is employed to resolve PCIe failures and establish efficient communication channels. High-speed interfaces such as the PCIe interface can fail and require robust training for efficient communication. Thus, PCIe link retraining mechanisms can periodically retrain the PCIe links to ensure a robust PCIe interface. In some aspects, PCIe link retraining is executed periodically to ensure robust PCIe interface. This PCIe link retraining can result in optimal link equalization to ensure robust and efficient communication using the PCIe interface. Systems and methods for monitoring strain applied to a cable are disclosed. One or more cables that are part of an immersion cooling system and connected to corresponding one or more computing components such as a server include a housing and transmission medium. A strain detector is disposed on the housing of the cable and communicatively coupled to a processor or other controller. The strain detector experiences a strain corresponding to the cable and a resistance of the strain detector changes based on the change in strain and the resistivity of the strain detector. The processor correlates a cable strain based on the change in resistance of the strain detector.

Description

DYNAMIC PCIe LINK RETRAINING IN IMMERSION FLUIDCROSS-REFERENCE TO RELATED APPLICATION

[0001] This Application claims the benefit of provisional U.S. Application No. 63 / 573,309, filed on April 2, 2024, and entitled “DYNAMIC PCIe LINK RETRAINING IN IMMERSION FLUID” and U.S. Application No. 63 / 637,574, filed on April 23, 2024, and entitled “ACTIVE SENSING FOR FIBER OPTIC PROTECTION” which are hereby incorporated by reference in their entireties.

[0002] In cases where the present application conflicts with a document incorporated by reference, the present application controls.BACKGROUND

[0003] Section 1: The invention relates generally to PCIe link training and dynamic retraining of at least one PCIe link in Al server systems used in dynamic environments of immersion fluid.

[0004] Peripheral Component Interconnect Express (PCIe) is a standard used to enable high-speed serial communication between the CPU and its peripheral components. PCIe is often used with graphics processing units (GPUs), solid-state drives (SSD) to send and receive data with the CPU. Al server systems rely on PCIe links to communicate between CPUs, between CPUs and GPUs, solid-state drives, PCIe switches and other system components. As such, the PCIe interface requires bandwidth to efficiently transmit data.

[0005] Implementing computing systems and computations can require cooling, for example, immersion cooling of system components using a tank containing a liquid (i.e. coolant liquid) used to maintain appropriate temperatures for system components. These computing environments can include cloud computing, systems implementing machine learning tasks, or systems implementing high-power computations, including run-time computations for processes associated with artificial intelligence and inference. These systems further implement and rely on high-speed interfaces such as PCIe devices. As such, robust functioning of PCIe components is integral in carrying out tasks in such computing environments.

[0006] The implementation of PCIe functionality involves PCIe links used when system devices are powered on, and further involves PCIe link training, which consists of receiverdetection, polling, and configuration for link equalization. It is integral for high-speed interfaces such as PCIe devices to go through robust link training during power-on state, to efficiently carry out tasks in a computing environment. With respect to immersion cooling systems, PCIe link training and dynamic PCIe link retraining may be necessary based on the monitored properties of an immersion cooling environment, including but not limited to, the cooling fluid and characteristic associated with the computing components in the system.

[0007] PCIe failures can occur when, for example, system components (i.e. servers, GPU, CPU) communicating with the PCIe undergo fluctuations as a result of, for example, changes in state of the immersion fluid, state of system components, state of system components from steady state to maximum power state. Fluctuations within an immersion fluid system can result in PCIe failure. Such failures can include signal loss during transmission, based on factors such as channel quality and transmission rates. The present invention provides a solution addressing PCIe failure in immersion cooling environments.

[0008] Section 2: As feature sizes and transistor sizes have decreased for computing hardware such as integrated circuits (ICs) including chips and semiconductor dies, the amount of heat generated by a single chip, such as a microprocessor, has increased. Computing hardware that has traditionally been air cooled has evolved to levels of power consumption requiring more heat dissipation than can be provided by air alone. In some cases, immersion cooling of ICs in a tank containing a coolant fluid is employed to maintain ICs at appropriate operating temperatures.

[0009] One type of immersion cooling is two-phase immersion cooling, in which heat from a semiconductor die is high enough to boil the coolant fluid. The boiling creates a coolant fluid vapor in the tank, which is condensed by cooling coils back to liquid form. Heat from the semiconductor dies can then be sunk into the liquid-to-gas and gas-to-liquid phase transitions of the coolant fluid with the result that the semiconductor dies are kept at an acceptable temperature.

[0010] Improvements to computational performance require associated improvements to computing infrastructure such as data transfer pathways. Fiber optic cables are frequently used for communicative coupling in computational systems due to their high data transfer rates and low latencies. However, these fiber optic cables can be fragile and easily damaged. Further, they may be costly and time consuming to repair.SUMMARY

[0011] Section 1: PCIe devices may go through a link equalization process to establish stable connection among the devices of a computing environment (i.e. immersion cooling environment). A training mechanism process is implemented, during power-up of the system with the link training seeking to establish link equalization. The one or more PCIe devices begin link training, leading to the PCIe devices connecting from an endpoint to the root complex. The link training process includes configuration of one or more PCIe data rates. Upon configuration of the PCIe devices during the initial training mechanism process, the PCIe links can be assessed to be in a number of states, including but not limited to a normal operation state, low-power states or recovery state. In general, normal operation state is where data and packets are sent and received, with the root complex and endpoint devices successfully communicating between each other. In systems in immersion cooling environments, once a threshold data rate has been reached, signal distortion is probable, leading to error rates impacting communication performance.

[0012] Over time, the training mechanism can become unreliable because of a number of factors affecting an immersion cooling environment, including but not limited to, degradation and contamination. In fact, contamination in systems within an immersion cooling environment can often lead to severe degradation of the PCIe links. In such cases, PCIe failures can occur, affecting the functionality of the entire system (i.e. immersion fluid system) and often requiring reboot or loss of data. A mechanism for retraining of the PCIe link would avoid the PCIe failure by establishing new parameters and / or new margins associated with the PCIe links. This results in efficient communication, avoids errors, and in general provides better functionality for systems relying on the communication channels of a PCIe interface.

[0013] The present disclosure is directed to providing a solution through PCIe link monitoring and retraining in dynamic environments of immersion fluid.

[0014] The present disclosure includes communication between components (i.e. CPUs and GPUs) within an Al server architecture which uses PCIe devices. High speed interfaces such as the PCIe interface can fail and require robust training for efficient communication. Thus, PCIe link retraining mechanisms can periodically retrain the PCIe links to ensure a robust PCIe interface.

[0015] In some aspects, the techniques described herein relate to a non-transitory computer-readable medium containing instructions thereon, the instructions configuring at least one processor to: determine, for a system, at least a first state of the system in an immersion cooling environment, wherein the system includes at least two computing components; determine, based at least in part on the first state, a link between the at least two computing components, the link established through at least one communication channel; determine at least one data rate value associated with the at least one communication channel; transmit, based at least in part on the at least one data rate value, data between the at least two computing components using the at least one communication channel; adjust, based at least in part on a second state of the system in the immersion cooling environment, at least one predetermined value associated with the link; and transmit, based on the adjustment, data between the at least two computing components.

[0016] In some aspects, the techniques described herein relate to a method implemented by at least one processor for configuring a computing system, the method including: determining, for the computing system, at least a first state of the computing system in an immersion cooling environment, wherein the computing system includes at least two computing components; determining, based at least in part on the first state, a link between the at least two computing components, the link established through at least one communication channel; determining at least one data rate value associated with the at least one communication channel; transmitting, based at least in part on the at least one data rate value, data between the at least two computing components using the at least one communication channel; adjusting, based at least in part on a second state of the computing system in the immersion cooling environment, at least one predetermined value associated with the link; and transmitting, based on the adjustment, data between the at least two computing components.

[0017] In some aspects, PCIe link retraining is executed periodically to ensure robust PCIe interface. This PCIe link retraining can result in optimal link equalization and / or other forms to ensure robust and efficient communication using the PCIe interface.

[0018] In some aspects, the system includes the CPU, GPU and an Al server architecture in which the PCIe layer is implemented. The PCIe components include devices directed to PCI root board and endpoints.

[0019] In some aspects, the characteristics of the PCIe channels are assessed to determine the rate at which data is to be transmitted. The PCIe data rate, along with compatibility with PCIe devices determines that data is received and transmitted with each device able to interpret the received data and respond accordingly.

[0020] In some aspects, the root port of the PCIe interface is the CPU and the endpoint of the PCIe interface is the GPU device.

[0021] In some aspects, equalization is achieved at both the transmitter and the receiver in a high-speed serial-data channel, based on adjustments to the data rate. In some aspects, equalization is achieved after training the link, with periodic adjustments for optimal equalization at lane rates. The optimal equalization schema considers channel characteristics and non-linear equalization techniques, including but not limited to, the Decision Feedback Equalizer (DFE). Additional techniques for efficient configuration of high-speed serial links include using continuous time linear equalization (CTLE). CTLE can contribute to better signal performance on the receiver side, through boosting of higher frequencies at the receiver.

[0022] In some aspects, retraining the PCIe link dynamically is based on equalization parameter negotiations. Dynamic equalization through dynamic retraining of the PCIe link allows for flexibility and efficiency when adjusting equalization settings for optimal link equalization.

[0023] In some aspects, the techniques described herein relate to an apparatus for monitoring immersion cooling fluid. The system includes a controller and a sensor operably connected to the controller. Characteristics associated with the immersion cooling fluid are determined to make adjustments to the dynamic retraining PCIe link mechanism.

[0024] In some aspects, the techniques described herein relate to a method for monitoring the immersion cooling environment to trigger the dynamic PCIe retraining mechanism. The triggering can include, but is not limited to, factors such as signal loss, server failure, reboot of the system, system errors, changes associated with the tank or immersion fluid and / or changes in immersion fluid state.

[0025] In some aspects, the techniques described herein applies to standard PCIe interfaces, and other computing environments, including server systems and generalcomputing environments not related to cooling systems. Additionally, the dynamic PCIe link training can be implemented in generic computing systems with or without server components.

[0026] In another aspect, the present application relates to one or more non-transitory computer readable media encoded with software comprising computer executable instructions and, when executed by a processor of a computer, operable to carry out the method according to various embodiments described herein.

[0027] All combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are part of the inventive subject matter disclosed herein. The terminology used herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.

[0028] Section 2: State-of-the-art computing systems process and transfer high volumes of data with extremely low latency and transmission durations, and fiber optic cables are often used to provide infrastructure for high bandwidth, low latency data transfer. In particular, immersion cooling systems provide additional cooling required to dispose of waste heat that scales as a function of computational speed. This enables higher efficiency performance for processes such as artificial intelligence (Al) training and inference, mathematical model calculations, graphics processing, and complex simulations such as finite element analysis (FEA).

[0029] Fiber optic cables used in high-performance computing systems such as immersion cooled computing systems may be made of relatively inflexible or brittle materials such as silica (glass), which may be subjected to relatively large bending radii without risk of damaging the fiber optic conduit. However, if a bend is too tight (e.g., too small of a radius), the fiber optic conduit in the cable may crack or otherwise be damaged. A damaged cable in an immersion cooling system is problematic and expensive to service and may be difficult to diagnose and fix. For example, a single damaged cable may introduce latency to a system that significantly erodes performance, but identifying which particular cable is damaged may be time consuming and difficult in a system that may have hundreds or thousands ofsuch cables. Such repairs may require ceasing operation of the relevant computing system while the repairs are made, which may cost hundreds of thousands of dollars in lost revenue.

[0030] The present technology is directed toward active sensing for detecting strain on fiber optic cables using strain gauges woven into a sheath material. A bend in the fiber optic cable may be detected as a change in the resistance of the strain gauge and may be received and processed by computing hardware. Detected instances of excess strain may cause one or more notifications to be sent to a technician, repair robot, or similar entity. This may prevent instances of damage and potential destruction of costly fiber optic cables.

[0031] In some aspects, the techniques described herein relate to a system for reducing strain on one or more cables, the system including: the one or more cables, each cable including a flexible housing surrounding a transmission medium; a strain detector disposed on the one or more cables; and a processor communicatively coupled to the strain detector; wherein the processor is configured to: measure a first resistance of the strain detector; and correlate a difference between the first resistance and a baseline resistance of the strain detector with a cable strain applied to at least one of the one or more cables.

[0032] In some aspects, the techniques described herein relate to a system, wherein: the strain detector includes a plurality of segments; and each cable of the one or more cables includes at least one segment of the plurality of segments.

[0033] In some aspects, the techniques described herein relate to a system, wherein each segment of the plurality of segments is communicatively coupled to the processor; and the processor is configured to measure a respective resistance of each segment of the plurality of segments and determine a cable strain applied to a portion of a respective cable on which each respective segment is disposed.

[0034] In some aspects, the techniques described herein relate to a system, wherein the strain detector is not disposed on an entire length of the one or more cables.

[0035] In some aspects, the techniques described herein relate to a system, wherein the processor is configured to measure at least one of the first resistance or the baseline resistance using a Wheatstone bridge.

[0036] In some aspects, the techniques described herein relate to a system, wherein the difference between the first resistance and the baseline resistance is between about 0.05% and about 0.5%.

[0037] In some aspects, the techniques described herein relate to a system, wherein the baseline resistance is about 2 kOhm and the difference between the first resistance and the baseline resistance is between about 1 Ohm and about 10 Ohm.

[0038] In some aspects, the techniques described herein relate to a system, wherein the strain detector includes a copper wire.

[0039] In some aspects, the techniques described herein relate to a system, wherein the copper wire includes a diameter of about 0.05 mm.

[0040] In some aspects, the techniques described herein relate to a system, further including an immersion cooling container at least partially filled with immersion cooling liquid; wherein the one or more cables are at least partially immersed in the immersion cooling liquid.

[0041] In some aspects, the techniques described herein relate to a system, wherein: the flexible housing includes a mesh; and the strain detector is woven into the mesh.

[0042] In some aspects, the techniques described herein relate to a system, further including: A radio-frequency identification (RFID) tag communicatively coupled to the processor and configured to transmit information about at least one of the one or more cables or the strain detector to the processor.

[0043] In some aspects, the techniques described herein relate to a system, wherein at least one of the one or more cables or the strain detector includes a color coding or a numbering associating the one or more cables with the strain detector.

[0044] In some aspects, the techniques described herein relate to a system, wherein the processor is further configured to transmit a notification to an entity in response to the cable strain applied to at least one of the one or more cables exceeding a threshold.

[0045] In some aspects, the techniques described herein relate to a system, wherein the entity includes at least one of a user, a system controller, or a facility controller.

[0046] In some aspects, the techniques described herein relate to a method for reducing strain on one or more cables, the method including: deforming one or more cables disposed within an immersion cooling system; measuring, by a processor, a first resistance of a strain detector disposed on the one or more cables; correlating, by the processor, a difference between the first resistance and a baseline resistance of the strain detector with a cable strainapplied to at least one of the one or more cables; and transmitting, by the processor, a notification to an entity in response to the cable strain applied to at least one of the one or more cables exceeding a threshold.

[0047] In some aspects, the techniques described herein relate to a method, wherein: the strain detector includes a plurality of segments; and each cable of the one or more cables includes at least one segment of the plurality of segments.

[0048] In some aspects, the techniques described herein relate to a method, wherein each segment of the plurality of segments is communicatively coupled to the processor; and the processor is configured to measure a respective resistance of each segment of the plurality of segments and determine a cable strain applied to a portion of a respective cable on which each respective segment is disposed.

[0049] In some aspects, the techniques described herein relate to a method, wherein the strain detector is not disposed on an entire length of the one or more cables.

[0050] In some aspects, the techniques described herein relate to a method, wherein the processor is configured to measure at least one of the first resistance or the baseline resistance using a Wheatstone bridge.

[0051] In some aspects, the techniques described herein relate to a method, wherein the difference between the first resistance and the baseline resistance is between about 0.05% and about 0.5%.

[0052] In some aspects, the techniques described herein relate to a method, wherein the baseline resistance is about 2 kOhm and the difference between the first resistance and the baseline resistance is between about 1 Ohm and about 10 Ohm.

[0053] In some aspects, the techniques described herein relate to a method, wherein the strain detector includes a copper wire.

[0054] In some aspects, the techniques described herein relate to a method, wherein the copper wire includes a diameter of about 0.05 mm. 25.

[0055] In some aspects, the techniques described herein relate to a method, wherein: the immersion cooling system further includes an immersion cooling container at least partially filled with immersion cooling liquid; and the one or more cables are at least partially immersed in the immersion cooling liquid.

[0056] In some aspects, the techniques described herein relate to a method, wherein: the one or more cables each include a flexible housing surrounding a transmission medium; the flexible housing includes a mesh; and the strain detector is woven into the mesh.

[0057] In some aspects, the techniques described herein relate to a method, further including: receiving, by the processor, information about at least one of the one or more cables or the strain detector from a radio -frequency identification (RFID) tag communicatively coupled to the processor.

[0058] In some aspects, the techniques described herein relate to a method, wherein at least one of the one or more cables or the strain detector includes a color coding or a numbering associating the one or more cables with the strain detector.

[0059] In some aspects, the techniques described herein relate to a method, further including: transmitting, by the processor, a notification to an entity in response to the cable strain applied to at least one of the one or more cables exceeding a threshold.

[0060] In some aspects, the techniques described herein relate to a method, wherein the entity includes at least one of a user, a system controller, or a facility controller.

[0061] All combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are part of the inventive subject matter disclosed herein. The terminology used herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The skilled artisan will understand that the drawings primarily are for illustrative purposes and are not intended to limit the scope of the inventive subject matter described herein. The drawings are not necessarily to scale; in some instances, various aspects of the inventive subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference characters generally refer to like features (e.g., functionally similar and / or structurally similar elements).

[0063] FIG. 1.1 illustrates an example of an immersion-cooling environment for computation in accordance with the present technology.

[0064] FIG. 1.2 illustrates a flowchart method for the immersion cooling environment in accordance with the present technology.

[0065] FIG. 1.3 depicts aspects of an immersion cooling system.

[0066] FIG. 2.1 A illustrates an immersion cooling system in accordance with the present invention.

[0067] FIG. 2. IB illustrates immersion cooling system in an exemplary state of maintenance.

[0068] FIG. 2.1C illustrates immersion cooling system further including a strain detector.

[0069] FIG. 2.2 depicts a Wheatstone bridge used to determine resistance and / or strain across a strain detector.

[0070] FIG. 2.3 depicts aspects of an immersion cooling system for dissipating heat from one or more heat-generating components such as semiconductor die packages via immersion cooling.DETAILED DESCRIPTION

[0071] Section 1: The present technology is directed toward a dynamic PCIe link retraining in an immersion fluid system and / or an immersion cooling environment. A training mechanism process is implemented, during power-up of a system with the link training seeking to establish link equalization. The training mechanism can over time become unreliable, because of a number of factors affecting an immersion fluid environment, including but not limited to, degradation and contamination. The present invention is directed to providing a solution through PCIe link monitoring and retraining in dynamic environments of immersion fluid.

[0072] FIG. 1.1 illustrates an example of an immersion-cooling environment 100 for computation in accordance with the present technology. System 100 includes numerous elements for purposes of illustration rather than limitation. System 100 may include the same, more, or fewer elements configured in the same or different manner in other implementations .

[0073] System 100 may be disposed within an immersion-cooled computing environment, where some or all the components of system 100 are immersed in cooling fluid. System components additionally include a number of storage devices and solid-state storage devices 101. System 100 can include, for example, PCI root complex 104, PCIe endpoints 107, PCIe switches 102, non-volatile memory express (NVME) solid state drives 101, data processing unit (DPU) 103, PCH 105, Cedar Modules 106 and NVLink Switches 108. DPU 103 may be an NVIDIA BlueField 3™ BMC software DPU and may control one or more operations of system 100. The system 100 includes a baseboard management controller (BMC) 113 which can be implemented in a variety of ways including as a Field Programmable Gate Array (‘FPGA’) 114, a Programmable Logic Chip (PLC), an Application Specific Integrated Circuit (ASIC), System-on-Chip (SOC) or any computing devices that includes discrete components such as a processing device, central processing unit, computer memory or various adapters.

[0074] In some implementations, NVRAM devices may be configured to receive, from the CPUs, data to be stored in the storage drives. In implementations, storage drives 101 may refer to any device configured to record data. The data communication links described herein are collectively illustrated by data communication links 109 and may be implemented through a PCIe interface. Controller 110 may include processing devices 104, which can include one or more general-purpose processing devices 104, such as a microprocessor, CPU or the like. The processing device may be connected to the GPUs 107 via cedar modules 106 and data communication links, which may be embodied as a PCIe link. Switches 102 are coupled to processing devices and PCH 105 via a data communications link.

[0075] Switches 102 may create multiple endpoints out of a single endpoint, thereby enabling multiple devices to share an endpoint. The switch may, for example, be a PCIe switch and presents multiple PCIe connections to the devices in system 100. Data communications links may be PCIe interfaces or may be based on other communication standards.

[0076] System 100 may include a baseboard management controller (BMC 113) configured to manage, control the operation of, and monitor components of system 100 and may function as the central management agent for a server or computing system. BMC 113 may be responsible for and configured to manage debugging, fault handling,firmware updating for components, thermal and power management, telemetry, and other suitable functions. BMC 113 may include at least one processor and at least one non- transitory computer-readable medium, the medium containing instructions configuring the processor to perform one or more functions outlined herein.

[0077] The PCIe switches 102 may be communicatively coupled to NVME SSDs 101 and a DPU 103, through communications channels, the communications channel including peripheral PCIe channels. The PCIe switches 102 direct data between the processors 104 and the PCIe endpoints (i.e. storage 101 and DPU 103). Each PCIe switch includes multiple ports for connecting to the components in the system. Other communications channels may be implemented between devices including one or more inter-integrated circuit channels (I2C channel 111), platform environment control interface channels (PECI channel 112), local area network / ethernet channels (LAN channel 113), power management bus (PMbus) channels, system management bus (SMbus) channels, compute express link (CXL) channels, serial (e.g., RS-232) channels, universal serial bus (USB) channels, universal asynchronous receiver / transmitter (UART) channels, management component transport protocol (MCTP) channels, or any suitable communications channels. Controller 110 further includes PCH 105 connected to NVMe SSDs 101 through communications channels 109. Additionally, system 100 may be stored in one or more racks. In various embodiments, the racks may have four slots, eight slots, fourteen slots, sixteen slots, thirty-two slots, or other suitable number of slots.

[0078] PCIe links support communication between any two endpoints, further allowing concurrent access across multiple endpoints. PCIe devices communicate via a logical connection called an interconnect or link. A link is a point-to-point communication channel between two PCIe ports allowing both of them to send and receive PCI requests (configuration, I / O, or memory read / write) and interrupts. At the physical level, a link is composed of one or more lanes. PCIe endpoint devices, such as GPUs 107, can connect to the endpoints NVlinks 108 and generally reside at the bottom of the branch of the PCIe tree topology.

[0079] System 100 may include one or more processors and / or one or more memory modules. Examples of processors that may be included in system 100 are microprocessor (e.g., a central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), data processing unit (DPU), and the like), a digital signal processing (DSP)die, an artificial intelligence (Al) accelerator, an application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), programmable logic controller (PLC), or any suitable processing circuitry. Examples of one or more memory modules that may be included in the system 100 include a dynamic random access memory (DRAM) module, a dual in-line memory module (DIMM), a static random access memory (SRAM) module, a flash memory module, a solid-state drive (SSD), a non-volatile random access memory (NVRAM) module, a read-only memory (ROM) module (such as a floating-gate ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), one-time programmable ROM (OTPROM), or the like), or any suitable type of memory module.

[0080] The architecture of system 100 further includes GPUs 107, which are communicatively connected to the CPUs 104 through the Cedar Modules 106. Controller 105 is connected to storage drives 101. The PCIe switch is connected to the NVLink Switches 108. System 100 may include processors 104, which may include CPU head nodes configured to manage one or more computational operations of system 100 such as data storage, task management, GPU allocation, temperature control, computations, or the like. System 100 may additionally include a platform controller hub (PCH 105), which may control data paths, clocking (e.g., operating the system clock), input / output functions, and other operations of system 100. In an embodiment, PCH 105 may be communicatively coupled to storage drives 101 through a PCIe connection 109.

[0081] System 100 may include a plurality of memory modules, including but not limited to NVME storage 101 (e.g., solid-state drives). These may be used to store data used by CPUs 104 and GPUs 107 (e.g., results of calculations, artificial intelligence models, data one which one or more calculations is to be performed, or the like), firmware for one or more components of system 100 including, CPUs 104. GPUs 107, or other components, status or operational information of one or more components of system 100, and the like.

[0082] System 100 may include a plurality of sensors which may be configured to detect a variety of system parameters related to the operation of system 100 including voltage, current, temperature, flow rate (e.g., air or liquid fluid flow rates), humidity, contaminant presence, dielectric constant, permittivity, permeability, or the like. These sensors may monitor a status of system 100 or one or more components of system 100 and control or alter operational parameters of system 100 or the one or more components.

[0083] System 100 may include GPUs 107, PCIe switches 102, and various modules such as cedar modules 107. PCIe switches 102 may control data routing and similar data flow paths from components such as PCH 105 and CPUs 104. Cedar modules 107 may provide an interconnection between GPUs 107 through PCIe channels.

[0084] The operational parameter of the at least one component of system 100 may include an operating temperature, a cooling operation (for example a flow of cooling fluid to transfer heat from system 100), a power level, a voltage, a frequency, a current, a power, a period, a temperature, a resistance, a state of charge, a filtration rate, or the like. For example, an identifier may indicate that system 100 is an immersion cooling server and therefore may indicate to GPUs 107 that they may operate with a first power level, a first temperature, a first frequency, or the like. In an alternative embodiment, an identifier may indicate that a system 100 is an air-cooled server, and that GPUs 107 may operate with a second power level lower than the first power level, a second temperature lower than the first temperature, a second frequency lower than the first frequency, or the like.

[0085] Fig. 1.2 is a flowchart of an example method for dynamic PCIe link retraining in an immersion fluid environment. At step 210, a connection is determined between root complex and PCIe endpoint devices, including for example, CPUs 104 and the GPUs 107. Establishing the connection allows for PCIe devices to send and receive data based on a predetermined PCIe data rate. At step 220, at least one data rate value is determined, the data rate value associated with a communications channel associated with, for example, a PCIe interface. At step 230, data is transmitted to at least one component of the plurality of components using the at least one communications channel. If the components coupled to the communications channel, including the PCIe devices are associated with a different PCIe interface (Gen 3), the PCIe devices will make adjustments through initial tuning and further fine tuning to reach optimal link equalization. This link equalization can result in efficient adjustments to the data rate associated with the communication channels and is caried out through initial and fine tuning resulting in, for example, less errors associated with data transmission, including stable and efficient transmission of data at an optimal data rate. This negotiation process leads to an optimal combination of equalization presets, for overall optimal link performance. Step 240 goes on to disclose adjusting, based at least in part on a second state associated with the at least one component of theplurality of components, the at least one data rate value. Step 250 transmits using the adjusted data rate value.

[0086] FIG. 1.3 depicts aspects of an immersion cooling system 300 for dissipating heat from one or more heat-generating components such as semiconductor die packages 305 via immersion cooling. Each package 305 can include one or more semiconductor dies that produce heat when the system is in operation. The immersion cooling system 300 in the illustrated example of FIG. 1.3 is a two-phase immersion cooling system, though the invention may also be implemented in a single-phase immersion cooling system. Immersion cooling system 300 may include some or all of the components of the immersion cooling system 100.

[0087] Immersion cooling system 300 includes a container such as tank 320 filled, at least in part, with immersion cooling liquid 364. The immersion cooling system 300 can further include at least one chiller 380 that flows a heat-transfer fluid through at least one condenser coil 370 that is disposed in the tank 320 and headspace 308. Condenser coil 370 and chiller 380 may be part of a heat exchanger. The packages 305 can be mounted on one or more printed circuit boards (PCBs) 357 that are immersed, at least in part, in the immersion cooling liquid 364. Immersion-cooling system 300 may further include a filter 375 disposed adjacent to the tank 320.

[0088] Filter 375 may include a filtration media, a housing, and a pump configured to force immersion cooling liquid 364 through filter 375 to remove contaminants, particulates, or other impurities that may be added to immersion cooling liquid 364 during use. Filter 375 may be housed outside of tank 320 while being in fluidic communication with immersion cooling liquid 364 in tank 320. Alternatively, filter 375 may be submerged within immersion cooling liquid 364 inside of tank 320.

[0089] Immersion cooling liquid 364 may be a hydrocarbon, a fluoroketone, an oil, or a similar dielectric liquid that will act as an insulator while simultaneously transferring heat from package 305 more efficiently than air. An example of immersion cooling liquid 364 is Novec™ 649 produced by 3M™. An exemplary immersion cooling liquid 364 used in accordance with embodiments of the present invention may have a dielectric constant baseline value of about 1.8-2 at a frequency of about 1 kHz.

[0090] In an embodiment of the invention, immersion cooling liquid 364 may be considered unacceptably contaminated if the dielectric constant and / or dielectric losstangent of immersion cooling fluid being used in an immersion cooling system 300 differs by a threshold amount as compared to unused or pure immersion cooling liquid 364. For example, immersion cooling liquid 364 may be considered unacceptably contaminated or degraded if the dielectric constant and / or dielectric loss tangent differs by a threshold of 10% or more as compared to unused or pure immersion cooling liquid 364. In an embodiment, a dielectric constant and / or dielectric loss tangent variation threshold may be 20%, 15%, 5%, 3%, 1%, or any suitable threshold.

[0091] Contamination of the immersion cooling liquid 364 and resulting changes to dielectric constant and / or dielectric loss tangent may alter or negatively impact operation of components within immersion cooling liquid 364 including semiconductor die(s) 350. An altered dielectric constant and / or dielectric loss tangent may result in undesirable cross-talk between components on a PCB, additional noise or reduction in signal strength transmitted along exposed wires of a PCB or semiconductor die(s) 350 submerged in immersion fluid, and / or signal dissipation through the immersion cooling liquid 364. Signal loss may be severe enough that two elements may be effectively represented as being separated by an open circuit despite being physically connected. In an embodiment, a dielectric constant and / or dielectric loss tangent variation threshold may be selected based on an observed or inferred effect on one or more submerged semiconductor die(s) 350. For example, an increase in PCIe bit error rate above an error rate baseline may be correlated with an increase in dielectric constant and / or dielectric loss tangent above a dielectric constant and / or dielectric loss tangent baseline. Accordingly, operation of semiconductor die(s) 350 may be throttled or suspended when a dielectric constant and / or dielectric loss tangent of immersion cooling liquid 364 exceeds a predetermined threshold.

[0092] Changes to dielectric constant and / or dielectric loss tangent may be caused by contaminants within immersion cooling liquid 364. In some cases, changes to dielectric constant and / or dielectric loss tangent may be reversed by filtering the contaminants from immersion cooling liquid 364. In some embodiments, upon detecting an increase in dielectric constant and / or dielectric loss tangent of immersion cooling liquid 364, controller 302 may instruct filter 375 to increase filtration throughput or notify a user that an immersion cooling liquid 364 filtration media may need to be replaced. If a dielectric constant and / or dielectric loss tangent exceeds a predetermined threshold, controller 302 may throttle or shut down one or more semiconductor die(s) 350, generate a notificationthat immersion cooling liquid 364 should be replaced, trigger an alarm, etc.

[0093] The illustrated example of FIG. 1.3 is not intended to be to scale. The immersion cooling system 300 may house and provide immersion cooling liquid 364 to tens, hundreds, or even thousands of packages 305. In some cases, the immersion cooling system 300 can be small (e.g., the size of a floor unit air conditioner, approximately 1 meter high, 0.5 meter width, 0.5 meter depth or length). In some implementations, the immersion cooling system can be large (e.g., the size of a van or larger, approximately 2.5 meters high, 2.5 meters width, 4 meters depth or length).

[0094] The immersion cooling system 300 can also include a controller 302 (e.g., a microcontroller, programmable logic controller (PLC), microprocessor, field- programmable gate array, logic circuitry, memory, or some combination thereof) to manage system operation. Controller 302 can perform various system functions such as monitoring temperatures of system components, cooling fluid level, tank access, chiller operation etc. The controller 302 can further issue commands to control system operation such as executing a start-up sequence, executing a shut-down sequence, assigning workloads among the packages, changing cooling fluid level, changing the temperature of the heat-transfer fluid circulated by the chiller 380, etc. In some implementations, controller 302 can include (or itself be) a baseboard management controller (BMC) 304. That is, the BMC 304 may monitor and control all aspects of system operation for the immersion cooling system 300 in addition to monitoring and controlling workloads of the semiconductor dies 350 in the packages 305 cooled by the system. The immersion cooling system 300 can also include a network interface controller (NIC 303) to allow the system to communicate over a network, such as a local area network or wide area network. The immersion cooling system 300 can further include a fluid sensor array 390 having a plurality of fluid sensors 310. Fluid sensors 310 may include one or more leak detection sensors at least partially submerged in immersion cooling liquid 364.

[0095] The semiconductor die(s) 350 can be mounted on and attached to a printed circuit board (PCB) 355 (sometimes referred to as a substrate) in device package 305. The package 305 can be made commercially available as an off-the-shelf (OTS) product. The package 305 can be used for single-phase or two-phase immersion cooling of at least one semiconductor die 350, such as a microprocessor (e.g., a central processing unit (CPU) and / or graphics processing unit (GPU)), voltage regulator (VR), high bandwidth memory(HBM), a digital signal processing (DSP) die, an artificial intelligence (Al) accelerator, an application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), and / or other densely patterned semiconductor die.

[0096] In the two-phase immersion cooling system 300 of FIG. 1.3, heat flows from the semiconductor die 350 where it is generated into the heat spreader 352. The heat spreader 352 is in thermal contact with an immersion cooling liquid 364 that can flow over and extract heat from the heat spreader 352. The amount of heat delivered by the heat spreader 352 to the immersion cooling liquid 364 is enough to boil the immersion cooling liquid 364 that contacts the heat spreader 352 (creating bubbles 365 and potentially creating froth 367 when bubbles 365 reach the surface of immersion cooling liquid 364). The vapor 366 from the boiled immersion cooling liquid 364 can be cooled and condensed back to liquid droplets 368, for example, by the condenser coil 370. The heat-transfer fluid, such as chilled water, from the chiller 380 can be circulated through the condenser coil 370 to lower the temperature of the condenser coil 370 below the condensation point in the headspace 308 of the tank 320. As a result, vapor 366 condenses on exterior surfaces of the condenser coil 370 and liquid droplets 368 from the condensed vapor can drip and / or flow back to the immersion cooling liquid 364. Although a single condenser coil 370 is depicted in FIG. 1.3, there can be a plurality of condenser coils 370 in tank 320 to condense the vapor 366 into droplets. Some or all of the condenser coils 370 may or may not be located directly over the PCBs 357. Instead, the condenser coil(s) 370 can be located near one or more walls of the tank 320, such that the condenser coil(s) 370 are not directly over the PCBs 357 on which the packages 305 are mounted.

[0097] To improve thermal performance in two-phase immersion cooling system 300, the heat spreader 352 can include a boiling enhancement coating (BEC) on at least one surface. The BEC can be formed from copper or a copper alloy and can be porous, for example, though BECs can take various forms. In some cases, the BEC is a micro porous copper coating having a thickness from approximately or exactly 50 microns to 500 microns thick (which may be produced by electroplating and / or etching). In some implementations, the BEC comprises a mesh copper layer bonded (e.g., via resistance heating) to at least an outer surface of the heat spreader 352. In some cases, the BEC is applied as particulates to at least one smooth surface of the heat spreader 352 and then subsequently sintered to adhere to one another and to the heat spreader 352. The BECprovides an improved surface area to contact the immersion cooling liquid 364 and can increase the heat transfer coefficient from the heat spreader 352 to the immersion cooling liquid 364 by up to a factor of 15 versus a smooth surface on the heat spreader 352. Accordingly, BECs can increase thermal conductivity to, and accelerate the boiling of, the immersion cooling liquid 364.

[0098] Further implementations of boiling enhancement coatings and enclosures are possible. Additional arrangements, applications, and methods of use of boiling enhancement coatings and enclosures, including with semiconductor dies and 3DIC stacks, are described in the below U.S. Patent Applications.

[0099] U.S. Patent Application No. 18 / 327,615, filed June 1, 2023 and entitled "Boiler Enhancement Coatings with Active Boiling Management,” discloses heat spreader and boiling enhancement enclosure architectures thermally and / or mechanically coupled to one or more semiconductor dies or logic ICs that may be used for passive and / or active management of immersion cooling fluid boiling, including through the use of valves to control pressure of boiling immersion cooling fluid within a boiling enhancement chamber, particularly in paragraphs

[0018] -

[0039] and FIGS. 3-5B. The entirety of U.S. Patent Application No. 18 / 327,615 is incorporated herein by reference.

[0100] U.S. Provisional Patent Application No. 63 / 500,167, filed May 4, 2023 and entitled “Direct to Chip Heat Spreader and Boiler Enhancement Coatings for Microelectronics,” discloses heat spreader and BECs thermally and / or mechanically coupled to one or more semiconductor dies, logic ICs, and / or 3DIC stacks, particularly in paragraphs

[0015] -

[0033] and FIGS. 2A-4. BEC form factors may include graphite heat spreader architectures, vapor chambers, heat pipes, copper plates, fins, and the like. BEC form factors may be thermally and / or mechanically coupled to the one or more semiconductor dies, logic ICs, and / or 3DIC stacks through a thermally conductive epoxy, and may have varying dimensions relative to a surface to which the semiconductor dies and / or logic ICs are mounted. The entirety of U.S. Provisional Patent Application No. 63 / 500,167 is incorporated herein by reference.

[0101] U.S. Patent Application No. 18 / 460,091, filed September 1, 2023 and entitled “Direct to Chip Application of Boiling Enhancement Coating,” discloses BECs and methods for applying BECs to semiconductor dies, logic ICs, and / or 3DIC stacks in accordance with the present technology. In particular, paragraphs

[0024] -

[0046] and FIGS.2A-5 disclose embodiments of BEC layers, adhesives, solders, sintering, laser ablation, meshes, and other BECs and BEC application methods. The entirety of U.S. Patent Application No. 18 / 460,091 is incorporated herein by reference.

[0102] U.S. Provisional Patent Application No. 63 / 506,945, filed June 8, 2023 and entitled “Vapor- Shedding Structures for Boiler Plates in Two-Phase Immersion Cooling Systems,” discloses structures that may be thermally and / or mechanically coupled to computing hardware such as one or more semiconductor dies, logic ICs, and / or 3DIC stacks to enable the shedding of immersion cooling vapors generated from the boiling of immersion cooling fluid during operation of the computing hardware. In particular, paragraphs

[0021] -

[0039] and FIGS. 3A-5 disclose vapor-shedding structures including varying porosities, constituent materials, and geometries relative to the computing hardware on which they are mounted. The entirety of U.S. Provisional Patent Application No. 63 / 506,945 is incorporated herein by reference.

[0103] U.S. Provisional Application No. 63 / 513,828, filed July 14, 2023 and entitled “Grinding Apparatuses and Methods for Mechanically Modifying Surfaces of Processors to Promote Boiling of a Coolant Liquid,” discloses methods for creating boiling enhancement modifications to surfaces such as the surfaces of computing hardware such as one or more semiconductor dies, logic ICs, and / or 3DIC stacks, particularly in paragraphs

[0036] -

[0095] and FIGS. 2A-8. For example, grooves, patterns, gouges, trenches, or other structures may be added to a surface or lid of a processor, semiconductor die, logic IC, 3DIC stack component, and / or BEC to encourage nucleation sites for bubbles of immersion cooling vapor to form during a cooling process, thus decreasing the thermal resistance between the processor, semiconductor die, logic IC, and / or 3DIC stack component and the surrounding immersion cooling fluid. The entirety of U.S. Provisional Application No. 63 / 513,828 is incorporated herein by reference.

[0104] U.S. Provisional Patent Application No. 63 / 513,829, filed July 14, 2023 and entitled “Electrical Connector Having a Heater to Facilitate Boiling of a Coolant Liquid to Improve Signal Integrity in Immersion Cooling Environment,” discloses heaters for promoting boiling of immersion cooling fluid near electrical connectors such as connections between components of a 3DIC stack and enable improved impedances at those connectors, particularly in paragraphs

[0019] -

[0052] and FIGS. 1A-3B. The entirety of U.S. Provisional Patent Application No. 63 / 513,829 is incorporated herein by reference.

[0105] U.S. Provisional Patent Application No. 63 / 603,242, filed November 28, 2023 and entitled “Woven Boiler Enhancement Coatings,” provides additional examples of BECs including woven BECs with variable weave patterns, densities, attachment mechanisms, and materials (including copper and tungsten) that may be attached to computing hardware such as one or more semiconductor dies, logic ICs, and / or 3DIC stacks in order to promote more efficient heat transfer and immersion cooling vapor nucleation, particularly in paragraphs

[0031] -

[0055] and FIGS. 3-7. The entirety of U.S. Provisional Patent Application No. 63 / 603,242 is incorporated herein by reference.CONCLUSION

[0106] While various inventive embodiments have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is deemed to be within the scope of the inventive embodiments described herein. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the inventive teachings is / are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation, many equivalents to the specific inventive embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed. Inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the inventive scope of the present disclosure.

[0107] Also, various inventive concepts may be embodied as one or more methods, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which mayinclude performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

[0108] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0109] The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.”

[0110] The phrase “and / or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and / or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and / or” clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to “A and / or B”, when used in conjunction with open-ended language such as “comprising” can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); etc.

[0111] As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of’ or “exactly one of,” or, when used in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.” “Consisting essentially of,” when used in the claims, shall have its ordinary meaning as used in the field of patent law.

[0112] As used herein in the specification and in the claims, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified. Thus, as a nonlimiting example, “at least one of A and B” (or, equivalently, “at least one of A or B,” or, equivalently “at least one of A and / or B”) can refer, in one embodiment, to at least one, optionally including more than one, A, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in yet another embodiment, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other elements); etc.

[0113] In the claims, as well as in the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases “consisting of’ and “consisting essentially of’ shall be closed or semi-closed transitional phrases, respectively, as set forth in the United States Patent Office Manual of Patent Examining Procedures, Section 2111.03.

[0114] Section 2: FIG. 2.1 A illustrates an immersion cooling system 2100 in accordance with the present technology. In particular, immersion cooling system 2100 may include an immersion cooling container 2130 and lid 2132. Immersion cooling container 2130 may be at least partially filled with immersion cooling liquid 2140, which may be a two-phase immersion cooling liquid. Immersion cooling system 2100 may include controller 2102 configured to control and / or determine one or more operational parameters of immersion cooling system 2100 and / or components included in immersion cooling system 2100 such as servers 110 and cables 120. Controller 2102 may be analogous to controller 2302 in FIG. 2.3, and may include a programmable logic controller (PEC),microprocessor, field-programmable gate array (FPGA), logic circuitry, memory module, or some combination thereof.

[0115] Controller 2102 may be communicatively coupled with one or more controllers external to immersion cooling system 2100 such as a cloud-based control plane, facility management control system, or any suitable digital or human controller. Controller 2102 may be configured to transmit a notification to an external controller based on one or more conditions of immersion cooling system 2100.

[0116] Immersion cooling system 2100 may include servers 110, for example server 2110a, server 2110b, server 2110c, and server 1 lOd. Servers 110 may be high-performance computing servers such as graphics processing unit (GPU) servers configured for performing artificial intelligence (Al) training or inference, video processing, mathematical model computation, or any suitable computation. Servers 110 may include one or more logic ICs mounted on a printed circuit board (PCB). A PCB may include traces, ball grid arrays (BGAs), sensors, leads, inputs, outputs, networking hardware including switches and cables, and / or additional electrical components.

[0117] Controller 2102 and / or logic ICs disposed on servers 110 may include one or more processors, microprocessors, central processing units (CPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other logic ICs configured to perform calculations. Each logic IC may additionally or alternatively be embodied as a system-on-a-chip (SoC), three-dimensional integrated circuit (3DIC) stack, central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), data processing unit (DPU), voltage regulator (VR), high bandwidth memory (HBM), digital signal processor (DSP), artificial intelligence (Al) accelerator, and / or other densely patterned semiconductor die.

[0118] Controller 2102 and / or logic ICs disposed on servers 110 may further include one or more memory modules such as a dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, solid-state memory (SSD), non-volatile random access memory (NVRAM), read-only memory (ROM, such as a floating-gate ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), one-time programmable ROM (OTPROM), or the like), or any suitable type of memory module. Controller 2102 may include one or more registers, data buffers, inputs and / or outputs (e.g., inter-integrated circuit ports, serialports, busses, parallel ports, wireless transmitters and receivers, universal serial bus (USB) ports, controller area network (CAN) busses, etc.), sensors (e.g., temperature, voltage, current, or similar sensors), and the like.

[0119] Controller 2102 may be a baseboard management controller (BMC) configured to control one or more operational parameters of servers 110 including operational parameters of logic ICs disposed on servers 110. For example, controller 2102 may throttle, rate limit, or depower logic ICs. Controller 2102 may control a voltage, a current, a resistance, a capacitance, a temperature, a flow rate, a frequency, a power, a storage of energy, an operation of a battery, or any suitable operational parameter of logic ICs.

[0120] Controller 2102 may be communicatively coupled to servers 110 through server cables 120 and / or network cable 2122. Server cables 120 may include server cable 2120a communicatively coupling server 2110a to controller 2102, server cable 2120b communicatively coupling server 2110b to controller 2102, server cable 2120c communicatively coupling server 2110c to controller 2102, and server cable 2120d communicatively coupling server 2110d to controller 2102. Server cables 120 and / or network cable 2122 may be high-speed cables such as fiber-optic cables, micro coaxial cables, twinax cables, or any suitable cables, wires, or links for communication.

[0121] Network cable 2122 may be analogous to each of server cables 120 and serve as a main cable to which each of cables 120 are connected. Each of server cables 120 and / or network cable 2122 may include a flexible housing that surrounds a transmission medium. The flexible housing may include rubber, thermopolymers, fluoropolymers, polyvinyl chloride, or the like. In an embodiment, the flexible housing may include a flexible mesh. The flexible mesh may include a plurality of interwoven fibers and strain detector 2160 may be woven into the mesh.

[0122] Each of server cables 120 and / or network cable 2122 may include a transmission medium such as a conduit made of glass, silica, fluorozirconate, fluoroaluminate, chalcogenide, sapphire, or other material through which information in the form of light may be transmitted. A transmission medium may include any suitable material such as copper, aluminum, carbon, or the like.

[0123] Servers 110, server cables 120 and network cable 2122 may be partially or fully immersed in immersion cooling liquid 2140. During operation, heat generated by logic ICs or other components of immersion cooling system 2100 may cause a portion of immersion cooling liquid 2140 to boil and change to a vapor phase. The vapor may rise into an upper portion of immersion cooling container 2130 and contact one or more condenser tubes, which may cause the vapor to recondense into immersion cooling liquid 2140.

[0124] FIG. 2. IB illustrates immersion cooling system 2100 in an exemplary state of maintenance. In an embodiment, a lifting mechanism 2150 such as a crane or lift is used to remove server 21 lOd from immersion cooling container 2130 for repair or replacement. However, the lifting of server 21 lOd has caused server cable 2120d to become taut, placing an increased strain on the cable and potentially damaging or breaking the cable. For example, at cable attachment point 2124a (where server cable 2120d connects to server 2110d) and cable attachment point 2124b (where server cable 2120d connects to network cable 2122), the cable may experience unacceptably small bending radii, causing damage or breaking the cable connection entirely. In the absence of an indication of a strain placed on server cable 2120d, a human or electronic operator of lifting mechanism 2150 may not be aware of the need to cease lifting operations.

[0125] FIG. 2.1C illustrates immersion cooling system 2100 further including a strain detector 2160. Strain detector 2160 may provide an indication of an amount of strain and / or stress that server cables 120 and / or network cable 2122 are under. Each of server cable 2120a, server cable 2120b, server cable 2120c, server cable 2120d, and network cable 2122 may include one or more strain detector 2160. Strain detector 2160 may be communicatively coupled to controller 2102 or any suitable computing device configured to detect an input or property from strain detector 2160 such as resistance.

[0126] Strain detector 2160 may include a wire (e.g., made of copper) of a known R'A resistivity in an unstrained state. Resistivity p is defined as p = — , where R is the resistanceof the wire, A is the cross-sectional area of the wire, and L is the length of the wire. As the wire is deformed and strained under a load (e.g., a tensile load), the wire will deform and R, A, and L will change as a function of the load. A change in A and L for a given load may be determined for a given wire. Thus, the measured change in resistance may indicate a strain applied to the strain detector 2160 (and therefore to a cable to which the strain detector2160 is attached). Because strain detector 2160 is fixed with respect to a cable (such as server cable 2120d), the strain applied to strain detector 2160 will provide an indication of how much strain is applied to server cable 2120d. Controller 2102 may monitor one or more resistivities associated with strain detector 2160 and determine if a strain on one or more portions of server cables 120 exceeds a threshold.

[0127] Strain detector 2160 may include a wire having a diameter of about 0.01 mm, about 0.02 mm, about 0.03 mm, about 0.04 mm, about 0.05 mm, about 0.06 mm, about 0.07 mm, about 0.08 mm, about 0.09 mm, about 0.1 mm, about 0.2 mm, about 0.5 mm, about 0.8 mm, about 1 mm, between about 0.01 mm and about 0.05 mm, between about 0.03 mm and about 0.08 mm, between about 0.05 mm and about 0.1 mm, between about 0.07 mm and about 0.2 mm, between about 0.1 mm and about 0.5 mm, between about 0.2 mm and about 0.8 mm, between about 0.5 mm and about 1 mm, or any suitable diameter.

[0128] Strain detection means 2160 may include multiple segments, each segment associated with a portion of server cables 120. For example, strain detector 2160 may include a segment disposed on server cable 2120a and configured to provide an indication of a strain applied to server cable 2120a, a segment disposed on server cable 2120b and configured to provide an indication of a strain applied to server cable 2120b, a segment disposed on server cable 2120c and configured to provide an indication of a strain applied to server cable 2120c, a segment disposed on server cable 2120d and configured to provide an indication of a strain applied to server cable 2120d, and so on.

[0129] Strain detector 2160 may include a plurality of segments of approximately equal length, for example a plurality of segments approximately 10 cm in length, with each segment of the plurality of segments having an individual resistance. A segment of strain detector 2160 may be any suitable length, for example about 0.5 cm, about 1 cm, about 2 cm, about 5 cm, about 8 cm, about 10 cm, about 15 cm, about 20 cm, about 25 cm, about 50 cm, about 75 cm, about 100 cm, about 125 cm, about 150 cm, about 200 cm, about 250 cm, about 500 cm, about 10 m, about 15 m, about 20 m, between about 0.5 cm and about 2 cm, between about 1 cm and about 5 cm, between about 2 cm and about 8 cm, between about 5 cm and about 10 cm, between about 8 cm and about 15 cm, between about 10 cm and about 20 cm, between about 15 cm and about 25 cm, between about 20 cm and about 50 cm, between about 25 cm and about 75 cm, between about 50 cm and about 100 cm, between about 75 cm and about 125 cm, between about 100 cm and about 150 cm, betweenabout 125 cm and about 200 cm, between about 150 cm and about 250 cm, between about 200 cm and about 500 cm, between about 250 cm and about 750 cm, between about 0.5 m and about 2 m, between about 1 m and about 5 m, between about 2 m and about 10 m, or any suitable length. In an embodiment, strain detector 2160 may include a plurality of segments having a plurality of lengths.

[0130] Strain detection means 2160 may include one or more segments that are not disposed on an entire length of a cable such as server cables 120. For example, strain detector 2160 may include one or more segments disposed at a connection point for each cable of server cables 120, where a cable is likely to bend, where a cable has relatively less flexibility, where a portion of a cable is fixed in place, but an adjacent portion of the cable is free to move, and the like, but strain detector 2160 may be absent from an area of a cable unlikely to excessively deform or bend. For example, strain detector 2160 may not be disposed on portions of server cables 120 that are substantially fixed, substantially straight, and / or substantially unlikely to bend. This may reduce cost and / or system complexity while not detrimentally affecting an ability to monitor cable bending in immersion cooling system 2100.

[0131] Additionally or alternatively, a reflectivity or ultrasonic sensor may be used to measure a particular location of a change in resistance (and therefore resistivity) due to a bending of the strain detector 2160 and cable to which it is attached.

[0132] A resistivity for strain detector 2160 may be about 5e-9 Q-m, about le-8 firn, about 2e-8 Q-m, about 3e-8 Q-m, about 4e-8 Q-m, about 5e-8 Q-m, about 6e-8 Q-m, about 7e-8 Q-m, about 8e-8 Q-m, about 9e-8 Q-m, about le-7 Q-m, about 2e-7 Q-m, about 3e-7 Q-m, 4e-7 Q-m, about 5e-7 Q-m, about 6e-7 Q-m, about 7e-7 Q-m, about 8e-7 Q-m, about 9e-7 Q-m, about le-6 Q-m, about 5e-6 Q-m, about le-5 Q-m, about 5e-5 Q-m, between about 5e-9 Q-m and about 5e-8 Q-m, between about le-8 Q-m and about 8e-8 Q- m, between about 4e-8 Q-m and about le-7 Q-m, between about 8e-8 Q-m and about 3e-7 Q-m, between about le-7 Q-m and about 5e-7 Q-m, between about 3e-7 Q-m and about 8e- 7 Q-m, between about 5e-7 Q-m and about 9e-7 Q-m, between about 7e-7 Q-m and about 3e-6 Q-m, between about le-6 Q-m and about 5e-6 Q-m, between about 3e-6 Q-m and about 7e-6 Q-m, between about 5e-6 Q-m and about le-5 Q-m, between about 8e-6 Q-m and about 5e-5 Q-m, or any suitable resistivity value. The resistance for strain detector 2160 may varyas a function of the strain applied to strain detector 2160 and the resistivity of the material of strain detector 2160.

[0133] Controller 2102 may be configured to determine if a strain applied to a cable such as one or more of server cables 120 exceeds a threshold based on the resistance of the strain detector 2160 while the strain is applied to the cable and strain detector 2160. For example, controller 2102 may determine that a strain applied to one or more of server cables 120 is equal to or greater than a threshold by correlating a change in resistance of strain detector 2160 (expressed as a percentage difference between a strained and unstrained state) to a strain applied to one or more of server cables 120.

[0134] For example, controller 2102 may determine that a strain of a cable on which strain detector 2160 is disposed exceeds a threshold based on a measured resistance of strain detector 2160 increasing compared to a baseline (unstretched) resistance of strain detector 2160 by about 0.05%, about 0.1%, about 0.2%, about 0.3%, about 0.5%, about 0.8%, about 1%, about 2%, between about 0.05% and about 0.08%, between about 0.075% and about 0.125%, between about 0.1% and about 0.5%, between about 0.2% and about 0.6%, between about 0.3% and about 0.8%, between about 0.5% and about 1%, between about 1% and about 2%, or any suitable percentage.

[0135] For example, controller 2102 may determine that a baseline resistance of strain detector 2160 is about 2 kOhm and that a measured resistance of strain detector 2160 under strain (e.g., bending, stretching, or other deformation) differs from 2 kOhm by about 1 Ohm to about 10 Ohms.

[0136] Controller 2102 may be configured to transmit a notification to one or more entities in response to a resistance of strain detector 2160 indicating that a strain of a cable such as one or more of server cables 120 exceeds a threshold. For example, in response to a resistance of strain detector 2160 indicating that a strain of a cable exceeds a threshold, controller 2102 may transmit a notification to a user such as a technician or a facility manager, an immersion cooling system controller such as a PLC, a facility controller such as a control plane, or any suitable entity. The notification may indicate that a strain of a cable exceeds a threshold and that a related activity (such as removing a server to which the cable is connected) should be stopped. In an embodiment, controller 2102 may be communicatively coupled to a lifting mechanism 150 configured to remove a server from an immersion cooling system (e.g., a crane). Lifting mechanism 2150 may lift a server suchas server 2110d out of immersion cooling system 2100. Lifting mechanism 2150 may receive a notification from controller 2102 in response to a strain on server cable 2120d exceeding a threshold based on an indication from strain detector 2160 that the strain applied to strain detector 2160 and server cable 2120d exceeds a threshold.

[0137] One or more cables may include a number, color, or similar coding indicating a cable. Each strain detector 2160 may similarly include a corresponding coding indicating that a strain detector 2160 measures strain of a particular cable of a plurality of cables. Cables and associated strain detector 2160 may be numbered, color coded, or otherwise marked based on connection to a particular server, portion of immersion cooling system 2100, function, or the like. For example, server cable 2120a and associated portion of strain detector 2160 may be colored red to indicate that it connects to server 2110a, server cable 2120b and associated portion of strain detector 2160 may be colored blue to indicate that it connects to server 2110b, server cable 2120c and associated portion of strain detector 2160 may be colored orange to indicate that it connects to server 2110c, and server cable 2120d and associated portion of strain detector 2160 may be colored green to indicate that it connects to server 21 lOd.

[0138] Additionally or alternatively, one or more cables may include a radiofrequency identification (RFID) tag. Each RFID tag on a cable may be communicatively coupled to a strain detector 2160 and transmit information indicating a permissible or current bend radius for a particular cable, an installation location of the cable and / or strain detector 2160, a current amount of strain applied to the cable and / or strain detector 2160, or any suitable information. An RFID tag may be wirelessly coupled to controller 2102, and controller 2102 may be configured to receive information from the RFID tag and transmit that information to a technician, additional controller (e.g., an immersion cooling system programmable logic controller (PEC), computing system control plane, or other suitable controller), or other suitable entity to provide relevant information about the cable and / or strain detector 2160.

[0139] In some implementations, a Wheatstone bridge 2210 may be used to determine a resistivity, voltage, resistance, impedance, capacitance, etc., across a strain detector 2200 as depicted in FIG. 2.2. The Wheatstone bridge 2210 may be electrically altered in response to at least one of a resistivity, a current, a voltage, a resistance, an impedance, a capacitance, or other suitable electrical parameter of the strain detector 2200.For example, a resistance across a portion of the Wheatstone bridge 2210 may be altered in response to a change in strain of the strain detector 2200.

[0140] The Wheatstone bridge 2210 can be operated to sense an unknown impedance (R4 in this example). For the example of FIG. 2.2, real resistances are used though impedances (which can include capacitance and inductance) can be used more generally. In a Wheatstone bridge 2210, three of the four resistances Ri, R2, R3 (or impedances) for resistive elements, are known, whereas the fourth resistance R4 (or impedance) for the variable resistive element is unknown and to be determined. A current source, which can be direct current (DC) or alternating current (AC), drives currents through the two circuit branches of the Wheatstone bridge 2210. One way to operate the Wheatstone bridge 2210 is to select Ri = R2 = R3 = R. If R4 = R, then the Wheatstone bridge 2210 is balanced and a voltage at voltage measurement node VA equals a voltage at voltage measurement node VB. Deviations of R4 from R results in VA i VB. Measuring the voltage between measurement nodes VA and VB, (which may, e.g., be done by controller 202) can then indicate the difference of R4 for the variable resistive element from R. Solving straightforward circuit equations can provide the value of R4 for the resistive element.

[0141] When the Wheatstone bridge 2210 is implemented in the impedance sensor, resistive elements can be implemented with integrated or discrete resistors, for example. The variable resistive element may comprise, at least in part, the strain detector 2200 and its associated resistance. In that regard, the variable resistive element can be implemented as a pair of separated electrodes across which the strain detector 2200 is connected. The resistance (and therefore strain) of the strain detector 2200 (e.g., determined via resistance R4 and / or impedance) can be determined, at least in part, by an amount of strain applied to a cable to which strain detector 2200 is attached.

[0142] FIG. 2.3 depicts aspects of an immersion cooling system 2300 for dissipating heat from one or more heat-generating components such as semiconductor die packages 2305 via immersion cooling. Each package 2305 can include one or more semiconductor dies that produce heat when the system is in operation. The immersion cooling system 2300 in the illustrated example of FIG. 2.3 is a two-phase immersion cooling system, though the invention may also be implemented in a single-phase immersion cooling system.

[0143] Immersion cooling system 2300 includes a container such as tank 320 filled, at least in part, with immersion cooling liquid 2364. The immersion cooling system 2300 can further include at least one chiller 2380 that flows a heat-transfer fluid through at least one condenser tube 2370 that is disposed in the tank 320 and headspace 2308. Condenser tubes 2370 and chiller 2380 may be part of a heat exchanger. The packages 2305 can be mounted on one or more printed circuit boards (PCBs) 2357 that are immersed, at least in part, in the immersion cooling liquid 2364. Immersion-cooling system 2300 may further include a filter 2375 disposed adjacent to the tank 320.

[0144] Filter 2375 may include a filtration media, a housing, and a pump configured to force immersion cooling liquid 2364 through filter 2375 to remove contaminants, particulates, or other impurities that may be added to immersion cooling liquid 2364 during use. Filter 2375 may be housed outside of tank 320 while being in fluidic communication with immersion cooling liquid 2364 in tank 320. Alternatively, filter 2375 may be submerged within immersion cooling liquid 2364 inside of tank 320.

[0145] Immersion cooling liquid 2364 may be a hydrocarbon, a fluoroketone, an oil, or a similar dielectric liquid that will act as an insulator while simultaneously transferring heat from package 2305 more efficiently than air. Examples of immersion cooling liquid 2364 are Novec™ 649, Novec™ 7000, and Novec™ 7100 produced by 3M™. An exemplary immersion cooling liquid 2364 used in accordance with embodiments of the present technology may have a dielectric constant baseline value of about 1.8-2 at a frequency of about 1 kHz.

[0146] In an embodiment of the invention, immersion cooling liquid 2364 may be considered unacceptably contaminated if the dielectric constant and / or dielectric loss tangent of immersion cooling fluid being used in an immersion cooling system 2300 differs by a threshold amount as compared to unused or pure immersion cooling liquid 2364. For example, immersion cooling liquid 2364 may be considered unacceptably contaminated or degraded if the dielectric constant and / or dielectric loss tangent differs by a threshold of 10% or more as compared to unused or pure immersion cooling liquid 2364. In an embodiment, a dielectric constant and / or dielectric loss tangent variation threshold may be 20%, 15%, 5%, 3%, 1%, or any suitable threshold.

[0147] Contamination of the immersion cooling liquid 2364 and resulting changes to dielectric constant and / or dielectric loss tangent may alter or negatively impactoperation of components within immersion cooling liquid 2364 including semiconductor die(s) 2350. An altered dielectric constant and / or dielectric loss tangent may result in undesirable cross-talk between components on a PCB, additional noise or reduction in signal strength transmitted along exposed wires of a PCB or semiconductor die(s) 2350 submerged in immersion fluid, and / or signal dissipation through the immersion cooling liquid 2364. Signal loss may be severe enough that two elements may be effectively represented as being separated by an open circuit despite being physically connected. In an embodiment, a dielectric constant and / or dielectric loss tangent variation threshold may be selected based on an observed or inferred effect on one or more submerged semiconductor die(s) 2350. For example, an increase in PCIe bit error rate above an error rate baseline may be correlated with an increase in dielectric constant and / or dielectric loss tangent above a dielectric constant and / or dielectric loss tangent baseline. Accordingly, operation of semiconductor die(s) 2350 may be throttled or suspended when a dielectric constant and / or dielectric loss tangent of immersion cooling liquid 2364 exceeds a predetermined threshold.

[0148] Changes to dielectric constant and / or dielectric loss tangent may be caused by contaminants within immersion cooling liquid 2364. In some cases, changes to dielectric constant and / or dielectric loss tangent may be reversed by filtering the contaminants from immersion cooling liquid 2364. In some embodiments, upon detecting an increase in dielectric constant and / or dielectric loss tangent of immersion cooling liquid 2364, controller 2302 may instruct filter 2375 to increase filtration throughput or notify a user that an immersion cooling liquid 2364 filtration media may need to be replaced. If a dielectric constant and / or dielectric loss tangent exceeds a predetermined threshold, controller 2302 may throttle or shut down one or more semiconductor die(s) 2350, generate a notification that immersion cooling liquid 2364 should be replaced, trigger an alarm, etc.

[0149] Further examples of sensors and methods for immersion cooling contamination monitoring may include probes for monitoring immersion cooling liquid parameters such as dielectric constant and dielectric loss tangent, and processors configured to identify trends in sensor data, model immersion cooling system behavior as a function of contamination, and alter operations of immersion cooling systems based on detected levels and / or states of contamination may be found in U.S. Provisional PatentApplication 63 / 516,748, filed July 31, 2023 and entitled “Di-Electric Monitoring of Immersion Fluid During Cooling Operation,” the entirety of which is incorporated herein by reference.

[0150] The illustrated example of FIG. 2.3 is not intended to be to scale. The immersion cooling system 2300 may house and provide immersion cooling liquid 2364 to tens, hundreds, or even thousands of packages 2305. In some cases, the immersion cooling system 2300 can be small (e.g., the size of a floor unit air conditioner, approximately 1 meter high, 0.5 meter width, 0.5 meter depth or length). In some implementations, the immersion cooling system can be large (e.g., the size of a van or larger, approximately 2.5 meters high, 2.5 meters width, 4 meters depth or length).

[0151] The immersion cooling system 2300 can also include a controller 2302 (e.g., a microcontroller, programmable logic controller (PFC), microprocessor, field- programmable gate array, logic circuitry, memory, or some combination thereof) to manage system operation. Controller 2302 can perform various system functions such as monitoring temperatures of system components, cooling fluid level, tank access, chiller operation etc. The controller 2302 can further issue commands to control system operation such as executing a start-up sequence, executing a shut-down sequence, assigning workloads among the packages, changing cooling fluid level, changing the temperature of the heat-transfer fluid circulated by the chiller 2380, etc. In some implementations, controller 2302 can include (or itself be) a baseboard management controller (BMC) 2304. That is, the BMC 2304 may monitor and control all aspects of system operation for the immersion cooling system 2300 in addition to monitoring and controlling workloads of the semiconductor dies 2350 in the packages 2305 cooled by the system. The immersion cooling system 2300 can also include a network interface controller (NIC 2303) to allow the system to communicate over a network, such as a local area network or wide area network. The immersion cooling system 2300 can further include a fluid sensor array 2390 having a plurality of fluid sensors 2310. Fluid sensors 2310 may include one or more leak detection sensors at least partially submerged in immersion cooling liquid 2364.

[0152] The semiconductor die(s) 2350 and can be mounted on and attached to a printed circuit board (PCB) 355 (sometimes referred to as a substrate) in device package 2305. The package 2305 can be made commercially available as an off-the-shelf (OTS) product. The package 2305 can be used for single -phase or two-phase immersion coolingof at least one semiconductor die 2350, such as a microprocessor (e.g., a central processing unit (CPU) and / or graphics processing unit (GPU)), voltage regulator (VR), high bandwidth memory (HBM), a digital signal processing (DSP) die, an artificial intelligence (Al) accelerator, an application-specific integrated circuit (ASIC), field- programmable gate array (FPGA), and / or other densely patterned semiconductor die.

[0153] In the two-phase immersion cooling system 2300 of FIG. 2.3, heat flows from the semiconductor die 2350 where it is generated into the heat spreader 2352. The heat spreader 2352 is in thermal contact with an immersion cooling liquid 2364 that can flow over and extract heat from the heat spreader 2352. The amount of heat delivered by the heat spreader 2352 to the immersion cooling liquid 2364 is enough to boil the immersion cooling liquid 2364 that contacts the heat spreader 2352 (creating bubbles 2365 and potentially creating froth 2367 when bubbles 2365 reach the surface of immersion cooling liquid 2364). The vapor 2366 from the boiled immersion cooling liquid 2364 can be cooled and condensed back to liquid droplets 2368, for example, by the condenser tube 2370. The heat-transfer fluid, such as chilled water, from the chiller 2380 can be circulated through the condenser tube 2370 to lower the temperature of the condenser tube 2370 below the condensation point in the headspace 2308 of the tank 320. As a result, vapor 2366 condenses on exterior surfaces of the condenser tube 2370 and liquid droplets 2368 from the condensed vapor can drip and / or flow back to the immersion cooling liquid 2364. There may be a plurality of condenser tubes 2370 in tank 320 to condense the vapor 2366 into droplets. Some or all of the condenser tubes 2370 may or may not be located directly over the PCBs 2357. Instead, the condenser tube(s) 2370 can be located near one or more walls of the tank 320, such that the condenser tube(s) 2370 are not directly over the PCBs 2357 on which the packages 2305 are mounted.

[0154] To improve thermal performance in two-phase immersion cooling system 2300, the heat spreader 2352 can include a boiling enhancement coating (BEC) on at least one surface. The BEC can be formed from copper or a copper alloy and can be porous, for example, though BECs can take various forms. In some cases, the BEC is a micro porous copper coating having a thickness from approximately or exactly 50 microns to 500 microns thick (which may be produced by electroplating and / or etching). In some implementations, the BEC comprises a mesh copper layer bonded (e.g., via resistance heating) to at least an outer surface of the heat spreader 2352. In some cases, the BEC isapplied as particulates to at least one smooth surface of the heat spreader 2352 and then subsequently sintered to adhere to one another and to the heat spreader 2352. The BEC provides an improved surface area to contact the immersion cooling liquid 2364 and can increase the heat transfer coefficient from the heat spreader 2352 to the immersion cooling liquid 2364 by up to a factor of 15 versus a smooth surface on the heat spreader 2352. Accordingly, BECs can increase thermal conductivity to, and accelerate the boiling of, the immersion cooling liquid 2364.

[0155] Further implementations of boiling enhancement coatings and enclosures are possible. Additional arrangements, applications, and methods of use of boiling enhancement coatings and enclosures, including with semiconductor dies and 3DIC stacks, are described in the below U.S. Patent Applications.

[0156] U.S. Patent Application No. 18 / 327,615, filed June 1, 2023 and entitled "Boiler Enhancement Coatings with Active Boiling Management,” discloses heat spreader and boiling enhancement enclosure architectures thermally and / or mechanically coupled to one or more semiconductor dies or logic ICs that may be used for passive and / or active management of immersion cooling fluid boiling, including through the use of valves to control pressure of boiling immersion cooling fluid within a boiling enhancement chamber, particularly in paragraphs

[0018] -

[0039] and FIGS. 3-5B. The entirety of U.S. Patent Application No. 18 / 327,615 is incorporated herein by reference.

[0157] U.S. Provisional Patent Application No. 63 / 500,167, filed May 4, 2023 and entitled “Direct to Chip Heat Spreader and Boiler Enhancement Coatings for Microelectronics,” discloses heat spreader and BECs thermally and / or mechanically coupled to one or more semiconductor dies, logic ICs, and / or 3DIC stacks, particularly in paragraphs

[0015] -

[0033] and FIGS. 2A-4. BEC form factors may include graphite heat spreader architectures, vapor chambers, heat pipes, copper plates, fins, and the like. BEC form factors may be thermally and / or mechanically coupled to the one or more semiconductor dies, logic ICs, and / or 3DIC stacks through a thermally conductive epoxy, and may have varying dimensions relative to a surface to which the semiconductor dies and / or logic ICs are mounted. The entirety of U.S. Provisional Patent Application No. 63 / 500,167 is incorporated herein by reference.

[0158] U.S. Patent Application No. 18 / 460,091, filed September 1, 2023 and entitled “Direct to Chip Application of Boiling Enhancement Coating,” discloses BECsand methods for applying BECs to semiconductor dies, logic ICs, and / or 3DIC stacks in accordance with the present technology. In particular, paragraphs

[0024] -

[0046] and FIGS. 2A-5 disclose embodiments of BEC layers, adhesives, solders, sintering, laser ablation, meshes, and other BECs and BEC application methods. The entirety of U.S. Patent Application No. 18 / 460,091 is incorporated herein by reference.

[0159] U.S. Provisional Patent Application No. 63 / 506,945, filed June 8, 2023 and entitled “Vapor- Shedding Structures for Boiler Plates in Two-Phase Immersion Cooling Systems,” discloses structures that may be thermally and / or mechanically coupled to computing hardware such as one or more semiconductor dies, logic ICs, and / or 3DIC stacks to enable the shedding of immersion cooling vapors generated from the boiling of immersion cooling fluid during operation of the computing hardware. In particular, paragraphs

[0021] -

[0039] and FIGS. 3A-5 disclose vapor-shedding structures including varying porosities, constituent materials, and geometries relative to the computing hardware on which they are mounted. The entirety of U.S. Provisional Patent Application No. 63 / 506,945 is incorporated herein by reference.

[0160] U.S. Provisional Application No. 63 / 513,828, filed July 14, 2023 and entitled “Grinding Apparatuses and Methods for Mechanically Modifying Surfaces of Processors to Promote Boiling of a Coolant Liquid,” discloses methods for creating boiling enhancement modifications to surfaces such as the surfaces of computing hardware such as one or more semiconductor dies, logic ICs, and / or 3DIC stacks, particularly in paragraphs

[0036] -

[0095] and FIGS. 2A-8. For example, grooves, patterns, gouges, trenches, or other structures may be added to a surface or lid of a processor, semiconductor die, logic IC, 3DIC stack component, and / or BEC to encourage nucleation sites for bubbles of immersion cooling vapor to form during a cooling process, thus decreasing the thermal resistance between the processor, semiconductor die, logic IC, and / or 3DIC stack component and the surrounding immersion cooling fluid. The entirety of U.S. Provisional Application No. 63 / 513,828 is incorporated herein by reference.

[0161] U.S. Provisional Patent Application No. 63 / 513,829, filed July 14, 2023 and entitled “Electrical Connector Having a Heater to Facilitate Boiling of a Coolant Liquid to Improve Signal Integrity in Immersion Cooling Environment,” discloses heaters for promoting boiling of immersion cooling fluid near electrical connectors such as connections between components of a 3DIC stack and enable improved impedances atthose connectors, particularly in paragraphs

[0019] -

[0052] and FIGS. 1A-3B. The entirety of U.S. Provisional Patent Application No. 63 / 513,829 is incorporated herein by reference.

[0162] U.S. Provisional Patent Application No. 63 / 603,242, filed November 28, 2023 and entitled “Woven Boiler Enhancement Coatings,” provides additional examples of BECs including woven BECs with variable weave patterns, densities, attachment mechanisms, and materials (including copper and tungsten) that may be attached to computing hardware such as one or more semiconductor dies, logic ICs, and / or 3DIC stacks in order to promote more efficient heat transfer and immersion cooling vapor nucleation, particularly in paragraphs

[0031] -

[0055] and FIGS. 3-7. The entirety of U.S. Provisional Patent Application No. 63 / 603,242 is incorporated herein by reference.Conclusion

[0163] While various inventive embodiments have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is deemed to be within the scope of the inventive embodiments described herein. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the inventive teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific inventive embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed. Inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the inventive scope of the present disclosure.

[0164] Also, various inventive concepts may be embodied as one or more methods, of which an example has been provided. The acts performed as part of the method may beordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

[0165] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0166] The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.”

[0167] The phrase “and / or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and / or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and / or” clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to “A and / or B”, when used in conjunction with open-ended language such as “comprising” can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); etc.

[0168] As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of’ or “exactly one of,” or, when used in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.” “Consisting essentially of,” when used in the claims, shall have its ordinary meaning as used in the field of patent law.

[0169] As used herein in the specification and in the claims, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified. Thus, as a nonlimiting example, “at least one of A and B” (or, equivalently, “at least one of A or B,” or, equivalently “at least one of A and / or B”) can refer, in one embodiment, to at least one, optionally including more than one, A, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in yet another embodiment, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other elements); etc.

[0170] In the claims, as well as in the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases “consisting of’ and “consisting essentially of’ shall be closed or semi-closed transitional phrases, respectively, as set forth in the United States Patent Office Manual of Patent Examining Procedures, Section 2111.03.

Claims

CLAIMS1 . A non-transitory computer-readable medium containing instructions thereon, the instructions configuring at least one processor to: determine, for a system, at least a first state of the system in an immersion cooling environment, wherein the system comprises at least two computing components; determine, based at least in part on the first state, a link between the at least two computing components, the link established through at least one communication channel; determine at least one data rate value associated with the at least one communication channel; transmit, based at least in part on the at least one data rate value, data between the at least two computing components using the at least one communication channel; adjust, based at least in part on a second state of the system in the immersion cooling environment, at least one predetermined value associated with the link; and transmit, based on the adjustment, data between the at least two computing components.

2. The medium of claim 1, wherein adjusting the at least one predetermined value comprises modifying a data rate value or at least one channel value associated with the at least one communication channel.

3. The medium of claim 1, wherein the adjusting of the at least one predetermined value associated with link results in altering at least one characteristic of a signal associated with the at least one communication channel.

4. The medium of claim 1, wherein the at least two computing components comprise at least one CPU, at least one GPU, at least one solid state storage device and at least one PCIe device.

5. The medium of claim 1, wherein the immersion cooling environment comprises at least one tank, at least one controller, at least one sensor and coolant liquid, with at least the first state or the second state related to at least one of the at least one tank, theat least one controller, the at least one sensor and the coolant liquid.

6. The medium of claim 5, wherein the at least one sensor is configured to detect at least one system parameter related to voltage, current, temperature, flow rate, humidity, contaminant presence, dielectric constant, permittivity, or permeability.

7. The medium of claim 5, wherein: the at least one sensor is configured to monitor a status of the system or at least one of the at least two computing components; and at least the first state or the second state being associated with at least one operational parameter comprising at least one of a voltage, a frequency, a current, a power, a period, a temperature, a fluid flow, a resistance, a state of charge, or a filtration rate.

8. A method implemented by at least one processor for configuring a computing system, the method comprising: determining, for the computing system, at least a first state of the computing system in an immersion cooling environment, wherein the computing system comprises at least two computing components; determining, based at least in part on the first state, a link between the at least two computing components, the link established through at least one communication channel; determining at least one data rate value associated with the at least one communication channel; transmitting, based at least in part on the at least one data rate value, data between the at least two computing components using the at least one communication channel; adjusting, based at least in part on a second state of the computing system in the immersion cooling environment, at least one predetermined value associated with the link; and transmitting, based on the step of adjusting, data between the at least two computing components.

9. A system for reducing strain on one or more cables, the system comprising: the one or more cables, each cable comprising a flexible housing surrounding a transmission medium; a strain detector disposed on the one or more cables; and a processor communicatively coupled to the strain detector; wherein the processor is configured to: measure a first resistance of the strain detector; and correlate a difference between the first resistance and a baseline resistance of the strain detector with a cable strain applied to at least one of the one or more cables.

10. The system of claim 9, wherein: the strain detector comprises a plurality of segments; and each cable of the one or more cables comprises at least one segment of the plurality of segments.

11. The system of claim 10, wherein each segment of the plurality of segments is communicatively coupled to the processor; and the processor is configured to measure a respective resistance of each segment of the plurality of segments and determine a cable strain applied to a portion of a respective cable on which each respective segment is disposed.

12. The system of claim 9, wherein the strain detector is not disposed on an entire length of the one or more cables.

13. The system of claim 9, wherein the processor is configured to measure at least one of the first resistance or the baseline resistance using a Wheatstone bridge.

14. The system of claim 9, wherein the difference between the first resistance and the baseline resistance is between about 0.05% and about 0.5%.

15. The system of claim 9, wherein the baseline resistance is about 2 kOhm and the difference between the first resistance and the baseline resistance is between about 1 Ohmand about 10 Ohm.

16. The system of claim 9, wherein the strain detector comprises a copper wire.

17. The system of claim 16, wherein the copper wire comprises a diameter of about 0.05 mm.

18. The system of claim 9, further comprising an immersion cooling container at least partially filled with immersion cooling liquid; wherein the one or more cables are at least partially immersed in the immersion cooling liquid.

19. The system of claim 9, wherein: the flexible housing comprises a mesh; and the strain detector is woven into the mesh.

20. The system of claim 9, further comprising:A radio-frequency identification (RFID) tag communicatively coupled to the processor and configured to transmit information about at least one of the one or more cables or the strain detector to the processor.

21. The system of claim 9, wherein at least one of the one or more cables or the strain detector comprises a color coding or a numbering associating the one or more cables with the strain detector.

22. The system of claim 9, wherein the processor is further configured to transmit a notification to an entity in response to the cable strain applied to at least one of the one or more cables exceeding a threshold.

23. The system of claim 22, wherein the entity comprises at least one of a user, a system controller, or a facility controller.

24. A method for reducing strain on one or more cables, the method comprising:deforming one or more cables disposed within an immersion cooling system; measuring, by a processor, a first resistance of a strain detector disposed on the one or more cables; correlating, by the processor, a difference between the first resistance and a baseline resistance of the strain detector with a cable strain applied to at least one of the one or more cables; and transmitting, by the processor, a notification to an entity in response to the cable strain applied to at least one of the one or more cables exceeding a threshold.

25. The method of claim 24, wherein: the strain detector comprises a plurality of segments; and each cable of the one or more cables comprises at least one segment of the plurality of segments.

26. The method of claim 25, wherein each segment of the plurality of segments is communicatively coupled to the processor; and the processor is configured to measure a respective resistance of each segment of the plurality of segments and determine a cable strain applied to a portion of a respective cable on which each respective segment is disposed.

27. The method of claim 24, wherein the strain detector is not disposed on an entire length of the one or more cables.

28. The method of claim 24, wherein the processor is configured to measure at least one of the first resistance or the baseline resistance using a Wheatstone bridge.

29. The method of claim 24, wherein the difference between the first resistance and the baseline resistance is between about 0.05% and about 0.5%.

30. The method of claim 24, wherein the baseline resistance is about 2 kOhm and the difference between the first resistance and the baseline resistance is between about 1 Ohm and about 10 Ohm.

31. The method of claim 24, wherein the strain detector comprises a copper wire.

32. The method of claim 31, wherein the copper wire comprises a diameter of about 0.05 mm.

33. The method of claim 24, wherein: the immersion cooling system further comprises an immersion cooling container at least partially filled with immersion cooling liquid; and the one or more cables are at least partially immersed in the immersion cooling liquid.

34. The method of claim 24, wherein: the one or more cables each comprise a flexible housing surrounding a transmission medium; the flexible housing comprises a mesh; and the strain detector is woven into the mesh.

35. The method of claim 24, further comprising: receiving, by the processor, information about at least one of the one or more cables or the strain detector from a radio-frequency identification (RFID) tag communicatively coupled to the processor.

36. The method of claim 24, wherein at least one of the one or more cables or the strain detector comprises a color coding or a numbering associating the one or more cables with the strain detector.

37. The method of claim 24, further comprising: transmitting, by the processor, a notification to an entity in response to the cable strain applied to at least one of the one or more cables exceeding a threshold.

38. The method of claim 37, wherein the entity comprises at least one of a user, a system controller, or a facility controller.

Images