Constructing a granular / discerning reward model (RM) for RLHF / rlaif

Abstract

Generating and training an artificial intelligence reward model is disclosed. A reward model is trained using a first dataset and providing one or more prompts to two pre-trained AI models such as transformer models or large language models. The two pre-trained AI models each return a response to the prompt. The responses are then ranked as winner-loser pairs using the reward model and the ranked winner-loser pairs are added to the first dataset. The process of providing one or more prompts to two pre-trained AI models may be iterated. The reward model is then re-trained using the updated first dataset.

Description

CONSTRUCTING A GRANULAR / DISCERNINGREWARD MODEL (RM) FOR RLHF / RLAIFCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This Application claims the benefit of U.S. Provisional Patent Application 63 / 585,362, filed September 26, 2023, and entitled “Method and Apparatus for Constructing a Granular / Discerning Reward Model (RM) for RLHF / RLAIF,” which is hereby incorporated by reference in its entirety.

[0002] In cases where the present application conflicts with a document incorporated by reference, the present application controls.BACKGROUND

[0003] As feature sizes and transistor sizes have decreased and more transistors are packed into integrated circuits (ICs) including chips and semiconductor dies, the amount of heat generated by a single IC, such as a microprocessor, has increased. ICs that once were air cooled have evolved into chips needing more heat dissipation than can be provided by air alone. In some cases, immersion cooling of ICs in a tank containing a coolant liquid is employed to maintain ICs at appropriate operating temperatures.

[0004] One type of immersion cooling is two-phase immersion cooling, in which heat from an IC boils the coolant liquid. The boiling creates a coolant-liquid vapor in the tank, which is condensed by cooling coils back to liquid form. Heat from the ICs can then be sunk into the liquid-to-gas phase transitions of the coolant liquid. This allows ICs to be run at a faster rate and allows for more computationally intensive processes than were previously possible without immersion cooling.

[0005] A particularly attractive use case for immersion cooling is in the training and use of artificial intelligence (Al) systems and models. These Al systems typically require costly and time-consuming training. Additionally, Al model training may have extremely high computational demands including high bandwidth, low latency, and large-scale data storage capabilities, which may require efficient waste heat removal that traditional server cooling systems are unable to provide. Accordingly, improvements to training of Al systems are needed to reduce training time and cost,increase Al model accuracy, and reduce or more efficiently utilize computational resources including waste heat remediation systems.OVERVIEW

[0006] The present disclosure relates to transformer artificial intelligence (Al) models such as large language model (LLM) training methods and apparatuses. In a typical LLM training lifecycle, a base Al model is pre-trained using an appropriate training dataset. The pre-trained base Al model may then be finetuned by a two-step process: (1 ) supervised fine tuning (SFT) followed by (2) one or both of reinforcement learning from human feedback (RLHF) or reinforcement learning from artificial intelligence feedback (RLAIF). In the SFT step, the base model (which has been pre-trained on a large dataset or corpus of data; for example, data scraped from the internet) is finetuned using supervised machine learning on a high-quality dataset (for example, a human annotated dataset known to be of high accuracy). These pre-trained datasets can be hundreds of gigabytes (GB) or terabytes (TB) in size.

[0007] The RLHF / RLAIF step then utilizes one or more additional datasets to train a reward model (RM). These datasets may be annotated by humans (in an RLHF process) or by other specialized Al models (in an RLAIF process). RLHF processes in particular may require large groups of trained humans and therefore can cost millions of dollars and take months for hundreds of human annotators. Further, these reward models are typically trained in a winner-loser format; in other words, an RM is presented with two responses to a given prompt and trained to identify one prompt response as “100% correct” and the other response as “0% correct.” However, two winner-loser prompt responses may be qualitatively similar: for example, a first prompt response may be “95% correct” while a second prompt response may be “90% correct.” The winner-loser model therefore has the disadvantage of potentially skewing the accuracy of the RM by more heavily weighting acceptable responses as unacceptable or incorrect, and may reduce the overall quality of responses provided by the RM.

[0008] The inventors have accordingly developed a technique for creating a specialized RM by restructuring SFT and RLHF datasets in concert and generatingan RM using existing Al models such that while the RM is still trained using a winnerloser model, the finetuned RM can discern between qualitatively similar prompt responses with much higher accuracy, or likewise identify that two qualitatively similar prompt responses may be similarly acceptable. This allows for a fully automated reinforcement learning / RM loop as a replacement for the RLHF loop and may eliminate the need for human annotated datasets. This technology may be fully integrable with chatbots such as OpenAI’s ChatGPT or Google’s BARD and may enable a reinforcement learning as a service (RLaaS) offering. This disclosure may additionally carry an advantage of being visible to an end user and therefore may provide improved clarity to a user of an LLM about a quality of a related RM or LLM as a product.

[0009] In some aspects, the techniques described herein relate to a method for training a reward model for machine learning (ML), the method including generating, by a processor, a reward model dataset by segmenting a first plurality of responses to a first prompt into a first winner-loser pair, the first plurality of responses including at least a first response having a first distance on a first side of a reward model classification boundary and at least a second response having a second distance on a second side of the reward model classification boundary. The reward model classification boundary separates a reward space with one or responses on the first side of the reward model classification boundary and defined as winners, and one or more responses on the second side of the reward model classification boundary and defined as losers. The method further includes training, by the processor, the reward model using the reward model dataset. The method further includes providing, by the processor, a prompt to a plurality of trained ML models and receiving a second plurality of responses and automatically segmenting, by the processor using the reward model, the second plurality of responses into a second winner-loser pair. The second plurality of responses includes at least a third response having a third distance on the first side of the reward model classification boundary less than the first distance and at least a fourth response having a fourth distance on the second side of the reward model classification boundary less than the second distance. The method further includes adding, by the processor, the second winner-loser pair to the reward model dataset. The method further includesiteratively repeating, by the processor, the acts of providing, automatically segmenting, and adding one or more times. The method further includes training, by the processor, the reward model using the updated reward model dataset.

[0010] In some aspects, the techniques described herein relate to a method for training a reward model for machine learning, wherein the plurality of trained ML models are large language models.

[0011] In some aspects, the techniques described herein relate to a method for training a reward model for machine learning, wherein the plurality of trained ML models are transformer models.

[0012] In some aspects, the techniques described herein relate to a method for training a reward model for machine learning, wherein the reward model is trained using a logistic regression function.

[0013] All combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are part of the inventive subject matter disclosed herein. The terminology used herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The skilled artisan will understand that the drawings primarily are for illustrative purposes and are not intended to limit the scope of the inventive subject matter described herein. The drawings are not necessarily to scale; in some instances, various aspects of the inventive subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference characters generally refer to like features (e.g., functionally similar and / or structurally similar elements).

[0015] FIG. 1 illustrates a method for training a machine learning (ML) model or agent.

[0016] FIG. 2 illustrates exemplary datasets that may be used to train and / or improve a reward model in accordance with the disclosure.

[0017] FIG. 3 illustrates a method for iteratively segmenting outputs with a reward model and training the reward model in accordance with the disclosure.

[0018] FIG. 4 illustrates a method for training a reward model in accordance with the disclosure.

[0019] FIG. 5 illustrates an environment for expanding a reward model dataset after the reward model dataset has been created.

[0020] FIG. 6 depicts aspects of an immersion-cooling system for dissipating heat from one or more semiconductor die packages via immersion cooling and performing high-computational-power tasks such as training LLMs.

[0021] FIG. 7 is a graphical depiction of a classification boundary of a reward model in accordance with the present disclosure.

[0022] Figure 8 is a block diagram showing some of the components typically incorporated in at least some of the computer systems and other devices on which the techniques described herein operate.DETAILED DESCRIPTION

[0023] FIG. 1 illustrates a method for training a machine learning (ML) model or agent using an immersion-cooled server or other suitable device(s) having a processor. Method 100 may include acts 110-140. In act 110, a processor may collect demonstration or other training data and use the collected data to train a supervised policy. In act 112, a sample prompt may be selected, by the processor, from a prompt dataset. The sample prompt may be then labeled by a labeler running on the processor in act 114, which provides an appropriate response to the prompt. The label may further be paired with the prompt by the processor, which creates a prompt-response pair. Multiple prompt-response pairs may be combined by the processor to form a dataset, which may then be used to train an ML agent. In act 116, the ML model may be fine tuned with a supervised fine tuning (SFT) dataset by the processor.

[0024] In act 120, the processor may collect statistically similar data and train a reward model using the statistically similar data. In act 122, the processor may sample a prompt and several possible responses to the prompt. In act 124, theprocessor may rank several possible responses from best to worst. The reward model may then be trained by the processor using the ranked responses in act 126.

[0025] Those skilled in the art will appreciate that the acts shown in Figure 1 and in each of the flow diagrams discussed below may be altered in a variety of ways. For example, the order of the acts may be rearranged; some acts may be performed in parallel; shown acts may be omitted, or other acts may be included; a shown act may be divided into subacts, or multiple shown acts may be combined into a single act, etc.

[0026] In act 130, the processor may then optimize the policy against the reward model using a proximal policy optimization (PPO) reinforcement learning model. To accomplish this, a new prompt may be sampled from the prompt dataset (the new prompt not having been used in act 112 or any prior act) by the processor at act 132. In act 134, a PPO model may be initialized from a supervised policy and in act 136, the policy may generate an output corresponding to the new prompt sampled in act 132. In act 138, the processor may utilize the reward model to then calculate a reward for the output generated in the previous act. Finally, the processor may use the reward to update the policy using the PPO in act 140.

[0027] FIG. 2 illustrates exemplary SFT and reinforcement learning with human feedback (RLHF) datasets that may be used to train and / or improve a large language model (LLM) or reward model in accordance with the invention. This technology may utilize LLMs that have been pre-trained with a first dataset, such as a large-scale training dataset received from an appropriate source. Examples of sources of large-scale training datasets include the internet, social media networks, internet service providers (ISPs), telecommunications companies, corporations, universities, and the like. In an embodiment, an exemplary LLM may undergo supervised fine tuning prior to the application of one or more methods outlined herein.

[0028] FIG. 2 additionally illustrates examples of a dataset utilized in the training of an exemplary LLM. The techniques herein may utilize one or more SFT datasets, one or more RLHF datasets, or one or more combined / augmented datasets. One or more combined / augmented datasets may equivalently be rewardmodel datasets for training a reward model, which may then be used to train a reinforcement learning (RL) agent or environment.

[0029] An SFT dataset may include two prompts, P1 and P2, each of which may generate five potential responses. Prompt P1 may generate responses R11 , R21 , R31 , R41 , and R51. Prompt P2 may generate responses R12, R22, R32, R42, and R52. An RLHF may be a dataset generated by a labeler (which can be human or Al) and include two responses to an input prompt, the two responses being ranked as winner-loser. FIG. 2 illustrates an exemplary RLHF dataset including a prompt PT having responses R1 and R21’ as well as prompt P2’ having responses R12’ and R22’.

[0030] A plurality of prompts and responses from an SFT dataset may be rearranged as an NC2 (n choose 2) winner-loser dataset for a list of responses to a given prompt. For example, an SFT dataset in FIG. 2 may be rearranged as two 5C2 winner-loser datasets. For a prompt having five responses, 5C2 gives ten possible winner-loser combinations of responses. However, it may be preferable to exclude responses appearing for second or subsequent times, or responses appearing as the winner in one pair but the loser in another pair.

[0031] The combined / augmented dataset in FIG. 2 may represent a dataset where responses from prompt P1 in an SFT dataset are ranked as winner-loser. A human labeler or Al model may compare responses and generate labels for the combined / augmented dataset. For example, responses R11 and R51 to prompt P1 in the combined / augmented dataset may be compared, with R11 labeled the winner and R51 labeled the loser. Likewise, responses R21 and R41 to prompt P1 in the combined / augmented dataset may be compared, with R21 labeled the winner and R41 labeled the loser.

[0032] While Figure 2 and each of the table diagrams discussed below show a table whose contents and organization are designed to make them more comprehensible by a human reader, those skilled in the art will appreciate that actual data structures used to store this information may differ from the table shown, in that they, for example, may be organized in a different manner; may contain more or less information than shown; may be compressed, encrypted, and / or indexed; may contain a much larger number of rows than shown, etc. Additionally, in someembodiments, rather than storing the data shown in the table diagrams in tables, it is stored in semi-structured or unstructured data stores, such as JSON objects.

[0033] Conventional reinforcement learning may include an RL agent (such as a transformer model or LLM) and an environment or simulator to which the reinforcement learning agent is exposed. The environment and / or simulator may be referred to as the reward model (RM). The reward model may provide prompts to which the RL agent may provide responses.

[0034] FIG. 3 illustrates a method for iteratively segmenting outputs with a reward model and training the reward model in accordance with the disclosure. In a first act 310, a prompt and a plurality of related outputs are sampled from a suitable data source. The data source may be different from a source of training data such as pre-training dataset or an SFT dataset. This may have the advantage of not overfitting the data and skewing outputs of the reward model. The outputs may then be ranked using a labeler or similar classifier in act 320. The labeler may use logistic regression or classification to rank the outputs. The outputs may be ranked on any suitable scale, for example 0-5, 0-1 , 0-20, or the like. In an embodiment, the outputs may be divided into pairs and labeled as winner-loser. The outputs may be classified, ranked, or labeled using keywords or other data attributes as necessary.

[0035] Once the outputs are labeled, the reward model may be trained in act 330 using the labeled output data and the associated input prompts used to generate the output data. The reward model may additionally be trained iteratively using newly labeled data. In act 340, outputs including one or more pluralities of responses from one or more trained machine learning models are segmented into one or more winner-loser pairs using the reward model. The segmented one or more winner-loser pairs may then be added to a reward model training dataset and the reward model training dataset may be used to retrain or further train the reward model.

[0036] FIG. 4 illustrates a training method in accordance with the disclosure. In an exemplary training method 400, raw data is obtained, for example from the internet, one or more training data sources, from scraping websites, from information generated by one or more users, and the like. The raw data may be cleaned and / or arranged into a predetermined format for ease or speed of processing. Relevant features of the raw data may be extracted and arranged to create a training dataset.The relevant features may be arranged as a feature matrix. A machine learning (ML) model, also referred to as an ML agent, may be trained with or fitted to the data in act 410 such that inputs to the ML agent produce outputs in accordance with the training dataset. For example, a dataset may be used to tune parameters or weights of nodes in a model such that inputs to the ML model analogous to inputs from the dataset produce outputs from the ML model matching outputs in the dataset.

[0037] The training dataset may be tuned, for example by one or more humans, to ensure the correct inferences are indicated by the dataset. For example, in the event that an element of data in a dataset has been corrupted or accidentally altered, a human may correct the element of data to ensure the wrong inference is not made by the ML model.

[0038] A validation dataset may be input into the ML model to test the ML model for accuracy. The validation dataset may have a known set of inputs and outputs, and if the ML model is properly trained, the ML model will produce the same or similar output to the validation dataset given the inputs from the validation dataset. The data from the validation dataset may be deliberately withheld from training or ML model parameter tuning acts to prevent overfitting of the ML model, which may occur when an ML model indicates relationships between input and output data that do not hold in general.

[0039] After the RL agent is trained with a training dataset, it may be fine tuned using an SFT dataset in act 420. An SFT dataset may include a ranking metric for grading responses, for example a rating of 0-4 or 0-20 allowing for multiple responses to be graded from a single prompt. This may have the downside of repeating prompts. Instead, the SFT dataset may be restructured from a multiple response SFT dataset into a smaller number of winner-loser combinations. For example, a winner loser grouping for prompt P1 may be segmented as R11 -> winner, R51-> loser; R21 -> winner, R41 -> loser; R31 is ignored. This may prevent a prompt from being repeated unnecessarily. The present disclosure has the additional advantage of reducing the risk of repeating responses from the same instance. For example, traditional ML model training using an SFT dataset may grade R11 -> winner, R21 -> loser; R21 -> winner, R31 -> loser. However, this may repeatresponses (R21 appears as both winner and loser), increasing time for training and potentially decreasing response accuracy.

[0040] Upon completion of SFT training, in act 430 a first iteration of a reward model dataset may be manually created or created by a processor using a promptresponse dataset (e.g., a combined / augmented dataset). The combined / augmented dataset may be created by restructuring a multi-response dataset into a winner-loser format with NC2 winner-loser combinations. FIG. 1 illustrates a reward model dataset including prompts PT, P2’, P1 , and P2, along with responses R11’, R21’, R12’, R22’, R11 , R51 , R21 , R41 , R12, R52, R22, and R42. A first plurality of responses may include at least a first response having a first distance above a reward model classification boundary and at least a second response having a second distance below the reward model classification boundary. The reward model classification boundary may separate a reward space with one or responses classified above the reward model classification boundary and defined as winners, and one or more responses classified below the reward model classification boundary and defined as losers. An exemplary reward model classification boundary is illustrated in greater detail with respect to FIG. 7. A first iteration of the reward model dataset may then be used to train a reward model.

[0041] In act 440, the reward model may then be used recursively to further generate a plurality of additional responses from a plurality of high-quality trained ML models, for example LLMs such as Google BARD or OpenAI’s ChatGPT. The responses from each of the plurality of high-quality LLMs are then automatically segmented in act 450 as winner-loser and added to the combined / augmented reward model dataset. After a predetermined number of iterations, the reward model may be trained using the combined / augmented reward model dataset in act 460, where the combined / augmented reward model dataset has been updated with the plurality of responses from high-quality LLMs.

[0042] In this way, a reward model dataset such as a combined / augmented reward model dataset may be recursively generated using high-quality promptresponse data, which may then be used to train the RL agent with a much higher granularity than using traditional RLAIF / RLHF alone. Additionally, this process maybe performed automatically by a processor utilizing Al feedback, eliminating a reliance on expensive and time-consuming human labeling.

[0043] FIG. 5 illustrates an environment for expanding a reward model dataset after the reward model dataset has been created. A prompt is presented to a plurality of LLMs of known high quality. For example, two LLMs such as Google’s BARD and OpenAI’s ChatGPT may then produce a response, giving two responses R1 and R2. The responses may be automatically ranked by a labeler as winner-loser and added to the reward model dataset. This process may be repeated as many times as desired to build a complete reward model dataset. This method has the benefit of increasing the size and granularity of the reward model dataset.

[0044] FIG. 6 depicts aspects of an immersion-cooling system 600 for dissipating heat from one or more semiconductor die packages 605 via immersion cooling and performing high-computational-power tasks such as training LLMs. Immersion-cooling system 600 and computing hardware (such as semiconductor die(s) 650) contained within immersion-cooling system 600 may be used to perform one or more methods in accordance with the present invention. For example, immersion-cooling system 600 and the computing hardware contained therein may be used to perform response segmentation, receive responses, evaluate a distance of a response from a reward model classification boundary, classify responses, alter a dataset (including by adding, removing, or changing data within a dataset), provide one or more prompts to one or more ML models, ML model training, LLM prompt generation, dataset generation, or any suitable operation related to ML models. Each package 605 can include one or more semiconductor dies that produce heat when the system is in operation. The immersion-cooling system 600 in the illustrated example of FIG. 6 is a two-phase immersion-cooling system, though the invention may also be implemented in a single-phase immersion-cooling system.

[0045] The immersion-cooling system 600 includes a container such as tank 620 filled, at least in part, with immersion cooling liquid 664. The immersion-cooling system 600 can further include at least one chiller 680 that flows a heat-transfer fluid through at least one condenser coil 670 that is disposed in the tank 620 and headspace 608. Condenser coil 670 and chiller 680 may be part of a heat exchanger. The packages 605 can be mounted on one or more printed circuit boards(PCBs) 657 that are immersed, at least in part, in the immersion cooling liquid 664. Immersion-cooling system 600 may further include a filter 675 disposed adjacent to the tank 620.

[0046] Filter 675 may include a filtration media, a housing, and a pump configured to force immersion cooling liquid 664 through filter 675 to remove contaminants, particulates, or other impurities that may be added to immersion cooling liquid 664 during use. Filter 675 may be housed outside of tank 620 while being in fluidic communication with immersion cooling liquid 664 in tank 620. Alternatively, filter 675 may be submerged within immersion cooling liquid 664 inside of tank 620.

[0047] Immersion cooling liquid 664 may be a hydrocarbon, a fluoroketone, an oil, or a similar dielectric liquid that will act as an insulator while simultaneously transferring heat from package 605 more efficiently than air. An example of immersion cooling liquid 664 is Novec™ 649 produced by 3MTM. An exemplary immersion cooling liquid 664 used in accordance with embodiments of the present invention may have a dielectric constant baseline value of about 1 .8-2 at a frequency of about 1 kHz.

[0048] In an embodiment of the invention, immersion cooling liquid 664 may be considered unacceptably contaminated if the dielectric constant and / or dielectric loss tangent of immersion cooling fluid being used in an immersion-cooling system 600 differs by a threshold amount as compared to unused or pure immersion cooling liquid 664. For example, immersion cooling liquid 664 may be considered unacceptably contaminated or degraded if the dielectric constant and / or dielectric loss tangent differs by a threshold of 10% or more as compared to unused or pure immersion cooling liquid 664. In an embodiment, a dielectric constant and / or dielectric loss tangent variation threshold may be 20%, 15%, 5%, 3%, 1 %, or any suitable threshold.

[0049] Contamination of the immersion cooling liquid 664 and resulting changes to dielectric constant and / or dielectric loss tangent may alter or negatively impact operation of components within immersion cooling liquid 664 including semiconductor die(s) 650. An altered dielectric constant and / or dielectric loss tangent may result in undesirable cross-talk between components on a PCB,additional noise or reduction in signal strength transmitted along exposed wires of a PCB or semiconductor die(s) 650 submerged in immersion fluid, and / or signal dissipation through the immersion cooling liquid 664. Signal loss may be severe enough that two elements may be effectively represented as being separated by an open circuit despite being physically connected. In an embodiment, a dielectric constant and / or dielectric loss tangent variation threshold may be selected based on an observed or inferred effect on one or more submerged semiconductor die(s) 650. For example, an increase in PCIe bit error rate above an error rate baseline may be correlated with an increase in dielectric constant and / or dielectric loss tangent above a dielectric constant and / or dielectric loss tangent baseline. Accordingly, operation of semiconductor die(s) 650 may be throttled or suspended when a dielectric constant and / or dielectric loss tangent of immersion cooling liquid 664 exceeds a predetermined threshold.

[0050] Changes to dielectric constant and / or dielectric loss tangent may be caused by contaminants within immersion cooling liquid 664. In some cases, changes to dielectric constant and / or dielectric loss tangent may be reversed by filtering the contaminants from immersion cooling liquid 664. In some embodiments, upon detecting an increase in dielectric constant and / or dielectric loss tangent of immersion cooling liquid 664, controller 602 may instruct filter 675 to increase filtration throughput or notify a user that an immersion cooling liquid 664 filtration media may need to be replaced. If a dielectric constant and / or dielectric loss tangent exceeds a predetermined threshold, controller 602 may throttle or shut down one or more semiconductor die(s) 650, generate a notification that immersion cooling liquid 664 should be replaced, trigger an alarm, etc.

[0051] The illustrated example of FIG. 6 is not intended to be to scale. The immersion-cooling system 600 may house and provide immersion cooling liquid 664 to tens, hundreds, or even thousands of packages 605. In some cases, the immersion-cooling system 600 can be small (e.g., the size of a floor unit air conditioner, approximately 1 meter high, 0.5 meter width, 0.5 meter depth or length). In some implementations, the immersion-cooling system can be large (e.g., the size of a van or larger, approximately 2.5 meters high, 2.5 meters width, 4 meters depth or length).

[0052] The immersion-cooling system 600 can also include a controller 602 (e.g., a microcontroller, programmable logic controller (PLC), microprocessor, field- programmable gate array, logic circuitry, memory, or some combination thereof) to manage system operation. Controller 602 can perform various system functions such as monitoring temperatures of system components, cooling fluid level, tank access, chiller operation etc. The controller 602 can further issue commands to control system operation such as executing a start-up sequence, executing a shutdown sequence, assigning workloads among the packages, changing cooling fluid level, changing the temperature of the heat-transfer fluid circulated by the chiller 680, etc. In some implementations, the controller 602 can include (or itself be) a baseboard management controller (BMC) 604. That is, the BMC 604 may monitor and control all aspects of system operation for the immersion-cooling system 600 in addition to monitoring and controlling workloads of the semiconductor dies 650 in the packages 605 cooled by the system. The immersion-cooling system 600 can also include a network interface controller (NIC) 603 to allow the system to communicate over a network, such as a local area network or wide area network. The immersioncooling system 600 can further include a fluid sensor array 690 having a plurality of fluid sensors 610. Fluid sensors 610 may include one or more leak detection sensors at least partially submerged in immersion cooling liquid 664.

[0053] The semiconductor die(s) 650 and can be mounted on and attached to a printed circuit board (PCB) 655 (sometimes referred to as a substrate) in device package 605. The package 605 can be made commercially available as an off-the- shelf (OTS) product. The package 605 can be used for single-phase or two-phase immersion cooling of at least one semiconductor die 650, such as a processor or microprocessor (e.g., a central processing unit (CPU) and / or graphics processing unit (GPU)), voltage regulator (VR), high bandwidth memory (HBM), a digital signal processing (DSP) die, an application-specific integrated circuit (ASIC), field- programmable gate array (FPGA), and / or other densely patterned semiconductor die.

[0054] Semiconductor die(s) 650 or similar computing hardware may be used to perform one or more processes or method acts described herein. For example, semiconductor die(s) 650 may be used to perform response segmentation, receivingresponses, evaluating a distance of a response from a reward model classification boundary, classifying responses, altering a dataset (including by adding, removing, or changing data within a dataset), providing one or more prompts to one or more ML models, ML model training, LLM prompt generation, dataset generation, or any suitable operation related to ML models.

[0055] In the two-phase immersion-cooling system 600 of FIG. 6, heat flows from the semiconductor die 650 where it is generated into the heat spreader 652. The heat spreader 652 is in thermal contact with an immersion cooling liquid 664 that can flow over and extract heat from the heat spreader 652. The amount of heat delivered by the heat spreader 652 to the immersion cooling liquid 664 is enough to boil the immersion cooling liquid 664 that contacts the heat spreader 652 (creating bubbles 665 and potentially creating froth 667 when bubbles 665 reach the surface of immersion cooling liquid 664). The vapor 666 from the boiled immersion cooling liquid 664 can be cooled and condensed back to liquid droplets 668, for example, by the condenser coil 670. The heat-transfer fluid, such as chilled water, from the chiller 680 can be circulated through the condenser coil 670 to lower the temperature of the condenser coil 670 below the condensation point in the headspace 608 of the tank 620. As a result, vapor 666 condenses on exterior surfaces of the condenser coil 670 and liquid droplets 668 from the condensed vapor can drip and / or flow back to the immersion cooling liquid 664. Although a single condenser coil 670 is depicted in FIG. 6, there can be a plurality of condenser coils 670 in tank 620 to condense the vapor 666 into droplets. Some or all of the condenser coils 670 may or may not be located directly over the PCBs 657. Instead, the condenser coil(s) 670 can be located near one or more walls of tank 620, such that the condenser coil(s) 670 are not directly over the PCBs 657 on which the packages 605 are mounted.

[0056] To improve thermal performance in two-phase immersion-cooling system 600, the heat spreader 652 can include a boiling enhancement coating (BEC) on at least one surface. The BEC can be formed from copper or a copper alloy and can be porous, for example, though BECs can take various forms. In some cases, the BEC is a micro porous copper coating having a thickness from approximately or exactly 50 microns to 600 microns thick (which may be producedby electroplating and / or etching). In some implementations, the BEC comprises a mesh copper layer bonded (e.g., via resistance heating) to at least an outer surface of the heat spreader 652. In some cases, the BEC is applied as particulates to at least one smooth surface of the heat spreader 652 and then subsequently sintered to adhere to one another and to the heat spreader 652. The BEC provides an improved surface area to contact the immersion cooling liquid 664 and can increase the heat transfer coefficient from the heat spreader 652 to the immersion cooling liquid 664 by up to a factor of 15 versus a smooth surface on the heat spreader 652. Accordingly, BECs can increase thermal conductivity to, and accelerate the boiling of, the immersion cooling liquid 664.

[0057] The immersion cooling systems disclosed herein may meet various safety and environmental standards set forth by various organizations including, but not limited to, the International Electrotechnical Commission (IEC), and the European Union (EU). For example, the immersion cooling systems disclosed herein may meet the requirements defined by the IEC standard IEC 62368-1 entitled, “Audio / video, information and communication technology equipment - Part 1 : Safety requirements,” which standard is hereby incorporated herein by reference. In another example, the immersion cooling systems disclosed herein may meet the requirements defined by the EU Restriction on Hazardous Substances (RoHS) directive, which standard is hereby incorporated herein by reference.

[0058] FIG. 7 is a graphical depiction of a classification boundary of a reward model in accordance with the present disclosure. The classification boundary defines a selectable metric for segmenting one or more responses into winners and losers. Prompts PT and P2’ may represent data from a traditional RLHF dataset classified or segmented into winner-loser pairs. The two pairs of responses R127R22’ and R117R21’ are separated by relatively large distances above and below a reward model classification boundary as compared to R21 / R41 and R11 / R51 . This may indicate that the responses R127R22’ and R117R21’ are far apart in quality and may further indicate that a reward model tuned using these two response pairs (and similar prompt-response data) may be less able to discern between two responses that are closer in quality, such as R21 / R41 and R11 / R51. The present disclosure provides a reward model with improved granularity compared to traditionaltechniques. By generating winner-loser pairs with a relatively smaller separation in reward space and training a reward model using a dataset incorporating these winner-loser pairs, an ML model trained using the reward model may exhibit greater accuracy, consistency, and reliability when compared to an ML model trained using other methods.

[0059] Figure 8 is a block diagram showing some of the components typically incorporated in at least some of the computer systems and other devices on which the techniques described herein operate. In various embodiments, these computer systems and other devices 800 can include server computer systems, cloud computing platforms or virtual machines in other configurations, desktop computer systems, laptop computer systems, netbooks, mobile phones, personal digital assistants, televisions, cameras, automobile computers, electronic media players, etc. In various embodiments, the computer systems and devices include zero or more of each of the following: a processor 801 for executing computer programs and / or training or applying machine learning models, such as a CPU, GPU, TPU, NNP, FPGA, or ASIC; a computer memory 802 — such as RAM, SDRAM, ROM, PROM, etc. — for storing programs and data while they are being used, including the facility and associated data, an operating system including a kernel, and device drivers; a persistent storage device 803, such as a hard drive or flash drive for persistently storing programs and data; a computer-readable media drive 804, such as a floppy, CD-ROM, or DVD drive, for reading programs and data stored on a computer-readable medium; and a network connection 805 for connecting the computer system to other computer systems to send and / or receive data, such as via the Internet or another network and its networking hardware, such as switches, routers, repeaters, electrical cables and optical fibers, light emitters and receivers, radio transmitters and receivers, and the like. None of the components shown in Figure 8 and discussed above constitutes a data signal per se. While computer systems configured as described above are typically used to support the operations described herein, those skilled in the art will appreciate that they may be implemented using devices of various types and configurations, and having various components.Conclusion

[0060] While various inventive embodiments have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is deemed to be within the scope of the inventive embodiments described herein. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the inventive teachings is / are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation, many equivalents to the specific inventive embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed. Inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the inventive scope of the present disclosure.

[0061] Also, various inventive concepts may be embodied as one or more methods, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

[0062] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0063] The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.”

[0064] The phrase “and / or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e. , elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and / or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and / or” clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to “A and / or B”, when used in conjunction with open-ended language such as “comprising” can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); etc.

[0065] As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of” or “exactly one of,” or, when used in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.” “Consisting essentially of,” when used in the claims, shall have its ordinary meaning as used in the field of patent law.

[0066] As used herein in the specification and in the claims, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations ofelements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, “at least one of A and B” (or, equivalently, “at least one of A or B,” or, equivalently “at least one of A and / or B”) can refer, in one embodiment, to at least one, optionally including more than one, A, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in yet another embodiment, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other elements); etc.

[0067] In the claims, as well as in the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,” and the like are to be understood to be open- ended, i.e. , to mean including but not limited to. Only the transitional phrases “consisting of” and “consisting essentially of” shall be closed or semi-closed transitional phrases, respectively, as set forth in the United States Patent Office Manual of Patent Examining Procedures, Section 2111.03.

[0068] These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.

Claims

CLAIMS1 . A method for training a reward model for machine learning (ML), the method comprising: generating, by a processor, a reward model dataset by segmenting a first plurality of responses to a first prompt into a first winner-loser pair, the first plurality of responses comprising at least a first response having a first distance on a first side of a reward model classification boundary and at least a second response having a second distance on a second side of the reward model classification boundary, the reward model classification boundary separating a reward space with one or responses on the first side the reward model classification boundary and defined as winners, and one or more responses on the second side of the reward model classification boundary and defined as losers; training, by the processor, the reward model using the reward model dataset; providing, by the processor, a prompt to a plurality of trained ML models and receiving a second plurality of responses; automatically segmenting, by the processor using the reward model, the second plurality of responses into a second winner-loser pair, the second plurality of responses comprising at least a third response having a third distance on the first side of the reward model classification boundary less than the first distance and at least a fourth response having a fourth distance on the second side of the reward model classification boundary less than the second distance; adding, by the processor, the second winner-loser pair to the reward model dataset; iteratively repeating, by the processor, the acts of providing, automatically segmenting, and adding one or more times; and training, by the processor, the reward model using the updated reward model dataset.

2. The method of claim 1 , wherein the plurality of trained ML models comprises large language models.

3. The method of claim 1 , wherein the plurality of trained ML models comprises transformer models.

4. The method of claim 1 , wherein the reward model is trained using a logistic regression function.

5. The method of claim 1 , wherein the processor is disposed within an immersion cooling fluid during operation.

6. One or more computer memories collectively having contents configured to cause a computing system to perform a method, the method comprising: generating, by a processor, a reward model dataset by segmenting a first plurality of responses to a first prompt into a first winner-loser pair, the first plurality of responses comprising at least a first response having a first distance on a first side of a reward model classification boundary and at least a second response having a second distance on a second side of the reward model classification boundary, the reward model classification boundary separating a reward space with one or responses on the first side the reward model classification boundary and defined as winners, and one or more responses on the second side of the reward model classification boundary and defined as losers; training, by the processor, the reward model using the reward model dataset; providing, by the processor, a prompt to a plurality of trained ML models and receiving a second plurality of responses; automatically segmenting, by the processor using the reward model, the second plurality of responses into a second winner-loser pair, the second plurality of responses comprising at least a third response having a third distance on the first side of the reward model classification boundary less than the first distance and at least a fourth response having a fourth distance on the second side of the reward model classification boundary less than the second distance;adding, by the processor, the second winner-loser pair to the reward model dataset; iteratively repeating, by the processor, the acts of providing, automatically segmenting, and adding one or more times; and training, by the processor, the reward model using the updated reward model dataset.

7. The one or more computer memories of claim 6, wherein the plurality of trained ML models comprises large language models.

8. The one or more computer memories of claim 6, wherein the plurality of trained ML models comprises transformer models.

9. The one or more computer memories of claim 6, wherein the reward model is trained using a logistic regression function.

10. The one or more computer memories of claim 6, wherein the processor is disposed within an immersion cooling fluid during operation.

11. A computing system comprising: one of more processors; and one or more memories collectively having contents configured to cause a computing system to perform a method, the method comprising: generating, by a processor, a reward model dataset by segmenting a first plurality of responses to a first prompt into a first winner-loser pair, the first plurality of responses comprising at least a first response having a first distance on a first side of a reward model classification boundary and at least a second response having a second distance on a second side of the reward model classification boundary, the reward model classification boundary separating a reward space with one or responses on the first side the reward model classification boundary anddefined as winners, and one or more responses on the second side of the reward model classification boundary and defined as losers; training, by the processor, the reward model using the reward model dataset; providing, by the processor, a prompt to a plurality of trained ML models and receiving a second plurality of responses; automatically segmenting, by the processor using the reward model, the second plurality of responses into a second winner-loser pair, the second plurality of responses comprising at least a third response having a third distance on the first side of the reward model classification boundary less than the first distance and at least a fourth response having a fourth distance on the second side of the reward model classification boundary less than the second distance; adding, by the processor, the second winner-loser pair to the reward model dataset; iteratively repeating, by the processor, the acts of providing, automatically segmenting, and adding one or more times; and training, by the processor, the reward model using the updated reward model dataset.

12. The computing system of claim 11 wherein the plurality of trained ML models comprises large language models13. The computing system of claim 11 , wherein the plurality of trained ML models comprises transformer models.

14. The computing system of claim 11 , wherein the reward model is trained using a logistic regression function.

15. The computing system of claim 11 , wherein the processor is disposed within an immersion cooling fluid during operation.

Images