Efficient knowledge distillation framework for training machine-learned models

Abstract

A computer system that includes the following: one or more processors; and one or more non-transitory computer-readable media storing instructions executable by the one or more processors to cause the computing system to perform one or more operations, the operations comprising: Obtaining a respective input; Obtaining a corresponding output from a machine-learned student sequence processing model that corresponds to the respective input; Generating a multiscale refinement target configured to distill knowledge from a machine-learned teacher sequence processing model while simultaneously reinforcing preferred behavior of the machine-learned student sequence processing model. the multi-scale refinement goal includes the following: a first component based on a divergence metric that, for the respective input, characterizes a comparison of a multitude of predictions of the machine-learned student sequence processing model with a multitude of predictions of the machine-learned teacher sequence processing model; and a second component based on a reinforcement learning signal that is assigned to the respective output; and Updating the machine-learned student sequence processing model based on the multiscale refinement goal.

Description

AREA

[0001] The present disclosure generally relates to machine learning. More specifically, the present disclosure relates to efficiently distilling machine-learned models based on one or more machine-learned reference models. GENERAL STATE OF THE ART

[0002] A computer can receive input(s). The computer can execute instructions to process the input(s) to generate output(s) using a parameterized model. The computer can obtain feedback on its performance in generating outputs with the model. The computer can generate feedback by evaluating its performance. The computer can receive feedback from an external source. The computer can update parameters of the model based on the feedback to improve its performance. In this way, the computer can iteratively "learn" to generate the desired outputs. The resulting model is often called a machine-learned model. SUMMARY

[0003] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description below, or may be learned from the description, or may be learned by practice of the embodiments.

[0004] In one aspect, the present disclosure provides an example computing system. The example computing system may include one or more processors and one or more non-transitory computer-readable media storing instructions executable by the one or more processors to cause the computing system to perform one or more example operations. The example operations may include obtaining a respective input. The example operations may include obtaining a respective output corresponding to the respective input from a machine-learned student sequence processing model. The example operations may include generating a multi-scale refinement target configured toDistill knowledge from a machine-learned teacher sequence processing model while reinforcing preferred behavior of the machine-learned student sequence processing model. The multi-scale refinement objective may include a first component based on a divergence metric,which indicates, for the respective input, a comparison of a plurality of predictions of the machine-learned student sequence processing model with a plurality of predictions of the machine-learned teacher sequence processing model. The multi-scale refinement objective may include a second component based on a reinforcement learning signal associated with the respective output. The example operations may include updating the machine-learned student sequence processing model based on the multi-scale refinement objective. In one aspect, the present disclosure provides an example of one or more non-transitory computer-readable media storing a machine-learned student sequence processing model comprised of a larger,machine-learned teacher sequence processing model. In the example for one or more non-transitory computer-readable media, the machine-learned student sequence processing model was trained by: obtaining a respective input; obtaining a respective output corresponding to the respective input from the machine-learned student sequence processing model; generating a multi-scale refinement objective configured to distill knowledge from the machine-learned teacher sequence processing model while reinforcing preferred behavior of the machine-learned student sequence processing model, wherein the multi-scale refinement objective includes: a first component based on a divergence metric,which, for the respective input, characterizes a comparison of a plurality of predictions of the machine-learned student sequence processing model with a plurality of predictions of the machine-learned teacher sequence processing model; and a second component based on a reinforcement learning signal associated with the respective output; and updating the machine-learned student sequence processing model based on the multi-scale refinement objective.

[0005] In one example aspect, the present disclosure provides an example computer-implemented method for training a machine-learned student sequence processing model. The example method may include obtaining a respective input. The example method may include obtaining a respective output corresponding to the respective input from the machine-learned student sequence processing model. The example method may include generating a multi-scale refinement target configured to distill knowledge from a machine-learned teacher sequence processing model while reinforcing preferred behavior of the machine-learned student sequence processing model.In the example method, the multi-scale refinement objective may include a first component based on a divergence metric that characterizes, for the respective input, a comparison of a plurality of predictions of the machine-learned student sequence processing model with a plurality of predictions of the machine-learned teacher sequence processing model. In the example method, the multi-scale refinement objective may include a second component based on a reinforcement learning signal associated with the respective output. The example method may include updating the machine-learned student sequence processing model based on the multi-scale refinement objective.

[0006] In the example method, the divergence metric may be evaluated using a student value generated by the machine-learned student sequence processing model for one or more portions of the respective output based on the respective input, wherein the value corresponds to a student probability of the one or more portions of the respective output that is dependent on the respective input. In the example method, the divergence metric may be evaluated using a teacher value generated by the machine-learned teacher sequence processing model for the one or more portions of the respective output based on the respective input, wherein the teacher value corresponds to a teacher probability of the one or more portions of the respective output that is dependent on the respective input.The example method may include, for each section of a plurality of sections of the respective output: determining a section-specific divergence metric characterizing a similarity between a student probability distribution over a set of candidate output sections and a teacher probability distribution over the set of candidate output sections, wherein each of the student probability distribution and the teacher probability distribution is dependent on the respective input and one or more sections of the respective output preceding the section; and aggregating the plurality of section-specific divergence metrics for the respective output to obtain the first component.

[0007] In the exemplary method, the teacher probability distributions for each of the section-specific divergence metrics can be generated at least partially in parallel by the machine-learned teacher sequence processing model.

[0008] In the exemplary method, the multi-scale refinement objective may include one or more weighting parameters that weight the respective contributions of the first component and the second component.

[0009] In the exemplary method, the reinforcement learning signal may include data indicating human feedback on an overall quality of the respective output.

[0010] In the example method, the reinforcement learning signal may include data indicative of a score generated by a machine-learned reward model. In the example method, the score may indicate an overall quality of the respective output. In the example method, evaluating the divergence metric may include determining a value of a mixture distribution corresponding to a mixture of a student probability distribution of the machine-learned student sequence processing model and a teacher probability distribution of the machine-learned teacher sequence processing model. In the example method, evaluating the divergence metric may include calculating a first divergence component indicative of a divergence of the student probability distribution with respect to the mixture distribution.In the example method, evaluating the divergence metric may include calculating a second divergence component that indicates a divergence of the teacher probability distribution with respect to the mixture distribution. In the example method, evaluating the divergence metric may include evaluating the divergence metric based on a combination of the first divergence component and the second divergence component.

[0011] In the example method, evaluating the divergence metric based on the first divergence component and the second divergence component may include calculating a weighted combination of the first divergence component and the second divergence component using a weighting parameter. In the example method, adjusting the weighting parameter may cause the divergence metric to interpolate between a mode-seeking behavior and a mean-seeking behavior. The example method may include adjusting the weighting parameter based on a desired output diversity for a task type. In the example method, the weighting may be a hyperparameter learned during training.

[0012] In the example method, the machine-learned teacher sequence processing model may not have been trained using reinforcement learning.

[0013] In the exemplary method, the machine-learned student sequence processing model may have been fine-tuned to achieve a baseline performance threshold prior to training with the multi-scale refinement objective.

[0014] In the example method, the machine-learned student sequence processing model may be characterized by a first number of parameters. In the example method, the machine-learned teacher sequence processing model may be characterized by a second number of parameters. In the example method, the second number of parameters may be greater than the first number of parameters. In the example method, the second number of parameters may be at least 30 times the first number of parameters.

[0015] The example method may include receiving a request from a client computing system to perform an inference task based on the input data. The example method may include obtaining the respective input from the input data. The example method may include generating the respective output using the machine-learned student sequence processing model. The example method may include returning output data based on the respective output to the client computing system and in response to the request. The example method may include receiving feedback data from the client computing system. The example method may include determining the reinforcement learning signal based on the feedback data.

[0016] The example method may include receiving the request and returning the output data in an online process. The example method may include obtaining the plurality of predictions of the machine-learned teacher sequence processing model and updating the machine-learned student sequence processing model based on the multi-scale refinement objective in an offline process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] A detailed discussion of embodiments directed to a person of ordinary skill in the art is set forth in the description, which refers to the accompanying figures, in which: Fig. 1 shows a block diagram of an example system according to example embodiments of the present disclosure; Fig. 2A shows a block diagram of an example training system according to example embodiments of the present disclosure; Fig. 2B shows a block diagram of an example training system according to example embodiments of the present disclosure; Fig. 2C shows a block diagram of an example training system according to example embodiments of the present disclosure; Fig. 2D shows a block diagram of an example training system according to example embodiments of the present disclosure; Fig. 3 shows a flowchart of an example training sequence according to example embodiments of the present disclosure; Fig. 4 shows a flowchart of an example sequence for calculating a divergence metric according to example embodiments of the present disclosure; Fig. 5 is a flowchart illustrating an example method for training a machine-learned model according to example implementations of aspects of the present disclosure; Fig. 6 is a graph showing example test results for an example implementation according to the present disclosure; Fig. 7 is a block diagram of an example processing flow for using a machine-learned model(s) to process an input(s) to generate an output(s), in accordance with example implementations of aspects of the present disclosure; Fig. 8 is a block diagram of an example sequence processing model according to example implementations of aspects of the present disclosure; Fig. 9 is a block diagram of an example technique for ingesting an example input sequence for processing by a sequence processing model, according to example implementations of aspects of the present disclosure; Fig. 10 is a block diagram of an example model development platform according to example implementations of aspects of the present disclosure; Fig. 11 is a block diagram of an example training workflow for training a machine-learned model according to example implementations of aspects of the present disclosure; Fig. 12 is a block diagram of an inference system for operating one or more machine-learned models to perform inference in accordance with example implementations of aspects of the present disclosure; Fig. 13 is a block diagram of an example networked computing system according to example implementations of aspects of the present disclosure; Fig. 14 is a block diagram of an example computing device according to example implementations of aspects of the present disclosure; and Fig. Figure 15 is a block diagram of an example computing device according to example implementations of aspects of the present disclosure. Reference numbers repeated across multiple figures are intended to identify different implementations of the same elements. DETAILED DESCRIPTIONOverview

[0018] In general, the present disclosure is directed to systems for efficiently distilling a machine-learned model based on a reference model. Knowledge distillation includes techniques for training a smaller student model to learn from a more powerful expert teacher model. Knowledge distillation can be useful for reducing the size of machine-learned models to reduce their inference overhead and storage requirements. However, in some cases, the smaller student model may not be sufficiently powerful to fully match the teacher output distribution. This misalignment can lead to situations where the student model cannot sufficiently learn from the teacher model, resulting in suboptimal distillation results.Advantageously, example implementations of the present disclosure can efficiently train a student model based on a larger teacher model while mitigating the effects of output distribution misalignment. In this way, example implementations can, for example, train a machine-learned model capable of performing inference with reduced computational effort.

[0019] Advantageously, to support the alignment of the student model and the teacher model, example implementations according to the present disclosure may train the student model by using the outputs of the teacher model to "correct" outputs organically generated by the student model. For example, the student model may include a machine-learned sequence processing model configured to generate sequences of information based on input context information. The student model may generate new data items in the sequence. The teacher model may process the sequence generated by the student model and, for each data item generated by the student model, generate an example of how the teacher model would have processed the sequence in that step.In this way, the student model can, for example, explore its own output space by organically generating sequences while receiving feedback on its performance from the teacher model.

[0020] Example implementations according to the present disclosure may also incorporate knowledge distillation updates with reinforcement learning. For example, a reinforcement learning scheme may be applied to a model as a form of fine-tuning to refine an alignment of a model's behavior with a desired performance profile. For example, conventional approaches might generally distill a larger reference model as a final step, such as using the outputs of a finalized teacher model (e.g., a teacher model that has completed its training). However, the student model may diverge from the behavior of the teacher model, so the student model could benefit from further fine-tuning itself.

[0021] Advantageously, knowledge distillation training signals can be combined with reinforcement learning signals to compute a multi-scale refinement objective, allowing the student model to be fine-tuned with a reinforcement learning signal while simultaneously receiving a correction from the teacher model. Both training signals can be based on the same inference(s). For example, reinforcement learning feedback can be determined based on an output from the student model, and the same output from the student model can be processed by the teacher model to generate the corrective distillation signal. In this way, example implementations of the present disclosure can achieve, for example, improved efficiency in training smaller, more powerful machine-learned models by allowing both distillation and reinforcement learning to use the same inference runs.

[0022] Furthermore, the multi-scale refinement objective can be "multi-scale" in that it provides feedback signals for a student model output at varying scales. For example, a teacher model can evaluate each section of a student model output. A micro-scale feedback signal can be computed based on the teacher model's section-by-section evaluation. For example, for a student model configured to autoregressively generate information sequences, a teacher model can predict how it would have handled each prediction scenario faced by the student (e.g., the predictions the teacher would have made given the context the student processed in that step). A macro-scale feedback signal can be computed based on a reinforcement learning reward associated with the entire output.For example, the entire output of the student model can be processed by a reward generator, which generates a reward that indicates the quality of the output. For example, the reward generator can issue a reward based on how well an output satisfies a query or statement provided in an input. In this way, for example, a multi-scale refinement objective can provide a training signal that spans a range of granularities.

[0023] Some conventional reinforcement learning techniques combine a reinforcement learning signal with a comparison to an anchor model, which is a control version of the currently trained model. This is intended to prevent the currently trained model from deviating too far from the anchor's performance. In other words, such techniques can hinder the learning of new information by biasing training updates toward the previous control point.

[0024] In contrast, joint distillation and reinforcement learning can accelerate the learning process for the currently trained model by providing the model with new information at multiple scales. Distillation feedback can correct individual reasoning errors of the currently trained model, while reinforcement learning can provide an overall quality signal for the model output. In this way, for example, both feedback signals can work together to allow the currently trained model to receive granular corrections—more generally, high-quality output—in the context of a larger goal. Furthermore, by directly fine-tuning the student model during distillation, a teacher model that has not been fine-tuned (or has been fine-tuned in fewer iterations) can be used for distillation. This can achieve significant efficiencies in various implementations.For example, performing inference using the large teacher model can be computationally expensive. Performing update iterations can also be expensive. By distilling the teacher model into the student model along with fine-tuning the student model (e.g., with a reinforcement learning signal), the benefit of fine-tuning can be granted directly to the student model rather than flowing indirectly through distillation. In this way, for example, improved student model performance can be achieved with greater efficiency.

[0025] The corrective feedback from the teacher model can be obtained by comparing an output of the student model and an output of the teacher model. The output can be a distribution over an output vocabulary. A student model can be trained based on a comparison between an output distribution of the student model and an output distribution of the teacher model. The output distributions can characterize probabilities over a set of possible output values. The comparison can include a divergence metric calculated between a student probability distribution and a reference probability distribution. In some cases, a student model can be trained by optimizing an objective function that includes the divergence metric (e.g., by minimizing a divergence between the student and teacher models).Example divergence metrics described herein can distill student models in a manner robust to discrepancies between the student and the teacher.

[0026] In some cases, a reinforcement learning signal may be generated based on an evaluation of one or more outputs generated by the student. In such cases, the student may be trained using an objective function based on a combination of the reinforcement learning signal and the comparison between the student probability distribution(s) and the teacher probability distribution(s). In some cases, this technique may allow reinforcement learning to be performed concurrently with model distillation, using the same input data and based on the same outputs generated by the student. As described above, this may have the technical effect and benefit of reducing the computational overhead (e.g., power costs) of performing model distillation and reinforcement learning by reducing the total number of inferences that must be made by the student model.

[0027] Example implementations of the present disclosure achieve various technical effects and advantages using a multi-scale refinement objective. In some cases, the systems and methods of the present disclosure may achieve better technical performance than prior systems and methods for a variety of common machine learning tasks (e.g., training models, distributing models, summarization, mathematical reasoning, and machine translation). Furthermore, in some cases, the systems and methods of the present disclosure may achieve similar technical performance with reduced computational effort (e.g., reduced power consumption) compared to prior systems and methods.

[0028] In some example experiments involving summarization, machine-learned models trained according to the present disclosure achieved higher scores for summarization metrics (e.g., greater bigram overlap between model-generated summaries and reference summaries) than similar prior models (e.g., of the same size and architecture) trained in other ways. Furthermore, some machine-learned models trained according to the present disclosure achieved higher implication scores than a much larger machine-learned model (e.g., with 12 times the parameters) trained in another way. This improved technical performance may, in some cases, allow the systems and methods of the present disclosure to perform the same tasks of prior work using computationally less intensive machine-learned models (e.g.,models with a smaller number of parameters) than previous work (e.g., achieve the same bigram overlap and implication scores).

[0029] In some example experiments involving summarization, machine-learned models trained according to the present disclosure achieved similar or better performance (e.g., higher bigram overlap scores) using fewer training iterations than previous machine-learned models. In some cases, machine-learned models of the present disclosure can be trained on less than one percent (e.g., 0.5 percent) of the training examples used to train previous machine-learned models of the same size and architecture and can still achieve better technical performance than the fully trained (100 percent of training examples) previous models. Thus, the systems and methods of the present disclosure can reduce computational overhead (e.g., power consumption) and other costs (e.g.,Reduce the data acquisition costs associated with training machine-learned models by enabling similar performance with fewer training iterations and less training data.

[0030] Furthermore, in some example experiments involving summarization, machine-learned models trained according to the present disclosure achieved similar or better performance (e.g., similar bigram overlap scores) compared to larger models distilled in other ways. For example, a 77 million-parameter model trained according to the present disclosure may, in some cases, achieve technical performance (e.g., bigram overlap score) similar to a 250 million-parameter model trained according to prior methods. It is understood that a smaller model may operate with lower computational overhead (e.g., reduced power costs) compared to a larger model.Thus, the systems and methods of the present disclosure may reduce the computational effort associated with operating a machine-learned model by enabling similar performance using a computationally less intensive (e.g., smaller) model.

[0031] In some example experiments involving machine translation, machine-learned models trained according to the present disclosure achieved higher scores for translation quality metrics (e.g., metrics comparing model-generated translations to human-generated reference translations) than similar prior models (e.g., of the same size and architecture) trained in other ways. This improved result was observed for multiple model sizes (e.g., 77 million parameters, 250 million parameters). Furthermore, since the performance of the machine-learned model scales with model size, it should be understood that the methods of the present disclosure may also enable machine translation of similar quality using smaller models than prior training methods.Thus, the systems and methods of the present disclosure may reduce the computational effort associated with running a machine-learned model by enabling similar machine translation performance using a computationally less intensive (e.g., smaller) model.

[0032] In some example experiments involving mathematical reasoning, machine-learned models trained according to the present disclosure achieved higher accuracy scores on a mathematical reasoning dataset than similar prior models (e.g., of the same size and architecture) trained in other ways. Furthermore, in some example experiments, machine-learned models trained according to the present disclosure achieved similar or higher accuracy scores compared to much larger prior models (e.g., 77 million parameter model of the present disclosure versus 20 billion parameter model, 250 million parameter model of the present disclosure versus 137 billion parameter model). It is understood that a smaller model may operate with less computational overhead (e.g., reduced power costs) compared to a larger model.Thus, the systems and methods of the present disclosure may reduce the computational effort associated with running a machine-learned model by enabling similar performance of mathematical reasoning using a computationally less intensive (e.g., smaller) model.

[0033] It is understood that these results are provided only as an example. Those skilled in the art will recognize that the systems and methods of the present disclosure can be applied to other machine learning tasks and can produce similar technical effects and advantages when applied.

[0034] A technical effect of example implementations of the present disclosure is increased energy efficiency when performing operations using machine-learned models, thereby improving the operation of computers implementing such models. For example, example implementations may provide more energy-efficient runtime execution or inference. In some scenarios, increased energy efficiency may result in less energy being used to perform a given task (e.g., less energy spent maintaining the model in memory, less energy spent performing computations within the model, etc.). In some scenarios, increased energy efficiency may result in more tasks (e.g.,a larger number of tasks, more complex tasks, the same task but with greater accuracy or precision, etc.) can be completed.

[0035] In another example aspect, example implementations may provide more energy-efficient training operations or model updates. In some scenarios, increased energy efficiency may result in less energy being used to perform a given number of update iterations (e.g., less energy spent maintaining the model in memory, less energy spent performing computations within the model such as computing gradients, backpropagating a loss, etc.). In some scenarios, increased energy efficiency may result in more update iterations being completed for a given energy budget (e.g., a larger number of iterations, etc.).In some scenarios, the increased expressiveness provided by model architectures and training techniques of the present disclosure may allow a given level of functionality to be achieved in fewer training iterations, thereby expending a lower energy budget. In some scenarios, the increased expressiveness provided by model architectures and training techniques of the present disclosure may allow an expanded level of functionality to be achieved in a given number of training iterations, thereby expending a lower energy budget.

[0036] In this way, for example, the improved energy efficiency of example implementations of the present disclosure may reduce an amount of pollution or other waste associated with the implementation of machine-learned models and systems, thereby advancing the field of machine learning and artificial intelligence as a whole. The amount of pollution may be reduced overall (e.g., in absolute terms) or on a normalized basis (e.g., energy per task, per model size, etc.). For example, an amount of CO2 released (e.g., by a power source) in connection with the training and execution of machine-learned models may be reduced by implementing more energy-efficient training or inference operations. An amount of heat stress in an environment (e.g.,by the processors / memory locations) can be reduced by implementing more energy-efficient training or inference operations.

[0037] With reference to the figures, exemplary embodiments of the present invention will now be discussed in more detail. Exemplary architectures

[0038] Fig. 1 shows a block diagram of an example system according to example embodiments of the present disclosure. Input data 102 may be processed by a student model 104. Student model 104 may generate student-generated values ​​106 (e.g., values ​​106-1, 106-2, 106-3) based on input data 102. A teacher model 108 may process input data 102 and student-generated values ​​106-1 to generate teacher-generated values ​​110, which may provide a reference point for evaluating student-generated values ​​106-2. A reinforcement learning evaluator 112 may further process student-generated values ​​106-3 (e.g., with respect to input data 102) to evaluate a quality of the output of student model 104. Reinforcement learning evaluator 112 may generate a reinforcement learning signal 114.The training system(s) 116 may process the student-generated values ​​106-2, the teacher-generated values ​​110, and the reinforcement learning signal(s) 114 to generate model updates 118 to train the student model 104 (e.g., using a multi-scale refinement loss).

[0039] The input data 102 may be or include various data types. The input data 102 may include text data, image data, audio data, or combinations thereof. In general, the input data 102 may be or include any data type represented in a computer-readable format. The input data 102 may be or be represented by serialized or sequential data elements. The student model 104 may include one or more machine-learned models. The student model 104 may include various model architectures. An example model architecture for the student model 104 may include a sequence processing model architecture (e.g., a transformer model). For example, the student model 104 may be configured to receive an input sequence and generate an output sequence.For example, the student model 104 may be configured to generate an output sequence in which elements of the output sequence are predicted based on the elements of the input sequence.

[0040] The student model 104 may or may not be pre-trained. For example, the student model 104 may be a pre-trained model (e.g., pre-trained through large-scale unsupervised learning). The student model 104 may be fine-tuned using one or more fine-tuning datasets. The student model 104 may be untrained, such as a model with randomly initialized weights.

[0041] The student-generated values ​​106-1, 106-2, 106-3 (or generally 106) may be, or be based on, any value or values ​​generated by the student model 104. The student-generated values ​​106 may be final output values ​​or intermediate output values. For example, for a student model 104 configured for natural language processing, a final output value may be a word, and an intermediate output value may be a probability associated with that word. For example, a student model 104 may be more generally configured with a set of candidate outputs (e.g., a vocabulary, a set of classes / categories, tokens, other output item(s), etc.), such that a final output value may be the output item itself (e.g., the word in the vocabulary, the class label, an image mask, etc.) and an intermediate output value may be one or more probabilities distributed across the set of candidate outputs.In general, the student model 104 may generate several different final outputs and several different intermediate output values. In one example, intermediate output values ​​may reflect the internal decision-making of the student model 104, while the final output values ​​may reflect the final decision of the student model 104 (e.g., regarding a prediction, etc.).

[0042] The student-generated values ​​106-1, 106-2, 106-3 can all be the same. At least one of the student-generated values ​​106-1, 106-2, 106-3 can be different from others. For example, the student-generated values ​​106-1 can include data in a format consistent with the input data 102 so that the teacher model 108 can process the input data 102 and the student-generated values ​​106-1 together. The student-generated values ​​106-1 can include data in a format interpretable by the teacher model 108. The student-generated values ​​106-2 can include data in a format consistent with the teacher-generated values ​​110 to facilitate comparison between them. The student-generated values ​​106-3 may include data in a format interpretable by the reinforcement learning evaluator 112.In one example, the student-generated values ​​106-3 may include data in a format consistent with the input data 102 such that the reinforcement learning evaluator 112 may process the input data 102 and the student-generated values ​​106-3 to generate a reward.

[0043] The teacher model 108 may include one or more machine-learned models. The teacher model 108 may include various model architectures. An example model architecture for the teacher model 108 may include a sequence processing model architecture (e.g., a transformer model). For example, the teacher model 108 may be configured to receive an input sequence and generate an output sequence. For example, the teacher model 108 may be configured to generate an output sequence in which elements of the output sequence are predicted based on the elements of the input sequence.

[0044] The teacher model 108 may be the same as or different from the student model 104. The teacher model 108 may use one or more model components or model architectures like the student model 104. The teacher model 108 may be completely different from the student model 104. The teacher model 108 may be larger than the student model 104. The student model 104 may have a number of parameters that is smaller (e.g., 5 times, 10 times, 20 times, 30 times, 50 times, 100 times smaller, etc.) than a number of parameters of the teacher model 108. The student model 104 may be characterized by a computational overhead (e.g., inference overhead, pre-training overhead, fine-tuning overhead, memory usage, etc.) that is less than a computational overhead of the teacher model 108.

[0045] The teacher model 108 can be trained on similar data as the student model 104. For example, the teacher model 108 can be a pre-trained model that has already been trained using one or more data sets. The student model 104 can also be previously trained on the same or different data sets.

[0046] The teacher-generated values ​​110 may be, or be based on, any value(s) generated by the teacher model 108. The teacher-generated values ​​110 may be final output values ​​or intermediate output values. For example, for a teacher model 108 configured for natural language processing, a final output value may be a word, and an intermediate output value may be a probability associated with that word. For example, a teacher model 108 may be more generally configured with a set of candidate outputs (e.g., a vocabulary, a set of classes / categories, tokens, other output item(s), etc.), such that a final output value may be the output item itself (e.g., the word in the vocabulary, the class label, an image mask, etc.) and an intermediate output value may be one or more probabilities distributed across the set of candidate outputs.In general, the teacher model 108 may generate several different final outputs and several different intermediate output values. In one example, intermediate output values ​​may reflect the internal decision-making of the teacher model 108, while the final output values ​​may reflect the final decision of the teacher model 108 (e.g., regarding a prediction, etc.).

[0047] In general, the student-generated values ​​106 and the teacher-generated values ​​110 may include values ​​selected or configured to facilitate a comparison between the performance of the student model 104 and the teacher model 108 (e.g., values ​​of similar dimensionality, values ​​corresponding to a similar step in an inference process, etc.). For example, the type of values ​​of each may be selected or obtained by the student and the teacher to represent a decision or the reasoning behind a decision made by the model(s). In this way, for example, a comparison of the values ​​may indicate a relative difference in performance between the models in decision-making.

[0048] A comparison between the student-generated values ​​106 and the teacher-generated values ​​110 can facilitate small-scale, granular feedback to the student model 104 regarding its performance. For example, an exemplary set of student-generated values ​​can correspond to a sequence of output items generated based on the input data 102. The teacher model 108 can process the input data 102 and the student-generated values ​​104 to step through the sequence output by the student model 104 to evaluate which output items the teacher model 108 would have considered at each connection point instead of the item(s) actually output by the student model 104.

[0049] The reinforcement learning evaluator 112 may facilitate broader feedback for the student model 104 more generally. For example, the reinforcement learning evaluator 112 may process the student-generated values ​​106-3 to evaluate an overall quality of the student-generated values ​​106-3. The reinforcement learning evaluator 112 may evaluate the student-generated values ​​106-3 with respect to the input data 102 to evaluate how well the student-generated values ​​106-3 respond to or implement the input data 102 overall.

[0050] The reinforcement learning signal 114 may be or include a reward or cost generated by the reinforcement learning evaluator 112 based on the student-generated values ​​106-3. For example, a reward may indicate how well the student-generated values ​​106-3 respond to or implement a query or instructions specified by the input data 102. The reinforcement learning evaluator 112 may generate the reinforcement learning signal 114 using a machine-learned reward model. For example, a machine-learned reward model may be trained to process an input and an output and generate a reward value associated with the output.

[0051] In various examples, the reinforcement learning signal 114 may include a signal indicating a quality (e.g., high quality, low quality) associated with one or more actions taken by the student model 104 (e.g., one or more student-generated values ​​106-3). In some cases, the quality may be a performance quality associated with one or more tasks (e.g., mathematical accuracy score in a mathematical reasoning task, actual accuracy score in a text generation task, implication score associated with a summarization task, artistic quality score in a creative content generation task, etc.). In some cases, the reinforcement learning signal 114 may include a signal indicating a desirability of one or more outcomes caused by the student model 104 (e.g.,For example, the reinforcement learning signal 114 may include a numerical indicator indicating the quality of an output sequence. In some cases, the reinforcement learning signal 114 may be based on input data 102 and one or more student-generated values ​​106-3.

[0052] A reinforcement learning implementation may include presenting at least one of input data 102 or student-generated values ​​106-3 to a human user to obtain human feedback on the at least one of input data 102 or student-generated values ​​106-3. For example, a computing system may control the student model 104 to perform one or more tasks (e.g., tasks executed or realized using an output(s) of the student model 104) for a human user. The computing system may determine an interaction with a user interface element that indicates feedback associated with the performance of the one or more tasks. The reinforcement learning signal 114 may be generated based on the feedback.

[0053] The training system(s) 116 may be or include one or more software, firmware, or hardware components configured to process student-generated values ​​106-2, teacher-generated values ​​110, and the reinforcement learning signal 114 and generate model updates 118. For example, the training system(s) 116 may calculate a multi-scale refinement objective to evaluate performance of the student model 104 across multiple output scales.

[0054] For example, a component of a multi-scale refinement objective may be based on a comparison of the student-generated value(s) 106-2 and the teacher-generated value(s) 110. For example, the training system(s) 116 may compare the student-generated value(s) 106-2 and the teacher-generated value(s) 110 to evaluate how well the individual output(s) of the student model 104 align with the preferred output(s) of the teacher model 108.

[0055] Another component of a multi-scale refinement objective may be based on the reinforcement learning signal(s) 114. For example, the training system(s) 116 may evaluate a received reward or cost indicated by the reinforcement learning signal(s) 114 to determine a loss. Based on the multi-scale refinement objective, the training system(s) 116 may generate or otherwise output one or more model updates 118. The model updates 118 may include updates to one or more parameters of the student model 104. For example, the model updates 118 may include updating one or more parameters of the student model 104 to optimize a value of an objective that includes the multi-scale refinement objective.In this way, the training system 116 can perform efficient model distillation and reinforcement learning based, for example, on student-generated values ​​104 generated from the same input data 102. Fig. 2A through 2D are block diagrams of an exemplary system for performing distillation along with reinforcement learning for autoregressive sequence processing models. For example, student model 104 may be an autoregressive sequence processing model. Teacher model 108 may be an autoregressive sequence processing model.

[0056] In a first iteration, which Fig. As shown in Figure 2A, an input sequence 200 may include input data 102. The student model 104 may process the input sequence 200 and generate a student distribution 202-s of values. An output value 204 may be obtained based on the student distribution 202-s.

[0057] In a second iteration, which Fig. 2B, the input sequence 200 may include input data 102 and the previously generated portion, the output value 204. The student model 104 may process the updated input sequence 200, including the input data 102 and the output value 204, to generate a student distribution 206-s. Based on the student distribution 206-s, an output value 208 may be obtained. Referring to Fig. 2C, the teacher model 108 may process the complete input sequence 200. The teacher model 108 may generate the teacher distribution 202-t and the teacher distribution 206-t to provide comparison points for the student distributions 202-s and 206-s.

[0058] The input sequence 200 may provide a context for one or more predictions by the student model 104 or the teacher model 108. For example, one or more predictions by the student model 104 or the teacher model 108 may depend on context data in the input sequence 200. The input sequence 200 may include the input data 102. The input sequence 200 may be iteratively updated with generated outputs from the student model 104. In this way, the student model 104 may, for example, process its previous outputs as context over multiple iterations and thereby generate new outputs dependent on the previous outputs. In this way, the student model 104 may, for example, autoregressively generate multiple output values ​​to form an output sequence.

[0059] The student distribution 202-s may reflect the internal decision-making of the student model 104 regarding the input sequence 200. For example, the student distribution 202-s may indicate probability values ​​associated with one or more possible output values ​​of the student model 104 that depend on the input data 102. For example, a student model 104 may be configured to generate an output selected from a set of candidate outputs. The selected output may be selected based on the student distribution 202-s.

[0060] For example, the student distribution 202-s may indicate a score or value generated by the student model 104 that indicates a goodness of a given output candidate with respect to the input sequence 200. The student distribution 202-s may reflect a relative evaluation of the output candidates, so that a specific candidate can be selected by comparing it with the values ​​or scores for the other candidates.

[0061] Although various examples are described herein where the teacher model 108 processes output(s) generated by the student model 104 to correct the student model 104 (e.g., "on-policy" training), it should be understood that in some cases the student model 104 may process output(s) generated by the teacher model 108 (e.g., "off-policy" training "supervised" by the teacher model 108). In some cases, the training of the student model 104 may be performed using a mixture of on-policy and off-policy training examples. In some cases, a randomly sampled indicator value may be used to determine whether to use an on-policy sample or an off-policy sample (e.g., based on a relationship between the indicator value and a threshold, such as to use an on-policy sample when the randomly sampled value is above a threshold).

[0062] In some cases, the student distribution 202-s may correspond to a discrete probability distribution or a continuous distribution (e.g., a probability distribution function). In some cases, the student distribution 202-s may have a plurality of partitions. In some cases, the partitions may correspond to a plurality of items of an output vocabulary of the student model 104. In some cases, the student distribution 202-s may include one or more values ​​(e.g., probability values). In some cases, a respective value of the student distribution 202-s may include or correspond to a respective prediction (e.g., prediction of the next token, prediction of masked speech, prediction of the expected output). Thus, in some cases, the student distribution 202-s may be a discrete probability distribution represented by a plurality of probability values ​​(e.g.,predictions), each of which is assigned to a large number of vocabulary words.

[0063] For example, the Student distribution 202-s may be a probability distribution P over an output space S. The probability values ​​P may be generated by the Student model 104. The probability values ​​P may be generated by the Student model 104 based on the input sequence 200. The probability value for a specific value s ∈ S may indicate a probability for the value s based on the input sequence 200. The probability value for a specific value s ∈ S may indicate a probability that the value s follows the input sequence 200 in a combined sequence (e.g., a context window).

[0064] The output value 204 may be a value generated by the student model 104. The output value 204 may be selected from a set of candidate output values ​​based on the student distribution 202-s. The output value 204 may be, include, or otherwise represent various data types, such as a data type that is the same as or different from the data represented by the input data 102.

[0065] In this way, for example, the student distribution 202-s may indicate predictions associated with one or more possible output values, such that the student distribution 202-s may correspond to the reasoning of the student model 104 about the potential outputs and an assessment of the quality of the outputs. While the output value 204 may indicate a selected decision, other predictions (e.g., generated values) associated with other candidates may reveal other information about the internal reasoning of the student model 104.

[0066] The student distribution 206-s may reflect the internal decision-making of the student model 104 regarding the input sequence 200. For example, the student distribution 206-s may indicate probability values ​​associated with one or more possible output values ​​of the student model 104 that depend on the input data 102 and the output value 204. For example, a student model 104 may be configured to generate an output selected from a set of candidate outputs. The selected output may be selected based on the student distribution 202-s. For example, the student distribution 206-s may indicate a score or value generated by the student model 104 that indicates a goodness of a given output candidate with respect to the input sequence 200.The student distribution 206-s may reflect a relative evaluation of the output candidates, so that a specific candidate can be selected by comparing it with the values ​​or scores for the other candidates.

[0067] In some cases, the student distribution 206-s may correspond to a discrete probability distribution or a continuous distribution (e.g., a probability distribution function). In some cases, the student distribution 206-s may have a plurality of partitions. In some cases, the partitions may correspond to a plurality of items of an output vocabulary of the student model 104. In some cases, the student distribution 206-s may include one or more values ​​(e.g., probability values). In some cases, a respective value of the student distribution 206-s may include or correspond to a respective prediction (e.g., prediction of the next token, prediction of masked speech, prediction of the expected output). Thus, in some cases, the student distribution 206-s may be a discrete probability distribution represented by a plurality of probability values ​​(e.g.,predictions), each of which is assigned to a large number of vocabulary words.

[0068] For example, the Student distribution 206-s may be a probability distribution P over an output space S. The probability values ​​P may be generated by the Student model 104. The probability values ​​P may be generated by the Student model 104 based on the input sequence 200. The probability value for a specific value s ∈ S may indicate a probability for the value s based on the input sequence 200. The probability value for a specific value s ∈ S may indicate a probability that the value s follows the input sequence 200 in a combined sequence (e.g., a context window).

[0069] The output value 208 may be a value generated by the student model 104. The output value 208 may be selected from a set of candidate output values ​​based on the student distribution 206-s. The output value 208 may be, include, or otherwise represent various data types, such as a data type that is the same as or different from the data represented by the input data 102 and the output value 204.

[0070] In this way, for example, the student distribution 206-s may indicate predictions associated with one or more possible output values, such that the student distribution 206-s may correspond to the reasoning of the student model 104 about the potential outputs and an assessment of the goodness of the outputs with respect to the input data 102 and the output value 204. While the output value 208 may indicate a selected decision, other predictions (e.g., generated values) associated with other candidates may reveal other information about the internal reasoning of the student model 104. Similarly, the teacher distribution 202-t may be a distribution generated based on the input data 102. In this way, the teacher distribution 202-t may, for example, represent the reasoning of the teacher model 108 with respect to the same information that was available to the student model 104 when generating the student distribution 202-s.In this way, for example, the teacher distribution 202-t may directly correspond to or otherwise provide a reference comparison point for the student distribution 202-s.

[0071] The teacher distribution 206-t may be a distribution generated based on the input data 102 and the output value 204. In this way, the teacher distribution 206-t may, for example, represent the reasoning of the teacher model 108 with respect to the same information that was available to the student model 104 when generating the student distribution 206-s. In this way, the teacher distribution 206-t may, for example, directly correspond to or otherwise provide a reference comparison point for the student distribution 206-s.

[0072] In some implementations, the teacher distribution(s) can be generated with a lower temperature setting compared to the temperature setting used to generate the student distribution(s). For example, a temperature parameter can influence the shape of the distribution. For example, the probability of predicting the nth token y_n, p(y_n|x), can be determined using softmax with temperature gamma: p(yn|x)=exp(zn / γ)∑i=1Mexp(zi / γ), where z_n can be a logit score for a token y_n. A greedy sample can correspond to a zero temperature. A temperature sample can correspond to a nonzero temperature.

[0073] Any one or more of the student or teacher distributions may be partial or complete distributions. For example, in some situations, a complete distribution over the entire output space may not be known. For example, a top-k set of output values ​​and their corresponding probabilities might be known. The distribution may be based on the top-k set. K may be 1 or more than 1.

[0074] Advantageously, teacher model 108 may generate teacher distributions 202-t and 206-t in parallel. Teacher model 108 may generate teacher distributions 202-t and 206-t by reusing one or more calculations. For example, teacher model 108 may generate teacher distributions 202-t and 206-t without performing multiple forward inference passes. For example, a teacher model 108 may compute one or more matrices that store or encode meanings of input sequence 200 (e.g., a KV cache of a transformer model) and relationships between them. Teacher model 108 may use such values ​​to compute both teacher distributions 202-t and 206-t. Furthermore, the teacher model 108 can generate (an) output distribution(s) by processing an input sequence 200 already generated by the student model 104, without itself generating autoregressive output values.In this way, example implementations of the present disclosure may, for example, reduce or avoid computational overhead for obtaining feedback from a teacher model 108 (which may, for example, be computationally more expensive to implement than the student model 104).

[0075] Section-level divergence metrics 212 and 214 may indicate a difference or similarity between the distributions for a given set of sections of an input(s). For example, the student distribution 202-s and the teacher distribution 202-t may be generated based on one or more sections of a sequence that precede the output value 204. In this way, for example, the student distribution 202-s and the teacher distribution 202-t may be evaluated to obtain a section-level divergence metric 212 that evaluates a directionality of the distribution(s) generated for those sections that precede the output value 204. Similarly, the student distribution 206-s and the teacher distribution 206-t may be generated based on one or more sections of a sequence that precede the output value 208.In this way, for example, the student distribution 206-s and the teacher distribution 206-t may be evaluated to obtain a bin-level divergence metric 214 that evaluates alignment of the distribution(s) generated for those bins preceding the output value 208.

[0076] Section-level divergence metrics 212 and 214 can be combined or aggregated to determine an overall divergence metric. Section-level divergence metrics 212 and 214 can be summed. Section-level divergence metrics 212 and 214 can be averaged (e.g., a weighted average). Other aggregation techniques can be used.

[0077] The section-level divergence metrics 212 and 214 may be combined by aggregating the divergences of the distributions at the respective inference steps. For example, each prediction step may be associated with a probability distribution from the student model and a probability distribution from the teacher model. Aggregated divergence may involve combining divergences from each inference step. Aggregated divergence may involve averaging divergences from each inference step. An exemplary aggregated divergence may be described as follows: D(pT‖pSθ)(y|x):=1Ly∑n=1LyD(pT(⋅|y <n,x)‖pSθ(⋅|y<n,x)), for an input sequence x and an output sequence y, where L_y is the length of a sequence y (e.g., a number of inference steps), P_T is a teacher probability distribution for a next segment of y, P_S is a student probability distribution for a next segment of y.

[0078] The bin-level divergence metrics 212 and 214 may be calculated based on individual values ​​from the distributions (e.g., values ​​obtained from discrete distributions) generated based on one or more bins of the inputs. For example, a student value may be obtained from the student distribution and a teacher value may be obtained from the teacher distribution. These values ​​may be compared, and the comparisons may be aggregated across the distribution(s) to calculate an overall divergence value for the pair of distributions.

[0079] An example divergence metric uses a ratio of the student value and the teacher value. In some cases, calculating the ratio may involve dividing the reference value by the student value. In some cases, an asymmetric (e.g., non-commutative, directed) divergence metric (e.g., Kullback-Leibler divergence) may be calculated using an asymmetric (e.g., non-commutative) function of the value and the reference value.

[0080] In some cases, the divergence metric may be calculated by summing a plurality of comparison values ​​across a plurality of partitions of the probability distribution. In some cases, calculating a respective comparison value may involve multiplying a logarithm of a respective ratio by a respective reference value associated with the respective ratio.

[0081] In some cases, the divergence metric may include a metric that induces mode-seeking behavior when used in an objective function (e.g., a divergence metric where minimizing the divergence metric between a learned probability distribution and a teacher probability distribution would result in a learned probability value being zero whenever a corresponding reference probability value is zero). In some cases, the divergence metric may correspond to a Kullback-Leibler divergence, a Jensen-Shannon divergence, a modified Kullback-Leibler divergence, or a modified Jensen-Shannon divergence. In some cases, a Kullback-Leibler divergence can be computed in both directions (e.g., from a reference distribution to a student distribution, or from a student distribution to a reference distribution).In some cases, the Kullback-Leibler divergence can be calculated in a direction that, when used in an objective function, induces mode-seeking behavior. In some cases, this direction may be referred to as "reverse CL," and calculating a "reverse CL" metric may involve calculating a sum over a vocabulary of (student probability value * log(student probability value divided by teacher probability value)), where a respective teacher probability value may be a probability value (e.g., a prediction probability of the next token) associated with a token / vocabulary word in a teacher vocabulary, and a respective student probability value may be associated with a similar (e.g., same) token / vocabulary word in a student vocabulary that may be similar to (e.g., may overlap with, may be the same as) the teacher vocabulary.

[0082] In some cases, calculating the first divergence may involve summing a plurality of comparison values ​​across a plurality of bins of a distribution. In some cases, a comparison value may include or correspond to a ratio between a respective student probability value and a respective teacher probability value associated with a respective bin (e.g., log(student value / teacher value)). In some cases, a respective bin may be associated with a respective output candidate in an output space of the student model 104. In some cases, the first divergence may correspond to a Kullback-Leibler divergence or a modified Kullback-Leibler divergence. In some cases, the first divergence may correspond to a Kullback-Leibler divergence from the student distribution to the mixture ratio. An exemplary KL divergence may be described as follows: DKL(P‖Q)=∑c∈CP(c)logP(c)Q(c) where P and Q are respective discrete distributions with C bins.

[0083] In some cases, a mixture-based divergence metric (e.g., Jensen-Shannon) can interpolate between mode-seeking and mean-seeking divergence metrics. For example, aligning a student distribution with a teacher distribution based on a mode-seeking divergence metric can result in the student distribution learning to allocate probability mass broadly across the output space, even for regions of the output space that might have low probability under the teacher distribution. This can increase the diversity of the student model's output. However, for student models with relatively limited explanatory power (compared to the teacher model), mode-seeking divergence can help the student model focus its explanatory power on a narrower region of the output space around a mode of the teacher's probability distribution.This can increase aspects of the student model's accuracy by prioritizing alignment with regions of higher probability of the teacher distribution. However, focusing on a narrower region of the output space can reduce output diversity.

[0084] By enabling interpolation between mode-seeking and mean-seeking divergence metrics, a mixture-based divergence metric can enable optimization of a diversity-performance trade-off for different tasks. For example, creative tasks may prioritize diversity, so training a student model to be more creative can use a divergence metric configured to provide mean-seeking behavior. Other tasks (e.g., data retrieval, generating tool calls for tool usage, etc.) may benefit from higher accuracy or precision, so such a student model can use a divergence metric configured to provide more mode-seeking behavior. In general, the balance of mode and mean behavior in training can be optimized for different tasks. For example, a learnable hyperparameter can guide such interpolation.Advantageously, a reinforcement learning signal can provide an independent training signal that can be used to assist in the optimization of the interpolation.

[0085] Calculating a mixture-based divergence metric may include determining a mixture distribution and calculating a first divergence between a student distribution and the mixture distribution. Calculating the divergence metric may include calculating a second divergence between a teacher distribution and the mixture distribution. Calculating the divergence metric may include combining the first and second divergences. In some cases, the mixture distribution may correspond to a mixture of a student probability distribution and a teacher probability distribution. An example mixture-based divergence metric may be described as follows: DJSD(β)(P‖Q)=βDKL(P‖βP+(1−β)Q)+(1−β)DKL(Q‖βP+(1−β)Q) where beta specifies an interpolation parameter (which can be, for example, a learnable hyperparameter).

[0086] In some cases, the mixture distribution can be obtained by interpolating between a student distribution and a teacher distribution based on a weight (e.g., 0.1, 0.9, 0.5, etc.). In some cases, the mixture distribution can correspond to a weighted combination of a student distribution and a teacher distribution (e.g., 0.1 * student + 0.9 * reference). In some cases, the weight can be a hyperparameter that can be learned during training.

[0087] In some cases, evaluating a divergence metric based on the first divergence and the second divergence may involve calculating a weighted combination of the first divergence and the second divergence. In some cases, the weight may be a hyperparameter that may be learned during training. In some cases, the weight used here may be the weight used to generate the mixture distribution. As an illustrative example, in some cases corresponding to a weight of 0.1, a mixture distribution may correspond to 0.1 * student distribution 222 + (1 - 0.1) * teacher distribution 224, and a weighted combination of divergences may correspond to (1 - 0.1) * first divergence + 0.1 * second divergence.

[0088] The output-level reinforcement learning signal 220 may be generated by the reinforcement learning evaluator 112 based on human feedback (e.g., RLHF). The output-level reinforcement learning signal 220 may be generated by the reinforcement learning evaluator 112 based on an output of a machine-learned model (e.g., RLAIF).

[0089] The multi-scale refinement objective 222 may be based on a combination of the divergence metric(s) and the reinforcement learning signal(s). The combination may be weighted using a weighting parameter. An example expression of the multi-scale refinement objective 222 may be as follows: Ex∼X[(1−α)Ey∼pSθ(⋅|x)[r(y)]−αEy∼pS(⋅|x)[D(pT‖pSθ)(y|x)]], where α ∈ [0, 1] controls the magnitude of the distillation loss relative to the RL objective. In some implementations, a multi-scale refinement objective 222 can facilitate maximizing a reward or other performance metric while simultaneously updating the more granular reasoning capabilities directly via distillation. Exemplary procedures

[0090] Fig. 3 shows a flow diagram of an exemplary method 300 for distilling a machine-learned model from a reference model according to embodiments of the present disclosure. The exemplary method 300 may be performed by one or more computing systems (e.g., one or more computing systems as described with respect to Fig. 1 to 15). Although Fig. While Figure 3 shows steps performed in a specific order for illustration and discussion, the methods of the present disclosure are not limited to the specific order or arrangement illustrated. The various steps of the example method 300 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0091] In some cases, method 300 may include obtaining an output from a machine-learned student sequence processing model at 302. The output may include, for example, an output value 204, 208.

[0092] In some cases, method 300 may include obtaining, at 304, a student probability distribution from a machine-learned student sequence processing model (e.g., a student model 104) and a teacher probability distribution from a machine-learned teacher sequence processing model (e.g., a teacher model 108). The teacher probability distribution may be, for example, a teacher distribution 202-t or 206-t. The first probability distribution may be, for example, a student probability distribution 202-s, 206-s.

[0093] In some cases, method 300 may include calculating a divergence metric at 306. In some cases, step 306 may be performed by training system(s) 116. In some cases, the divergence metric may be, for example, the section-level divergence metric 212 or 214.

[0094] In some cases, method 300 may include obtaining a gain learning signal at 308. In some cases, the gain learning signal may be a gain learning signal 114, 216.

[0095] In some cases, method 300 may include updating the first machine-learned sequence processing model at 310. In some cases, step 310 may include training system(s) 116 performing a model update 118 on a student model 104. In some cases, step 310 may include updating the first machine-learned sequence processing model based on an objective function (e.g., performing a gradient update to optimize an objective function). In some cases, step 310 may be performed without backpropagation through the output generation process of step 302. In some cases, the objective function may include the multi-scale refinement objective 222.

[0096] Fig. 4 shows a flow diagram of an exemplary method 400 for joint distillation and reinforcement learning according to embodiments of the present disclosure. The exemplary method 400 may be performed by one or more computing systems (e.g., one or more computing systems as described with respect to Fig. 1 to 15). Although Fig. While Figure 4 shows steps performed in a specific order for illustration and discussion, the methods of the present disclosure are not limited to the specific order or arrangement illustrated. The various steps of the exemplary method 400 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0097] In some cases, the example method 400 may include obtaining a respective input at 402. The input may include the input sequence 200.

[0098] In some cases, the example method 400 may include obtaining, at 404, a respective output corresponding to the respective input from a machine-learned student sequence processing model. The respective output may include an output value 204, 208.

[0099] In some cases, the example method 400 may include, at 406, generating a multi-scale refinement objective configured to distill knowledge from a machine-learned teacher sequence processing model while reinforcing preferred behavior of the machine-learned student sequence processing model. In some cases, the multi-scale refinement objective may include a first component based on a divergence metric characterizing a comparison of a plurality of predictions of the machine-learned student sequence processing model with a plurality of predictions of the machine-learned teacher sequence processing model (e.g., a student distribution); and a second component based on a reinforcement learning signal associated with the respective output.

[0100] In some cases, the example method 400 may include at 408 updating the machine-learned student sequence processing model based on the multi-scale refinement objective.

[0101] In some implementations of the example method 400, the divergence metric is evaluated using a Student score generated by the machine-learned Student sequence processing model for one or more portions of the respective output based on the respective input. For example, the Student score may correspond to a Student probability of the one or more portions of the respective output that depends on the respective input. An example Student probability may be an element of a distribution of probabilities output by the machine-learned Student sequence processing model. The distribution of probabilities may be, for example, a Student distribution 202-s, 206-s, etc.

[0102] In some implementations of the example method 400, the divergence metric is evaluated using a teacher value generated by the machine-learned teacher sequence processing model for the one or more portions of the respective output based on the respective input. For example, the teacher value may correspond to a teacher probability of the one or more portions of the respective output that depends on the respective input. An example teacher probability may be an element of a distribution of probabilities output by the machine-learned teacher sequence processing model. The distribution of probabilities may be, for example, a teacher distribution 202-t, 206-t, etc.

[0103] The example method 400 may include determining a section-specific divergence metric for each of a plurality of sections of the respective output. The section-specific divergence metric may characterize a similarity between a student probability distribution over a set of candidate output sections and a teacher probability distribution over the set of candidate output sections. The section-specific divergence metric may be section-specific because each of the student probability distribution and the teacher probability distribution may depend on the respective input and one or more sections of the respective output preceding that specific section. The example method 400 may include aggregating the plurality of section-specific divergence metrics for the respective output to obtain the first component.

[0104] In some implementations of the example method 400, the teacher probability distributions for each of the section-specific divergence metrics may be generated at least partially in parallel by the machine-learned teacher sequence processing model. For example, because the machine-learned teacher sequence processing model may process the sequence already generated by the machine-learned student processing model to generate distributions, the machine-learned teacher sequence processing model may avoid autoregressively (e.g., sequentially) generating its own output sequence.

[0105] In some implementations of the example method 400, the objective includes one or more weighting parameters that weight the respective contributions of the first component and the second component of the multi-scale refinement objective. In this way, for example, a strength of distillation may be adjusted relative to a strength of the reinforcement learning signal. In this way, for example, training may prioritize granular "correctness" with overall "goodness" of the output.

[0106] In some implementations of the example method 400, the reinforcement learning signal includes data indicative of human feedback regarding an overall quality of the respective performance. The human feedback may be received via one or more user interfaces. The human feedback may be received in real time upon generation of the output(s) using the machine-learned student sequence processing model (e.g., in response to a client system receiving the output(s) generated using the machine-learned student sequence processing model).

[0107] In some implementations of the example method 400, the reinforcement learning signal includes data indicative of a score generated by a machine-learned reward model, where the score indicates an overall quality of the respective output. For example, a machine-learned reward model may evaluate an overall quality of an output(s) generated using the machine-learned student sequence processing model, such as whether the output(s) answer a question, perform a requested task, contain requested information, promote a state of a multi-sequence process, etc.

[0108] In some implementations of the example method 400, evaluating the divergence metric includes determining a value of a mixture distribution corresponding to a mixture of a student probability distribution of the machine-learned student sequence processing model and a teacher probability distribution of the machine-learned teacher sequence processing model. In some implementations of the example method 400, evaluating the divergence metric includes calculating a first divergence component indicating a divergence of the student probability distribution with respect to the mixture distribution. In some implementations of the example method 400, evaluating the divergence metric includes calculating a second divergence component indicating a divergence of the teacher probability distribution with respect to the mixture distribution.In some implementations of the example method 400, evaluating the divergence metric includes evaluating the divergence metric based on a combination of the first divergence component and the second divergence component. For example, a divergence metric based on a mixture distribution may be a Jensen-Shannon divergence.

[0109] In some implementations of the example method 400, evaluating the divergence metric based on the first divergence component and the second divergence component may include calculating a weighted combination of the first divergence component and the second divergence component using a weighting parameter. In some implementations of the example method 400, adjusting the weighting parameter causes the divergence metric to interpolate between a mode-seeking behavior and a mean-seeking behavior. For example, the example method 400 may include adjusting the weighting parameter based on a desired output diversity for a task type. In some implementations of the example method 400, the weighting is a hyperparameter learned during training.

[0110] In some implementations of the example method 400, the machine-learned teacher sequence processing model has not been trained using reinforcement learning. For example, an ultimate goal (e.g., expenditures that increase a reward) may be applied directly to the student model, such as instead of initially training the teacher model to increase the reward (possibly with significant effort). Even if the teacher model is initially trained to increase the reward, in some situations, distillation from the teacher alone may be insufficient to obtain a student model that optimally increases the reward.

[0111] In some implementations of the example method 400, the machine-learned student sequence processing model was fine-tuned to achieve a baseline performance threshold prior to training with the multi-scale refinement objective.

[0112] In some implementations of the example method 400, the machine-learned student sequence processing model is characterized by a first number of parameters, the machine-learned teacher sequence processing model is characterized by a second number of parameters, and the second number of parameters is greater than the first number of parameters. For example, the second number of parameters may be at least 30 times the first number of parameters.

[0113] The example method 400 may include receiving a request from a client computing system to perform an inference task based on the input data. For example, a host computing system may provide a machine learning service that performs inference(s) (or outputs based thereon) in response to client requests. A client request may include some input data for which a corresponding output is desired. The example method 400 may include obtaining the respective input from the input data. The example method 400 may include generating the respective output using the machine-learned student sequence processing model. The example method 400 may include returning output data based on the respective output to the client computing system and in response to the request.The example method 400 may include receiving feedback data from the client computing system. For example, a host computing system may receive an approval, rejection, edit, or other feedback associated with the provided output data. The example method 400 may include determining the reinforcement learning signal based on the feedback data.

[0114] The example method 400 may include receiving the request and returning the output data in an online process. The example method 400 may include obtaining the plurality of predictions of the machine-learned teacher sequence processing model and updating the machine-learned student sequence processing model based on the multi-scale refinement objective in an offline process.

[0115] An example method for determining a divergence metric may involve determining a mixture distribution. In some cases, the mixture distribution may correspond to a mixture of a teacher probability distribution (e.g., teacher distribution 202-t, 206-t) and a student probability distribution (e.g., student distribution 202-s, 206-s). In some cases, the mixture distribution may be obtained by interpolating between the reference probability distribution and the student probability distribution based on a weight (e.g., 0.1, 0.9, 0.5, etc.). In some cases, the mixture distribution may correspond to a weighted combination of the reference probability distribution and the student probability distribution (e.g., 0.1 * student + 0.9 * reference). In some cases, the weight may be a hyperparameter that may be learned during training.

[0116] An example method for determining a divergence metric may include calculating a first divergence between a Student distribution and the mixture distribution. In some cases, calculating the first divergence may include summing a plurality of comparison values ​​across a plurality of partitions of the probability distribution. In some cases, a comparison value may include or correspond to a ratio between a respective Student probability value and a respective mixture probability value associated with a respective partition (e.g., log(Student Value / Mix Value)). In some cases, a respective partition may be associated with a respective token in a vocabulary of the Student model 104. In some cases, the first divergence may correspond to a Kullback-Leibler divergence or a modified Kullback-Leibler divergence.In some cases, the first divergence may correspond to a Kullback-Leibler divergence from the student distribution to the mixing ratio.

[0117] An example method for determining a divergence metric may include calculating a second divergence between a teacher distribution and the mixture distribution. In some cases, calculating the second divergence may include summing a plurality of comparison values ​​across a plurality of partitions of the probability distribution. In some cases, a comparison value may include or correspond to a ratio between a respective teacher probability value and a respective mixture probability value associated with a respective partition (e.g., log(teacher value / mixture value)). In some cases, a respective partition may be associated with a respective token in a vocabulary of the teacher model 210. In some cases, the second divergence may correspond to a Kullback-Leibler divergence or a modified Kullback-Leibler divergence.In some cases, the first divergence may correspond to a Kullback-Leibler divergence from the teacher distribution to the mixing ratio.

[0118] An example method for determining a divergence metric may include evaluating a divergence metric based on the first divergence and the second divergence. In some cases, step 408 may include calculating a weighted combination of the first divergence and the second divergence. The weighted combination may be a JSD metric. In some cases, the weight may be a hyperparameter that may be learned during training. In some cases, the weight used in step 408 may be the weight used in step 402. As an illustrative example, in some cases corresponding to a weight of 0.1, a mixture distribution may correspond to 0.1 * student distribution + (1 - 0.1) * teacher distribution, and a weighted combination of divergences may correspond to (1 - 0.1) * first divergence + 0.1 * second divergence.

[0119] Fig. 5 shows a flowchart of a method 500 for training one or more machine-learned models according to aspects of the present disclosure. For example, an example machine-learned model may include a student model 104.

[0120] One or more portions of the example method 500 may be implemented by a computing system including one or more computing devices, such as the computing systems described with respect to the other figures. Each respective portion of the example method 500 may be performed by any (or any combination of) one or more computing devices. Furthermore, one or more portions of the example method 500 may be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. Fig. 5 shows elements performed in a specific order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any methods discussed herein may be variously adapted, rearranged, expanded, omitted, combined, or modified without departing from the scope of the present disclosure. Fig. 5 is described with reference to elements / terminology described with respect to other systems and figures for exemplary illustrative purposes and are not intended to be limiting. One or more portions of the exemplary method 500 may additionally or alternatively be performed by other systems.

[0121] At 502, the example method 500 may include obtaining a training instance. A set of training data may include a plurality of training instances distributed across multiple datasets (e.g., a training dataset, a validation dataset, or a test dataset). A training instance may be labeled or unlabeled. Although referred to as a "training" instance in the example method 500, it should be understood that runtime inference may form training instances when a model is trained using an evaluation of the model's performance on that runtime instance (e.g., online training / learning). Example data types for the training instance and various associated tasks are described throughout this disclosure.At 504, the example method 500 may include processing the training instance using one or more machine-learned models to generate an output. The output may be obtained directly from the one or more machine-learned models or may be a downstream result of a chain of processing operations that includes an output from the one or more machine-learned models.

[0122] At 506, the example method 500 may include receiving a score signal associated with the output. The score signal may be obtained using a loss function. Various definitions of the loss may be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, contrastive loss, or various other loss functions. The score signal may be computed using known ground truth labels (e.g., supervised learning), predicted or estimated labels (e.g., semi- or self-supervised learning), or without labels (e.g., unsupervised learning). The score signal may be a reward (e.g., for reinforcement learning). The reward may be computed using a machine-learned reward model configured to generate rewards based on a received output(s).The reward can be calculated using feedback data that describes human feedback on the output(s).

[0123] At 508, the example method 500 may include updating the machine-learned model using the score signal. For example, in some embodiments, values ​​for parameters of the machine-learned model(s) may be learned using various training or learning techniques, such as, for example, backpropagation. For example, the score signal may be backpropagated from the output (or other source of the score signal) through the machine-learned model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the score signal with respect to the parameter value(s). For example, a system(s) may be trained end-to-end with one or more machine-learned models. Gradient descent techniques may be used to iteratively update the parameters over a number of training iterations.In some implementations, performing error backpropagation may involve performing truncated backpropagation over time. The example method 500 may involve implementing a number of generalization techniques (e.g., weight decay, dropouts, etc.) to improve the generalization ability of the currently trained models.

[0124] In some implementations, the example method 500 may be implemented to train a machine-learned model from an initialized state to a fully trained state (e.g., when the model has a desired performance profile, such as based on accuracy, precision, recall, etc.).

[0125] In some implementations, the example method 500 may be implemented for specific stages of a training procedure. For example, in some implementations, the example method 500 may be implemented to pre-train a machine-learned model. For example, pre-training may involve large-scale training on potentially noisy data to achieve a broad base of performance levels across a variety of tasks / data types. In some implementations, the example method 500 may be implemented to fine-tune a machine-learned model. For example, fine-tuning may involve smaller-scale training on higher-quality (e.g., labeled, curated, etc.) data. Fine-tuning may affect all or a portion of the parameters of a machine-learned model.For example, different sections of the machine-learned model can be "frozen" for certain training stages. For example, parameters associated with an embedding space can be "frozen" during fine-tuning (e.g., to retain information learned from a broader domain(s) than those present in the fine-tuning dataset(s). An example fine-tuning approach involves reinforcement learning. Reinforcement learning can be based on user feedback on model performance during use. Example results

[0126] Fig. Figure 6 shows results from an example test using a multi-scale refinement objective to train a machine-learned student model for a summarization task. In summarization, model-generated summaries are generally intended to be factually consistent with their input documents. However, distillation alone may not improve factual consistency, as even large models can hallucinate and generate inconsistent summaries.

[0127] Fig. Figure 6 provides two baselines. RLEF* corresponds to a T5 base model trained with the RLAIF procedure of Roit et al. (2023), where the student is regularized to the original student model itself instead of the teacher. The teacher corresponds to the T5-XL model (12x larger than the T5 base).

[0128] The labels indicated with alpha values ​​refer to exemplary implementations of the present disclosure in which a multi-scale refinement objective of the following formulation was used to distill knowledge from T5-XL and simultaneously train with an RLAIF reward from a textual implication score from a T5-XXL NLI classifier with weighting parameter alpha: Ex∼X[(1−α)Ey∼pSθ(⋅|x)[r(y)]−αEy∼pS(⋅|x)[D(pT‖pSθ)(y|x)]], where JSD was used with a 0.9 beta.

[0129] As in Fig. As shown in Figure 6, joint distillation with RL fine-tuning improves factual consistency compared to the much larger teacher model, while strong improvements in summarization quality are obtained for the distilled student model compared to RLEF* alone.

[0130] Other exemplary test results are discussed below.

[0131] An example training algorithm is shown below in pseudocode:

[0132] In some example summarization experiments according to the present disclosure, a student model 104 may be distilled from a teacher model 108 according to Algorithm 1 based on portions of one or more training examples from one or more summarization datasets (e.g., 1,000 examples, 10,000 examples, 50,000 examples, 200,000 examples, etc.). In some cases, the student model 104 may be distilled without using ground-truth summaries. In some cases, a student model 104 distilled according to the present disclosure (e.g., student data fraction = 1, divergence = Jensen-Shannon based on 90 percent mixing) using only 1,000 training examples and without using ground truth summaries may outperform previous models of the same size trained on 200,000 training examples with ground truth summaries of the same summarization dataset (e.g.,achieve higher bigram overlap scores). An even greater improvement (e.g., 6x greater) may be possible if a student model 104 according to the present disclosure is trained on a larger number (e.g., 10,000, 50,000) of training examples.

[0133] In some example summarization experiments according to the present disclosure, student models 104 of different sizes may be compared to models of the same size trained according to other methods. In such experiments, models trained according to the present disclosure may consistently outperform models of the same size trained according to other methods. In some experiments according to the present disclosure, a 77 million parameter model distilled according to the present disclosure achieved similar performance (e.g., similar bigram overlap score) as a 250 million parameter model distilled according to other methods.

[0134] In some example summarization experiments according to the present disclosure, student models 104 trained according to Algorithm 1 may be compared to larger pre-trained models (e.g., a 540 billion transformer model called with a few prompts). In some experiments, a 77 million parameter student model 104 may achieve similar performance to a pre-trained model with approximately 7,000 times as many parameters.

[0135] In some example summarization experiments according to the present disclosure, student models 104 may be trained according to the methods described herein based on a combination of the divergence metrics of Algorithm 1 and a reinforcement learning signal. In some experiments involving summarization, the reinforcement learning signal may be a textual implication feedback score indicating whether a summary of student model 104 is implied by the input context it is summarizing. In some cases, student model 104 may be trained using an objective function that includes both a divergence metric and a reinforcement learning signal. In some cases, the objective function may be a weighted sum (e.g., (1 minus alpha) * (RL signal) + alpha * divergence).In some experiments, a student model 104 trained with a high alpha of 0.5 may achieve a higher implication score than the teacher model 108 from which it learned, even if the teacher model 108 is much larger (e.g., 38 times as many parameters) than the student model 104. In other experiments, a student model 104 trained with a low alpha of 0.05 may achieve much higher implication scores than the teacher model 108 (e.g., more than 40 percent implied versus less than 20 percent implied) and still achieve a substantial improvement in summarization quality (more than 3 points improvement in the bigram overlap score) over a baseline model not distilled according to the present disclosure.

[0136] In some example experiments according to the present disclosure, various values ​​were used as the softmax temperature of the teacher model 108. In some experiments, the student models 104 trained using a teacher temperature below 1.0 (e.g., below 0.5) showed a significant improvement on summarization tasks (e.g., higher bigram overlap score) compared to the student models 104 trained with a teacher temperature of 1.0.

[0137] In some examples of machine translation experiments according to the present disclosure, 77 million and 250 million parameter student models 104 were trained according to Algorithm 1 using various mode-seeking divergence metrics (e.g., "inverse KL" Kullback-Leibler divergence, modified Jensen-Shannon divergence based on 90 percent mixing, modified Jensen-Shannon divergence based on 10 percent mixing, etc.). In such experiments, the student models 104 trained according to the present disclosure performed better on machine translation tasks (e.g., higher scores on one or more machine translation metrics based on n-gram overlap between model-generated translations and human-generated reference translations) than models distilled according to other methods.

[0138] In some example mathematical reasoning experiments according to the present disclosure, the input data 102 may include one or more short and / or mental prompting examples adapted to a mathematical reasoning. In some cases, respective input data 102 may further include a mathematical reasoning problem from a mathematical reasoning dataset (e.g., appended to an input context immediately after a short or mental prompt). In such experiments, 250-million-parameter student models 104 trained according to the present disclosure (e.g., student data fraction = 1, "inverse-KL" Kullback-Leibler divergence) outperformed models of the same size trained with other methods (e.g., higher mathematical accuracy).In some example experiments, 250 million parameter student models trained according to the present disclosure achieved similar performance to a 137 billion parameter model (548 times larger) that was prompted using a mental prompt.

[0139] These and other exemplary results are further described in Agarwal et al., Generalized Knowledge Distillation for Auto-regressive Language Models, arXiv:2306.13649v2 (October 3, 2023), which is hereby incorporated by reference. Examples of machine-learned models

[0140] Fig. 7 is a block diagram of an example processing flow for using a machine-learned model(s) 1 to process an input(s) 2 to generate an output(s) 3.

[0141] The machine-learned model(s) 1 may be or include one or more machine-learned models or model components. Examples of machine-learned models may include neural networks (e.g., deep neural networks). Examples of machine-learned models may include nonlinear models or linear models. Examples of machine-learned models may use other architectures instead of or in addition to neural networks. Examples of machine-learned models may include decision tree-based models, support vector machines, hidden Markov models, Bayesian networks, linear regression models, k-means clustering models, etc.Example neural networks may include feedforward neural networks, recurrent neural networks (RNNs), including long short-term memory (LSTM)-based recurrent neural networks, convolutional neural networks (CNNs), diffusion models, generative adversarial networks, or other forms of neural networks. Examples of neural networks may include deep neural networks. Some examples of machine-learned models may utilize an attention mechanism, such as self-attention. For example, some machine-learned models may include multi-headed self-attention models.

[0142] The machine-learned model(s) 1 may include a single or multiple instances of the same model configured to operate on data from the input(s) 2. The machine-learned model(s) 1 may include an ensemble of different models that can cooperatively interact to process data from the input(s) 2. For example, the machine-learned model(s) 1 may employ a structure with a mixture of experts. See, e.g., Zhou et al., Mixture-of-Experts with Expert Choice Routing, ARXIV:2202.09368v2 (October 14, 2022).

[0143] The machine-learned model(s) 1 may be an example of a student model 104 or a teacher model 108. Various configurations of the machine-learned model(s) 1 described herein should also be understood to describe various configurations of a student model 104 or a teacher model 108.

[0144] Input(s) 2 may generally contain or otherwise represent different data types. Input(s) 2 may contain one data type or many different data types. Output(s) 3 may be data of the same type(s) or different data types compared to input(s) 2. Output(s) 3 may contain one data type or many different data types.

[0145] Example data types for input(s) 2 or output(s) 3 include natural language text data, software code data (e.g., source code, object code, machine code, or any other form of computer-readable instructions or programming languages), machine code data (e.g., binary code, assembly code, or other forms of machine-readable instructions directly executable by a central processing unit of a computer), assembly code data (e.g., low-level programming languages ​​that use symbolic representations of machine code instructions to program a processing unit), genetic data or other chemical or biochemical data, image data, audio data, audiovisual data, haptic data, biometric data, medical data, financial data, statistical data, geographical data, astronomical data, historical data, sensor data in general (e.g.,Digital or analog values, such as voltage values ​​or other absolute or relative level measurements from a real or artificial input, such as an audio sensor, light sensor, displacement sensor, etc., etc. The data can be raw or processed and in any format or schema.

[0146] In multimodal inputs 2 or outputs 3, example combinations of data types include image data and audio data, image data and natural language data, natural language data and software code data, image data and biometric data, sensor data and medical data, etc. It is understood that any combination of data types may be present in an input 2 or an output 3.

[0147] An example input 2 may include one or more data types, such as the example data types mentioned above. An example output 3 may include one or more data types, such as the example data types mentioned above. The data type(s) of input 2 may be the same as or different from the data type(s) of output 3. It should be understood that the example data types mentioned above are provided for illustrative purposes only. Data types contemplated within the scope of the present disclosure are not limited to the examples mentioned above. Example of machine-learned sequence processing models

[0148] Fig. 8 is a block diagram of an example implementation of an example machine-learned model configured to process sequences of information. For example, an example implementation of machine-learned model(s) 1 may include machine-learned sequence processing model(s) 4. An example system may pass input(s) 2 to sequence processing model(s) 4. Sequence processing model(s) 4 may include one or more machine-learned components. Sequence processing model(s) 4 may process the data from input(s) 2 to obtain input sequence 5. Input sequence 5 may include one or more input elements 5-1, 5-2, ..., 5-M, etc., obtained from input(s) 2.The sequence processing model 4 may process the input sequence 5 using a prediction layer(s) 6 to generate an output sequence 7. The output sequence 7 may include one or more output elements 7-1, 7-2,..., 7-N, etc., generated based on the input sequence 5. The system may generate the output(s) 3 based on the output sequence 7.

[0149] The sequence processing model(s) 4 may include one or more components of the machine-learned model configured to ingest, generate, or otherwise reason about sequences of information. For example, some example sequence processing models in the text domain are referred to as “Large Language Models” or LLM (Large Language Model). See, e.g., PaLM 2 Technical Report, GOOGLE, https: / / ai.google / static / documents / palm2techreport.pdf (nd). Other example sequence processing models may operate in other domains, such as, for example, image domains, see, e.g., Dosovitskiy et al., An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale, ARXIV:2010.11929v2 (June 3, 2021), audio domains, see, e.g., B. Agostinelli et al., MusicLM: Generating Music From Text, ARXIV:2301.11325v1 (26 January 2023), biochemical domains, see e.g. Jumper et al., Highly Accurate Protein Structure Prediction with AlphaFold, 596 Nature 583 (August 26, 2021). The sequence processing model(s) 4 may process one or more data types simultaneously. The sequence processing model(s) 4 may include relatively large models (e.g., more parameters, computationally expensive, etc.), relatively small models (e.g., fewer parameters, computationally simple, etc.), or both.

[0150] In general, the sequence processing model(s) 4 may obtain the input sequence 5 using data from the input(s) 2. For example, the input sequence 5 may include a representation of data from the input(s) 2 in a format understood by the sequence processing model(s) 4. One or more machine-learned components of the sequence processing model(s) 4 may ingest the data from the input(s) 2, parse the data into pieces compatible with the processing architectures of the sequence processing model(s) 4 (e.g., via "tokenization"), and project the pieces into an input space associated with the prediction layer(s) 6 (e.g., via "embedding").

[0151] The sequence processing model(s) 4 may receive the data from the input(s) 2 and parse the data into a sequence of elements to obtain the input sequence 5. For example, a portion of the input data from the input(s) 2 may be decomposed into parts that together represent the contents of the portion of input data. The parts may provide the elements of the sequence.

[0152] Elements 5-1, 5-2,..., 5-M can, in some cases, represent building blocks for capturing or expressing meaningful information in a specific data domain. For example, the elements can describe "atomic units" across one or more domains. For example, for a textual input source(s), the elements can correspond to groups of one or more words or subword components, such as sentences of one or more characters.

[0153] For example, elements 5-1, 5-2,..., 5-M may represent tokens obtained using a tokenizer. For example, a tokenizer may process a given portion of an input source and output a series of tokens (e.g., corresponding to input elements 5-1, 5-2,..., 5-M) that represent the portion of the input source. Various approaches to tokenization may be used. For example, a textual input source(s) may be tokenized using a byte pair encoding (BPE) technique. See, for example, B. Kudo et al., SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, PROCEEDINGS OF THE 2018 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING (System Demonstrations), pages 66-71 (October 31-November 4, 2018), https: / / aclanthology.org / D18-2012.pdf.(An) image-based input source(s) can be tokenized by extracting and serializing patches from an image.

[0154] In general, any data type can be serialized and processed to the input sequence 5. It is understood that the Fig. 7, the element(s) 5-1, 5-2,..., 5-M shown may be the tokens or their embedded representations.

[0155] The prediction layer(s) 6 may predict one or more output elements 7-1, 7-2,..., 7-N based on the input elements. The prediction layer(s) 6 may include one or more machine-learned model architectures, such as one or more layers of learned parameters that manipulate and transform the input(s) to extract higher-level meaning from and relationships between the input elements 5-1, 5-2,..., 5-M. In this way, for example, the prediction layer(s) 6 may predict new output element(s) in light of the context provided by the input sequence 5.

[0156] Prediction layer(s) 6 can evaluate associations between segments of input sequence 5 and a specific output element. These associations can predict the probability that a specific output follows the input context. As an example, consider the text block: "The carpenter's toolbox was small and heavy. It was full of _." Prediction layer(s) 6 can identify that "He" refers back to "toolbox" by determining a relationship between the respective embeddings. The example prediction layer(s) 6 can also link "He" to the attributes of the toolbox, such as "small" and "heavy." Based on these associations, prediction layer(s) 6 can, for example, assign a higher probability to the word "nails" than to the word "sawdust."

[0157] A transformer is an example architecture that can be used in the prediction layer(s) 4. See, e.g., Vaswani et al., Attention Is All You Need, ARXIV:1706.03762v7 (August 2, 2023). A transformer is an example of a machine-learning model architecture that uses an attention mechanism to compute associations between elements within a context window. The context window may include a sequence containing the input sequence 5 and optionally one or more output elements 7-1, 7-2,..., 7-N. A transformer block may include one or more attention layers and one or more post-attention layers (e.g., feedforward layers, such as a multi-layer perceptron).

[0158] Prediction layer(s) 6 may include other machine-learned model architectures in addition to or instead of transformer-based architectures. For example, recurrent neural networks (RNNs) and long-short-term memory models (LSTMs), as well as convolutional neural networks (CNNs), may also be used. In general, prediction layer(s) 6 may utilize various types of artificial neural networks capable of understanding or generating sequences of information.

[0159] The output sequence 7 may include or otherwise represent the same or different data types as the input sequence 5. For example, the input sequence 5 may represent text data and the output sequence 7 may represent text data. The input sequence 5 may represent image, audio, or audiovisual data, and the output sequence 7 may represent text data (e.g., describing the image, audio, or audiovisual data). It should be understood that the prediction layer(s) 6 and any other interstitial model components of the sequence processing model(s) 4 may be configured to receive a variety of data types in the input sequence(s) 5 and output a variety of data types in the output sequence(s) 7.

[0160] The output sequence 7 can have various relationships to the input sequence 5. The output sequence 7 can be a continuation of the input sequence 5. The output sequence 7 can be complementary to the input sequence 5. The output sequence 7 can translate, transform, augment, or otherwise modify the input sequence 5. The output sequence 7 can respond to, evaluate, confirm, or otherwise react to the input sequence 5. The output sequence 7 can implement an instruction provided via the input sequence 5 (or describe instructions for implementation).

[0161] The output sequence 7 can be generated autoregressively. For example, for some applications, an output of one or more prediction layers 6 can be passed through one or more output layers (e.g., a softmax layer) to obtain a probability distribution over an output vocabulary (e.g., a text or symbol vocabulary) that depends on a set of input items in a context window. In this way, for example, the output sequence 7 can be generated autoregressively by sampling a likely next output item, adding the item to the context window, regenerating the probability distribution based on the updated context window, sampling a likely next output item, and so on.

[0162] Output sequence 7 can also be generated non-autoregressively. For example, multiple output elements of output sequence 7 can be predicted jointly without any explicit sequential dependence on each other. See, for example, Saharia et al., Non-Autoregressive Machine Translation with Latent Alignments, arXiv:2004.07437v3 (November 16, 2020).

[0163] The output sequence 7 may include one or more sections or one or more elements. In an example content generation configuration, the output sequence 7 may include multiple elements corresponding to multiple sections of a generated output sequence (e.g., a textual sentence, values ​​of a discretized waveform, computer code, etc.). In an example classification configuration, the output sequence 7 may include a single element associated with a classification output. For example, an output "vocabulary" may include a set of classes into which an input sequence is to be classified. For example, a vision transformer block may pass information about a latent state to a multi-layer perceptron, which outputs a probable class value associated with an input image.

[0164] Fig. 9 is a block diagram of an exemplary technique for ingesting an exemplary input sequence 8. The input sequence 8 may include various functional elements that are part of the model infrastructure, such as an element 8-0 obtained from a task indicator 9 that signals to any model(s) processing the input sequence 8 that a specific task is currently being performed (e.g., to support adapting a performance of the model(s) to that specific task). The input sequence 8 may include various data elements from different data modalities. For example, an input modality 10-1 may include a modality of data. A data sequence model 11-1 may process data from the input modality 10-1 to project the data into a format compatible with the input sequence 8 (e.g.,one or more vectors dimensioned according to the dimensions of the input sequence 8) to obtain elements 8-1, 8-2, 8-3. Another input modality 10-2 may include a different modality of data. A data sequence model 11-2 may project data from the input modality 10-2 into a format compatible with the input sequence 8 to obtain elements 8-4, 8-5, 8-6. Another input modality 10-3 may include yet another modality of data. A data sequence model 11-3 may project data from the input modality 10-3 into a format compatible with the input sequence 8 to obtain elements 8-7, 8-8, 8-9.

[0165] The input sequence 8 may be the same as or different from the input sequence 5. The input sequence 8 may be a multimodal input sequence containing elements representing data from different modalities using a common dimensional representation. For example, an embedding space may have P dimensions. The input sequence 8 may be configured to contain a plurality of elements having P dimensions. In this way, example implementations may, for example, facilitate information extraction and reasoning across different data modalities by projecting data into elements in the same embedding space for comparison, combination, or other computations between them.

[0166] For example, elements 8-0,..., 8-9 may specify specific locations within a multidimensional embedding space. Some elements may map to a set of discrete locations in the embedding space. For example, elements corresponding to discrete elements of a predetermined vocabulary of tokens may map to discrete locations in the embedding space associated with those tokens. Other elements may be continuously distributed throughout the embedding space. For example, some data types may be decomposed into continuously defined sections (e.g., image patches) that can be described using continuously distributed locations within the embedding space.

[0167] In some implementations, the expressiveness of the embedding space may not be limited to meanings associated with a concrete set of tokens or other building blocks. For example, a continuous embedding space may encode a spectrum of high-level information. A single piece of information (e.g., a token) may be mapped to a concrete point in this space: For example, a token for the word "dog" may be projected to an embedded value pointing to a concrete location in the embedding space associated with dog-related information. Similarly, an image patch of an image of a dog on grass may also be projected into the embedding space. In some implementations, the projection of the dog's image may be similar to the projection of the word "dog," while also sharing similarity with a projection of the word "grass," while potentially being distinct from both.In some implementations, the projection of the image patch may not be precisely aligned with a single projection of a single word. In some implementations, the projection of the image patch may be aligned with a combination of the projections of the words "dog" and "grass." In this way, for example, a higher-level embedding space can encode information that may be independent of the data modalities in which the information is expressed.

[0168] The task indicator 9 may include a model or model component configured to identify a task currently being performed and to insert an input value, represented by element 8-0, into the input sequence 8, which signals which task is currently being performed. For example, the input value may be provided as a data type associated with an input modality and projected along with that input modality (e.g., the input value may be a textual task label embedded in the input along with other text data; the input value may be a pixel-based representation of a task embedded in the input along with other image data; etc.). The input value may be provided as a data type that is different from, or at least independent of, other input(s).For example, the input value represented by element 8-0 can be learned within a continuous embedding space.

[0169] The input modalities 10-1, 10-2 and 10-3 may be associated with different data types (e.g., as described above with respect to input(s) 2 and output(s) 3).

[0170] The data sequence models 11-1, 11-2, and 11-3 may be the same or different from each other. The data sequence models 11-1, 11-2, and 11-3 may be adapted to each respective input modality 10-1, 10-2, and 10-3. For example, a textual data sequence model may subdivide a section of input text and project the subdivisions into element(s) in the input sequence 8 (e.g., elements 8-1, 8-2, 8-3, etc.). An image data sequence model may subdivide an input image and project the subdivisions into element(s) in the input sequence 8 (e.g., elements 8-4, 8-5, 8-6, etc.). A data sequence model of any data type can partition an input of that arbitrary data type and project the partitions into element(s) in the input sequence 8 (e.g., elements 8-7, 8-8, 8-9, etc.).

[0171] The data sequence models 11-1, 11-2, and 11-3 can be part of the machine-learned sequence processing model(s) 4. The data sequence models 11-1, 11-2, and 11-3 can be trained jointly with the machine-learned sequence processing model(s) 4 or trained independently. The data sequence models 11-1, 11-2, and 11-3 can be trained end-to-end with the machine-learned sequence processing model(s) 4. Exemplary development platform for machine-learned models

[0172] Fig. 10 is a block diagram of an example model development platform 12 that can facilitate the creation, adaptation, and refinement of example machine-learned models (e.g., machine-learned model(s) 1, sequence processing model(s) 4, etc.). The model development platform 12 can provide a number of different toolkits that developer systems can use in developing new or adapted machine-learned models.

[0173] The model development platform 12 can provide one or more model libraries 13 that contain building blocks for new models. The model libraries 13 can include one or more pre-trained foundation models 13-1 that can provide a backbone of computational performance across various tasks. The model libraries 13 can include one or more pre-trained expert models 13-2 that can focus on performance in specific domains. The model libraries 13 can include various model primitives 13-3 that can provide sub-architectures or components (optionally pre-trained) that can be assembled in various arrangements as desired.

[0174] The model development platform 12 can receive selections of various model components 14. The model development platform 12 can forward selected model components 14 to a workbench 15, which combines selected model components 14 into a development model 16.

[0175] The workbench 15 can facilitate further refinement and adaptation of the development model 16 by utilizing a number of different toolkits integrated into the model development platform 12. For example, the workbench 15 can facilitate the alignment of the development model 16 with a desired performance profile for various tasks using a model alignment toolkit 17.

[0176] The model alignment toolkit 17 may provide a number of tools to cause the development model 16 to generate outputs aligned with desired behavioral properties. Alignment may involve increasing the accuracy, precision, recall, etc. of model outputs. Alignment may involve enforcing output styles, an output schema, or other preferred properties of model outputs. Alignment may be general or domain-specific. For example, a pre-trained baseline model 13-1 may begin with an initial level of performance across multiple domains. Alignment of the pre-trained baseline model 13-1 may involve improving performance in a specific domain of information or tasks (e.g., even at the expense of performance in another domain of information or tasks).

[0177] The model alignment toolkit 17 may integrate one or more datasets 17-1 for aligning the development model 16. A curated dataset(s) 17-1 may include labeled or unlabeled training data. The dataset(s) 17-1 may be obtained from public datasets. The dataset(s) 17-1 may be obtained from private datasets associated with one or more developer systems for aligning customized machine-learned models customized for private use cases.

[0178] Pretraining pipelines 17-2 may include a machine-learned model training workflow configured to update the development model 16 over large, potentially noisy datasets. Pretraining may, for example, utilize unsupervised learning techniques (e.g., denoising, etc.) to process a large number of training instances to update model parameters from an initialized state and achieve a desired baseline performance. The pretraining pipelines 17-2 may utilize unlabeled datasets in the dataset(s) 17-1 to perform pretraining. The workbench 15 may implement a pretraining pipeline 17-2 to pretrain the development model 16.

[0179] Fine-tuning pipelines 17-3 may include a machine-learned model training workflow configured to refine the model parameters of the development model 16 using higher-quality data. The fine-tuning pipelines 17-3 may update the development model 16 by performing supervised training with a labeled dataset(s) in the dataset(s) 17-1. The fine-tuning pipelines 17-3 may update the development model 16 by performing reinforcement learning using reward signals from user feedback signals. The workbench 15 may implement a fine-tuning pipeline 17-3 for fine-tuning the development model 16.

[0180] Prompt libraries 17-4 may include sets of inputs configured to induce behavior aligned with the desired performance criteria. Prompt libraries 17-4 may include few prompts (e.g., inputs providing examples of desired model outputs to precede a desired runtime query), mental prompts (e.g., inputs providing step-by-step reasoning within the examples to facilitate thorough reasoning by the model), and the like.

[0181] Example prompts can be retrieved from an available repository of prompt libraries 17-4. Example prompts can be contributed by one or more developer systems using Workbench 15.

[0182] In some implementations, pre-trained or fine-tuned models can achieve satisfactory performance without examples in the inputs. For example, null prompts can include inputs that lack examples. Null prompts can occur within a domain within a training dataset or outside the training domain(s).

[0183] Prompt libraries 17-4 may include one or more prompt engineering tools. Prompt engineering tools may provide workflows for retrieving or learning optimized prompt values. Prompt engineering tools may facilitate the direct learning of prompt values ​​(e.g., input element values) based on one or more training iterations. The workbench 15 may implement prompt engineering tools in the development model 16.

[0184] The prompt libraries 17-4 may include prompt generation pipelines. For example, inputs may be generated using the development model 16 itself or other machine-learned models. In this way, for example, a first model may process information about a task and generate input for a second model to be processed to perform a step of the task. The second model may be the same as or different from the first model. The workbench 15 may implement prompt generation pipelines in the development model 16.

[0185] The prompt libraries 17-4 may include pipelines for context injection. For example, performance of the development model 16 for a specific task may be improved if an additional context is provided for performing the task. The prompt libraries 17-4 may include software components configured to identify the desired context, retrieve the context from an external source (e.g., a database, a sensor, etc.), and add the context to the input prompt. The workbench 15 may implement context injection pipelines in the development model 16.

[0186] Although various training examples described herein refer to "pre-training" and "fine-tuning" with respect to the model development platform 12, it should be understood that the model alignment toolkit 17 may generally support a wide variety of training techniques adapted for training a wide variety of machine-learned models. Example training techniques may correspond to the example training method 800 described above.

[0187] The model development platform 12 may include a model plug-in toolkit 18. The model plug-in toolkit 18 may include a variety of tools configured to augment the functionality of a machine-learned model by integrating the machine-learned model with other systems, devices, and software components. For example, a machine-learned model may utilize tools to increase performance quality where appropriate. For example, deterministic tasks may be outsourced to dedicated tools instead of performing the task probabilistically with an increased risk of error. For example, instead of autoregressively predicting the solution to a system of equations, a machine-learned model may recognize a tool to request the solution and pass the system of equations to the appropriate tool.The tool can be a conventional equation-solving system that can operate deterministically to solve the system of equations. The tool's output can be returned in response to the original query. In this way, using the tool can allow some example models to focus on the strengths of machine-learned models—e.g., understanding an intention in an unstructured request for a task—while augmenting the model's performance by offloading certain tasks to a more focused tool for rote application of deterministic algorithms to a well-defined problem.

[0188] The model plug-in toolkit 18 may include validation tools 18-1. The validation tools 18-1 may include tools that can parse and validate the output(s) of a machine-learned model. The validation tools 18-1 may include developed heuristics that specify certain thresholds to be applied to model outputs. For example, the validation tools 18-1 may anchor the outputs of machine-learned models to structured data sources (e.g., to mitigate "hallucinations"). The model plug-in toolkit 18 may include tooling packages 18-2 for implementing one or more tools, which may include scripts or other executable code that can be executed in parallel with the development model 16. The tooling packages 18-2 may include one or more inputs configured to cause machine-learned model(s) to implement the tools (e.g.,a few prompts that cause a model to issue tool calls in the correct syntax, etc.). For example, the tooling packages 18-2 may contain fine-tuning training data for training a model to use a tool.

[0189] The model plug-in toolkit 18 may include interfaces for calling external application programming interfaces (APIs) 18-3. For example, in addition to or instead of implementing tool calls or tool code directly with the development model 16, the development model 16 may be aligned with output statements that initiate API calls to send or retrieve data via external systems.

[0190] The model plug-in toolkit 18 can be integrated with the prompt libraries 17-4 to create a catalog of available tools for use with the development model 16. For example, a model can receive a catalog of available tools in an input, and the model can generate an output that selects a tool from the available tools and initiates a tool call to use the tool.

[0191] The model development platform 12 may include a computational optimization toolkit 19 for optimizing the computational performance of the development model 16. For example, model compression tools 19-1 may enable the development model 16 to be reduced in size while maintaining a desired level of performance. For example, model compression 19-1 may include quantization workflows, weight pruning, and sparsification techniques, etc. Hardware acceleration tools 19-2 may facilitate the configuration of model storage and execution formats for optimal operation on different hardware resources. For example, hardware acceleration 19-2 may include tools for optimally partitioning models for distributed processing across multiple processing units for increased bandwidth, lower unified memory requirements, etc.Distillation tools 19-3 can provide for the training of lighter-weighted models based on the knowledge encoded in the development model 16. For example, the development model 16 can be a high-performance, large machine-learned model optimized using the model development platform 12. To obtain a lighter-weighted model for operation in resource-constrained environments, a smaller model can be a "student model" that learns to imitate the development model 16 as a "teacher model."

[0192] In this way, for example, the investment in learning the parameters and configurations of the development model 16 can be efficiently transferred to a smaller model for more efficient inference.

[0193] The workbench 15 may implement one, more, or none of the toolkits implemented in the model development platform 12. The workbench 15 may output an output model 20 based on the development model 16. The output model 20 may be a deployment version of the development model 16. The output model 20 may be a development or training checkpoint of the development model 16. The output model 20 may be a distilled, compressed, or otherwise optimized version of the development model 16.

[0194] The workbench 15 may implement training techniques according to the present disclosure to distill (e.g., distillation 19-3) and simultaneously perform reinforcement learning (e.g., via fine-tuning paths 17-3) using a multi-scale refinement objective according to example implementations of the present disclosure.

[0195] Fig. 11 is a block diagram of an example training flow for training a machine-learned development model 16. One or more portions of the example training flow may be implemented by a computing system including one or more computing devices, such as the computing systems described with respect to the other figures. Each respective portion of the example training flow may be performed by any (or any combination of) one or more computing devices. Furthermore, one or more portions of the example training flow may be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. Fig. 11 shows elements performed in a specific order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any methods discussed herein may be variously adapted, rearranged, expanded, omitted, combined, or modified without departing from the scope of the present disclosure. Fig. 11 is described with reference to elements / terminology described with respect to other systems and figures for exemplary illustration purposes and are not to be understood as limiting. One or more portions of the exemplary training sequence may be additionally or alternatively performed by other systems.

[0196] Initially, the development model 16 may remain as an initialized model 21 in an initial state. The development model 16 may be initialized with weight values. Initial weight values ​​may be random or based on an initialization scheme. The initial weight values ​​may be based on previous pretraining for the same or a different model.

[0197] The initialized model 21 may be pre-trained in a pre-training stage 22. The pre-training stage 22 may be implemented using one or more pre-training pipelines 17-2 on data from the dataset(s) 17-1. Pre-training may be omitted, for example, if the initialized model 21 is already pre-trained (e.g., the development model 16 contains, is based on, or is based on a pre-trained baseline model or an expert model).

[0198] The pre-trained model 23 can then be a new version of the development model 16, which can remain as the development model 16 or as a new development model. The pre-trained model 23 can be the initial state if the development model 16 was already pre-trained. The pre-trained model 23 can be fine-tuned in a fine-tuning stage 24. The fine-tuning stage 24 can be implemented using one or more fine-tuning pipelines 17-3 on data from the dataset(s) 17-1. Fine-tuning can be omitted, for example, if a pre-trained model shows satisfactory performance, if the model has already been fine-tuned, or if other tuning approaches are preferred.

[0199] The fine-tuned model 29 can then be a new version of the development model 16, which can remain as the development model 16 or as a new development model. The fine-tuned model 29 can be the initial state if the development model 16 has already been fine-tuned. The fine-tuned model 29 can be subjected to refinement with user feedback 26. For example, the refinement with user feedback 26 can involve reinforcement learning, optionally based on human feedback from human users of the fine-tuned model 25. Since reinforcement learning can be a form of fine-tuning, it is understood that the fine-tuning stage 24 can subsume the refinement with user feedback 26 stage. The refinement with user feedback 26 can produce a refined model 27.The refined model 27 can be issued to a downstream system(s) 28 for deployment or further development.

[0200] In some implementations, computational optimization operations may be applied before, during, or after each stage. For example, the initialized model 21 may undergo computational optimization 29-1 (e.g., using the computational optimization toolkit 19) before the pre-training stage 22. The pre-trained model 23 may undergo computational optimization 29-2 (e.g., using the computational optimization toolkit 19) before the fine-tuning stage 24. The fine-tuned model 25 may undergo computational optimization 29-3 (e.g., using the computational optimization toolkit 19) before refinement with user feedback 26. The refined model 27 may undergo computational optimization 29-4 (e.g., using the computational optimization toolkit 19) before output to the downstream system(s) 28. The computational optimization(s) 29-1,..., 29-4 can all be the same, all be different, or at least involve some different optimization techniques. Example of an inference system of a machine-learned model

[0201] Fig. 12 is a block diagram of an inference system for operating one or more machine-learned models 1 to perform inference (e.g., for training, for deployment, etc.). A model host 31 may receive machine-learned model(s) 1. The model host 31 may host one or more model instances 31-1, which may be one or more instances of one or more models. The model host 31 may host model instance(s) 31-1 using the available computing resources 31-2 associated with the model host 31.

[0202] The model host 31 may receive a distilled machine-learned model 1. For example, the model host 31 may implement a smaller version of a larger model. For example, the model host 31 may implement (e.g., store, execute, etc.) a machine-learned student model distilled from a larger machine-learned teacher model according to example aspects of the present disclosure.

[0203] The model host 31 can perform inference on behalf of one or more clients 32. The client(s) 32 can transmit an input request 33 to the model host 31. Using the input request 33, the model host 31 can obtain the input(s) 2 for input to the machine-learned model(s) 1. The machine-learned model(s) 1 can process the input(s) 2 to generate the output(s) 3. Using the output(s) 3, the model host 31 can return an output payload 34 in response to the input request 33 from the client(s) 32. The output payload 34 can include or be based on the output(s) 3.

[0204] The model host 31 may utilize various other resources and tools to augment the inference task. For example, the model host 31 may communicate with tool interfaces 35 to facilitate the use of tools by the model instance(s) 31-1. The tool interfaces 35 may include local or remote APIs. The tool interfaces 35 may include built-in scripts or other software functionality. The model host 31 may enable online learning interface(s) 36 to facilitate ongoing improvements to the machine-learned model(s) 1. For example, the online learning interface(s) 36 may be used within reinforcement learning loops to retrieve user feedback on inferences fulfilled by the model host 31. The model host 31 can access runtime data source(s) 37 to augment the input(s) 2 with further context information.For example, the runtime data source(s) 37 may include a knowledge graph 37-1 that facilitates the structured retrieval of information associated with the input request(s) 33 (e.g., a search engine service). The runtime data source(s) 37 may include public or private, external, or local database(s) 37-2 that may store information associated with the input request(s) 33 to augment the input(s) 2. The runtime data source(s) 37 may include account data 37-3 that may be retrieved in association with a user account corresponding to a client 32 to customize the behavior of the model host 31 accordingly.

[0205] The model host 31 may be implemented by one or more computing devices or one or more computing systems. The client(s) 2 may be implemented by one or more computing devices or one or more computing systems, which may include computing devices or systems shared with the model host 31.

[0206] For example, the model host 31 may operate on a server system that provides a machine learning service to client device(s) operating client(s) 32 (e.g., over a local or wide area network). The client device(s) may be end-user devices used by individuals. The client device(s) may be server systems operating client(s) 32 to provide various functions as a service to downstream end-user devices.

[0207] In some implementations, the model host 31 may operate on the same device or system as the client(s) 32. The model host 31 may be a machine learning service executing on the device to provide machine learning functionality to one or more applications operating on a client device, which may include an application implementing client(s) 32. The model host 31 may be part of a same application as the client(s) 32. For example, the model host 31 may be a subroutine or submethod implemented by part of an application, and the client(s) 32 may be another subroutine or submethod that enables the model host 31 to perform inference functions within the application. It is understood that the model host 31 and the client(s) 32 may have different configurations.

[0208] The model instance(s) 31-1 may include one or more machine-learned models available for performing inference. The model instance(s) 31-1 may include weights or other model components stored in persistent storage, temporarily cached, or loaded into high-speed memory. The model instance(s) 31-1 may include multiple instances of the same model (e.g., for parallel execution of multiple requests on the same model). The model instance(s) 31-1 may include instance(s) of different models. The model instance(s) 31-1 may include cached intermediate states of an active or inactive model or active or inactive models used to accelerate the inference of these models.For example, an inference session with a specific model can generate a significant amount of computational results that can be reused for future inference runs (e.g., using a KV cache for transformer-based models). These computational results can be stored in association with that inference session, allowing the session to run more efficiently upon resumption.

[0209] The computational resource(s) 31-2 may include one or more processors (central processing units, graphics processing units, tensor processing units, machine learning accelerators, etc.) connected to one or more storage devices. The computational resource(s) 31-2 may include a dynamic pool of available resources shared with other processes. The computational resource(s) 31-2 may include storage devices large enough for an entire model instance to fit into a single storage instance. The computational resource(s) 31-2 may also partition a model instance(s) across multiple storage devices (e.g., through data parallelization or tensor parallelization, etc.).This can be done to increase parallelism or to run a large model using multiple memory devices that individually may not be able to fit the entire model into memory.

[0210] Input request 33 may include data for input(s) 2. Model host 31 may process input request 33 to obtain input(s) 2. Input(s) 2 may be obtained directly from input request 33 or retrieved using input request 33. Input request 33 may be communicated to model host 31 via an API.

[0211] The model host 31 can perform inference on batches of input requests 33 in parallel. For example, a model instance 31-1 can be configured with an input structure that has a batch dimension. Separate input(s) 2 can be distributed across the batch dimension (e.g., rows of an array). The separate input(s) 2 can contain completely different contexts. The separate input(s) 2 can be multiple inference steps of the same task. The separate input(s) 2 can be staggered within an input structure so that any given inference cycle can operate on different sections of the respective input(s) 2. In this way, for example, the model host 31 can perform inference on the batch in parallel, so that the output(s) 3 can also contain the batch dimension and return the inference results for the batch input(s) 2 in parallel.In this way, for example, batches of an input request(s) 33 can be processed in parallel for a higher throughput of the output payload 34.

[0212] The output payload 34 may include or be based on the output(s) 3 of the machine-learned model(s) 1. The model host 31 may process the output(s) 3 to obtain the output payload 34. This may involve chaining multiple rounds of inference (e.g., iterative, recursive, over the same model(s), or different models) to arrive at a final output for a task to be returned in the output payload 34. The output payload 34 may be transmitted to the client(s) 32 via an API.

[0213] (An) online learning interface(s) 36 can facilitate reinforcement learning of a machine-learned model(s) 1. The online learning interface(s) 36 can facilitate reinforcement learning with human feedback (RLHF). The online learning interface(s) 36 can facilitate federated learning of a machine-learned model(s) 1.

[0214] The model host 31 may execute the machine-learned model(s) 1 to perform inference for different tasks using different data types. For example, various different inputs 2 and outputs 3 may be used for various different tasks. In some implementations, the input(s) 2 may be or otherwise represent image data. The machine-learned model(s) 1 may process the image data to generate an output. As one example, the machine-learned model(s) 1 may process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) 1 may process the image data to generate an image segmentation output.As another example, the machine-learned model(s) 1 may process the image data to generate an image classification output. As another example, the machine-learned model(s) 1 may process the image data to generate an image data modification output (e.g., a change in the image data, etc.). As another example, the machine-learned model(s) 1 may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine-learned model(s) 1 may process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) 1 may process the image data to generate a prediction output. In some implementations, the task is a computer vision task.In some cases, the input(s) include two pixel data for one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, where each score corresponds to a different object class and represents the probability that the one or more images show an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images and, for each region, a probability that the region shows an object of interest.As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in the one or more images, a respective probability for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines, for each pixel in the one or more images, a respective depth value. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, motion of the scene shown at the pixel between the images in the network input.

[0215] In some implementations, the input(s) 2 may be or otherwise represent natural language data. The machine-learned model(s) 1 may process the natural language data to generate an output. As one example, the machine-learned model(s) 1 may process the natural language data to generate a speech coding output. As another example, the machine-learned model(s) 1 may process the natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) 1 may process the natural language data to generate a translation output. As another example, the machine-learned model(s) 1 may process the natural language data to generate a classification output.As another example, the machine-learned model(s) 1 may process the natural language data to generate a text segmentation output. As another example, the machine-learned model(s) 1 may process the natural language data to generate a semantic intent output. As another example, the machine-learned model(s) 1 may process the natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is of higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) 1 may process the natural language data to generate a prediction output (e.g., one or more predicted next chunks of natural language content).

[0216] In some implementations, the input(s) 2 may be or otherwise represent speech data (e.g., data describing spoken natural language, such as audio data, text data, etc.). The machine-learned model(s) 1 may process the speech data to generate an output. As one example, the machine-learned model(s) 1 may process the speech data to generate a speech recognition output. As another example, the machine-learned model(s) 1 may process the speech data to generate a speech translation output. As another example, the machine-learned model(s) 1 may process the speech data to generate a latent embedding output. As another example, the machine-learned model(s) 1 may process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.).As another example, the machine-learned model(s) 1 may process the speech data to generate an upscaled speech output (e.g., speech data that is of higher quality than the input speech data, etc.). As another example, the machine-learned model(s) 1 may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine-learned model(s) 1 may process the speech data to generate a prediction output.

[0217] In some implementations, the input(s) 2 may be or otherwise represent latent coding data (e.g., a latent spatial representation of an input, etc.). The machine-learned model(s) 1 may process the latent coding data to generate an output. As one example, the machine-learned model(s) 1 may process the latent coding data to generate a recognition output. As another example, the machine-learned model(s) 1 may process the latent coding data to generate a reconstruction output. As another example, the machine-learned model(s) 1 may process the latent coding data to generate a search output. As another example, the machine-learned model(s) 1 may process the latent coding data to generate a post-clustering output.As another example, the machine-learned model(s) 1 may process the latent coding data to generate a prediction output.

[0218] In some implementations, input(s) 2 may be or otherwise represent statistical data. Statistical data may be, represent, or otherwise include computed and / or calculated data from another data source. Machine-learned model(s) 1 may process the statistical data to generate an output. As one example, machine-learned model(s) 1 may process the statistical data to generate a detection output. As another example, machine-learned model(s) 1 may process the statistical data to generate a prediction output. As one example, machine-learned model(s) 1 may process the statistical data to generate a classification output. As one example, machine-learned model(s) 1 may process the statistical data to generate a segmentation output.As one example, the machine-learned model(s) 1 may process the statistical data to generate a visualization output. As another example, the machine-learned model(s) 1 may process the statistical data to generate a diagnostic output.

[0219] In some implementations, the input(s) 2 may be or otherwise represent sensor data. The machine-learned model(s) 1 may process the sensor data to generate an output. As one example, the machine-learned model(s) 1 may process the sensor data to generate a detection output. As another example, the machine-learned model(s) 1 may process the sensor data to generate a prediction output. As another example, the machine-learned model(s) 1 may process the sensor data to generate a classification output. As another example, the machine-learned model(s) 1 may process the sensor data to generate a segmentation output. As another example, the machine-learned model(s) 1 may process the sensor data to generate a visualization output.As another example, the machine-learned model(s) 1 may process the sensor data to generate a diagnostic output. As another example, the machine-learned model(s) 1 may process the sensor data to generate a detection output. In some implementations, the machine-learned model(s) 1 may be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data, and the output may include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task.In another example, the task may involve generating an embedding for input data (e.g., input audio or visual data). In some cases, the input includes audio data representing a spoken utterance, and the task is a speech recognition task. The output may include text output mapped to the spoken utterance. In some cases, the task involves encrypting or decrypting input data. In some cases, the task involves a microprocessor performance task, such as branch prediction or memory address translation.

[0220] In some implementations, the task is a generative task, and the machine-learned model(s) 1 may be configured to output content generated with respect to the input(s) 2. For example, the input(s) 2 may be or otherwise represent data from one or more modalities that encodes context for generating additional content.

[0221] In some implementations, the task may be a text completion task. The machine-learned model(s) 1 may be configured to process input(s) 2 representing text data and generate output(s) 3 representing additional text data that completes a text sequence that includes the input(s) 2. For example, the machine-learned model(s) 1 may be configured to generate output(s) 3 to complete a sentence, paragraph, or section of text that follows from a section of text represented by the input(s) 2.

[0222] In some implementations, the task may be an instruction-following task. The machine-learned model(s) 1 may be configured to process input(s) 2 representing instructions for performing a function and generate output(s) 3 that promotes a goal of fulfilling the instruction function (e.g., at least one step of a multi-scale procedure for performing the function). The output(s) 3 may represent data of the same modality or a different modality than the input(s) 2. For example, the input(s) 2 may represent text data (e.g., natural language instructions for a task to be performed), and the machine-learned model(s) 1 may process the input(s) 2 to generate the output(s) 3 representing text data in response to the instructions (e.g.,Natural language answers, programming language answers, machine language answers, etc.). The input(s) 2 may represent image data (e.g., image-based instructions for a task to be performed, possibly accompanied by text instructions), and the machine-learned model(s) 1 may process the input(s) 2 to generate output(s) 3 that represent text data in response to the instructions (e.g., natural language answers, programming language answers, machine language answers, etc.). One or more output(s) 3 may be generated iteratively or recursively to sequentially process and fulfill steps toward fulfilling the requested functionality. For example, an initial output may be executed by an external system or processed by the machine-learned model(s) 1 to complete an initial step of performing a function.Multiple steps can be performed, producing a final output that responds to the initial instructions.

[0223] In some implementations, the task may be a question-answering task. The machine-learned model(s) 1 may be configured to process input(s) 2 representing a question to be answered and generate output(s) 3 that furthers a goal of returning an answer to the question (e.g., at least one step of a multi-scale procedure for performing the function). The output(s) 3 may represent data of the same modality or a different modality than the input(s) 2. For example, the input(s) 2 may represent text data (e.g., natural language instructions for a task to be performed), and the machine-learned model(s) 1 may process the input(s) 2 to generate the output(s) 3 representing text data in response to the question (e.g.,Natural language answers, programming language answers, machine language answers, etc.). The input(s) 2 may represent image data (e.g., image-based instructions for a task to be performed, possibly accompanied by text instructions), and the machine-learned model(s) 1 may process the input(s) 2 to generate output(s) 3 that represent text data in response to the question (e.g., natural language answers, programming language answers, machine language answers, etc.). One or more output(s) 3 may be generated iteratively or recursively to sequentially process and fulfill steps toward answering the question. For example, an initial output may be executed by an external system or processed by the machine-learned model(s) 1 to perform an initial step of obtaining an answer to the question (e.g.,Querying a database, performing a calculation, running a script, etc.). Multiple steps can be performed, producing a final output that answers the question.

[0224] In some implementations, the task may be an image generation task. The machine-learned model(s) 1 may be configured to process input(s) 2 representing a context regarding a desired portion of the image content. The context may include text data, image data, audio data, etc. The machine-learned model(s) 1 may be configured to generate output(s) 3 representing image data showing imagery relevant to the context. For example, the machine-learned model(s) 1 may be configured to generate pixel data of an image. Values ​​for a channel(s) associated with the pixels in the pixel data may be selected based on the context (e.g., based on a probability determined based on the context).

[0225] In some implementations, the task may be an audio generation task. The machine-learned model(s) 1 may be configured to process input(s) 2 representing a context related to a desired portion of the audio content. The context may include text data, image data, audio data, etc. The machine-learned model(s) 1 may be configured to generate output(s) 3 representing audio data related to the context. For example, the machine-learned model(s) 1 may be configured to generate waveform data in the form of an image (e.g., a spectrogram). Values ​​for the channel(s) associated with pixels of the image may be selected based on the context.The machine-learned model(s) 1 may be configured to generate waveform data in the form of a sequence of discrete samples of a continuous waveform. Values ​​of the sequence may be selected based on the context (e.g., based on a probability determined based on the context).

[0226] In some implementations, the task may be a data generation task. The machine-learned model(s) 1 may be configured to process input(s) 2 representing a context regarding a desired portion of data (e.g., data from different data domains, such as sensor data, image data, multimodal data, statistical data, etc.). The desired data may, for example, be synthetic data for training other machine-learned models. The context may include any data type(s). The machine-learned model(s) 1 may be configured to generate output(s) 3 representing data aligned with the desired data. For example, the machine-learned model(s) 1 may be configured to generate data values ​​for populating a dataset.Values ​​for the data object(s) can be selected based on the context (e.g., based on a probability determined based on the context). Examples of computing systems and devices

[0227] Fig. 13 is a block diagram of an example networked computing system that can perform aspects of the example implementations of the present disclosure. The system can include a number of computing devices and systems communicatively coupled via a network 49. An example computing device 50 is described to provide an example of a computing device that can perform any aspect of the present disclosure (e.g., implementing the model host 31, the client(s) 32, or both). An example server computing system 60 is described as an example of a server computing system that can perform any aspect of the present disclosure (e.g., implementing the model host 31, the client(s) 32, or both). The computing device 50 and the server computing system(s) 60 can cooperate (e.g.,via the network 49) to perform any aspect of the present disclosure (e.g., implementing the model host 31, the client(s) 32, or both). A model development platform system 70 is an example system that may host or fulfill the model development platform(s) 12 for developing machine-learned models. A third-party system(s) 80 is / are an example system(s) with which any of the computing device 50, server computing system(s) 60, or model development platform system(s) 70 may cooperate in performing various aspects of the present disclosure (e.g., incorporating third-party tools, accessing third-party databases or other resources, etc.).

[0228] The network 49 may be any type of communications network, such as a local area network (e.g., intranet), a wide area network (e.g., Internet), or a combination thereof, and may include any number of wired or wireless connections. In general, communication over the network 49 may occur over any type of wired or wireless connection, using a variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), or protection schemes (e.g., VPN, Secure HTTP, SSL). The network 49 may also be implemented over a system bus. For example, one or more devices or one or more systems of the Fig. 13 may be co-located with, contained by, or otherwise integrated with one or more other devices or systems. Computing device 50 may be any type of computing device, such as a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a portable computing device, an embedded computing device, a server computing device, a virtual machine running on a host device, or any other type of computing device. Computing device 50 may be a client computing device. Computing device 50 may be an end-user computing device.Computing device 50 may be a provisioned service computing device that provides a service to an end user (who may use another computing device to interact with computing device 50).

[0229] Computing device 50 may include one or more processors 51 and a memory 52. ​​The processor(s) 51 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. Memory 52 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 52 may store data 53 and instructions 54 that may be executed by the processor(s) 51 to cause the computing device 50 to perform operations. The operations may implement one or more of the features described herein. The operations may implement example methods and techniques described herein.

[0230] Computing device 50 may also include one or more input components that receive user input. For example, a user input component may be a touch-sensitive component (e.g., a touch-sensitive screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other examples of user input components include a microphone, a camera, a lidar, a physical keyboard, or other buttons or other means by which a user can provide user input.

[0231] Computing device 50 may store or include one or more machine-learned models 55. Machine-learned models 55 may include one or more machine-learned models 1, such as a sequence processing model 4. Machine-learned models 55 may include one or more model instances 31-1. The machine-learned model(s) 55 may be received from server computing system(s) 60, model development platform system 70, third-party system(s) 80 (e.g., an application distribution platform), or developed locally on computing device 50. The machine-learned model(s) 55 may be loaded into memory 52 and used or otherwise implemented by processor(s) 51. The computing device 50 may implement multiple parallel instances of the machine-learned model(s) 55.

[0232] The server computing system(s) 60 may include one or more processors 61 and memory 62. The processor(s) 61 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 62 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 62 may store data 63 and instructions 64 executable by the processor(s) 61 to cause the server computing system(s) 60 to perform operations. The operations may implement one or more of the features described herein.The operations may implement example methods and techniques described herein.

[0233] In some implementations, server computing system 60 includes or is otherwise implemented by one or more server computing devices. In cases where server computing system 60 includes multiple server computing devices, such server computing devices may operate according to sequential computing architectures, parallel computing architectures, or a combination thereof.

[0234] The server computing system 60 may store or otherwise include one or more machine-learned models 65. The machine-learned model(s) 65 may be the same as or different from the machine-learned model(s) 55. The machine-learned models 65 may include one or more machine-learned models 1, such as a sequence processing model 4. The machine-learned models 65 may include one or more model instances 31-1. The machine-learned model(s) 65 may be received from the computing device 50, the model development platform system 70, third-party system(s) 80, or developed locally on the server computing system(s) 60. The machine-learned model(s) 65 may be loaded into memory 62 and used or otherwise implemented by the processor(s) 61.The server computing system(s) 60 may implement multiple parallel instances of the machine-learned model(s) 65.

[0235] In an example configuration, machine-learned models 65 may be included in or otherwise stored and implemented by the server computing system 60 to establish a client-server relationship with the computing device 50 for performing model inferences. For example, the server computing system(s) 60 may implement the model host 31 on behalf of the client(s) 32 on the computing device 50. For example, the machine-learned models 65 may be implemented by the server computing system 60 as part of a web service (e.g., a remote machine-learned model hosting service, such as an online interface for performing machine-learned model operations over a network on the server computing system(s) 60). For example, the server computing system(s) 60 may communicate with the computing device 50 over a local intranet or internet connection.For example, computing device 50 may be a workstation or endpoint in communication with server computing system(s) 60, wherein the implementation of machine-learned models 65 is managed by server computing system(s) 60 to remotely perform inference (e.g., for runtime or training operations), with the output(s) returned to computing device 50 (e.g., distributed, streamed, etc.). Machine-learned models 65 may work cooperatively or interoperably with machine-learned models 55 on computing device 50 to perform various tasks.

[0236] The model development platform system(s) 70 may include one or more processors 71 and a memory 72. The processor(s) 71 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 72 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 72 may store data 73 and instructions 74 executable by the processor(s) 71 to cause the model development platform system(s) 70 to perform operations. The operations may implement one or more of the features described herein.The operations may implement exemplary methods and techniques described herein. Example operations include the functionality described herein with respect to the model development platform 12. This and other functionality may be implemented by developer tool(s) 75.

[0237] Third-party system(s) 80 may include one or more processors 81 and memory 82. The processor(s) 81 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 82 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 82 may store data 83 and instructions 84 that may be executed by the processor(s) 81 to cause the third-party system(s) 80 to perform operations. The operations may implement one or more of the features described herein.The operations may implement example methods and techniques described herein. Example operations include the functionality described herein with respect to tools and other external resources invoked when training or performing inference with the machine-learned model(s) 1, 4, 16, 20, 55, 65, etc. (e.g., third-party resource(s) 85).

[0238] Fig. 13 illustrates an example arrangement of computing systems that may be used to implement the present disclosure. Other computing system configurations may also be used. For example, in some implementations, one or both of computing system 50 or server computing system(s) 60 may implement all or a portion of the operations of model development platform system 70. For example, computing system 50 or server computing system(s) 60 may implement developer tool(s) 75 (or extensions thereof) to develop, update / train, or refine machine-learned models 1, 4, 16, 20, 55, 65, etc. using one or more techniques described herein with respect to model alignment toolkit 17. In this way, for example, computing system 50 or server computing system(s) 60 mayfor model personalization / adaptation as permitted by user data preference selection) develop, update / train, or refine machine-learned models.

[0239] Fig. 14 is a block diagram of an example computing device 98 operating in accordance with example embodiments of the present disclosure. Computing device 98 may be a user computing device or a server computing device (e.g., computing device 50, server computing device(s) 60, etc.).

[0240] Computing device 98 may implement model host 31. For example, computing device 98 may include a number of applications (e.g., applications 1 through N). Each application may include its own machine learning library and machine-learned model(s). For example, each application may include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. As shown in Fig. As illustrated in Figure 14, each application may communicate with a number of other components of the computing device, such as one or more sensors, a context manager, a device state component, or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0241] Fig. 15 is a block diagram of an example computing device 99 that operates according to example embodiments of the present disclosure. Computing device 99 may be the same as or different from computing device 98. Computing device 99 may be a user computing device or a server computing device (e.g., computing device 50, server computing device(s) 60, etc.). Computing device 98 may implement model host 31. For example, computing device 99 may include a number of applications (e.g., applications 1 through N). Each application may communicate with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.In some implementations, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a common API for all applications).

[0242] The central intelligence layer can include a number of machine-learned models. For example, as in Fig. 15 illustrates, a respective machine-learned model may be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine-learned model. For example, in some implementations, the central intelligence layer may provide a single model for all applications. In some implementations, the central intelligence layer is included in or otherwise implemented by an operating system of the computing device 99.

[0243] The central intelligence layer may communicate with a central device data layer. The central device data layer may be a central repository of data for the computing device 99. As in Fig. As illustrated in Figure 15, the central device data layer may communicate with a number of other components of the computing device, such as one or more sensors, a context manager, a device state component, or additional components. In some implementations, the central device data layer may communicate with each device component using an API (e.g., a private API).

[0244] According to one embodiment, an exemplary computing system is provided for training a machine-learned student sequence processing model. The computing system may perform operations including: obtaining a respective input; obtaining a respective output corresponding to the respective input from a machine-learned student sequence processing model;Generating a multi-scale refinement objective configured to distill knowledge from a machine-learned teacher sequence processing model while reinforcing preferred behavior of the machine-learned student sequence processing model, the multi-scale refinement objective comprising: a first component based on a divergence metric indicating, for the respective input, a comparison of a plurality of predictions of the machine-learned student sequence processing model with a plurality of predictions of the machine-learned teacher sequence processing model; and a second component based on a reinforcement learning signal associated with the respective output; and updating the machine-learned student sequence processing model based on the multi-scale refinement objective. Additional Revelation

[0245] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as to actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions between and among components. For example, the processes discussed herein may be implemented using a single device or component or multiple cooperating devices or components. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.While the present subject matter has been described in detail with reference to various specific exemplary embodiments, each example is provided for the purpose of illustration, not limitation, of the disclosure. Those skilled in the art, after receiving the foregoing, may readily make modifications, variations, and equivalents to such embodiments. Accordingly, the present disclosure does not preclude the incorporation into the present subject matter of such modifications, variations, or additions as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may be used in another embodiment to yield yet another embodiment. Therefore, the present disclosure is intended to cover such modifications, variations, and equivalents.

[0246] Aspects of the disclosure have been described with respect to illustrated embodiments thereof. Any and all features of the following claims may be combined or rearranged in any manner, including combinations of claims not explicitly enumerated in combination, since the exemplary claim dependencies recited herein are not to be construed as limiting the scope of possible feature combinations as disclosed herein. The scope of the present disclosure is accordingly exemplary rather than restrictive, and the disclosure does not preclude the incorporation into the present subject matter of such modifications, variations, or additions as would be readily apparent to one of ordinary skill in the art. Furthermore, terms are described herein using lists of example elements connected by joins such as "and," "or," "but," etc.It is understood that such links are provided for illustrative purposes only. Clauses and other sequences of elements connected by a specific link such as "or" may refer to "and / or," "at least one of," "any combination of," and so on, exemplary elements listed therein. Terms such as "based on" should be understood as "at least partially based on."

[0247] The term "may" should be understood to refer to a possibility of a feature in various implementations, not to prescribe a capability that is necessarily present in every implementation. For example, the phrase "X can perform Y" should be understood to mean that X has the potential to be configured to perform Y in various implementations, not to mean that in every case X must always be able to perform Y. It should be understood that in various implementations, X may not be able to perform Y and remain within the scope of the present disclosure.

[0248] The term "could" should be understood to refer to a possibility of a feature in various implementations, not to prescribe a capability that is necessarily present in every implementation. For example, the phrase "X could perform Y" should be understood to imply that X has the potential to be configured to perform Y in various implementations, not to imply that X must always be able to perform Y in every case. It should be understood that in various implementations, X may not be able to perform Y and remain within the scope of the present disclosure. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Roit et al. (2023

[0127] Agarwal et al., Generalized Knowledge Distillation for Auto-regressive Language Models, arXiv:2306.13649v2 (October 3, 2023

[0139] Zhou et al., Mixture-of-Experts with Expert Choice Routing, ARXIV:2202.09368v2 (October 14, 2022

[0142] Dosovitskiy et al., An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale, ARXIV:2010.11929v2 (June 3, 2021

[0149] Agostinelli et al., MusicLM: Generating Music From Text, ARXIV:2301.11325v1 (January 26, 2023

[0149] Jumper et al., Highly Accurate Protein Structure Prediction with AlphaFold, 596 Nature 583 (26. August 2021

[0149] Kudo et al., SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, PROCEEDINGS OF THE 2018 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING (System Demonstrations), Seiten 66-71 (31. Oktober-4. November 2018), https: / / aclanthology.org / D18-2012.pdf

[0153] Vaswani et al., Attention Is All You Need, ARXIV:1706.03762v7 (2. August 2023

[0157] Saharia et al., Non-Autoregressive Machine Translation with Latent Alignments, arXiv:2004.07437v3 (16. November 2020

[0162]

Claims

[1] Computing system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions executable by the one or more processors to cause the computing system to perform one or more operations, the operations comprising: Obtaining a respective input; Obtaining a respective output corresponding to the respective input from a machine-learned student sequence processing model; Generating a multi-scale refinement objective configured to distill knowledge from a machine-learned teacher sequence processing model while reinforcing preferred behavior of the machine-learned student sequence processing model, where the multi-scale refinement objective includes: a first component based on a divergence metric that characterizes, for the respective input, a comparison of a plurality of predictions of the machine-learned student sequence processing model with a plurality of predictions of the machine-learned teacher sequence processing model; and a second component based on a reinforcement learning signal associated with the respective output; and Updating the machine-learned student sequence processing model based on the multi-scale refinement objective. [2] The computing system of claim 1, wherein the divergence metric is evaluated using: a Student value generated by the machine-learned Student sequence processing model for one or more sections of the respective output based on the respective input, the value corresponding to a Student probability of the one or more sections of the respective output that is dependent on the respective input; and a teacher value generated by the machine-learned teacher sequence processing model for the one or more sections of the respective output based on the respective input, wherein the teacher value corresponds to a teacher probability of the one or more sections of the respective output that is dependent on the respective input. [3] The computing system of claim 1, wherein the operations comprise: for each section of a multitude of sections of the respective edition: Determining a bin-specific divergence metric that characterizes a similarity between a student probability distribution over a set of candidate output bins and a teacher probability distribution over the set of candidate output bins, wherein each of the student probability distribution and the teacher probability distribution depends on the respective input and one or more bins of the respective output preceding the bin; and Aggregate the plurality of section-specific divergence metrics for the respective output to obtain the first component. [4] The computing system of claim 3, wherein the teacher probability distributions for each of the section-specific divergence metrics are generated at least partially in parallel by the machine-learned teacher sequence processing model. [5] The computing system of claim 1, wherein the multi-scale refinement objective comprises one or more weighting parameters that weight the respective contributions of the first component and the second component. [6] The computing system of claim 1, wherein the reinforcement learning signal comprises data indicative of human feedback on an overall quality of the respective output. [7] The computing system of claim 1, wherein the reinforcement learning signal comprises data indicative of a score generated by a machine-learned reward model, the score indicative of an overall quality of the respective output. [8] The computing system of claim 1, wherein evaluating the divergence metric comprises: Determining a value of a mixture distribution corresponding to a mixture of a student probability distribution of the machine-learned student sequence processing model and a teacher probability distribution of the machine-learned teacher sequence processing model; Calculating a first divergence component that characterizes a divergence of the student probability distribution with respect to the mixture distribution; Calculating a second divergence component that characterizes a divergence of the teacher probability distribution with respect to the mixture distribution; and Evaluate the divergence metric based on a combination of the first divergence component and the second divergence component. [9] The computing system of claim 8, wherein evaluating the divergence metric based on the first divergence component and the second divergence component comprises: Calculating a weighted combination of the first divergence component and the second divergence component using a weighting parameter. [10] The computing system of claim 9, wherein adjusting the weighting parameter causes the divergence metric to interpolate between a mode search behavior and a mean search behavior. [11] The computing system of claim 10, wherein the operations comprise: Adjusting the weighting parameter based on a desired output diversity for a task type. [12] The computing system of claim 11, wherein the weight is a hyperparameter learned during training. [13] The computing system of claim 1, wherein the machine-learned teacher sequence processing model was not trained using reinforcement learning. [14] The computing system of claim 13, wherein the machine-learned student sequence processing model has been fine-tuned to achieve a baseline performance threshold prior to training with the multi-scale refinement objective. [15] A computing system according to claim 1, wherein: the machine-learned student sequence processing model is characterized by a first number of parameters; the machine-learned teacher sequence processing model is characterized by a second number of parameters; and the second number of parameters is greater than the first number of parameters. [16] The computing system of claim 15, wherein the second number of parameters is at least 30 times the first number of parameters. [17] The computing system of claim 1, wherein the operations comprise: Receiving a request to perform an inference task based on input data from a client computing system; Obtaining the respective input from the input data; Generating the respective output using the machine-learned student sequence processing model; Returning output data to the client computing system based on the respective output and in response to the request; Receiving feedback data from the client computing system; and Determine the reinforcement learning signal based on the feedback data. [18] The computing system of claim 17, wherein the operations comprise: Receiving the request and returning the output data in an online process; and Obtaining the plurality of predictions of the machine-learned teacher sequence processing model and updating the machine-learned student sequence processing model based on the multi-scale refinement objective in an offline process. [19] One or more non-transitory computer-readable media storing instructions executable by one or more processors for causing a computing system to perform one or more operations, the operations comprising: Obtaining a respective input; Obtaining a respective output corresponding to the respective input from a machine-learned student sequence processing model; Generating a multi-scale refinement objective configured to distill knowledge from a machine-learned teacher sequence processing model while reinforcing preferred behavior of the machine-learned student sequence processing model, where the multi-scale refinement objective includes: a first component based on a divergence metric that characterizes, for the respective input, a comparison of a plurality of predictions of the machine-learned student sequence processing model with a plurality of predictions of the machine-learned teacher sequence processing model; and a second component based on a reinforcement learning signal associated with the respective output; and Updating the machine-learned student sequence processing model based on the multi-scale refinement objective. [20] One or more non-transitory computer-readable media according to claim 19, wherein the divergence metric is evaluated using: a Student value generated by the machine-learned Student sequence processing model for one or more sections of the respective output based on the respective input, the value corresponding to a Student probability of the one or more sections of the respective output that is dependent on the respective input; and a teacher value generated by the machine-learned teacher sequence processing model for the one or more sections of the respective output based on the respective input, wherein the teacher value corresponds to a teacher probability of the one or more sections of the respective output that is dependent on the respective input. [21] One or more non-transitory computer-readable media according to claim 19, wherein the operations comprise: for each section of a multitude of sections of the respective edition: Determining a bin-specific divergence metric that characterizes a similarity between a student probability distribution over a set of candidate output bins and a teacher probability distribution over the set of candidate output bins, wherein each of the student probability distribution and the teacher probability distribution depends on the respective input and one or more bins of the respective output preceding the bin; and Aggregate the plurality of section-specific divergence metrics for the respective output to obtain the first component. [22] One or more non-transitory computer-readable media according to claim 21, wherein the teacher probability distributions for each of the section-specific divergence metrics are generated at least partially in parallel by the machine-learned teacher sequence processing model. [23] One or more non-transitory computer-readable media according to claim 19, wherein the multi-scale refinement objective comprises one or more weighting parameters that weight the respective contributions of the first component and the second component. [24] One or more non-transitory computer-readable media according to claim 19, wherein the reinforcement learning signal comprises data indicative of human feedback on an overall quality of the respective output. [25] One or more non-transitory computer-readable media according to claim 19, wherein the reinforcement learning signal comprises data indicative of a score generated by a machine-learned reward model, the score indicative of an overall quality of the respective output. [26] One or more non-transitory computer-readable media according to claim 19, wherein evaluating the divergence metric comprises: Determining a value of a mixture distribution corresponding to a mixture of a student probability distribution of the machine-learned student sequence processing model and a teacher probability distribution of the machine-learned teacher sequence processing model; Calculating a first divergence component that characterizes a divergence of the student probability distribution with respect to the mixture distribution; Calculating a second divergence component that characterizes a divergence of the teacher probability distribution with respect to the mixture distribution; and Evaluate the divergence metric based on a combination of the first divergence component and the second divergence component. [27] One or more non-transitory computer-readable media according to claim 26, wherein evaluating the divergence metric based on the first divergence component and the second divergence component comprises: Calculating a weighted combination of the first divergence component and the second divergence component using a weighting parameter. [28] One or more non-transitory computer-readable media according to claim 27, wherein adjusting the weighting parameter causes the divergence metric to interpolate between a mode search behavior and a mean search behavior. [29] One or more non-transitory computer-readable media according to claim 28, wherein the operations comprise: Adjusting the weighting parameter based on a desired output diversity for a task type. [30] One or more non-transitory computer-readable media according to claim 29, wherein the weight is a hyperparameter learned during training. [31] One or more non-transitory computer-readable media according to claim 19, wherein the machine-learned teacher sequence processing model was not trained using reinforcement learning. [32] One or more non-transitory computer-readable media according to claim 31, wherein the machine-learned student sequence processing model has been fine-tuned to achieve a baseline performance threshold prior to training with the multi-scale refinement objective. [33] One or more non-transitory computer-readable media according to claim 19, wherein: the machine-learned student sequence processing model is characterized by a first number of parameters; the machine-learned teacher sequence processing model is characterized by a second number of parameters; and the second number of parameters is greater than the first number of parameters. [34] One or more non-transitory computer-readable media according to claim 33, wherein the second number of parameters is at least 30 times the first number of parameters. [35] One or more non-transitory computer-readable media according to claim 19, wherein the operations comprise: Receiving a request to perform an inference task based on input data from a client computing system; Obtaining the respective input from the input data; Generating the respective output using the machine-learned student sequence processing model; Returning output data to the client computing system based on the respective output and in response to the request; Receiving feedback data from the client computing system; and Determine the reinforcement learning signal based on the feedback data. [36] One or more non-transitory computer-readable media according to claim 35, wherein the operations comprise: Receiving the request and returning the output data in an online process; and Obtaining the plurality of predictions of the machine-learned teacher sequence processing model and updating the machine-learned student sequence processing model based on the multi-scale refinement objective in an offline process.

Images