Calibrating distillation

Abstract

Techniques are provided for calibrating distillation learning from a teacher model to a student model. In particular, the present disclosure presents systems and methods that provide high quality and high speed convergence. That is, the proposed example system not only enables the distillation loss to be minimized at a probability mean in the probability domain of the teacher prediction distribution, but also provides fine (e.g., symmetric and / or strong) convex losses around logi t and / or optimal values in the probability domain (e.g., including away from the minimum) to encourage the rapid convergence (e.g., fast convergence) of gradient-based methods. And regardless of the distance from the minimum value).

Description

Technical Field

[0001] The present disclosure relates generally to machine learning and more particularly to techniques for calibrating distillation learning from a teacher model to a student model. Background Art

[0002] In machine learning, knowledge distillation generally refers to the process of transferring knowledge (e.g., via distillation training) from a teacher model to a student model. Typically, although not necessarily, the teacher model will be larger than the student model (e.g., in terms of the number of parameters). Specifically, although large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity may not be fully utilized or needed in all cases. For example, since smaller models are less expensive to evaluate, they can be deployed on less powerful hardware (such as mobile devices). More generally, student models can be designed to be simpler, train faster, and / or be deployed according to deployment (e.g., system constraints) restrictions. The teacher model does not have to comply with such restrictions and can take more time to train. Therefore, there are many situations where knowledge distillation from a teacher model to a student model can bring benefits. Summary of the invention

[0003] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0004] An example aspect of the present disclosure relates to a computing system for performing distillation training with improved computational efficiency. The computing system includes: one or more processors; a teacher model, the teacher model including a teacher model body, a teacher logit head, and a teacher prediction head, wherein the teacher model body is configured to process inputs to generate a teacher intermediate representation, wherein the teacher logit head is configured to process the teacher intermediate representation to generate a teacher logit value, and wherein the teacher prediction head is configured to process the teacher logit value to generate a teacher probability value; a student model, the student model including a student model body, a first student logit head, a second student logit head, and a student prediction head, wherein the student model body is configured to process inputs to generate a student intermediate representation, wherein the first student logit head is configured to process the student intermediate representation to generate a first student logit value, wherein the second student logit head is configured to process the student intermediate representation to generate a second student logit value, and wherein the student prediction head is configured to process the first student logit value and the second student logit value to generate a student probability value; and one or more non-transitory computer-readable media, the one or more non-transitory computer-readable media collectively storing instructions that, when executed by the one or more processors, cause the computing system to perform operations. These operations include: evaluating a first loss function based on a teacher logit value and a first student logit value; modifying one or more parameters of at least a first student logit head based on the first loss function; evaluating a second different loss function based on a teacher probability value and a student probability value; and modifying one or more parameters of at least a second student logit head based on the second loss function.

[0005] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media that collectively store: a machine learning student model, wherein: the machine learning student model includes a student model body, a first student logit head, a second student logit head, and a student prediction head, the student model body being configured to process an input to generate a student intermediate representation, the first student logit head being configured to process the student intermediate representation to generate a first student logit value, the second student logit head being configured to process the student intermediate representation to generate a second student logit value, the student prediction head being configured to process the first student logit value and the second student logit value to generate a student probability value, the first student logit head having been trained using a first loss function that evaluates the first student logit value and the teacher logit value generated by a teacher model, and the second student logit head having been trained using a second loss function that evaluates the student probability value and the teacher probability value generated by the teacher model; and instructions for running the machine learning student model to process the input to generate the student probability value.

[0006] Another example aspect of the present disclosure relates to a computing system for performing distillation training with improved computational efficiency, the computing system comprising: one or more processors; a teacher model, the teacher model comprising a teacher model body, a teacher logit head, and a teacher prediction head, wherein the teacher model body is configured to process an input to generate a teacher intermediate representation, wherein the teacher logit head is configured to process the teacher intermediate representation to generate a teacher logit value, and wherein the teacher prediction head is configured to process the teacher logit value to generate a teacher probability value; a plurality of student models, wherein each student model comprises a student model body, a first student logit head, and a second student logit head, wherein the student model The model body is configured to process an input to generate a student intermediate representation, wherein a first student logit head is configured to process the student intermediate representation to generate a first student logit value, wherein a second student logit head is configured to process the student intermediate representation to generate a second student logit value; a student ensemble prediction head, the student ensemble prediction head is configured to generate a student probability value according to a plurality of first student logit values ​​and a plurality of second student logit values ​​from a plurality of student models; and one or more non-transitory computer-readable media, the one or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more processors, cause a computing system to perform operations. These operations include, for each student model in a plurality of student models: evaluating a first loss function based on a teacher logit value and a first student logit value; modifying one or more parameters of at least the first student logit head based on the first loss function; evaluating a second different loss function based on the teacher probability value and the student probability value; and modifying one or more parameters of the second student logit head of each student model based on the second loss function.

[0007] Another example aspect of the present disclosure relates to a computing system for performing distillation training with improved computational efficiency. The computing system includes: one or more processors; a teacher model, the teacher model including a teacher model body, a first teacher scoring head, and a second teacher scoring head, wherein the teacher model body is configured to process an input to generate a teacher intermediate representation, wherein the first teacher scoring head is configured to process the teacher intermediate representation to generate a first teacher scoring value in a first scoring domain, and wherein the second teacher scoring head is configured to process the first teacher scoring value to generate a second teacher scoring value in a second scoring domain, wherein the second scoring domain corresponds to a target of the teacher model; a student model, the student model including a student model body, a first student scoring head, a second student scoring head, and a third student scoring head, wherein The student model body is configured to process an input to generate a student intermediate representation, wherein a first student scoring head is configured to process the student intermediate representation to generate a first student score value in a first scoring domain, wherein a second student scoring head is configured to process the student intermediate representation to generate a second student score value in the first scoring domain, and wherein a third student scoring head is configured to process the first student score value and the second student score value to generate a third student score value in a second scoring domain; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations. The operations include evaluating a first loss function based on a first teacher score value and a first student score value; modifying one or more parameters of at least the first student scoring head based on the first loss function; evaluating a second different loss function based on a second teacher score value and a third student score value; and modifying one or more parameters of at least the second student scoring head based on the second loss function.

[0008] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0011] Figure 1 A graphical diagram showing an example forward pass during an example calibration distillation training method according to an example embodiment of the present disclosure is shown.

[0012] Figure 2A graphical diagram illustrating an example backward pass during an example calibration distillation training method according to an example embodiment of the present disclosure is shown.

[0013] Figure 3 A graphical diagram illustrating an example backward pass during an example calibration distillation training method according to an example embodiment of the present disclosure is shown.

[0014] Figure 4 A graphical diagram showing an example forward pass during inference according to an example embodiment of the present disclosure is shown.

[0015] FIG. 5A to FIG. 5C A graphical diagram illustrating an example calibration distillation training method applied to multiple student models simultaneously according to an example embodiment of the present disclosure is shown.

[0016] Fig. 6A A block diagram of an example computing system is depicted according to an example embodiment of the present disclosure.

[0017] Figure 6B A block diagram of an example computing device is depicted according to an example embodiment of the present disclosure.

[0018] Figure 6C A block diagram of an example computing device is depicted according to an example embodiment of the present disclosure.

[0019] Reference numerals that are repeated across multiple drawings are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0020] Overview

[0021] In general, the present disclosure relates to techniques for calibrating distillation learning from a teacher model to a student model. Specifically, the present disclosure proposes systems and methods that provide high-quality and high-speed convergence. That is, the proposed method enables the loss to converge quickly and then be calibrated to converge to the correct optimal value. For example, the proposed system enables the distillation loss to be minimized at the probability mean in the probability domain of the teacher's prediction distribution (e.g., as an appropriate scoring rule), while also providing a refined (e.g., symmetric and / or strong) convex loss around the optimal value in the logit and / or probability domain (e.g., including away from the minimum) to encourage the rapid convergence of gradient-based methods (e.g., regardless of the distance from the minimum). As an example, when optimizing the logistic loss, it is best to converge to the probability mean. However, the described method can also be applied to other losses to ensure convergence to the correct minimum point (whatever it may be) at a fast convergence rate by ensuring a strong convex loss or a refined convex loss. The method has particular benefits when the proposed method is applied to a distribution of examples that the teacher performs the same as the student.

[0022] The proposed system can promote the above benefits by performing distillation training according to a two-stage (or approach) method. In the first stage or approach, a distillation loss that gives good convergence can be used, such as L1, L2, or quantile regression-based distillation. For example, the loss can be applied in the logical space between the teacher and the first head of the student. In the second stage, the prediction can be calibrated towards the expected optimal value, for example by applying a cross-entropy loss calibration. For example, the loss can be applied in the probability space between the teacher and the student, where the student probability has been generated at least in part using the student's second different head. These two stages can be applied together in both the forward path and the reverse path.

[0023] More specifically, multiple losses and configurations have been proposed and considered for knowledge distillation. A major aspect of distillation is the ability of the teacher to express the enhancement of examples. Specifically, the student can only express a single prediction for a series of examples, since the features are only available to the teacher. On the other hand, the teacher has access to many feature / parameter dimensions that the student does not have access to. This allows the teacher to produce a distribution of predicted values ​​for a series of examples, which, according to the student, are summarized into a single prediction.

[0024] Choosing an appropriate loss is important for improving the quality of distillation and the speed of convergence. Specifically, the distillation loss should be minimized at the probability mean in the probability domain of the teacher's predicted distribution of the series of examples that the student regards as one example to minimize the cross entropy loss objective. If a different loss is optimized, the point on the distribution where the loss is minimized may be different. In addition, the loss should be a fine convex (e.g., preferably symmetric convex, and even more preferably strongly convex) loss around any such optimal value (including far away from the minimum) in the logit and probability domains to encourage fast convergence of gradient-based methods, regardless of whether we are closer or farther away from the minimum. However, none of the known or practical methods in the art that attempt to use a single loss can fully satisfy these two properties.

[0025] Given that existing methods fail to satisfy these two properties, the present disclosure provides systems and methods that satisfy the above requirements using a method that operates on two stages (eg, which may correspond to two loss paths flowing through two different loss heads).

[0026] In the first stage, the training system may apply a first distillation loss (e.g., squared loss) in logit space to allow fast convergence, but not necessarily to the correct minimum (e.g., to the logit mean, which for many skewed teacher distributions is farther from the origin than the probability mean).

[0027] In the second stage, the training system calibrates the predictions using a second distillation loss (e.g., a cross entropy loss) to pull the minimum toward the correct mean (e.g., in the probability domain). The calibration loss may not be a refined convex loss, but because it operates on a loss that generates faster convergence to a minimum that is usually close to the desired minimum, it only needs to refine the predictions toward the desired minimum.

[0028] The first stage and the second stage can be performed sequentially or simultaneously (e.g., in parallel). The proposed system is general and various losses can be used in the two stages. For example, L1 or quantile regression (QR) distillation loss can be used in the first stage.

[0029] The present disclosure provides a variety of technical effects and benefits. As an example, using the proposed method to perform distillation learning can improve training efficiency (e.g., using fewer training cycles or processing iterations to achieve faster convergence). This can reduce the consumption of computing resources, such as processor usage, memory usage, and / or network bandwidth usage.

[0030] As another example technical effect and benefit, a model trained according to the proposed method can provide better results, such as more accurate results. This can improve the performance of the model and its implementing computing system with respect to many different tasks. Therefore, the system and method of the present disclosure can improve the operation of the computer.

[0031] As yet another example technical effect and benefit, the present disclosure enables more general use of student models that have been distilled from teacher models. Typically, student models are smaller (e.g., in terms of storage size) and / or run faster (e.g., require less computation, such as fewer processor operations). This can result in reduced consumption of computing resources, such as processor usage, memory usage, and / or network bandwidth usage. The teacher model can be trained once offline and used for multiple student models to be deployed or experimented with.

[0032] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.

[0033] Example Discussion of Distillation Losses

[0034] The example implementation of the present disclosure is applicable to a system that distills a teacher signal into a student signal, and we specifically want the student model to achieve the minimum cross entropy log loss on its test data. Let ti denote the teacher's prediction for example i in the logit domain, and si denote the student's prediction for the same example. One possible approach is direct label distillation. However, ti and si can also express the difference of logit pairs used to rank the distillation. Let is the teacher's prediction in the probability domain, and let is the student's prediction for example i. For logistic regression, the signal in the probability domain is related to the signal in the logical domain through the Logistic (Sigmoid) function

[0035]

[0036] Where sigma represents the logic function. For the binary logic case, let y i ∈{0, 1} is the true label of example i. Then, the logistic loss of the student’s prediction for example i is given by the following equation:

[0037]

[0038] The distillation loss is defined between a teacher signal and a student signal, where in deep networks, the backpropagated gradients are usually but not always propagated only to the student's network and features, so that the student learns the teacher's predictions (and in many cases also the true label loss). The example description in this article focuses only on the distillation loss for the teacher's predictions.

[0039] Cross-entropy distillation can be achieved by applying a distillation loss to the student predictions.

[0040]

[0041] To make it consistent with the teacher's score label A temperature parameter γ can be introduced for the temperature cross entropy logistic loss given by the following equation

[0042]

[0043] The temperature essentially stretches or compresses both the teacher and student sigmoids by the same scaling factor, and is also used to scale the loss. The expression in (4) is obtained by using (1) to and Replace the mathematical operation of (3) with the corresponding scaled sigmoid.

[0044] The squared loss uses the L2 norm of the difference. In logit, it is given by the following equation:

[0045]

[0046] and is given in probability by the following equation:

[0047]

[0048] Similarly, the L1 norm distillation loss can be defined as:

[0049]

[0050] The temperature-scaled probit distillation loss can be defined by equation (3) (scaled by temperature γ), where the probability and is equal to the normal cumulative density function (CDF) with the standard deviation equal to the temperature. (This view can be analogous to viewing the logistic predicted probability as the CDF value of the logit of the logistic distribution). The probit probability is given by the following equation:

[0051]

[0052] where Φ(.) is the standard normal CDF and erf(.) is the standard error function given by:

[0053]

[0054] The Huber loss connects the squared loss at and near the minimum with the linear loss away from the minimum. The trade-off between these two components is determined by the parameter β. For distillation, the loss is given by the following equation:

[0055]

[0056] Available in and A similar functional form is used above to distill the probability with Huber loss. The larger β is, the closer the loss is to quadratic, and the smaller β is, the closer the loss is to L1.

[0057] Quantile regression-based distillation does not directly connect student signal beliefs s i and the teacher signal belief. Conversely, for each quantile τ in the quantile value set {τ}, for the teacher signal t i Create a separate loss. The loss is related to the function q τ (s i ) is related. As the output of the deep network, q τ (s i ) can be defined as

[0058] q τ (s i )=ω τ s i +b τ (11)

[0059] where w τ and b τ are the link weights and biases, which are also derived from the teacher signal t i Learned from.

[0060] More generally, s ican be connected to q via a matrix of link weights and bias vectors τ (s i ). For example, S i can be a vector at some layer of the deep network (e.g., it could be the second-to-last layer connected to the output), and w τ can be a vector of learned weights, where b τ is the scalar bias. However, other configurations are possible. The QR distillation loss is then defined as the sum of all assigned quantiles {τ}, given by the following equation:

[0061]

[0062] Where I(.) is the indicator function and ReLU(.) is the rectified linear unit ReLU(x)=max(x,0).

[0063] Training for the distillation loss learns the tau quantile-specific parameter w τ and b τ (Or, if the network definition is different, more general parameters are learned.) This training also learns the student (logit) signal s i , the student signal can be the observed parameters, or the latent parameters of the network's internal beliefs about the student's prediction for example l0git. If the set of quantiles {τ} is symmetric (in the sense that if τ is included in the set, then 1-τ is also included in the set), then using this loss produces a loss that is minimized at the median of the teacher distribution. If {τ} = {0.5}, then the loss in (12) reduces to (a scaled version of) the loss in equation (7). The more quantiles, the smoother the loss, or it can be piecewise linear with smaller gradient jumps between segments. A variation similar to equation (12) can be used in the probabilistic domain, replacing ti and S respectively. i Replace with and

[0064] The individual quantile loss components of QR distillation can be smoothed by using a smoother function such as x^2, [ReLU(x)]^2, SmeLU, SmeLU_beta(x), swish(x), softplus(x), or other functions. When using SmeLU,

[0065]

[0066] The QR distillation loss is given by the following equation:

[0067]

[0068] If the two do not match for the teacher distribution, using a smoother function than the ReLU function will push the minimum from the median to the mean (in this case in the logit domain).

[0069] However, none of the above loss functions alone provide the desired combination of properties: (1) fast minimization to an optimum at the mean of the teacher predictions in the probability domain, and (2) a fine convex loss surface over the entire range in both the logit and probability domains. While using the Huber loss in the logit (or smoothed QR regression distillation) for certain teacher prediction distributions can provide some compromise, these methods require different tuning for different example sets and different teacher distributions, and it is not clear how to do this generally.

[0070] Example Calibration Distillation Technique

[0071] In view of the failures of the above-mentioned existing loss methods, the present disclosure proposes a system and method that provides high-quality and high-speed convergence. That is, the proposed example system enables the distillation loss to be minimized at the probability mean in the probability domain of the teacher prediction distribution, while also providing a refined (e.g., symmetric and / or strong) convex loss around the optimal value in the logit and / or probability domain (e.g., including far away from the minimum) to encourage fast convergence of gradient-based methods (e.g., regardless of the distance from the minimum).

[0072] The proposed system can promote these benefits by performing distillation training according to a two-stage (or approach) method. In the first stage or approach, a distillation loss that gives good convergence can be used, such as L1, L2 or QR distillation. For example, the loss can be applied in the logical space between the teacher and the first head of the student. In the second stage, the prediction can be calibrated towards the expected optimal value, for example by applying a cross-entropy loss calibration. For example, the loss can be applied in the probability space between the teacher and the student, where the student probabilities have been generated at least in part using the student's second different head.

[0073] Figures 1 to 3 Example aspects of the proposed training method are shown. In the first stage, a faster convergence loss can be applied to the student model's top predictions of the teacher. The result can then be passed to another loss that observes the input signal of the first loss as well as the predictions of the first loss and uses its additional parameters to calibrate the predictions of the second loss.

[0074] In some implementations, calibration may control at least link weight parameters and bias parameters that multiply the neuron activations of the penultimate deep network layer of the student and add the predictions to the first loss. More complex solutions may add more layers or parameters to handle the calibration loss. Both loss heads may apply only distillation loss, or may also apply a loss related to the true label. Backpropagation from the student to the teacher may be stopped, and backpropagation from calibrated predictions to pre-calibrated predictions (e.g., which apply a fast convergence loss) may also be stopped. The network itself may typically be set to learn from a fast learning first loss, and backpropagation but not necessarily from the calibrated loss output may be blocked. However, configurations that allow updates from either loss to the main student network may also be applied.

[0075] More specifically, Figure 1 A graphical diagram of an example forward pass during an example calibration distillation training method is shown; and Figure 2 shows a graphical diagram of an example backward pass according to a first back-propagation scheme, and Figure 3 A graphical diagram of an example backward pass according to the second back-propagation scheme is shown.

[0076] First reference Figure 1 , a distillation training scheme may be applied to distill knowledge from the teacher model 12 to the student model 14. Models 12 and 14 may include multiple heads. Each of these "heads" may be a single prediction operator connected to the previous (previous) layers of the model, and / or may include multiple hidden layers that are ultimately connected to a single prediction operator, where the weights and biases of these layers are learnable. Thus, a "head" may include a single prediction operator (e.g., a logit prediction operator, a softmax operator, etc.) or multiple neural network layers that produce such an operator.

[0077] The teacher model 12 may include a teacher model body 16, a teacher logit head 18, and a teacher prediction head 20. The teacher model body 16 may be configured to process inputs (e.g., training inputs 22) to generate a teacher intermediate representation (shown at 24). The teacher logit head 18 may be configured to process the teacher intermediate representation 24 to generate a teacher logit value 26. The teacher prediction head 20 may be configured to process the teacher logit value 26 to generate a teacher probability value 28.

[0078] The student model 14 may include a student model body 30, a first student logit head 32, a second student logit head 34, and a student prediction head 36. The student model body 30 may be configured to process an input (e.g., a training input 22) to generate a student intermediate representation (shown generally at 38). The first student logit head 32 may be configured to process the student intermediate representation 38 to generate a first student logit value 40. The second student logit head 34 may be configured to process the student intermediate representation 38 to generate a second student logit value 42. The student prediction head 36 may be configured to process the first student logit value 40 and the second student logit value 42 to generate a student probability value 44.

[0079] As an example, Figure 1 As shown, the student model 14 may be configured to add the first student logit value 40 and the second student logit value 42 to generate a combined logit value. The student prediction head 36 may be configured to process the combined logit value to generate a student probability value 44.

[0080] According to one aspect of the present disclosure, multiple loss functions may be used to train the two heads 32 and 34 of the student model 14. Specifically, Figure 1 As shown, the training system may evaluate a first loss function 46 based on the teacher logit value 26 and the first student logit value 40. The training system may evaluate a second, different loss function 48 based on the teacher probability value 28 and the student probability value 44.

[0081] Various loss functions may be used to satisfy the desired properties discussed herein. As an example, the first loss function 46 may be or include one of a squared loss, a Huber loss, a smoothed quantile loss, a quantile regression loss, or a smoothed loss. For example, the first loss function 46 may be L p Loss function.

[0082] In general, in some implementations, the first loss function 46 may converge faster than the second loss function 48. In some implementations, the second loss function 48 may converge to a point that gives a minimum at some point of the loss suitable for use with respect to the teacher's prediction distribution. If a cross-entropy logistic loss is used in the training system, such a point is the mean of the distribution probabilities of the teacher's predictions for a series of examples that appear as a single example to the student because the features are used in the teacher's predictions but not in the student's predictions. For example, the second loss function 48 may be one or both of the following: symmetrically convex or strongly convex around a convergence optimum. As an example, the second loss function 48 may be or include a cross-entropy loss function.

[0083] The two loss functions 46 and 48 may be used to train the student model 14 according to a variety of different back-propagation methods. Figure 2 A first example back-propagation method is shown in Figure 2 , the first loss function 46 is used only to modify or otherwise train the first student logit head 32 (e.g., back propagation of the loss function 46 stops at the base of the head 32). (The head may itself be a network including several hidden layers.) However, the second loss function 48 may be used to modify or otherwise train both the second student logit head 34 and the student model body 30.

[0084] Figure 3 A second example back-propagation method is shown in Figure 3 , the first loss function 46 is used to modify or otherwise train both the first student logit head 32 and the student model body 30. However, the second loss function 48 is only used to modify or otherwise train the second student logit head 34 (e.g., back propagation of the loss function 48 stops at the base of the head 34).

[0085] Figure 4 FIG. 1 shows a graphical diagram of an example forward pass during inference according to an example embodiment of the present disclosure. Specifically, the forward pass through the student model at the inference stage may follow the same procedure as in Figure 1 The forward pass through the student model during the training phase is shown in the same way, except that the input is the inference input instead of the training input, and also differs in that the teacher is not included in the inference.

[0086] One example application of the proposed method is to improve the cross entropy log loss.Thus, in some implementations, the student prediction head may be or include a logistic function, and the student probability value may be or include a logistic regression output.

[0087] Another example application is pairs or lists of examples, where methods such as those proposed here can be applied to a pairwise / listwise ranking loss. For ranking (e.g., ranking of individual examples), applying a strong logit loss (e.g., such as a squared loss on the logit, or a loss such as QR distillation) may sometimes be sufficient to meet the ranking objective (e.g., this may not necessarily be consistent with a cross entropy loss).

[0088] Another aspect of the present disclosure relates to applying the proposed method to student model ensembles. More specifically, some systems use ensembles where each component of the model is trained independently and distillation is applied independently. The final prediction then averages the individual predictions (or uses a mixture of experts).

[0089] like Figure 5AAs shown, the proposed method can also be applied to student ensembles. The ensemble can contain any number (n) of student models. Specifically, for each student model in the ensemble, an independent distillation loss can be applied relative to the teacher's prediction. Specifically, a first loss function 46 (e.g., an initial fast convergence distillation loss) can be applied to each student model individually (e.g., at the first second logit head 1-n). However, a second calibration loss 48 can be applied on top of the ensemble average prediction. Specifically, Figure 1 and Figure 5A For comparison, a second loss function 48 that applies the calibration loss effect may be employed over the ensemble, rather than applying the calibration loss on the individual component models.

[0090] Therefore, when applied to integration, the first loss 46 and the second loss 48 may be similar to Figure 2 and Figure 3 For example, in some implementations, the second loss function 48 may only be responsible for updating the biases of the link weights and matrix multiplication weights applied to the top-level set of different integrated components, which weights and biases may be concatenated into a layer, or assembled separately and then added together to generate the final calibration output. Figure 5A The circles in (which combine the second student logit values ​​1-n) can then be interpreted as the connections of the hidden layers closest to the output of the ensemble components, on which the link weights are applied to generate a residual signal that calibrates the final prediction. The stop gradient operation prevents the updates from being propagated to each of the networks that make up the student body, thereby retaining only the calibrated updates to the link weights and biases of the calibration matrix multiplication. The final logit value can then be added to the ensemble uncalibrated prediction value to produce the calibrated prediction value. Figure 5B and Figure 5C Two different example back-propagation methods are shown that may be applied.

[0091] Now collective reference Figures 1 to 5C Although the example embodiments are described with reference to applying the first loss function and the second loss function at the logit and probability domains, alternative example implementations of the present disclosure may also be applied to other scoring domains, such as a regression domain that does not use probability.

[0092] Example Apparatus and Systems

[0093] Fig. 6A A block diagram of an example computing system 100 is depicted, in accordance with an example embodiment of the present disclosure. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0094] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0095] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 executed by the processor 112 to cause the user computing device 102 to perform operations.

[0096] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a Transformer model). Reference Figure 1 5 discusses an example machine learning model 120.

[0097] In some implementations, one or more machine learning models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel distillation across multiple instances of a teacher and / or student).

[0098] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine learning model 140 may be implemented by the server computing system 140 as part of a web service. Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0099] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices by which a user can provide user input.

[0100] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0101] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0102] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the model 140 may be or may otherwise include a variety of machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a Transformer model). Reference Figure 15 discusses an example model 140 .

[0103] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0104] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 154 may store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0105] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques (e.g., such as error back propagation). For example, a loss function may be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over multiple training iterations.

[0106] In some implementations, performing error back-propagation may include performing truncated back-propagation through time.The model trainer 160 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization capabilities of the model being trained.

[0107] Specifically, model trainer 160 may train machine learning models 120 and / or 140 based on training data set 162. In some implementations, if the user has provided consent, the training examples may be provided by user computing device 102. Thus, in such implementations, model 120 provided to user computing device 102 may be trained by training computing system 150 based on user-specific data received from user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0108] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 may be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more computer executable instruction sets stored in a tangible computer readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0109] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted over any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0110] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0111] In some implementations, the input to the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, encoded representation of the image data, hashing of the image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change to the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0112] In some implementations, the input to the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0113] In some implementations, the input to the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech translation output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a predicted output.

[0114] In some implementations, the input of the machine learning model of the present disclosure may be latent coded data (e.g., a latent space representation of an input, etc.). The machine learning model may process the latent coded data to generate an output. As an example, the machine learning model may process the latent coded data to generate a recognition output. As another example, the machine learning model may process the latent coded data to generate a reconstruction output. As another example, the machine learning model may process the latent coded data to generate a search output. As another example, the machine learning model may process the latent coded data to generate a re-clustering output. As another example, the machine learning model may process the latent coded data to generate a prediction output.

[0115] In some implementations, the input to the machine learning model of the present disclosure may be statistical data. Statistical data may be, represent, or otherwise include data calculated and / or computed from some other data source. The machine learning model may process statistical data to generate an output. As an example, the machine learning model may process statistical data to generate an identification output. As another example, the machine learning model may process statistical data to generate a prediction output. As another example, the machine learning model may process statistical data to generate a classification output. As another example, the machine learning model may process statistical data to generate a segmentation output. As another example, the machine learning model may process statistical data to generate a visualization output. As another example, one or more machine learning models may process statistical data to generate a diagnostic output.

[0116] In some implementations, the input to one or more machine learning models of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, one or more machine learning models may process the sensor data to generate a detection output.

[0117] In some cases, the machine learning model may be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data, and the output may include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task may include generating an embedding for the input data (e.g., input audio or visual data).

[0118] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data for one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the likelihood that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, a corresponding likelihood for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel in one of the input images, the motion of the scene depicted at the pixel between the images in the network input.

[0119] In some cases, the input includes audio data representing spoken utterances, and the task is a speech recognition task. The output may include a text output mapped to the spoken utterance. In some cases, the task includes encrypting or decrypting the input data. In some cases, the task includes a microprocessor performance task, such as branch prediction or memory address translation.

[0120] Fig. 6A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training data set 162. In such implementations, the model 120 may be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0121] Figure 6B Depicted is a block diagram of an example computing device 10 performing in accordance with an example embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0122] Computing device 10 includes multiple applications (e.g., Application 1 to Application N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0123] like Figure 6B As shown, each application can communicate with multiple other components of the computing device (e.g., such as one or more sensors, context managers, device state components, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0124] Figure 6C Depicted is a block diagram of an example computing device 50 performing in accordance with an example embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0125] The computing device 50 includes a plurality of applications (e.g., Application 1 to Application N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0126] The central intelligence layer includes multiple machine learning models. For example, Figure 6C As shown, a corresponding machine learning model may be provided for each application, and the corresponding machine learning model may be managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 50 or is otherwise implemented by the operating system.

[0127] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for computing devices 50. Figure 6C As shown, the central device data layer can communicate with multiple other components of the computing device (e.g., such as one or more sensors, context managers, device state components, and / or additional components). In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0128] Additional disclosure

[0129] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitions of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0130] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, may easily produce changes, modifications, and equivalents to these embodiments. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet further embodiments. Therefore, the present disclosure is intended to cover these changes, modifications, and equivalents.

Claims

1. A computing system for performing distillation training with improved computational efficiency, the computing system comprising: one or more processors; a teacher model, the teacher model comprising a teacher model body, a teacher logit head, and a teacher prediction head, wherein the teacher model body is configured to process an input to generate a teacher intermediate representation, wherein the teacher logit head is configured to process the teacher intermediate representation to generate a teacher logit value, and wherein the teacher prediction head is configured to process the teacher logit value to generate a teacher probability value; a student model, the student model comprising a student model body, a first student logit head, a second student logit head, and a student prediction head, wherein the student model body is configured to process an input to generate a student intermediate representation, wherein the first student logit head is configured to process the student intermediate representation to generate a first student logit value, wherein the second student logit head is configured to process the student intermediate representation to generate a second student logit value, and wherein the student prediction head is configured to process the first student logit value and the second student logit value to generate a student probability value; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: Evaluating a first loss function based on the teacher logit value and the first student logit value; modifying one or more parameters of at least the first student-logit head based on the first loss function; evaluating a second different loss function based on the teacher probability value and the student probability value; and One or more parameters of at least the second student-logit head are modified based on a second loss function.

2. The computing system of claim 1, wherein the first loss function comprises one of a squared loss, a Huber loss, a smoothed quantile loss, a quantile regression loss, or a smoothed loss.

3. A computing system as claimed in any preceding claim, wherein the first loss function comprises L p Loss function.

4. A computing system as described in any preceding claim, wherein the first loss function converges faster than the second loss function.

5. A computing system as described in any preceding claim, wherein the second loss function includes an appropriate scoring rule that minimizes at expected statistical data points in an appropriate domain of the predictive distribution produced by the teacher.

6. The computing system of any preceding claim, wherein one or both of the first loss function and the second loss function are one or both of the following: symmetrically convex or strongly convex around a convergence optimum.

7. A computing system as described in any preceding claim, wherein the second loss function comprises a cross-entropy loss function that gives a minimum value at the predicted mean probability with respect to the predicted distribution.

8. The computing system of any preceding claim, further comprising: One or more parameters of the student subject are modified based on the first loss function.

9. The computing system of any one of claims 1 to 7, further comprising: One or more parameters of the student subject are modified based on the second loss function.

10. A computing system as claimed in any preceding claim, wherein: The student model is configured to add the first student logit value and the second student logit value to generate a combined logit value; and The student prediction head is configured to process the combined logit values ​​to generate the student probability values.

11. The computing system of any preceding claim, wherein the student prediction head comprises a logistic function and the student probability value comprises a logistic regression output.

12. The computing system of any preceding claim, wherein the teacher probability values ​​are stored in and accessed from a non-transitory computer-readable medium for use in training the student model.

13. One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media collectively storing: A machine learning student model where: The machine learning student model includes a student model body, a first student logit head, a second student logit head and a student prediction head, The student model body is configured to process the input to generate a student intermediate representation, The first student logit head is configured to process the student intermediate representation to generate a first student logit value, The second student logit head is configured to process the student intermediate representation to generate second student logit values, The student prediction head is configured to process the first student logit value and the second student logit value to generate a student probability value, The first student logit head has been trained using a first loss function that evaluates the first student logit values ​​and teacher logit values ​​generated by a teacher model, and The second student logit head has been trained using a second loss function that evaluates the student probability values ​​and the teacher probability values ​​generated by the teacher model; as well as Instructions for running the machine learning student model to process inputs to generate the student probability values.

14. The one or more non-transitory computer-readable media of claim 13, wherein: The machine learning student model is configured to add the first student logit value and the second student logit value to generate a combined logit value; and The student prediction head is configured to process the combined logit values ​​to generate the student probability values.

15. The one or more non-transitory computer-readable media of claim 13 or 14, wherein the first loss function comprises a squared loss function and the second loss function comprises a cross entropy loss function.

16. A computing system for performing distillation training with improved computational efficiency, the computing system comprising: one or more processors; a teacher model, the teacher model comprising a teacher model body, a teacher logit head, and a teacher prediction head, wherein the teacher model body is configured to process an input to generate a teacher intermediate representation, wherein the teacher logit head is configured to process the teacher intermediate representation to generate a teacher logit value, and wherein the teacher prediction head is configured to process the teacher logit value to generate a teacher probability value; a plurality of student models, wherein each student model comprises a student model body, a first student logit head, and a second student logit head, wherein the student model body is configured to process an input to generate a student intermediate representation, wherein the first student logit head is configured to process the student intermediate representation to generate a first student logit value, wherein the second student logit head is configured to process the student intermediate representation to generate a second student logit value; a student ensemble prediction head configured to generate a student probability value according to a plurality of the first student logit values ​​and a plurality of the second student logit values ​​from the plurality of student models; one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: For each student model in the plurality of student models: Evaluating a first loss function based on the teacher logit value and the first student logit value; modifying one or more parameters of at least the first student-logit head based on the first loss function; evaluating a second different loss function based on the teacher probability value and the student probability value; and One or more parameters of the second student logit head of each student model are modified based on the second loss function.

17. The computing system of claim 16, wherein the first loss function comprises one of a squared loss, a Huber loss, a smoothed quantile loss, a quantile regression loss, or a smoothed loss.

18. The computing system of claim 16 or 17, wherein the first loss function comprises L p Loss function.

19. The computing system of claim 16, 17 or 18, wherein the first loss function converges faster than the second loss function.

20. The computing system of claim 16, 17, 18, or 19, wherein the second loss function converges to a point that gives a minimum loss with respect to the distribution of teacher predictions over examples that appear identical to the student model.

21. A computing system for performing distillation training with improved computational efficiency, the computing system comprising: one or more processors; a teacher model, the teacher model comprising a teacher model body, a first teacher rating head, and a second teacher rating head, wherein the teacher model body is configured to process an input to generate a teacher intermediate representation, wherein the first teacher rating head is configured to process the teacher intermediate representation to generate a first teacher rating value in a first rating domain, and wherein the second teacher rating head is configured to process the first teacher rating value to generate a second teacher rating value in a second rating domain, wherein the second rating domain corresponds to an objective of the teacher model; a student model, the student model comprising a student model body, a first student scoring head, a second student scoring head, and a third student scoring head, wherein the student model body is configured to process an input to generate a student intermediate representation, wherein the first student scoring head is configured to process the student intermediate representation to generate a first student score value in the first scoring domain, wherein the second student scoring head is configured to process the student intermediate representation to generate a second student score value in the first scoring domain, and wherein the third student scoring head is configured to process the first student score value and the second student score value to generate a third student score value in the second scoring domain; as well as one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: evaluating a first loss function based on the first teacher rating value and the first student rating value; modifying one or more parameters of at least the first student-scoring head based on the first loss function; evaluating a second different loss function based on the second teacher rating value and the third student rating value; and One or more parameters of at least the second student-scoring head are modified based on a second loss function.

22. The computing system of claim 21, wherein the first scoring domain comprises a logit domain, and wherein the second scoring domain comprises a probability domain.

Images