Epistemological Machine Learning Models

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for processing an input using a epistemological machine learning model that improves the quality of an output produced by a base machine learning model.

Description

[Technical field]

[0001] This specification relates to using machine learning models to process inputs. [Background technology]

[0002] As an example, a neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to another layer in the network, for example, the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current value of a respective set of weights. Summary of the Invention [Means for solving the problem]

[0003] This specification describes a system, implemented as a computer program on one or more computers, that performs machine learning tasks by processing inputs using an epistemic machine learning model.

[0004] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0005] Traditional machine learning models, e.g., neural networks, generate marginal predictions that cannot be used to distinguish between aleatory and epistemic uncertainty: the output of a traditional machine learning model for a given input that is ambiguous, i.e., assigns significant probability mass to two or more possible outputs, does not indicate whether the output is ambiguous due to aleatory or epistemic uncertainty.

[0006] Aleatory uncertainty refers to uncertainty that cannot be resolved by training a machine learning model on more or different data, for example because the input is ambiguous and cannot be precisely classified or precisely modeled.

[0007] Epistemic uncertainty refers to uncertainty that can be ameliorated with additional data, e.g., by training a machine learning model on more data or on different data, i.e., if the model had been trained differently, it might have produced a more accurate prediction for a given input.

[0008] The lack of ability to distinguish between these two types of uncertainty can prevent traditional models from being deployed in many real-world situations that require reliable predictions.

[0009] For example, a robot or other agent can be more effectively controlled if the output of a policy model for the agent indicates whether the model is uncertain about what action to take in a given state because the state is ambiguous, or because the model has not been trained on appropriate data.

[0010] As another example, image classification outputs could be more usefully incorporated into real-world systems that make decisions based on the output if information was available indicating whether the underlying image was ambiguous or whether the model needed to be trained on more data to make valid predictions.

[0011] This specification describes techniques for extending a traditional machine learning model (called a "base machine learning model") with an epistemic machine learning model. The use of the epistemic model allows the final output generated based on the outputs of the traditional model and the epistemic model to provide an indication of whether a given output is generated as a result of epistemic uncertainty or aleatory uncertainty.

[0012] Additionally, epistemic models generally consume significantly less resources than base models, which may be large-scale language models, large-scale computer vision models, or other models with millions, and possibly billions, of parameters. Thus, extending the base model with outputs generated by the epistemic model can provide information about the uncertainty in the predictions of the base model with relatively little additional computational overhead.

[0013] In particular, the described techniques for generating the final output better reflect uncertainty than existing techniques while being significantly more computationally efficient than existing techniques because incorporating uncertainty information using the described techniques requires only one or more forward passes through one or more models that consume a relatively small amount of computational resources, rather than requiring additional forward passes through the base model, which may be computationally expensive.

[0014] For example, the described techniques can provide improved uncertainty indications compared to ensembles of multiple models. In particular, the described techniques can outperform ensembles of hundreds of models at a computational cost less than that of an ensemble with two models.

[0015] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram of an example machine learning system. [Diagram 2] FIG. 1 illustrates an example architecture of uncertainty indications provided by outputs of traditional and epistemological models. [Diagram 3] FIG. 1 illustrates an example of operations performed by a machine learning system. [Figure 4] FIG. 1 is a flow diagram of an example process for processing model inputs. [Diagram 5] FIG. 1 illustrates an example of results achieved by a machine learning system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Like reference numbers and designations in the various drawings indicate like elements.

[0018] This specification describes a system, implemented as a computer program on one or more computers, that performs machine learning tasks by processing inputs using epistemological machine learning models.

[0019] The machine learning tasks performed by the system may be any suitable machine learning tasks.

[0020] For example, the machine learning task may be a computer vision task (also referred to as an "image processing task"). In other words, the epistemic machine learning model includes a convolutional neural network, or a different type of neural network (e.g., a transformer-based neural network), configured to receive at least one input image (such as an image captured by a camera, a still camera, or a video camera) and process the input image to generate a model output for the input image, i.e., perform some kind of image processing task. In this specification, processing the input image refers to using the machine learning model to process intensity values ​​of pixels of the image.

[0021] For example, the task may be image classification, and the output generated by the system for a given image may be a respective score for each of a set of object categories, each score representing the estimated likelihood that the image contains an image of an object belonging to that category.

[0022] As another example, the task may be image embedding generation, and the output generated by the system may be a numerical embedding of the input image.

[0023] As yet another example, the task may be object detection, and the output generated by the system may identify locations in the input image, e.g., bounding boxes or other geometric regions within the image, in which particular types of objects are indicated.

[0024] As yet another example, the task may be image segmentation, and the output generated by the system may define, for each pixel of the input image, to which of multiple categories the pixel belongs.

[0025] More generally, however, the task may be any of a variety of tasks, including tasks that process inputs other than images.

[0026] As an example, if the input to the system is an Internet resource (e.g., a web page), document, or document portion, or features extracted from an Internet resource, document, or document portion, the task may be to classify the resource or document, i.e., the output generated by the model for a given Internet resource, document, or document portion may be a score for each of a set of topics, with each score representing an estimated likelihood that the Internet resource, document, or document portion is about that topic.

[0027] As another example, if the input to the system is impression context features for a particular advertisement, the output generated by the model may be a score representing the estimated likelihood that a particular advertisement will be clicked on.

[0028] As another example, if the inputs to the system are features of a personalized recommendation for a user, e.g., one or more of features characterizing the context for the recommendation, features characterizing previous actions taken by the user, or features characterizing the user, the output generated by the model may be a score for each of a set of content items, each score representing an estimated likelihood that the user will respond favorably to being recommended that content item. The content items may be, for example, videos, software applications, electronic documents such as e-books, advertisements, etc.

[0029] As another example, if the input to the system is a sequence of text in one language, the output produced by the model may be a score for each of a set of several texts in another language, with each score representing an estimated likelihood that that text in that other language is an appropriate translation from the input text into that other language.

[0030] As another example, the task may be an audio processing task. For example, if the input to the system is a sequence representing an oral utterance (e.g., as captured by a microphone), the output generated by the system may be a score for each of a set of several texts, each score representing an estimated likelihood that the text is a correct transcript for the utterance. As another example, the task may be a keyword spotting task, where if the input to the system is a sequence representing an oral utterance, the output generated by the system may indicate whether a particular word or phrase (a "hot word") was spoken in the utterance. As another example, if the input to the system is a sequence representing an oral utterance, the output generated by the system may identify the natural language in which the utterance was spoken.

[0031] As another example, a task may be a natural language processing or understanding task that operates on a sequence of text in some natural language, such as an entailment task, a paraphrase task, a text similarity task, a sentiment task, a sentence completion task, a grammaticality task, etc.

[0032] As another example, the task may be a text-to-speech task, where the input is text in a natural language, or features of text in a natural language, and the model output is a spectrogram or other data that defines the audio of the text as spoken in the natural language.

[0033] As another example, the task may be a health prediction task, where the input is electronic health record data for a patient and the output is a prediction related to the patient's future health, e.g., a predicted treatment that should be prescribed for the patient, the likelihood that an adverse health event will occur for the patient, or a predicted diagnosis for the patient.

[0034] As another example, the task may be an agent control task, where the inputs are observations that characterize the state of the environment and the outputs define actions to be performed by the agent in response to the observations.

[0035] For example, the output may be a respective score for each action in a set of actions that can be performed by the agent, each action in the set of actions may be assigned a respective Q-value, a probability distribution over the set of actions may be specified, or the actions to be performed by the agent may be directly regressed.

[0036] The agent may be, for example, a mechanical agent, e.g., a robot or autonomous vehicle that interacts with a real-world environment (where the observations may include the output of one or more sensors, e.g., a camera, that detect the real-world environment and the actions may be for the mechanical agent to move within the environment and / or change its configuration), a software agent, e.g., a computer simulation of a robot or autonomous vehicle that interacts in a computer simulation of a real-world environment, a control system for an industrial facility (e.g., controlling the heating and / or cooling systems of an industrial facility or controlling manufacturing equipment within the facility and the observations may comprise temperature and / or power consumption observations of one or more locations within the facility), a system for distributing resources (e.g., power or computing resources) among multiple systems configured to employ those resources, or a control system that controls different types of agents.

[0037] In some cases, the machine learning task is a combination of multiple individual machine learning tasks, i.e., the neural network is configured to perform multiple different individual machine learning tasks, e.g., two or more of the machine learning tasks described above. For example, the neural network may be configured to perform multiple individual natural language understanding tasks. In some cases, the network input may include an identifier for the individual natural language understanding task to be performed at the network input. As another example, the neural network may be configured to perform multiple individual image processing tasks or computer vision tasks, i.e., by processing a single input image to generate outputs for multiple different individual image processing tasks in parallel.

[0038] 1 is a diagram of an example machine learning system 100. Machine learning system 100 is an example of a system, implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described below may be implemented.

[0039] The system 100 receives model input 102 for a machine learning task, e.g., input data for one of the tasks described above (e.g., image data if the task is an image processing task), and processes the model input to generate a final output 112 for the task.

[0040] In particular, when the system 100 receives a new model input 102, the system 100 processes the new model input 102 using the base machine learning model 110 to generate a base output 114 for the machine learning task for the new model input 102.

[0041] For example, when the machine learning task is a classification task, the base output 114 may include a respective logit, e.g., a respective score, for each of a set of categories for the classification task, where the logit indicates the likelihood that a new model input is in that category.

[0042] As another example, when the machine learning task is a regression task, the base output 114 may include one or more regressed values.

[0043] The base machine learning model 110 may be any suitable model configured and trained to perform a machine learning task.

[0044] Examples of machine learning models include random forests, support vector machines, gradient boosted trees, decision forests, decision trees, logistic regression models, and neural networks.

[0045] While conventional systems would provide a base output 114 of the base model 110 as the final output 112 for the model input 102, the system 100 also utilizes a cognitive machine learning model 120 to improve the quality of the base output 114 to more effectively take into account uncertainty in the predictions generated by the base model 110.

[0046] In particular, the system 100 samples a set of one or more indexes 104 from a reference distribution over the possible indexes.

[0047] In some implementations, each possible index is a scalar value and the reference distribution is, for example, a uniform distribution over a discrete set of possible indexes or a Gaussian distribution over the space of possible indexes.

[0048] In some other implementations, each possible index is a vector with a fixed number of dimensions, and the reference distribution is, for example, a multidimensional Gaussian distribution over the multidimensional space of possible indices, or a uniform distribution over a discrete set of multidimensional vectors.

[0049] In general, the sampling of the index is independent of the new model input 102. That is, the system 100 samples from the same fixed reference distribution for each model input processed by the system 100.

[0050] In some cases, the system 100 samples a fixed number of indexes for each input. In some other cases, the system 100 may determine how many indexes to sample for a given input based on some criteria. For example, the system 100 may sample the maximum number of indexes that would result in a computational cost that does not exceed the amount of available computation to generate the final output 112 for the new model input 102.

[0051] For each of one or more sampled indexes 104 in the set, the system 100 uses an epistemological machine learning model 120 to process epistemological inputs including the sampled index 104 and (i) new model inputs 102, (ii) data derived from the new model inputs 102, or (iii) both, to generate an epistemological output 124 for the machine learning task for the new model inputs 102.

[0052] Each epistemological output 124 has the same format as the base output, i.e., it has the number and type of values ​​required by the machine learning task.

[0053] That is, each epistemological input includes new model inputs, data derived from the new model inputs, or both, and additional input-sampled indices that are independent of the new model inputs.

[0054] In some cases, as described in more detail below, the data derived from the new model input may be an intermediate representation of the model input 102 generated by the base model 110 while processing the model input 102.

[0055] The system 100 then generates a final output 112 for the machine learning task from at least the base output 114 for the machine learning task and the epistemological output 124 for the set of one or more sampled indexes.

[0056] Unlike the epistemological input, the new model input processed by the base model 110 does not include any sampled indices. Thus, because the base output 114 does not depend on the sampled indices, when the system 100 samples multiple indices, the system 100 still only needs to perform one forward pass through the base model 110 to generate the final output 112.

[0057] In some implementations, the system 100 also generates a previous output using a previous epistemological machine learning model that has not been trained to process previous inputs, including the sampled indexes and (i) new model inputs, (ii) data derived from the new model inputs, or (iii) both, and generates a final output 112 using the base output, the epistemological output, and the previous output.

[0058] The epistemological model 120 will also be referred to herein as a learnable epistemological model to distinguish it from previous epistemological machine learning models, which are not learnable, i.e., because they have not received any training.

[0059] Like the base model, the epistemological model 120, and, when used, the prior model, may be any suitable machine learning model and may be the same type of model or a different type of model compared to the base model 110.

[0060] In general, the epistemological model 120 , and the prior models when used, consume significantly less resources than the base model 110 .

[0061] For example, the base model 110 may be a large language model, a large computer vision model, such as a convolutional neural network or a vision transformer, or other model with millions and possibly billions of parameters, while the epistemological model and the prior model are relatively smaller neural networks with significantly fewer parameters, e.g., thousands or hundreds of thousands of parameters. For example, the proportion of variable parameters that define the epistemological model 120 and, if any, the prior model, may be at least a factor of 1000, or at least 10,000 times less than the variable parameters that define the base model 110.

[0062] Thus, by extending the base model 110 with the epistemological model 120, and possibly with outputs generated by previous models, it is possible to provide information about the uncertainty in the predictions of the base model 110 with relatively little additional computational overhead.

[0063] In particular, the described technique for generating the final output 112 better reflects uncertainty than existing techniques while being significantly more computationally efficient than existing techniques.

[0064] The reason for this is that incorporating uncertainty information does not require an additional forward pass through the base model 110, which is computationally expensive, but rather only requires one or more additional forward passes through one or more models that consume a relatively small amount of computational resources, i.e., through the epistemological model 120 and possibly the previous model.

[0065] When a single index 104 is sampled, the system 100 generates a combined output by combining the base output, the epistemological output, and possibly the previous output. For example, the system may calculate a sum of the base output, the epistemological output, and possibly the previous output. As another example, the system may assign weights to the base output, the epistemological output, and possibly the previous output, and then calculate a weighted sum of those outputs.

[0066] The system then provides the combined output as the final output 112. As described below, even when only a single index 104 is sampled, the use of the sampled index and epistemic model improves how well the combined output reflects epistemic uncertainty, for example, because the combined output more accurately models the joint probability distribution than the base output.

[0067] When multiple indexes are sampled, the system 100 generates a respective combined output for each sampled index, as described above.

[0068] The system 100 can then use each combined output to generate a final output 112 for the machine learning task in any of a variety of ways.

[0069] For example, the system 100 may generate an uncertainty estimate from each combined output and provide the uncertainty estimate as part of the final output 112.

[0070] The uncertainty estimate may be based, for example, on a measure of the spread of the distribution of the outputs, such as the variance or standard deviation.

[0071] Alternatively, the uncertainty estimates may instead be based on individual uncertainty estimates for the logits for each class, or for each regression value, that are generated based on the variance of the corresponding logits or regression values ​​across the combined output, i.e., the system may calculate a separate individual uncertainty estimate for each logit or regression value included in the combined output.

[0072] As another example, the system 100 may calculate a measure of central tendency, such as the minimum, maximum, or average, of the combined outputs and then provide the resulting measure as part of the final output 112.

[0073] That is, the system 100 may provide the resulting measure as the final combined output, and in some cases may also provide an uncertainty measure (an "uncertainty estimate") along with the final combined output as the final output 112, or may use the uncertainty measure to modify the final combined output and then provide the modified combined output (with or without the uncertainty measure) as the final output 112.

[0074] The epistemological machine learning model 120, and possibly previous machine learning models, may be utilized to enhance the operation of the system 100 in any of a variety of ways by extending the machine learning model.

[0075] For example, in many applications, e.g., safety-critical applications such as controlling a robot or autonomous vehicle or performing classification of medical images or images captured by a sensor of a robot, autonomous vehicle, or other agent, the system 100 may need to employ a risk management scheme that penalizes outputs that are uncertain. In these applications, the system 100 may generate a respective corrector for each logit or regression value from the combined output and uncertainty estimates. The corrector reduces the logit or regression value by assigning a penalty to logit or regression values ​​that vary more across the sampled index, and then uses the corrected output as the final output for the task. For example, the corrector for a given logit or regression value may be the output of a function applied to the individual uncertainty estimates for the logit or regression value, e.g., an increasing function that scales the logit or regression value or normalizes the logit or regression value. As another example, the corrector may be a fixed value that is assigned only to the subset of logits or regression values ​​with the highest uncertainty estimates, or only to the subset of logits or regression values ​​with uncertainty estimates above a threshold.

[0076] As another example, it is often important to explore the space of possible actions, environments, or both, e.g., during training of a model for controlling an agent or for making content recommendations.

[0077] In some of these implementations, the system can generate a respective modifier for each logit or regression value from the combined output that gives a bonus to values ​​that vary more across the sampled indexes, i.e., increases their value, and then uses the modified output as the final output for the task. For example, the bonus for a given logit or regression value can be the output of a function applied to the individual uncertainty estimates for the logit or regression value, e.g., an increasing function that scales or normalizes the logit or regression value. As another example, the bonus can be a fixed value that is assigned only to the subset of logits or regression values ​​with the highest uncertainty estimates, or only to the subset of logits or regression values ​​with uncertainty estimates above a threshold.

[0078] In other of these implementations, when the output includes respective scores, e.g., Q-values ​​or logits, for actions to be performed by the agent, the system can select actions using a measure of uncertainty for the action rather than using a measure of central tendency for the combined output, so that, for example, actions with a higher measure of uncertainty are more likely to be selected than actions with a lower measure of uncertainty.

[0079] In other words, in any of the above implementations where exploration is important, the system uses the final output to select actions that will cause the agent to explore the environment.

[0080] As another example, the system can determine how to generate additional training data for the base model, the epistemological model, or both, based on the uncertainty estimates calculated from the combined outputs. For example, when the combined output for a given new model input has high uncertainty, the system can query to generate more training examples that are similar to the new model input. As a particular example, when controlling an agent, the system can control the agent to reach a state of the environment characterized by observations that are similar to those that gave rise to the high uncertainty estimates.

[0081] As another example, the system may be performing a process of generating a final output as part of training the base model, the epistemological model, or both, e.g., during online learning to fine-tune a pre-trained model. In this example, for example, to enable the system to adapt faster to new data in which the model is uncertain, and thus perform online learning in fewer training steps and while consuming fewer computational resources, e.g., fewer processor cycles and less memory, the system may determine a learning rate to be used when training on new model inputs based on the uncertainty measure for the combined output. That is, the system may assign a higher learning rate to model inputs having an uncertainty measure that indicates higher uncertainty.

[0082] As another example, in safety-critical applications, the system may generate an alert to be provided to a user of the system based on an uncertainty measure calculated from the combined outputs, i.e., the system may alert the user when the uncertainty measure exceeds a threshold, indicating that the outputs are unlikely to be trustworthy.

[0083] FIG. 2 shows an example of a conventional neural network 210, e.g., exemplary model inputs with a base model 110, and an epistemological model, and possibly an extension with a previous model, to generate an epistemological neural network (ENN) 220.

[0084] In the example of Figure 2, the model input is an image and the task is image classification, specifically, it requires classifying the input image as either an image of a rabbit (R) or an image of a duck (D).

[0085] In example 200, only the conventional neural network 210 is used, which produces outputs including a score of 0.5 for R and a score of 0.5 for D.

[0086] In this example, it is not clear from the output whether the image is an ambiguous image, i.e., whether the image is of a rabbit or a duck, or whether there is insufficient data for the conventional neural network 210 to accurately classify the image, i.e., the image is ambiguous, or the image is not ambiguous but the conventional neural network 210 has not been trained on enough training data or the right type of training data to accurately classify the image.

[0087] In particular, as shown in Figure 2, given a single image, the network 210 outputs marginal predictions that assign probabilities to two classes. When the probabilities are 0.5 each, as in example 200, it remains unclear whether this is because the image is ambiguous and users' opinions about the correct classification are evenly split, or whether the neural network would decide for one class if it were trained on more data.

[0088] To illustrate this, the two tables to the right of the neural network output represent possible joint distributions over the labels of two equal images: one cell represents the joint probability of assigning R to both images, another cell represents the joint probability of assigning R to the first image and D to the second image, another cell represents the joint probability of assigning D to the first image and R to the second image, and another cell represents the joint probability of assigning D to both images.

[0089] Both tables correspond to uniform marginal distributions. The first table shows unavoidable ambiguities that do not become resolved through training on additional data, e.g., because conditioning on the first image label does not change the second distribution. The second table shows that additional training should resolve uncertainties, e.g., because, conditional on the first label, the distribution of the second label assigns all probabilities to the same outcomes as the first.

[0090] FIG. 2 also illustrates the effect of augmenting a conventional neural network 210 with a cognitive machine learning model that receives an index z sampled from a reference distribution to generate an ENN 220.

[0091] Example 230 shows that when an image is ambiguous, sampling different z does not change the probability for the image, i.e., the probability remains 0.5 for R and 0.5 for D, even with multiple sample z, as shown in Figure 2. This indicates that the image is ambiguous and that further training will not improve the performance of ENN 220 in correctly classifying the image, i.e., the joint probability table below assigns equal probabilities of 0.25 to each cell.

[0092] Example 250 shows that when the images are unambiguous but the ENN 220 is trained on insufficient data, the probability for R and the probability for D depend on the sampled z. In the example of FIG. 2, the ENN 220 assigns a probability of 1 to R when z is less than or equal to 0, and assigns a probability of 1 to D when z is greater than 0. This dependency on z indicates that there is epistemic uncertainty in the data on which the ENN 220 was trained, rather than an inherent unresolvable ambiguity in the input. In other words, the joint probability table below assigns a joint probability of 1 to both images being R when z is less than or equal to 0, and a joint probability of 1 to both images being D when z is greater than 0.

[0093] Therefore, by using the ENN 220 instead of simply the conventional neural network 210, the system can generate a final output that accurately characterizes the uncertainty in the output generated by the system, and the reliability of that output.

[0094] More generally, for inputs x1, ..., x T Given, the joint prediction is,probability

[0095]

number

[0096] For each class combination y1, ..., y T Conventional neural networks are not designed to provide joint predictions, but joint predictions can be generated by multiplying the marginal predictions.

[0097]

number

[0098] however,

[0099]

number

[0100] x t In the combined output for y t This is the score for.

[0101] However, this representation models each outcome as independent and therefore cannot distinguish ambiguity from a lack of data. ENNs address this by allowing more expressive simultaneous predictions through integrating over an epistemic index z.

[0102]

number

[0103] where the integral is over z,

[0104]

number

[0105] x t In the combined output for y t This is the score for.

[0106] This integration introduces dependencies such that the joint predictions are not necessarily simply the product of marginals.

[0107] Examples 230 and 250 provide simple examples of how two different ENNs can use epistemic indices to distinguish between the types of uncertainty described above.

[0108] In example 230, the ENN produces marginal predictions that do not vary with z, and therefore the resulting joint predictions are simply the independent product of marginals. This corresponds to an "accidental" or "irreducible" form of uncertainty that cannot be resolved with the data.

[0109] On the other hand, example 250 shows an ENN that makes different predictions depending on the sign of the epistemic index. This corresponds to the "epistemic" or "reducible" uncertainties that can be resolved in the data. In this case, integrating a 2x2 matrix over z produces a diagonal matrix with 1 / 2 on each diagonal element.

[0110] Prior to using the base model 110 and the epistemological model 120, the system 100 or another training system trains the base model 110 and the epistemological model 120 on training data.

[0111] In some implementations, the base model 110 is pre-trained, for example, in an appropriate objective function for the machine learning task. After the base model 110 is trained, the training system trains the epistemological model 120 while keeping the base model 110 fixed. Keeping the base model 110 fixed refers to keeping the values ​​of the parameters of the base model 110 fixed, i.e., fixed to the same values, throughout the training of the epistemological model 120. For example, the training system can train the epistemological model 120 in the same objective function for the machine learning task that was used to train the base model 110.

[0112] In some other implementations, the training system jointly trains the base model 110 and the epistemological model 120 on an objective function for a machine learning task. That is, during training, updates to variable parameters of the base model may be substantially simultaneous or interleaved with corresponding updates to the epistemological model. For example, during simultaneous training, gradients of the objective function for the same training input may be used to update the base model and the epistemological model.

[0113] Examples of objective functions may include cross-entropy loss, squared error loss, negative log-likelihood loss, etc. In some cases, the task loss function may also include one or more additional terms that do not depend on the label for a given input, e.g., auxiliary loss terms, regularization terms, etc.

[0114] As a particular example, the objective function may be a marginal objective function that depends only on a single input. For example, learning may be performed based on minimizing the objective function, and training of the base model may be performed on a set of input-output training examples ({x t ,y t}), the objective function is such that the input to the epistemological machine learning model 120 is the sum of (x t , z), and z is a sample of one or more indices, then the output of the epistemological machine learning model 120 is y t It may be such that it increases the likelihood that

[0115] Even though the objective function is a marginal objective function, the final output generated by the system 100 can accurately model the joint probability after training due to the inclusion of the epistemic machine learning model 120. An example of the performance of the system 100 in modeling the joint probability after training on a marginal objective function is shown below with reference to FIG.

[0116] If the training system trains the base model 110 and the epistemological model 120 together, and the epistemological model 120 receives as input the internal representation generated by the base model 110, then to improve the stability of the training, the training system may apply a stop gradient function in the internal representation. "Stop gradient" indicates that the arguments to the function are treated as fixed when computing the gradient.

[0117] Before training, the variation of the combined output as a function of z reflects the prior uncertainty in the prediction. Since the base model 110 does not depend on z, this variation must be derived from the epistemological model 120. As explained above, the training system can induce this initial variation by incorporating the base model.

[0118] That is, the prior model represents prior uncertainty and has no trainable parameters. The learnable epistemological model may be initialized to output values ​​close to 0, for example using Glorot initialization or a different initialization scheme, but is then trained such that the combined output produces statistically plausible predictions for all possible values ​​of z. The variation of the combined input x as a function of z indicates predictive epistemological uncertainty, as in example 250.

[0119] FIG. 3 illustrates an example of the operation of the system 100 when the system 100 includes a base model 110, a (learnable) epistemological model 120, and a prior epistemological model 310.

[0120] In particular, in the example of FIG. 3, the system 100 receives a model input 102 and processes the model input 102 using a base model 110 to generate a base output for the model input 102 .

[0121] As part of processing the model input 102, the base model 110 generates features for the model input 102. The features for the model input 102 are an internal representation of the model input 102 generated by the base model 110.

[0122] For example, when the base model 110 is a neural network, the features may be the output of one or more hidden layers of the neural network. As a particular example, the features may be the output of the last hidden layer of the neural network, e.g., the last convolutional layer in the neural network, the last transformer layer in the neural network, etc.

[0123] The learnable epistemological model 120 processes the features generated by the base model 110 and the indexes z sampled from the reference distribution Pz to generate an epistemological output.

[0124] As explained above, the epistemological model 120 is generally less computationally expensive than the base model 110. That is, the epistemological model 120 is more computationally efficient than the base model 110, for example, because the epistemological model 120 has fewer weights than the base model 110.

[0125] For example, the epistemological model 120 may be a multi-layer perceptron (MLP) that processes a concatenation of features and index z. As another example, the epistemological model 120 may be a convolutional neural network that processes a spatial grid of features concatenated with another spatial grid generated by broadcasting index z.

[0126] The prior epistemological model 310 processes the features generated by the base model 110 and the index z to generate a prior output.

[0127] As explained above, the prior model 310 is not learnable and is incorporated into the system 100 from the beginning of the training of the epistemological model 120 to induce variability with respect to the index z.

[0128] That is, the previous machine learning model 110 has weights that are fixed to randomly initialized values. In other words, the system 100, or other training system, initializes the weights of the previous model 310 at the beginning of the training of the epistemological model 120, and then keeps the weights fixed throughout and after the training of the epistemological model 120.

[0129] The previous model 310 may have any of a variety of architectures.

[0130] As an example, the prior model 310 may have the same architecture as the epistemological model 120, but with weights that are held fixed rather than adjusted during training. In some of these examples, the weights of the prior model and the epistemological model are initialized to different values ​​at the beginning of training.

[0131] As another example, the previous model 310 may be composed of multiple individual models.

[0132] For example, the prior model 310 may include an ensemble of a first prior machine learning model that receives epistemological input, i.e., the same input as the epistemological model 120, and a second prior machine learning model that receives new model input and possibly one or more sampled indices.

[0133] In this example, the first prior model may have the same architecture as the epistemological model 120.

[0134] The ensemble of second prior models may have any suitable architecture, e.g., convolutional neural networks, MLPs, etc. For example, when the model input is high-dimensional, e.g., large images or other sensor data, and the internal representation received by the epistemological model is a lower-dimensional representation of the input, utilizing an ensemble of second prior models can improve the performance of the system. That is, utilization of an ensemble of small models that are not trained but receive the complete input can allow a more computationally efficient epistemological model to be effectively trained.

[0135] As another example, the prior model 310 may be an ensemble of multiple models that each receive epistemic inputs. In this example, the prior models 310 in the ensemble may all have the same architecture but may have weights initialized differently.

[0136] Then, for each z, the system 100 combines the base output, the epistemological output, and the previous output to generate a combined output. For example, the system can sum the base output, the epistemological output, and the previous output to generate the combined output. When there are multiple samples of z, the system 100 can include an uncertainty measure in the final output in addition to the final combined output generated from all of the samples. As yet another example, the system 100 can use the uncertainty measure to modify the final combined output to generate the final output.

[0137] 4 is a flow diagram of an exemplary process 400 for processing model inputs to generate final model outputs. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a machine learning system suitably programmed in accordance with this specification, such as machine learning system 100 shown in FIG. 1, can perform process 400.

[0138] The system receives new model input (step 402).

[0139] The system processes the new model inputs using the base machine learning model to generate base outputs for the machine learning task for the new model inputs (step 404).

[0140] The system samples a set of one or more indexes from a reference distribution over the indexes (step 406).

[0141] For each of one or more sampled indexes in the set, the system uses an epistemological machine learning model to process epistemological inputs including the sampled index and (i) new model inputs, (ii) data derived from the new model inputs, or (iii) both, to generate epistemological outputs for the machine learning task for the new model inputs (step 408).

[0142] The system generates a final output for the machine learning task from at least the base output for the machine learning task and the epistemological output for the set of one or more sampled indexes (step 410).

[0143] In particular, as described above, the system may also generate a previous output using a previous epistemological machine learning model that has not been trained to process previous inputs, including sampled indices and (i) new model inputs, (ii) data derived from the new model inputs, or (iii) both, and generates a final output using the base output, the epistemological output, and the previous output.

[0144] FIG. 5 shows example results 500 achieved by the machine learning system 100 in comparison to two conventional systems.

[0145] In particular, FIG. 5 shows results achieved by the machine learning system 100 (“epinet”) in comparison to a single base neural network (“ResNet”) and an ensemble of base neural networks (“ensemble”) in an image classification task.

[0146] Figure 5 shows the performance of systems with different total number of parameters in terms of three different performance measures: classification accuracy, marginal log loss, and joint log loss. Marginal log loss measures the logarithm of the values ​​of the marginal loss function on which the model was trained, e.g., the cross-entropy. Joint log loss measures the logarithm of the values ​​of the joint loss function, which measures the joint probability assigned to a sequence of inputs (and is not the loss function on which the model was trained).

[0147] As can be seen in Figure 5, given the same number of parameters, EpiNet achieves similar classification accuracy and marginal log-loss as the other two systems, but achieves significantly lower (better) joint log-loss than the other two systems. In particular, the system achieves significantly better joint log-loss than ensembles with significantly more parameters (and thus significantly higher computational cost).

[0148] Because many real-world tasks require accurate simultaneous predictions, for example to enable real-world decision-making or exploration, the results in FIG. 5 indicate that system 100 can be effectively used for real-world decision-making at a fraction of the computational cost of larger ensembles of models.

[0149] In particular, as explained above, marginal prediction does not distinguish between ambiguity in the underlying inputs and lack of data. However, in order to generate accurate joint predictions, the predictions generated by the models need to effectively take into account the input ambiguity. Thus, models that make accurate joint predictions can be effectively incorporated into real-world tasks, e.g., safety-critical tasks such as robotics or autonomous driving, while models that can make accurate marginal predictions but cannot make accurate joint predictions are not. As shown from the results in FIG. 5, the system 100 can be effectively used for real-world decision making at a fraction of the computational cost of larger ensembles of models, due to the improvement in joint prediction log loss (as well as comparable performance in classification accuracy and marginal log loss) compared to conventional systems.

[0150] This specification uses the term "configured" in connection with systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action during operation. A computer program or programs configured to perform a particular operation or action means that the program or programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0151] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, or in a combination of one or more of them, including the structures disclosed herein and their structural equivalents. The subject matter described herein may be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or sequential access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0152] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In some cases, an apparatus may include, in addition to hardware, code that creates an execution environment for a computer program, e.g., code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0153] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any type of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may correspond to a file in a file system, but need not. A program may be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., a file that stores one or more modules, subprograms, or portions of code. A computer program may be deployed to run on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0154] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular way, or at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0155] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine, and in other cases, multiple engines may be installed and running on the same computer or computers.

[0156] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., FPGAs or ASICs, or a combination of special purpose logic circuitry and one or more programmed computers.

[0157] A computer suitable for executing a computer program may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. In general, the central processing unit will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated in special-purpose logic circuitry. In general, a computer will also include one or more mass storage devices, for example, magnetic disks, magneto-optical disks, or optical disks, for storing data, or be operatively coupled to receive data therefrom, or transfer data thereto, or both. However, a computer need not have such devices. Moreover, a computer may be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, for example, a universal serial bus (USB) flash drive, to name just a few examples.

[0158] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks.

[0159] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may be used to provide for user interaction as well, e.g., feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, speech input, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a response message from the user in return.

[0160] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling common and computationally intensive parts of, e.g., machine learning training or production, e.g., inference, workloads.

[0161] The machine learning model may be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the Jax framework.

[0162] An embodiment of the subject matter described herein may be implemented in a computing system that includes back-end components, e.g., as a data server, or includes middleware components, e.g., an application server, or includes front-end components, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user may interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0163] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device, e.g., to display data to and receive user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., results of user interaction, may be received at the server from the device.

[0164] Although the present specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Some features described in the present specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in some combinations, and may even be initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination, or a variation of a subcombination.

[0165] Similarly, although operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequence illustrated, or that all of the illustrated operations be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0166] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes illustrated in the accompanying figures do not necessarily require the particular order illustrated, or sequence, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0167] 100 Machine Learning Systems, Systems 102 Model Input, New Model Input 104 Index 110 Base Machine Learning Model, Base Model, Previous Machine Learning Model 112 Final Output 114 Base Output 120 Epistemological Machine Learning Models, Epistemological Models, Learnable Epistemological Models 124 Epistemic output 200, 230, 250, 500 examples 210 Conventional Neural Networks, Networks 220 Epistemological Neural Network (ENN), ENN 310 Pre-epistemological models, Pre-models

Claims

1. A method implemented by one or more computers, comprising: receiving new model input; processing the new model inputs using a base machine learning model to generate base outputs for a machine learning task for the new model inputs; Sampling a set of one or more indexes from a reference distribution over the indexes; For each of the one or more sampled indexes in the set, using an epistemological machine learning model to process an epistemological input comprising the sampled index and (i) the new model input, (ii) data derived from the new model input, or (iii) both, to generate an epistemological output for the machine learning task for the new model input; generating a final output for the machine learning task from at least the base output for the machine learning task and the epistemological output for the set of one or more sampled indexes; A method comprising:

2. The method of claim 1 , wherein the new model input does not include any of the sampled indices.

3. 3. The method of claim 1, wherein the base machine learning model generates an internal representation of the new model input while generating the base output, the data derived from the model input is the internal representation, and the epistemological input comprises the sampled index and (i) the internal representation, or (ii) both the internal representation and the new model input.

4. 4. The method of claim 3, wherein the base machine learning model is a neural network and the internal representation is an output of one or more hidden layers of the neural network.

5. The method of claim 4 , wherein the internal representation is the output of a last hidden layer of the neural network.

6. The method further comprising: For each of the one or more sampled indexes, using a previous machine learning model to process the sampled index and a previous input comprising (i) the new model input, (ii) the data derived from the new model input, or (iii) both, to generate a previous output for the machine learning task for the new model input, the previous machine learning model having weights fixed to randomly initialized values. Further comprising: generating the final output for the machine learning task, generating a final output for the machine learning task from the base output for the machine learning task, the epistemological output for the one or more sampled indexes, and the previous output for the one or more sampled indexes.

6. The method according to any one of claims 1 to 5, comprising:

7. The previous machine learning model, (i) a first prior machine learning model that receives the epistemological input; and (ii) an ensemble of second, previous machine learning models that receives the new model input and the one or more sampled indices; and The method of claim 6, comprising:

8. 8. The method of claim 1, wherein the epistemological machine learning model is more computationally efficient than the base machine learning model.

9. the set includes only one sampled index, and generating the final output for the machine learning task comprises: combining at least the base output for the machine learning task and the epistemological output for the sampled index to generate a combined output for the machine learning task; using the combined output as the final output for the machine learning task; 9. The method of any one of claims 1 to 8, comprising:

10. the set includes a plurality of sampled indexes, and generating the final output for the machine learning task comprises: for each index, combining at least the base output for the machine learning task and the epistemological output for the sampled index to generate a combined output for the machine learning task; generating the final output for the machine learning task from the combined output for the plurality of indexes; 10. The method of any one of claims 1 to 9, comprising:

11. The method of claim 10 , wherein generating the final output comprises calculating a measure of central tendency of the combined output.

12. generating an uncertainty measure for the final output from the combined outputs for the plurality of indexes.

12. The method of claim 10 or claim 11, further comprising:

13. the base output and the epistemological output each comprising a respective logit for each of a plurality of classes, and combining at least the base output for the machine learning task and the epistemological output for the sampled index to generate the final output for the machine learning task; for each class, adding at least the logits for the class from the base output and the logits for the class from the epistemological output to generate a combined logit for the class.

13. The method according to any one of claims 9 to 12, comprising:

14. the base output and the epistemological output each comprising a respective regression value, and combining at least the base output for the machine learning task and the epistemological output for the sampled index to generate the final output for the machine learning task; adding said respective values ​​from at least said base output and said epistemological output to generate a combined regression value.

14. The method according to any one of claims 9 to 13, comprising:

15. 15. The method of claim 1, wherein the epistemological machine learning model is trained to optimize an objective function for the machine learning task while holding the base machine learning model fixed.

16. 15. The method of claim 1, wherein the epistemological machine learning model and the base machine learning model are trained together to optimize an objective function for the machine learning task.

17. 17. The method of claim 1, wherein the new model input is one or more images and the machine learning task is a computer vision task.

18. 20. The method of claim 17, wherein the computer vision task is image classification and the final output comprises a respective score for each of a plurality of object categories.

19. 17. The method of claim 1, wherein the new model inputs are observations characterizing a state of an environment and the final outputs define actions to be performed by an agent interacting with the environment.

20. 20. The method of claim 19, wherein the environment is a real-world environment, the agent is a mechanical agent, and the observations comprise data from one or more sensors configured to sense the environment.

21. using the final output to select an action that will cause the agent to explore the environment.

21. The method of claim 19 or claim 20, further comprising:

22. 1. A system comprising: One or more computers; one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform each of the operations of the method of any one of claims 1 to 21; A system comprising:

23. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform each of the operations of the method of any one of claims 1 to 21.

Images