Detecting neural network hallucinations using in-context learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing an input using a neural network to generate one or more first candidate outputs with corresponding first likelihoods, generating one or more sequences of additional candidate outputs and a respective second likelihood for each additional candidate output in the sequence, for each of the first candidate outputs and for each additional candidate output, by processing the input, the first candidate output, and any additional candidate outputs preceding the additional candidate output in the sequence using the neural network, determining a measure of uncertainty using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs among the sequences of additional candidate outputs, and determining, based on the measure of uncertainty, whether to provide any of the candidate outputs as a response to the input.

Description

[0001] Attorney Docket No.45288-0469WO1 DETECTING NEURAL NETWORK HALLUCINATIONS USING IN-CONTEXT LEARNING BACKGROUND [1] This specification relates to processing data using machine learning models. [2] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model. [3] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non- linear transformation to a received input to generate an output. SUMMARY [4] This specification describes a system implemented as computer programs on one or more computers in one or more locations that can, in response to a given input, determine whether to abstain from responding to the given input or to provide an output generated by a machine learning model, e.g., a neural network, in response to the input. [5] The system can use a neural network that is configured to generate multiple candidate outputs for a given input to determine a measure of uncertainty for each of the generated candidate outputs as a proxy for confidence in the response. In this specification, a candidate output refers to a potential answer or response for the input. The system can then use the measure of uncertainty to determine whether to provide any of the candidate outputs as a response to the input or to abstain from responding. In this specification, abstaining refers to providing an abstention response that is not generated using any of the one or more candidate outputs, or refraining from providing any response to the input. [6] More specifically, the system can determine a measure of uncertainty by including additional candidate outputs as example answers for the input over a number of processing iterations. The system can generate the additional candidate outputs for inclusion as example answers in future processing iterations by generating a sequence of candidate outputs over the number of processing iterations using previously generated candidate outputs. For example, the Attorney Docket No.45288-0469WO1 system can generate a sequence of candidate outputs by (i) generating a first candidate output by processing an input using the neural network, and (ii) including the first candidate output in the next input to the neural network as an example answer to the input. Next, the system can (iii) process the next input using the neural network to generate a second candidate output, and can (iv) include the first and second candidate outputs in the next input as example answers to the input. Then, the system can (v) process the next input using the neural network to generate a third candidate output, which can be included as an additional example in the input in the next processing iteration, and so on. [7] In particular, the system can condition the output of the neural network on previously generated candidate outputs by, in each processing iteration, processing the input and any previously generated candidate outputs as potential answers to generate the additional candidate outputs. In this context, the additional candidate outputs provide additional information to guide the generation of the next candidate outputs using the neural network. In the case that the neural network is a language processing neural network, the candidate outputs included as examples in the next input are referred to as context. [8] At each processing iteration, the system can determine the corresponding likelihoods of the candidate outputs in the sequence of candidate outputs. After generating a sequence of candidate outputs, the system can determine the measure of uncertainty by evaluating the corresponding pseudo-likelihood values of the sequences at each of the processing iterations. [9] In this specification, a corresponding likelihood refers to the likelihood of a candidate output at a particular processing iteration, e.g., determined based on a relative probability for the candidate output compared to other potential candidate outputs at that processing iteration. Relatedly, the pseudo-likelihood values of the sequences of candidate outputs refers to an approximation of the actual joint likelihood based on the product of the corresponding likelihoods for each of the candidate outputs in each of the sequences at each of the processing iterations. In this context, the term “pseudo” is used to distinguish between the approximation of the joint likelihood distribution and the actual joint likelihood distribution, given the standard conditioning applied using the chain-rule, which yields the product of corresponding likelihoods, is replaced with an input prompt function of the conditioning variables.

[0010] That is, the system can determine how independent the distributions of the candidate outputs generated at each of the positions in the sequence are, e.g., by quantifying the extent to Attorney Docket No.45288-0469WO1 which the corresponding likelihood values for each of the candidate outputs change given the inclusion of additional candidate outputs as example answers for the input over each of the processing iterations using the pseudo-likelihood of the sequences at that iteration.

[0011] According to a first aspect there is provided a method for receiving an input, wherein the input comprises a query, processing the input using a neural network to generate one or more first candidate outputs and a corresponding first likelihood for each of the first candidate outputs, wherein each candidate output is a possible response to the input, for each of the first candidate outputs, generating one or more sequences of one or more additional candidate outputs and a respective second likelihood for each additional candidate output in the sequence, the generating comprising, for each additional candidate output, processing the input, the first candidate output, and any additional candidate outputs preceding the additional candidate output in the sequence using the neural network to generate the additional candidate output and the corresponding second likelihood for the additional candidate output, determining a measure of uncertainty using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs among the sequences of one or more additional candidate outputs for each of the first candidate outputs, and determining, based on the measure of uncertainty, whether to provide any of the candidate outputs as the response to the input.

[0012] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0013] Generally, machine learning models are not able to generate outputs that are always accurate. Depending on the task, detecting errors in prediction can sometimes be difficult, e.g., for an end-user of the machine learning model. For example, one type of error that is difficult to detect is a hallucination produced by a generative neural network, e.g., a language processing neural network. In this specification, a hallucination is when a generative neural network outputs an incorrect answer as a response to an input prompt, which can be hard for a user to reliably detect. Detecting prediction error is particularly difficult in cases when there are multiple possible correct responses, as relying on thresholding a measure of general uncertainty can improperly discount instances when the neural network correctly identifies some or all of the possible correct responses as having relatively high likelihoods.

[0014] The system of this specification can identify uncertain outputs of a neural network by processing an input and one or more additional candidate outputs using the neural network to Attorney Docket No.45288-0469WO1 determine a measure of uncertainty in the response. Since the system determines the measure of uncertainty as a measure of epistemic uncertainty based on the change in the corresponding likelihoods for the candidate outputs over the number of processing iterations used to generate multiple sequences of candidate outputs, the system can more accurately determine whether to abstain even in the case where multiple possible responses are correct, relative to other abstention techniques that use a general measure of uncertainty which does not account for the possibility of multiple correct answers.

[0015] The system can reduce the use of computational resources generally required to implement an abstention policy since the system requires no architecture change or finetuning of the neural network. More specifically, the system can determine the measure of uncertainty using in-context learning. In this specification, in-context learning involves generalizing from the growing set of examples of candidate outputs iteratively included in the input at each processing iteration.

[0016] In particular, basing the measure of uncertainty on the inference of the neural network obviates the need to implement any model architecture change, e.g., accommodating an additional neural network or neural network head in memory, to generate an uncertainty score for each candidate output. Moreover, the system precludes the need to finetune the neural network or to train a separate classifier to detect factuality in candidate outputs, thereby reducing the computational resources relative to updating the weights of the neural network or training another neural network from scratch to detect whether an output is factually correct. In the case that the neural network is a language processing neural network or visual language model, e.g., networks that can include millions or billions of model parameters, the system can dramatically reduce the use of computational resources by implementing the abstention policy without modifying the architecture or finetuning a pretrained neural network.

[0017] Furthermore, the techniques of this specification can reliably identify uncertain outputs in both single and multi-answer responses using the same abstention policy based on a measure of epistemic uncertainty. In this specification, epistemic uncertainty is uncertainty arising from a lack of knowledge about the ground truth output, e.g., the correct response to the input. A high measure of epistemic uncertainty can indicate that the neural network has low confidence in the output. In particular, epistemic uncertainty can arise based on a limited set of training data or the limitations of a given model architecture, e.g., the number of parameter values in the neural network. Defining an abstention policy using the techniques of this specification is more effective for multi-response Attorney Docket No.45288-0469WO1 answers than other abstention policy methods since the system can determine whether to provide an abstention response based on a measure of epistemic uncertainty determined using the corresponding likelihoods for the candidate outputs in the sequences of candidate outputs, rather than a measure of general uncertainty.

[0018] More specifically, in cases when multiple possible responses are correct for a given input, a general measure of uncertainty can be large due to high aleatoric uncertainty, which captures the inherent uncertainty in the output data distribution, instead of the epistemic uncertainty from a lack of knowledge about the ground truth. When multiple responses are correct, simply estimating the general uncertainty in the output and comparing the general uncertainty to a threshold can be insufficient, as a perfect, e.g., ground-truth, predictor can have large aleatoric uncertainty and no epistemic uncertainty, while a completely useless predictor can have large epistemic uncertainty, even if the total amount of uncertainty of the two predictors are the same. Rather than determining a measure of general uncertainty that includes irreducible randomness inherent to the output distribution (aleatoric uncertainty), the system can determine when the uncertainty is large from a lack of knowledge about the ground truth (epistemic uncertainty).

[0019] Moreover, many existing approaches for uncertainty quantification, e.g., approaches that evaluate the log-likelihood of candidate output responses, estimate the entropy, perform ensembling, or conformal prediction, aim to detect only one dominant candidate output as a proxy for confidence in the response, and therefore are not effective at determining when to abstain given a multi-response answer. Instead of selecting a dominant response based on a threshold criterion, the system can identify that the likelihood of other correct answers remains relatively large and stable despite the inclusion of additional candidate outputs in the input as examples.

[0020] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG. 1 is a system diagram of an example abstention policy inference system that can determine a measure of uncertainty using sequences of candidate outputs.

[0022] FIG. 2 depicts an example generation of a sequence of candidate outputs. Attorney Docket No.45288-0469WO1

[0023] FIG. 3 provides an overview of changing likelihood behavior as a result of including multiple candidate outputs in the input at each processing iteration.

[0024] FIG.4 depicts example precision-recall results of the example abstention inference system of FIG. 1 and a semantic-entropy-based baseline system.

[0025] FIG. 5 is a flow diagram of an example process for determining whether to provide candidate outputs as a response to an input using a measure of uncertainty determined using sequences of candidate outputs.

[0026] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0027] FIG. 1 shows an example abstention policy inference system 100. The abstention policy inference system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0028] The system 100 can receive an input 105, e.g., from a user of the system 100 or an upstream system. For example, the input 105 can be formatted in the style of a prompt, e.g., a directive instruction, e.g., a question, statement, code snippet, or example. The input 105 can include one or more of text, image, audio, or video inputs. In some cases, the input 105 can be multimodal. The image and / or video may be an image and / or video of a real-world environment. Such an image or video may have been captured by a camera. The audio may be audio of a real-world environment. Such audio may have been captured by a microphone.

[0029] The system 100 can implement an abstention policy for determining whether to provide a candidate output 184 generated by a neural network 120 that is configured to generate multiple different outputs for the given input 105 as a response 180 to the given input 105. In particular, the system 100 can determine whether to abstain from responding to the input 105 by determining a measure of uncertainty 170 in the response and comparing the measure of uncertainty 170 to a threshold. The system 100 can use the measure of uncertainty 170 to prevent prediction error, e.g., instead of providing an incorrect response when the neural network 120 has low confidence in a candidate output, the system can use the measure of uncertainty 170 to provide an abstention response. Attorney Docket No.45288-0469WO1

[0030] The system 100 can process the input 105 using a neural network 120 that is configured to generate multiple candidate outputs for a given input. The system 100 can then determine the measure of uncertainty 170 by generating multiple sequences of candidate outputs 130 using the candidate outputs generated by the neural network 120 in each of a number of processing iterations. In particular, the system can determine a corresponding likelihood for each candidate output generated at a particular processing iteration and, at the final processing iteration, can determine the measure of uncertainty 170 based on the pseudo-likelihoods of the generated sequences characterizing how independent the candidate outputs in the sequences are based on the corresponding likelihood for the candidate output generated at each of the processing iterations, as is described in more detail below.

[0031] In this specification, a corresponding likelihood refers to the likelihood of a candidate output at a particular processing iteration, e.g., determined based on a relative probability for the candidate output compared to other potential candidate outputs at that processing iteration. Relatedly, the pseudo-likelihood values of the sequences of candidate outputs refers to an approximation of the actual joint likelihood based on the product of the corresponding likelihoods for each of the candidate outputs in each of the sequences at each of the processing iterations.

[0032] In particular, the system 100 can use in-context learning to generate multiple sequences of candidate outputs 130. In-context learning is a technique for causing a neural network to perform a specific task at the time of inference, e.g., as opposed to finetuning the neural network 120 for the specific task using a finetuning dataset. In this case, the system 100 can perform in-context learning by iteratively including one or more candidate outputs generated in a previous processing iteration as example responses in the next input to the neural network 120.

[0033] In particular, for each sequence of candidate outputs 130, the system 100 can successively process the input 105 and additional candidate outputs generated in previous processing iterations over a number of processing iterations to generate a sequence of candidate outputs 130. For example, the system 100 can process a first input using the neural network 120 in a first processing iteration: Prompt: Consider the following question: [Q1] Response: [A1] In this case, A1 is the first candidate output.

[0034] Then, in a next processing iteration, the system 100 can include A1 in the next input as an example answer in the context: Attorney Docket No.45288-0469WO1 Prompt: Consider the following question: [Q1] Response: [A2] One answer to the question [Q1] is [A1]. Answer the question. In this case, A2 is one of the multiple candidate outputs generated in the second processing iteration. Generally, A2 is not the same candidate output as A1, but, in the case that the neural network 120 generates a candidate output already represented in the context, the system 100 can repeat the candidate output in the context.

[0035] Likewise, the system 100 can include A2 in the next input as an additional example answer in the context: Prompt: Consider the following question: [Q1] Response: [A3] One answer to the question [Q1] is [A1]. Another answer to the question [Q1] is [A2]. Answer the question.

[0036] Thus, for each processing iteration, the system 100 can include additional candidate outputs generated during each previous processing iteration as examples in the context in order to increase the sequence of candidate outputs: Prompt: Consider the following question: [Q1] One answer to the question [Q1] is [A1]. Another answer to the question [Q1] is [A2]. Another answer to the question [Q1] is [A3]. … Another answer to the question [Q1] is [AN]. Answer the question. In this case, the sequence of candidate outputs includes A1, A2, A3…, AN.

[0037] The neural network 120 can have any appropriate number of neural network layers (e.g., 1 layer, 5 layers, or 10 layers) of any appropriate type (e.g., fully-connected layers, attention layers, convolutional layers, etc.) connected in any appropriate configuration (e.g., as a linear sequence of layers, or as a directed graph of layers). In particular, the neural network 120 can be configured to process an input sequence of varying lengths.

[0038] In some cases, the neural network 120 can be a generative neural network. For example, the neural network 120 can be a language processing neural network. A language processing neural Attorney Docket No.45288-0469WO1 network is an auto-regressive network that is configured to sequentially process the contents of an input and trained to perform next element prediction, e.g., to define a likelihood score distribution over a next set of elements. In particular, the neural network 120 can be referred to as an auto- regressive neural network when the neural network 120 auto-regressively generates an output sequence of tokens. More specifically, the auto-regressively generated output is created by generating each particular token in the output sequence conditioned on a current input sequence that includes any tokens that precede the particular token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token.

[0039] For example, the neural network 120 can be an auto-regressive Transformer-based neural network that includes (i) a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0040] In this example, the neural network 120 can have any of a variety of Transformer-based neural network architectures. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Elsen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d’Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a Attorney Docket No.45288-0469WO1 unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

[0041] Generally, to apply the self-attention operation, each attention block uses one or more attention heads. Each attention head generates a set of queries, a set of keys, and a set of values, and then applies any of a variety of variants of query-key-value (QKV) attention, e.g., a dot product attention function or a scaled dot product attention function, using the queries, keys, and values to generate an output. Each query, key, or value can be a vector that includes one or more vector elements. When there are multiple attention heads, the attention block then combines the outputs of the multiple attention heads, e.g., by concatenating the outputs and, optionally, processing the concatenated outputs through a linear layer.

[0042] As an example, the neural network 120 can be a visual language model (VLM) that can be configured to process an image or sequence of images in a video to generate an intermediate representation of the image and perform an image processing task. In this case, the tokens can represent any of text, audio, image, etc. inputs. For example, the visual language model can be a contrastive language-image pre-training (CLIP) model, a vision transformer (ViT), a unified image-to-image translation (UNIT) model, or an attention generative adversarial network (AttnGAN).

[0043] In more detail, the system 100 can determine whether to abstain from generating a response 180 to the input 105 by determining a measure of uncertainty 170 using the corresponding likelihoods of each candidate output generated using the neural network 120 at each of the number of processing iterations involved in generating the sequences of candidate outputs 130.

[0044] In this specification, determining the corresponding likelihoods of candidate outputs generated using the neural network 120 refers to obtaining the likelihoods of each of the candidate outputs from the neural network 120 in a particular processing iteration. That is, at each processing iteration, the system 100 can obtain the corresponding likelihood 136 for the candidate outputs from the neural network 120. Attorney Docket No.45288-0469WO1

[0045] For example, the system 100 can obtain the corresponding likelihood 136 for the candidate output generated at each processing iteration based on the likelihood or pseudo-likelihood values determined by one of the final layers of the neural network 120. For example, the system 100 can sample the candidate output(s) after a final softmax layer of the neural network 120 and can obtain the corresponding likelihood values as the output of the softmax layer, e.g., the log probability of the logits assigned to each candidate output.

[0046] In some cases, e.g., the case that the neural network 120 is an autoregressive neural network, determining the corresponding likelihood can involve obtaining the corresponding likelihood of each token in a sequence of output tokens defining a candidate output and aggregating the corresponding likelihoods of each token in the output sequence of tokens as the corresponding likelihood for the candidate output. In this case, the likelihood value for the candidate output at a particular processing iteration are determined as a function, e.g., by determining the log probability of the logits of each element of the output sequence defining the candidate output at the processing iteration and multiplying the log probabilities.

[0047] In particular, the system 100 can process the input 105 using the neural network 120 to generate one or more first candidate output(s) 132 with corresponding likelihoods, e.g., in a first processing iteration.

[0048] In the case that the neural network 120 is a language processing neural network that is configured to define a likelihood score distribution over a next set of elements, e.g., an output sequence of elements that represents probabilities for each element of the output sequence, the language processing neural network can generate the first candidate output(s) 132 by processing the input 105 and generating multiple candidate outputs by sampling different output elements for each element of the output sequence. For example, the neural network 120 can sample the N most likely candidate outputs generated in a particular iteration based on the probability values at each element of the output sequence as the first candidate output(s) 132, e.g., by performing top-k sampling.

[0049] The system 100 can then determine, as a proxy for confidence, to what extent the corresponding likelihoods for each of the first candidate output(s) change based on the inclusion of additional candidate output(s) as example answers in the input. The system can determine which candidate outputs to include as example answers by generating a sequence of candidate outputs. Attorney Docket No.45288-0469WO1 At each processing iteration, the system can determine the corresponding likelihoods 136 for each new candidate output in the sequence, as is described in more detail below.

[0050] The system 100 can generate multiple sequences of candidate outputs 130. In some cases, the system 100 can generate a single sequence of candidate outputs for multiple first candidate outputs. In other cases, the system 100 can generate multiple sequences of candidate outputs for a single first candidate output.

[0051] For each sequence, the system 100 can consecutively extend the sequence of candidate outputs 130 in each processing iteration by including the candidate outputs generated by the neural network 120 in the previous processing iteration as additional example answers for the input 105. That is, the system 100 can generate an additional candidate output for inclusion as an additional example in the next input by including any previously generated candidate outputs in the input to the neural network 120.

[0052] For example, the system 100 can generate a sequence of candidate outputs 130 for each first candidate output 132 generated in the first processing iteration. More specifically, for each first candidate output 132, the system 100 can process a next input that includes the input 105 and the first candidate output 132 to generate one or more additional candidate outputs 134 with respective corresponding likelihoods 136. Then, to add a next candidate output to the sequence, the system 100 can process a next input that includes the input 105, the first candidate output 132, and the one or more additional candidate outputs 134 generated in the second processing iteration to generate one or more additional candidate outputs 134. In each processing iteration, the system 100 can process a next input that includes the input 105, the first candidate output 132, the one or more additional candidate outputs 134 generated in the second processing iteration, the one or more additional candidate outputs 134 generated in the third processing iteration, and so on. This process is depicted in FIG. 2.

[0053] In the case that the neural network 120 is a language processing neural network, the system 100 can iteratively process the input 105, the first candidate output 132, and any additional candidate outputs 134 in the increasing sequence of candidate outputs 130 as context, e.g., additional information that informs the generation of the output of the language processing neural network, to generate a subsequent output at each of a number of processing iterations. In particular, the system 100 can include the one or more candidate examples in the context window for the input 105 to the language processing neural network. Attorney Docket No.45288-0469WO1

[0054] For example, the system 100 can process an input 105 that includes the question “What is the national instrument of Ireland?” using the neural network 120 to generate first candidate outputs 132, e.g., “the harp”, “the Uilleann pipes”, and “the guitar”. In this case, the next input for each of the first candidate outputs 132 can be: “What is the national instrument of Ireland? An example answer for this question is the harp” for the first candidate output “the harp”, “What is the national instrument of Ireland? An example answer for this question is the Uilleann pipes” for the first candidate output “the Uilleann pipes”, and “What is the national instrument of Ireland? An example answer for this question is the guitar” for the first candidate output “the guitar”.

[0055] As another example, the system 100 can process an input 105 that includes the question “Name a city in Greece” using the neural network 120 to generate first candidate outputs 132, e.g., “Athens”, “Thessaloniki”, and “Patras”. In this case, the next input for each of the first candidate outputs 132 can be: “Name a city in Greece. An example answer for this question is Athens” for the first candidate output “Athens”, “Name a city in Greece. An example answer for this question is Thessaloniki” for the first candidate output “Thessaloniki”, and “Name a city in Greece. An example answer for this question is Patras” for the first candidate output “Patras”. In the next iteration, each of the sequences can include an additional candidate output from the neural network 120.

[0056] In the case that the neural network 120 generates a candidate output that is already included as an example in the sequence of candidate outputs 130, the system 100 can repeat the candidate output as an example answer. For example, in the sequence for first candidate output “Athens”, in the case the neural network 120 generates “Athens” in a future processing iteration, the system can include “An example answer for this question is Athens. Another example answer for this question is Athens”, since repeating information in the context of a language processing neural network is not necessarily redundant.

[0057] In some cases, the system 100 can include multiple candidate outputs in the context, e.g., from a particular processing iteration. For example, the system 100 can have sampled four additional candidate outputs A2, A3, A4, and A5 for the first candidate input A1 in the second processing iteration for inclusion in the context for the third processing iteration. In the example described, the next input can be: Consider the following question: [Q1] One answer to the question [Q1] is [A1]. (from first iteration) Attorney Docket No.45288-0469WO1 Another answer to the question [Q1] is [A2]. (from second iteration) Another answer to the question [Q1] is [A3]. (from second iteration) Another answer to the question [Q1] is [A4]. (from second iteration) Another answer to the question [Q1] is [A5]. (from second iteration)

[0058] Alternatively, the system 100 can receive the first candidate output(s) 132 and the corresponding likelihood 136 for the first candidate output(s) 132, e.g., from an external system 190. In this case, the system 100 can generate the sequences of candidate outputs 130 by providing network inputs and receiving network outputs from the external system 190. For example, the systems 100 and 190 can communicate by transmitting data over a network connection, e.g., the Internet, an intranet, a cloud-based network, etc. In particular, the system 100 can provide network inputs to the external system 190 over the network and the external system 190 can process the network inputs using the second neural network 195 to generate additional candidate outputs 134 that are transmitted to the system 100 at each processing iteration in order to generate the sequences of candidate outputs 130, following the process as described above.

[0059] The system 100 can evaluate a measure of uncertainty 170 for the candidate output(s) generated for the input 105 using the corresponding likelihoods 136 for the first candidate output(s) 132 and the additional candidate outputs 134 generated at each processing iteration among the multiple sequences of candidate outputs 130. In particular, the system 100 can determine whether the corresponding likelihoods 136 change in a meaningful way based on the inclusion of the additional candidate outputs 134 as examples in the input.

[0060] More specifically, the system 100 can determine that the neural network 120 has high confidence about the knowledge stored in its parameters if the inclusion of additional candidate outputs 134 in the input for a processing iteration does not meaningfully change the candidate outputs generated in the processing iteration. That is, the system 100 can determine that the neural network 120 has high confidence about the knowledge stored in its parameters for the input 105 if the neural network 120 is insensitive to the inclusion of additional information.

[0061] In contrast, the system 100 can determine that the neural network 120 has low confidence about the knowledge stored in its parameters for the input 105, if the inclusion of additional candidate outputs meaningfully changes the output. In this context, a meaningful change is quantified based on the change of the corresponding likelihood for each candidate output over the processing iterations. In particular, the system can determine a measure of the interaction of the Attorney Docket No.45288-0469WO1 candidate outputs in the sequence at each of the processing iterations. More specifically, the stability of the corresponding likelihoods for each candidate output over a number of processing iterations depends on the inclusion of additional information in the context. For example, if the inclusion of different candidate outputs in only a few processing iterations causes the corresponding likelihood for a candidate output to rapidly decrease, then the neural network 120 has low confidence in the response.

[0062] In more detail, the output distribution generated by the neural network 120 for the candidate output(s) should not meaningfully change based on sampled candidate output(s) that are included in the input as examples used in generating the sequences of candidate outputs 130. For example, the system 100 can determine the measure of uncertainty 170 for a response 180 by approximating a discrepancy between an approximation of the neural network output distribution and an approximation of the ground truth output distribution determined using the corresponding likelihoods 136. In some cases, the system 100 can determine the measure of uncertainty 170 as a divergence between the approximations of the neural network output distribution and the ground truth output distribution determined based on the corresponding likelihoods 136 among the sequences of candidate outputs 130. By determining the divergence using the approximated distributions, the system 100 can provide a lower bound on the measure of uncertainty 170 in the response 180.

[0063] In particular, the system 100 can use the sequences of candidate outputs 130 to construct a pseudo-likelihood output distribution without evaluating the entire set of possible candidate outputs for an input, which is uncountable. More specifically, each of the sequences of candidate outputs 130 can be considered as independent samples from the neural network output distribution Q~, e.g., represented as where Ykn is a candidate in the kth sequence of candidate outputs. In more detail, each sample from Q~is assumed to be conditionally independent, and the pseudo- joint distribution over responses given a query x based on the independent sequences of candidate outputs is given as Attorney Docket No.45288-0469WO1 where F is the prompting mechanism. More explicitly, sampling from Q~is formally stated as:

[0064] Since the sequences of candidate outputs 130 are independent samples from the neural network output distribution, the system 100 can determine the measure of uncertainty 170 by aggregating the corresponding likelihoods obtained for each processing iteration in the sequences of candidate outputs 130 to approximate the neural network output distribution and to approximate the ground truth output distribution.

[0065] In particular, the system 100 can aggregate the corresponding likelihoods of each respective candidate output across the k sequences of candidate outputs 130 to generate aggregated likelihoods 162 for each candidate output by summing the corresponding likelihood of each jth instance of a candidate output Yiin the n processing iterations, e.g.,

[0066] In some cases, the system 100 can determine the unique candidate outputs in the sequences of candidate outputs, e.g., in order to aggregate likelihoods of candidate outputs that are lexically different but semantically equivalent. In particular, the system 100 can cluster the candidate outputs, e.g., using any appropriate clustering technique, to ensure that candidate outputs with similar meaning are considered to be the same candidate output.

[0067] For example, the system 100 can cluster the candidate outputs in the sequences of candidate outputs by computing a similarity score between each pair of candidate outputs Yiand Yjin the sequences of candidate outputs 130 using a similarity function S, e.g., S(Yi, Yj), and generating clusters of candidate outputs based on the similarity scores. The system 100 can then apply a similarity threshold τ to the similarity scores to determine whether each pair of candidate outputs should be associated together in the same cluster, e.g., to be considered as the same candidate output. For example, the system 100 can create clusters according to S(Yi, Yj) ≥ τ .

[0068] For example, the corresponding likelihoods for candidate are clustered together according to the similarity function S can be summed together to represent the corresponding likelihood of the semantically equivalent output, e.g., for each candidate output Yijin each cluster Di : ∑^ Q^Y^^^ . Attorney Docket No.45288-0469WO1

[0069] In this case, by reducing redundancies among the candidate outputs generated by the neural network, the system 100 can ensure the aggregated likelihoods 162 accurately reflect the distribution over unique candidate outputs.

[0070] In some cases, the system 100 can process pairs of candidate outputs using a similarity neural network to determine respective pairwise similarity scores for each of the pairs of candidate outputs in the sequences of candidate outputs 130. The similarity neural network can have any appropriate neural network architecture that can be configured to process a pair of candidate outputs to generate a respective pairwise similarity score for the pair of candidate outputs. In particular, the similarity neural network can have any appropriate number of neural network layers (e.g., 1 layer, 5 layers, or 10 layers) of any appropriate type (e.g., fully-connected layers, attention layers, convolutional layers, etc.) connected in any appropriate configuration (e.g., as a linear sequence of layers, or as a directed graph of layers).

[0071] In some cases, the similarity neural network can be a language processing neural network, e.g., a different language processing neural network than the language processing neural network, in the case that the neural network 120 is implemented as a language processing neural network. In this case, the similarity neural network can measure the semantic similarity between text responses. For example, for X='What is the population of London', and Y='8,980,000' and Y'='8,900,000', a sensible similarity function can assign a large number to the similarity score.

[0072] An example of determining pairwise similarity scores by prompting a language processing neural network with an input that includes a pair of candidate outputs and an instruction to determine the similarity between the pair of candidate outputs, e.g., on a standardized scale from 1 to 5, 1 to 10, etc., to predict the pairwise similarity score and performing clustering is described in more detail in “Mitigating LLM Hallucinations via Conformal Abstention” (Abbasi-Yadkori, Yasin, et. al., 10.48550 / arXiv.2405.01563), and is also described in U.S. App. No. 63 / 548,839 “Conformal Abstention for Language Processing Neural Networks” and WIPO PCT App. No. PCT / US2025 / 013726.

[0073] The system 100 can then normalize the likelihood of candidate output Yi, e.g. unique candidate output Yi, e.g., Q~(j instances of Y1 over iterations 1:n), in each iteration using the aggregated likelihood 162 for Yi to generate an approximation of a joint distribution of the candidate outputs, e.g., a pseudo-joint distribution, as the neural network output distribution: Attorney Docket No.45288-0469WO1 In more detail, in the case that the system 100 any redundancies in the candidate outputs, the system 100 can normalize the candidate outputs based on the unique set of candidate outputs determined by performing the clustering.

[0074] The system 100 can estimate the ground truth distribution as a product distribution of the normalized aggregated likelihoods of candidate outputs over each respective processing iteration for each of the sequences of candidate outputs 130. In particular, the system 100 can generate the product distribution by taking the product of a respective marginal sum of corresponding likelihoods for each candidate output i appearing at each particular position in the sequence.

[0075] More specifically, the system can determine the respective marginal sums computed for each candidate output at each position in the sequence by summing each of the likelihoods in the sequence that correspond with the first candidate output or each of the likelihoods in the sequence that correspond with any additional candidate outputs at the particular position in the sequence, e.g.,

[0076] As an example, the In this case, the system 100 can determine the mutual information estimate using the approximation of the joint and product distributions described above. In particular, the system 100 can determine a measure of divergence between the approximation of the neural network output distribution, e.g., the joint distribution of candidate outputs, and the approximation of the ground truth distribution, e.g., the product distribution of candidate outputs.

[0077] For example, the mutual information estimation can be calculated as a Kullback-Leibler (KL) divergence between the joint and product distributions. More specifically, the resulting KL divergence of the approximated distributions can provide a lower bound on the mutual information of the ground truth output distributions.

[0078] For example, the KL divergence can be computed as Attorney Docket No.45288-0469WO1 where ^ is a stabilization parameter, e.g., which is added to promote numerical stability of the calculation using the sequences of candidate outputs 130.

[0079] As another example, the KL divergence can be computed using an additional stabilization parameter, e.g., as where ^1 and ^2

[0080] The system 100 can use the measure of uncertainty 170 to determine whether to provide any of the candidate outputs 184 as a response 180 to the input or to provide an abstention output 182, e.g., outputting “I don’t know” or another natural language phrase that indicates that the neural network 120 is not confident in any generated response, or not providing any response. In particular, the system 100 can determine whether to provide any candidate outputs 184 as a response to the input based on whether the measure of uncertainty 170 satisfies one or more criteria, e.g., an abstention threshold value 175.

[0081] For example, the system 100 can compare the measure of uncertainty 170 to an abstention threshold value 175 to determine whether the system 100 should provide any of the candidate outputs generated by the neural network 120 as response 180. In the case that the measure of uncertainty 170 does not satisfy the abstention threshold value 175, the system 100 can provide an abstention output 182.

[0082] In the case that the measure of uncertainty 170 satisfies the abstention threshold value 175, the system can provide one of the candidate outputs as response 180. As an example, the system can select a first candidate output from the sequence with the lowest measure of uncertainty, e.g., the first candidate output with the greatest corresponding likelihood stability over the processing iterations, or greedily according to the relative likelihoods of the first candidate outputs.

[0083] The abstention threshold value 175 can be determined in a variety of ways. For example, the system 100 can receive the abstention threshold value 175, e.g., as a user input. As another example, the abstention threshold value 175 can be determined using a hyperparameter search. As yet another example, the system 100 can calibrate an abstention policy parameterized by the abstention threshold value 175 using a calibration set of input-response pairs, e.g., as described in “Mitigating LLM Hallucinations via Conformal Abstention” (Abbasi-Yadkori, Yasin, et. al., Attorney Docket No.45288-0469WO1 10.48550 / arXiv.2405.01563), and is also described in U.S. App. No. 63 / 548,839 “Conformal Abstention for Language Processing Neural Networks” and WIPO PCT App. No. PCT / US2025 / 013726.

[0084] In more detail, a counterpart abstention calibration system, or another calibration system, can calibrate the abstention threshold value 175 by tuning the value 175 using a set of example prompt-response pairs, e.g., calibration dataset Dn = {(X1,Y1),...,(Xn,Yn)}. As an example, an input prompt in the calibration database can be a query, and the corresponding response in the prompt- response pair can be the correct response to the request or an abstention response, e.g., which indicates that it is incorrect for the system to provide a response to the input prompt.

[0085] As an example, the calibration system can calibrate the abstention threshold value 175 in accordance with minimizing the rate of abstention while bounding the expected risk of the policy by a risk tolerance criterion ^, which specifies the upper bound on the allowable risk in the response: where abstention policy a threshold value γ 175 and is the expected loss resulting from the abstention policy.

[0086] Given the calibration dataset, the calibration system can determine the abstention threshold value 175 in an approximately optimal solution to the equation above, e.g., based on the risk tolerance criterion ^. In particular, the calibration system can determine respective abstention policies parameterized by the abstention threshold value 175 by calibrating each policy with different risk tolerances.

[0087] Since the abstention policy inference system 100 can determine whether to provide an abstention response based on a measure of epistemic uncertainty determined using the corresponding likelihoods for the candidate outputs, rather than a measure of general uncertainty, the system 100 can be reliably used for inputs with both single and multi-answer inputs.

[0088] In particular, in the case that the neural network 120 is a language processing neural network, the system 100 can be used to mitigate hallucination, for both single and multi-answer queries. In this context, hallucination is when a language processing neural network outputs an incorrect answer, which can be hard for a user to reliably detect. In this case, instead of providing Attorney Docket No.45288-0469WO1 a candidate output 184 that is an incorrect response the system 100 can provide the abstention output 182.

[0089] FIG. 2 depicts an example of generating a sequence of candidate outputs using a neural network 250. For example, the abstention policy inference system 100 of FIG. 1 can generate a sequence of candidate outputs for a first candidate output using the processing iterations shown in FIG. 2.

[0090] In particular, the system can use in-context learning to generate a sequence of candidate outputs by processing an input including the previously generated candidate outputs in the sequence using the neural network 250 in each processing iteration.

[0091] For example, in the first processing iteration 202, the system can process an input 210 that includes a query 212 using the neural network 250 to generate a candidate output A and corresponding likelihood A 214. In the particular example depicted, candidate output A is the first candidate output and is used as the first element to establish the sequence of candidate outputs depicted in FIG. 2.

[0092] For each subsequent iteration, the system can iteratively process the query 212 and any additional candidate outputs in the sequence of candidate outputs 130 as additional examples to generate a subsequent output. As an example, in the second processing iteration 204, the system can process an input 220 that includes the query 212 and candidate output A 222 using the neural network 250 to generate additional candidate output B and respective corresponding likelihood for candidate output B 224.

[0093] As another example, in the third processing iteration 206, the system can process an input 230 that includes the query 212, candidate output A 222, and candidate output B 232 using the neural network 250 to generate additional candidate output C and respective corresponding likelihood for candidate output C 234.

[0094] More specifically, an input for iteration t+1 can include the query 212, candidate output A, and any additional candidate outputs that were previously generated in the sequence of candidate outputs 130, e.g., candidate output B, candidate output C, etc. That is, in processing iteration N 208, the system can process an input 290 that includes the query 212, candidate output A 222, candidate output B 232, candidate output C 242, and so on, e.g., through candidate output N-1292 using the neural network 250 to generate the output that includes candidate output N+1 and corresponding likelihood for candidate outputs N 294. Attorney Docket No.45288-0469WO1

[0095] While depicted here as only generating a single candidate output in the processing iteration to simplify the illustration of the sequence of candidate outputs, the neural network 250 is configured to generate multiple candidate outputs for a given input and the system can sample multiple candidate outputs at each generation, e.g., as is described with respect to FIG. 1. In particular, the system can add multiple additional candidate outputs as examples to the next input.

[0096] By iteratively processing the first candidate output, e.g., candidate output A, and any additional candidate outputs generated in the sequence of candidate outputs, e.g., candidate output B, C, … N, as examples in the next input and obtaining corresponding likelihood values for each candidate output in the sequence at each iteration from the neural network, the system can determine the extent to which the corresponding likelihood values meaningfully change based on the examples included in each input, e.g., by estimating the mutual information, as described above by generating multiple sequences of candidate outputs.

[0097] In this case, at the end of the single generated sequence, the system can determine the pseudo-likelihood of the sequence 226 by multiplying each of the corresponding likelihoods for each of the candidate outputs generated in the sequence. However, this is a simplification, since multiple sequences are needed to generate an approximation of the actual joint likelihood, e.g., based on the product of the corresponding likelihoods for each of the candidate outputs in each of the sequences at each of the processing iterations.

[0098] FIG. 3 illustrates example results of an abstention policy inference system. In particular, FIG.3 provides an overview of how the relative likelihood for candidate outputs can change based on the inclusion of additional candidate outputs as examples in the input, e.g., to provide an intuition into the process described in FIGS. 1 and 2.

[0099] The graphs in rows 310, 320, and 330 illustrate the relative likelihood on the y-axis for a correct candidate output (e.g., there are multiple correct outputs in row 330) over each processing iterations on the x-axis. For ease of understanding, the relative likelihood on the y-axis for a correct candidate output can designate the relative likelihood that the candidate output is correct for the input, e.g., based on the conditional normalized probability, with one representing the highest relative probability that the candidate output is correct.

[0100] In this case, the system can include the single additional candidate output identified below each graph as an example in each next input. For example, for graph 312, the system can repeatedly include “Paris” as the example answer, e.g., “One answer to this question is Paris. Another answer Attorney Docket No.45288-0469WO1 to this question is Paris…” and so on at future iterations for each next input. Likewise, for graphs 322 and 332, the system can repeatedly include “Uillean pipes” and “Manchester”, respectively as the example answers in each next input.

[0101] The relative likelihood of the correct candidate output in the first iteration and the additional candidate output repeated as an example is also depicted below the x-axis in each graph. In particular, the system can determine the measure of uncertainty for the correct candidate output based on whether the relative likelihood for the candidate output changes in a meaningful way based on the inclusion of the additional candidate output as an example. Here, the graphs depict the behavior of a correct candidate output over the processing iterations to demonstrate how the inclusion of an additional candidate output can impact the neural network’s output certainty.

[0102] For example, the graphs 312, 314, and 316 in row 310 depict examples in which the system processes an input that has a single correct answer, e.g., London in 312, George Washington in 314, and John Steinbeck in 316. The aleatoric uncertainty is low for each input, since there are not multiple correct responses for the respective inputs. In this case, the system includes a single additional incorrect candidate output as an example in the input.

[0103] In graphs 312 and 314, the behavior of the relative likelihood for the correct candidate output demonstrates that the neural network has high certainty in the correct candidate output, e.g., since the relative likelihood of the correct candidate output remains high across many iterations, before decreasing slightly after many iterations of including an incorrect candidate output as an example in the input. In contrast, in graph 316, the behavior of the relative likelihood for the correct candidate output demonstrates the neural network has less certainty in the correct candidate output, e.g., since the relative likelihood of the correct candidate output, John Steinbeck, decreases to zero after iteratively prompting the neural network with the incorrect example of “Ernest Hemingway” as a potential answer.

[0104] As another example, the graphs 322, 324, and 326 in row 320 depict examples in which the system processes an input that has a single correct answer, e.g., the harp in 312, Ralph Fiennes in 314, and a rat in 316. The aleatoric uncertainty is low for each input, since there are not multiple correct responses for the respective inputs. Similarly to row 310, in this case, the system includes a single additional incorrect candidate output as an example in the input.

[0105] In graphs 322, 324, and 326, the behavior of the relative likelihood for the correct candidate output demonstrates that the neural network has low certainty in the correct candidate Attorney Docket No.45288-0469WO1 output, e.g., since the relative likelihood of the correct candidate output decreases to zero after only a few iterations of including an incorrect candidate output as an example in the input.

[0106] As yet another example, the graphs 332, 334, and 336 in row 330 depict examples in which the system processes an input that has multiple correct answers. The aleatoric uncertainty is high for each input, since there are multiple correct responses for the respective inputs. In this case, the graphs depict a relative likelihood for one of the possible correct candidate outputs, e.g., London in 332, banana in 334, and wine in 336, and the system includes a single additional correct candidate output as an example in the input.

[0107] In graphs 332, 334, and 336, the behavior of the relative likelihood for one of the possible correct candidate outputs demonstrates that the neural network retains high certainty over approximately 20 processing iterations, e.g., by maintaining an average relative likelihood around 70% over the first 20 processing iterations. While the relative likelihood eventually decreases to zero by the 100thiteration, the relative likelihood does not change in a meaningful way for a significant number of iterations, thereby demonstrating the applicability of the methods of this specification to evaluating the uncertainty of multi-answer inputs based on the relative likelihood for each candidate output.

[0108] FIG. 4 depicts example precision-recall results of an example abstention inference system and a semantic entropy baseline system.

[0109] In particular, graphs 400 and 450 depict precision-recall (PR) curves for a system implemented as described using the TriviaQA+WordNet and AmbigQA+WordNet datasets. The TriviaQA dataset is a question-answer dataset obtained from Wikipedia and other web sources, e.g., as described in further detail in Joshi, M., et. al. “TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension” (Transactions of the Association for Computational Linguistics (ACL 2017)), and the AmbiQA dataset is a question answer dataset that includes ambiguously phrased questions, e.g., as described in further detail in Min, S., et. al. “AmbigQA: Answering ambiguous open-domain questions.” (Conference on Empirical Methods in Natural Language Processing (EMNLP 2020)). Both can be augmented with additional questions by leveraging relationships represented in the WordNet dataset to obtain a significant number of multi-label queries.

[0110] In this context, recall is the percentage of queries where the system does not abstain, and precision is the percentage of correct decisions among these queries. Each point on the curve Attorney Docket No.45288-0469WO1 corresponds to a particular abstention threshold value. On the recall area depicted in graphs 400 and 450, the system described, which uses a mutual information (MI) estimate as a measure of uncertainty, outperforms a semantic-entropy-based (SE) baseline system, e.g., as implementing the method detailed in Kuhn, L, et. al. “Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation” (International Conference on Learning Representations (ICLR 2023)) on the high-entropy multi-label queries. In particular, apart from low recall values, the performance of the semantic entropy system degrades noticeably with the addition of extra multi-label data.

[0111] FIG. 5 is a flow diagram of an example process 500 for determining whether to provide candidate outputs as a response to an input using a measure of uncertainty determined using sequences of candidate outputs. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, an abstention policy inference system, e.g., the abstention policy inference system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.

[0112] The system can receive an input (step 510), and the system can process the input using a neural network to generate one or more first candidate outputs and a corresponding first likelihood for each of the first candidate outputs (step 520). For example, the input can include an image, audio, or video. The image and / or video may be an image and / or video of a real-world environment. Such an image or video may have been captured by a camera. The audio may be audio of a real-world environment. Such audio may have been captured by a microphone. In some cases, the neural network can be a language processing neural network or a visual language model. In other cases, the neural network can be a multi-modal generative model.

[0113] The system can then generate, for each of the first candidate outputs, one or more sequences of one or more additional candidate outputs and a respective second likelihood for each additional candidate output in the sequence (step 530). In particular, the system can construct a sequence of candidate outputs by processing the input, the first candidate output, and any additional candidate outputs preceding the additional candidate output in the sequence using the neural network to generate the additional candidate output and the corresponding second likelihood for the additional candidate output.

[0114] The system can determine a measure of uncertainty using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs among the Attorney Docket No.45288-0469WO1 sequences of candidate outputs (step 540). In particular, the system can determine the measure of uncertainty by generating an aggregated likelihood , e.g., the pseudo-likelihood, based on the candidate outputs appearing in the sequence, and determining the measure of uncertainty using the aggregated likelihoods. For example, the system can generate the aggregated likelihood using the first likelihood for the first candidate output and the second likelihoods for the additional candidate outputs generated in the sequence of additional candidate outputs for the first candidate output.

[0115] In some cases, the system 100 can determine a set of unique candidate outputs among the sequences of candidate outputs. For example, the system can cluster the first candidate output and any additional candidate outputs among the sequences of candidate outputs generated for each of the first candidate outputs. In this case, the system can determine the mutual information estimate using the set of unique candidate outputs.

[0116] In some cases, the measure of uncertainty can be a mutual information estimate. In this case, the system can determine the mutual information estimate by normalizing the aggregated likelihoods to generate an approximation of a joint distribution of candidate outputs of the first candidate output and any additional candidate outputs, and generating an approximation of a product distribution of candidate outputs for each of the first candidate outputs and any additional candidate outputs using the aggregated likelihoods. The system can then determine the mutual information estimate using the approximation of the joint distribution of candidate outputs and the approximation of a product distribution of candidate outputs, e.g., by determining a measure of divergence between the approximation of the joint distribution of candidate outputs and the approximation of the product distribution of candidate outputs. In some cases, the mutual information estimate can be parameterized using one or more stabilization parameters. As an example, the measure of divergence can be a Kullback-Leibler (KL) divergence.

[0117] For example, the system can generate the approximation of the product distribution by determining a respective marginal sum of corresponding likelihoods for each candidate output for each processing of the input, the first candidate output, and any additional candidate outputs preceding the additional candidate output in the sequence at each processing iteration. That is, the system can determine the marginal distribution of the responses at the first, second, third, etc. positions in the sequence. More specifically, the system can determine the sum of first likelihoods for the first candidate output or the sum of second likelihoods for the additional candidate outputs depending on the appearance of the first candidate output or the additional candidate output at the Attorney Docket No.45288-0469WO1 particular position in the sequence of candidate outputs. The system can then determine a product of the respective marginal sums computed for each candidate output at each position as the approximation of the product distribution of candidate outputs.

[0118] The system can determine whether to provide any of the candidate outputs as a response to the input based on the measure of uncertainty (step 550). For example, the system can determine whether to provide one or more of the candidate outputs based on whether the measure of uncertainty satisfies a criterion, e.g., based on an abstention threshold value. In particular, the system, or another calibration system, can have determined the abstention threshold value by calibrating an abstention policy using a calibration set of input-response pairs.

[0119] As an example, the system can determine to provide one or more of the candidate outputs as a response when the measure of uncertainty satisfies a criterion, and the system can determine not to provide any of the candidate outputs as a response to the input when the measure of uncertainty does not satisfy the criterion. For example, the system can select a first candidate output from the sequence with the lowest measure of uncertainty, or greedily according to the relative likelihoods of the first candidate outputs. As another example, in response to determining not to provide any of the candidate outputs, the system can provide an abstention response that is not generated using any of the one or more candidate outputs, or can refrain from providing any response to the input.

[0120] In some cases, the system can obtain the one or more first candidate outputs and a corresponding first likelihood for each of the first candidate outputs as output from a second neural network, e.g., from an external system. In this case, the system can generate the sequence of one or more additional candidate outputs and a respective second likelihood for each additional candidate output in the sequence by generating a network input that includes the input, the first candidate output, and any additional candidate outputs preceding the additional candidate output in the sequence, and providing the network input as input to the second neural network to obtain the additional candidate output and the corresponding second likelihood for the additional candidate output. The system can then determine the measure of uncertainty using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs, e.g., obtained from the second neural network in each processing iteration, and can determine whether to provide any of the candidate outputs based on the measure of uncertainty. Attorney Docket No.45288-0469WO1 As an example, the system can provide an indication of whether to provide any of the candidate outputs as the response to the input, e.g., to the external system.

[0121] In some cases, the system can be used to control a robot interacting in an environment, e.g., a virtual or physical environment. For example, the input can include an observation characterizing a state of the environment being interacted with by the robot, e.g., an image of the environment, a natural language instruction to the robot, or both, and each first and additional candidate output can specify a respective action to be performed by the robot in response to the observation. In this case, determining, based on the measure of uncertainty, whether to provide any of the candidate outputs as a response to the input includes determining whether to use any of the respective actions to control the robot in response to the observation. In particular, in response to determining to use one of the respective actions to control the robot, the system can control the robot using one of the respective actions.

[0122] In some cases, the task that the neural network is used to perform may relate to multimodal data. The neural network may be considered a multimodal machine leaning model. In the discussion below, the machine learning model may be referred to simply as “the model”. As used herein a “modality” refers to a type of data, and thus a multimodal machine learning model is one that can process multiple different types of data. The model may receive, as an input, multimodal data. The model may then provide, as an output, information relating to (e.g., derived from) the multimodal data. The input may be a query, for example a text-based question with an associated data item (e.g., an image). The output may be representative of a response to the query. The output may be in the form of text (e.g., a transcription), an audio utterance (e.g., a simulated spoken utterance), or any other means of conveying information. The multimodal data may include one or more of: a text data item, image data item, audio data item (representing values of a digitized audio waveform, e.g., a time sequence of waveform-representing elements), sensor data (representing values of a digitized sensor waveform). An image data item may comprise image data representing a still or moving image, i.e., an image or video, e.g., an image or video that has been captured using a camera. Elements of the image data may comprise monochrome or color pixels of the image or video. As defined herein an “image” includes a point cloud, e.g., from a LIDAR system, and a “pixel” includes a point of the point cloud. Similarly “video” includes a time sequence of point clouds. Objects in the image or video may comprise objects, e.g., physical objects, represented by the image or video. Attorney Docket No.45288-0469WO1

[0123] Text may be received, e.g., as a series of encoded characters, e.g., UTF-8 encoded characters; such “characters” can include Chinese and other similar characters, as well as logograms, syllabograms and the like. The multimodal machine learning model can include a text encoder that processes a sequence of text to represent the text as a series of text tokens from a vocabulary of text tokens, e.g., that each represent words, wordpieces or characters in a natural or computer language. The computer language may be any formal language used to communicate with a computer, e.g., a markup language, or a command or configuration language, or a data exchange language such as JSON, or a programming language.

[0124] The model output may comprise any form of output appropriate to the machine learning task performed by the multimodal machine learning model. For example the model output may comprise text in a natural or computer language that defines a result of the task, e.g., for tasks such as image captioning, visual question answering, or object detection or instance segmentation. Also or instead the model output may comprise data defining an image, video or audio object, e.g., in a generative task; or the model output may comprise non-textual action selection data for selecting an action to be performed by an agent controlled by the model. As another example the model output may also or instead define an intermediate step to be performed during the task, e.g., a call to a software API for a software tool that is used when performing the task; the multimodal input may then receive an output from the software tool that is used to generate a final model output that performs the task. A few particular examples of model output are given later.

[0125] Such a multimodal model (e.g., a VLM) can be trained using very large (but possibly noisy) datasets in which text is paired with an image and / or with one or more other types of data, e.g., audio data, or data relating to the operation of an agent acting in an environment to perform a variety of tasks. Such a model is can be trained, e.g., using self-supervised learning. The pairing can often be imperfect, and the training dataset can, but may not, include any actual examples of a particular task to be performed, but nonetheless an ability to perform a particular task can emerge. There are many examples of suitable, publically available training datasets. A trained VLM may be referred to as a pre-trained VLM.

[0126] Some example multimodal machine learning models with which the techniques described herein may be used include: Flamingo (Alayrac et al. arXiv:2204.14198); ALIGN (Jia et al., arXiv:2102.05918); PaLI (Chen et al. arXiv:2209.06794) and PaLI-X (Chen et al. arXiv:2305.18565); and Gemini (“Gemini: A Family of Highly Capable Multimodal Models, Attorney Docket No.45288-0469WO1 Gemini Team, Google”). These references also include indications of training datasets that may be used to train the respective models.

[0127] In general, performing an image processing task using a multimodal neural network such as a visual language model (VLM), that includes an (a trained) image encoder neural network, can involve providing an image to the image encoder neural network to generate a representation of the image, in particular of features of the image. The representation of the image is then processed to perform the image processing task. In general techniques for processing image representations to perform a wide range of image processing tasks are well known. In general a video processing task can be performed by processing the representations of multiple images representing a sequence video frames.

[0128] An example involves generating an output that requires reasoning, e.g., spatio-temporal reasoning, to respond to a natural language query input, e.g., relating to a moving image (video). For example such a query may require predictive reasoning (“what will happen next”), counterfactual reasoning (“what would happen in a different circumstance”), explanatory reasoning (“why did something happen”), or causal reasoning generally. For example the image representation can be used to detect objects in the video frames and provide information relating to the detected objects in response to a query. The query may comprise, for example, a request for a prediction of a future event or state relating to one or more of the objects (e.g., “will objects X and Y collide?”), or a request for conditional or counterfactual information relating to one or more of the objects (e.g., “what event would [not] happen if object X is modified, moved or absent?”), or a request for analysis of the video frames to determine a property or characteristic of one or more of the objects (e.g., “how many objects of type Z are moving?”). The output may, for example, be in the form of a yes / no answer, or may define a probability distribution over a set of possible answers; or the response may define the location of an object. Such systems can be used to predict whether or not two objects will collide, or how this may be avoided. The output may be used e.g., to provide a warning and / or to control motion of one or more of the objects

[0129] Example tasks for multimodal models such as a Vision-Language Model (VLM) are described below.

[0130] In general a multimodal machine learning model can be trained to perform any sort of machine learning task or tasks. After the multimodal machine learning model has been trained it can be deployed for use in performing the machine learning task(s). For instance, the machine Attorney Docket No.45288-0469WO1 learning model can be deployed in an environment that enables users to provide requests for the machine learning model to process specified multimodal inputs to generate corresponding model outputs. Users can provide the requests, e.g., by way of a user interface or through an application programming interface (API). The requests can be transmitted from a user device (e.g., over a data communication network, e.g., the internet) to one or more computers implementing the machine learning model, e.g., in a data center. The machine learning model can process multimodal inputs specified by user requests to generate corresponding model outputs, and then transmit the model outputs to user devices (e.g., over a data communication network).

[0131] The following describes an intuition about why implementing the techniques described above can be useful in determining whether or not to provide an output from a multimodal machine learning model.

[0132] For example, in the case that an answer A1 has appeared many times in training data, then the parameters of the multimodal machine learning model will be learned such that the corresponding likelihood of A1, e.g., the softmax value of A1 from the output layer of the multimodal machine learning model, is large and will remain large despite the inclusion of additional candidate outputs as potential answers in the input in a sequence of candidate outputs, thereby indicating a high measure of certainty in the answer A1. As another example, in the case that an answer A1 did not appear many times in training data, the parameters of the multimodal machine learning model will be learned such that the softmax of the output layer of the multimodal machine learning model for A1 is not large or decreases over the course of generating the sequence of candidate outputs by including the additional candidate outputs as potentially answers in the input, thereby indicating a low measure of certainty in the answer A1.

[0133] In the case that the multimodal machine learning model is a language processing neural network, the sensitivity of the language processing neural network can be observed in a single attention head. For example, a self-attention head with context X and condition Z | X can be defined as where Wq, Wk, and Wv are the query, key, and value matrices, Z represents the set of examples included in the context, ETrepresents the end of the query, d is the dimension of each query in the query matrix, and X is a candidate output, e.g., answer A1. Attorney Docket No.45288-0469WO1

[0134] In this case, if an element of the input lies within principal components of the key-query matrix product, then the output will be generated according to the value matrix. On the other hand, if an element of the prompt has little overlap with principal components, and appears often in the examples included in the context, it is more likely to be copied from the input as the response.

[0135] For example, if X has appeared many times in training, then the parameters Wq and Wk will be learned such that X is the response. As another example, if X has not appeared many times in the training data, and vector Y, e.g., A2 which is an incorrect response but still a candidate output, appears in m rows of the context Z, then Y will be the response. .

[0136] In some to be performed by the multimodal machine learning model can be described by part or all of the sequence of text in the multimodal input to the model. For example in a multimodal input that includes an image such a prompt might specify “Generate a caption”, “Generate a description”, “Answer the following question: [about the image or video]”, or “Detect a person”. Where the model is used for an agent control task a prompt may define “Take the knife out of the drawer”, or “Q: What action should the robot take to take the knife out of the drawer?”. Also or instead such a prompt may give one or more examples of a task to be performed. A multimodal machine learning model can be trained on multiple natural and / or computer languages and the prompt may then specify a language to use.

[0137] A few examples of some machine learning tasks that can be performed by a model trained as described herein follow. The tasks described below may be tasks that require spatial awareness or other context from the image or video. For example, a prompt may ask “What is the object in the top left corner?”.

[0138] As one example the task may comprise an object or action detection task. A task-specific training data item may comprise an image or video containing one or more objects or actions, and a sequence of text. The sequence of text may describe or otherwise label the object(s) or action(s) and may include text giving bounding box coordinates for the object(s) or action(s). After training, when the model is used in inference, the model output may comprise or represent text that Attorney Docket No.45288-0469WO1 describes or otherwise labels detected object(s) or action(s) in the image input, and may include bounding-box coordinates for the detected object(s) or action(s), e.g., "102090100 cat 2030100 100 dog”.

[0139] As another example the task may comprise a classification task, e.g., an object or action classification task. A task-specific training data item may comprise an image or video item containing one or more objects or actions and a sequence of text. The sequence of text may describe or otherwise classify the object(s) or action(s). After training, when the model is used in inference, the model output may comprise data, e.g., text, that classifies the object(s) or action(s) in the image input into one of a plurality of classes.

[0140] As another example the task may comprise an image or video describing task, e.g., a captioning task (which, as used here, includes an audio description task to explain what is happening in a video). A task-specific training data item may comprise an image or video and a sequence of text describing the image or video. After training, when the model is used in inference, the model output may comprise data, e.g., text, describing the image or video. For example the model output may provide a caption or description or it may count objects in the image or video, or it may provide some other form of description.

[0141] As another example the task may comprise an image or video question-answering task. A task-specific training data item may comprise an image or video and a sequence of text that describes the image or video. After training, when the model is used in inference, the model output may comprise data, e.g., text, that answers a question about the second modality input specified in a prompt sequence of text, e.g., as described above. This may be used, e.g., to answer questions about visual plots and charts or about sounds.

[0142] As another example the task may comprise a character or word recognition task, e.g., an OCR (optical character recognition) task. A task-specific training data item may comprise an image or video and a sequence of text that includes text that is depicted in the image or video, or that is represented as speech in the audio item. After training, when the model is used in inference, the model output may comprise text that represents characters or words in the second modality input, e.g., in a natural language.

[0143] As another example the task may comprise a still or moving image generation task; Google DeepMind Gemini is an example of a system that can generate an image output. As another example, an image such as a plot or chart may be decoded from one or more (language) tokens Attorney Docket No.45288-0469WO1 generated by the system. A training data item for such a system may comprise an image or video and a sequence of text that describes the image or video. After training, when the model is used in inference, the model output may comprise data for an image or video, e.g., image data defining values for pixels of a still or moving image, and the sequence of text in the multimodal input to the model may describe or characterize the image or video to be generated.

[0144] As another example the task may comprise a computer language text generation task. A task-specific training data item may comprise an image or video and a sequence of text in a computer language for generating the image or video. After training, when the model is used in inference, the model output may comprise text in the or another computer language for generating or rendering an image or video, e.g., a web page, plot, or chart.

[0145] As one example, the model receives input in the form of an image data item and a text data item. The text data item may be a question about the image data item. The image data item may, for example, contain an image of one or more objects, and each object may belong to a particular class. The text data item may be a question regarding the one or more objects, the class of said objects, the position of an object or feature within the image (e.g., a pixel location within the image) or any other characteristic of the objects or of the image.

[0146] In an example using multimodal data, the input comprises an image of a person wearing a cap and driving a car. The input also comprises the question “what is the person wearing?”. The model can provide an answer to the question. In the example, a correct answer would be “a cap”. Other correct answers in the example could include “headwear”, “a hat” or “a baseball cap”. An incorrect answer could be “a car”. The correct answers relate to characteristics of the image (e.g., class of objects in the image) but, importantly, are contextualized in light of the question. As such, the correct answer relates to objects in a class relevant to ‘wearable clothing’ and disregards answers relevant to objects in unrelated classes (e.g., the car). Applying the methods described herein to this example, answers relating to hat, headwear, caps etc. would have a large similarity score and would therefore be more likely to be returned as an answer. Applying the methods described herein to this example, answers relating to the car would likely have a low similarity with the majority of other answers and so the model would likely abstain from providing this as an answer.

[0147] In another example using multimodal data, the input comprises a video (e.g., multiple images forming frames of a moving model) of a car driving near a pedestrian. The input also Attorney Docket No.45288-0469WO1 comprises the question “will the car hit the pedestrian?”. The model provides, as an output, a prediction of whether the car will or will not hit the pedestrian. The prediction may be made with reference to a physical model, explicit or implicit comparison to similar scenarios, or any other information that the model can understand from the video. Such scenario predictions are of particular use for control of autonomous vehicles or driver assistance features. It can be imagined that in vehicle control applications such as these, the risk tolerance may be lower and so it may be beneficial to set the abstention threshold and similarity threshold values to ensure a lower risk tolerance.

[0148] In another example, the input to the model may comprise a text data item, an image data item, or both. The input may be a query related to the selection of or generation of an image with particular characteristics. For example, the query may be a text-based request for the selection or generation of an image of an object of a particular class. The query may include an image of an object of a particular class and a text-based request for the selection or generation of an image of an object having the same class as the object in the image in the query. The model provides, as an output, an image responsive to the query, e.g., an image of an object of the particular class. The output image may be selected from a set of candidate images. The candidate images may be images generated by the model. Using the models described herein, the model is more likely to provide an image that accurately represents the type of image requested and is more likely to abstain rather than provide an incorrect image. A generated image of an object can be considered a simulation of said object. The object may be a real-world object, for example a real-world object belonging to a particular real-world class.

[0149] The model may be trained on training images, for example images which represent real- world objects. When a model that is trained to generate images is provided with a task to generate an image, it processes an input to select values (e.g., RGB, intensity, gamma) for pixels of an image. The pixel values are selected and arranged such that the resultant image, formed by the pixels when displayed, represents an object. The object represented is related to the information contained within the query. The generated image may share characteristics with (e.g., pixel values and arrangements) with training images upon which it was trained. For example, a generated image of a cat will share some characteristics (e.g., pixel values) with training data images depicting cats.

[0150] In another example of a computer language text generation task a task-specific training data item may comprise an image or video and a sequence of text in a computer language for Attorney Docket No.45288-0469WO1 performing a task in relation to the image or video, e.g., a data processing task that involves analyzing the content of the image or video to provide a result of the analysis or, e.g., a search to search for information relating to the content of the image or video. The computer language in the model output may comprise computer language for invoking a function or calling one or more external APIs. Merely as one example, such an output may be formatted as a JSON object. As previously, the sequence of text in the multimodal input may define the task to be performed and the second modality input may comprise, e.g., an image or video in relation to which the task is to be performed, e.g., a task that involves manipulation of particular types of data that may benefit from access to an API such as mathematical data, date / time related data, scientific data, recent data that may post-date training of the model (that may be accessed by a search function or API), and so forth. After training, when the model is used in inference, the model output may comprise text in the or another computer language for performing a task, e.g., as described above, in relation to an image or video in the second modality input. The method may then include using the text in the computer language to perform the task.

[0151] In general where the model output comprises text this may be provided as speech representing the text.

[0152] In some implementations the machine learning task comprises an agent control task in which the agent interacts with an environment to perform the agent control task. In these implementations the multimodal input includes an observation characterizing the environment. For example the multimodal input can include a sequence of text that defines the task to be performed by the agent and the image can represent an observation of the environment, e.g., captured by a camera or other imaging device from a real-world environment. A task-specific training data item may comprise a sequence of text representing one or more actions of the agent, and an image observation of the environment. After training, when the model is used in inference, the model output comprises an action selection output, e.g., including text, that is used to select one or more actions to be performed by the agent in the environment in response to the observation. As another example the action selection output may also or instead define one or more low-level skills, e.g., from a vocabulary of previously learnt skills. As before, the sequence of text in the multimodal input to the model may describe the task to be performed, e.g., “What action should the robot take to [perform task]”. Attorney Docket No.45288-0469WO1

[0153] In some agent control implementations, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. For example, the agent may be a robot or other mechanical agent interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment. In these implementations, the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment. The actions may define control signals to control the robot or other mechanical agent, e.g., positions, torques, or other control signals for the parts of the mechanical agent, or higher-level control commands.

[0154] For example, the system can be used to control a robot. In this case, the input can be an observation characterizing a state of an environment that is being interacted with by a robot, and each candidate output can specify a respective action to be performed by the robot in response to the observation. For example, the input observation can be an image of the environment or a sensor input characterizing the environment. As an additional example, the output can specify one or more of positions, torques, or other controllable signals for the robot in response to the observation. In particular, the output can be a natural language instruction to the robot, e.g., specifying target positions, e.g., the output may define an action as text such as “A: 132114128525156”, that canbe converted into a control signal for a mechanical agent, such as a robot, e.g., “^^ =^0.1, −0.2,0^ ^^ = ^10^ , 25^ , −7^^” to achieve a target position. The system can determinewhether to provide any of the candidate outputs as a response to the input based on the measure of uncertainty, e.g., to determine whether to use any of the respective actions to control the robot in response to the observation.

[0155] In some agent control implementations the agent may be a human agent and the environment may be a real-world environment. For example the agent can be a human user of a digital assistant such as a smart speaker, smart display, or some other device that is used to instruct the user to perform actions. The task may be any real-world task that the user wishes to perform. The observations may be obtained from an observation capture subsystem, e.g., a monitoring system such as a video camera or sound capture system, to capture visual observations of the user Attorney Docket No.45288-0469WO1 performing the task. The actions may comprise instructions in the form of, e.g., text, image, video, or audio data such as speech, that guide the user in performing the task.

[0156] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0157] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine- generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud- based environments where components reside on different machines or within a cloud infrastructure.

[0158] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and Attorney Docket No.45288-0469WO1 specialized processing hardware such as field-programmable gate arrays (FPGAs) or application- specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0159] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.

[0160] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple Attorney Docket No.45288-0469WO1 engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of AI and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0161] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in AI and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0162] Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the AI model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be Attorney Docket No.45288-0469WO1 implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.

[0163] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence.

[0164] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0165] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0166] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a Attorney Docket No.45288-0469WO1 front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0167] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0168] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the Attorney Docket No.45288-0469WO1 combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0169] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0170] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0171] In addition to the embodiments described above, the following embodiments are also innovative:

[0172] Embodiment 1 is a method comprising: receiving an input, wherein the input comprises a query; processing the input using a neural network to generate one or more first candidate outputs and a corresponding first likelihood for each of the first candidate outputs, wherein each candidate output is a possible response to the input; for each of the first candidate outputs, generating one or more sequences of one or more additional candidate outputs and a respective second likelihood for each additional candidate output in the sequence, the generating comprising, for each additional candidate output: processing the input, the first candidate output, and any additional candidate outputs preceding the additional candidate output in the sequence using the neural network to generate the additional candidate output and the corresponding second likelihood for the additional candidate output; Attorney Docket No.45288-0469WO1 determining a measure of uncertainty using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs among the sequences of one or more additional candidate outputs for each of the first candidate outputs; and

[0173] determining, based on the measure of uncertainty, whether to provide any of the candidate outputs as the response to the input.

[0174] Embodiment 2 is the method of embodiment 1, wherein determining whether to provide any of the candidate outputs as the response to the input comprises: determining to provide one or more of the candidate outputs as a response to the input when the measure of uncertainty satisfies a criterion.

[0175] Embodiment 3 is the method of embodiment 2, wherein determining whether to provide any of the candidate outputs as the response to the input comprises: determining not to provide any of the candidate outputs as a response to the input when the measure of uncertainty does not satisfy the criterion.

[0176] Embodiment 4 is the method of any one of embodiments 2-3, wherein the criterion is based on an abstention threshold value.

[0177] Embodiment 5 is the method of embodiment 3, further comprising, in response to determining not to provide any of the candidate outputs: providing an abstention response that is not generated using any of the one or more candidate outputs; or refraining from providing any response to the input.

[0178] Embodiment 6 is the method of any one or embodiments 1-5, wherein determining the measure of uncertainty using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs among the sequences of one or more additional candidate outputs for each of the first candidate outputs comprises: generating, from the first likelihood for the first candidate output and the second likelihoods for the additional candidate outputs in the sequences of additional candidate outputs, an aggregated likelihood; and determining the measure of uncertainty using the aggregated likelihoods.

[0179] Embodiment 7 is the method of any one of embodiments 1-6, wherein the measure of uncertainty is a mutual information estimate. Attorney Docket No.45288-0469WO1

[0180] Embodiment 8 is the method of embodiment 7, when dependent on embodiment 6, wherein determining the mutual information estimate using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs further comprises: normalizing the aggregated likelihoods to generate an approximation of a joint distribution of candidate outputs for the first candidate outputs and any additional candidate outputs; generating an approximation of a product distribution of candidate outputs for each of the first candidate output and any additional candidate outputs using the aggregated likelihoods; and determining the mutual information estimate using the approximation of the joint distribution of candidate outputs and the approximation of a product distribution of candidate outputs.

[0181] Embodiment 9 is the method of embodiment 8, further comprising: determining a set of unique candidate outputs among the first candidate output and any additional candidate outputs among the sequences of one or more additional candidate outputs; and determining the mutual information estimate using the set of unique candidate outputs.

[0182] Embodiment 10 is the method of any one of embodiments 8-9, wherein generating an approximation of a product distribution of candidate outputs using the aggregated likelihoods comprises: for each processing of the input, the first candidate output, and any additional candidate outputs preceding the additional candidate output in the sequence using the neural network, determining a respective marginal sum of corresponding likelihoods for each candidate output at a first position in the sequence; and determining a product of the respective marginal sums for each candidate output as the approximation of the product distribution of candidate outputs at each position in the sequence as the approximation of the product distribution of candidate outputs.

[0183] Embodiment 11 is the method of embodiment 10, wherein the respective marginal sum of corresponding likelihoods for each candidate output comprises: the sum of first likelihoods for the first candidate output; or the sum of the second likelihoods for the additional candidate outputs.

[0184] Embodiment 12 is the method of any one of embodiments 8-9, wherein computing the mutual information estimate comprises determining a measure of divergence between the Attorney Docket No.45288-0469WO1 approximation of the joint distribution of candidate outputs and the approximation of the product distribution of candidate outputs.

[0185] Embodiment 13 is the method of embodiment 12, wherein the mutual information estimate is parameterized using a stabilization parameter.

[0186] Embodiment 14 is the method of any one of embodiments 1-13, wherein the neural network is a language processing neural network.

[0187] Embodiment 15 is the method of any one of embodiments 1-14, wherein the input comprises an image, audio, or a video.

[0188] Embodiment 16 is the method of any one of embodiments 1-15, wherein the neural network is a multi-modal generative model.

[0189] Embodiment 17 is the method of any one of embodiments 1-16, wherein the neural network is a visual language model neural network.

[0190] Embodiment 18 is the method of any one of embodiments 1-17, wherein the input comprises an observation characterizing a state of an environment being interacted with by a robot, wherein each first and additional candidate output specifies a respective action to be performed by the robot in response to the observation, and wherein determining, based on the measure of uncertainty, whether to provide any of the candidate outputs as a response to the input comprises determining whether to use any of the respective actions to control the robot in response to the observation. The environment may be a real-world environment, and the robot a mechanical agent. The respective action to be performed by the robot may comprise one of: an object manipulation task or an environment navigation task.

[0191] Embodiment 19 is the method of embodiment 18, wherein the input further comprises a natural language instruction to the robot.

[0192] Embodiment 20 is the method of any one of embodiments 18-19, wherein the observation comprises an image of the environment. The image may be generated using an image sensor configured to sense a real-world environment. The image sensor may be a camera, for example.

[0193] Embodiment 21 is the method of any one of embodiments 18-20, further comprising:

[0194] in response to determining to use one of the respective actions to control the robot, controlling the robot using one of the respective actions. Attorney Docket No.45288-0469WO1

[0195] Embodiment 22 is the method of any one of embodiments 1-21, when dependent on embodiment 4, wherein the abstention threshold value has been determined by calibrating an abstention policy using a calibration set of input-response pairs.

[0196] Embodiment 23 is a method comprising: receiving an input, wherein the input comprises a query; obtaining, as output from a second neural network, one or more first candidate outputs for an input to the neural network and a corresponding first likelihood for each of the candidate outputs, wherein each candidate output is a possible response to the input; for each of the first candidate outputs, generating one or more sequences of one or more additional candidate outputs and a respective second likelihood for each additional candidate output in the sequence, the generating comprising, for each additional candidate output: generating a network input that comprises the input, the first candidate output, and any additional candidate outputs preceding the additional candidate output in the sequence; providing the network input as input to the second neural network to obtain, as output, the additional candidate output and the corresponding second likelihood for the additional candidate output; determining a measure of uncertainty using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs among the sequences of one or more additional candidate outputs for each of the first candidate outputs; and determining, based on the measure of uncertainty, whether to provide any of the candidate outputs as a response to the input.

[0197] Embodiment 24 is the method of embodiment 23, further comprising providing an indication of whether to provide any of the candidate outputs as the response to the input.

[0198] Embodiment 25 is the method of any one of embodiments 1-24, wherein the input comprises audio representing a spoken utterance and the candidate outputs are one or more of graphemes, characters, or words that represents the utterance.

[0199] Embodiment 26 is the method of any one of embodiments 1-25, wherein the input comprises an image and the candidate outputs are text that describe the image.

[0200] Embodiment 27 is the method of any one of embodiments 1-25, wherein the input comprises a text-based query and an image, and the candidate outputs are representative of a response to the query and in the form of text or an audio utterance. Attorney Docket No.45288-0469WO1

[0201] Embodiment 28 is a system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of embodiments 1 to 27.

[0202] Embodiment 29 is a computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any one of embodiments 1 to 27.

[0203] What is claimed is:

Claims

Attorney Docket No.45288-0469WO1 CLAIMS 1. A method comprising: receiving an input, wherein the input comprises a query; processing the input using a neural network to generate one or more first candidate outputs and a corresponding first likelihood for each of the first candidate outputs, wherein each candidate output is a possible response to the input; for each of the first candidate outputs, generating one or more sequences of one or more additional candidate outputs and a respective second likelihood for each additional candidate output in the sequence, the generating comprising, for each additional candidate output: processing the input, the first candidate output, and any additional candidate outputs preceding the additional candidate output in the sequence using the neural network to generate the additional candidate output and the corresponding second likelihood for the additional candidate output; determining a measure of uncertainty using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs among the sequences of one or more additional candidate outputs for each of the first candidate outputs; and determining, based on the measure of uncertainty, whether to provide any of the candidate outputs as the response to the input.

2. The method of claim 1, wherein determining whether to provide any of the candidate outputs as the response to the input comprises: determining to provide one or more of the candidate outputs as a response to the input when the measure of uncertainty satisfies a criterion.

3. The method of claim 2, wherein determining whether to provide any of the candidate outputs as the response to the input comprises: determining not to provide any of the candidate outputs as a response to the input when the measure of uncertainty does not satisfy the criterion.

4. The method of claim 2 or claim 3, wherein the criterion is based on an abstention threshold value.Attorney Docket No.45288-0469WO1 5. The method of claim 3, further comprising, in response to determining not to provide any of the candidate outputs: providing an abstention response that is not generated using any of the one or more candidate outputs; or refraining from providing any response to the input.

6. The method of any preceding claim, wherein determining the measure of uncertainty using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs among the sequences of one or more additional candidate outputs for each of the first candidate outputs comprises: generating, from the first likelihood for the first candidate output and the second likelihoods for the additional candidate outputs in the sequences of additional candidate outputs, an aggregated likelihood; and determining the measure of uncertainty using the aggregated likelihoods.

7. The method of any preceding claim, wherein the measure of uncertainty is a mutual information estimate.

8. The method of claim 7, when dependent on claim 6, wherein determining the mutual information estimate using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs comprises: normalizing the aggregated likelihoods to generate an approximation of a joint distribution of candidate outputs for the first candidate outputs and any additional candidate outputs; generating an approximation of a product distribution of candidate outputs for each of the first candidate output and any additional candidate outputs using the aggregated likelihoods; and determining the mutual information estimate using the approximation of the joint distribution of candidate outputs and the approximation of a product distribution of candidate outputs.

9. The method of claim 8, further comprising:Attorney Docket No.45288-0469WO1 determining a set of unique candidate outputs among the first candidate output and any additional candidate outputs among the sequences of one or more additional candidate outputs for each of the first candidate outputs; and determining the mutual information estimate using the set of unique candidate outputs.

10. The method of any one of claims 8-9, wherein generating an approximation of a product distribution of candidate outputs using the aggregated likelihoods comprises: for each processing of the input, the first candidate output, and any additional candidate outputs preceding the additional candidate output in the sequence using the neural network, determining a respective marginal sum of corresponding likelihoods for each candidate output at a first position in the sequence; and determining a product of the respective marginal sums for each candidate output at each position in the sequence as the approximation of the product distribution of candidate outputs.

11. The method of claim 10, wherein the respective marginal sum of corresponding likelihoods for each candidate output comprises: the sum of first likelihoods for the first candidate output; or the sum of the second likelihoods for the additional candidate outputs.

12. The method of any one of claims 8-9, wherein computing the mutual information estimate comprises determining a measure of divergence between the approximation of the joint distribution of candidate outputs and the approximation of the product distribution of candidate outputs.

13. The method of claim 12, wherein the mutual information estimate is parameterized using one or more stabilization parameters.

14. The method of any preceding claim, wherein the neural network is a language processing neural network.Attorney Docket No.45288-0469WO1 15. The method of any preceding claim, wherein the input comprises an image, audio, or a video.

16. The method of any preceding claim, wherein the neural network is a multi-modal generative model.

17. The method of any preceding claim, wherein the neural network is a visual language model neural network.

18. The method of any preceding claim, wherein the input comprises an observation characterizing a state of an environment being interacted with by a robot, wherein each first and additional candidate output specifies a respective action to be performed by the robot in response to the observation, and wherein determining, based on the measure of uncertainty, whether to provide any of the candidate outputs as a response to the input comprises determining whether to use any of the respective actions to control the robot in response to the observation.

19. The method of claim 18, wherein the input further comprises a natural language instruction to the robot.

20. The method of any one of claims 18-19, wherein the observation comprises an image of the environment.

21. The method of any one of claims 18-20, further comprising: in response to determining to use one of the respective actions to control the robot, controlling the robot using one of the respective actions.

22. The method of any preceding claim, when dependent on claim 4, wherein the abstention threshold value has been determined by calibrating an abstention policy using a calibration set of input-response pairs.

23. A method comprising:Attorney Docket No.45288-0469WO1 receiving an input, wherein the input comprises a query; obtaining, as output from a second neural network, one or more first candidate outputs for an input to the neural network and a corresponding first likelihood for each of the candidate outputs, wherein each candidate output is a possible response to the input; for each of the first candidate outputs, generating one or more of sequences of one or more additional candidate outputs and a respective second likelihood for each additional candidate output in the sequence, the generating comprising, for each additional candidate output: generating a network input that comprises the input, the first candidate output, and any additional candidate outputs preceding the additional candidate output in the sequence; providing the network input as input to the second neural network to obtain, as output, the additional candidate output and the corresponding second likelihood for the additional candidate output; determining a measure of uncertainty using the first likelihoods for the first candidate outputs and the second likelihoods for the additional candidate outputs among the sequences of one or more additional candidate outputs for each of the first candidate outputs; and determining, based on the measure of uncertainty, whether to provide any of the candidate outputs as the response to the input.

24. The method of claim 23, further comprising providing an indication of whether to provide any of the candidate outputs as the response to the input.

25. The method of any preceding claim, wherein the input comprises audio representing a spoken utterance and the candidate outputs are one or more of graphemes, characters, or words that represents the utterance.

26. The method of any preceding claim, wherein the input comprises an image and the candidate outputs are text that describe the image.

27. The method of any preceding claim, wherein the input comprises a text-based query and an image, and the candidate outputs are representative of a response to the query and in the form of text or an audio utterance.Attorney Docket No.45288-0469WO1 28. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of claims 1 to 27.

29. A computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any one of claims 1 to 27.

Images