Conformal abstention for neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for calibrating an abstention policy to mitigate prediction errors by a neural network. In one aspect, a system comprises receiving an input prompt, processing the input prompt using a neural network to generate a plurality of candidate outputs for the input prompt, determining a similarity score that characterizes a similarity of each of the candidate outputs to each other candidate output, determining whether the similarity score satisfies an abstention threshold value, and in response to determining that the similarity score satisfies a criterion based on an abstention threshold value being satisfied, providing one or more of the candidate outputs as a generated response to the input prompt.

Description

CONFORMAL ABSTENTION FOR NEURAL NETWORKSCROSS-REFERENCE TO RELATED APPLICATION[1] This application claims priority to U.S. Application No. 63 / 548,839, filed February 1, 2024 and U.S. Application No. 63 / 549,306, filed February 2, 2024, the disclosures of which are incorporated herein by reference.BACKGROUND[2] This specification relates to processing data using machine learning models.[3] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.[4] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.SUMMARY[5] This specification describes a system implemented as computer programs on one or more computers in one or more locations that can determine whether to abstain from responding or provide an output generated by a machine learning model, e.g., a neural network, in response to an input prompt. In particular, the system can apply an abstention policy to mitigate prediction errors when predicting and providing an output from a neural netw ork that is configured to generate multiple outputs given the same input prompt. More specifically, the system can determine whether to abstain when it is likely that the neural network is going to make a prediction error in responding to the prompt.[6] Generally, machine learning models that are trained to perform useful tasks are not able to generate outputs that are always accurate. Depending on the task, detecting errors in prediction can sometimes be difficult, e.g., for an end-user. For example, one type of error that is difficult to detect is a hallucination produced by a generative neural network, e.g.. a language processing neural network. In this specification, a hallucination is when a generative neural netw ork confidently outputs an incorrect answer as response to an input prompt, which can be hard for a user to reliably detect.[7] More specifically, the system can sample and compare candidate outputs of the neural network, e.g., k candidate outputs, that w ere generated for the same input prompt and determinea similarity score for the candidate outputs that characterizes a similarity' of each of the candidate outputs to each other candidate output as a proxy of a measure of confidence in response. More specifically, the system can use the similarity score to determine whether the neural network is susceptible to produce a prediction error, e.g., by comparing the similarity score to an abstention threshold value.[8] Additionally, the techniques of this specification specify the calibration of the abstention policy without direct access to the underlying parameters of the neural network. In particular, the system of this specification, or another calibration system, can calibrate the abstention policy in accordance with a specified risk tolerance criterion, e.g., the risk tolerance can be quantified as a rate of unfiltered hallucinations, by updating one or more parameterizing values of an abstention policy. As an example, the abstention policy can include an abstention threshold value and, optionally, a similarity threshold value.[9] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0010] The techniques of this specification enable the determination of whether a neural network that is configured to generate multiple different outputs for a given input prompt should abstain from responding to the input prompt in order to prevent prediction error. In particular, the system can implement an abstention policy that is able to mitigate prediction errors while maintaining a low abstention rate in accordance with a determined similarity score. In particular, the system can determine a similarity score for each response generated by the neural network for a particular input prompt by sampling and comparing candidate outputs of the neural network to each other as a proxy measure of confidence in response that can be compared to an abstention threshold value.

[0011] Additionally, the techniques of this specification provide for the calibration of the abstention policy, e.g., using the system of this specification, or another calibration system. More specifically, the calibration of the abstention policy does not require training or finetuning of the neural network, or another model, e.g., the training of a separate classifier to detect various properties such as factuality, toxicity, etc. in a response or the training of an ensemble of models to estimate the confidence in response, thereby reducing computational resources relative to jointly training the neural network, or another model, or an ensemble of models and calibrating the abstention policy. In particular, training an ensemble of general neural network models can be prohibitively computationally expensive. The techniques of this specification describe the calibrating of an abstention policy using only query access to a neuralnetwork to generate candidate outputs for different input prompts that can be used to determine the similarity score.

[0012] Moreover, the calibration can be performed with a calibration dataset that is much smaller than the dataset used to train the neural network, thereby enabling easy maintenance and facilitating updates of the abstention policy parameters. Calibration can even be performed online since it does not require any updates to the neural network. Since the quality of the abstention policy is defined by the calibration data, calibrating the abstention policy in an online setting can enhance the performance of the abstention policy through the inclusion of more data.

[0013] The calibration of this specification can be more effective at calibrating an abstention policy for prediction error with respect to longer responses generated by a neural network, e.g., relative to other methods used to calibrate for prediction error. In particular, methods that rely on maximizing log-probabilities are less effective for uncertainty quantification for neural networks since (i) incorrect and correct responses can have similar log-likelihoods, e.g., loglikelihoods within the same range of values, and (ii) the log-likelihood of a response is dependent on the length of the response. In contrast, defining an abstention policy using the techniques of this specification is more effective for longer responses than other methods since it does not rely on the generated log-likelihoods and does not require setting a given sequence length for response in order to calibrate the abstention policy. Relatedly, in an example implementation that involves calibrating an abstention policy for a generative neural network using the system of this specification, calibration performed with the system achieves better performance relative to other methods for preventing hallucination of generative neural networks, e.g.. log-probability and greedy sampling methods, e.g., sampling the highest probability token for each token, top-10 log-probability max, and other variations using sequence embeddings exact match / word overlap, etc.

[0014] Additionally, the calibration system can calibrate an abstention policy in accordance with a specified risk tolerance criterion, e.g., the risk tolerance can be quantified as a rate of unfiltered hallucinations, using conformal abstention by updating one or more parameterizing values of an abstention policy. More specifically, the specified risk tolerance criterion can provide an upper bound on the risk for optimizing the abstention policy. The system can guarantee the risk of the abstention policy remains below the risk tolerance criterion to achieve an approximately optimal solution to minimizing both the rate of abstention and the risk of the abstention policy through one or more of a conformal risk control or leam-then-test approach.Thus, the abstention policy can provide a theoretical guarantee on risk minimization through the calibrated policy based onthe specified risk tolerance criterion.

[0015] Furthermore, the system provides for flexibility in defining how the risk is calculated, e.g., with a rule or using an additional model. For example, a user can specify the risk tolerance criterion, independent of the risk calculation, neural network architecture, and underlying data distribution. The user can set the overall risk tolerance criterion to assert a particular desired upper bound on risk and the calibration system can calibrate the parameters of the abstention policy accordingly, e.g., the abstention threshold value and the similarity threshold value. As a particular example for a generative neural network, the calibration system can minimize the expected rate of abstentions while ensuring the expected risk of hallucination remains below the risk tolerance criterion.

[0016] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG. 1 A is a system diagram of an example abstention policy inference system that uses a calibrated abstention policy to determine whether to provide an output as response to an input prompt.

[0018] FIG. IB is block diagram of an example abstention policy calibration system.

[0019] FIG. 2 illustrates how the example abstention policy inference system of FIG. 1A can determine a similarity score that can be compared to an abstention threshold value to determine whether to provide an output as response to an input prompt.

[0020] FIG. 3 is a flow diagram of an example process for determining whether to provide one or more candidate outputs of a neural network based on a calibrated abstention policy.

[0021] FIG. 4 is a flow' diagram of an example process for calibrating an abstention policy for a neural network.

[0022] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0023] FIG. 1A shows an example abstention policy inference system 100. The abstention policy inference system 100 is an example of a system implemented as computer programs onone or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0024] The abstention policy inference system 100 can apply a calibrated abstention policy for any neural network that is configured to generate multiple candidate outputs for a given input prompt, e.g., by determining a similarity score that characterizes a similarity of each of the candidate outputs to each other candidate output as is described in more detail below.

[0025] In particular, the abstention policy inference system 100 can implement an abstention policy that defines when the system 100 should provide an abstention output 162 in response to an input prompt 110, e.g., instead of providing an incorrect response. As an example, the input prompt 110 can be a directive instruction from a user, e.g., a question, statement, code snippet, or example, that can be processed by the neural network 120 to generate a number of candidate outputs 125, e.g., candidate responses to the input prompt 110. For example, an input prompt 110 can be a request for information, e.g., “What city is Vallecito Elementary School located in?” or “How many calories worth of food does a 20 pound cat eat per day?”.

[0026] The system can 100 generally calibrate the abstention policy for a neural network 120 that is configured to generate multiple candidate outputs 125 for a given input prompt, e.g., the input prompt 110. That is, the neural network 120 can have any appropriate number of neural network layers (e.g., 1 layer, 5 layers, or 10 layers) of any appropriate type (e.g., fully- connected layers, attention layers, convolutional layers, etc.) connected in any appropriate configuration (e.g.. as a linear sequence of layers, or as a directed graph of layers).

[0027] In some cases, the neural network 120 is a generative model, e.g., a generative- adversarial network or an autoregressive language processing network. As an example, the neural network 120 can have a recurrent neural network architecture. More specifically, the neural network 120 can be a recunent neural network (RNN), long short-term memory (LSTM), or gated-recurrent unit (GRU). As another example, the neural network model 125 can be transformer-based, e.g., an encoder-decoder transformer, an encoder-only transformer, or a decoder-only transformer, configured to perform parallel processing of the contents of the multimodal input using a multi-headed attention mechanism.

[0028] As a particular example, the neural network 120 can be a language processing neural network. In this specification, a hallucination is a particular type of neural network prediction error that occurs when a generative processing neural network, e.g., a language processing neural network, confidently outputs an incorrect answer as response to an input prompt 110, which can be hard for a user to reliably detect.

[0029] A language processing neural network is an auto-regressive network that is configured to sequentially process the contents of an input and trained to perform next element prediction, e.g., to define a likelihood score distribution over a next set of elements. In particular, the neural network can be referred to as an auto-regressive neural network when the neural network auto- regressively generates an output sequence of tokens. More specifically, the auto-regressively generated output is created by generating each particular token in the output sequence conditioned on a current input sequence that includes any tokens that precede the particular token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token.

[0030] For example, the language processing neural network can be an auto-regressive Transformer-based neural network that includes (i) a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0031] In this example, the language processing neural network can have any of a variety of Transformer-based neural network architectures e.g., an encoder-decoder transformer, an encoder-only transformer, or a decoder-only transformer. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022; J.W. Rae. S. Borgeaud, T. Cai,K. Millican, J. Hoffmann, H. F. Song. J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Eisen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini. L. Sifre, L. Martens. X.L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac. E. Lockhart. S. Osindero. L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. arXiv preprint arXiv: 1910.10683, 2019; Daniel Adiwardana, Minh-ThangLuong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like opendomain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry. Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.

[0032] Generally, to apply the self-attention operation, each attention block uses one or more attention heads. Each attention head generates a set of queries, a set of keys, and a set of values, and then applies any of a variety of variants of query-key-value (QKV) attention, e.g., a dot product attention function or a scaled dot product attention function, using the queries, keys, and values to generate an output. Each query, key, value can be a vector that includes one or more vector elements. When there are multiple attention heads, the attention block then combines the outputs of the multiple attention heads, e.g., by concatenating the outputs and, optionally, processing the concatenated outputs through a linear layer.

[0033] More specifically, a language processing neural network is configured to generate multiple outputs 125 for each input prompt in a single processing iteration. Since the language processing neural network is configured to define a likelihood score distribution over a next set of elements, e.g., an output sequence of elements that represents probabilities for each element of the output sequence, the language processing neural network can generate a number of candidate outputs 125 by processing the input prompt 110 and sampling different output elements for each element of the output sequence. In some cases, the language processing neural network can generate and sample the N most likely responses based on the probability values as the candidate outputs 125, e.g., by performing top-k sampling.

[0034] In some cases, the language processing neural network is a visual language model (VLM) that can be configured to process an input prompt 110 that includes an image or sequence of images in a video to generate an intermediate representation of the image and perform an image processing task. For example, the visual language model can be a contrastive language-image pre-training (CLIP) model, a vision transformer (ViT). a unified image-to- image translation (UNIT) model, or an attention generative adversarial network (AttnGAN).

[0035] As an example, the image processing task can involve generating an output that requires reasoning, e.g., spatio-temporal reasoning, to respond to a natural language query input, e.g., relating to a moving image (video). For example, such a query may require predictive reasoning ("what will happen next"), counterfactual reasoning f‘what would happen in a different circumstance”), explanatory reasoning (‘'why did something happen”), or causalreasoning generally. For example, the image representation can be used to detect objects in the video frames and provide information relating to the detected objects in response to a query, e.g., a request for a prediction of a future event or state relating to one or more of the objects (e.g., “will objects X and Y collide?”), or a request for conditional or counterfactual information relating to one or more of the objects (e.g., “what event would [not] happen if object X is modified, moved or absent?”), or a request for analysis of the video frames to determine a property or characteristic of one or more of the objects (e.g., “how many objects of type Z are moving?”). The output may, for example, be in the form of a yes / no answer, or may define a probability distribution over a set of possible answers; or the response may define the location of an obj ect. Such a base network can be used to predict whether or not two obj ects will collide, or how this may be avoided. The output may be used e.g., to provide a warning, to control motion of one or more of the objects, or both.

[0036] As another example, the neural network can be a diffusion model that is configured to sequentially refine an initial state representing the input prompt 110 through a sequence of transformations that add noise to a data sample to generate an output. In some cases, the neural network can be implemented as a latent diffusion model that includes an encoder block to map images to a lower-dimensional latent space before applying the sequence of transformations and a decoder block to map from the lower-dimensional latent space back into image space. As an example, the neural network can be implemented as a U-Net that includes skip connections that allow the neural network to combine both coarse features from the beginning of the sequence of transformations and fine features from the end of the sequence of transformations to improve the generated image quality.

[0037] More specifically, the system 100 can apply an abstention policy using an abstention policy subsystem 130 to determine whether to generate an abstention output 162 instead of providing a candidate output 164 as a response 160. In particular, the system 100 can process the candidate outputs 125 using the abstention policy subsystem 130 to determine a measure of similarity, e.g., an overall similarity score 155, between each of the candidate outputs as a proxy measure of confidence in the response. The subsystem 130 can then use the similarity score 155 to determine whether the neural network 120 is susceptible to produce a prediction error, e.g., a hallucination, for a specific input prompt 110 by comparing the similarity score to an abstention threshold value 134.

[0038] The abstention policy can include one or more calibrated abstention policy parameters 132. For example, the abstention policy can include an abstention threshold value 134. As another example, the abstention policy can include an abstention threshold value 134 and asimilarity threshold value 136. In particular, the abstention parameters 135 deployed in the abstention policy inference system 100 are calibrated values. For example, a counterpart abstention calibration system, or another calibration system, can calibrate the abstention policy parameters 132 by tuning the values of the parameters 135 using a set of example promptresponse pairs, as will be described in more detail with respect to FIGS. IB and 4.

[0039] In some cases, the system 100 can apply an abstention policy associated with a specified risk tolerance. In this context, the subsystem 130 can determine whether to provide an actual candidate output 1 4 as a response 160 based on whether the measure of similarity between each of the candidate outputs satisfies a specific risk tolerance criterion that was used to calibrate the parameterizing values of the abstention policy. For example, a counterpart abstention calibration system, or another calibration system, can calibrate the abstention policy parameters 132 in accordance with a specified risk tolerance criterion, e.g., the risk tolerance can be quantified as the total number of hallucinations resulting from applying the abstention policy with the abstention policy parameters at a particular calibration iteration, as will be described in more detail with respect to FIGS. IB and 4. In this case, the abstention policy can provide a theoretical guarantee on risk minimization through the specified risk tolerance criterion.

[0040] In particular, the abstention policy subsystem 130 can compare the candidate outputs 125 generated for a given input prompt 110 using the neural network 120, e.g., a subset of k sampled candidate outputs, by determining an overall similarity score 155 for the candidate outputs 125 that characterizes a similarity of each of the candidate outputs to each other candidate output. The subsystem 130 can then compare the similarity score 155 to the abstention threshold value 134 to determine whether a response should be provided as the response 160, e.g., to a user.

[0041] For example, the abstention policy subsystem 130 can process the candidate outputs 125 using a similarity neural network 140 to determine respective pairwise similarity' scores 145 for each of the pairs of candidate outputs 125. The subsystem 130 can then combine the pairwise similarity scores 145 into the overall similarity score 155.

[0042] The similarity neural network 140 can have any appropriate neural network architecture that can be configured to process a pair of candidate outputs to generate a respective pairwise similarity score for the pair of candidate outputs. In particular, the similarity neural network 140 can have any appropriate number of neural network layers (e.g., 1 layer, 5 layers, or 10 layers) of any appropriate type (e.g., fully-connected layers, attention layers, convolutionallayers, etc.) connected in any appropriate configuration (e.g., as a linear sequence of layers, or as a directed graph of layers).

[0043] In some cases, the similarity neural network 140 can be a language processing neural network, e.g., a different language processing neural network than the language processing neural network, in the case that the neural network 120 is implemented as a language processing neural network. In particular, the pairwise similarity scores 143 can be computed by prompting a second language processing neural network with an input that includes a pair of candidate outputs and an instruction to determine the similarity between the pair of candidate outputs, e.g., on a standardized scale from 1 to 5, 1 to 10, etc., to predict the pairwise similarity' score. For example, a prompt can be formulated as:“Given a context, measure the similarity of a reference text with 10 comparison texts on a scale of 1-10. Your response should be 10 numbers, where the i-th number is the similarity of the reference text with the i-th comparison text. Here are a context, a reference text, and 10 comparison texts for you to evaluate: Context: personalized prompt Reference Text: responseComparison Textl: responsel Comparison Text2: response2 Comparison Text3: responses Comparison Text4: response4 Comparison Text5: responses Comparison Text6: response6 Comparison Text7: response? Comparison Text8: responses Comparison Text9: response9 Comparison TextlO: responselO Y our response should be a list of 10 comma separated numbers.’'

[0044] In some cases, the similarity neural network 140 can be a lightweight neural network, e.g., a lightweight model with a smaller, less complicated, or both architecture relative to the neural network 120. In particular, the similarity neural network 140 can have fewer trained parameters, e.g., lOx less, lOOx less, etc., than the neural network 120. In other cases, the similarity neural network 140 can be the same neural network as the neural network 120.

[0045] As an example, in the simplified case where four candidate outputs are generated, to generate all of the possible pairwise similarity scores, the subsystem 130 can calculate the similarity scores between the candidate outputs 1:2, 2: 1, 1 :3, 3: 1, 1 :4, 4: 1, 2:3, 3:2, 2:4, 4:2, 3:4, and 4:3. As another example, the subsystem 130 can calculate the similarity scores between candidate outputs 1:2-4. 2:3-4, and 3:4. e.g., since the pairwise similarity score between the same pair of outputs generally should not depend on the order. In particular, this provides amore computationally efficient way of generating similarity' scores for multiple candidate outputs. This is especially of use in situations where the computer / processor performing the similarity scoring has a limited computational power (e.g., such as on personal devices like mobile phones, smart watches, etc.).

[0046] The subsystem 130 can generate a set of pairwise similarity' scores 145 by processing the candidate outputs 125 using the similarity neural network 140. The subsystem 130 can then determine an overall similarity score 155 that measures the similarity between the candidate output and each other candidate outputs using the pairwise similarity scores 145, e.g., the overall similarity' score 145 compares each particular candidate output with the other candidate outputs, in contrast to the pairwise similarity' score.

[0047] In the particular example depicted, the subsystem 130 can process the pairwise similarity’ scores 145 using a clustering engine 150 to determine the overall similarity score 155. In this case, the clustering engine 150 can cluster the pairwise similarity' scores 145, e.g., based on respective pairwise similarity' scores relative to a common candidate output, and use the clusters to determine the overall similarity score 145.

[0048] For example, the clustering engine 150 can determine the count of the cluster with the largest number of candidate outputs as the similarity' score or calculate a measure of entropy as the similarity score. As another example, in the case that the abstention policy parameter(s) 132 include a similarity threshold value 136, the subsystem 130 can determine a count of how many pairwise similarity scores are greater than the similarity threshold value 136. An example for how a clustering engine 150 can determine the overall similarity score 155 will be described in more detail with respect to FIG. 2.

[0049] The abstention policy subsystem 130 can then compare the overall similarity' score 155 to the abstention threshold value 134. As an example, in the case that the overall similarity score 155 is greater than the abstention threshold value 134, the system 100 can output a candidate output 164 from the language processing network 164. In particular, the subsystem 130 can select a candidate output 164 as the response 160, e.g., from the largest cluster or any of the clusters with pairwise similarity scores relative to a common candidate output, above the similarity threshold value 136.

[0050] As another example in the case that the overall similarity score 155 is less than the abstention threshold value 134, the system 100 can output an abstention response 162 by outputting a response that indicates a lack of knowledge or confidence, e.g., by outputting “I don’t know” or another natural language phrase that indicates that the neural network 120 is not confident in any generated response, or not providing any response for the input prompt110 instead of an incorrect response. In some cases, the abstention response 162 can also be accompanied by a suggestion of where a user can look to determine the answer.

[0051] For example, a counterpart abstention policy calibration system, or another calibration system, can calibrate the abstention policy parameter(s) 132 in accordance with a specified risk tolerance criterion. FIG. IB depicts an example abstention policy calibration system 165 that can calibrate the abstention policy parameter(s) used by the example abstention policy inference system 100 in accordance with a specified risk tolerance criterion a 175.

[0052] In particular, the abstention policy calibration system 165 can obtain a calibration set of example prompt-response pairs 170, e.g., calibration dataset Dn= {(Xi,Yi),...,(Xn,Yn)}. As an example, an input prompt in the calibration dataset 170 can be a request for information, and the corresponding response in the prompt-response pair can be the correct response to the request or an abstention response, e.g., which indicates that it is incorrect for the system to provide a response to the input prompt.

[0053] Generally, the calibration dataset 170 can be much smaller than the training dataset that is used to train the neural network. The system 165 can also obtain the risk tolerance criterion a 175, e.g., by way of an input to the system 165 from a user, or from another system. In particular, the risk tolerance criterion 175 specifies the upper bound on the allowable risk in the response. In some cases, one or more users of the system 165 can configure the system 165 to calibrate respective abstention policies, e.g., based on different risk tolerances.

[0054] As an example, the system 165 can calibrate the abstention policy parameter(s) 132 in accordance with minimizing the rate of abstention while bounding the expected risk of the policy by the risk tolerance criterion 175: min I:i 'v ) . subieei to-tor ' "is the expected rate of abstention under abstention policy a parameterized by parameters y andis the expected loss resulting from the abstention policy. Given the calibration dataset 170, the system 165 can determine the values of the abstention policy parameter(s) 135 in an approximately optimal solution to the equation above.

[0055] In the particular example depicted, the abstention policy parameter(s) 132 includes the abstention policy threshold A and similarity threshold / ?. In this case, the abstention policy is a binary abstention policy. More specifically, the binary abstention policy A(X, / ?, A) indicates whether the system 165 should provide a candidate output or an abstention output as responseto the input prompt, e.g., A(X, ?, A) = 1 denotes that the system 165 should abstain from responding to input prompt X because it is likely that the neural network has generated an output that is a prediction error, e.g., a hallucination.

[0056] More specifically, the abstention policy calibration system 165 can calibrate the abstention policy parameter(s) 132 by updating the parameter(s)132 with black-box neural network access 195. That is, the abstention policy calibration system 165 can calibrate the abstention policy parameter(s) 132 without training or finetuning the neural network, or even having access to any of the underlying neural network parameters. As described in FIG. 1A, the system 165 can calibrate any neural network that is configured to generate multiple candidate outputs. In some cases, the neural network is a pretrained or fine-tuned language processing neural network that can be included in a different system external to the system 165.

[0057] For a binary abstention policy, e.g., the calibration system 165 can perform conformal prediction as the calibration procedure 180 in order to update the parameter(s) 132 that define when the inference system should output an abstention response, instead of an incorrect response. More specifically, conformal prediction refers to associating a prediction with a confidence interval that defines the probability that a candidate output includes the correct output.

[0058] By ensuring the measure of risk of generated responses for the calibration set 170 are guaranteed to not exceed an upper bound on risk provided by the specified risk tolerance criterion a 175, conformal prediction provides a formal theoretical guarantee on the risk that the inference system that deploys the calibrated abstention policy parameter(s) 132 will be able to manage, e.g., by providing a confidence guarantee which holds with high probability' over samples from the same domain as the calibration set 170. with some certainty. More specifically, the specified risk tolerance criterion 175 provides an upper bound on the risk that is used to optimize the abstention policy parameter(s) 132 such that, when used in inference, the abstention policy will, with high probability7, provide responses that do not violate the risk tolerance criterion 175.

[0059] The calibration system 165 can guarantee the risk remains below the risk tolerance criterion 175 with high probability through any appropriate conform prediction approach. One example of such an approach includes techniques described in more detail in ‘‘Conformal Risk Control”, Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tai Schuster. arXiv, 2023. and “Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control”. Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candes. Michael I. Jordan, and Lihua Lei, arXiv, 2022.

[0060] For example, in the case that / 3 is fixed and the system is used to calibrate A, according to a conformal risk control framework, as disclosed by Angelopoulos, the expected risk of lambda can be bound over the calibration data 170:foris the average loss over the calibration dataset 170. The guarantee in expectation can be converted toa guarantee in probability, namely thatv' ■ — ■ for any failure probability 5.This result can be shown to hold with high probability through amplification.

[0061] In the case that the loss is non-monotonic in / ?, the leam-then-test approach, as disclosed by Angelopoulos, can be used to bound the failure probability' 5 with upper confidence bound p, even if the loss is non-monotonic:In this case, the p can be determined by solving a simple optimization problem, e.g., using Empirical Bernstein’s inequality or Bemoulli-relative-entropy inequality.

[0062] The risk tolerance criterion 175 can be specified in any appropriate way, e.g., the calibration system 165 allows for flexibility in defining how the measure of risk is calculated. For example, the measure of risk can be specified by a logic-encoded rule or by using an additional model to predict the measure of risk. In particular, a user of the system 165 can specify the risk tolerance criterion 175 and the method used to determine the measure of risk, e.g., by configuring the model to use a particular rule or an additional risk model. In some cases, the user can provide a risk machine learning model (not depicted) that can be configured to process the input prompts from the calibration set 170 and predict an associated risk tolerance for the calibration set 170.

[0063] In the case that a risk machine learning model is used to predict the risk tolerance criterion 175, the risk machine learning model can have any appropriate machine learning architecture, e.g., a random forest, a support vector machine, a decision tree, linear regression model, or a neural network, that can be configured to process an input prompt to generate a measure of risk. For instance, the risk machine learning model can be a neural network with any appropriate number of neural network layers (e.g., 1 layer, 5 layers, or 10 layers) of any appropriate type (e.g., fully -connected layers, attention layers, convolutional layers, etc.)connected in any appropriate configuration (e.g., as a linear sequence of layers, or as a directed graph of layers).

[0064] In particular, the calibration system 165 can perform a calibration procedure, e.g., the calibration procedure 180. More specifically, the system 165 can perform conformal prediction as the calibration procedure 180 to quantify the response uncertainty' and determine whether the system 100 should output an abstention response 162 based on the abstention policy 190 implemented at a current calibration iteration and can update the abstention policy parameter(s) 132, e.g., in accordance with minimizing a loss function with respect to the specified risk tolerance criterion a 175 at the calibration iteration.

[0065] In particular, the calibration system 165 can augment each prompt-response pair (X, Y) in the calibration dataset Dnby sampling k samples from the candidate outputs generated from the neural network, e.g., by processing the query X using the black-box neural network access 195.

[0066] As described with respect to FIG. 1A, the system 165 can determine a measure of pairwise similarity between each pair of the candidate outputs. For example, the system 165 can use a text similarity function s(x, Y, Y’) to measure the similarity between text responsesY and Y’. In particular, Y and Y’ can be considered to contain the same content if s(x, Y, Y’) > / ?. For example, for X='What is the population of London', and Y='8,980,000' and Y'='8,900,000', a sensible similarity function can assign a large number to s(X; Y,Y').

[0067] In some cases, the similarity function is contextual, meaning that the determined measure of pairwise similarity can depend on the context prompt X. In this case, the responsesY and Y' above could be considered as dissimilar in the case that Y and Y’ were generated from two different prompts.

[0068] As described with respect to FIG. 1A, the system 165 can then cluster the candidate outputs based on the pairwise similarity measures to determine the overall similarity score. The system 165 can determine the overall similarity score that indicates how similar the candidate responses are to each other as a proxy measure for confidence in the response.

[0069] For example, the system 165 can determine the size of the largest cluster as the overall score. As another example, the system 165 can determine whether a count of the candidate responses is above a similarity threshold level. As yet another example, the system 165 can define the overall similarity score to be a measure of entropy of the clustering, e.g., a crossentropy. In this case, the system 165 can define a probability’ distribution over the candidate responses in each of the clusters based on the cluster size and can calculate, e.g.. by assigninga probability mass to each cluster that is proportional to the cluster size, the entropy of the distribution.

[0070] At each of a number of calibration iterations, the system 165 can then update the parameters 132 based on an objective function. In particular, the system 165 can update the values of the abstention policy parameter(s) 132 by calculating and backpropagating gradients of the loss function, e.g., using the update rule of any appropriate gradient descent optimization algorithm, e.g., RMSprop or Adam.

[0071] More specifically, the system 165 can determine a loss function that counts the number of instances where the calibration system 165 provided an output in accordance with the values of the abstention policy parameters at the current calibration iteration, but should have abstained, e.g.,where 1 is an indicator function, e.g., that is, where the loss is 1 in the case that the system 165 provided an output, but should have abstained, and m(X, A) is the response provided by the system 165.

[0072] In this case, the system 165 can then define the average loss as:and the conditional risk as the expected loss over the calibration set, test input prompts Xi, and the corresponding correct responses Yi.

[0073] In this case, the system 165 can update and A in accordance with minimizing the loss function, e.g., such that R( / ? , Aa . For example, the system 165 can implement a conformal risk control approach to determine A as the minimum value such thatwhich provides a formal guarantee that R(A, ft) is less than the risk tolerance for a calibration set of n prompt-response pairs, as is described more in Abbasi-Yadkori, Y., et. al., "Mitigating LLM Hallucinations via Conformal Abstention” (arXiv:2405.01563vl).

[0074] In some cases, the system 165 can additionally include a second term in the loss function to penalize the number of instances where the policy abstains, where the system 165 could have responded. As an example, in this case, the total loss can be:where C is a positive constant that weights the relative importance between the two loss terms. As described above, the system 165 can update and A in accordance with minimizing the loss function, e.g., such that

[0075] In some cases, the system, e.g., system 165 in calibration or system 100 in inference, can be configured to generate the candidate outputs 125 in parallel. While described below with respect to the calibration process and the abstention policy calibration system 165. system 165, the abstention policy inference system 100, or both can leverage parallelized compute to enhance the response latency and reduce the bandwidth necessary to generate the response.

[0076] In particular, the system 165 can process multiple input prompts, e.g., from the calibration set 170, using the neural network to generate a number of candidate outputs. Generating a number of candidate outputs for each input prompt may be a relatively computationally intensive task. In some cases, each candidate output may require its own inference step (e.g., implementing an instance of the neural network to generate an output). In this case, the compute required to generate multiple candidate outputs scales based on the number of candidate outputs generated.

[0077] To enhance the response latency and reduce the bandwidth, the generation of the candidate outputs may be parallelized. For example, a first processor (e.g., neural network processing hardware) may receive an input prompt and generate a first candidate output for the input prompt and a second processor may receive the input prompt and generate a second candidate output for the input prompt in parallel. As such, the process of generating multiple candidate outputs can be distributed and hence can be performed over a reduced time period compared to generating each candidate output sequentially.

[0078] For example, the neural network may be associated with (e.g., stored on, run on, located on) a first computing system. The number of candidate outputs may be generated at the first computing system. The first computing system may comprise a server or other computing system with relatively high computational resources available. The first computing system may comprise a parallel computing system comprising at least two sets of neural net ork processing hardware configured to be operable in parallel. Each of the number of candidate outputs may be generated using an instance of the neural network implemented on a correspondingprocessor of the first computing system. Generating the number of candidate outputs may include processing the input prompt using a first instance of the neural network on a first of the at least two sets of neural network processing hardware to generate a first candidate output and processing the input prompt using a second instance of the neural network to generate a second candidate output. The first computing system may receive the input from a client system, e.g., a user device, for example as a query’.

[0079] In an example embodiment, the similarity neural network may be associated with (e.g.. stored on, run on, located on) a second computing system that is different to the first computing system. The step of determining a similarity' score may be performed at the second computing system. The second computing system may be a client system, e.g., a user device. Determining whether the similarity’ score satisfies an abstention threshold value may additionally be performed at the second computing system or may be performed at a third computing system, which may be a client system.

[0080] In another embodiment, the similarity' neural network may be associated with (e.g., stored on, run on, located on) the first computing system. As an example, in this case, the similarity neural network can be run on a different processor (e.g., neural network processing hardware) compared to the processor(s) on which the neural network is run.

[0081] The computing system and / or processor(s) on which the neural network is run may be larger than the computing system and / or processor(s) on which the similarity neural network is run. Larger may be interpreted as, for example, having a larger memory (e.g.. for storing neural network parameters) and / or more computational power.

[0082] In general, it may be undesirable to add additional computational burden to the processor generating the plurality' of candidate outputs (e.g., at the first computer or processor). Therefore, the steps of running the abstention policy (determining similarity scores and determining whether the similarity score satisfies an abstention threshold value) may be decoupled from the generation of responses so as to distribute this computational burden. Designing the method in this way enables efficient leveraging of the computational resources of the available computing systems to improve the reliability and / or accuracy of generated responses. In particular, the plurality of candidate outputs may be generated in parallel and therefore in a time efficient manner, and the additional computational requirement of running the abstention policy can be run externally.

[0083] This yields additional benefits, for example the abstention threshold and associated parameters (e.g., risk tolerance, penalty constants, loss functions, greediness, desired abstention rate, any parameter associated with the similarity neural network) can be updatedwithout any update to the neural network. As such, the abstention policy can be updated without any requirement to retrain the neural network. Decoupling the updating of the abstention policy from the neural network therefore reduces the need for training and retraining of the neural network, as well as enabling a more flexible and individual approach to risk for each user of the neural network.

[0084] For example, a first user and second user may each query the neural network with the same prompt but each have a different risk tolerance for answers. The first user may wish for the abstention policy to be stricter, abstaining more often so as to avoid incorrect answers with a higher certainty. The second user may wish for a more relaxed abstention policy which rarely abstains, and the second user therefore wishes to tolerate a higher risk regarding the correctness of answers. Decoupling the abstention policy from the neural network enables a neural network to provide answers to multiple users, with different risk appetites, without requiring any retraining for each user. Furthermore, the decoupling enables the abstention policy to be updated based on up-to-date data, leading to more accurate abstention selection, without requiring the neural netw ork to be trained.

[0085] FIG. 2 illustrates how the example abstention policy inference system of FIG. 1A can determine an overall similarity score relative to a common candidate output, that can be compared to an abstention threshold value to determine whether to provide an output as response to an input prompt.

[0086] The system can sample and compare candidate outputs of a neural network, e.g., k candidate outputs, that were generated for the same input prompt. More specifically, the clustering engine 150 can determine the overall similarity' score by clustering the pairwise similarity scores calculated between each of the candidate outputs.

[0087] In the particular example depicted, the clustering engine can cluster the candidate outputs by pairwise similarity score relative to a particular common candidate output, e.g., cluster A 210 represents candidate outputs with a determined pairwise similarity score greater than five for the common candidate output, cluster B 220 represents candidate outputs with a determined pairwise similarity score greater than six for the common candidate output, and cluster C 230 represents candidate outputs with a determined pairwise similarity score greater than seven for the common candidate output. Likewise, the clustering engine 150 can generate the clusters for each of the other candidate outputs.

[0088] In this case, the implemented abstention policy includes a similarity threshold value 240, which is has been calibrated to be six. e.g., such that the clustering engine 150 is restricted to determining the overall similarity score using the clusters that have pairwise similarity scoresrelative to the common candidate output which are greater than the similarity threshold value 240 of six. In particular, the clustering engine 150 can determine the overall similarity score using cluster B 220 and cluster C 230.

[0089] As an example, the clustering engine 150 can determine the overall similarity score as the total count of the largest cluster with pairwise similarity scores greater than the similarity threshold value. In this case, the overall similarity score is eight, since that is the size of cluster B 220, which includes more candidate outputs than cluster C 230.

[0090] As another example, the clustering engine 300 can determine the overall similarity score as the total count of all clusters with pairwise similarity scores greater than the similarity7threshold value. In this case, the overall similarity score is 13, since that is the aggregate size of cluster B 220 and cluster C 230.

[0091] As described with respect to FIG. 1A, the abstention policy subsystem can then compare the overall similarity score with the abstention threshold value to determine whether to provide an output as response to an input prompt. In particular, the abstention threshold value that is used to compare the overall similarity score can have been calibrated based on how the overall similarity score is calculated in order to provide a fair comparison. For example, in the first example described above, the abstention threshold value can have been calibrated as the total count of the largest cluster that is greater than the similarity threshold 240. As another example, in the second example described above, the abstention threshold value can have been calibrated as the total count of all clusters with pairwise similarity7scores greater than the similarity threshold value 240.

[0092] In the case that the overall similarity7score exceeds the abstention threshold value, the abstention policy subsystem can select one of the candidate outputs, e.g., from cluster B 220, in the case that the overall similarity score is determined as the total count of the largest cluster with pairwise similarity scores greater than the similarity threshold value 240, or from either cluster B 220 or cluster C, in the case that the overall similarity score is determined as the total count of all clusters with pairw ise similarity scores greater than the similarity threshold value 240.

[0093] FIG. 3 is a flow diagram of an example process for determining whether to provide one or more candidate outputs of a neural network based on a calibrated abstention policy. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, an abstention policy inference system, e.g., the abstention policy inference system 100 of FIG. 1 A, appropriately programmed in accordance with this specification, can perform the process 300.

[0094] The system can receive an input prompt (step 310), and can process the input prompt using a neural network to generate a number of candidate outputs for the input prompt (step 320). In some cases, the system can generate the number of candidate outputs using the first neural network and can identify a first number of the initial candidate outputs as the number of candidate outputs, e.g., by selecting a subset of candidate outputs based on any appropriate criteria. For example, the neural network can be a language processing neural network. In this case, the input prompt can include a directive instruction, e.g.. a question, statement, code snippet, or example. As a further example, the neural network can be a trained visual language model that includes an image encoder neural network and a text encoder. In this case, the input prompt, the generated candidate outputs for the input prompt, or both can include an image.

[0095] As an example, the system can generate the number of candidate outputs using a parallel computing system, e.g., that includes at least three sets of neural network processing hardware that is coupled to be operable in parallel. In this case, the system can implement a first instance of the first neural network on a first of the sets of neural network processing hardware, and can implement a second instance of the first neural network on a respective second of the sets of neural network processing hardware. The system can then sample from the first instance and the second instance of the first neural network to generate one or more candidate outputs in parallel.

[0096] The system can determine a similarity score that characterizes a similarity of each of the candidate outputs to each other candidate output (step 330). In particular, the system can generate, using a second neural network, a respective pairwise similarity score for each of a number of pairs of candidate outputs, e.g., for all of the outputs or a subset of the outputs. In some cases, the second neural network can have fewer parameters than the first neural network, e.g., the second neural network can be a lightweight model with a smaller, less complicated, or both architecture relative to the first neural network. In the case that the system generates the candidate outputs using a parallel computing system, the system can implement a first instance of the second neural network on, e.g., a third of the sets of neural network processing hardware, and use the first instance of the second neural network to generate the respective pairwise similarity score for each of the number of pairs of candidate outputs.

[0097] For example, the system can process an input for each pair of candidate outputs that includes an instruction to generate the pairwise similarity score and the pair of candidate outputs using the second neural network to generate the respective pairwise similarity score for the pair of candidate outputs. As another example, the system can process an input that includes an instruction to generate the respective pairwise similarity scores and the number of pairs ofcandidate outputs using the second neural network to generate, e.g., in a single processing iteration, the respective pairwise similarity’ scores for each of the number of pairs.

[0098] The system can then generate the similarity score using the respective pairwise similarity scores. For example, the similarity score can be a contextual similarity' score that measures a contextual similarity between the candidate outputs in accordance with a context provided by the input prompt, e.g., by conditioning the similarity between the candidate outputs based on the input prompt. As an example, the system can generate the similarity score based on a similarity threshold value. An example for calibrating a similarity threshold value as part of an abstention policy will be described in more detail with respect to FIG. 4.

[0099] In some cases, the system can generate the similarity score by determining a set of clusters of the candidate outputs based on respective pairwise similarity scores relative to a common candidate output, e.g., where each cluster includes only candidate outputs with pairwise similarity scores for the common candidate output greater than a similarity threshold value. As an example, in this case, the system can determine the similarity’ score based on a size of a largest cluster. As another example, in this case, the system can determine the similarity score based on an entropy of one or more of the clusters, e.g., by assigning a probability mass to each cluster that is proportional to the cluster size and calculating the entropy of the distribution. In other cases, the system can generate the similarity score from the respective pairwise similarity scores by determining a count of the pairwise similarity scores for each candidate output that are greater than a similarity threshold value. In this case the system can identify the largest count as the similarity score.

[0100] The system can determine whether the similarity score satisfies a criterion based on an abstention threshold value (step 340). In particular, the system can compare the similarity score to the abstention threshold value. More specifically, the abstention threshold value can have been determined by calibrating an abstention policy on a calibration set of prompt-response pairs, each having a corresponding risk value indicative of an incorrectness, e.g., a ground truth label of whether the abstention response should have been provided, of the response in the pair. An example for calibrating an abstention value as part of an abstention policy will be described in more detail with respect to FIG. 4.

[0101] In response to determining that the similarity score satisfies the criterion, e.g., is greater than the abstention threshold value, the system can provide one or more of the candidate outputs as a generated response to the input prompt (step 350). As an example, in the case that the input prompt includes image data and text data, the generated response can include information representative of the image data and the text data. As another example, in the case that the inputprompt includes first image data and text data, the generated response can include second image data representative of the first image data processed according to contextual information contained in the text data. In contrast, in response to determining that the similarity score does not satisfy the criterion, e.g., is less than the abstention threshold value, the system can provide an abstention response that is not generated using any of the candidate outputs or the system can abstain from responding, e.g., by not providing any response to the input prompt.

[0102] FIG. 4 is a flow diagram of an example process for calibrating an abstention policy for a neural network. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, an abstention policy calibration system appropriately programmed in accordance with this specification, e.g., the abstention policy calibration system 165 of FIG. 1. can perform the process 400 to generate the calibrated abstention policy used by the abstention policy inference system 100 of FIG. 1 A.

[0103] The system can obtain a set of prompt-response pairs, each having a corresponding risk value indicative of an incorrectness of the response in the pair (step 410). For example, the corresponding risk value for each prompt-response pair can have been determined using a risk prediction model or rule. In some cases, the system can be configured to receive the risk prediction model or rule based on a user input to the system.

[0104] The system can process one or more prompts of the set of prompt-response pairs using the neural network to generate a predicted response in accordance with a current abstention threshold value (step 420). An example for how the system can process the one or more prompts to generate a predicted response is described in more detail with respect to FIG. 3.

[0105] The system can determine the count of instances when the abstention policy should have abstained from responding and did not (step 430). In particular, the count of instances can specify a generated response for a first prompt of a first prompt-response pair in the case that the corresponding risk value for the first prompt-response pair indicates an incorrect response. In some cases, the system can additionally determine the count of instances when the abstention policy should not have abstained and did. In this case, the count of instances can specify an abstention response for a first prompt of a prompt-response pair in the case that the corresponding nsk value for the first prompt-response pair indicates a correct response.

[0106] The system can adjust the abstention threshold value in accordance with minimizing the count of instances when the abstention policy should have abstained from responding and did not while ensuring a measure of risk under the abstention policy satisfies a target risk tolerance criterion (step 440). In particular, the abstention threshold value can be a minimum abstention threshold value, e.g., if the similarity score described in FIG. 3 is less than theabstention threshold value, then the system can provide an abstention response. In some cases, the abstention policy further includes a similarity’ threshold value, e.g., that is used to determine the similarity’ score as described in FIG. 3. In this case, the system can adjust the similarity threshold value, in accordance yvith the optimization described above as part of the calibration process.

[0107] In particular, the system can use a loss function indicative of the count of instances when the abstention policy should have abstained from responding and did not. In some cases, the system can additionally use a second loss function indicative of the count of instances when the abstention policy should not have abstained from responding and did. For example, the measure of risk can be determined in accordance yvith the risk prediction model or rule used to determine the corresponding risk value for each prompt. In some cases, the measure of risk under the abstention policy is the expected value of the loss function for the one or more predicted responses.

[0108] The methods described herein can be executed (for example as a computer program) on one or more computers that are located at one site or distributed across multiple sites and interconnected by a data communication network. A computer may be referred to as a computing system. Computers suitable for the execution of the methods described in this specification can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The elements of a computer are a central processing unit for performing or executing instructions and one or more memory’ devices for storing instructions and data. The central processing unit and the memory- can be supplemented by, or incorporated in. special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g.. a universal serial bus (USB) flash drive, to name just a feyv.

[0109] The methods described herein can be executed (for example as a computer program) on a parallel computing system. A parallel computing system may comprise one or more processors configured to be operable in parallel. The processors may comprise hardware, for example neural network processing hardyvare. In some arrangements, the processors may beassociated with the same computer. In other arrangements, the processors may be associated with multiple computers. Embodiments of the methods described herein are specifically adapted for operation on parallel computing systems. Parallelization can help increase computation speed, for example wall-clock time of computations. Parallelization can also be used to aid in pipelining tasks when multiple tasks are distributed across different processing systems.

[0110] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0111] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device. A user device may be referred to as a client device, or more simply a device.

[0112] Any computing system has a finite amount of computing resources. Computing resources may include, for example, computational power, computation speed, memory capacity, electrical power, bandwidth, queue length, circuit area usage, cooling requirements, or any other resource. A back-end component, for example a server, is typically configured to have a relatively large amount of computing resource compared to a client component, for example a user device. For example, a server may have a larger memory portion, higher compute etc.

[0113] The task that the first neural network is used to perform may relate to multimodal data. The first neural network may be considered a multimodal machine leaning model. In thediscussion below, the machine learning model may be referred to simply as “the model”. As used herein a “modality’” refers to a type of data, and thus a multimodal machine learning model is one that can process multiple different types of data. The model may receive, as an input, multimodal data. The model may then provide, as an output, information relating to (e.g., derived from) the multimodal data. The input may be a query, for example a text-based question with an associated data item (e.g., an image). The output may be representative of a response to the query. The output may be in the form of text (e.g., a transcription), an audio utterance (e.g., a simulated spoken utterance), or any other means of conveying information. The multimodal data may include one or more of: a text data item, image data item, audio data item (representing values of a digitized audio waveform, e.g., a time sequence of waveformrepresenting elements), sensor data (representing values of a digitized sensor waveform). An image data item may comprise image data representing a still or moving image, i.e., an image or video, e.g., an image or video that has been captured using a camera. Elements of the image data may comprise monochrome or color pixels of the image or video. As defined herein an “image” includes a point cloud, e.g., from a LIDAR system, and a “pixel” includes a point of the point cloud. Similarly “video” includes a time sequence of point clouds. Objects in the image or video may comprise objects, e.g., physical objects, represented by the image or video. [H4] Text may be received, e.g., as a series of encoded characters, e.g., UTF-8 encoded characters; such “characters” can include Chinese and other similar characters, as well as logograms, syllabograms and the like. The multimodal machine learning model can include a text encoder that processes a sequence of text to represent the text as a series of text tokens from a vocabulary of text tokens, e.g., that each represent words, wordpieces or characters in a natural or computer language. The computer language may be any formal language used to communicate with a computer, e.g., a markup language, or a command or configuration language, or a data exchange language such as JSON, or a programming language

[0115] The model output may comprise any form of output appropriate to the machine learning task performed by the multimodal machine learning model. For example the model output may comprise text in a natural or computer language that defines a result of the task, e.g.. for tasks such as image captioning, visual question answering, or object detection or instance segmentation. Also or instead the model output may comprise data defining an image, video or audio object, e.g., in a generative task; or the model output may comprise non-textual action selection data for selecting an action to be performed by an agent controlled by the model. As another example the model output may also or instead define an intermediate step to be performed during the task, e.g., a call to a software API for a software tool that is used whenperforming the task; the multimodal input may then receive an output from the software tool that is used to generate a final model output that performs the task. A few particular examples of model output are given later.

[0116] Such a multimodal model (e.g., a VLM) can be trained using very large (but possibly noisy) datasets in which text is paired with an image and / or with one or more other ty pes of data, e.g., audio data, or data relating to the operation of an agent acting in an environment to perform a variety of tasks. Such a model is can be trained, e.g., using self-supervised learning. The pairing can often be imperfect, and the training dataset can, but may not, include any actual examples of a particular task to be performed, but nonetheless an ability' to perform a particular task can emerge. There are many examples of suitable, publically available training datasets. A trained VLM may be referred to as a pre-trained VLM.

[0117] Some example multimodal machine learning models with which the techniques described herein may be used include: Flamingo (Alayrac et al. arXiv:2204. 14198); ALIGN (Jia et al., arXiv:2102.05918); PaLI (Chen et al. arXiv:2209.06794) and PaLI-X (Chen et al. arXiv:2305.18565); and Gemini ("‘Gemini: A Family of Highly Capable Multimodal Models, Gemini Team. Google”). These references also include indications of training datasets that may be used to train the respective models.[H8] In general, performing an image processing task using a multimodal neural network such as a visual language model (VLM). that includes an (a trained) image encoder neural network, can involve providing an image to the image encoder neural network to generate a representation of the image, in particular of features of the image. The representation of the image is then processed to perform the image processing task. In general techniques for processing image representations to perform a wide range of image processing tasks are well known. In general a video processing task can be performed by processing the representations of multiple images representing a sequence video frames.[H9] An example involves generating an output that requires reasoning, e.g., spatio-temporal reasoning, to respond to a natural language query input, e g., relating to a moving image (video). For example such a query may require predictive reasoning (“what will happen next”), counterfactual reasoning (“what would happen in a different circumstance”), explanatory reasoning (“why did something happen”), or causal reasoning generally. For example the image representation can be used to detect objects in the video frames and provide information relating to the detected objects in response to a query. The query may comprise, for example, a request for a prediction of a future event or state relating to one or more of the objects (e.g., “will objects X and Y collide?”), or a request for conditional or counterfactual informationrelating to one or more of the objects (e.g., “what event would [not] happen if object X is modified, moved or absent?"’), or a request for analysis of the video frames to determine a property or characteristic of one or more of the objects (e.g., “how many objects of type Z are moving?”). The output may, for example, be in the form of a yes / no answer, or may define a probability distribution over a set of possible answers; or the response may define the location of an object. Such systems can be used to predict whether or not two objects will collide, or how this may be avoided. The output may be used e.g.. to provide a warning and / or to control motion of one or more of the objects

[0120] Example tasks for multimodal models such as a Vision-Language Model (VLM) are described below.

[0121] In general a multimodal machine learning model can be trained to perform any sort of machine learning task or tasks. After the multimodal machine learning model has been trained it can be deployed for use in performing the machine learning task(s). For instance, the machine learning model can be deployed in an environment that enables users to provide requests for the machine learning model to process specified multimodal inputs to generate corresponding model outputs. Users can provide the requests, e.g., by way of a user interface or through an application programming interface (API). The requests can be transmitted from a user device (e.g., over a data communication network, e.g., the internet) to one or more computers implementing the machine learning model, e.g., in a data center. The machine learning model can process multimodal inputs specified by user requests to generate corresponding model outputs, and then transmit the model outputs to user devices (e.g., over a data communication network).

[0122] In some implementations, after training, a particular task that is to be performed by the multimodal machine learning model can be described by part or all of the sequence of text in the multimodal input to the model. For example in a multimodal input that includes an image such a prompt might specify “Generate a caption”, “Generate a description”, “Answer the following question: [about the image or video]”, or “Detect a person”. Where the model is used for an agent control task a prompt may define “Take the knife out of the drawer”, or “Q: What action should the robot take to take the knife out of the drawer?”. Also or instead such a prompt may give one or more examples of a task to be performed. A multimodal machine learning model can be trained on multiple natural and / or computer languages and the prompt may then specify a language to use.

[0123] A few examples of some machine learning tasks that can be performed by a model trained as described herein follow. The tasks described below may be tasks that require spatialawareness or other context from the image or video. For example, a prompt may ask “What is the object in the top left comer?’;

[0124] As one example the task may comprise an object or action detection task. A taskspecific training data item may comprise an image or video containing one or more objects or actions, and a sequence of text. The sequence of text may describe or otherwise label the object(s) or action(s) and may include text giving bounding box coordinates for the object(s) or action(s). After training, when the model is used in inference, the model output may comprise or represent text that describes or otherwise labels detected object(s) or action(s) in the image input, and may include bounding-box coordinates for the detected object(s) or action(s). e.g., "10 20 90 100 cat 20 30 100 100 dog”.

[0125] As another example the task may comprise a classification task, e.g.. an object or action classification task. A task-specific training data item may comprise an image or video item containing one or more objects or actions and a sequence of text. The sequence of text may describe or otherwise classify the object(s) or action(s). After training, when the model is used in inference, the model output may comprise data, e.g., text, that classifies the object(s) or action(s) in the image input into one of a plurality of classes.

[0126] As another example the task may comprise an image or video describing task, e.g., a captioning task (which, as used here, includes an audio description task to explain what is happening in a video). A task-specific training data item may comprise an image or video and a sequence of text describing the image or video. After training, when the model is used in inference, the model output may comprise data, e.g., text, describing the image or video. For example the model output may provide a caption or description or it may count objects in the image or video, or it may provide some other form of description.

[0127] As another example the task may comprise an image or video question-answering task. A task-specific training data item may comprise an image or video and a sequence of text that describes the image or video. After training, when the model is used in inference, the model output may comprise data, e.g., text, that answers a question about the second modality input specified in a prompt sequence of text, e g., as described above. This may be used, e.g., to answer questions about visual plots and charts or about sounds.

[0128] As another example the task may comprise a character or word recognition task, e.g., an OCR (optical character recognition) task. A task-specific training data item may comprise an image or video and a sequence of text that includes text that is depicted in the image or video, or that is represented as speech in the audio item. After training, when the model is usedin inference, the model output may comprise text that represents characters or words in the second modality input, e.g., in a natural language.

[0129] As another example the task may comprise a still or moving image generation task; Google DeepMind Imagen 3 is an example of a system that can generate an image output. As another example For example an image such as a plot or chart may be decoded from one or more (language) tokens generated by the system. A training data item for such a system may comprise an image or video and a sequence of text that describes the image or video. After training, when the model is used in inference, the model output may comprise data for an image or video, e.g., image data defining values for pixels of a still or moving image, and the sequence of text in the multimodal input to the model may describe or characterize the image or video to be generated.

[0130] As another example the task may comprise a computer language text generation task. A task-specific training data item may comprise an image or video and a sequence of text in a computer language for generating the image or video. After training, when the model is used in inference, the model output may comprise text in the or another computer language for generating or rendering an image or video, e.g.. a web page, plot, or chart.

[0131] As one example, the model receives input in the form of an image data item and a text data item. The text data item may be a question about the image data item. The image data item may, for example, contain an image of one or more objects, and each object may belong to a particular class. The text data item may be a question regarding the one or more objects, the class of said objects, the position of an object or feature within the image (e.g., a pixel location within the image) or any other characteristic of the objects or of the image.

[0132] In an example using multimodal data, the input comprises an image of a person wearing a cap and driving a car. The input also comprises the question "‘what is the person wearing?'; The model can provide an answer to the question. In the example, a correct answer would be “a cap”. Other correct answers in the example could include “headwear”, “a hat” or “a baseball cap”. An incorrect answer could be “a car”. The correct answ ers relate to characteristics of the image (e.g., class of objects in the image) but, importantly, are contextualized in light of the question. As such, the correct answer relates to objects in a class relevant to ‘wearable clothing’ and disregards answers relevant to objects in unrelated classes (e.g., the car). Applying the methods described herein to this example, answers relating to hat, headwear, caps etc. would have a large similarity score and would therefore be more likely to be returned as an answer. Applying the methods described herein to this example, answers relating to the car would likelyhave a low similarity with the majority of other answers and so the model would likely abstain from providing this as an answer.

[0133] In another example using multimodal data, the input comprises a video (e.g., multiple images forming frames of a moving model) of a car driving near a pedestrian. The input also comprises the question “will the car hit the pedestrian?”. The model provides, as an output, a prediction of whether the car will or will not hit the pedestrian. The prediction may be made with reference to a physical model, explicit or implicit comparison to similar scenanos, or any other information that the model can understand from the video. Such scenario predictions are of particular use for control of autonomous vehicles or driver assistance features. It can be imagined that in vehicle control applications such as these, the risk tolerance may be lower and so it may be beneficial to set the abstention threshold and similarity threshold values to ensure a lower risk tolerance.

[0134] In another example, the input to the model may comprise a text data item, an image data item, or both. The input may be a query' related to the selection of or generation of an image with particular characteristics. For example, the query may be a text-based request for the selection or generation of an image of an object of a particular class. The query' may include an image of an object of a particular class and a text-based request for the selection or generation of an image of an object having the same class as the object in the image in the query . The model provides, as an output, an image responsive to the query, e.g., an image of an object of the particular class. The output image may be selected from a set of candidate images. The candidate images may be images generated by the model. Using the models described herein, the model is more likely to provide an image that accurately represents the type of image requested and is more likely to abstain rather than provide an incorrect image. A generated image of an object can be considered a simulation of said object. The object may be a real-world object, for example a real-world object belonging to a particular real-world class.

[0135] The model may be trained on training images, for example images which represent real- world objects. When a model that is trained to generate images is provided with a task to generate an image, it processes an input to select values (e.g., RGB, intensity, gamma) for pixels of an image. The pixel values are selected and arranged such that the resultant image, formed by the pixels when displayed, represents an object. The object represented is related to the information contained within the query'. The generated image may share characteristics with (e.g., pixel values and arrangements) with training images upon which it was trained. For example, a generated image of a cat will share some characteristics (e.g.. pixel values) with training data images depicting cats.

[0136] In another example of a computer language text generation task a task-specific training data item may comprise an image or video and a sequence of text in a computer language for performing a task in relation to the image or video, e.g., a data processing task that involves analyzing the content of the image or video to provide a result of the analysis or, e.g., a search to search for information relating to the content of the image or video. The computer language in the model output may comprise computer language for invoking a function or calling one or more external APIs. Merely as one example, such an output may be formatted as a JSON object. As previously, the sequence of text in the multimodal input may define the task to be performed and the second modality input may comprise, e.g., an image or video in relation to which the task is to be performed, e.g., a task that involves manipulation of particular types of data that may benefit from access to an API such as mathematical data, date / time related data, scientific data, recent data that may post-date training of the model (that may be accessed by a search function or API), and so forth. After training, when the model is used in inference, the model output may comprise text in the or another computer language for performing a task, e.g., as described above, in relation to an image or video in the second modality input. The method may then include using the text in the computer language to perform the task.

[0137] In general where the model output comprises text this may be provided as speech representing the text.

[0138] In some implementations the machine learning task comprises an agent control task in which the agent interacts with an environment to perform the agent control task. In these implementations the multimodal input includes an observation characterizing the environment. For example the multimodal input can include a sequence of text that defines the task to be performed by the agent and the image can represent an observation of the environment, e.g., captured by a camera or other imaging device from a real-world environment. A task-specific training data item may comprise a sequence of text representing one or more actions of the agent, and an image observation of the environment. After training, when the model is used in inference, the model output comprises an action selection output, e.g., including text, that is used to select one or more actions to be performed by the agent in the environment in response to the observation. As an illustration the model output 122 may define an action as text such asC'A: 132 114 128 5 25 156’’, that can be converted into a control signal for a mechanical agent, such as a robot, e.g., "AT = [0.1, —0.2,0] AR = [10°, 25°, —7°]” . As another example the action selection output may also or instead define one or more low-level skills, e.g.. from a vocabulary of previously leamt skills. As before, the sequence of text in the multimodal inputto the model may describe the task to be performed, e.g., "What action should the robot take to [perform task]’;

[0139] In some agent control implementations, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real- world environment to perform the task. For example, the agent may be a robot or other mechanical agent interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment. In these implementations, the observations may include, e.g.. one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment. The actions may define control signals to control the robot or other mechanical agent, e.g., positions, torques, or other control signals for the parts of the mechanical agent, or higher-level control commands.

[0140] In some agent control implementations the agent may be a human agent and the environment may be a real-world environment. For example the agent can be a human user of a digital assistant such as a smart speaker, smart display, or some other device that is used to instruct the user to perform actions. The task may be any real-world task that the user wishes to perform. The observations may be obtained from an observation capture subsystem, e.g.. a monitoring system such as a video camera or sound capture system, to capture visual observations of the user performing the task. The actions may comprise instructions in the form of, e.g., text, image, video, or audio data such as speech, that guide the user in performing the task.

[0141] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0142] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry’, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD). a storage medium, a random or serial access memory’ device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0143] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all ty pes of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0144] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiledor interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.

[0145] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is A pically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of Al and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0146] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in Al and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relyingsolely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0147] Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the Al model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high- performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.

[0148] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory' devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence.

[0149] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods caninclude acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0150] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0151] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middlew are serv er or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app. through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0152] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of netw ork, such as a local area netw ork (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send dataor feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0153] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0154] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0155] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0156] What is claimed is:

Claims

CLAIMS1. A computer-implemented method comprising: receiving an input prompt; processing the input prompt using a neural network to generate a plurality of candidate outputs for the input prompt; determining a similarity score that characterizes a similarity of each of the candidate outputs to each other candidate output; determining whether the similarity score satisfies a criterion based on an abstention threshold value; and in response to determining that the similarity score satisfies the criterion, providing one or more of the candidate outputs as a generated response to the input prompt.

2. The method of claim 1 , further comprising, in response to determining that the similarity score does not satisfy the criterion, providing an abstention response that is not generated using any of the candidate outputs; or not providing any response to the input prompt.

3. The method of claim 1 or 2, wherein determining the similarity score that measures the similarity between the candidate outputs comprises: generating, using a second neural network, a respective pairwise similarity score for each of a plurality of pairs of candidate outputs; and generating the similarity score using respective pairwise similarity scores.

4. The method of claim 3, wherein generating, using the second neural network, a respective pairwise similarity score for each of a plurality of pairs of candidate outputs comprises, for each pair of candidate outputs: processing an input comprising an instruction to generate the pairw ise similarity score and the pair of candidate outputs using the second neural netw ork to generate the respective pairwise similarity score for the pair of candidate outputs.

5. The method of claim 3, wherein generating, using the second neural network, a respective pairwise similarity7score for each of a plurality of pairs of candidate outputs comprises:processing an input comprising an instruction to generate the respective pairwise similarity scores and the plurality of pairs of candidate outputs using the second neural network to generate the respective pairwise similarity scores for the plurality of pairs of candidate outputs.

6. The method of claim 4 or 5, wherein generating the similarity score from the respective pairwise similarity scores comprises: determining a set of clusters of the candidate outputs based on respective pairwise similarity scores relative to a common candidate output.

7. The method of claim 6, wherein each cluster includes only candidate outputs w ith pairwise similarity scores greater than a similarity threshold value for the common candidate output.

8. The method of claim 6 or 7, further comprising: determining the similarity score based on a size of a largest of the clusters; or determining the similarity score based on an entropy of one or more of the clusters.

9. The method of claim 4 or 5, wherein generating the similarity score from the respective pairwise similarity scores comprises determining a largest value of a count that, for each of the candidate outputs, counts how' many of the pairw ise similarity scores for the candidate output are greater than a similarity threshold value.

10. The method of any one of claims 1-9, wherein the similarity' score comprises a contextual similarity' score that measures a contextual similarity between the candidate outputs in accordance with a context provided by the input prompt.

11. The method of any one of claims 1-10, wherein processing the input prompt using a neural netw ork to generate a plurality of candidate outputs for the input prompt further comprises: processing the input prompt using the neural network to generate a plurality of initial candidate outputs; andidentifying a first number of the initial candidate outputs as the plurality of candidate outputs.

12. The method of any preceding claim, wherein the abstention threshold value has been determined by calibrating an abstention policy on a calibration set of prompt-response pairs, each having a corresponding risk value indicative of an incorrectness of the response in the pair.

13. The method of claim 12, wherein the corresponding risk value for each prompt-response pair in the calibration set has been determined using a risk prediction model.

14. The method of claim 12 or 13, wherein calibrating comprises adjusting the abstention threshold value in accordance with an optimization comprising, for the prompt-response pairs in the calibration data set: minimizing a count of instances when the abstention policy should have abstained from responding and did not while ensuring a measure of risk under the abstention policy satisfies a target risk tolerance criterion.

15. The method of claim 14, further comprising determining the count of instances when the abstention policy should have abstained from responding and did not. wherein the count of instances specifies a generated response for a first prompt of a first prompt-response pair when the corresponding risk value for the first prompt-response pair indicates an incorrect response.

16. The method of claim 15, wherein minimizing the count of instances when the abstention policy should have abstained from responding and did not comprises using a loss function indicative of the count of instances when the abstention policy should have abstained from responding and did not.

17. The method of claim 16, wherein the measure of risk under the abstention policy comprises an expectation of the loss function.

18. The method of any of claims 12-17, when dependent on any one of claims 7 or 9, wherein the abstention policy further comprises the similarity threshold value.

19. The method of claim 18, wherein calibrating further comprises adjusting the similarity threshold value in accordance with the optimization.

20. The method of any of claims 14-19, wherein the optimization further comprises minimizing a count of instances when the abstention policy should not have abstained and did.

21. The method of claim 20, further comprising determining the count of instances when the abstention policy should not have abstained from responding and did. wherein the count of instances specifies an abstention response for a first prompt of a first prompt-response pair when the corresponding risk value for the first prompt-response pair indicates a correct response.

22. The method of claim 21, wherein minimizing the count of instances when the abstention policy should not have abstained from responding and did comprises using a second loss function indicative of the count of instances when the abstention policy should not have abstained from responding and did.

23. The method of any of claims 14-22, wherein the abstention threshold value is a minimum abstention threshold value.

24. The method of claim 4, or any claim dependent therefrom, wherein the second neural network has fewer parameters than the first neural network.

25. The method of claim 4, or any claim dependent therefrom, implemented on a parallel computing system comprising at least three sets of neural network processing hardware coupled to be operable in parallel and wherein: processing the input prompt using a neural network to generate the plurality of candidate outputs for the input prompt comprises: implementing a first instance of the first neural network on a first of the sets of neural network processing hardware; implementing a second instance of the first neural network on a second of the sets of neural network processing hardware; andsampling from the first instance and the second instance of the first neural network to generate a first candidate output and second candidate output in parallel; and generating, using the second neural network, the respective pairwise similarity score for each of a plurality of pairs of candidate outputs comprises implementing a first instance of the second neural network on a third of the sets of neural network processing hardware.

26. The method of any preceding claim, wherein: the first neural network is a trained visual language model including an image encoder neural network and a text encoder; the input prompt, the generated candidate outputs to the input prompt, or both comprises an image.

27. The method of any preceding claim, wherein: the input prompt comprises image data and text data and the candidate outputs comprise information representative of the image data and text data; or the input prompt comprises first image data and text data and the candidate outputs comprise second image data, the second image data representative of the first image data processed according to contextual information contained in the text data.

28. A computer-implemented method comprising performing conformal abstention to determine an abstention threshold value, the method comprising: receiving a calibration set of prompt-response pairs, each having a corresponding risk value indicative of an incorrectness of the response in the pair; and determining, using the calibration set, an abstention policy for a neural network, wherein the abstention policy defines when the neural network should abstain from providing a response to a given input, and wherein determining the abstention policy comprises: minimizing a count of instances when the abstention policy should have abstained from responding and did not while ensuring a measure of risk under the abstention policy satisfies a target risk tolerance criterion.

29. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of claims 1-28.

30. A computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any one of claims 1-28.

Images