Shape preserving abstention for neural networks

Abstract

Methods, systems, and apparatus for calibrating abstention strategies to mitigate prediction errors in neural networks, including computer programs encoded on a computer storage medium. In one aspect, a system includes: receiving an input cue; processing the input cue using a neural network to generate a plurality of candidate outputs in response to the input cue; determining a similarity score characterizing the similarity between each of the candidate outputs and each of the other candidate outputs; determining whether the similarity score satisfies an abstention threshold; and, in response to determining that the similarity score satisfies a criterion based on satisfying the abstention threshold, providing one or more of the candidate outputs as a generated response to the input cue.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Application No. 63 / 548,839, filed February 1, 2024, and U.S. Application No. 63 / 549,306, filed February 2, 2024, the disclosures of which are incorporated herein by reference. Background Technology

[0003] This manual relates to using machine learning models to process data.

[0004] Machine learning models receive input and generate outputs based on those inputs, such as predicted outputs. Some machine learning models are parametric models and generate outputs based on the received inputs and the values ​​of the model's parameters.

[0005] Some machine learning models are deep models, which employ multiple layers to generate outputs from received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the Invention

[0006] This specification describes a system implemented as a computer program on one or more computers at one or more locations, the system being able to determine, in response to an input prompt, whether to abstain and not respond or to provide an output generated by a machine learning model (e.g., a neural network). Specifically, the system can apply an abstention strategy to mitigate prediction errors when predicting and providing outputs from a neural network configured to generate multiple outputs given the same input prompt. More specifically, the system can determine whether to abstain when the neural network is likely to produce a prediction error in response to the prompt.

[0007] Generally, machine learning models trained for useful tasks cannot always generate accurate outputs. Depending on the task, for example, for the end user, detecting errors in predictions can sometimes be difficult. One type of error that is difficult to detect, for example, is the illusion produced by generative neural networks (e.g., language processing neural networks). In this specification, an illusion refers to a situation where a generative neural network confidently outputs an incorrect answer as a response to an input prompt, a situation that may be difficult for a user to reliably detect.

[0008] More specifically, the system can sample and compare candidate outputs (e.g., k candidate outputs) generated by the neural network for the same input cue, and determine a similarity score representing the similarity between each candidate output and each other candidate output as a proxy for a measure of confidence in the response. More specifically, the system can use the similarity score to determine whether the neural network is prone to prediction errors, for example, by comparing the similarity score with a dropout threshold.

[0009] Additionally, this specification provides techniques for calibrating abstention policies without direct access to the underlying parameters of the neural network. Specifically, the system of this specification or another calibration system can calibrate an abstention policy by updating one or more parameterized values ​​of the abstention policy based on a specified risk tolerance criterion—for example, risk tolerance can be quantified as the ratio of unfiltered illusions. As an example, the abstention policy may include an abstention threshold and optionally a similarity threshold.

[0010] Specific embodiments of the subject matter described in this specification may be implemented in order to achieve one or more of the following advantages.

[0011] The technology described herein enables the determination of whether a neural network configured to generate multiple distinct outputs for a given input cue should abstain from responding to the input cue in order to prevent prediction errors. Specifically, the system can implement an abstention strategy that mitigates prediction errors while maintaining a low abstention rate based on a determined similarity score. In particular, the system can determine a similarity score for each response generated by the neural network for a specific input cue as a proxy measure of the confidence level of the response by sampling and comparing the candidate outputs of the neural network, which can be compared with an abstention threshold.

[0012] Additionally, the techniques described herein provide for calibrating abstention strategies, for example, using the system described herein or another calibration system. More specifically, calibrating abstention strategies does not require training or fine-tuning of a neural network or another model, such as training a separate classifier to detect various properties in the response (e.g., authenticity, harmfulness, etc.) or training an ensemble of models to estimate confidence in the response, thereby reducing computational resources and calibrating the abstention strategy compared to jointly training a neural network or another model or an ensemble of models. In particular, training an ensemble of general neural network models can be computationally prohibitively expensive. The techniques described herein describe calibrating abstention strategies using only query access to a neural network to generate candidate outputs for different input cues, which can be used to determine similarity scores.

[0013] Furthermore, calibration can be performed using a calibration dataset that is much smaller than the dataset used to train the neural network, thereby enabling easy maintenance and facilitating updates to the abstention policy parameters. Calibration can even be performed online, as it does not require any updates to the neural network. Since the quality of the abstention policy is defined by the calibration data, calibrating the abstention policy in an online environment can enhance its performance by including more data.

[0014] For longer responses generated by neural networks, the calibration described in this specification can, for example, calibrate the abandonment strategy against prediction errors more effectively than other methods used to calibrate prediction errors. In particular, methods relying on maximizing log probabilities are less effective for quantifying the uncertainty of neural networks because (i) incorrect and correct responses may have similar log-likelihoods, e.g., log-likelihoods within the same range of values, and (ii) the log-likelihood of a response depends on the length of the response. In contrast, for longer responses, defining the abandonment strategy using the techniques described in this specification is more effective than other methods because it does not depend on the generated log-likelihood and does not require setting a given sequence length for the response to calibrate the abandonment strategy. Relatedly, in example implementations involving calibrating the abandonment strategy of a generative neural network using the system described in this specification, calibration using said system achieves better performance than other methods used to prevent the illusion of generative neural networks (e.g., log-probability and greedy sampling methods, such as sampling the highest probability word for each word, the top 10 log-probability maximums, and other variations using sequence embedding exact matching / word overlap, etc.).

[0015] Additionally, the calibration system can calibrate the abstention policy using conformal abstention by updating one or more parameterized values ​​of the abstention policy based on a specified risk tolerance criterion. For example, the risk tolerance can be quantified as the ratio of unfiltered illusions. More specifically, the specified risk tolerance criterion can provide an upper bound for the risk of optimizing the abstention policy. The system can guarantee that the risk of the abstention policy remains below the risk tolerance criterion to achieve an approximate optimal solution that minimizes both the abstention rate and the risk of the abstention policy through one or more implementations of conformal risk control or a learn-before-test approach. Therefore, the abstention policy can provide a theoretical guarantee of risk minimization based on the specified risk tolerance criterion through a calibrated policy.

[0016] Furthermore, the system offers flexibility in defining how risk is calculated, for example, by leveraging rules or using additional models. For instance, a user can specify risk tolerance criteria independently of risk calculation, neural network architecture, and underlying data distribution. A user can set an overall risk tolerance criterion to assert a specific expected upper bound on the risk, and the calibration system can accordingly calibrate parameters of the abstention policy, such as the abstention threshold and similarity threshold. As a specific example of a generative neural network, the calibration system can minimize the expected abstention rate while ensuring that the expected illusory risk remains below the risk tolerance criterion.

[0017] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0018] Figure 1A This is a system diagram of an example abstention policy reasoning system that uses a calibrated abstention policy to determine whether to provide an output as a response to an input prompt.

[0019] Figure 1B This is a block diagram of an example abstention policy calibration system.

[0020] Figure 2 It shows Figure 1A An example of an abstention strategy reasoning system could determine a similarity score, which could be compared with an abstention threshold to determine whether to provide an output as a response to an input prompt.

[0021] Figure 3 This is a flowchart of an example process for determining whether to provide one or more candidate outputs of a neural network based on a calibrated abstention policy.

[0022] Figure 4 This is a flowchart of an example process for calibrating the abstention strategy of a neural network.

[0023] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0024] Figure 1A An example abstention policy reasoning system 100 is shown. The abstention policy reasoning system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which the systems, components and techniques described below are implemented.

[0025] The abstention policy reasoning system 100 can apply a calibrated abstention policy to any neural network configured to generate multiple candidate outputs in response to a given input cue, for example, by determining a similarity score that characterizes the similarity of each of the candidate outputs to each of the other candidate outputs, as described in more detail below.

[0026] Specifically, the abstention strategy reasoning system 100 can implement an abstention strategy that defines when the system 100 should provide an abstention output 162 in response to input prompt 110, for example, instead of providing an incorrect response. As an example, input prompt 110 can be an indicative instruction from a user, such as a question, statement, code snippet, or example, which can be processed by neural network 120 to generate multiple candidate outputs 125, such as candidate responses to input prompt 110. For example, input prompt 110 can be a request for information, such as, “What city is Vallecito Elementary School located in?” or “How many calories worth of food does a 20-pound cat eat per day?”

[0027] System 100 can generally calibrate the abandonment strategy of neural network 120, which is configured to generate multiple candidate outputs 125 in response to a given input cue (e.g., input cue 110). That is, neural network 120 can have any suitable type (e.g., fully connected layers, attention layers, convolutional layers, etc.) of any suitable number of neural network layers (e.g., 1 layer, 5 layers, or 10 layers) connected in any suitable configuration (e.g., a linear sequence of layers or a directed graph as layers).

[0028] In some cases, neural network 120 is a generative model, such as a generative adversarial network or an autoregressive language processing network. As an example, neural network 120 may have a recurrent neural network architecture. More specifically, neural network 120 may be a recurrent neural network (RNN), a long short-term memory (LSTM), or a gated recurrent unit (GRU). As another example, neural network model 125 may be transformer-based, such as an encoder-decoder transformer, an encoder-only transformer, or a decoder-only transformer, configured to use a multi-head attention mechanism for parallel processing of the content of multimodal inputs.

[0029] As a specific example, neural network 120 may be a language processing neural network. In this specification, illusion is a specific type of neural network prediction error that occurs when a generative processing neural network (e.g., a language processing neural network) confidently outputs an incorrect answer as a response to input prompt 110, and such prediction error may be difficult for a user to detect reliably.

[0030] A language processing neural network is an autoregressive network configured to process input sequentially and trained to predict the next element (e.g., a likelihood score distribution defined on the next set of elements). Specifically, a neural network can be called an autoregressive neural network when it autoregressively generates an output sequence of lexical units. More specifically, the autoregressively generated output is created by generating each specific lexical unit in the output sequence conditioned on the current input sequence, which includes any lexical units in the output sequence preceding the specific lexical unit; that is, lexical units that have been generated for any previous position in the output sequence preceding the specific lexical unit.

[0031] For example, a language processing neural network may be an autoregressive Transformer-based neural network that includes (i) multiple attention blocks, each of which applies a self-attention operation; and (ii) an output subnetwork that processes the output of the last attention block to generate a score distribution.

[0032] In this example, the language processing neural network can have any of a variety of Transformer-based neural network architectures, such as encoder-decoder transformer, encoder-only transformer, or decoder-only transformer. Examples of such architectures include those described in the following literature: J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, LA Hendricks, J. Welbl, A. Clark et al., Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; JWRae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HF Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, LA Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N.McAleese, A.Wu, E. Elsen, SM Jayakumar, E. Buchatskaya, D. Budden, E.Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, XL Li, A. Kuncoro, A.Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M.Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik,I.Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, BA. Hechtman, L. Weidinger, I. Gabriel, WS. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving, Scaling language models: Methods, analysis & insights from traininggopher, CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer (Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer), arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, Towards a human-like open-domain chatbot, CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al., Language models are few-shot learners, arXiv preprint arXiv:2005.14165, 2020.

[0033] Generally, to apply self-attention operations, each attention block uses one or more attention heads. Each attention head generates a set of queries, a set of keys, and a set of values, and then applies any of the various variants of query-key-value (QKV) attention (e.g., a dot product attention function or a scaled dot product attention function) to generate output using the queries, keys, and values. Each query, key, and value can be a vector comprising one or more vector elements. When multiple attention heads are present, the attention block then combines the outputs of the multiple attention heads, for example, by concatenating the outputs and optionally processing the concatenated outputs through a linear layer.

[0034] More specifically, the language processing neural network is configured to generate multiple outputs 125 for each input cue in a single processing iteration. Since the language processing neural network is configured to define a likelihood score distribution over the next set of elements (e.g., the output sequence of elements, representing the probability for each element of the output sequence), it can generate multiple candidate outputs 125 by processing the input cue 110 and sampling distinct output elements for each element of the output sequence. In some cases, the language processing neural network can generate N most probable responses based on probability values ​​and sample them as candidate outputs 125, for example, by performing top-k sampling.

[0035] In some cases, language processing neural networks are visual language models (VLMs), which can be configured to process input cues, including images or image sequences in a video, to generate intermediate representations of the images and perform image processing tasks. For example, a visual language model can be a contrastive language-image pre-trained (CLIP) model, a visual transformer (ViT), a unified image-to-image translation (UNIT) model, or an attention-based generative adversarial network (AttnGAN).

[0036] As an example, image processing tasks may involve generating outputs that require inferences (e.g., spatiotemporal inferences) in response to natural language query inputs, such as those relating to moving images (videos). For example, such queries might require predictive inferences ("what will happen next"), counterfactual inferences ("what would happen in a different circumstance"), explanatory inferences ("why did something happen"), or general causal inferences. For example, image representation can be used to detect objects in video frames and provide information related to the detected objects in response to queries, such as requests for predictions of future events or states associated with one or more objects (e.g., "Will objects X and Y collide?"), requests for conditional or counterfactual information associated with one or more objects (e.g., "What event would [not] happen if object X is modified, moved, or absent?"), or requests to analyze video frames to determine the properties or characteristics of one or more objects (e.g., "How many objects of type Z are moving?"). The output can be, for example, in the form of a yes / no answer, or can be defined as a probability distribution over a set of possible answers; or the response can define the location of the objects. Such a basic network can be used to predict whether two objects will collide or how this can be avoided. The output can be used, for example, to provide warnings, control the movement of one or more objects, or both.

[0037] As another example, the neural network can be a diffusion model configured to sequentially refine the initial state of the input cue 110 by adding noise to data samples to generate the output. In some cases, the neural network can be implemented as a latent diffusion model, which includes an encoder block for mapping the image to a low-dimensional latent space before applying the sequence of transformations, and a decoder block for mapping back from the low-dimensional latent space to the image space. As an example, the neural network can be implemented as a U-Net, which includes skip connections that allow the neural network to combine coarse features from the beginning of the sequence of transformations with fine features from the end of the sequence of transformations to improve the quality of the generated image.

[0038] More specifically, system 100 can use the abstention policy subsystem 130 to apply an abstention policy to determine whether to generate an abstention output 162 instead of providing a candidate output 164 as a response 160. In particular, system 100 can use the abstention policy subsystem 130 to process candidate outputs 125 to determine a measure of similarity between each of the candidate outputs (e.g., an overall similarity score 155) as a substitute for a measure of confidence in the response. Subsystem 130 can then use the similarity score 155 to determine whether the neural network 120 is prone to prediction errors (e.g., hallucinations) for a particular input cue 110 by comparing the similarity score with an abstention threshold 134.

[0039] The abstention policy may include one or more calibrated abstention policy parameters 132. For example, the abstention policy may include abstention threshold 134. As another example, the abstention policy may include abstention threshold 134 and a similarity threshold 136. In particular, the abstention parameter 135 deployed in the abstention policy inference system 100 is a calibrated value. For example, a corresponding abstention calibration system or another calibration system may calibrate the abstention policy parameter 132 by using the value of the parameter 135 tuned to a set of example cue-response pairs, as shown regarding Figure 1B and Figure 4 A more detailed description.

[0040] In some cases, system 100 can apply an abstention policy associated with a specified risk tolerance. In this context, subsystem 130 can determine whether to provide an actual candidate output 164 as a response 160 based on whether a measure of the similarity between each of the candidate outputs satisfies a specific risk tolerance criterion used to calibrate the parameterized values ​​of the abstention policy. For example, a corresponding abstention calibration system or another calibration system can calibrate the abstention policy parameters 132 according to a specified risk tolerance criterion, such as risk tolerance being quantified as the total number of illusions generated due to the application of an abstention policy with abstention policy parameters at a particular calibration iteration, as denoted by... Figure 1B and Figure 4 To describe in more detail. In this case, the abstention strategy can provide a theoretical guarantee of risk minimization through a specified risk tolerance criterion.

[0041] Specifically, the abstention strategy subsystem 130 can compare candidate outputs 125 by determining an overall similarity score 155 representing the similarity between each of the candidate outputs 125 (e.g., a subset of k sampled candidate outputs) generated by the neural network 120 for a given input cue 110 and each of the other candidate outputs. The subsystem 130 can then compare the similarity score 155 with an abstention threshold 134 to determine whether a response should be provided to the user, for example, as response 160.

[0042] For example, the abstention strategy subsystem 130 can use a similarity neural network 140 to process the candidate outputs 125 to determine the corresponding pairwise similarity score 145 for each pair of candidate outputs 125. Then, the subsystem 130 can combine the pairwise similarity scores 145 into an overall similarity score 155.

[0043] The similarity neural network 140 can have any suitable neural network architecture that can be configured to process a pair of candidate outputs to generate corresponding pairwise similarity scores for the pair of candidate outputs. In particular, the similarity neural network 140 can have any suitable number of neural network layers (e.g., 1 layer, 5 layers, or 10 layers) of any suitable type (e.g., fully connected layers, attention layers, convolutional layers, etc.) connected in any suitable configuration (e.g., a linear sequence of layers or a directed graph as layers).

[0044] In some cases, the similarity neural network 140 may be a language processing neural network, for example, a language processing neural network different from the one implemented as a language processing neural network when neural network 120 is implemented as such. Specifically, a pairwise similarity score 143 can be calculated by prompting the second language processing neural network with input including a pair of candidate outputs and instructions for determining the similarity between the pair of candidate outputs (e.g., on a normalized scale of 1 to 5, 1 to 10, etc.) to predict the pairwise similarity score. For example, the prompt may be formulated as follows: “Given a context, measure the similarity of a reference text with 10comparison texts on a scale of 1-10. Your response should be 10 numbers,where the i-th number is the similarity of the reference text with the i-thcomparison text. Here are a context, a reference text, and 10 comparisontexts for you to evaluate: Context: personalized prompt Reference Text: response Comparison Text1: response1 Comparison Text2: response2 Comparison Text3: response3 Comparison Text4: response4 Comparison Text5: response5 Comparison Text6: response6 Comparison Text7: response7 Comparison Text8: response8 Comparison Text9: response9 Comparison Text10: response10 Your response should be a list of 10 comma separated numbers. (Given context, measure the similarity between the reference text and 10 comparative texts on a scale of 1 to 10. Your response should consist of 10 numbers, where the i-th number represents the similarity between the reference text and the i-th comparative text. Here is the context, the reference text, and the 10 comparative texts for your evaluation:) Context: Personalized prompts Reference text: Response Compare text 1: response 1 Compare text 2: response 2 Compare text 3: response 3 Compare text 4: response 4 Compare text 5: response 5 Compare text 6: response 6 Compare text 7: response 7 Compare text 8: response 8 Compare text 9: response 9 Compare text 10: response 10 Your response should be a comma-separated list of 10 numbers. In some cases, the similarity neural network 140 can be a lightweight neural network, for example, a lightweight model with a smaller, less complex, or smaller and less complex architecture compared to neural network 120. Specifically, the similarity neural network 140 can have fewer (e.g., 10 times fewer, 100 times fewer, etc.) trained parameters than neural network 120. In other cases, the similarity neural network 140 can be the same neural network as neural network 120.

[0045] As an example, in the simplified case of generating four candidate outputs, to generate all possible pairwise similarity scores, subsystem 130 can compute similarity scores between candidate outputs 1:2, 2:1, 1:3, 3:1, 1:4, 4:1, 2:3, 3:2, 2:4, 4:2, 3:4, and 4:3. As another example, subsystem 130 can compute similarity scores between candidate outputs 1:2 to 4, 2:3 to 4, and 3:4, for example, because pairwise similarity scores between the same pair of outputs generally should not depend on the order. In particular, this provides a more computationally efficient way to generate similarity scores for multiple candidate outputs. This is especially useful when the computer / processor performing the similarity scoring has limited computing power (e.g., on personal devices such as mobile phones and smartwatches).

[0046] Subsystem 130 can generate a set of pairwise similarity scores 145 by processing candidate outputs 125 using a similarity neural network 140. Subsystem 130 can then determine an overall similarity score 155, which uses the pairwise similarity scores 145 to measure the similarity between a candidate output and each other candidate output; for example, the overall similarity score 145 compares each specific candidate output to other candidate outputs, compared to the pairwise similarity scores.

[0047] In the specific example depicted, subsystem 130 may use clustering engine 150 to process pairwise similarity scores 145 to determine an overall similarity score 155. In this case, clustering engine 150 may, for example, cluster the pairwise similarity scores 145 based on the corresponding pairwise similarity scores relative to the common candidate outputs, and use the clustering to determine the overall similarity score 145.

[0048] For example, clustering engine 150 can determine the count of the cluster with the largest number of candidate outputs as a similarity score, or calculate a measure of entropy as a similarity score. As another example, if the abstention policy parameter 132 includes a similarity threshold 136, subsystem 130 can determine how many pairs of similarity scores are greater than the similarity threshold 136. [The remaining text appears to be incomplete and requires further context.] Figure 2 A more detailed example of how clustering engine 150 can determine an overall similarity score of 155.

[0049] Then, the abstention strategy subsystem 130 can compare the overall similarity score 155 with the abstention threshold 134. As an example, if the overall similarity score 155 is greater than the abstention threshold 134, the system 100 can output a candidate output 164 from the language processing network 164. In particular, the subsystem 130 can select the candidate output 164 as a response 160, for example, from the largest cluster or any cluster that has a pairwise similarity score relative to the common candidate output that is higher than the similarity threshold 136.

[0050] As another example, if the overall similarity score 155 is less than the abstention threshold 134, the system 100 may output an abstention response 162 by outputting a response indicating a lack of knowledge or confidence (e.g., by outputting "I don't know" or another natural language phrase indicating that the neural network 120 has no confidence in any generated response) or by not providing any response to the input prompt 110 instead of providing an incorrect response. In some cases, the abstention response 162 may also be accompanied by suggestions on where the user can search to determine the answer.

[0051] For example, the corresponding abstention policy calibration system or another calibration system can calibrate the abstention policy parameter 132 according to the specified risk tolerance criterion. Figure 1B An example abstention policy calibration system 165 is described, which can be configured according to a specified risk tolerance criterion. 175 is used to calibrate the abstention policy parameters used by the example abstention policy inference system 100.

[0052] Specifically, the abstention policy calibration system 165 can obtain a calibration set of sample cue-response pairs 170, for example, calibration dataset D. n = {(X1, Y1), ..., (X n Y n As an example, the input prompt in calibration dataset 170 may be a request for information, and the corresponding response in the prompt-response pair may be a correct response to the request or an abstention response, for example, the abstention response indicating that the system is providing an incorrect response to the input prompt.

[0053] Generally, the calibration dataset 170 can be much smaller than the training dataset used to train the neural network. System 165 can also obtain risk tolerance criteria, for example, through inputs to system 165 from a user or from another system. 175. In particular, risk tolerance criterion 175 specifies an upper bound on the permissible risk in a response. In some cases, one or more users of system 165 may configure system 165 to calibrate corresponding abstention strategies, for example, based on different risk tolerances.

[0054] As an example, system 165 can calibrate the abstention policy parameter 132 based on minimizing the abstention rate while simultaneously limiting the expected risk of the policy according to risk tolerance criterion 175:

[0055] in It is the expected abstention rate under the abstention strategy a parameterized by parameter γ, and This is the expected loss caused by the abstention policy. Given a calibration dataset 170, system 165 can determine the value of the abstention policy parameter 135 in the approximate optimal solution of the above equation.

[0056] In the specific example depicted, the abstention policy parameter 132 includes the abstention policy threshold. and similarity threshold In this case, the abstention strategy is a binary abstention strategy. More specifically, the binary abstention strategy A(X, , The system 165 indicates whether it should provide a candidate output or a blank output as a response to the input prompt, for example, A(X, , = 1 indicates that system 165 should abstain from responding to the input prompt X, because the neural network has likely already generated an output that is a prediction error, such as a hallucination.

[0057] More specifically, the abstention policy calibration system 165 can calibrate the abstention policy parameters 132 by utilizing the black-box neural network access 195 to update parameters 132. That is, the abstention policy calibration system 165 can calibrate the abstention policy parameters 132 without training or fine-tuning the neural network, or even without accessing any of the underlying neural network parameters. Figure 1A As described, system 165 can calibrate any neural network configured to generate multiple candidate outputs. In some cases, the neural network can be a pre-trained or fine-tuned language processing neural network that can be included in a different system outside of system 165.

[0058] For a binary abstention strategy, for example, calibration system 165 can perform conformal prediction as a calibration process 180 to update parameters 132 that define when the inference system should output an abstention response instead of an incorrect response. More specifically, conformal prediction refers to associating the prediction with a confidence interval that defines the probability that a candidate output includes the correct output.

[0059] This is achieved by ensuring that the risk measure of the generated response of calibration set 170 does not exceed the risk tolerance criterion specified by the given risk tolerance criteria. The upper bound provided by 175, conformal prediction, offers a formal theoretical guarantee that an inference system deploying calibrated abstention policy parameters 132 will be able to manage risk, for example, by providing a confidence guarantee with a certain degree of certainty that holds with a high probability on samples from the same domain as calibration set 170. More specifically, the specified risk tolerance criterion 175 provides an upper bound on the risk used to optimize the abstention policy parameters 132 such that, when used in inference, the abstention policy will provide a response with a high probability that does not violate risk tolerance criterion 175.

[0060] The calibration system 165 can guarantee with high probability that the risk remains below the risk tolerance criterion 175 using any appropriate conformal prediction method. An example of this approach includes techniques described in more detail in the following literature: “Conformal Risk Control”, Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster, arXiv, 2023; and “Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control”, Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candes, Michael I. Jordan, and Lihua Lei, arXiv, 2022.

[0061] For example, in It is fixed and the system is used for calibration. In this case, according to the conformal risk control framework disclosed by Angelopoulos, the expected risk of λ can be limited on the calibration data 170:

[0062] for ,in This is the average loss on the calibration dataset 170. The guarantee in terms of expectation can be translated into a guarantee in terms of probability, i.e., for any failure probability δ, By amplifying the data, it can be proven that this result is true with a high probability.

[0063] Regarding the loss In the case of non-monotonic loss, the learn-before-test approach, as disclosed by Angelopoulos, can be used to limit the failure probability δ using the upper bound of confidence μ, even if the loss is non-monotonic:

[0064] In this case, μ can be determined by solving a simple optimization problem, for example, using the empirical Bernstein inequality or the Bernoulli relative entropy inequality.

[0065] Risk tolerance criterion 175 can be specified in any suitable manner; for example, calibration system 165 allows for flexible definition of how the measure of risk is calculated. For instance, the measure of risk can be specified by logically encoded rules or by predicting the measure of risk using an additional model. Specifically, a user of system 165 can specify risk tolerance criterion 175 and the method used to determine the measure of risk, for example, by configuring a model to use specific rules or an additional risk model. In some cases, a user can provide a risk machine learning model (not depicted) that can be configured to process input cues from calibration set 170 and predict the associated risk tolerance of calibration set 170.

[0066] When using a risk machine learning model to predict risk tolerance criterion 175, the risk machine learning model can have any suitable machine learning architecture that can be configured to process input cues to generate a measure of risk, such as a random forest, support vector machine, decision tree, linear regression model, or neural network. For example, the risk machine learning model can be a neural network with any suitable type (e.g., fully connected layers, attention layers, convolutional layers, etc.) of any suitable type (e.g., fully connected layers, attention layers, convolutional layers, etc.) connected in any suitable configuration (e.g., a linear sequence as layers or a directed graph as layers).

[0067] Specifically, calibration system 165 can perform a calibration process, such as calibration process 180. More specifically, system 165 can perform conformal prediction as calibration process 180 to quantify response uncertainty and determine whether system 100 should output an abstention response 162 based on the abstention strategy 190 implemented at the current calibration iteration, and can do so, for example, according to a risk tolerance criterion relative to a specified risk tolerance at the time of calibration iteration. 175 Minimize the loss function to update the abstention policy parameters 132.

[0068] Specifically, the calibration system 165 can augment the calibration dataset D by sampling k samples from the candidate outputs generated by the neural network. n Each hint-response pair (X, Y) in the dataset is processed, for example, by using a black-box neural network to access 195 to process query X.

[0069] Such as about Figure 1A As described, system 165 can determine a measure of pairwise similarity between each pair of candidate outputs. For example, system 165 can use a text similarity function s(x, Y, Y') to measure the similarity between text responses Y and Y'. Specifically, if s(x, Y, Y') ≥ If Y and Y' are identical, then Y and Y' can be considered to contain the same content. For example, for X = "What is the population of London" and Y = "8,980,000" and Y' = "8,900,000", a reasonable similarity function can assign the larger number to s(X; Y,Y').

[0070] In some cases, the similarity function is context-dependent, meaning that a deterministic measure of pairwise similarity can depend on the contextual cue X. In this case, where Y and Y' are generated from two different cues, the responses Y and Y' described above could be considered dissimilar.

[0071] Such as about Figure 1A As described, system 165 can then cluster the candidate outputs based on pairwise similarity metrics to determine an overall similarity score. System 165 can determine an overall similarity score, which indicates how similar the candidate responses are to each other, as a metric for the confidence level of the response.

[0072] For example, System 165 can determine the size of the largest cluster as the overall score. As another example, System 165 can determine whether the count of candidate responses is above a similarity threshold level. As yet another example, System 165 can define the overall similarity score as a measure of the entropy (e.g., cross-entropy) of the clusters. In this case, System 165 can define the probability distribution of each cluster on the candidate responses based on the cluster size, and can calculate the entropy of the distribution, for example, by assigning a probability mass proportional to the cluster size to each cluster.

[0073] At each of the multiple calibration iterations, system 165 can then update parameter 132 based on the objective function. In particular, system 165 can update the value of the abandonment policy parameter 132 by computing and backpropagating the gradient of the loss function, for example, using the update rule of any suitable gradient descent optimization algorithm (e.g., RMSprop or Adam).

[0074] More specifically, system 165 can determine a loss function that counts the number of instances where the calibration system 165 provided output based on the value of the abstention policy parameter at the current calibration iteration but should have abstained, for example:

[0075] Where 1 is an indicator function, for example, that is, where the loss is 1 in the case where system 165 provides an output but should have abstained, and m(X, The response is provided by system 165.

[0076] In this case, system 165 can then define the average loss as:

[0077] Furthermore, conditional risk is defined as occurring within the calibration set and test input prompt X. i and the corresponding correct response Y i The expected losses.

[0078]

[0079] In this case, system 165 can minimize the loss function (e.g., make R( , ) ≤ To update and For example, System 165 can implement conformal risk control methods to determine... As the minimum value, , This provides R( , Formal guarantees of risk tolerance for calibration sets of fewer than n cue-response pairs are described in more detail in “Mitigating LLM Hallucinations via Conformal Abstention” by Abbasi-Yadkori, Y. et al. (arXiv:2405.01563v1).

[0080] In some cases, system 165 may additionally include a second term in its loss function to penalize the number of instances where the policy abstains when system 165 could have responded. As an example, in this case, the total loss could be:

[0081] Where C is a positive constant weighted by the relative importance of the two loss terms. As described above, system 165 can minimize the loss function (e.g., make R( , ) ≤ To update and .

[0082] In some cases, the system (e.g., system 165 in calibration or system 100 in inference) can be configured to generate candidate outputs 125 in parallel. Although the calibration process and the abstention policy calibration system 165 are described below, system 165, the abstention policy inference system 100, or both can utilize parallelized computation to enhance response latency and reduce the bandwidth required to generate the response.

[0083] Specifically, system 165 can use a neural network to process multiple input cues (e.g., from calibration set 170) to generate multiple candidate outputs. Generating multiple candidate outputs for each input can be a relatively computationally intensive task. In some cases, each candidate output may require its own inference steps (e.g., implementing an instance of a neural network to generate the output). In this case, the computation required to generate multiple candidate outputs is scaled based on the number of candidate outputs generated.

[0084] To improve response latency and reduce bandwidth, the generation of candidate outputs can be parallelized. For example, a first processor (e.g., neural network processing hardware) can receive an input cue and generate a first candidate output for the input cue, and in parallel, a second processor can receive the input cue and generate a second candidate output for the input cue. Therefore, the process of generating multiple candidate outputs can be distributed and thus can be performed in a reduced time period compared to generating each candidate output sequentially.

[0085] For example, the neural network may be associated with a first computing system (e.g., stored on the first computing system, running on the first computing system, or located on the first computing system). Multiple candidate outputs may be generated at the first computing system. The first computing system may include a server or other computing system with relatively high availability of computing resources. The first computing system may include a parallel computing system comprising at least two sets of neural network processing hardware configured to operate in parallel. Each of the multiple candidate outputs may be generated using an instance of the neural network implemented on a corresponding processor of the first computing system. Generating multiple candidate outputs may include: processing an input prompt using a first instance of the neural network on a first set of at least two sets of neural network processing hardware to generate a first candidate output; and processing an input prompt using a second instance of the neural network to generate a second candidate output. The first computing system may receive input from a client system (e.g., a user device), for example, as a query.

[0086] In an example embodiment, the similarity neural network may be associated with a second computing system (e.g., stored on, run on, or located on a second computing system), which is different from the first computing system. The step of determining the similarity score can be performed at the second computing system. The second computing system may be a client system, such as a user device. Determining whether the similarity score meets a withdrawal threshold may additionally be performed at the second computing system, or may be performed at a third computing system, which may be the client system.

[0087] In another embodiment, the similarity neural network may be associated with a first computing system (e.g., stored on the first computing system, running on the first computing system, or located on the first computing system). As an example, in this case, the similarity neural network may run on a different processor than the processor on which the neural network runs (e.g., neural network processing hardware).

[0088] The computing system and / or processor on which a neural network runs can be larger than the computing system and / or processor on which a similar neural network runs. Larger can be interpreted as having, for example, more memory (e.g., for storing neural network parameters) and / or more computing power.

[0089] Generally, it may not be desirable to add additional computational burden to the processor generating multiple candidate outputs (e.g., at a first computer or processor). Therefore, the steps of running the abstention policy (determining the similarity score and whether the similarity score meets the abstention threshold) can be decoupled from response generation to distribute this computational burden. Designing the method in this way allows for efficient use of the computational resources of the available computing system to improve the reliability and / or accuracy of the generated response. In particular, multiple candidate outputs can be generated in parallel and therefore in a time-efficient manner, and the additional computational requirements of running the abstention policy can be handled externally.

[0090] This yields additional benefits, such as the ability to update the abstention threshold and associated parameters (e.g., risk tolerance, penalty constant, loss function, greediness, expected abstention rate, and any parameters associated with the similarity neural network) without requiring any updates to the neural network. Therefore, the abstention policy can be updated without any requirement to retrain the neural network. Decoupling the update of the abstention policy from the neural network reduces the need for training and retraining the neural network, and allows for a more flexible and individualized approach to risk for each user of the neural network.

[0091] For example, a first user and a second user can each use the same prompts to query the neural network, but each has a different risk tolerance for the answer. The first user might prefer a stricter abstention policy, abstaining more frequently to avoid incorrect answers with greater certainty. The second user might prefer a more lenient abstention policy with fewer abstentions, and therefore, tolerate a higher risk regarding the correctness of the answer. Decoupling the abstention policy from the neural network allows the network to provide answers to multiple users with different risk preferences without any retraining for each user. Furthermore, decoupling allows the abstention policy to be updated based on the latest data, resulting in more accurate abstention choices without training the neural network.

[0092] Figure 2 It shows Figure 1A An example of an abstention strategy reasoning system can determine an overall similarity score relative to common candidate outputs, which can be compared with an abstention threshold to determine whether to provide an output as a response to an input prompt.

[0093] The system can sample and compare candidate outputs (e.g., k candidate outputs) generated by the neural network for the same input cue. More specifically, the clustering engine 150 can determine the overall similarity score by clustering the pairwise similarity scores computed between each of the candidate outputs.

[0094] In the specific example depicted, the clustering engine can cluster candidate outputs based on pairwise similarity scores relative to specific common candidate outputs. For example, cluster A 210 represents candidate outputs with determined pairwise similarity scores greater than five for common candidate outputs, cluster B 220 represents candidate outputs with determined pairwise similarity scores greater than six for common candidate outputs, and cluster C 230 represents candidate outputs with determined pairwise similarity scores greater than seven for common candidate outputs. Similarly, clustering engine 150 can generate clusters for each of the other candidate outputs.

[0095] In this scenario, the implemented abstention strategy includes a similarity threshold 240, which has been calibrated to six, for example, such that the clustering engine 150 is restricted to using clusters with pairwise similarity scores relative to common candidate outputs that have a similarity threshold 240 greater than six to determine the overall similarity score. Specifically, the clustering engine 150 may use cluster B 220 and cluster C 230 to determine the overall similarity score.

[0096] As an example, clustering engine 150 can determine the overall similarity score as the total count of the largest clusters whose pairwise similarity scores are greater than a similarity threshold. In this case, the overall similarity score is eight, because this is the size of cluster B 220, which includes more candidate outputs than cluster C 230.

[0097] As another example, clustering engine 300 can determine the overall similarity score as the total count of all clusters whose pairwise similarity scores are greater than a similarity threshold. In this case, the overall similarity score is 13, because this is the aggregation size including clusters B 220 and C 230.

[0098] Such as about Figure 1A As described, the abstention policy subsystem can then compare the overall similarity score with an abstention threshold to determine whether to provide an output as a response to the input prompt. Specifically, the abstention threshold used for comparing the overall similarity scores may have been calibrated based on how the overall similarity score is calculated to provide a fair comparison. For example, in the first example described above, the abstention threshold may have been calibrated to the total count of the largest clusters with a similarity score greater than the similarity threshold of 240. As another example, in the second example described above, the abstention threshold may have been calibrated to the total count of all clusters with pairwise similarity scores greater than the similarity threshold of 240.

[0099] If the overall similarity score exceeds the abstention threshold, and if the overall similarity score is determined to be the total count of the largest clusters with pairwise similarity scores greater than the similarity threshold 240, the abstention strategy subsystem may, for example, select one of the candidate outputs from cluster B220, or if the overall similarity score is determined to be the total count of all clusters with pairwise similarity scores greater than the similarity threshold 240, select one of the candidate outputs from cluster B220 or cluster C.

[0100] Figure 3 This is a flowchart of an example process for determining whether to provide one or more candidate outputs of a neural network based on a calibrated abstention policy. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, an abstention policy inference system appropriately programmed according to this specification (e.g., Figure 1A The abstention strategy reasoning system 100) can perform process 300.

[0101] The system can receive an input prompt (step 310) and can use a neural network to process the input prompt to generate multiple candidate outputs in response to the input prompt (step 320). In some cases, the system can use a first neural network to generate multiple candidate outputs and can identify a first number of initial candidate outputs as said multiple candidate outputs, for example, by selecting a subset of candidate outputs based on any suitable criterion. For example, the neural network can be a language processing neural network. In this case, the input prompt can include indicative instructions, such as questions, statements, code snippets, or examples. As yet another example, the neural network can be a trained visual language model including an image encoder neural network and a text encoder. In this case, the input prompt, the generated candidate outputs in response to the input prompt, or both can include an image.

[0102] As an example, the system can use a parallel computing system to generate multiple candidate outputs. For instance, the parallel computing system includes at least three sets of neural network processing hardware coupled together to operate in parallel. In this case, the system can implement a first instance of a first neural network on a first set of the neural network processing hardware, and a second instance of the first neural network on a corresponding second set of the neural network processing hardware. The system can then sample from the first and second instances of the first neural network to generate one or more candidate outputs in parallel.

[0103] The system can determine a similarity score characterizing the similarity between each of the candidate outputs and each of the other candidate outputs (step 330). Specifically, the system can use a second neural network to generate the corresponding pairwise similarity score for each pair of candidate outputs (e.g., all outputs or a subset of outputs). In some cases, the second neural network can have fewer parameters than the first neural network; for example, the second neural network can be a lightweight model with a smaller, less complex, or smaller and less complex architecture relative to the first neural network. In cases where the system uses a parallel computing system to generate candidate outputs, the system can implement a first instance of the second neural network on a third set of neural network processing hardware, and use the first instance of the second neural network to generate the corresponding pairwise similarity score for each pair of candidate outputs.

[0104] For example, the system can use a second neural network to process an input including instructions to generate a pair of candidate outputs and a pair of candidate outputs for each pair of candidate outputs, to generate a corresponding pair of candidate outputs. As another example, the system can use a second neural network to process an input including instructions to generate corresponding pair of candidate outputs and multiple pairs of candidate outputs, to generate a corresponding pair of similarity scores for each of the multiple pairs, for example, in a single processing iteration.

[0105] The system can then use the corresponding pairwise similarity scores to generate a similarity score. For example, the similarity score could be a contextual similarity score, which measures the contextual similarity between candidate outputs based on the context provided by the input prompt, for example, by conditionalizing the similarity between candidate outputs based on the input prompt. As an example, the system could generate a similarity score based on a similarity threshold. (The text then abruptly shifts to a different topic: "Regarding...") Figure 4 A more detailed description of an example of calibrating a similarity threshold as part of an abstention strategy.

[0106] In some cases, the system can generate a similarity score by determining a set of clusters of candidate outputs based on their corresponding pairwise similarity scores relative to the common candidate outputs. For example, each cluster may include only candidate outputs whose pairwise similarity scores relative to the common candidate outputs are greater than a similarity threshold. As an example, in this case, the system can determine the similarity score based on the size of the largest cluster. As another example, in this case, the system can determine the similarity score based on the entropy of one or more clusters, for example, by assigning a probability mass proportional to the cluster size to each cluster and calculating the entropy of the distribution. In other cases, the system can generate a similarity score from the corresponding pairwise similarity scores by determining the count of pairwise similarity scores greater than a similarity threshold for each candidate output. In this case, the system can identify the largest count as the similarity score.

[0107] The system can determine whether the similarity score meets the criteria based on an abstention threshold (step 340). Specifically, the system can compare the similarity score to the abstention threshold. More specifically, the abstention threshold may have been determined by calibrating the abstention strategy on a calibration set of cue-response pairs, each cue-response pair having a corresponding risk value indicating the inaccuracy of the response in the pair, e.g., whether a baseline true label should have been provided for an abstention response. [The remaining text appears to be incomplete and requires further context.] Figure 4 A more detailed description of an example of calibrating abstention values ​​as part of an abstention strategy.

[0108] In response to determining that the similarity score meets the criteria (e.g., greater than the abstention threshold), the system may provide one or more of the candidate outputs as a generated response to the input prompt (step 350). As an example, if the input prompt includes image data and text data, the generated response may include information representing the image data and text data. As another example, if the input prompt includes first image data and text data, the generated response may include second image data representing the first image data processed based on contextual information contained in the text data. Conversely, in response to determining that the similarity score does not meet the criteria (e.g., less than the abstention threshold), the system may provide an abstention response not generated using any of the candidate outputs, or the system may abstain without responding, for example, by not providing any response to the input prompt.

[0109] Figure 4 This is a flowchart of an example process for calibrating a relinquishment policy for a neural network. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a relinquishment policy calibration system appropriately programmed according to this specification (e.g., relinquishment policy calibration system 165 of Figure 1) can perform process 400 to generate a relinquishment policy by... Figure 1A The abstention strategy reasoning system 100 uses a calibrated abstention strategy.

[0110] The system can obtain a set of prompt-response pairs, each prompt-response pair having a corresponding risk value indicating the inaccuracy of the response in the pair (step 410). For example, a risk prediction model or rule may have been used to determine the corresponding risk value for each prompt-response pair. In some cases, the system can be configured to receive a risk prediction model or rule based on user input to the system.

[0111] The system can use a neural network to process one or more cues in a set of cue-response pairs to generate a predicted response based on the current abstention threshold (step 420). About Figure 3 It describes in more detail an example of how the system can process one or more prompts to generate a predicted response.

[0112] The system can determine the count of instances where a waiver policy should have waived but did not, and instead responded (step 430). Specifically, if the corresponding risk value of the first prompt-response pair indicates an incorrect response, the instance count can specify the generated response for the first prompt in the first prompt-response pair. In some cases, the system can additionally determine the count of instances where a waiver policy should not have waived but did. In this case, if the corresponding risk value of the first prompt-response pair indicates a correct response, the instance count can specify the waiver response for the first prompt in the prompt-response pair.

[0113] The system can adjust the abstention threshold (step 440) by minimizing the count of instances where abstention should have been avoided but was avoided, while simultaneously ensuring that the measure of risk under the abstention policy meets the target risk tolerance criterion. Specifically, the abstention threshold can be a minimum abstention threshold, for example, if... Figure 3 If the similarity score described in the code is less than the abstention threshold, the system can provide an abstention response. In some cases, the abstention strategy also includes, for example, methods for determining... Figure 3 The similarity threshold described in the similarity score. In this case, the system can adjust the similarity threshold based on the optimization described above as part of the calibration process.

[0114] Specifically, the system may use a loss function that indicates the count of instances that should have abstained but did not, and instead responded. In some cases, the system may additionally use a second loss function that indicates the count of instances that should not have abstained under the abstention policy but did, and did not, respond. For example, the measure of risk may be determined based on a risk prediction model or rule used to determine the corresponding risk value for each prompt. In some cases, the measure of risk under the abstention policy is the expected value of the loss function for one or more predicted responses.

[0115] The methods described herein can be executed (e.g., as a computer program) on one or more computers located at a single site or distributed across multiple sites and interconnected via a data communication network. A computer may also be referred to as a computing system. A computer suitable for performing the methods described herein may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit (CPU). Generally, the CPU receives instructions and data from read-only memory or random access memory, or both. The elements of a computer are the CPU for making or executing instructions and one or more memory devices for storing instructions and data. The CPU and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Generally, a computer will also include one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or operatively coupled to receive data from or transfer data to said one or more mass storage devices, or both. However, a computer need not have such devices. In addition, a computer can be embedded in another device (e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few).

[0116] The methods described herein can be executed on a parallel computing system (e.g., as a computer program). A parallel computing system may include one or more processors configured to operate in parallel. Processors may include hardware, such as neural network processing hardware. In some arrangements, processors may be associated with the same computer. In other arrangements, processors may be associated with multiple computers. Embodiments of the methods described herein are specifically adapted for operation on parallel computing systems. Parallelization can help improve computational speed, for example, calculating clockwork time. Parallelization can also be used to assist in task pipelined operation when multiple tasks are distributed across different processing systems.

[0117] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or app that a user can interact with through an implementation of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. Components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0118] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship is established by computer programs running on respective computers that establish a client-server relationship between them. In some embodiments, the server sends data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device. The user device may be referred to as a client device or more simply as a device.

[0119] Every computing system has a limited amount of computing resources. Computing resources can include, for example, computing power, computing speed, memory capacity, power, bandwidth, queue length, circuit area usage, cooling requirements, or any other resources. Compared to client components (e.g., user devices), backend components (e.g., servers) are typically configured with a relatively large amount of computing resources. For example, servers may have a larger memory footprint, higher computing power, etc.

[0120] The task performed by the first neural network can be related to multimodal data. The first neural network can be considered a multimodal machine learning model. In the following discussion, the machine learning model can be simply referred to as a "model". As used herein, "modality" refers to the type of data, and therefore a multimodal machine learning model is a model capable of processing multiple different types of data. The model can receive multimodal data as input. The model can then provide information related to (e.g., derived from) the multimodal data as output. The input can be a query, such as a text-based question with associated data items (e.g., an image). The output can represent a response to the query. The output can be in the form of text (e.g., transcription), audio utterance (e.g., simulated spoken utterance), or any other means of conveying information. Multimodal data can include one or more of the following: text data items, image data items, audio data items (representing values ​​of digitized audio waveforms, e.g., time series of waveform representation elements), and sensor data (representing values ​​of digitized sensor waveforms). Image data items can include image data representing still or moving images (i.e., images or videos, e.g., images or videos already captured using a camera). Elements of image data can include monochrome or color pixels of an image or video. As defined herein, an "image" includes, for example, a point cloud from a LiDAR system, and a "pixel" includes points within that point cloud. Similarly, a "video" includes a time series of point clouds. Objects in an image or video may include objects represented by the image or video, such as physical objects.

[0121] Text can be received, for example, as a series of encoded characters (e.g., characters encoded in UTF-8); such "characters" can include Chinese and other similar characters, as well as logograms, syllabograms, etc. Multimodal machine learning models can include a text encoder that processes sequences of text to represent the text as a series of text lexical units from a vocabulary, for example, text lexical units each representing a word, phrase, or character in a natural or computer language. The computer language can be any form of language used for communicating with a computer, such as a markup language, or a command or configuration language, or a data exchange language (such as JSON), or a programming language.

[0122] Model output can include any form of output suitable for a machine learning task performed by the multimodal machine learning model. For example, model output can include text in natural or computer language defining the outcome of the task, such as for tasks like adding descriptive text to images, visual question answering, or object detection or instance segmentation. And or alternatively, model output can include data defining image, video, or audio objects, for example, in a generation task; or model output can include non-textual action selection data for selecting actions to be performed by an agent controlled by the model. As another example, model output can also, or alternatively, define intermediate steps to be performed during the task, such as calls to software APIs for software tools used in performing the task; the multimodal input can then receive output from said software tool, which is used to generate the final model output for performing the task. Several specific examples of model output are given later.

[0123] Such multimodal models (e.g., VLMs) can be trained using very large (but potentially noisy) datasets, where text is paired with images and / or with one or more other types of data (e.g., audio data or data related to the actions of an agent acting in an environment to perform various tasks). Such models can be trained, for example, using self-supervised learning. The pairings may often be imperfect, and the training dataset may include, but may not include, any actual examples of the specific task to be performed, but the ability to perform the specific task may emerge. Many examples of suitable publicly available training datasets exist. A trained VLM can be referred to as a pre-trained VLM.

[0124] Some example multimodal machine learning models that can be used with the techniques described in this paper include: Flamingo (Alayrac et al., arXiv:2204.14198); ALIGN (Jia et al., arXiv:2102.05918); PaLI (Chen et al., arXiv:2209.06794) and PaLI-X (Chen et al., arXiv:2305.18565); and Gemini (“Gemini: A Family of Highly Capable Multimodal Models, Gemini Team, Google”). These citations also include instructions on training datasets that can be used to train the respective models.

[0125] Generally, performing image processing tasks using multimodal neural networks (such as visual language models (VLMs)) that include (trained) image encoder neural networks can involve feeding images to the image encoder neural network to generate representations of the images, particularly representations of the image features. The image representations are then processed to perform the image processing task. Techniques for processing image representations to perform a wide range of image processing tasks are generally well-known. Typically, video processing tasks can be performed by processing representations of multiple images representing a sequence of video frames.

[0126] Examples involve generating outputs that require inferences (e.g., spatiotemporal inferences) in response to natural language query inputs, such as those relating to moving images (videos). For example, such queries might request predictive inferences ("what will happen next"), counterfactual inferences ("what would happen in a different circumstance"), explanatory inferences ("why did something happen"), or general causal inferences. For instance, image representations could be used to detect objects in video frames and, in response to a query, provide information related to the detected objects. Queries may include, for example, requests for predictions of future events or states related to one or more objects (e.g., "Will objects X and Y collide?"), requests for conditional or counterfactual information related to one or more objects (e.g., "What event would [not] happen if object X is modified, moved, or absent?"), or requests to analyze video frames to determine the properties or characteristics of one or more objects (e.g., "How many objects of type Z are moving?"). Output may be, for example, in the form of a yes / no answer, or may be defined as a probability distribution over a set of possible answers; or the response may define the position of the object. Such systems can be used to predict whether two objects will collide or how this can be avoided. The output can be used, for example, to provide warnings and / or control the movement of one or more objects.

[0127] The following describes example tasks of multimodal models such as the Visual Language Model (VLM).

[0128] Generally, a multimodal machine learning model can be trained to perform one or more machine learning tasks of any kind. Once trained, it can be deployed for use in performing these tasks. For example, the model can be deployed in an environment that allows a user to provide a request for the model to process specified multimodal inputs to generate corresponding model outputs. This request can be made, for example, through a user interface or an application programming interface (API). The request can be sent from a user device (e.g., via a data communication network, such as the Internet) to one or more computers implementing the model, for example, in a data center. The model can then process the multimodal inputs specified by the user request to generate corresponding model outputs and then send those outputs to the user device (e.g., via a data communication network).

[0129] In some implementations, after training, the specific task to be performed by the multimodal machine learning model can be described by a portion or all of a sequence of text in the model's multimodal input. For example, in multimodal input including images, such a prompt could specify "Generate a caption," "Generate adescription," "Answer the following question: [about the image, or video item]," or "Detect a person." In the case of the model being used for agent control tasks, the prompt could be defined as "Take the knife out of the drawer," or "Q: What action should the robot take to take the knife out of the drawer?" And, or alternatively, such a prompt could provide one or more examples of the task to be performed. The multimodal machine learning model can be trained on multiple natural and / or computer languages, and the prompt can then specify the language to be used.

[0130] Below are a few examples of machine learning tasks that can be performed by models trained as described in this article. The tasks described below can be those requiring spatial awareness or other context from images or videos. For example, a prompt could ask, “What is the object in the top left corner?”

[0131] As an example, a task may include object or action detection. Task-specific training data items may include images or videos containing one or more objects or actions, along with sequences of text. The text sequence may describe or otherwise label the objects or actions and may include text giving the bounding box coordinates of the objects or actions. After training, when the model is used for inference, the model output may include or represent text describing or otherwise labeling the detected objects or actions in the images, and may include the bounding box coordinates of the detected objects or actions, such as "10 20 90 100cat 20 30 100 100 dog".

[0132] As another example, the task can include a classification task, such as an object or action classification task. Task-specific training data items can include images or videos containing one or more objects or actions, as well as sequences of text. The text sequence can describe the objects or actions or otherwise classify them. After training, when the model is used for inference, the model output can include data that classifies objects or actions in the image input into one of several classes, such as text.

[0133] As another example, the task may include an image or video description task, such as a task that adds descriptive text (as used herein, which includes an audio description task to explain what is happening in a video). Task-specific training data items may include images or videos and sequences of text describing the images or videos. After training, when the model is used for inference, the model output may include data describing the images or videos, such as text. For example, the model output may provide descriptive text or descriptions, or it may count objects in an image or video, or it may provide some other form of description.

[0134] As another example, the task could include an image or video question-answering task. Task-specific training data items could include images or videos and sequences of text describing those images or videos. After training, when the model is used for inference, the model output could include data (e.g., text) that answers questions about a second modal input specified in the sequence of prompts about the text, as described above. This could be used, for example, to answer questions about visual drawing and charts or about sound.

[0135] As another example, the task may include a character or word recognition task, such as an OCR (Optical Character Recognition) task. Task-specific training data items may include images or videos and sequences of text, including text depicted in the images or videos or represented as speech in audio items. After training, when the model is used for inference, the model output may include text representing characters or words from a second modal input, for example, in natural language.

[0136] As another example, the task can include still or moving image generation; Google DeepMindImagen 3 is an example of a system capable of generating image output. As yet another example, an image, such as a drawing or chart, can be decoded from one or more (language) lexical units generated by the system. Training data items for such a system can include images or videos and sequences of text describing the images or videos. After training, when the model is used for inference, the model output can include image or video data, such as image data defining the pixel values ​​of still or moving images, and sequences of text in the model's multimodal inputs that can describe or characterize the images or videos to be generated.

[0137] As another example, the task may include a computer language text generation task. Task-specific training data items may include images or videos and sequences of text in the computer language used to generate the images or videos. After training, when the model is used for inference, the model output may include text in the computer language or another computer language used to generate or render images or videos (e.g., web pages, drawings, or charts).

[0138] As an example, the model receives input in the form of image data items and text data items. The text data items can be questions about the image data items. For example, an image data item can contain images of one or more objects, and each object can belong to a specific class. The text data items can be questions about one or more objects, the class of said objects, the location of the objects or features within the image (e.g., pixel locations within the image), or any other characteristic of the objects or the image.

[0139] In the example using multimodal data, the input includes an image of a person wearing a cap and driving a car. The input also includes the question "what is the person wearing?". The model can provide an answer to the question. In this example, the correct answer would be "cap". Other correct answers in the example could include "headwear", "brimmed hat", or "baseball cap". An incorrect answer could be "car". The correct answer is related to the characteristics of the image (e.g., the class of the object in the image), but importantly, it is contextualized in light of the question. Therefore, correct answers relate to objects in the class related to 'wearable clothing', and answers related to objects in irrelevant classes (e.g., car) are ignored. Applying the method described in this paper to this example, answers related to brimmed hats, headwear, caps, etc., will have high similarity scores and are therefore more likely to be returned as an answer. Applying the method described in this paper to this example, answers related to cars will likely have low similarity to most other answers, and therefore the model will likely abstain from providing this as an answer.

[0140] In another example using multimodal data, the input includes a video of a car driving near a pedestrian (e.g., multiple images of frames forming a motion model). The input also includes the question "Will the car hit the pedestrian?". The model provides a prediction as output of whether the car will hit or not the pedestrian. The prediction can be made with reference to a physical model, explicit or implicit comparisons to similar scenarios, or any other information the model can understand from the video. Such scenario predictions are particularly useful for controlling autonomous vehicles or driver assistance features. It is conceivable that in vehicle control applications such as these, risk tolerance may be low, and therefore it may be beneficial to set a forfeit threshold and a similarity threshold to ensure low risk tolerance.

[0141] In another example, the input to the model may include text data items, image data items, or both. The input may be a query related to selecting or generating an image with specific characteristics. For example, the query may be a text-based request to select or generate an image of an object of a specific class. The query may include an image of an object of a specific class and a text-based request to select or generate an image of an object of the same class as the object in the image in the query. The model provides an image as output in response to the query, such as an image of an object of a specific class. The output image may be selected from a set of candidate images. Candidate images may be images generated by the model. Using the model described herein, the model is more likely to provide an image that accurately represents the type of the requested image and is more likely to reject rather than provide an incorrect image. The generated image of the object can be considered a simulation of the object. The object may be a real-world object, such as a real-world object belonging to a specific real-world class.

[0142] The model can be trained on training images (e.g., images representing real-world objects). When given the task of generating images, the model, trained to generate images, processes the input to select values ​​for the pixels of the image (e.g., RGB, intensity, gamma). The pixel values ​​are selected and arranged such that the final image representation of the object is formed when the pixels are displayed. The represented object is related to information contained within the query. The generated image may share characteristics (e.g., pixel values ​​and arrangement) with the training images on which the generated image was trained. For example, a generated image of a cat will share some characteristics (e.g., pixel values) with training data images depicting cats.

[0143] In another example of a computer language text generation task, task-specific training data items may include images or videos and sequences of computer-language text for performing tasks related to those images or videos, such as data processing tasks involving analyzing the content of images or videos to provide results of analysis, or searches for information related to the content of images or videos. The computer language in the model output may include computer language used to invoke functions or call one or more external APIs. As an example only, such output may be formatted as a JSON object. As previously described, the sequence of text in the multimodal input may define the task to be performed, and the second modal input may include, for example, images or videos related to the task to be performed, such as tasks involving manipulating specific types of data that may benefit from access to APIs (e.g., mathematical data, date / time-related data, scientific data, recent data that may be later than the model's training (which can be accessed through search functions or APIs), etc.). After training, when the model is used for inference, the model output may include text in the computer language or another computer language for performing tasks related to the images or videos in the second modal input (e.g., as described above). Methods may then include using the computer-language text to perform the task.

[0144] Generally, when the model output includes text, this can be provided as speech representing the text.

[0145] In some implementations, the machine learning task includes an agent control task, where the agent interacts with the environment to perform the agent control task. In these implementations, multimodal inputs include observations representing the environment. For example, multimodal inputs may include a sequence of text defining the task to be performed by the agent, and images may represent observations of the environment captured, for example, by a camera or other imaging device from a real-world environment. Task-specific training data items may include a sequence of text representing one or more actions of the agent, as well as image observations of the environment. After training, when the model is used for inference, the model output includes an action selection output, for selecting one or more actions to be performed by the agent in response to observations, such as text. For example, model output 122 may define actions as text, such as “A:132 114 128 5 25 156”, which can be converted into control signals for a mechanical agent (such as a robot), for example, “ As another example, the action selection output can also, or alternatively, define one or more low-level skills, such as a vocabulary of skills learned previously. As mentioned earlier, a sequence of text in the model's multimodal input can describe the task to be performed, such as "What action should the robot take to [perform task]".

[0146] In some implementations of agent control, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment. Examples include robots or autonomous or semi-autonomous land, air, or sea vehicles that operate or navigate through the environment. Actions are performed by the mechanical agent within the real-world environment to complete a task. For instance, the agent may be a robot or other mechanical agent that interacts with the environment to perform a specific task (e.g., locating or manipulating an object of interest within the environment, moving the object of interest to a specified location within the environment, or navigating to a specified destination within the environment). In these implementations, observation may include one or more of the following: images, object position data, and sensor data used to capture observations as the agent interacts with the environment. Actions may define control signals used to control the robot or other mechanical agent, such as position, torque, or other control signals for parts of the mechanical agent, or higher-level control commands.

[0147] In some implementations of intelligent agent control, the intelligent agent can be a human agent, and the environment can be a real-world environment. For example, the intelligent agent can be a human user of a digital assistant (such as a smart speaker, smart display, or some other device used to instruct the user to perform actions). The task can be any real-world task the user wishes to perform. Observations can be obtained from an observation capture subsystem (e.g., a monitoring system such as a video camera or sound capture system) used to capture visual observations of the user performing the task. Actions can include instructions guiding the user to perform the task in the form of, for example, text, images, video, or audio data (such as speech).

[0148] In this specification, the term "configured" is used in connection with computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or combinations thereof that enable it to perform specific operations or actions during operation. For example, configuring a system may involve installing a software library with specific algorithms, updating firmware with new instructions for data manipulation, or adding hardware components to gain enhanced processing power. Similarly, one or more computer programs are "configured" to perform those intended operations or actions when executed by a computing device or hardware, containing instructions that cause the device to perform specific operations or actions.

[0149] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuit systems, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be implemented as one or more computer programs, which are essentially modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a computing device or hardware. The storage medium can be a storage device such as a hard disk drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination thereof. Additionally or alternatively, program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by the computing device or hardware. Furthermore, implementations can leverage emerging technologies such as quantum computing or neuromorphic computing for specific applications and can be deployed in distributed or cloud-based environments, where components reside on different machines or within a cloud infrastructure.

[0150] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, computing devices or hardware may also include code that creates the execution environment for computer programs. This code can take the form of processor firmware, protocol stacks, database management systems, operating systems, or combinations of these elements. In a general-purpose computing (GPGPU) environment where code specifically designed for GPU execution (often referred to as kernels or shaders) is employed, embodiments can particularly benefit from leveraging the parallel processing capabilities of the GPU. Similarly, TPUs excel at running optimized tensor operations, which are crucial for many machine learning algorithms. By utilizing these accelerators and their specialized programming models, systems can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in fields such as computer vision, natural language processing, and robotics.

[0151] Computer programs (also known as software, applications, modules, scripts, code, or simply programs) can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. They can be deployed in various forms, such as standalone programs, modules, components, subroutines, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., a script within a markup language document), residing in a dedicated file, or distributed across multiple coordination files (e.g., files storing modules, subroutines, or code segments). Computer programs can execute on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected via data communication networks. The specific implementation of a computer program can involve a combination of traditional programming languages ​​and programming for GPGPUs or TPUs utilizing specialized languages ​​or libraries designed for them, depending on the chosen hardware platform and desired performance characteristics.

[0152] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components mounted on one or more computers, which may be located at a single site or distributed across multiple locations. In some cases, one or more dedicated computers may be used for a particular engine, while in others, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of AI and machine learning may include data preprocessing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of an engine will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0153] The processes and logic flows described in this specification can be executed by one or more programmable computers that run one or more computer programs to perform functions by manipulating input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be used to implement the concurrent execution of various aspects of these processes and logic flows, thereby significantly accelerating the process. This approach offers significant advantages for computationally intensive tasks common in AI and machine learning applications, such as matrix multiplication, convolution, and other operations exhibiting high parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedup and efficiency gains can be achieved compared to relying solely on CPUs. Alternatively, or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), to achieve even higher performance or energy efficiency in specific use cases.

[0154] Computers capable of executing computer programs can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators typically work in conjunction with the CPU, handling specialized computations, while the CPU manages overall system operation and other tasks. Typically, the CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The components of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of the processing unit and memory will depend on factors such as the complexity of the AI ​​model, the amount of data being processed, and the desired performance and latency requirements. Implementations can be carried out on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. Systems can include storage devices such as hard disk drives, SSDs, or flash memory for persistent data storage.

[0155] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and storage devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or durability.

[0156] To facilitate user interaction, embodiments of the subjects described in this specification can be implemented on a computing device equipped with a display device (such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display) for presenting information to the user. Input can be provided by the user through various means, including a keyboard, touchscreen, voice commands, gesture recognition, or other input modalities, depending on the specific device and application. Additional input methods may include sound, voice, or tactile input, while feedback to the user may take the form of visual, auditory, or tactile feedback. Furthermore, the computer can interact with the user by exchanging documents with the user's device or application. This may involve sending web content or data in response to a request, or sending and receiving text messages or other forms of messages via a mobile device or messaging platform. The choice of input and output modalities will depend on the specific application and the desired form of user interaction.

[0157] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow or JAX. These frameworks provide comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0158] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These components may include: backend components, such as backend servers or cloud-based infrastructure; optional middleware components, such as middleware servers or application programming interfaces (APIs) for facilitating communication and data exchange; and frontend components, such as client devices having a user interface through which users can interact with the implemented subject matter, a web browser, or an app. For example, the described functionality may be implemented only on the client device (e.g., for on-device machine learning) or deployed as a combination of frontend and backend components for more complex applications. Where present, these components can be interconnected using any form or medium of digital data communication, such as communication networks, like local area networks (LANs) or wide area networks (WANs), including the Internet. The specific system architecture and the selection of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0159] Computing systems can include clients and servers that can be geographically separated and interact via communication networks. The specific type of network (such as a local area network (LAN), wide area network (WAN), or the Internet) will depend on the accessibility and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. Depending on the nature of the data being exchanged and the system's security requirements, these protocols can include HTTP, TCP / IP, or other specialized protocols. In some embodiments, the server transmits data or instructions to a user device acting as a client, such as a computer, smartphone, or tablet. The client device can then process the received information, display the results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interaction between the user and the system, enabling a wide range of applications and functionalities.

[0160] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from a claimed combination may be removed from said combination, and the claimed combination may involve sub-combinations or variations thereof.

[0161] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or requiring all shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0162] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims may be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer implementation method, comprising: Receive input prompts; The input prompt is processed using a neural network to generate multiple candidate outputs for the input prompt; Determine a similarity score that represents the similarity between each of the candidate outputs and each other candidate output; The similarity score is determined based on the abstention threshold to determine whether it meets the criteria. as well as In response to determining that the similarity score satisfies the criterion, one or more of the candidate outputs are provided as a generated response to the input prompt.

2. The method of claim 1, further comprising, in response to determining that the similarity score does not meet the criterion, Provide an abstention response that is not generated using any of the candidate outputs; or No response is provided to the input prompt.

3. The method of claim 1 or 2, wherein determining the similarity score that measures the similarity between the candidate outputs comprises: A second neural network is used to generate the corresponding pairwise similarity score for each of the multiple candidate outputs; as well as The similarity score is generated using the corresponding pairwise similarity scores.

4. The method of claim 3, wherein using the second neural network to generate a corresponding pairwise similarity score for each of the plurality of candidate outputs includes, for each pair of candidate outputs: The second neural network is used to process inputs including instructions for generating the pairwise similarity scores and a pair of candidate outputs to generate the corresponding pairwise similarity scores for the pair of candidate outputs.

5. The method of claim 3, wherein using the second neural network to generate a corresponding pairwise similarity score for each of the plurality of candidate outputs comprises: The second neural network is used to process inputs including instructions for generating the corresponding pairwise similarity scores and the multiple pairs of candidate outputs to generate the corresponding pairwise similarity scores of the multiple pairs of candidate outputs.

6. The method of claim 4 or 5, wherein generating the similarity score from the corresponding pairwise similarity scores comprises: The cluster set of the candidate outputs is determined based on the corresponding pairwise similarity scores relative to the common candidate outputs.

7. The method of claim 6, wherein each cluster includes only candidate outputs having pairwise similarity scores for the common candidate outputs that are greater than a similarity threshold.

8. The method of claim 6 or 7, further comprising: The similarity score is determined based on the size of the largest value in the cluster. or The similarity score is determined based on the entropy of one or more of the clusters.

9. The method of claim 4 or 5, wherein generating the similarity score from the corresponding pairwise similarity scores comprises: A maximum count is determined, and for each candidate output, the count is used to count how many of the pairwise similarity scores of that candidate output are greater than a similarity threshold.

10. The method of any one of claims 1 to 9, wherein the similarity score includes a context similarity score, the context similarity score measuring the context similarity between the candidate outputs based on the context provided by the input prompt.

11. The method of any one of claims 1 to 10, wherein using a neural network to process the input cue to generate a plurality of candidate outputs for the input cue further comprises: The neural network is used to process the input prompts to generate multiple initial candidate outputs; as well as The first number of initial candidate outputs are identified as the plurality of candidate outputs.

12. The method of any of the preceding claims, wherein the abstention threshold has been determined by calibrating the abstention policy on a calibration set of cue-response pairs, each cue-response pair having a corresponding risk value indicating the inaccuracy of the response in the pair.

13. The method of claim 12, wherein the corresponding risk value for each cue-response pair in the calibration set has been determined using a risk prediction model.

14. The method of claim 12 or 13, wherein calibration includes adjusting the abstention threshold according to optimization, the optimization including for the cue-response pairs in the calibration dataset: Minimize the count of instances where the abstention strategy should have resulted in abstention but instead resulted in a response, while ensuring that the measure of risk under the abstention strategy meets the target risk tolerance criterion.

15. The method of claim 14, further comprising determining the count of instances in which the abstention strategy should have abstained without responding but instead responded, wherein the count of instances specifies a generated response to a first prompt in the first prompt-response pair when the corresponding risk value of the first prompt-response pair indicates an incorrect response.

16. The method of claim 15, wherein minimizing the count of instances in which the abstention policy should have abstained but did not, and instead responded, comprises using a loss function that indicates the count of instances in which the abstention policy should have abstained but did not, and instead responded.

17. The method of claim 16, wherein the measure of risk under the abstention strategy includes the expectation of the loss function.

18. The method of any one of claims 12 to 17 when subordinate to any one of claims 7 or 9, wherein the waiver strategy further includes the similarity threshold.

19. The method of claim 18, wherein the calibration further comprises adjusting the similarity threshold according to the optimization.

20. The method of any one of claims 14 to 19, wherein the optimization further comprises minimizing the count of instances in which the abstention policy should not have abstained but did abstained.

21. The method of claim 20, further comprising determining the count of instances in which the abstention strategy should not have abstained but did abstain without responding, wherein the count of instances specifies an abstention response to a first prompt in the first prompt-response pair when the corresponding risk value of the first prompt-response pair indicates a correct response.

22. The method of claim 21, wherein minimizing the count of instances in which the abstention policy should not have abstained but actually abstained without responding includes using a second loss function, the second loss function indicating the count of instances in which the abstention policy should not have abstained but actually abstained without responding.

23. The method of any one of claims 14 to 22, wherein the abstention threshold is a minimum abstention threshold.

24. The method of claim 4 or any claim dependent on claim 4, wherein the second neural network has fewer parameters than the first neural network.

25. The method of claim 4 or any claim dependent to claim 4, wherein the method is implemented on a parallel computing system comprising at least three sets of neural network processing hardware coupled to be capable of operating in parallel, and wherein: Using a neural network to process the input prompt to generate the plurality of candidate outputs for the input prompt includes: A first instance of the first neural network is implemented on a first set of the set of neural network processing hardware; A second instance of the first neural network is implemented on a second set of the neural network processing hardware; and Samples are taken from the first instance and the second instance of the first neural network to generate a first candidate output and a second candidate output in parallel; and Using the second neural network to generate the corresponding pairwise similarity score for each of the multiple candidate outputs includes implementing a first instance of the second neural network on a third set of the set of neural network processing hardware.

26. The method as claimed in any of the preceding claims, wherein: The first neural network is a trained visual language model that includes an image encoder neural network and a text encoder; The input prompt, the generated candidate output of the input prompt, or both include an image.

27. The method as claimed in any of the preceding claims, wherein: The input prompt includes image data and text data, and the candidate output includes information representing the image data and the text data; or The input prompt includes first image data and text data, and the candidate output includes second image data, which represents the first image data processed based on contextual information contained in the text data.

28. A computer-implemented method comprising performing conformal abandonment to determine an abandonment threshold, the method comprising: Receive a calibration set of prompt-response pairs, each prompt-response pair having a corresponding risk value indicating the inaccuracy of the response in the pair; as well as The calibration set is used to determine the abandonment strategy of the neural network, wherein the abandonment strategy defines when the neural network should abandon its response to a given input, and wherein determining the abandonment strategy includes: Minimize the count of instances where the abstention strategy should have resulted in abstention but instead resulted in a response, while ensuring that the measure of risk under the abstention strategy meets the target risk tolerance criterion.

29. A system comprising: One or more computers; And one or more storage devices storing instructions that, when executed by the one or more computers, are operable to cause the one or more computers to perform the method as described in any one of claims 1 to 28.

30. A computer storage medium encoded with a computer program, the computer program including instructions that, when executed by a data processing device, are operable to cause the data processing device to perform the method as described in any one of claims 1 to 28.

Images