Automatic evaluation of alignment with human preferences

Abstract

Described is a computing apparatus (500) for evaluating a scoring model (302), the computing apparatus (500) being configured to: obtain, for each of a plurality of items (204a, 204b, 204c, 204d) in an item set (204), an item score (205a, 205b, 205c, 205d); obtain, for each of the plurality of items (204a, 204b, 204c, 204d) in the item set (204), a scoring model-generated score (303a, 303b, 303c, 303d), the scoring model-generated score (303a, 303b, 303c, 303d) having been determined by a scoring model (302); and correlate the item score (205a, 205b, 205c, 205d) and the scoring model-generated score (303a, 303b, 303c, 303d); and evaluate the scoring model (302) in dependence on the correlation (301). In this way, the accuracy of the scoring model (302) may be evaluated based on a comparison to the item score (205a, 205b, 205c, 205d).

Description

[0001] AUTOMATIC EVALUATION OF ALIGNMENT WITH HUMAN PREFERENCES

[0002] FIELD OF THE INVENTION

[0003] This invention relates to an apparatus and method for evaluating a scoring model.

[0004] BACKGROUND

[0005] It’s generally accepted (in business and research) that the best Language Models / Personal Assistants (LMs / PAs) are aligned with Human Preferences (HP). The efficient and automatic evaluation of HP is still an unsolved problem. Figure 1 is a table comparing the main differences between prior art methods, and the present application. The table summarises and contrasts the most relevant methods of LM / PA evaluation, compared with the present system and broken down into individual steps (left to right). Details of HumanRankEval (the present system) is described herein.

[0006] All METHODS in the table may produce a numerical score that evaluates a particular aspect of LMs / PAs (GOAL). The JUDGE column specifies who determines the correct answer (ground truth). This could be humans or human-generated content (as a proxy), e.g. answers to a multiple-choice exam, a programming task, usage of tools / APIs and several other means to test the capabilities of LMs / PAs. In other cases, an LM / PA is the JUDGE. The METRIC column specifies the how to compute the final score given the JUDGE’S outputs, e.g., computing accuracy for a multiple-choice exam or computing pass@l for LMs / PAs that generate correct code for a given description.

[0007] Prior art systems may be categorised into the following three categories:

[0008] Human Evaluation - Present users with LM / PA outputs and ask them to score a) one or more aspects of HP, e.g. helpfulness, offensiveness, factual correctness, clarity, etc. or b) ask humans which of the presented outputs (usually from two different LMs / PAs) they prefer, or c) any combinations of a) and b) where humans directly judge the quality of LMs / PAs [Judging LLM-as-a-judge with MT- Bench and Chatbot Arena; and BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset], HPs are usually converted into Elo scores or Win-Rate percentages (scores are relative to other LMs / PAs). Automatic Evaluation (versus human text) - These methods may comprise evaluation tasks that have well-defined correct solutions to human-generated text(s). Tasks usually test multiple-choice questions [Measuring massive multitask language understanding], functional (code) correctness [Evaluating Large Language Models Trained on Code], success rate [AgentBench: Evaluating LLMs as Agents] with accuracy-like metrics on a range of tasks e.g. Tool / API usage [AgentBench: Evaluating LLMs as Agents], complex logical reasoning [Training verifiers to solve math word problems], school admission tests and math competitions [Agieval: A human-centric benchmark for evaluating foundation models]. The objective is not to test for HP, however, there is some correlation between HP and these metrics.

[0009] Automatic Evaluation (versus LM / PA text) - An emerging trend in automatic evaluation uses (extremely large and usually proprietary) LMs / PAs such as ChatGPT or GPT-4 as a faster substitute for human judgement. LMs / PAs are prompted to score the outputs of other LMs / PAs [Judging LLM- as-a-judge with MT -Bench and Chatbot Arena; and AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback] on a) aspects of human preferences, e.g. helpfulness, usefulness, safety, etc. or b) choosing one response over another. HP are converted into Elo scores or Win-Rate percentages (again, scores are relative to other LMs / PAs).

[0010] Prior art systems may have the following disadvantages:

[0011] Human Evaluation - This process may be costly, both in terms of time and money (logistics). Sourcing appropriate experts in multiple domains may be challenging plus human judges can be inconsistent and / or biased [Style Over Substance: Evaluation Biases for Large Language Models], Fast model iteration may be infeasible with this approach. Human evaluation requires testing multiple models as Elo scores (or Win-Rates) are relative to other LMs / PAs (no absolute scores).

[0012] Automatic Evaluation (versus human text) - Different tasks evaluate specific capabilities of LMs / PAs and they are useful to test models on such behaviours; however, they may not accurately estimate HP. Higher scores from these metrics may not mean humans will necessarily prefer these LMs / PAs.

[0013] Automatic Evaluation (versus LM / PA text) - This method (almost always) uses extremely large, proprietary LMs / PAs such as GPT-4 or ChatGPT. There may be no control over these LMs / PAs, they may be (and have been) shut down / changed at a short notice; there may be no transparency as we may not know the details of any training. Data privacy may be a problem. The agreement with humans is yet to be proven. It is desirable to develop an apparatus and method that overcomes the above problems.

[0014] SUMMARY

[0015] According to a first aspect, there is provided a computing apparatus for evaluating a scoring model, the computing apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code is executable by the one or more processors so that the computing apparatus is configured to: obtain, for each of a plurality of items in an item set, an item score; obtain, for each of the plurality of items in the item set, a scoring model-generated score, the scoring model-generated score having been determined by a scoring model; correlate the item score and the scoring model-generated score; and evaluate the scoring model in dependence on the correlation. In this way, the accuracy of the scoring model may be evaluated based on a comparison to the item score.

[0016] In some implementations, each of the plurality of items may comprise an output of an item model. In some implementations, the item set may comprise a plurality of items generated based on the same input by the item model. In this way, the accuracy of the scoring model may be evaluated when scoring an item model.

[0017] In some implementations, the item score may comprise one or more constituents. In some implementations, each constituent may comprise a characteristic of the item. In some implementations, the item score may comprise a composite of the constituents. In some implementations, the constituents may be based on one or more: of helpfulness, usefulness, safety, timeliness, clarity, and factual correctness of the item. In this way, the score may be broken down to cover different aspects of the item. The composition of the constituents may provide a simple summary of the aspects of the item.

[0018] In some implementations, the item score may be based on human preferences. In some implementations, the item score may have been determined by a human. In some implementations, the item score may have been determined by an item score model. In this way, the scoring model may be assessed based on its ability to score human preferences, and optionally against scores determined by a human. Alternatively, the scoring model may be assessed against scores determined by an item score model. In some implementations, the scoring model-generated score may comprise a simulated probability of the item being preferred by a human as determined by the scoring model. In this way, the scoring model may be assessed based on its ability to simulate human choices.

[0019] In some implementations, the computing apparatus may be configured to: select a sub-set of the items of the item set based on the item score, and obtain a scoring model-generated score for only the selected sub-set of items of the item set. In this way, the items may be filtered so that the scoring model is only required to determine scores for a sub-set of items, improving efficiency.

[0020] In some implementations, the computing apparatus may be configured to correlate the item score and the scoring model-generated score based on a product of the item score and the scoring model-generated score. In some implementations, the computing apparatus may be configured to correlate the item score and the scoring model-generated score based on a correlation function of the item score and the scoring model-generated score. In some implementations the correlation function may be a Pearson correlation. In some implementations the correlation function may be a Spearman correlation. In this way, the correlation may be carried out in an efficient manner.

[0021] In some implementations, the computing apparatus may be configured to generate the plurality of items using the item model. In this way, the apparatus may generate the items itself, providing a self-sufficient evaluation apparatus.

[0022] In some implementations, the computing apparatus may be configured to determine the scoring model-generated scores using the scoring model and based on the plurality of items. In this way, the apparatus may generate the scoring model-generated scores itself, providing a self- sufficient evaluation apparatus.

[0023] In some implementations, the item score may not be determined by the scoring model. In this way, the evaluation of the scoring model is not biased towards the giving itself a good evaluation.

[0024] In some implementations, the computing apparatus may be configured to obtain, for each of the plurality of items in the item set, a plurality of item scores. In some implementations, each of the plurality of item scores for an item in the item set may have been determined by a different human. In this way, the evaluation may be based on a range of different human’s preferences, which may test the scoring model’s ability to allow for different human types.

[0025] In some implementations, the computing apparatus may be configured to carry out the steps of any preceding statement for a plurality of inputs of the item model so as to evaluate the scoring model based on all the plurality of inputs. In this way, the evaluation may be based on a range of different inputs to the item model, which may test the scoring model’s ability to allow for different input types.

[0026] In some implementations, the item model may comprise a language model, the output items of the item model may comprise text, and the scoring model-generated scores may be numerical or ordinal scores. In some implementations, the scoring model may comprise a language model. In some implementations, language model may comprise one or more of a neural model, an artificial model, a statistical model, and a multi-modal language model. In this way, the scoring model evaluation may be applied to a language model setting.

[0027] In some implementations, the item model may comprise one or more of an image generation model, a sound generation model, a video generation model, and a physical item generation model. In some implementations, the scoring model may comprise one or more of an image model, a sound model, a video model, and a physical item model. In this way, the scoring model evaluation may be applied to a non-language model setting.

[0028] In some implementations, the computing apparatus may be configured to output the evaluation of the scoring model. In this way, the evaluation may be used for a further use. In some implementations, the computing apparatus may be configured to use the evaluation of the scoring model to train the scoring model. In this way, the evaluation may be used to improve the scoring model. In some implementations, the computing apparatus may be configured to use the evaluation of the scoring model to train a different model. In this way, the evaluation may be used to improve a different model.

[0029] According to a second aspect, there is provided a method for evaluating a scoring model, the method comprising steps of obtaining, for each of a plurality of items in an item set, an item score; obtaining, for each of the plurality of items in the item set, a scoring model-generated score, the scoring model-generated score having been determined by a scoring model; correlating the item score and the scoring model-generated score; and evaluating the scoring model in dependence on the correlation. In this way, the accuracy of the scoring model may be evaluated based on a comparison to the item score.

[0030] BRIEF DESCRIPTION OF THE FIGURES

[0031] The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0032] Figure 1 is a table comparing the differences between prior art methods, and the present application.

[0033] Figure 2 schematically illustrates an item generation pipeline of the present application.

[0034] Figure 3 schematically illustrates a scoring model evaluation pipeline of the present application.

[0035] Figure 4 shows an example of a computer implemented method for evaluating a scoring model.

[0036] Figure 5 shows an example of an apparatus configured to perform the methods described herein.

[0037] DETAILED DESCRIPTION

[0038] The apparatuses and methods described herein concern the evaluation of a scoring model.

[0039] Embodiments of the present invention may tackle one or more of the problems previously mentioned by: obtaining, for each of the plurality of items in the item set, a scoring modelgenerated score, and evaluating the scoring model in dependence on the correlation between the item score and the scoring model-generated score. In this way, the accuracy of the scoring model may be evaluated based on a comparison to the item score.

[0040] The following terms may be used throughout the application: Language Model (LM): a mechanism for representing language (natural and / or artificial) as a probability distribution over words / tokens capable of outputting text sequences and possibly outputting the probabilities of individual words / tokens in those text sequences.

[0041] Personal Assistants (PA): LMs that are typically (but not necessarily) tuned / trained to follow instructions in order to perform (usually to assist humans with) a variety of tasks such as coding, writing, question answering, summarisation and other related tasks (examples include ChatGPT or GPT-4).

[0042] Human Preferences (HP): refers to the quality / type of outputs that humans prefer to receive from PAs, typically expressed as choosing one out of multiple outputs / answers. NOTE'. Humans generally prefer outputs that are helpful, useful, inoffensive, clear, safe, correct, factual, etc. However, the present system may treat HP as a black box, no attribution, even knowledge of constituent scores mentioned above is necessary to evaluate HP.

[0043] The steps of the present application may be illustrated by the HumanRankEval row in Figure 1. The individual steps of HumanRankEval may provide a better evaluation of HP than previous work.

[0044] The present system may solve the following problems:

[0045] How to represent / capture HP in an automatic manner? Manual collection, e.g., via crowdsourcing may be extremely challenging as there is a need to capture the naturalness, spontaneity and open-endedness of outputs carefully scored by thousands of enthusiasts and experts on real-world topics / problems.

[0046] How to assign a score to LMs / PAs that correlates well with human preferences? It may be desirable to have an efficient means to determine this and adjust model training accordingly.

[0047] The present system may provide the following advantages:

[0048] A medium that can capture / represent HP.

[0049] May avoid costs of crowdsourcing this type of data.

[0050] May avoid risks of unnatural data being collected (difficult to find enough domain experts to mimic such open-ended questions and answers)

[0051] A process that accurately estimates LM’s / PA’s alignment with HP.

[0052] HumanRankEval scores may be well-aligned with human preferences (unlike common automatic benchmarks or LLMs-as-judges). Full control, transparency and privacy (unlike using proprietary and changing APIs) may be provided.

[0053] No sampling (usually stochastic) from the LM / PA may be required, making replication fast & stable.

[0054] The present system may solve the aforementioned problems through the following steps: Compile a dataset (from public sources or private sources) in a novel format that is able to capture / represent HP. This may take shape as multiple answers to information-seeking questions where each answer can be scored by (on average dozens of) experts in that domain. Users may naturally interact with each other and collectively determine each answer score, which can be used to compute our novel metric.

[0055] Query a scoring model LM / PA by gathering the probabilities of each answer (text, completion or output), under its current distribution (its trained weights).

[0056] Correlate the human scores and model probabilities to obtain the HumanRankEval score. A higher score means the scoring model is more aligned with human preferences (the desirable outcome).

[0057] HumanRankEval may be intended for fast and accurate automatic evaluation of LMs / PAs, at any stage of development, in order to estimate how well the LM / PA is aligned with HP (best PAs have high alignment). The metric can be used to adjust the training / development in an efficient and cost-effective manner.

[0058] The HumanRankEval may have access to the LM / PA outputs, i.e. token / word probabilities, not only the textual content such as found in web-based APIs. Therefore, the HumanRankEval metric may be suitable for LM / PA owners and / or open-source LMs / PAs that can be readily queried for such probabilities. Additionally, HP capture multi-dimensional qualities of LMs / PAs and HumanRankEval may simply provide a score for this composite, rather than for its constituents, e.g. helpfulness, usefulness, safety, timeliness, clarity, etc.

[0059] HumanRankEval can be used in products using LMs / PAs, e.g. a Programming or Writing Assistant (Copilot). Its biggest value is in time / cost savings, relative to developing and evaluating LMs / PAs by obtaining human feedback, which would be slow / expensive. HumanRankEval is not a replacement for human judgement (e.g. final decision to deploy the LM / PA to the end user), however, it is intended to accelerate the iterative development of LMs / PAs by enabling frequent evaluation (and experimentation) that is well-aligned with known HP.

[0060] The present system, HumanRankEval, may comprise the following steps: 1) Obtain human scores (Data Preparation), 2) Obtain LM / PA probabilities, 3) Correlate them to compute HumanRankEval score. The goal may be to evaluate LMs / PAs for alignment with HP.

[0061] Figure 2 schematically illustrates an item generation pipeline 200 of the present application.

[0062] A plurality of items 204a, 204b, 204c, 204d may be generated. The plurality of items 204a, 204b, 204c, 204d may form an item set 204. Each of the plurality of items 204a, 204b, 204c, 204d in the item set 204 my share a common feature. Each of the plurality of items 204a, 204b, 204c, 204d have been generated from the same input 203. In the example in Figure 2, four items 204a, 204b, 204c, 204d are shown. However, the number of items 204a, 204b, 204c, 204d may be varied depending on the implementation.

[0063] Each of the plurality of items 204a, 204b, 204c, 204d may comprise an output from an item model 201. An input 203 may be inputted into the item model 201 and the plurality of items 204a, 204b, 204c, 204d may be outputted. Each of the plurality of items 204a, 204b, 204c, 204d may have been generated from the same input 201. The item model 201 may form part of the apparatus such that the apparatus generates the plurality of items 204a, 204b, 204c, 204d. Alternatively, the item model 201 may be separate to the apparatus, and the plurality of items 204a, 204b, 204c, 204d may be received by the apparatus.

[0064] In the example shown in Figure 2, the input 203 may comprise a question 203. The output items may comprise answers 204a, 204b, 204c, 204d. The question 203 may have different answers 204a, 204b, 204c, 204d. In this case, the item model 201 may comprise a language model. As such, the input 203 and the outputs 204 comprise language. More generally, when the item model 201 comprises a language model, the output items 204 of the item model 201 may comprise text. The language model may comprise one or more of: a neural model, an artificial model, a statistical model, and a multi-modal language model. Alternatively, the item model 201 may be a non-language model. The item model 201 may comprise one or more of: an image generation model, a sound generation model, a video generation model, and a physical item generation model. In such case, the outputs 204 may comprise an image, a sound, a video, or a physical item respectively. The physical item may take the form of a 3D printed output.

[0065] In the case where an item model 201 is not used, the items 204a, 204b, 204c, 204d may have been generated by other means. For example, the items 204a, 204b, 204c, 204d could simply be physical objects or products. The items 204a, 204b, 204c, 204d may later be scored as described herein.

[0066] A plurality of item scores 205a, 205b, 205c, 205d may be obtained. The item scores 205a, 205b, 205c, 205d may correspond to the plurality of items 204a, 204b, 204c, 204d in the item set 204. There may be an item score 205 for each of the plurality of items 204a, 204b, 204c, 204d in the item set 204.

[0067] The plurality of item scores 205a, 205b, 205c, 205d may be obtained without needing to generate the plurality of item scores 205a, 205b, 205c, 205d. In other words, the plurality of item scores 205a, 205b, 205c, 205d may be received by the apparatus from another system. The plurality of item scores 205a, 205b, 205c, 205d may have been generated by a human. The human may generate an item score 205 for each of the plurality of items 204a, 204b, 204c, 204d in the item set 204. In other words, the human may have allocated an item score 205 to each of the plurality of items 204a, 204b, 204c, 204d.

[0068] Alternatively, the plurality of item scores 205a, 205b, 205c, 205d may have generated by an item score model. The item score model may generate an item score 205 for each of the plurality of items 204a, 204b, 204c, 204d in the item set 204. In other words, the item score model may have allocated an item score 205 to each of the plurality of items 204a, 204b, 204c, 204d. The item score model may comprise any scoring machine. The item score model may comprise a machine learning model. The item score model may output the item scores 205a, 205b, 205c, 205d to the apparatus. Alternatively, the item model 201 may form part of the apparatus, and the item model as a part of the apparatus may generate the plurality of item scores 205a, 205b, 205c, 205d. For each of the plurality of items 204a, 204b, 204c, 204d in an item set 204, a plurality of item scores 205a, 205b, 205c, 205d may be obtained. In the case of a human determining the item scores 205a, 205b, 205c, 205d, a plurality of different humans may each determine an item score 205 for each of the plurality of items 204a, 204b, 204c, 204d in an item set 204. In the case of an item score model determining the item scores 205a, 205b, 205c, 205d, a plurality of different item score models may each determine an item score 205 for each of the plurality of items 204a, 204b, 204c, 204d in an item set 204. Alternatively, the same item score model may determine a plurality item score 205 for each of the plurality of items 204a, 204b, 204c, 204d in an item set 204. In this later case, the item score model may use different criteria or metrics for each of the different item scores 205 for the same item 204a, 204b, 204c, 204d.

[0069] The item score 205 may comprise one or more constituents. Each constituent may comprise a characteristic of the item 205. In other words, the item score 205 may be based on one or more values associated with characteristics of the item 205. The item score 205 may comprise a composite of the constituents. The item score 205 may be made up from each of the constituents. The item score 205 may comprise a sum of the constituents. The item score 205 may comprise a product of the constituents. In this way, the composition of the constituents may provide a simple summary of the aspects of the item. The constituents may be based on one or more: of helpfulness, usefulness, safety, timeliness, clarity, and factual correctness of the item 205. For example, the item score 205 may comprise 6 for helpfulness, 2 for usefulness, 5 for safety, 3 for timeliness, 5 for clarity, and 4 factual correctness which may total in 25, e.g., the first item score 205a. In this way, the item score 205 may be tailored for use with human preferences.

[0070] A sub-set of the items 204 may be selected. A sub-set of the items 204 may be selected by a filter 202. The sub-set of the items 204 may be filtered based on their item score 205. For example, the items 204 the lowest item scores 205 may be filtered out. The items 204 with an item score 205 below an item score 205 threshold may be filtered out. Alternatively, a certain proportion of the items 204 with the lower item scores 205 may be filtered out. In the example shown in Figure 2, the items 204 with the four highest item scores 205 were kept after filtering. In this way, the more accurate items 204, based on their item scores 205, may be kept.

[0071] The details in Figure 2 show a specific embodiment of the item generation pipeline 200. Human-generated data (7,000 questions in total) may be gathered from public communities (StackOverflow, StackExchange). The data may be filtered 202 to select high-quality pairs (minimum of 5 votes from domain experts per answer, at least 40 votes per question, maximum of 4,000 characters for each question-answer pair, no duplicates, etc.). The apparatus may keep the top 4 answers 204 and their scores 205.

[0072] The sources of content, the number of questions and answers (or sentences, or any other multimodal content) and the filtering details may all vary. The quality score for answers / outputs may be expressed in a numerical way, e.g., integers or floats.

[0073] Figure 3 schematically illustrates a scoring model evaluation pipeline of the present application.

[0074] A plurality of scoring model-generated scores 303a, 303b, 303c, 303d may be obtained. The scoring model-generated scores 303a, 303b, 303c, 303d may correspond to the plurality of items 204a, 204b, 204c, 204d in the item set 204. There may be a scoring model-generated score 303 for each of the plurality of items 204a, 204b, 204c, 204d in the item set 204.

[0075] The plurality of scoring model-generated scores 303a, 303b, 303c, 303d may be obtained without needing to generate the plurality of scoring model-generated scores 303a, 303b, 303c, 303d. In other words, the plurality of scoring model-generated scores 303a, 303b, 303c, 303d may be received by the apparatus from another system. The plurality of scoring modelgenerated scores 303a, 303b, 303c, 303d may have been generated by a scoring model 302. The scoring model 302 may generate a scoring model-generated scores 303a, 303b, 303c, 303d for each of the plurality of items 204a, 204b, 204c, 204d in the item set 204. In other words, the scoring model 302 may have allocated a scoring model-generated scores 303 to each of the plurality of items 204a, 204b, 204c, 204d. The scoring model 302 may comprise a machine learning model. The scoring model 302 may output the scoring model-generated scores 303a, 303b, 303c, 303d to the apparatus. Alternatively, the scoring model 302 may form part of the apparatus, and the scoring model 302 as a part of the apparatus may generate the plurality of scoring model-generated scores 303a, 303b, 303c, 303d.

[0076] The scoring model-generated score 303a, 303b, 303c, 303d may comprise numerical or ordinal scores. In the case of numerical scores, the scoring model-generated scores 303a, 303b, 303c, 303d may comprise a simulated probability of the item 204a, 204b, 204c, 204d being preferred by a human as determined by the scoring model 302. In other words, the scoring model- generated scores 303a, 303b, 303 c, 303d may give a likelihood that out of all the items 204a, 204b, 204c, 204d, a human would select that item 204a, 204b, 204c, 204d. The probability value may be out of 1. The total probability of all the scoring model-generated scores 303a, 303b, 303c, 303d may sum to 1. Therefore, the probability of each scoring model-generated score 303a, 303b, 303c, 303d is equal to or less than 1. As shown in the example in Figure 2, the scoring model-generated scores 303a, 303b, 303c, 303d are 0.5, 0.11, 0.1, and 0.02. Item 204a has the highest scoring model-generated score 303a of 0.5, and so it is most likely to be selected by a human, as determined by the scoring model 302. Alternatively, a log-likelihood probability value may be used instead of a probability out of 1. In the case of ordinal scores, the scoring model 302 may rank each of the items 204a, 204b, 204c, 204d from most to least preferred by a human as determined by the scoring model 302.

[0077] In the example shown in Figures 2 and 3, the input 203 may comprise a question 203. The output items may comprise answers 204a, 204b, 204c, 204d. The question 203 may have different answers 204a, 204b, 204c, 204d. In this case, the scoring model 302 may comprise a language model. As such, scoring model 302 may be able to score the items 204a, 204b, 204c, 204d, which comprise text. The language model may comprise one or more of: a neural model, an artificial model, a statistical model, and a multi-modal language model.

[0078] Alternatively, in the case where the item model 201 is not a non-language model, the scoring model 302 may be a non-language model. The scoring model 302 may comprise one or more of an image model, a sound model, a video model, and a physical item model. As such, scoring model 302 may be able to score the items 204a, 204b, 204c, 204d, which may comprise an image, a sound, a video, or a physical item respectively. The physical item may take the form of a 3D printed output.

[0079] The item score 205 may not be determined by the scoring model 302. In other words, the scoring model 302 may not be the same model as the item score model. As such, the scoring model 302 is not used to generate both the item score 205 and the scoring model-generated score 303. In this way, the evaluation of the scoring model 302 is not biased towards the giving itself a good evaluation.

[0080] The item scores 205a, 205b, 205c, 205d and the scoring model-generated scores 303a, 303b, 303c, 303d may be correlated. In this way, the scoring model-generated scores 303a, 303b, 303c, 303d of the scoring model 302 may be compared against the item scores 205a, 205b, 205c, 205d. In the case of human generated item scores 205a, 205b, 205c, 205d, the scoring model-generated scores 303a, 303b, 303c, 303d of the scoring model 302 may be compared against human preferences.

[0081] As shown in Figure 3, the item scores 205a, 205b, 205c, 205d and the scoring model-generated scores 303a, 303b, 303c, 303d may be correlated based on a correlation function 301 of the item score 205a, 205b, 205c, 205d and the scoring model-generated score 303a, 303b, 303c, 303 d. In other words, for each of the items 204a, 204b, 204c, 204d, the correlation function 301 of the item score 205a, 205b, 205c, 205d and the scoring model-generated score 303a, 303b, 303c, 303d may be used. The correlation function 301 may comprise a Pearson correlation. The correlation function 301 may comprise a Spearman correlation.

[0082] Alternatively, the item scores 205a, 205b, 205c, 205d and the scoring model-generated scores 303a, 303b, 303c, 303d may be correlated based on a product of the item score 205a, 205b, 205c, 205d and the scoring model-generated score 303a, 303b, 303c, 303d. In other words, for each of the items 204a, 204b, 204c, 204d, the correlation may be a product of the item score 205a, 205b, 205c, 205d and the scoring model-generated score 303a, 303b, 303c, 303d.

[0083] In dependence on the correlation between the item scores 205a, 205b, 205c, 205d and the scoring model-generated scores 303a, 303b, 303c, 303d, the scoring model 302 may be evaluated. The closer the scoring model-generated scores 303a, 303b, 303c, 303d are to the preferred item scores 205a, 205b, 205c, 205d, the better the evaluation may be. The evaluation may comprise an evaluation score. The evaluation score may provide a metric for the accuracy, or suitability, of the scoring model 302.

[0084] The process described herein, with reference to Figure 2 and 3, may be used for a plurality of inputs 203 to the item model. In other words, a plurality of different inputs 203 may be used. Each of the inputs 203 may result in a plurality of outputs 204. Item scores 205 and scoring model-generated scores 303 may be obtained and / or generated for each of the outputs 204 generated from each of the inputs 203. Each of the inputs 203 may be used at once, or sequentially. The scoring model 302 may be evaluated based on the plurality of inputs 203. Using the example of questions 203 and answers 204 in Figures 2 and 3, a plurality of questions 203 may be asked to the item model 201 resulting in a plurality of answers 204 for each question 203. In this way, a larger range of inputs 203 may be used to evaluate the scoring model 203.

[0085] The evaluation of the scoring model 302 may be used for a range of further purposes. The evaluation of the scoring model 302 may be outputted. The evaluation of the scoring model 302 may be outputted for use by a different system. For example, a plurality of different scoring models 302 may be evaluated and compared. In this way, a preferred scoring model 302 may be selected for use.

[0086] The evaluation of the scoring model 302 may be used to train the scoring model 302. In the case of a machine learning scoring model 302, the evaluation of the scoring model 302 may be used to train the scoring model 302 through back-propagation. The training of the scoring model 302 based on the evaluation of the scoring model 302 may be automatic or manual. The training of the scoring model 302 may be continuous or periodic during use of the scoring model 302. The evaluation of the scoring model 302 may also be used to train different models. For example, other scoring models could be trained based on the evaluated scoring model 302.

[0087] The details in Figure 3 show a specific embodiment of the scoring model evaluation pipeline 300. The present system may obtain a probability score 303 from the LM / PA 302 for each answer 204 (or text sequence / output). Logits may be gathered for each token in the answer 204 from the LM / PA 302. Log and softmax functions may be used, and then an average of those probabilities or log-likelihoods 303 (using character rather than token sequence length) may be used.

[0088] Human scores (25, 17, 11, 5) 205 may be correlated with LM / PA probabilities (0.5, 0.11, 0.1, 0.02) 303 to obtain the HumanRankEval score. Correlation 301 may be computed with any equivalents of Pearson coefficient such as Spearman Rank or another correlation-like function (including using accuracy-like metrics, if applicable). A higher HumanRankEval score may indicate a better alignment with HP.

[0089] The present system may be adapted in the following ways:

[0090] The LM / PA probabilities 303 for each answer 204 (or any text sequence, or any output) could be obtained without access to the individual word / token scores, e.g., provide all answers 204 in the prompt and request the LM / PA 302 to provide the scores for all of them in its output (this could work for API-based LMs / PAs such as ChatGPT).

[0091] HumanRankEval could be extended to include the HP constituents such as helpfulness, safety, clarity, factual correctness, usefulness, timeliness, etc. if those are available.

[0092] HumanRankEval could be extended to evaluate specialist topics and / or multilingual applications (both natural and / or programming languages). This is not necessarily restricted to the goal of matching human preferences over these topics, but could be applied to matching any preference criteria.

[0093] HumanRankEval can be extended beyond language outputs to evaluate any kind of output based on preference, e.g., images or sounds (multi-modal). Examples include evaluating HP for Stable Diffusion images or Text-To- Speech (TTS) outputs or some other Generative Al content (Music, Video).

[0094] HumanRankEval could be used as a training loss, i.e., by backpropagating the correlation of human scores 205 and LM / PA probabilities 303 during instruction-tuning, for example.

[0095] HumanRankEval could gather answers / outputs 204 from LMs / PAs (instead of humans) with a well-known quality, using deterministic generation, then correlating their probabilities, i.e., score LM / PA outputs.

[0096] Figure 4 summarises an example of a method 400 for evaluating a scoring model. At step 401, the method 400 comprises obtaining, for each of a plurality of items in an item set, an item score. At step 402, the method 400 comprises obtaining, for each of the plurality of items in the item set, a scoring model-generated score, the scoring model-generated score having been determined by a scoring model. At step 403, the method 400 comprises correlating the item score and the scoring model-generated score. At step 404, the method 400 comprises evaluating the scoring model in dependence on the correlation.

[0097] An example of an apparatus 500 configured to implement the method 400 is schematically illustrated in Figure 5. The computing apparatus 500 may comprise the pipeline 200, 300. The apparatus 500 may be implemented on an electronic device, such as a computer, laptop, tablet, or smart phone.

[0098] The apparatus 500 comprises a processor 501 configured to process the datasets in the manner described herein. For example, the processor 501 may be implemented as a computer program running on a programmable device such as a Central Processing Unit (CPU). The apparatus 500 comprises a memory 502 which is arranged to communicate with the processor 501. Memory 502 may be a non-volatile memory. The processor 501 may also comprise a cache (not shown in Figure 5), which may be used to temporarily store data from memory 502. The apparatus 500 may comprise more than one processor 501 and more than one memory 502. The memory 502 may store data that is executable by the processor 501. The processor 501 may be configured to operate in accordance with a computer program stored in non-transitory form on a machine-readable storage medium. The computer program may store instructions for causing the processor to perform its methods in the manner described herein. The method steps described herein may be carried out by a computer-readable storage medium. The method steps described herein may be carried out by a computer program product.

[0099] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description, it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A computing apparatus (500) for evaluating a scoring model (302), the computing apparatus (500) comprising one or more processors (501) and a memory (502) storing in nontransient form data defining program code executable by the one or more processors (501), wherein the program code is executable by the one or more processors (501) so that the computing apparatus (500) is configured to: obtain, for each of a plurality of items (204a, 204b, 204c, 204d) in an item set (204), an item score (205a, 205b, 205c, 205d); obtain, for each of the plurality of items (204a, 204b, 204c, 204d) in the item set (204), a scoring model-generated score (303a, 303b, 303c, 303d), the scoring modelgenerated score (303a, 303b, 303c, 303d) having been determined by a scoring model (302); correlate the item score (205a, 205b, 205c, 205d) and the scoring modelgenerated score (303a, 303b, 303c, 303d); and evaluate the scoring model (302) in dependence on the correlation (301).

2. The computing apparatus (500) according to claim 1, wherein each of the plurality of items (204a, 204b, 204c, 204d) comprise an output of an item model (201), and wherein the item set (204) comprises a plurality of items (204a, 204b, 204c, 204d) generated based on the same input (203) by the item model (201).

3. The computing apparatus (500) according to claim 1 or 2, wherein the item score (205a, 205b, 205c, 205d) comprises one or more constituents, each constituent comprising a characteristic of the item (204a, 204b, 204c, 204d); and optionally wherein the item score (205a, 205b, 205c, 205d) comprises a composite of the constituents; and optionally wherein the constituents are based on one or more: of helpfulness, usefulness, safety, timeliness, clarity, and factual correctness of the item.

4. The computing apparatus (500) according to any preceding claim, wherein the item score (205a, 205b, 205c, 205d) is based on human preferences, and optionally wherein the item score (205a, 205b, 205c, 205d) has been determined by a human and / or the item score (205a, 205b, 205c, 205d) has been determined by an item score model.

5. The computing apparatus (500) according to any preceding claim, wherein the scoring model-generated score (303a, 303b, 303c, 303d) comprises a simulated probability of the item (204a, 204b, 204c, 204d) being preferred by a human as determined by the scoring model (302).

6. The computing apparatus (500) according to any preceding claim, wherein the apparatus (500) is configured to: select a sub-set of the items (204a, 204b, 204c, 204d) of the item set (204) based on the item score (205a, 205b, 205c, 205d), and obtain a scoring model-generated score (303a, 303b, 303c, 303d) for only the selected sub-set of items (204a, 204b, 204c, 204d) of the item set (204).

7. The computing apparatus (500) according to any preceding claim, wherein the apparatus (500) is configured to correlate the item score (205a, 205b, 205c, 205d) and the scoring modelgenerated score (303 a, 303b, 303 c, 303 d) based on one or more of: a product of the item score (205a, 205b, 205c, 205d) and the scoring model-generated score (303a, 303b, 303c, 303d); or a correlation function of the item score (205a, 205b, 205c, 205d) and the scoring modelgenerated score (303a, 303b, 303c, 303d), such as a Pearson correlation or a Spearman correlation.

8. The computing apparatus (500) according to any preceding claim, wherein the apparatus (500) is configured to: when dependent on claim 2, generate the plurality of items (204a, 204b, 204c, 204d) using the item model (201); and / or determine the scoring model-generated scores (303a, 303b, 303c, 303d) using the scoring model (302) and based on the plurality of items (204a, 204b, 204c, 204d).

9. The computing apparatus (500) according to any of claims 2 to 8, wherein the item score (205a, 205b, 205c, 205d) is not determined by the scoring model (302).

10. The computing apparatus (500) according to any preceding claim, wherein the apparatus (500) is configured to obtain, for each of the plurality of items (204a, 204b, 204c, 204d) in the item set (204), a plurality of item scores (205a, 205b, 205c, 205d), each of the plurality of itemscores (205a, 205b, 205c, 205d) for an item (204a, 204b, 204c, 204d) in the item set (204) having been determined by a different human.

11. The computing apparatus (500) according to any of claims 2 to 10, wherein the apparatus (500) is configured to carry out the steps of any preceding claim for a plurality of inputs (203) of the item model (201) so as to evaluate the scoring model (302) based on all the plurality of inputs (203).

12. The computing apparatus (500) according to any of claims 2 to 11, wherein the item model (201) comprises a language model, the output items (204a, 204b, 204c, 204d) of the item model comprise text, and the scoring model-generated scores are numerical or ordinal scores; and optionally wherein the scoring model (302) comprises a language model; and optionally wherein the language model comprises one or more of a neural model, an artificial model, a statistical model, and a multi-modal language model.

13. The computing apparatus (500) according to any of claims 2 to 11, wherein the item model (201) comprises one or more of: an image generation model, a sound generation model, a video generation model, and a physical item generation model; and optionally wherein the scoring model (302) comprises one or more of: an image model, a sound model, a video model, and a physical item model.

14. The computing apparatus (500) according to any preceding claim, wherein the apparatus (500) is configured to: output the evaluation of the scoring model (302); use the evaluation of the scoring model (302) to train the scoring model (302); and / or use the evaluation of the scoring model (302) to train a different model.

15. A method (400) for evaluating a scoring model, the method (400) comprising steps of: obtaining, for each of a plurality of items in an item set, an item score (401); obtaining, for each of the plurality of items in the item set, a scoring modelgenerated score, the scoring model-generated score having been determined by a scoring model (402); correlating the item score and the scoring model-generated score (403); andevaluating the scoring model in dependence on the correlation (404).

Images