Incremental minimum bayes decoding for code synthesis

Abstract

Described is a computing apparatus (600) for generating a sequential model output (403), the computing apparatus (600) being configured to: obtain a model input (101); using a model (102), generate a plurality of first candidate output sub-sequences (403-1, 403-2, 403-N) from the model input (101), each of the first candidate output sub-sequences (403-1, 403-2, 403-N) being code forming a first segment of the sequential model output (403); analyse each of the first candidate output sub-sequences (403-1, 403-2, 403-N) to select one of the first candidate output sub-sequences (403-1, 403-2, 403-N); and generate the sequential model output (403) based on the selected first candidate output sub-sequence (403-1, 403-2, 403-N) forming the first segment of the sequential mode output (403). In this way, the best first candidate output sub-sequence (403-1, 403-2, 403-N) may be selected for use in generating the sequential model output (403).

Description

[0001] INCREMENTAL MINIMUM BAYES DECODING FOR CODE SYNTHESIS

[0002] FIELD OF THE INVENTION

[0003] This invention relates to an apparatus and method for generating a sequential model output.

[0004] BACKGROUND

[0005] Structured prediction refers to the family of machine learning methods that aim to predict some structured output. Sequence prediction aims to progressively generate a sequence of actions. I.e., at any step, it aims to predict the next item in a sequence based on previous observations (input, context, and sequence produced so far).

[0006] A machine learning model (pretrained or otherwise), hereby referred to as the “Model”, can be adapted through a training process to perform such a task, taking as input the previous observations, and mapping them to the parameters of a probability or confidence distribution over a set of next actions. For language in particular, an example of such a model could be a Pretrained Language Model (PLM), given as input a textual user input and a dialogue context, and asked to produce a response. The response could be formed via a sequence of output actions each consisting of a character, word, or other subunit of language. The process of producing the output is also referred to as decoding or inference. Figure 1 schematically illustrates a prior art pipeline for generating a sequential model output. The input 101 is fed into the model 102. The model 102 outputs the sequence of actions in the sequential model output 103.

[0007] In order to pursue output quality, a system should ideally locate the mode of the model’s distribution, i.e., the output sequence with the highest probability according to the model. As calculating this over all potential sequences of actions may be intractable, a system may seek a point estimate of the most probable output under the model. One way to estimate the mode of the distribution, often used in generative language models, is maximum a posteriori (MAP) decoding, i.e., iteratively selecting the most probable next action given the sub-sequence of previous actions - as illustrated by Equation 1. pW = n?=i P(xi\x1-- xi-1) (1) However, MAP decoding and its approximations may underperform. Stahlberg and Byrne [Felix Stahlberg and Bill Byrne 2019 On NMT Search Errors and Model Errors: Cat Got Your Tongue?] show that the true mode may often inadequately short or empty and conclude that the success of MAP decoding depends on search biases introduced by hyperparameters such as beam size and length normalisation, which are not tuned to correlate with the MAP objective of locating the mode. These biases include a bias towards shorted sequences, an exposure bias towards prefixes that the model was exposed to during training, and a bias towards more frequent tokens due to a focus on competing on the token-level.

[0008] A particular shortcoming of MAP methods may be the assumption of a single best output existing, while in practice multiple good outputs exist. Since good outputs are likely to be similar to each other, they compete for the same part of probability mass, while a worse sequence may get a higher assigned probability due to having less competition.

[0009] Alternatively, instead of pursuing the mode, actions may be randomly sampled based on the distribution provided by the model. Often the model’s distribution can be further manipulated by methods like temperature scaling, top-k or nucleus sampling. The benefit of random sampling is that the resulting output may be more representative of a model’s capabilities than the narrow probability mass the mode represents, but this may not offer theoretical guarantees for output sequence quality.

[0010] Both decoding strategies may be compatible with over-generate and reranked methods, where multiple candidate outputs are generated (via sampling or beam search), and then reranked through an evaluation function to pick the best.

[0011] To address the shortcomings of MAP decoding, Minimum Bayes Risk (MBR) decoding was proposed [Vaibhava Goel and William J Byrne. 2000. Minimum bayes-risk automatic speech recognition. Comput. Speech Lang, and Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. NAACL], The MBR decoder may be thought of as selecting a consensus output sequence: For a given input, a number of potential outputs are randomly sampled. Afterwards, all of them are compared against each other with some similarity function. The output sequence that is closest on average to all other potential outputs, is deemed to be the best. Code Synthesis or text-to-code generation is the task of generating a Programming Language code based on a Natural Language description of the problem. Figure 2 schematically illustrates a prior art pipeline for code synthesis. Considering the generation of Programming Language code as a sequence of actions, Code Synthesis can be considered a sequence prediction problem, potentially approached with the aforementioned models and decoding methods. The Code Synthesis model employed here could be a Code PLM, i.e., a PLM pretrained on Programming Language code as well as natural language. The instructions 201 may be inputted into the code synthesis model 202. The code synthesis model 202 may generate and output the programming language code 203.

[0012] The task of Code Synthesis benefits from the fact that the generate Programming Language code is functional, i.e., it can be executed given some input and it will return some output. Following this, such Models can be evaluated on their functional behaviour through the use of Unit Tests (i.e., pairs of inputs and expected outputs); the generated Programming Language code is checked on whether given specific inputs, it returns the correct corresponding output.

[0013] Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. 2022 Natural Language to Code Translation with Execution. EMNLP introduced execution outputbased MBR decoding for Code PLMs on Code Synthesis tasks. This is illustrated in Figure 3. To perform MBR the system compares whether Model output codes exhibit the same behaviour when executed. Specifically, the system feeds an example prompt and input to a Code PLM (i.e. the CodeX model in the illustration), sample some Model output codes (Codes #1, #2, and #3), and execute each Model output code on a small number of Unit Tests (i.e. pairs of inputs and expected outputs) to approximate semantic equivalence; two Model output codes are considered equivalent if and only if all unit tests produce the same execution output across both Model outputs. The numbers above the dotted arcs bellow denote that semantic equivalence as the binary matching loss between different execution loss, i.e., 0 if the execution outputs are the same, 1 otherwise. The Bayes risk is the sum of the loss between each Model output code and the rest. From the candidate Model outputs, the ones with the lowest Bayes risk are considered the best.

[0014] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume , Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu and Oriol Vinyals. 2022. Competition-level code generation with AlphaCode. Science present AlphaCode, a large code PLM. To generate a solution for a given problem, they generate 1000 Model outputs and cluster them based on their behavior, i.e., their execution outputs over a small number of unit test inputs. Model outputs that exhibit the same behavior, i.e., produce consistent execution outputs, are placed in the same cluster. Then a representative Model output is randomly selected from the cluster, with a priority given to the more populous cluster. While the paper does not recognize it as such, this process is essentially the application of MBR on the Model outputs.

[0015] The two Prior Art approaches are similar enough to share the same disadvantages.

[0016] Mainly, the application of minimum Bayes risk (MBR) on Code PLMs may be insufficient as the average of Model outputs is not necessarily close to the Bayes optimal action. This is due to lexical or parse-tree similarity of Model outputs being misleading, and the fact that the length of valid Model outputs is variable (which in turn affects similarity estimations). The Prior Art may not account for these issues. Additionally, Model outputs may be evaluated on whether they are functional or share the same functional behaviour, but such signal is not easily available during inference (due to the lack of available Unit Tests).

[0017] It is desirable to develop an apparatus and method that overcomes the above problems.

[0018] SUMMARY

[0019] According to a first aspect, there is provided a computing apparatus for generating a sequential model output, the computing apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code is executable by the one or more processors so that the computing apparatus is configured to: obtain a model input; using a model, generate a plurality of first candidate output sub-sequences from the model input, each of the first candidate output sub-sequences being code forming a first segment of the sequential model output; analyse each of the first candidate output sub-sequences to select one of the first candidate output subsequences; and generate the sequential model output based on the selected first candidate output sub sequence forming the first segment of the sequential mode output. In this way, the best first candidate output sub-sequence may be selected for use in generating the sequential model output. This may improve the accuracy of the overall sequential model output.

[0020] In some implementations, the computing apparatus may be configured to: using the model, generate a plurality of second candidate output sub-sequences from the model input, each of the second candidate output sub-sequences being code forming a second segment of the sequential model output; analyse each of the second candidate output sub-sequences to select one of the second candidate output sub-sequences; and generate the sequential model output based on the selected second candidate output sub-sequence forming the second segment of the sequential mode output. In this way, two segments of the sequential model output may be generated through selection of one of the candidate output sub-sequences.

[0021] In some implementations, the second segment of the sequential model output may follow the first segment of the sequential model output. In this way, different segments at different points in the sequential model output may be generated. In some implementations, the second segment of the sequential model output may directly follow the first segment of the sequential model output. In this way, different segments at following points in the sequential model output may be generated.

[0022] In some implementations, the computing apparatus may be configured to generate the plurality of second candidate output sub-sequences based on the prefix from the selected first candidate output sub-sequence. In this way, the divergence by later candidate output sub-sequences for later segments from the optimal sequential model output may be reduced.

[0023] In some implementations, the computing apparatus may be configured to repeat the steps in any of statements above for any further segments of the sequential model output. In some implementations, the computing apparatus may be configured to repeat the steps in any of the statements above for all of any further segments of the sequential model output. In this way, all of the sequential model output may be generated from the candidate output sub-sequences.

[0024] In some implementations, the computing apparatus may be configured to determine a breakpoint in the sequential model output defining an end of one of the segments of the sequential model output. In this way, the segments of the sequential model output may be broken up. In some implementations, the computing apparatus may be configured to determine the breakpoint based on a feature in one or more of the candidate output sub-sequences forming the segment of the sequential model output. In some implementations, the feature may comprise one or more of an end of a line, brackets, changes in indentation, and a new code block. In this way, the features of the candidate output sub-sequence which indicate an effective breakpoint may be used to determine the breakpoint.

[0025] In some implementations, the computing apparatus may be configured to determine the breakpoint based on statistical analysis of the candidate output sub-sequences forming the segment of the sequential model output. In some implementations, the computing apparatus may be configured to determine the breakpoint based on statistical analysis of the following one or more candidate output sub-sequences forming the following segment of the sequential model output. In this way, the breakpoint may be determined automatically based on the contents of the candidate output sub-sequences, either of the current or following segment. This may improve efficiency of the model.

[0026] In some implementations, the computing apparatus may be configured to carry out the statistical analysis by analysing the distribution of the potential actions of the following one or more candidate output sub-sequences forming the following segment of the sequential model output. In this way, the execution output that the candidate output sub-sequence may produce is used to determine an efficient breakpoint.

[0027] In some implementations, the computing apparatus may be configured to analyse the distribution of the potential actions of the following one or more candidate output subsequences forming the following segment of the sequential model output to determine a confidence value. The confidence value may indicate the level of variation in the distribution of the potential actions of the following one or more candidate output sub-sequences forming the following segment of the sequential model output. In some implementations, the computing apparatus may be configured to determine the breakpoint in dependence on the confidence value being above a confidence value threshold. In this way, the level of confidence in the accuracy of the execution output that the candidate output sub-sequence may produce is used to determine an efficient breakpoint. In some implementations, the computing apparatus may be configured to analyse the candidate output sub-sequences to select one of the candidate output sub-sequences by means of the average of each candidate output sub-sequence action log probabilities or minimum bayes risk (MBR). In this way, the candidate output sub-sequence may be selected based on its ability to produce an accurate execution output.

[0028] In some implementations, the model input may comprise context. In some implementations, the model input may comprise a prefix sequence. In this way, further information may be provided to the model which can be used in the generation of the candidate output sub-sequences.

[0029] In some implementations, the model input may be obtained from a user. In this way, the apparatus can be used in system, such as a software package, in which the user can input and then receive an output. In some implementations, the model input may be obtained automatically. In this way, a user is not necessarily required to manually input to the model.

[0030] In some implementations, the model may be a language model. In some implementations, the model may be a pretrained language model. In some implementations, the model may be a code pretrained language model. In this way, the apparatus may be applied to a range of different models.

[0031] According to a second aspect, there is provided a method for generating a sequential model output, the method comprising steps of obtaining a model input; using a model, generating a plurality of first candidate output sub-sequences from the model input, each of the first candidate output sub-sequences being code forming a first segment of the sequential model output; analysing each of the first candidate output sub-sequences to select one of the first candidate output sub-sequences; and generating the sequential model output based on the selected first candidate output sub-sequence forming the first segment of the sequential mode output. In this way, the best first candidate output sub-sequence may be selected for use in generating the sequential model output. This may improve the accuracy of the overall sequential model output. BRIEF DESCRIPTION OF THE FIGURES

[0032] The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0033] Figure 1 schematically illustrates a prior art pipeline for generating a sequential model output.

[0034] Figure 2 schematically illustrates a prior art pipeline for code synthesis.

[0035] Figure 3 is a prior art code for calculating Bayes risk.

[0036] Figure 4 schematically illustrates an exemplary pipeline for generating a sequential model output of the present application.

[0037] Figure 5 shows an example of a computer implemented method for generating a sequential model output.

[0038] Figure 6 shows an example of an apparatus configured to perform the methods described herein.

[0039] DETAILED DESCRIPTION

[0040] The apparatuses and methods described herein concern the generation of a sequential model output.

[0041] Embodiments of the present invention may tackle one or more of the problems previously mentioned by: using a model to generate a plurality of first candidate output sub-sequences from the model input and generating the sequential model output based on the selected first candidate output sub-sequence forming the first segment of the sequential mode output. In this way, the best first candidate output sub-sequence may be selected for use in generating the sequential model output. This may improve the accuracy of the overall sequential model output.

[0042] The following acronyms will be used throughout:

[0043] Language Modelling: The process of representing language as a probability distribution over words; in the context of other words (for example, sentences or phrases), it is the process of modelling the probability distribution of the next word in Causal Language Modelling, or any hidden word in Masked Language Modelling.

[0044] Language Model (LM): An amalgamation of computer-based mathematical functions that perform the task of Language Modelling.

[0045] Pretrained Language Model (PLM): An amalgamation of computer-based mathematical functions that perform the task of Language Modelling, which have been engineered in such a way to reflect the probability distribution of specific underlying data.

[0046] Code Pretrained Language Model (CPLM): A pretrained language model that reflects the probability distribution over both Programming Language code data as well as natural language data.

[0047] Structured prediction (SP): A family of machine learning methods that aim to predict some structured output. Sequence prediction in particular, aims to progressively generate a sequence of actions, i.e. at any step, it aims to predict the next item in a sequence based on previous observations.

[0048] Natural Language Generation (NLG): The process of generating natural language phrases, sentences, paragraphs etc. by means of a Pretrained Language Model. More specifically, NLG in the setting of this Innovation is modeled as a structured (sequence) prediction problem, where the output is formed by a sequence of actions, each corresponding to production of a word.

[0049] Code Synthesis (CS): Similar to NLG, with the explicit target language being a computer language rather than natural language, i.e., code in a specific Programming Language. In the context of this Innovation, Code Synthesis shall encompass the task of presenting to a PLM a problem formulated in Natural Language, and the objective is to generate code in a Programming Language that solves the problem.

[0050] Machine Learning (ML): Any process of devising a statistical model for a well-defined task with the help of algorithms.

[0051] To avoid any confusion potentially arising from a Model producing Programming Language code which can in turn return some output, the output of the Model shall be defined as “Model outputs” which (in a Code Synthesis task) are sequences representing Programming Language code and “execution outputs” which are the outputs returned by “Model outputs” when tested on Unit Tests. Referring to Figure 1, the input 101 is fed into the model 102, and the model 102 outputs “model outputs” 103. The model outputs 103 can be executed to output execution outputs.

[0052] The present system aims to apply minimum Bayes risk (MBR) decoding to Model outputs. The present system aims to overcome the following technical problems with previous applications: Using functional behaviour (i.e., execution outputs) to compare Model outputs may be impractical, as Unit Tests may not be easily available during inference.

[0053] Using word or token overlap metrics to compare Model outputs may be unreliable since, due to the nature of code, different valid Model outputs could have very variable lengths. Simply normalizing the metrics over length has been shown to be sub-optimal.

[0054] Waiting for the generation of Model outputs to conclude before applying similarity metrics, may cause the Model outputs sequence to diverge too much due to errors early in the sequence. In other words, code generation models are sensitive to the code prefix, and poor prefixes can be unrecoverable (both because of errors and because of the resulting distribution being poor).

[0055] The present system introduces:

[0056] The concept of generating the output of an LLM by alternating between

[0057] 1) Generation: the Model incrementally generating multiple candidate segments of the output, and

[0058] 2) Selection: determining which of the candidate segments is going to be part of the final output. To determine each segment, the present system may specifically papply MBR over the candidate segments. Each segment may be generated following the determination of the previous segment.

[0059] This process may help with word or token overlap metric reliability when applying MBR as the variance between Model outputs segment lengths is narrower.

[0060] This process may also ensure that errors are caught early in the output sequence, recovering from a poor prefix before it can affect the rest of the sequence. strategies on how to split Model outputs into segments, i.e., when the model should consider that the sequence generation has reached a segment end point, and trigger the selection process. Segmentation may be manually pre-set (e.g., segment at the end of each line or code block) or determined automatically (e.g., based on some metric over the output probability distribution. Figure 4 schematically illustrates an exemplary pipeline 400 for generating a sequential model output 403 of the present application. To generate the model output 403, incremental MBR may be applied following the steps outlined below (and illustrated in Figure 4).

[0061] From a top-level view, the following steps are carried out:

[0062] • Generation: Given an optional input and optional context and a (potentially empty) prefix sequence, many continuations of the sequence may be sampled (until a breakpoint) as hypotheses (e.g., candidate output sub-sequences 403-1 to 403-N in Figure 4). Each sub-sequence may be sampled until a breakpoint.

[0063] • A natural breakpoint in code may be line breaks or closing brackets (‘ }’) or changes in indentation or the generation of a control word (e.g., if, while) which can be taken to indicate the start of a new code block. However, breakpoints could also be determined automatically by some process.

[0064] • Selection: Estimate the utility of the hypotheses. This may be achieved as in regular MBR, through a consensus. In other words, instead of seeking the argmax of Equation 2 as in regular greedy decoding (i.e., pick the most probable action given the prefix sequence), incremental MBR may seek the argmax of Equation 3, where a is subsequence of tokens representing a natural solution segment in the problem space, and u is a utility function (i.e. the consensus) to measure their quality. p(xi|x1- xi-1) (2) u(a x1-- xi-1(3)

[0065] A utility function to reach the consensus could be any similarity metric that can be applied over sequences, e.g., chrF++, BLEU, codeBLEU or CodeBERTScore.

[0066] • Add the highest utility hypothesis (e.g. candidate output sub-sequences 403-2.1 to 403-2.N in Figure 4) to the prefix and repeat until the end of the output solution is reached. The end of the output solution is signified through a stopping criterion.

[0067] A stopping criterion could be a specific token being generated, a pre-specified length limit, etc.

[0068] The pipeline 400 may comprise a model 102. A model input 101 may be obtained. The model input 101 may be received by the model 102. The model input 101 may be obtained from a user. The user may input the model input 101 to the model 102. The model input 101 may be obtained automatically. The model input 101 may been originally inputted by the user, and then inputted automatically to the model 102. The model input 101 may be inputted as a part of a software program. The model 102 may output “model outputs” 403. The model outputs 403 may be sequential model outputs 403. The model outputs 403 may be returned to the user. The model input 101 may be inputted to the model 102 and returned to the user as part of the software program for generating model outputs.

[0069] The model 102 may comprise a language model (ML). In this way, the language model may output a language model output 403. The model 102 may comprise a pretrained language model. In this way, the pretrained language model may be trained to output an accurate language model output 403. The model 102 may comprise a code pretrained language model. In this way, the code pretrained language model may output a code language model output 403.

[0070] The model input 101 may comprise an input to the model 102. The model input 101 may comprise context. The model input 101 may comprise a prefix sequence. The model input 101 may comprise instructions for generating a code language output. The model input 101 may comprise descriptions of a task or problem.

[0071] The model 102 may generate a plurality of candidate output sub-sequences 403-1, 403-2, 403- N. The model 102 may generate a plurality of candidate output sub-sequences 403-1, 403-2, 403-N from the model input 101. The plurality of candidate output sub-sequences 403-1, 403- 2, 403-N may be defined as first plurality of candidate output sub-sequences 403-1, 403-2, 403- N. The plurality of first candidate output sub-sequences 403-1, 403-2, 403-N are illustrated in Figure 4 by the first column. The notation 403-N, namely the N, designates the first column. Each of the plurality of first candidate output sub-sequences 403-1, 403-2, 403-N may be code. Each of the plurality of first candidate output sub-sequences 403-1, 403-2, 403-N may form a first segment 403 -X of the sequential model output 403. In other words, each of the plurality of first candidate output sub-sequences 403-1, 403-2, 403-N may provide possible options for the first segment 403-X of the sequential model output 403. Each of the plurality of first candidate output sub-sequences 403-1, 403-2, 403-N may be different from one another. Each of the plurality of first candidate output sub-sequences 403-1, 403-2, 403-N may be a different length from one another.

[0072] The plurality of first candidate output sub-sequences 403-1, 403-2, 403-N may be analysed to select one of the plurality of first candidate output sub-sequences 403-1, 403-2, 403-N. Figure 4 illustrates that the candidate output sub-sequence 403-2 is selected. Of course, any of the plurality of first candidate output sub-sequences 403-1, 403-2, 403 -N could have been selected. The best of the plurality of first candidate output sub-sequences 403-1, 403-2, 403-N may be selected. The plurality of first candidate output sub-sequences 403-1, 403-2, 403-N may be analysed by means of the average of each candidate output sub-sequence action log probabilities to select one first candidate output sub-sequences 403-2. The plurality of first candidate output sub-sequences 403-1, 403-2, 403-N may be analysed by means of minimum bayes risk (MBR) to select one first candidate output sub-sequences 403-2. The plurality of first candidate output sub-sequences 403-1, 403-2, 403-N may be analysed by means of the average of each candidate output sub-sequence action log probabilities and minimum bayes risk (MBR) to select one first candidate output sub-sequences 403-2.

[0073] The sequential model output 403 may be generated based on the selected first candidate output sub-sequence 403-2. The selected first candidate output sub-sequence 403-2 may form a part of the sequential model output 403. The selected first candidate output sub-sequence 403-2 may form the first segment 403 -X of the sequential model output 403.

[0074] The model 102 may generate a plurality of candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N. The model 102 may generate a plurality of candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N from the model input 101. The plurality of candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may be defined as second plurality of candidate output subsequences 403-2.1, 403-2.2, 403-2.N. The plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N are illustrated in Figure 4 by the second column. The notation 403- 2.N, namely the N, designates the second column. Each of the plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may be code. Each of the plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may form a second segment 403-Y of the sequential model output 403. In other words, each of the plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may provide possible options for the second segment 403-Y of the sequential model output 403. Each of the plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may be different from one another. Each of the plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may be a different length from one another. The plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may be analysed to select one of the plurality of second candidate output sub-sequences 403-2.1, 403- 2.2, 403-2.N. As an example, the candidate output sub-sequence 403-2.2 may be selected. Of course, any of the plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403- 2.N could have been selected. The best of the plurality of second candidate output subsequences 403-2.1, 403-2.2, 403-2.N may be selected. The plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may be analysed by means of the average of each candidate output sub-sequence action log probabilities to select one second candidate output sub-sequences 403-2. The plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may be analysed by means of minimum bayes risk (MBR) to select one second candidate output sub-sequences 403-2. The plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may be analysed by means of the average of each candidate output sub-sequence action log probabilities and minimum bayes risk (MBR) to select one second candidate output sub-sequences 403-2.2.

[0075] The sequential model output 403 may be generated based on the selected second candidate output sub-sequence 403-2.2. The selected second candidate output sub-sequence 403-2.2 may form a part of the sequential model output 403. The selected second candidate output subsequence 403-2.2 may form the second segment 403-Y of the sequential model output 403.

[0076] The second segment 403-X.Y may follow the first segment 403-X. In other words, the second segment 403-X.Y may comprise a later part of the sequential model output 403 than the first segment 403-X. As a result, the plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may form a later part of the sequential model output 403 than the plurality of first candidate output sub-sequences 403-1, 403-2, 403 -N. The second segment 403-X.Y may directly follow the first segment 403-X. In other words, the second segment 403-X.Y may comprise the part of the sequential model output 403 that is straight after the first segment 403- X. As a result, the plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403- 2.N may form the part of the sequential model output 403 that is straight after the plurality of first candidate output sub-sequences 403-1, 403-2, 403 -N.

[0077] The plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may be generated based on the prefix from the selected first candidate output sub-sequence 403-2.1, 403-2.2, 403-2.N. In other words, the prefix of the earlier candidates may be used to generate the later candidates for the sequential model output 403.

[0078] The above steps of the plurality of second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N may be repeated for one or more further segments of the sequential mode output 403. This will be illustrated in Figure 4 by further columns. For example, 403-2.2 may be selected and a further plurality of candidate output sub-sequences 403-2.2.1, 403-2.2.2, 403-2.2.N may be generated. The further plurality of candidate output sub-sequences 403-2.2.1, 403-2.2.2, 403-2.2.N may be analysed to select one of the further plurality of candidate output subsequences 403-2.2.1, 403-2.2.2, 403-2.2.N. The sequential model output 403 may be generated based on this selection, and so on. The process may be repeated for a number of further segments. The process may be repeated for all of the segments in the sequential model output 403. In this way, all of the segments in the sequential model output 403 may have a selected candidate output sub-sequence. The sequential model output 403 may then be generated based on the combination of all the selected candidate output sub-sequences. The end completion of the sequential model output 403 may be signified through a stopping criterion. The stopping criterion may comprise a specific token being generated. The stopping criterion may comprise a pre-specified length limit and / or time limit.

[0079] Preferably, the process may follow the order of

[0080] 1. Generate the first candidate output sub-sequences 403-1, 403-2, 403-N;

[0081] 2. Analyse the first candidate output sub-sequences 403-1, 403-2, 403-N to select one first candidate output sub-sequence 403-2;

[0082] 3. Generate the second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N;

[0083] 4. Analyse the second candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N to select one second candidate output sub-sequence 403-2.2;

[0084] 5. Generate a further plurality of candidate output sub-sequences;

[0085] 6. Analyse the further plurality of output sub-sequences to select one further candidate output sub-sequence;

[0086] 7. Generate the sequential mode output based on the selected candidate output subsequences.

[0087] Steps 5 and 6 may be repeated for all of the further candidate output sub-sequences.

[0088] Alternatively, the process may carry out steps 1, 3, and 5 followed by steps 2, 4, and 6. The process may also follow any other feasible order. By carrying out the segmented process above, this may help with word or token overlap metric reliability as the variance between output segment lengths is narrower. This process may also ensure that errors are caught early in the Model output sequence, recovering from a poor prefix before it can affect the rest of the sequence.

[0089] Each of the segments 403 -X, 403 -X.Y may have an end defined by a breakpoint. In other words, the end of the segments 403-X, 403X.Y may end in an endpoint. Consequently, each of the plurality of candidate output sub-sequences for the end point in question also end in the endpoint. In the example where the second segment 403 -X-Y directly follows the first segment 403-X, the breakpoint may separate the second segment 403 X-Y from the first segment 403-X. The breakpoint may apply to all of the candidate output sub-sequences that form that segment.

[0090] The breakpoint may be determined based on a feature in one or more of the candidate output sub-sequences 403-1, 403-2, 403-N, 403-2.1, 403-2.2, 403-2.N forming the segment 403-X, 403 -Y of the sequential model output 403. In other words, the breakpoint for the segment may be based on a feature in one or more of the candidate output sub-sequences that form that segment. The breakpoint may be based on a plurality of features in one or more of the candidate output sub-sequences. The same feature may appear in each of the candidate output subsequences for the segment. The feature may only appear in one or some of the candidate output sub-sequences for the segment. There may be more than one type of feature that is used to determine the breakpoint. For example, there may be a group of features, and if one or more of those features appear in one or more of the candidate output sub-sequences for the segment, then the breakpoint may be determined. The feature(s) may comprise one or more of: an end of a line of code, brackets in the code, changes in indentation in the code, and a new code block. Other features may also be suitable. Taking the example above, if the line of code ends in any of the candidate output sub-sequences for the segment, the breakpoint may be determined.

[0091] The breakpoint may be determined based on statistical analysis of the candidate output subsequences 403-1, 403-2, 403-N forming the segment 403-X of the sequential model output 403. Alternatively, or in addition, the breakpoint may be determined based on statistical analysis of the candidate output sub-sequences 403-2.1, 403-2.2, 403-2.N forming the segment 403-X. Y of the sequential model output 403. In other words, the statistical analysis may be performed on the current candidate output sub-sequences for the current segment, and based on the analysis the breakpoint of the segment may be determined. Alternatively, or in addition, the statistical analysis may be performed on the current candidate output sub-sequences for the following segment, and based on the analysis the breakpoint of the current segment may be determined. To perform statical analysis of the following segment, the following candidate outputs sub-sequences for the following segment may need to be determined, or estimated, beforehand.

[0092] The statistical analysis may comprise analysing the distribution of the potential actions of the candidate output sub-sequences. In the case of the breakpoint being based on the current segment, the distribution of the potential actions of the current candidate output sub-sequences may be analysed. In the case of the breakpoint being based on the following segment, the distribution of the potential actions of the following candidate output sub-sequences may be analysed. The potential actions may comprise the execution output of the candidate output subsequences. The potential actions may be determined, or estimated, to form the distribution. The potential actions may be determined, or estimated, to form the distribution before the statistical analysis is carrying out.

[0093] The statistical analysis of the candidate output sub-sequences may allow a confidence value to be determined. The confidence value may indicate the level of variation in the distribution of the potential actions. There may be a confidence value threshold. If the confidence value is above or below a certain threshold, then the breakpoint may be determined. In particular, if the confidence value of the current candidate output sub-sequences for the current segments is low, and below the threshold, then the breakpoint may be determined. In this way, the current candidate output sub-sequences will not continue to be generated if the confidence in them is low. Similarly, if the confidence value of the following candidate output sub-sequences for the current segments is high, and above the threshold, then the breakpoint may be determined. In this way, the current candidate output sub-sequence will not continue if the confidence in the following candidate output sub-sequences are high.

[0094] In summary, the model output segmentation can be performed using pre-defined rules (e.g., reaching some predefined token or the end of a line). However, automatic detection (e.g., through estimating the surprisal of the sequence - based on the distribution of potential actions) of breaking points may be domain-independent, i.e., not defined for a specific programming language. Automatic segmentation may also allow better detection of points of uncertainty in the sequence and may focus MBR decisions on them. The present system has been developed for use in Code PLMs but can be applied on any PLM or in general any model or process that iteratively constructs a sequential output. The benefit of the present system may stem from the assumption that the model output sequence can be meaningfully segmented into sub-sequences (via the identification of breakpoints or other process). As such the present system should be applicable and beneficial on any task where such an assumption can be reasonable made.

[0095] The present system is not limited to the application of MBR in the selection step, and any other reranking methods may be applied to perform the selection instead.

[0096] The present system is also not limited to the utility functions or breakpoint identification processes mentioned here.

[0097] Figure 5 summarises an example of a method 500 for generating a sequential model output. At step 501, the method 500 comprises obtaining a model input. At step 502, the method 500 comprises using a model, generating a plurality of first candidate output sub-sequences from the model input, each of the first candidate output sub-sequences being code forming a first segment of the sequential model output. At step 503, the method 500 comprises analysing each of the first candidate output sub-sequences to select one of the first candidate output subsequences. At step 504, the method 500 comprises generating the sequential model output based on the selected first candidate output sub-sequence forming the first segment of the sequential mode output. Steps 502 to 504 may be repeated for subsequent segments, as described here.

[0098] The algorithm of the present system may be applicable on any Model that predicts or generates full or partial sequential output (as illustrated in Figure 1). To condition the generation process, the Model may or may not require an input and / or context. The Model may be stored or accessible through a computer system, such as a PC, cloud compute server, mobile device or other computing device. An example of an apparatus 600 configured to implement the method 500 is schematically illustrated in Figure 6. The computing apparatus 600 may comprise the pipeline 400. The apparatus 600 may be implemented on an electronic device, such as a computer, laptop, tablet, or smart phone.

[0099] The apparatus 600 comprises a processor 601 configured to process the datasets in the manner described herein. For example, the processor 601 may be implemented as a computer program running on a programmable device such as a Central Processing Unit (CPU). The apparatus 600 comprises a memory 602 which is arranged to communicate with the processor 601. Memory 602 may be a non-volatile memory. The processor 601 may also comprise a cache (not shown in Figure 6), which may be used to temporarily store data from memory 602. The apparatus 600 may comprise more than one processor 601 and more than one memory 602. The memory 602 may store data that is executable by the processor 601. The processor 601 may be configured to operate in accordance with a computer program stored in non-transitory form on a machine-readable storage medium. The computer program may store instructions for causing the processor to perform its methods in the manner described herein. The method steps described herein may be carried out by a computer-readable storage medium. The method steps described herein may be carried out by a computer program product.

[0100] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description, it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A computing apparatus (600) for generating a sequential model output (403), the computing apparatus comprising one or more processors (601) and a memory (602) storing in non-transient form data defining program code executable by the one or more processors (601), wherein the program code is executable by the one or more processors (601) so that the computing apparatus is configured to: obtain a model input (101); using a model (102), generate a plurality of first candidate output sub-sequences (403-1, 403-2, 403-N) from the model input (101), each of the first candidate output sub-sequences (403-1, 403-2, 403-N) being code forming a first segment (403-X) of the sequential model output (403); analyse each of the first candidate output sub-sequences (403-1, 403-2, 403-N) to select one of the first candidate output sub-sequences (403-1, 403-2, 403-N); and generate the sequential model output (403) based on the selected first candidate output sub-sequence (403-2) forming the first segment (403-X) of the sequential mode output (403).

2. A computing apparatus (600) according to claim 1, wherein the apparatus (600) is configured to: using the model (102), generate a plurality of second candidate output subsequences (403-2.1, 403-2.2, 403-2.N) from the model input (101), each of the second candidate output sub-sequences (403-2.1, 403-2.2, 403-2.N) being code forming a second segment (403-X. Y) of the sequential model output (403); analyse each of the second candidate output sub-sequences (403-2.1, 403-2.2, 403-2.N) to select one of the second candidate output sub-sequences (403-2.1, 403-2.2, 403-2.N); and generate the sequential model output (403) based on the selected second candidate output sub-sequence (403-2.2) forming the second segment (403-X. Y) of the sequential mode output (403).

3. A computing apparatus (600) according to claim 2, wherein the second segment (403- X.Y) of the sequential model output (403) follows the first segment (403. X) of the sequential model output (403), and optionally wherein the second segment (403-X. Y) of the sequentialmodel output (403) directly follows the first segment (403. X) of the sequential model output (403).

4. A computing apparatus (600) according to claim 2 or 3, wherein the apparatus (600) is configured to generate the plurality of second candidate output sub-sequences (403-2.1, 403- 2.2, 403-2.N) based on the prefix from the selected first candidate output sub-sequence (403- 2).

5. A computing apparatus (600) according to any of claims 2 to 4, wherein the apparatus (600) is configured to repeat the steps in any of claims 2 to 4 for any further segments of the sequential model output (403), and optionally wherein the apparatus (600) configured to repeat the steps in any of claims 2 to 4 for all of any further segments of the sequential model output (403).

6. A computing apparatus (600) according to any preceding claim, wherein the apparatus (600) is configured to determine a breakpoint in the sequential model output (403) defining an end of one of the segments (403-X, 403-X.Y) of the sequential model output (403).

7. A computing apparatus (600) according to claim 6, wherein the apparatus (600) is configured to determine the breakpoint based on a feature in one or more of the candidate output sub-sequences (403-1, 403-2, 403-N, 403-2.1, 403-2.2, 403-2.N) forming the segment (403-X, 403-Y) of the sequential model output (403), and optionally wherein the feature comprises one or more of: an end of a line, brackets, changes in indentation, and a new code block.

8. A computing apparatus (600) according to claim 6 or 7, wherein the apparatus (600) is configured to determine the breakpoint based on statistical analysis of one or more of the candidate output sub-sequences (403-1, 403-2, 403-N) forming the segment (403-X) of the sequential model output (403); and / or the following one or more candidate output subsequences (403-2.1, 403-2.2, 403-2.N) forming the following segment (403-X.Y) of the sequential model output (403).

9. A computing apparatus (600) according to claim 8, wherein the apparatus (600) is configured to carry out the statistical analysis by analysing the distribution of the potentialactions of the following one or more candidate output sub-sequences (403-2.1, 403-2.2, 403- 2.N) forming the following segment (403-X.Y) of the sequential model output (403).

10. A computing apparatus (600) according to claim 9, wherein the apparatus (600) is configured to analyse the distribution of the potential actions of the following one or more candidate output sub-sequences (403-2.1, 403-2.2, 403-2.N) forming the following segment (403-X.Y) of the sequential model output (403) to determine a confidence value, the confidence value indicating the level of variation in the distribution of the potential actions of the following one or more candidate output sub-sequences (403-2.1, 403-2.2, 403-2.N) forming the following segment (403-X.Y) of the sequential model output (403); and optionally wherein the apparatus (600) is configured to determine the breakpoint in dependence on the confidence value being above a confidence value threshold.

11. A computing apparatus (600) according to any preceding claim, wherein the apparatus (600) is configured to analyse the candidate output sub-sequences (403-1, 403-2, 403 -N, 403- 2.1, 403-2.2, 403-2.N) to select one of the candidate output sub-sequences by means of the average of each candidate output sub-sequence action log probabilities or minimum bayes risk (MBR).

12. A computing apparatus (600) according to any preceding claim, wherein the model input (101) comprises context and / or a prefix sequence.

13. A computing apparatus (600) according to any preceding claim, wherein the model input (101) is obtained from a user, and / or wherein the model input (101) is obtained automatically.

14. A computing apparatus (600) according to any preceding claim, wherein the model (102) is a language model; and optionally wherein the model (102) is a pretrained language model; and optionally wherein the model (102) is a code pretrained language model.

15. A method (500) for generating a sequential model output, the method (500) comprising steps of: obtaining a model input (501);using a model, generating a plurality of first candidate output sub-sequences from the model input, each of the first candidate output sub-sequences being code forming a first segment of the sequential model output (502); analysing each of the first candidate output sub-sequences to select one of the first candidate output sub-sequences (503); and generating the sequential model output based on the selected first candidate output sub-sequence forming the first segment of the sequential mode output (504).

Images