Unlimited context length with maximum self-information compression

Abstract

Described is a computing apparatus (700) for generating an output from a model (104) configured to: obtain a part (101a, 101b, 101c, 101d) of a context input (101) as input tokens (103); generate embeddings (106) from the part (101a, 101b, 101c, 101d) of the context input (101) using the model (104), calculate the self-information of the input tokens (105) from the representation (201) of the embeddings (105); compress the embeddings (105) in dependence on the self-information of the input tokens (103); generate further output tokens (106) from the part (101a, 101b, 101c, 101d) of the context input (101) and the compressed embeddings using the model (104); and output a token output in dependence on the further output tokens (106. In this way, the improved further output tokens (106) may be used for the output, while not exceeding the context length limit of the model (104).

Description

[0001] UNLIMITED CONTEXT LENGTH WITH MAXIMUM SELF-INFORMATION COMPRESSION

[0002] FIELD OF THE INVENTION

[0003] This disclosure relates to an apparatus and method for generating an output from a model.

[0004] BACKGROUND

[0005] Processing very long contexts by large language models (LLMs) poses challenges like high computational cost, memory issues, and potential decrease in model performance due to less focused attention across the long sequence.

[0006] The entire context serves as the input and should thus be used / considered in order to generate the next tokens. However, current LLMs can have limitations in the maximum context length they can process, typically in the range of 4,000-32,000 tokens. Some commercial models can handle up to 200,000 tokens.

[0007] Going beyond this limit is considered technically impossible using the standard methodology. In addition, the generalization beyond the model trained capabilities to unseen sequence length is not clear.

[0008] To tackle this issue many approaches, utilize an additional algorithm to compress the original context and allow the model to use it. It is not trivial which parts of the original context are more important and which can be reduced.

[0009] Prior art work related to cache compression includes:

[0010] 1. LLM Lingua (from Microsoft) suggests using a Small LM model as a prompt compressor before passing the text to a larger and more expensive model. The small model removes tokens that got low perplexity and thus contribute less to the overall entropy. [H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang and L. Qiu, "LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models," ArXiv, 2023]

[0011] 2. Selective Context work that handles the context string and self-information concepts to filter out less informative tokens. [Y. Li, "Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of," arXiv, 2023]

[0012] 3. LM Infinite (from Meta) suggests keeping inthe KV Cache only a fixed number of “starting tokens” (i.e., from the beginning of the prompt) as well as a fixed window of “rear tokens” (i.e., latest tokens) while omitting all the “middle-tokens”. [C. Han, Q. Wang, W. Xiong, Y. Chen, H. Ji and S. Wang, "Lm-infinite: Simple on-the-fly length generalization for large language models," arXiv, 2023]

[0013] 4. Streaming LLM use a sliding window attention in order to reduce the KV Cache size [G. Xiao, Y. Tian, B. Chen, S. Han and M. Lewis, "Efficient Streaming Language Models with Attention Sinks," arXiv, 2023],

[0014] Some of the prior art solutions handle the prompt before it reaches the model. The advantage of such approach as it is independent of the language model itself and hence can be deployed in any scenario and infrastructure. However, such methods may miss the essential information that a model extract from the prompt. For example, an LLM which is fine-tuned to summarize legal documents might focus on different keywords than a medical model.

[0015] On the other side of the scope are approaches which manage the internal KV Cache variable within the model, i.e., the “memory” of the alreadv digested context. These approaches may need to be integrated to a specific model or family of models, because they need to understand the representation of this internal KV Cache. For example, LLaMA encodes positional information into the KV Cache while GPT do not.

[0016] Working on the prompt directly, leverage the model’s ability to represent information which is beneficial for the interpretation of the prompt. On the other hand, approaches which manipulate the KV Cache internally require a dedicated kernel for each model.

[0017] It is desirable to develop an apparatus and method that overcomes the above problems.

[0018] SUMMARY

[0019] According to a first aspect, there is provided a computing apparatus for generating an output from a model, the computing apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code is executable by the one or more processors so that the computing apparatus is configured to: obtain a part of a context input as input tokens; generate embeddings from the part of the context input using the model, calculate the self- information of the input tokens from the representation of the embeddings; compress the embeddings in dependence on the self-information of the input tokens; generate further output tokens from the part of the context input and the compressed embeddings using the model; and output a token output in dependence on the further output tokens. In this way, the improved further output tokens may be used for the output, while not exceeding the context length limit of the model.

[0020] In some implementations, the computing apparatus may be configured to obtain the part of the context input by splitting the context input into a plurality of parts. In this way, the context input may be broken down into chunks, which reduces the input size to the model.

[0021] In some implementations, the computing apparatus may be configured to split the context input into a plurality of parts each of which comprise a context length below a context length limit of the model. In this way, the context input may be broken down into chunks which is of a size that the model can handle.

[0022] In some implementations, the computing apparatus may be configured to repeat the steps of any of the statements for each of the parts of the context input. In this way, the output of the model may be based on the entire context input.

[0023] In some implementations the representation of the embeddings may be used to calculate the self-information of the input tokens are represented as a vector. In this way, the representation may be in a form which can be interpreted for the compression step.

[0024] In some implementations, the computing apparatus may be configured to, after the embeddings are generated, store the part of the context input and the embeddings in a keys and values cache. In this way, the part of the context input and the embeddings may be stored ready for the next iteration through the model.

[0025] In some implementations, the computing apparatus may be configured to compress the embeddings by removing input tokens with a self-information below a self-information threshold. In this way, tokens which are less useful for generating the further output tokens are not used in the next iterations through the model.

[0026] In some implementations, the computing apparatus may be configured to compress the embeddings by only removing input tokens in a specific portion of the embeddings. In this way, portions which are more useful for generating further output tokens, such as the final portion, may be kept irrespective of their self-information level. In some implementations, the self-information comprises the probability of an input token. In this way, tokens which appear with high frequency, which are less useful for generating further output tokens are not used in the next iteration through the model. In some implementations, the self-information comprises the similarly of an input token compared to other tokens. In this way, similar tokens, which are less useful for generating further output tokens are not used in the next iteration through the model.

[0027] In some implementations, the computing apparatus may be configured to compress the embeddings by removing input tokens such that the total context length of the part of the context input and the compressed embeddings is less than the context length limit of the model. In this way, it can be ensured that the input for the next iteration through the model does not exceed the context length limit of the model.

[0028] In some implementations, the computing apparatus may be configured to generate tokens from the part of the context input using the model. In this way, the model may produce the tokens as well as the further output tokens.

[0029] In some implementations, the computing apparatus may be configured to: generate further embeddings from the part of the context input and the compressed embeddings using the model, calculate the self-information of the input tokens from the representation of the further embeddings; compress the further embeddings in dependence on the self-information of the further output tokens; generate more further output tokens from the part of the context input and the compressed further embeddings using the model; and output the token output in dependence on the more further output tokens.

[0030] In some implementations, the model is a language model. In some implementations, the input context is text. In some implementations, the token output is text. In some implementations, the language model is configured to carry out one or more language model tasks of: question answering, information retrieval, text summarisation, and reading comprehension. In this way, the present system may be used for linguistic tasks.

[0031] According to a second aspect, there is provided a method for generating an output from a model, the method comprising steps of: obtaining a part of a context input as input tokens; generating embeddings from the part of the context input using the model, calculating the self- information of the input tokens from the representation of the embeddings; compressing the embeddings in dependence on the self- information of the input tokens; generating further output tokens from the part of the context input and the compressed embeddings using the model; and outputting a token output in dependence on the further output tokens. In this way, the improved further output tokens may be used for the output, while not exceeding the context length limit of the model.

[0032] BRIEF DESCRIPTION OF THE FIGURES

[0033] The present disclosure will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0034] Fig. 1 schematically illustrates an exemplary pipeline for generating an output from a model.

[0035] Fig. 2 schematically illustrates representations of embeddings.

[0036] Fig. 3 is a graph illustrating the maximum self-information correlation probability approach.

[0037] Fig. 4 schematically illustrates an exemplary pipeline for the maximum self-information correlation KV Cache approach.

[0038] Figs. 5a and 5b illustrate the output of the present system compared to the prior art.

[0039] Fig. 6 shows an example of a computer implemented method for generating a sequential model output. Fig. 7 shows an example of an apparatus configured to perform the methods described herein.

[0040] DETAILED DESCRIPTION

[0041] The apparatuses and methods described herein concern the generation of an output from a model.

[0042] Embodiments of the present disclosure may tackle one or more of the problems previously mentioned by: compressing the embeddings in dependence on the self-information of the input tokens; generating further output tokens from the part of the context input and the compressed embeddings using the model; and outputting a token output in dependence on the further output tokens. In this way, the improved further output tokens may be used for the output, while not exceeding the context length limit of the model.

[0043] The following terms will be used throughout:

[0044] Model - In the present context this may be a neural network model that is trained for a specific task (e.g., classification, regression, etc.).

[0045] Language Model (LM) - A type of machine learning model that is trained on large amounts of text data to understand and generate human language.

[0046] Large Language Model (LLM) - A large LM.

[0047] Natural Language Processing (NLP) - The implementation of models onto human language.

[0048] Generative Language Model (GLM) - An NLP model that takes as an input a sequence of words (or, one might say, a sequence of tokens) and generates an extension to this sequence. For example, a probable extension to the string “the rain in Spain” is “falls mainly on the plane”. The generating process is also termed auto-regressor due to the incremental process which uses the generated tokens in order to produce the following tokens. Technically, each such token is concatenated to the sequence of prompt + already generated incremental tokens.

[0049] Prompt / Context - The sequence of words used as the input to the Language model, based on which the completion will be generated.

[0050] Self-Information - In the field of Information theory, and in the context of the present application, is the amount of information contained in a cache. This amount can be physically measured in units of information, i.e., bits.

[0051] Key Value (KV) Cache - When running a transformer-based language model in an autoregressive mode it is inefficient to recompute all the transformer internal states every incremental step. Instead, the internal variables, namely the Keys and Values (and hence KV), are saved in a cache memory and then used in the next generative step.

[0052] Neural Network (NN) - A method in artificial intelligence that teaches computers to process data in a way that is inspired by the human brain.

[0053] This present system proposes a mechanism to reduce the size of the context by compressing the KV Cache of the LM. The present system may provide unlimited context length while preserving the maximum self-information of the tokens in the KV Cache.

[0054] The present system aims to solve the following problems: How to extend the context length beyond the sequence length limit that the model was trained on.

[0055] How to reduce computational and memory complexity related to KV Cache.

[0056] • Focusing the model, even when given a very long sequence length, which may lead to inferior performance due to less focused attention.

[0057] Fig. 1 schematically illustrates an exemplary pipeline 100 for generating an output from a model 104. The pipeline 100 may comprise the model 104.

[0058] The pipeline 100 aims to use the model 104 in order to produce the most informative representation 105 to the input tokens 103 in the context sentence. Using this representation 105, the context may be compressed. B denotes the number of tokens that the model 104 can handle. N 102a denotes the length of the input context 101.

[0059] For N<B, the model 104 can handle the full context. However, for B<N not all the input context 101 can fit in the model 104 and it must be compressed. The input context 101 may be divided into chunks of size C 102b, such that C<B. The model 104 may be fed chunk by chunk. When the limit KVCache.length + C >= B is reached, maximum self-information correlation (MSIC) compression may be performed.

[0060] In more detail, the pipeline 100 may obtain a context input 101. The context input 101 may be inputted by a user. The user may input the context input 101 on a device on which the pipeline 100 is running or is accessible. For example, the user may input the context input 101 on a device, and the pipeline 100 may receive the context input 101.

[0061] The context input 101 may comprise a context length N 102a. The context input 101 may be split into a plurality of parts 101a, 101b, 101c, 101 d. Fig. 1 illustrates an example with four parts 101a, 101b, 101c, 101 d. In practice, any number of parts 101a, 101b, 101c, 101 d may be generated. The model 104 may have a context length limit. The context input 101 may be split into parts 101a, 101b, 101c, 101 d each of which have a part context length C 102b below context length limit of the model 104. The pipeline 100 may then run on the part 101a, 101b, 101c, 101 d of the context input 101. In this way, even if the context input 101 context length N 102a is above the context length limit of the model 104, the model 104 can still be used.

[0062] The part 101a, 101b, 101c, 101 d of the context input 101 may be input into the model 104 as input tokens 103. The model 104 may generate output tokens 106 from the part 101a, 101b, 101c, 101 d of the context input 101. The output tokens 106 may form the desired output of the model 104 in response to the part 101a, 101b, 101c, 101 d of the context input 101. Put another way, in response to the part 101a, 101b, 101c, 10 Id of the context input 101, the model 104 may be configured to generate the output tokens 106. The output tokens 106 may continue the input tokens 103. Additionally, or alternatively, the model 104 may generate embeddings 105. The embeddings 105 may represent the input tokens 103.

[0063] After the embeddings 105 are generated, the part 101a, 101b, 101c, 101 d of the context input 101 and / or the embeddings 105 may be stored in a KV cache 107. The KV cache may store the part 101a, 101b, 101c, 101 d of the context input 101 and / or the embeddings 105 for use in later iterations of the model 104.

[0064] Fig. 2 schematically illustrates representations 200 of embeddings 105. The embeddings 105 may be represented as a vector 201a, 201b, 201c, 201d. There are four examples shown in Fig. 2. The single solid vector 201a may represent the word “John”. The five dashed vectors 201b may represent different punctuation types, which may be similar in meaning. The single dotted vector 201c may represent the word “reporter’ . The three dot / dash vectors 201 d may represent the similar words “was”, “were” and “will”. Put simply, the values of the vectors 201a, 201b, 201c, 201d may comprise characteristics about the input token 103 the embedding 105 representing. The embeddings 105 may be used to calculate the self-information of the input tokens 103. In particular, the representation 200 of the embeddings 105 may be used calculate the self-information of the input tokens 103. The ‘self-information’ of the input tokens 103 may be considered a metric for the ‘usefulness’ of the corresponding embedding 105 in generating the final output of the model 104.

[0065] The embeddings 105 may be compressed. The embeddings 105 may be compressed in dependence on the self- information of the input tokens 103. The self- information of the input tokens 103 may be used to control the compression of the embeddings 105. The embeddings 105 may be compressed by MSIC compression 108.

[0066] The embeddings 105 may be compressed by removing input tokens 103 with a self-information below a self-information threshold. Put another way, any embeddings 105 which correspond to an input token 103 with a Tow level’ of self-information may be removed. In this way, ‘non-useful’ embeddings 105 may be removed. The self-information threshold may be predetermined. The self-information threshold may be varied depending on the model 104.

[0067] Alternatively, or in addition, the embeddings 105 may be compressed by only removing input tokens 103 in a specific portion of the embeddings 105. For example, only input tokens 103 in the middle of the embeddings 105 may be removed, but input tokens 103 at each of the end of the embeddings 105 may be kept. Input tokens 103 in certain regions may be kept irrespective of their self-information level. Input tokens 103 towards the end of the embeddings 105 are often useful in the running of the subsequent part 101a, 101b, 101c, 101 d of the context input 101. The middle of the embeddings 105 may be the middle 50%, 60%, 70%, 80% or 90%.

[0068] The self-information may comprise the probability of an embedding 105out of the total embeddings 105 for the input tokens 103. The more probable embeddings 105 may have a lower self- information. The less probable embeddings 105 may have a higher self-information. Embeddings 105 with a probability above a probability threshold may be removed during compression.

[0069] Fig. 3 is a graph 300 illustrating the maximum self-information correlation probability approach. As shown in the graph 300, the self-information of an input token 103 may vary as an inverse log of probability. As an example, the probability threshold could be set at 0.2 as this includes the majority of the input tokens 103 with high self-information.

[0070] The maximum self-information correlation probability approach is based on in Information theory. For a sequence of input tokens T = {t1;... , tN] the joint probability distribution is denoted by P(T), and the self-information is 1(7") = — log2(P(T))

[0071] However, in practice the joint probability function of T is unknown.

[0072] Instead one may use the empirical probability P(t) of a single token t and hence the self-information is l(t) = where ftis a frequency of the token t. And the total information is / tot(T) = — £ log2(P(t))

[0073] The objective in MSIC is: select n<N tokens from T while maximizing total self-information.

[0074] The maximum contextual information problem may be formulated as in Equation 1 :

[0075] Note that Self-information is monotonically decreasing with P(t), thus, choosing n tokens with lowest P(t) achieves this objective. Alternatively, or in addition, to probability, the self-information may comprise the similarity of an embedding 105compared to other embeddings 105 for the token inputs 103. The more similar embeddings 105 may have a lower self-information. The less similar embeddings 105 may have a higher self- information. Embeddings 105 with a similarly above a probability threshold may be removed during compression.

[0076] Fig. 4 schematically illustrates an exemplary pipeline 400 for the maximum self-information correlation KV Cache approach.

[0077] Words may have similar contextual information but different textual representation (token wise). For example, a “seat” and a “chair” are totally different textually, but they represent a similar meaning.

[0078] Such similarity can be measured when words are represented by embeddings, as described herein with reference to Fig. 2.

[0079] These correlations may be taken into account when calculating MSIC. A language model may practically calculate the most efficient representation (for its task). Instead of measuring frequency as with the probability approach, the correlation between embeddings in KV cache are used to reflect their joint probability. MSIC may be formulated as in Equation 2:

[0080] Where cr(a, b) G [—1,1] is a correlation function between two tokens a and b.

[0081] Using the Key Cache matrix, a good embedding representation 200 of each input token 103 may be achieved, in the following way:

[0082] Use the sequence T as the input of the model 104. Since the model 104 cannot accept the full string, use chunk of input sequence as described herein with reference to Fig. 1. During its standard forward pass, the model 104 will create relevant word embeddings per input token 103 (per head, per layer) and store them in KV Cache.

[0083] When the MSIC compression is due (because some limit threshold was crossed) - use the Key Cache in order to calculate MSIC as follows:

[0084] As shown in Fig. 4, create a correlation matrix 403 between tokens by multiplying K 401 by itself 402. Then sum 404 each row of the correlation matrix 403 to get the input token similarity 405 to all the rest of the input tokens 103. This is depicted in Fig. 4.

[0085] As shown in Fig. 4, summing over each line gives you the correlation score per input token 103, from which the topmost input tokens 103 may be picked. To pick the most informative input tokens 103, simply pick top-n tokens.

[0086] The compression of the embeddings 105 may be carried out such that the total context length of the part 101a, 101b, 101c, 101 d of the context input 101 and the compressed embeddings 109 is less than the context limit of the model 104. In this way, the part 101a, 101b, 101c, 101 d of the context input 101 and the compressed embeddings 109 may be re-used by the model 104 for further iterations. The compression may thus be controlled by the objective of reducing the total context length of the part 101a, 101b, 101c, lOld of the context input 101 and the compressed embeddings 109 to below the context limit of the model 104.

[0087] The part 101a, 101b, 101c, 101 d of the context input 101 and the compressed embeddings 109 may be input into the model 104. The model 104 may be used to generate further output tokens 106. The model 104 may use the part 101a, 101b, 101c, 101 d of the context input 101 and the compressed embeddings 109 to generate the further output tokens 106. The further output tokens 106 may be improved over the output tokens 106 due to the extra information provided by the compressed embeddings 109.

[0088] The model 104 may output a token output 110. The token output 110 may be a part of the output which is returned to the user in response to the part 101a, 101b, 101c, 101 d of the context input 101. The model 104 may output the token output 110 in dependence on the further output tokens 106. The further output tokens 106 may form part of the token output 110.

[0089] The model 104 may also output further embeddings 105. The model 104 may generate further embeddings 105 from the part 101a, 101b, 101c, 101 d of the context input 101 and the compressed embeddings 109. The self-information of the input tokens 103 may be calculated from the representation 200 of the further embeddings 105. The further embeddings 105 may be compressed in dependence on the self-information of the input tokens 103. The model 104 may generate more further output tokens 106 from the part 101a, 101b, 101c, lOld of the context input 101 and the compressed further embeddings 109. The model 104 may output the token output 110 in dependence on the more further output tokens 106.

[0090] Put another way, the process described with reference to Fig. 1 , may be repeated for one or more iterations. The iterations may generate one or more further sets of output tokens 106. The token output 110 may be formed from one or more of the sets of output tokens 106. All of the sets of output tokens may be used to generate the token output 110. Alternatively, the most recent, or last generated, set of output tokens 106 may be used to generate the token output 110.

[0091] The process described in relation to Fig. 1 may be repeated for each of the parts 101a, 101b, 101c, 101 d of the context input 101. In this way, the entire context input 101 may be run through the model 104. This allows for a context input 101 which has a context length N 102a above the model 104 context length limit to still be run through the model 104 in parts 101a, 101b, 101c, lOld. The token output 110 for each part 101a, 101b, 101c, 101 d of the context input 101 may be combined to form the output of the model 104. The output of the model 104 is then returned to the user.

[0092] The model 104 may be a language model. The input context 101 may be text. The token output 110 may be text. In particular, the language model may be configured to carry out one or more language model tasks of: question answering, information retrieval, text summarisation, and reading comprehension.

[0093] Figs. 5a and 5b illustrate the output of the present system compared to the prior art. The memory efficiency of the present system may be straight forward. It may simply allow a model with a fixed batch size to handle a practically infinite sequence. It is possible that there may be some deterioration in the accuracy of such model. In order to evaluate the accuracy degradation, a benchmark that uses long sequence length is used. Two such benchmarks have been used to compare prior art against the present system: Longbench [Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang and J. Li, "LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding," arXiv, 2023] and LV-Eval [T. Yuan, X. Ning, D. Zhou, Z. Yang, S. Li, M. Zhuang, Z. Tan, Z. Yao, D. Lin, B. Li, G. Dai, S. Yan and Y. Wang, "LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K," arXiv, 2024] to compare MSIC result to other prior art algorithms, Streaming LLM, and LLM Lingua, as well as a Prompt-cutting technique suggested by [Bai et al.] that takes the first and last parts of the prompt according to the half of the KV cache budget. As a model ChatGLM-3 is used in all tests. As seen in Table 501 and Table 502, results of the present system outperform other technologies in most scenarios.

[0094] Fig. 6 summarises an example of a method 600 for generating a sequential model output. At step 601 , the method 600 comprises obtaining a part of a context input as input tokens. At step 602, the method 600 comprises generating embeddings from the part of the context input using the model. At step 603, the method 600 comprises calculating the self- information of the input tokens from the representation of the embeddings. At step 604, the method 600 comprises compressing the embeddings in dependence on the self-information of the input tokens. At step 605, the method 600 comprises generating further output tokens from the part of the context input and the compressed embeddings using the model. At step 606, the method 600 comprises outputting a token output in dependence on the further output tokens.

[0095] The algorithm of the present system may be applicable on any model that generates outputs. The model may be stored or accessible through a computer system, such as a PC, cloud compute server, mobile device or other computing device. An example of an apparatus 700 configured to implement the method 600 is schematically illustrated in Fig. 7. The computing apparatus 700 may comprise the pipeline 100. The apparatus 700 may be implemented on an electronic device, such as a computer, laptop, tablet, or smart phone.

[0096] The apparatus 700 comprises a processor 701 configured to process the datasets in the manner described herein. For example, the processor 701 may be implemented as a computer program running on a programmable device such as a Central Processing Unit (CPU). The apparatus 700 comprises a memory 702 which is arranged to communicate with the processor 701. Memory 702 may be a non-volatile memory. The processor 701 may also comprise a cache (not shown in Fig. 7), which may be used to temporarily store data from memory 702. The apparatus 700 may comprise more than one processor 701 and more than one memory 702. The memory 702 may store data that is executable by the processor 701. The processor 701 may be configured to operate in accordance with a computer program stored in non-transitory form on a machine-readable storage medium. The computer program may store instructions for causing the processor to perform its methods in the manner described herein. The method steps described herein may be carried out by a computer-readable storage medium. The method steps described herein may be carried out by a computer program product.

[0097] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present disclosure may consist of any such individual feature or combination of features. In view of the foregoing description, it will be evident to a person skilled in the art that various modifications may be made within the scope of the disclosure.

Claims

CLAIMS1. A computing apparatus (700) for generating an output from a model (104), the computing apparatus (700) comprising one or more processors (701) and a memory (702) storing in non-transient form data defining program code executable by the one or more processors (701), wherein the program code is executable by the one or more processors (701) so that the computing apparatus (700) is configured to: obtain a part (101a, 101b, 101c, 101d) ofa context input (101) as input tokens (103); generate embeddings (105) from the part (101a, 101b, 101c, lOld) of the context input (101) using the model (104), calculate the self-information of the input tokens (103) from the representation (201) of the embeddings (105); compress the embeddings (105) in dependence on the self-information of the input tokens (103); generate further output tokens (106) from the part (101a, 101b, 101c, 101 d) of the context input (101) and the compressed embeddings using the model (104); and output a token output (110) in dependence on the further output tokens (106).

2. The computing apparatus (700) according to claim 1, wherein the computing apparatus (700) is configured to obtain the part (101a, 101b, 101c, 101 d) of the context input (101) by splitting the context input (101) into a plurality of parts (101a, 101b, 101c, lOld).

3. The computing apparatus (700) according to claim 2, wherein the computing apparatus (700) is configured to split the context input (101) into a plurality of parts (101a, 101b, 101c, 101 d) each of which comprise a context length ( 102b) below a context length limit of the model (104).

4. The computing apparatus (700) according to claim 2 or 3, wherein the computing apparatus (700) is configured to repeat the steps of any of claims 1 to 3 for each of the parts (101a, 101b, 101c, 10 Id) of the context input (101).

5. The computing apparatus (700) according to any preceding claim, wherein the representation (201 ) of the embeddings (105) used to calculate the self-information of the input tokens (103) are represented as a vector (201a, 201b, 201c, 201d).

6. The computing apparatus (700) according to any preceding claim, wherein the computing apparatus (700) is configured to, after the embeddings (105) are generated, store the part (101a, 101b, 101c, lOld) of the context input (101) and the embeddings (105) in a keys and values cache (107).

7. The computing apparatus (700) according to any preceding claim, wherein the computing apparatus (700) is configured to compress the embeddings (105) by removing input tokens (103) with a self-information below a self-information threshold.

8. The computing apparatus (700) according to claim 7, wherein the computing apparatus (700) is configured to compress the embeddings (105) by only removing input tokens (103) in a specific portion of the embeddings (105).

9. The computing apparatus (700) according to any preceding claim, wherein the self-information comprises the probability of an input token (103), and / or wherein the self-information comprises the similarly of an input token (103) compared to other input tokens (103).

10. The computing apparatus (700) according to any of claims 7 to 9, wherein the computing apparatus (700) is configured to compress the embeddings (105) by removing input tokens (103) such that the total context length of the part (101a, 101b, 101c, 101 d) of the context input (101) and the compressed embeddings (105) is less than the context length limit of the model (104).

11. The computing apparatus (700) according to any preceding claim, wherein the computing apparatus (700) is configured to generate output tokens (106) from the part (101a, 101b, 101c, 101 d) of the context input (101) using the model (104).

12. The computing apparatus (700) according to any preceding claim, wherein the computing apparatus (700) is configured to: generate further embeddings (105) from the part (101a, 101b, 101c, 10 Id) of the context input (101) and the compressed embeddings (105) using the model (104), calculate the self- information of the input tokens (103) from the representation (201) of the further embeddings (105); compress the further embeddings (105) in dependence on the self-information of the input tokens (103); generate more further outputtokens (106) from the part (101a, 101b, 101c, 10 Id) of the context input (101) and the compressed further embeddings using the model (104); and output the token output (110) in dependence on the more further output tokens (106).

13. The computing apparatus (700) according to any preceding claim, wherein the model (104) is a language model, the input context (101) is text, and the token output (110) is text.

14. The computing apparatus (700) according to claim 13, wherein language model is configured to carry out one or more language model tasks of: question answering, information retrieval, text summarisation, and reading comprehension.

15. A method (600) for generating an output from a model, the method (600) comprising steps of: obtaining a part of a context input (601 ) as input tokens; generating embeddings from the part of the context input using the model (602), calculating the self-information of the input tokens from the representation of the embeddings (603); compressing the embeddings in dependence on the self-information of the input tokens (604); generating further output tokens from the part of the context input and the compressed embeddings using the model (605); and outputting a token output in dependence on the further output tokens (606).

Images