Electronic device for predicting tokens of output sequence and method thereof

Abstract

In one embodiment, a method includes predicting, by a decoder of an LLM and in response to an input sequence provided to an encoder of the LLM, s tokens of an output sequence. The method further includes accessing, for each of one or more attention layers of the LLM, a set of attention logits specific to that attention layer and used by the LLM to predict the n most recent tokens of the s tokens; determining, for each of the one or more attention layers and by a trained mask generation model, a layer-specific attention mask for the set of attention logits specific to that attention layer; and predicting, by the decoder of the LLM, the next m tokens of the output sequence using the set of attention logits as masked by the layer-specific attention mask for each layer.

Description

ELECTRONIC DEVICE FOR PREDICTING TOKENS OF OUTPUT SEQUENCE AND METHOD THEREOF

[0001] This application generally relates to techniques for memory efficient attention window expansion for trained LLMs.

[0002] A large language model (LLM) is a type of machine-learning model designed for natural language processing tasks, such as language generation. LLMs have many parameters and are trained on very large corpuses of natural-language input. LLMs typically have multiple layers of neural networks, each with parameters that can be tuned during training, and also have multiple attention layers, which focus on specific portions of a token sequence.

[0003] To perform natural-language tasks, LLMs operate on embedded token sequences, which are numerical representations of portions of natural language. There is a wide variance in the amount of text that one token can represent: a token can represent a single character, a part of a word (such as a suffix or prefix), a whole word, or a multiword phrase. Different LLMs can have different embeddings and tokenization of the same natural-language input, based on their architectures and training.

[0004] An LLM's context window determines how much an LLM's output can be influenced by the LLM's prior natural-language input and output, similar to how a phrase's meaning can be informed in conversation or in writing by nearby words that are not directly part of the phrase. By effectively managing larger contexts, LLMs can maintain coherence over long conversations, helping virtual assistants and chatbots to provide relevant and accurate responses based on dialog history. Moreover, the ability to sift through vast quantities of information to identify specific data points allows for more efficient knowledge discovery and decision-making in natural-language tasks. An LLM's context window requires computer memory, often GPU memory, to store relevant contextual information.

[0005] According to an embodiment of the disclosure, a method is disclosed. The method may include predicting, by a decoder of a Large Language Model (LLM), in response to an input sequence provided to an encoder of the LLM, s tokens of an output sequence. The method may include accessing, for each of one or more attention layers of the LLM, a first set of attention logits corresponding to each of the one or more attention layers, wherein the first set of attention logits are used by the LLM to predict n most recent tokens of the s tokens. The method may include determining, by a trained mask generation model, a layer-specific attention mask for the first set of attention logits corresponding to each of the one or more attention layers, based on the first set of attention logits. The method may include predicting, by the decoder of the LLM, next m tokens of the output sequence using, for each of the one or more attention layers, the first set of attention logits as masked by the layer-specific attention mask corresponding to each of the one or more attention layers.

[0006] According to an embodiment of the disclosure, an electronic device is disclosed. The electronic device may include at least one memory storing one or more instructions. The electronic device may include one or more processors including processing circuitry, wherein the one or more instructions executed by the one or more processors individually or collectively, to cause the electronic device to: predict, by a decoder of a Large Language Model (LLM), in response to an input sequence provided to an encoder of the LLM, s tokens of an output sequence. According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to access, for each of one or more attention layers of the LLM, a first set of attention logits corresponding to each of the one or more attention layers, wherein the first set of attention logits are used by the LLM to predict n most recent tokens of the s tokens. According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to determine, by a trained mask generation model, a layer-specific attention mask for the first set of attention logits corresponding to each of the one or more attention layers, based on the first set of attention logits. According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to predict, by the decoder of the LLM, next m tokens of the output sequence using, for each of the one or more attention layers, the first set of attention logits as masked by the layer-specific attention mask corresponding to each of the one or more attention layers.

[0007] According to an embodiment of the disclosure, a computer-readable medium containing instructions, wherein the instructions, when executed by at least one processor, cause the electronic device (100) to perform the method herein is disclosed.

[0008] Fig. 1 illustrates an example memory efficient attention window expansion method for trained LLMs.

[0009] Fig. 2 illustrates an example architecture for implementing the example method of Fig. 1.

[0010] Fig. 3 illustrates an example of a layer-specific attention mask output by a mask generation model.

[0011] Fig. 4 illustrates an example electronic device.

[0012] For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the various embodiments and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the present disclosure is thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the present disclosure as illustrated therein being contemplated as would normally occur to one skilled in the art to which the present disclosure relates.

[0013] It will be understood by those skilled in the art that the foregoing general description and the following detailed description are explanatory of the present disclosure and are not intended to be restrictive thereof.

[0014] Whether or not a certain feature or element was limited to being used only once, it may still be referred to as "one or more features" or "one or more elements" or "at least one feature" or "at least one element." Furthermore, the use of the terms "one or more" or "at least one" feature or element do not preclude there being none of that feature or element, unless otherwise specified by limiting language including, but not limited to, "there needs to be one or more..." or "one or more elements is required."

[0015] Reference is made herein to some "embodiments." It should be understood that an embodiment is an example of a possible implementation of any features and / or elements of the present disclosure. Some embodiments have been described for the purpose of explaining one or more of the potential ways in which the specific features and / or elements of the proposed disclosure fulfil the requirements of uniqueness, utility, and non-obviousness.

[0016] Use of the phrases and / or terms including, but not limited to, "a first embodiment," "a further embodiment," "an alternate embodiment," "one embodiment," "an embodiment," "multiple embodiments," "some embodiments," "other embodiments," "further embodiment", "furthermore embodiment", "additional embodiment" or other variants thereof do not necessarily refer to the same embodiments. Unless otherwise specified, one or more particular features and / or elements described in connection with one or more embodiments may be found in one embodiment, or may be found in more than one embodiment, or may be found in all embodiments, or may be found in no embodiments. Although one or more features and / or elements may be described herein in the context of only a single embodiment, or in the context of more than one embodiment, or in the context of all embodiments, the features and / or elements may instead be provided separately or in any appropriate combination or not at all. Conversely, any features and / or elements described in the context of separate embodiments may alternatively be realized as existing together in the context of a single embodiment.

[0017] Any particular and all details set forth herein are used in the context of some embodiments and therefore should not necessarily be taken as limiting factors to the proposed disclosure.

[0018] The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process or method that comprises a list of steps does not include only those steps but may include other steps not expressly listed or inherent to such process or method. Similarly, one or more devices or sub-systems or elements or structures or components proceeded by "comprises... a" does not, without more constraints, preclude the existence of other devices or other sub-systems or other elements or other structures or other components or additional devices or additional sub-systems or additional elements or additional structures or additional components.

[0019] It should be appreciated that the blocks in each flowchart and combinations of the flowcharts may be performed by one or more computer programs which include computer-executable instructions. The entirety of the one or more computer programs may be stored in a single memory or the one or more computer programs may be divided with different portions stored in different multiple memories.

[0020] Any of the functions or operations described herein can be processed by one processor or a combination of processors. The one processor or the combination of processors is circuitry performing processing and includes circuitry like an application processor (AP), a communication processor (CP), a graphical processing unit (GPU), a neural processing unit (NPU), a microprocessor unit (MPU), a system on chip (SoC), an IC, or the like.

[0021] The processor may include various processing circuitry and / or multiple processors.　 For example, as used herein, including the claims, the term "processor" may include various processing circuitry, including at least one processor, wherein one or more of at least one processor, individually and / or collectively in a distributed manner, may be configured to perform various functions described herein. As used herein, when "a processor", "at least one processor", and "one or more processors" are described as being configured to perform numerous functions, these terms cover situations, for example and without limitation, in which one processor performs some of recited functions and another processor(s) performs other of recited functions, and also situations in which a single processor may perform all recited functions. Additionally, the at least one processor may include a combination of processors performing various of the recited / disclosed functions, e.g., in a distributed manner.　 At least one processor may execute program instructions to achieve or perform various functions.

[0022] Embodiments of the present disclosure will be described below in detail with reference to the accompanying drawings.

[0023] For the sake of clarity, the first digit of a reference numeral of each component of the present disclosure is indicative of the Figure number, in which the corresponding component is shown. For example, reference numerals starting with digit "1" are shown at least in Figure 1. Similarly, reference numerals starting with digit "2" are shown at least in Figure 2

[0024] Increasing an LLM's context window improves the LLM's performance on natural-language tasks, but also increases the memory resources required to store that context. In addition, the memory used by an LLM is typically GPU memory, which can be particularly resource-intensive to scale. Moreover, when LLMs use deep attention mechanisms, which provide superior performance, the memory requirements scale quadratically with context length (i.e., doubling the context window requires 4 times as much memory). Thus, an LLM's context window is typically limited by the available memory to store context. This is particularly true for edge-deployed LLMs, (e.g., LLMs deployed on client devices such as personal computers, smartphones, etc.), which typically have fewer computational resources than do server-deployed LLMs, although both types of deployments have context windows that are constrained by memory limitations.

[0025] Apart from adding more memory, one approach for increasing an LLM's context window is to re-train the LLM with curated datasets that contain large inputs or outputs. However, LLM model training is a very expensive and resource-intensive process, and is often impractical. Other approaches use inference-time techniques to increase an LLM's context window without having to re-train the entire model. For instance, inference-time techniques may use an attention mask to increase the attention window, but such masks are either static (i.e., stay the same for each LLM layer and for each iteration) or are based on statistical evaluations of a particular LLM's architecture. In real-world use cases, these approaches suffer from poor performance.

[0026] In contrast, the techniques of this disclose use a variable, layer-specific attention mask computed by a trained mask-generation model during inference. As described below, these techniques can modify the layer-specific attention masks during inference as the attention logits in a particular layer evolve during inference iterations. As explained below, by using a variable mask that looks at attention logits at each layer for previous tokens generated, these techniques can focus each layer's attention window towards tokens that yield the largest attention logits, regardless of their position. These techniques do not require model retraining, and can be used in conjunction with other LLM optimizations. Moreover, the adaptive masked attention generation techniques described herein (in conjunction with sparse attention kernels, in an embodiment) increase the effective attention window of an LLM, for example enabling the model to maintain constant or near-constant context window memory overhead while achieving near loss-less long context performance, in an embodiment.

[0027] Fig. 1 illustrates an example memory efficient attention window expansion method for trained LLMs. Step 110 of the example method of Fig. 1 includes predicting, by a decoder of an LLM and in response to an input sequence provided to an encoder of the LLM, s tokens (e.g. predetermined number of tokens) of an output sequence. The input sequence may be typically text input (e.g., text provided by an end user, transcribed verbal input from a user, etc.), and the decoder may determine the corresponding natural-language input for the task. The s tokens (predetermined number of tokens) may represent s iterations of the inference task performed by the LLM decoder.

[0028] Fig. 2 illustrates an example architecture for implementing the example method of Fig. 1. In the example of Fig. 2, input 202 is provided to an LLM and is encoded by the LLM's encoder. The LLM's text-generation architecture is shown in Fig. 2, which includes an embedding layer 204 for embedding input 202. The LLM of Fig. 2 includes multiple transformer blocks 210, each of which include an RMS normalization layer 212, a multi-head attention layer 214, an RMS normalization layer 216, a feed forward layer 218, and an activation layer 220. The output of the transformer blocks may be sent to another RMS normalizing layer 222, a linear layer 224, and a softmax layer 226. The output probabilities 228 are then used to determine the final sequence that the LLM will generate in response to some input. The example of Fig. 2 illustrates an architecture of an LLM text-generation model, and illustrates only a single instance of various layers when in reality multiple such layers are used in an LLM decoder; this illustration is for example purposes only and the techniques described herein are not limited to the specific architecture shown in Fig. 2.

[0029] In the example of Fig. 2, sequence s 230 may represent the output sequence of the LLM after s inference iterations. At each inference iteration, each attention layer 214 determines a new set of attention logits based on the cached attention logit values for that layer stored in that layer's KV cache 250. In an embodiment, an attention layer 214 may write a new row of attention logits to its KV cache 250 during each iteration. An LLM has many attention layers, each with its own KV cache (or dedicated portion of a KV cache), and each of which has its own set of attention logits that will influence the overall output sequence s.

[0030] Step 120 of the example method of Fig. 1 includes accessing, for each of one or more attention layers of the LLM, a set of attention logits specific to (e.g. corresponding to) that attention layer and used by the LLM to predict the n most recent tokens of the s tokens in the output sequence. In other words, at the conclusion of the sth inference iteration (where s is one or more), then for each of one or more attention layers (e.g., all the attention layers, or just some of the attention layers of the LLM), the attention logits specific to (e.g. corresponding to) that attention layer and used to predict the most recent n tokens are accessed. As explained above, the attention logits are typically stored (in indexed form) in a KV cache, and the attention logits for a specific attention layer referenced in step 120 are typically accessed from the KV cache (although, in an embodiment, such logits may be accessed by logging those logits from the respective attention layer).

[0031] The most recent n tokens are accessed from the s tokens, where n is less than or equal to s. In an embodiment, n is a hyperparameter. In an embodiment, n may vary among iterations of the method of Fig. 1, and / or may vary among attention layers (e.g., n may take a different value for different attention layers). In an embodiment, n is a hyperparameter that may be the same for each of the one or more attention layers (e.g., all the attention layers) referenced in step 120.

[0032] Step 130 of the example method of Fig. 1 includes determining, for each of the one or more attention layers and by a trained mask generation model, a layer-specific attention mask for the set of attention logits specific to that attention layer, based on the accessed set of attention logits. In the example of Fig. 2, mask generation model (MGM) 240 includes a projection 242, which linearizes the matrix of logit values and, in an embodiment, concatenates sequence s 230 to that linearized vector. The attention logits for a particular layer accessed in step 120 are then input (e.g., after projection) to an encoder 244 and decoder 246, which are trained to output an attention mask based on the input attention logits. In an embodiment, the input attention logits may be represented via a heatmap.

[0033] In an embodiment, the encoder and decoder of a mask generation model may be vision transformers that are fine-tuned for the mask generation task. For instance, an MGM may be a compact transformer encoder adapted from ViT-base of the SAM ViT-H model, with 6 layers, hidden size of 512, 8 attention heads, and a feed-forward dimension of 2048, although other parameters may be used.

[0034] In the example of Fig. 2, MGM 240 takes as input a low-dimensional projection of the current token embeddings and outputs a sparse mask , where N is the sequence length. In an embodiment, the mask generation may be informed by a hyperparameter k, e.g, (M = TopK(Sigmoid(F),k)). In an embodiment, an MGM specifically interacts with the decoder-only masked attention layers of the VLM, i.e. the decoder-only masked LLM component of the VLM.

[0035] In an embodiment, an MGM may be trained on a diverse corpus of attention patterns sampled from various VLMs and tasks from a long-context subset of particular datasets, minimizing the loss, such as the binary cross-entropy loss, between the predicted mask and the ground truth sparse attention patterns. Other examples of loss mechanisms includes IOU (intersection over union or dice) loss, weighted (or focal) binary cross entropy or cross entropy loss, and KLD (Kullback-Leibler divergence) loss, and this disclosure contemplates that any suitable loss mechanism may be used. To adapt the MGM to specific VLMs and downstream tasks, an embodiment may employ reinforcement learning. For example, an embodiment may employ Odds-Ratio Preference Optimization (ORPO), which fine-tunes the MGM by recursively optimizing its parameters based on the output of the target VLM. The value of k in the TopK operation is, in an embodiment, dynamically adjusted based on the current context length and a target sparsity ratio r: k = max(kmin, min(kmax, round(r N))). The sparse attention operation may be computed as and O=AV, where Q, K, and V are the query, key, and value matrices respectively, and ⊙ denotes element-wise multiplication. To optimize this operation, an embodiment implements a revised CUDA Triton kernel that efficiently handles the sparse matrix multiplication and softmax operations. An embodiment may incorporate shared memory usage, warp-level primitives for efficient parallel reduction, and block-sparse matrix multiplication for the QKT operation.

[0036] In an embodiment, an MGM may be fine-tuned on a synthetic dataset created (e.g. generated) by sampling attention patterns from the LLM across various tasks and input sequences. For example, a dataset may include pairs (At-n:t-1, Mt), where At-n:t-1 are the attention logits from the previous n tokens, and Mt is the corresponding optimal attention mask at time (i.e., iteration) t.

[0037] Training may be performed, in an embodiment, using the Adam optimizer with a learning rate of, e.g., 1e-4 and a batch size of 8 (gradient accumulation to avoid OOM). Early stopping may be employed based on validation loss to prevent overfitting. Training may run for a number of epochs (e.g., 3), with cosine learning rate and warm up ratio of 0.1, with early stoppage if loss stabilizes.

[0038] To adapt VLM with MGM integration, an embodiment may use ORPO. For instance, starting from a LLaMA 3.2 90B Vision Instruct model, with the help of quantized ORPO an embodiments train and preference-align on a single 48GB GPU (A6000) with CPU offloading (through DeepSpeed ZeRO Stage 3), or on two 48 GB GPUs (2xA6000) for faster training.

[0039] For training, an embodiment may use a dataset with a number of samples (e.g., 1,200) compiled from an internal long context visual document retrieval dataset (extracting product information, UI context, and user flows from PDFs) to answer related questions from FAQ Question Answer pairs. An embodiment may first augment the dataset with negative samples (incomplete variants, and merged variants), and context mapping samples (samples with manual context retrieved and appended to questions), for a total of, e.g., 12,000 samples. An embodiment may then run training for 3 epochs with cosine learning rate scheduler, with learning rate starting at 1e-5, warmup ratio of 0.1, and batch-size of 8 (gradient accumulation to avoid OOM errors), using the 8-bit Adam optimizer, with early stoppage if loss stabilizes. For preference optimization in ORPO, an embodiment may use beta=0.1.

[0040] The above description regarding specific training approaches and parameters for particular mask generation models are for example purposes only, and are not exhaustive. This disclosure contemplates that other training parameters and procedures may be used to train a mask generation model based on input training and ground-truth data. For instance, an embodiment may take the output of a high-quality (e.g., GPT) LLM model for a given input and use as ground truth what the MGM model's sparse attention would be in order to generate the same sequence as generated by high-quality model. Moreover, while certain examples described above use a vision transformer architecture for the mask generation model, this disclosure contemplates that other encoder-decoder architectures may be used to output the layer-specific attention mask based on the attention logits for that layer from the previous n iterations.

[0041] Fig. 3 illustrates a simple example of a layer-specific attention mask output by a mask generation model. Each value is a logit, and its value is the attention of token r for token c (basically, the key (K) times the value (V)), where r is the row index and c is the column index. For instance, logit 302 has a value 2 and is indexed by r=1 and c=0. As illustrated in the example of Fig. 3, the mask generation model allocates attention in the mask in a non-uniform manner. For instance, in the example of Fig. 3, several logits are masked (i.e., are given the value 0). By modifying the attention mask to ignore regions with lower allocation, the techniques described herein increase the effective attention window for the LLM.

[0042] Step 140 of the example method of Fig. 1 includes predicting, by the decoder of the LLM, the next m tokens of the output sequence using, for each of the one or more attention layers, the set of attention logits as masked by the layer-specific attention mask for that layer. In an embodiment, each attention layer outputs logit values to the next layer, and these logit values are typically stored in the KV cache. Thus, a layer-specific mask is used to modify the logit values for its layer, which are then used by the subsequent layer during the m inference iterations.

[0043] In an embodiment, m is a tunable parameter that may determine how many iterations of the output sequence will be performed before the layer-specific mask is updated. In an embodiment, the attention mask is used to generate sparse attention logits for its corresponding layer. That set of sparse attention logits is then used for m iterations (i.e., to predict the next m tokens following the s tokens that have already been predicted). As illustrated in the example of Fig. 2, each layer's attention mask may be used to compress the KV cache, which stores the indexed value of the attention logits for a specific layer. In an embodiment, the indices of removed logits (i.e., logits given the value of 0 in the attention mask) may be preserved so that if the attention layer refers to those logits in the next m iterations, a zero value is returned for that reference.

[0044] The method of Fig. 1 may be repeated iteratively for a particular inference task. In an embodiment, n and m may have a fixed particular value during iterations of the method of Fig. 1 for a particular inference task. In an embodiment, either or both of n or m may vary among at least some of the iterations (e.g., m may be 16 in the first iteration, and may take a larger or smaller value in the next iteration, etc.).

[0045] In an embodiment, such as the example of Fig. 2, a mask generation model generates a layer-specific mask based on both (1) the attention logits for the previous n sequence iterations and (2) the previous sequence (e.g., sequence s 230). For example, the sequence s may be concatenated with the linearized vector of logit values, and this concatenated vector may be input via projection 242 to encoder / decoder 244 and 246 to generate the layer-specific attention mask. Doing so improves the attention allocation of the mask, for example by taking into account attention allocation across the full set of layers (as evidence by the sequence s), in addition to the layer-specific attention logits in the KV cache.

[0046] In an embodiment, an iteration of the example method of Fig. 1 may evaluate a deviation, or difference, between the attention in the sequence s and the masked attention for particular layer as identified by the masked attention logit values (e.g., the masked values in the KV cache for that layer). If the difference is greater than a threshold, this indicates that the layer-specific mask attenuated attention for a particular layer in areas that globally (looking at the attention output from all layers) are in fact relevant to the output sequence. For example, suppose token IDs 4-6 are given zero attention by a mask for a particular layer, but the sequence s indicates that the attention for IDs 4-6 is relatively high. This indicates that the layer-specific mask is deviating from the global attention values determined by the LLM's full text-generation model.

[0047] In such instances, an embodiment may revert that layer's attention logit values to their previous values (i.e., the values that were present before the most recent instance of the applied mask) and then regenerate a mask for those logits. In an embodiment, mask generation may be a stochastic process, such that recompression will alter the attention logit values from those that previously led to a too-large deviation. These approaches prevent masking from deviating a layer's attention logits too far from the overall attention values present in a particular sequence s. In an embodiment, these differences may be evaluated during each iteration of the example method of Fig. 1 (e.g., every m inference iterations). In other embodiments, these differences may be evaluated more frequently (e.g., at every inference iteration).

[0048] In an embodiment, by using global attention and model output to dynamically verify and optimize the eviction and scarification steps, an embodiment may outperform existing cache compression or dynamic attention approaches that employ fixed strategies, i.e. they tend to blindly follow their compression or sparsification strategies without verifying and dynamically adapting based on global attention and model output.

[0049] Algorithm 1 below, also referred to as the CALC-LLM (context-adaptive layerwise compression for LLMs) algorithm, illustrates a particular implementation of the example method of Fig. 1 that also evaluates the deviation between mask logit values and a sequence.

[0050]

[0051] In the example implementation of algorithm 1, k may be determined based on the amount of available memory to store context. In an embodiment, k may vary from layer to layer.

[0052] CALC-LLM combines dynamic sparse attention with adaptive KV-cache compression to enable efficient processing of long sequences in VLMs. This approach leverages a lightweight Mask Generation Model (MGM) to dynamically generate sparse attention patterns, coupled with a novel adaptive compression technique for the KV-cache. The adaptive KV-cache compression technique combines frequency-based and recency-based importance scoring. For each key-value pair (ki, vi) in the cache, an embodiment maintain a frequency counter fi and a timestamp ti of the last access. The importance score for each pair may be computed as

[0053] (1)

[0054] where balances frequency and recency, is the maximum frequency across all pairs, is the current timestamp, and is a sliding window size. Based on these scores, an embodiment apply a dynamic compression ratio to each pair: . The compression may be implemented using a combination of pruning (removing pairs if ) and quantization (quantizing to )) bits). An embodiment may use a modified version of the ZeroQuant algorithm for efficient quantization, adapting it to handle dynamic bit-widths and leveraging mixed-precision arithmetic to maintain accuracy while reducing memory footprint.

[0055] To integrate CALC-LLM with relevant architectures, an embodiment may replace the standard attention mechanism and KV-cache with the dynamic sparse attention and adaptive compression modules. The integration process involves, for example, initializing the MGM with pre-trained parameters, generating attention masks at regular intervals or when significant deviations are detected, applying the mask to attention logits, computing sparse attention output using a CUDA kernel, updating positional embeddings using dynamic NTK scaling, and updating and compressing the KV-cache. In an embodiment, compression may occur through eviction based on the MGM output.

[0056] An embodiment may employ one or more of several optimizations to maximize efficiency. For example, fused CUDA kernels combine multiple operations (e.g., mask generation with sparse attention, KV-cache compression and decompression) to reduce memory bandwidth usage. Mixed precision computation leverages FP16 for MGM computations and a combination of FP16 and FP32 for sparse attention and KV-cache operations, with careful management of accumulation to maintain numerical stability. In an embodiment, the precision of value may be Int8 if quantized (or int4 in alternative embodiments, the precision can go as low as needed), or BF16 in GPU when unquantized (in place of FP16, for instance because BF16 is more optimized for GPUs). Other GPU architectures can also use FP8 or FP4. A custom memory pool pre-allocates large chunks of GPU memory, implements a slab allocator for efficient small allocations, and uses memory defragmentation techniques to coalesce free spaces periodically. Adaptive hyper-parameter tuning monitors key metrics (perplexity, attention entropy, memory usage) during inference and adjusts hyper-parameters using Bayesian optimization, with a warm-up period to stabilize estimates.

[0057] To handle sequences longer than those seen during pre-training, an embodiment may employ a dynamic positional embedding interpolation technique using Neural Tangent Kernels (NTK) with frequency-scaled temperature. For a new sequence length , such embodiments compute interpolated embeddings as:

[0058] (2)

[0059] where is the NTK with frequency-scaled temperature: , and with learnable parameters and .

[0060] In an embodiment, the time complexity of CALC-LLM is whereNis the sequence length andris the sparsity ratio. The space complexity is , enabling processing of much longer sequences compared to standard transformers, which typically have a quadratic space complexity.

[0061] This disclosure contemplates that the hyperparametersn,m, andkmay take various values; for examplen= 128,m=16, andkmay be a dynamic value having a maximum of 64.

[0062] Fig. 4illustrates an example electronic device 400. In an embodiment, one or more electronic devices 400 perform one or more steps of one or more methods described or illustrated herein. In an embodiment, one or more electronic devices 400 provide functionality described or illustrated herein. In an embodiment, software running on one or more electronic devices 400 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. An embodiment may include one or more portions of one or more electronic devices 400. Herein, reference to an electronic device may encompass a computing device, and vice versa, where appropriate. Moreover, reference to an electronic device may encompass one or more electronic devicess, where appropriate.

[0063] This disclosure contemplates any suitable number of electronic devices 400. This disclosure contemplates electronic device 400 taking any suitable physical form. As example and not by way of limitation, electronic device 400 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, electronic device 400 may include one or more electronic devices 400; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more electronic devices 400 may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein. As an example and not by way of limitation, one or more electronic devices 400 may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more electronic devices 400 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.

[0064] In an embodiment, computer system 400 includes a processor 402, memory 404, an input / output (I / O) interface 408, a communication interface 410, and a bus 412. Although this disclosure describes and illustrates a particular electronic device having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable electronic device having any suitable number of any suitable components in any suitable arrangement.

[0065] In an embodiment, processor 402 includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor 402 may retrieve (or fetch) the instructions from an internal register, an internal cache, or memory 404; decode and execute them; and then write one or more results to an internal register, an internal cache, memory 404, or storage 406. In an embodiment, processor 402 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor 402 including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor 402 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory 404, and the instruction caches may speed up retrieval of those instructions by processor 402. Data in the data caches may be copies of data in memory 404 for instructions executing at processor 402 to operate on; the results of previous instructions executed at processor 402 for access by subsequent instructions executing at processor 402 or for writing to memory 404; or other suitable data. The data caches may speed up read or write operations by processor 402. The TLBs may speed up virtual-address translation for processor 402. In an embodiment, processor 402 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor 402 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor 402 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 402. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0066] In an embodiment, memory 404 includes main memory for storing instructions for processor 402 to execute or data for processor 402 to operate on. As an example and not by way of limitation, electronic device 400 may load instructions from storage 406 or another source (such as, for example, another electronic device 400) to memory 404. Processor 402 may then load the instructions from memory 404 to an internal register or internal cache. To execute the instructions, processor 402 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 402 may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor 402 may then write one or more of those results to memory 404. In an embodiment, processor 402 executes only instructions in one or more internal registers or internal caches or in memory 404 (as opposed to storage 406 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 404 (as opposed to storage 406 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor 402 to memory 404. Bus 412 may include one or more memory buses, as described below. In an embodiment, one or more memory management units (MMUs) reside between processor 402 and memory 404 and facilitate accesses to memory 404 requested by processor 402. In an embodiment, memory 404 includes random access memory (RAM). This RAM may be volatile memory, where appropriate Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory 404 may include one or more memories 404, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

[0067] In an embodiment, memory 404 may include GPU memory (VRAM). Model weights and KV cache are loaded into GPU memory in an embodiment, as described above.

[0068] In an embodiment, memory 404 may include mass storage for data or instructions. As an example and not by way of limitation, memory 404 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Memory 404 may include removable or non-removable (or fixed) media, where appropriate. Memory 404 may be internal or external to computer system 400, where appropriate. In an embodiment, memory 404 is non-volatile, solid-state memory. In an embodiment, memory 404 includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage 406 taking any suitable physical form. Memory 404 may include one or more storage control units facilitating communication between processor 402 and memory 404, where appropriate. Where appropriate, memory 404 may include one or more memories 404. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0069] In an embodiment, I / O interface 408 includes hardware, software, or both, providing one or more interfaces for communication between computer system 400 and one or more I / O devices. Electronic device 400 may include one or more of these I / O devices, where appropriate. One or more of these I / O devices may enable communication between a person and computer system 400. As an example and not by way of limitation, an I / O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I / O device or a combination of two or more of these. An I / O device may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interfaces 408 for them. Where appropriate, I / O interface 408 may include one or more device or software drivers enabling processor 402 to drive one or more of these I / O devices. I / O interface 408 may include one or more I / O interfaces 408, where appropriate. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0070] In an embodiment, communication interface 410 includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between electronic device 400 and one or more other electronic devices 400 or one or more networks. As an example and not by way of limitation, communication interface 410 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface 410 for it. As an example and not by way of limitation, electronic device 400 may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, electronic device 400 may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Electronic device 400 may include any suitable communication interface 410 for any of these networks, where appropriate. Communication interface 410 may include one or more communication interfaces 410, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

[0071] In an embodiment, bus 412 includes hardware, software, or both coupling components of computer system 400 to each other. As an example and not by way of limitation, bus 412 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus 412 may include one or more buses 412, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0072] Herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

[0073] Herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated otherwise by context. Moreover, "and" is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated otherwise by context.

[0074] This disclosure contemplates a system that includes one or more non-transitory computer readable storage media storing instructions; and one or more processors coupled to the one or more non-transitory computer readable storage media and operable to execute the instructions to perform certain functions includes embodiments in which those functions are performed by a single processor, embodiments in which those functions are performed by multiple processors that each perform all the functions, and embodiments in which those functions are performed by multiple processors (e.g., in separate computing devices) where each processor performs at least one function but less than all recited functions.

[0075] The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend.

[0076] According to an embodiment of the disclosure, a method is disclosed. The method may include predicting, by a decoder of an LLM and in response to an input sequence provided to an encoder of the LLM,stokens of an output sequence. The method may include accessing, for each of one or more attention layers of the LLM, a set of attention logits specific to that attention layer and used by the LLM to predict thenmost recent tokens of thestokens. The method may include determining, for each of the one or more attention layers and by a trained mask generation model, a layer-specific attention mask for the set of attention logits specific to that attention layer, based on the accessed set of attention logits. The method may include predicting, by the decoder of the LLM, the nextmtokens of the output sequence using, for each of the one or more attention layers, the set of attention logits as masked by the layer-specific attention mask for that layer.

[0077] According to an embodiment of the disclosure, the method may include determining the layer-specific attention mask for each of the one or more attention layers based on thestokens of the output sequence.

[0078] According to an embodiment of the disclosure, the method may include, after predicting themtokens, repeating the accessing, determining, and predicting steps for one or more additional iterations.

[0079] According to an embodiment of the disclosure, the method may include, for at least one of the additional iterations: determining, for each of the one or more attention layers, a difference between (1) the set of attention logits as masked by the layer-specific attention mask for that layer and (2) thestokens of the output sequence for that iteration. According to an embodiment of the disclosure, the method may include, when the difference is less than a threshold, then updating the attention logits for that layer using the layer-specific attention mask. According to an embodiment of the disclosure, the method may include, when the difference is greater than a threshold, then reverting the attention logits for that layer to that layer's attention logits from a previous iteration.

[0080] According to an embodiment of the disclosure, the method may include selecting the topkset of attention logits in each of the one or more layers to predict the nextmtokens of the output sequence.

[0081] According to an embodiment of the disclosure, the method may be performed by a client device that stores the LLM.

[0082] According to an embodiment of the disclosure, the method may include determining the layer-specific attention mask and predicting the nextmtokens using a CALC-LLM algorithm.

[0083] According to an embodiment of the disclosure, the set of attention logits may be accessed from a KV cache for the respective attention layer. The method may include compressing the KV cache for the respective attention layer based on the set of attention logits as masked by the layer-specific attention mask for that respective attention layer.

[0084] According to an embodiment of the disclosure, the trained mask generation model may include a vision transformer encoder and decoder.

[0085] According to an embodiment of the disclosure, a system is disclosed. The system may include one or more non-transitory computer readable storage media storing instructions; and one or more processors coupled to the one or more non-transitory computer readable storage media and operable to execute the instructions to predict, by a decoder of an LLM and in response to an input sequence provided to an encoder of the LLM,stokens of an output sequence. According to an embodiment of the disclosure, the one or more processors may be operable to execute the instructions to access, for each of one or more attention layers of the LLM, a set of attention logits specific to that attention layer and used by the LLM to predict thenmost recent tokens of thestokens. According to an embodiment of the disclosure, the one or more processors may be operable to execute the instructions to determine, for each of the one or more attention layers and by a trained mask generation model, a layer-specific attention mask for the set of attention logits specific to that attention layer, based on the accessed set of attention logits. According to an embodiment of the disclosure, the one or more processors may be operable to execute the instructions to predict, by the decoder of the LLM, the nextmtokens of the output sequence using, for each of the one or more attention layers, the set of attention logits as masked by the layer-specific attention mask for that layer.

[0086] According to an embodiment of the disclosure, the one or more processors may be operable to execute the instructions to determine the layer-specific attention mask for each of the one or more attention layers based on thestokens of the output sequence.

[0087] According to an embodiment of the disclosure, the one or more processors may be operable to execute the instructions to, after predicting themtokens, repeat the accessing, determining, and predicting steps for one or more additional iterations.

[0088] According to an embodiment of the disclosure, the one or more processors may be operable to execute the instructions to, for at least one of the additional iterations, determine, for each of the one or more attention layers, a difference between (1) the set of attention logits as masked by the layer-specific attention mask for that layer and (2) thestokens of the output sequence for that iteration. According to an embodiment of the disclosure, the one or more processors may be operable to execute the instructions to, when the difference is less than a threshold, then update the attention logits for that layer using the layer-specific attention mask. According to an embodiment of the disclosure, the one or more processors may be operable to execute the instructions to, when the difference is greater than a threshold, then revert the attention logits for that layer to that layer's attention logits from a previous iteration.

[0089] According to an embodiment of the disclosure, the one or more processors may be operable to execute the instructions to select the topkset of attention logits in each of the one or more layers to predict the nextmtokens of the output sequence. According to an embodiment of the disclosure, the one or more processors and the computer readable storage media are part of a client device that stores the LLM.

[0090] According to an embodiment of the disclosure, the one or more processors may be operable to execute the instructions to determine the layer-specific attention mask and predict the nextmtokens by executing a CALC-LLM algorithm.

[0091] According to an embodiment of the disclosure, one or more non-transitory computer readable storage media storing instructions are disclosed. According to an embodiment of the disclosure, one or more non-transitory computer readable storage media may be operable when executed by one or more processors to predict, by a decoder of an LLM and in response to an input sequence provided to an encoder of the LLM,stokens of an output sequence. According to an embodiment of the disclosure, one or more non-transitory computer readable storage media may be operable when executed by one or more processors to access, for each of one or more attention layers of the LLM, a set of attention logits specific to that attention layer and used by the LLM to predict thenmost recent tokens of thestokens. According to an embodiment of the disclosure, one or more non-transitory computer readable storage media may be operable when executed by one or more processors to determine, for each of the one or more attention layers and by a trained mask generation model, a layer-specific attention mask for the set of attention logits specific to that attention layer, based on the accessed set of attention logits. According to an embodiment of the disclosure, one or more non-transitory computer readable storage media may be operable when executed by one or more processors to predict, by the decoder of the LLM, the nextmtokens of the output sequence using, for each of the one or more attention layers, the set of attention logits as masked by the layer-specific attention mask for that layer.

[0092] According to an embodiment of the disclosure, one or more non-transitory computer readable storage media may include instructions that are operable when executed by one or more processors to, after predicting themtokens, repeat the accessing, determining, and predicting steps for one or more additional iterations. According to an embodiment of the disclosure, one or more non-transitory computer readable storage media may be part of a client device that stores the LLM. According to an embodiment of the disclosure, one or more non-transitory computer readable storage media may include instructions that are operable when executed by one or more processors to, determine the layer-specific attention mask and predict the nextmtokens using a CALC-LLM algorithm.

[0093] According to an embodiment of the disclosure, a method is disclosed. The method may include predicting, by a decoder of a Large Language Model (LLM), in response to an input sequence provided to an encoder of the LLM,stokens of an output sequence. The method may include accessing, for each of one or more attention layers of the LLM, a first set of attention logits corresponding to each of the one or more attention layers, wherein the first set of attention logits are used by the LLM to predictnmost recent tokens of the s tokens. The method may include determining, by a trained mask generation model, a layer-specific attention mask for the first set of attention logits corresponding to each of the one or more attention layers, based on the first set of attention logits. The method may include predicting, by the decoder of the LLM, nextmtokens of the output sequence using, for each of the one or more attention layers, the first set of attention logits as masked by the layer-specific attention mask corresponding to each of the one or more attention layers.

[0094] According to an embodiment of the disclosure, the method may include determining the layer-specific attention mask corresponding to each of the one or more attention layers based on thestokens of the output sequence. According to an embodiment of the disclosure, the method may include after predicting the next m tokens, for one or more additional iterations, accessing, for each of the one or more attention layers of the LLM, a second set of attention logits corresponding to each of the one or more attention layers, wherein the second set of attention logits are used by the LLM to predict thenmost recent tokens of themtokens. According to an embodiment of the disclosure, the method may include determining, by the trained mask generation model, the layer-specific attention mask for the second set of attention logits corresponding to each of the one or more attention layers, based on the second set of attention logits. According to an embodiment of the disclosure, the method may include predicting, by the decoder of the LLM, nextptokens of the output sequence using, for each of the one or more attention layers, the second set of attention logits as masked by the layer-specific attention mask corresponding to each of the one or more attention layers.

[0095] According to an embodiment of the disclosure, the method may include, for at least one of the one or more additional iterations: determining, for each of the one or more attention layers, a difference between (1) the set of attention logits as masked by the layer-specific attention mask for each of the one or more attention layers and (2) thestokens of the output sequence for at least one of the one or more additional iterations. According to an embodiment of the disclosure, the method may include, based on determining that the difference is less than a threshold, updating the attention logits for the first attention layer using the layer-specific attention mask. According to an embodiment of the disclosure, the method may include, based on determining that the difference is greater than the threshold, reverting the attention logits for the first attention layer to the first attention layer's attention logits from a previous iteration.

[0096] According to an embodiment of the disclosure, the method may include selecting the topkset of attention logits in each of the one or more attention layers to predict the nextmtokens of the output sequence. According to an embodiment of the disclosure, the method may include determining the layer-specific attention mask and predicting the nextmtokens using a CALC-LLM algorithm. According to an embodiment of the disclosure, the set of attention logits may be accessed from a KV cache for respective attention layer of the one or more attention layers. According to an embodiment of the disclosure, the method may include compressing the KV cache for the respective attention layer based on the set of attention logits as masked by the layer-specific attention mask for that respective attention layer.

[0097] According to an embodiment of the disclosure, the trained mask generation model may include a vision transformer encoder and decoder.

[0098] According to an embodiment of the disclosure, an electronic device is disclosed. The electronic device may include at least one memory storing one or more instructions. The electronic device may include one or more processors including processing circuitry, wherein the one or more instructions executed by the one or more processors individually or collectively, to cause the electronic device to: predict, by a decoder of a Large Language Model (LLM), in response to an input sequence provided to an encoder of the LLM, s tokens of an output sequence. According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to access, for each of one or more attention layers of the LLM, a first set of attention logits corresponding to each of the one or more attention layers, wherein the first set of attention logits are used by the LLM to predict n most recent tokens of the s tokens. According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to determine, by a trained mask generation model, a layer-specific attention mask for the first set of attention logits corresponding to each of the one or more attention layers, based on the first set of attention logits. According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to predict, by the decoder of the LLM, next m tokens of the output sequence using, for each of the one or more attention layers, the first set of attention logits as masked by the layer-specific attention mask corresponding to each of the one or more attention layers.

[0099] According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to determine the layer-specific attention mask corresponding to each of the one or more attention layers based on thestokens of the output sequence. According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to, after predicting the nextmtokens, for one or more additional iterations access, for each of the one or more attention layers of the LLM, a second set of attention logits corresponding to each of the one or more attention layers, wherein the second set of attention logits are used by the LLM to predict the n most recent tokens of the m tokens. According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to determine, by the trained mask generation model, the layer-specific attention mask for the second set of attention logits corresponding to each of the one or more attention layers, based on the second set of attention logits. According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to predict, by the decoder of the LLM, next p tokens of the output sequence using, for each of the one or more attention layers, the second set of attention logits as masked by the layer-specific attention mask corresponding to each of the one or more attention layers.

[0100] According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to: for at least one of the additional iterations, determine, for each of the one or more attention layers, a difference between (1) the set of attention logits as masked by the layer-specific attention mask for each of the one or more attention layers and (2) the s tokens of the output sequence for at least one of the one or more additional iterations. According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to, based on determining that the difference is less than a threshold, update the attention logits for the first attention layer using the layer-specific attention mask. According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to, based on determining that the difference is greater than the threshold, revert the attention logits for the first attention layer to the first attention layer's attention logits from a previous iteration.

[0101] According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to select the topkset of attention logits in each of the one or more attention layers to predict the nextmtokens of the output sequence.

[0102] According to an embodiment of the disclosure, the one or more instructions may be executed by the one or more processors individually or collectively, to cause the electronic device to determine the layer-specific attention mask and predict the nextmtokens by executing a CALC-LLM algorithm.

[0103] According to an embodiment of the disclosure, a computer-readable medium containing instructions, wherein the instructions, when executed by at least one processor, cause the electronic device to perform the method herein is disclosed.

Claims

1.A method (100) comprising:predicting (110), by a decoder of a Large Language Model (LLM), in response to an input sequence provided to an encoder of the LLM,stokens of an output sequence;accessing (120), for each of one or more attention layers of the LLM, a first set of attention logits corresponding to each of the one or more attention layers, wherein the first set of attention logits are used by the LLM to predictnmost recent tokens of the s tokens;determining (130), by a trained mask generation model, a layer-specific attention mask for the first set of attention logits corresponding to each of the one or more attention layers, based on the first set of attention logits; andpredicting (140), by the decoder of the LLM, nextmtokens of the output sequence using, for each of the one or more attention layers, the first set of attention logits as masked by the layer-specific attention mask corresponding to each of the one or more attention layers.2.The method of Claim 1, further comprising determining the layer-specific attention mask corresponding to each of the one or more attention layers based on thestokens of the output sequence.3.The method of any one of claims 1 to 2, further comprising:after predicting the next m tokens, for one or more additional iterations:accessing, for each of the one or more attention layers of the LLM, a second set of attention logits corresponding to each of the one or more attention layers, wherein the second set of attention logits are used by the LLM to predict thenmost recent tokens of themtokens;determining, by the trained mask generation model, the layer-specific attention mask for the second set of attention logits corresponding to each of the one or more attention layers, based on the second set of attention logits; andpredicting, by the decoder of the LLM, nextptokens of the output sequence using, for each of the one or more attention layers, the second set of attention logits as masked by the layer-specific attention mask corresponding to each of the one or more attention layers.4.The method of Claim 3, further comprising, for at least one of the one or more additional iterations:determining, for each of the one or more attention layers, a difference between (1) the set of attention logits as masked by the layer-specific attention mask for each of the one or more attention layers and (2) the s tokens of the output sequence for at least one of the one or more additional iterations;based on determining that the difference is less than a threshold, updating the attention logits for the first attention layer using the layer-specific attention mask; andbased on determining that the difference is greater than the threshold, reverting the attention logits for the first attention layer to the first attention layer's attention logits from a previous iteration.5.The method of any one of claims 1 to 4, further comprising selecting the topkset of attention logits in each of the one or more attention layers to predict the nextmtokens of the output sequence.6.The method of any one of claims 1 to 5, further comprising determining the layer-specific attention mask and predicting the nextmtokens using a CALC-LLM algorithm.7.The method of any one of claims 1 to 6, wherein the set of attention logits are accessed from a KV cache for respective attention layer of the one or more attention layers; andfurther comprising compressing the KV cache for the respective attention layer based on the set of attention logits as masked by the layer-specific attention mask for that respective attention layer.8.The method of any one of claims 1 to 7, wherein the trained mask generation model comprises a vision transformer encoder and decoder.9.An electronic device (400) comprising:at least one memory (404) storing one or more instructions; andone or more processors (402) including processing circuitry, wherein the one or more instructions executed by the one or more processors individually or collectively, to cause the electronic device (400) to:predict, by a decoder of a Large Language Model (LLM), in response to an input sequence provided to an encoder of the LLM, s tokens of an output sequence;access, for each of one or more attention layers of the LLM, a first set of attention logits corresponding to each of the one or more attention layers, wherein the first set of attention logits are used by the LLM to predict n most recent tokens of the s tokens;determine, by a trained mask generation model, a layer-specific attention mask for the first set of attention logits corresponding to each of the one or more attention layers, based on the first set of attention logits; andpredict, by the decoder of the LLM, next m tokens of the output sequence using, for each of the one or more attention layers, the first set of attention logits as masked by the layer-specific attention mask corresponding to each of the one or more attention layers.10.The electronic device (400) of Claim 9, wherein the one or more instructions executed by the one or more processors individually or collectively, to cause the electronic device (400) to: determine the layer-specific attention mask corresponding to each of the one or more attention layers based on thestokens of the output sequence.11.The electronic device (400) of any one of claims 9 to 10, wherein the one or more instructions executed by the one or more processors individually or collectively, to cause the electronic device (400) to: after predicting the nextmtokens, for one or more additional iterations:access, for each of the one or more attention layers of the LLM, a second set of attention logits corresponding to each of the one or more attention layers, wherein the second set of attention logits are used by the LLM to predict the n most recent tokens of the m tokens;determine, by the trained mask generation model, the layer-specific attention mask for the second set of attention logits corresponding to each of the one or more attention layers, based on the second set of attention logits; andpredict, by the decoder of the LLM, next p tokens of the output sequence using, for each of the one or more attention layers, the second set of attention logits as masked by the layer-specific attention mask corresponding to each of the one or more attention layers.12.The electronic device (400) of any one of claims 9 to 11, wherein the one or more instructions executed by the one or more processors individually or collectively, to cause the electronic device (400) to, for at least one of the additional iterations:determine, for each of the one or more attention layers, a difference between (1) the set of attention logits as masked by the layer-specific attention mask for each of the one or more attention layers and (2) the s tokens of the output sequence for at least one of the one or more additional iterations;based on determining that the difference is less than a threshold, update the attention logits for the first attention layer using the layer-specific attention mask; andbased on determining that the difference is greater than the threshold, revert the attention logits for the first attention layer to the first attention layer's attention logits from a previous iteration.13.The electronic device (400) of any one of claims 9 to 12, wherein the one or more instructions executed by the one or more processors individually or collectively, to cause the electronic device (400) to: select the topkset of attention logits in each of the one or more attention layers to predict the nextmtokens of the output sequence.14.The electronic device (400) of any one of claims 9 to 13, wherein the one or more instructions executed by the one or more processors individually or collectively, to cause the electronic device (400) to: determine the layer-specific attention mask and predict the nextmtokens by executing a CALC-LLM algorithm.15.A computer-readable medium containing instructions, wherein the instructions, when executed by at least one processor, cause the electronic device (400) to perform the method of any one of claims 1 to 8.

Images