Model inference optimization method and electronic device

Abstract

This application discloses a model inference optimization method and electronic device, relating to the field of model inference technology. The method includes: in the model pre-filling stage, determining a first set based on an attention probability matrix calculated from the input word sequence, and determining a second set based on the position index of each word in the input word sequence and a preset window, with caching differentiated according to the set type to which the position index belongs; in the model decoding stage, iteratively executing the loading method of historical key-value pairs based on whether the historical position index belongs to the set type to generate the current word and its new key-value pair, and updating the second set based on the current decoding round; when the termination condition is met, outputting the words generated in the current model decoding stage as the inference result sequence. This solves the problem in related technologies where it is difficult to balance inference quality and efficiency under limited resources, and achieves differentiated adaptation of key-value caching and features to balance inference quality and efficiency under limited resources.

Description

Technical Field

[0001] This application relates to the field of model reasoning technology, and in particular to a model reasoning optimization method and electronic device. Background Technology

[0002] In recent years, with the widespread application of models in tasks such as long text understanding, multi-turn dialogue, and chained reasoning, the computational efficiency and resource consumption issues in the inference stage have become increasingly prominent. During autoregressive generation, the model needs to cache the key and value matrices corresponding to the generated terms to form a key-value cache. However, with the increase in context length and the number of concurrent requests, the storage capacity and access bandwidth of the key-value cache quickly become the main bottlenecks restricting inference performance. Related technologies mainly employ two optimization paths: sparsification and quantization. Sparsification methods attempt to reduce cache size and computational cost by eliminating or skipping low-contribution terms, but the attention importance judgments they rely on often dynamically drift with the inference process; eliminated content may become key evidence again in subsequent steps, leading to decreased inference stability. Quantization methods compress the key-value cache into a low-bit representation to reduce storage and bandwidth requirements, but typically employ a uniform or weakly differentiated compression strategy for all terms. Under low-bit compression, errors are highly concentrated in a small number of high-weight positions, dominating end-to-end quality degradation. Summary of the Invention

[0003] This application provides a model inference optimization method and an electronic device to solve the technical problem in related technologies that model inference methods are difficult to balance inference quality and inference efficiency under limited resource budgets.

[0004] This application provides a model inference optimization method, which includes:

[0005] In response to being in the model pre-filling stage, the attention probability matrix is ​​determined based on the input word sequence corresponding to the inference request, and the column-direction aggregation calculation is performed on the attention probability matrix to determine the first set;

[0006] Based on the position index of each word in the input word sequence and the preset window, a second set is determined. The first set and the second set are composed of position indices; wherein, the position index is used to identify the storage location of the key-value pair corresponding to each word.

[0007] For the key-value pairs corresponding to the input word sequence, the attention calculation method and storage precision are determined according to whether the corresponding position index belongs to the first set or the second set. Based on the attention calculation method and storage precision of the key-value pairs, a mixed precision cache is established.

[0008] In response to being in the model decoding phase, iterative execution continues:

[0009] Based on whether the position index corresponding to the historical key-value pair belongs to the first set or the second set, determine the loading method for reading historical key-value pairs from the mixed precision cache. Based on the query vector of the current decoding round and the historical key-value pairs obtained based on the loading method, generate the current word element and the new key-value pair corresponding to the current word element.

[0010] Write the new key-value pair to the mixed-precision cache;

[0011] Update the second set based on the position index of the generated lexical unit in the current decoding round;

[0012] In response to the termination condition being met, the tokens generated in the current model decoding phase are output as the inference result sequence.

[0013] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the virtual machine migration method described in the following embodiments.

[0014] In response to being in the model pre-filling stage, the attention probability matrix is ​​determined based on the input word sequence corresponding to the inference request, and the column-direction aggregation calculation is performed on the attention probability matrix to determine the first set;

[0015] Based on the position index of each word in the input word sequence and the preset window, a second set is determined. The first set and the second set are composed of position indices.

[0016] For the key-value pairs corresponding to the input word sequence, the attention calculation method and storage precision are determined according to whether the corresponding position index belongs to the first set or the second set. Based on the attention calculation method and storage precision of the key-value pairs, a mixed precision cache is established.

[0017] In response to being in the model decoding phase, iterative execution continues:

[0018] Based on whether the position index corresponding to the historical key-value pair belongs to the first set or the second set, determine the loading method for reading historical key-value pairs from the mixed precision cache. Based on the query vector of the current decoding round and the historical key-value pairs obtained based on the loading method, generate the current word element and the new key-value pair corresponding to the current word element.

[0019] Write the new key-value pair to the mixed-precision cache;

[0020] Update the second set based on the position index of the generated lexical unit in the current decoding round;

[0021] In response to the termination condition being met, the tokens generated in the current model decoding phase are output as the inference result sequence.

[0022] The model inference optimization method provided in this application determines the attention probability matrix based on the input lexical sequence corresponding to the inference request during the model pre-filling stage, performs column-direction aggregation calculations to determine a first set, and determines a second set composed of position indices based on the position indices of each lexical in the input lexical sequence and a preset window. For key-value pairs corresponding to the input lexical sequence, the storage precision is determined based on whether their corresponding position indices belong to the first or second set, and a mixed-precision cache is established accordingly. This assigns differentiated storage precision to key-value pairs belonging to different sets, laying the data structure foundation for storage optimization and access efficiency improvement in the subsequent decoding stage. In response to iterative execution during the model decoding stage, the method determines the starting point based on whether the position indices of historical key-value pairs belong to the first set or the updated second set. The system loads historical key-value pairs from the mixed-precision cache and generates the current lexical and its corresponding new key-value pair based on the query vector of the current decoding round and the historical key-value pairs obtained using this loading method. This enables differentiated control over the historical cache access path. By writing the new key-value pairs into the mixed-precision cache, the system dynamically maintains the decoding generation and cache state, providing a complete historical context for subsequent generation steps. The system updates the second set based on the position index of the generated lexical in the current decoding round, ensuring continuous high-precision protection of the latest context. Finally, by responding to the termination condition of the inference result sequence and outputting the inference result sequence, the system completes end-to-end high-quality, low-latency long-context inference service. Without increasing the additional GPU memory budget, the system achieves coordinated optimization of inference quality, storage efficiency, and decoding speed in long-context inference of the model. Attached Figure Description

[0023] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic flowchart of a model inference optimization method provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of an attention heatmap;

[0026] Figure 3 This is a schematic diagram illustrating the distribution of vertical stripe features and column importance scores in an attention heatmap.

[0027] Figure 4 This is a diagram illustrating the relationship between the measurement index and sequence length.

[0028] Figure 5A flowchart illustrating a model inference optimization method provided in another embodiment of this application;

[0029] Figure 6 A flowchart illustrating a model inference optimization method provided in yet another embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the structure of a model inference optimization device provided in an embodiment of this application;

[0031] Figure 8 This is an internal structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0033] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0034] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] like Figure 1 As shown, one embodiment of this application provides a model inference optimization method, which specifically includes the following steps:

[0036] S1: In response to being in the model pre-filling stage, determine the attention probability matrix based on the input word sequence corresponding to the inference request, perform column-wise aggregation calculation on the attention probability matrix, and determine the first set.

[0037] Specifically, multi-head attention computation is performed on the input word sequence: This includes performing a dot product operation on the query vector and key vector corresponding to each attention head, dividing the dot product result by a scaling factor and inputting it into a normalized exponential function to obtain the single-head attention probability matrix corresponding to each attention head, and then fusing the single-head attention probability matrices corresponding to each attention head to form an attention probability matrix.

[0038] In the model pre-filling stage, this application first performs standard Transformer multi-head attention computation on the input word sequence. Each layer of the Transformer model contains multiple parallel attention heads. For a certain attention head in a certain layer, given the input sequence of that layer, where each word has a corresponding hidden state vector in that layer, and the length of the input sequence is a certain value.

[0039] This attention focus first uses three different projection matrices to linearly transform the hidden state of each word into a query vector, a key vector, and a value vector, respectively. The query vector, denoted as Q, is obtained by multiplying the hidden state of the current word by the query projection matrix and is used to calculate the similarity with all historical key vectors. The key vector, denoted as K, is obtained by multiplying the hidden states of historical words by the key projection matrix and is used to calculate the similarity with the current query vector. The value vector is used for subsequent weighted summation.

[0040] Subsequently, the dot product of each query vector and all key vectors is calculated to obtain the original attention score matrix. The dot product operation is the inner product of the query vector and the key vectors, and the result represents the original association strength between the current query and the historical keys. Each element in this matrix is ​​divided by a scaling factor, usually the square root of the key vector dimension, to maintain gradient stability. The scaled original attention score matrix is ​​then normalized row by row using the Softmax function, so that the sum of the attention weights in each row is 1, thus obtaining the single-head attention probability matrix corresponding to that attention head.

[0041] The single-head attention probability matrix is ​​a two-dimensional matrix. Its row indices correspond to the query positions, i.e., the currently used query terms; the column indices correspond to the key positions, i.e., the historical terms that are the focus of attention. Each element in the matrix represents the attention weight assigned by the model to a key term when generating a representation of a query term. The single-head attention probability matrix obtained from the above calculation process is essentially a probability distribution matrix formed by normalizing the similarity between the query and the key.

[0042] For each row of the matrix, all elements in that row form a probability distribution describing the degree of attention the corresponding query term receives from all historical terms in the entire sequence, and the sum of all elements in that row is 1. For each column of the matrix, all elements in that column describe the degree of attention received by the corresponding key term from all query terms in the sequence when that key term is the object of attention.

[0043] The above process is performed independently in each attention head, resulting in multiple single-head attention probability matrices. Since each attention head in the multi-head attention mechanism captures sequence dependencies from different representation subspaces, this application fuses these single-head attention probability matrices. Specifically, the fusion method can be element-wise averaging, weighted summation, or concatenating the output vectors of each head and then mapping them through an output projection layer to ultimately form a unified attention probability matrix.

[0044] After obtaining the attention probability matrix, this application breaks through the limitation of traditional techniques that use the matrix only for the current layer output calculation, and innovatively performs aggregate statistics on its column dimensions to quantify the overall attention level of each key position in the entire prefix sequence.

[0045] Specifically, the attention weights contained in each column of the attention probability matrix are summed, and the sum is used as the column importance score of the position index corresponding to each column. The product of the preset key position ratio parameter and the total number of position indices of the current attention head in the current layer is used as the selection quantity. From the position indices, a number of position indices are selected in descending order of column importance score to form the first set.

[0046] For each column in the attention probability matrix (corresponding to a fixed position index k), the attention weights of all query positions contained in that column for that key are summed to obtain the total attention received by each key position k across the entire sequence. This summation is the column importance score of the position index corresponding to each column. The total attention quality attracted by the key vector at key position k from all query positions reflects the semantic importance of that key position in the entire prefix sequence.

[0047] Based on the aforementioned column importance scores, this application constructs the first set using a Top-K selection mechanism. Specifically, this involves: First, determining the total number of position indices of the current attention head in the current layer, i.e., the length of the input lexical sequence, a known quantity determined during the pre-filling stage. Second, introducing a preset key position ratio parameter ρ (typically 1%-5%). This parameter is one of the core control knobs provided in this application, used to balance protection strength and compression benefits. A larger ρ value results in more positions in the first set, a higher proportion of keywords protected by full precision, and lower quantization error risk, but smaller caching compression benefits; conversely, a smaller ρ value results in greater compression benefits, but requires more precise identification of a few truly critical anchor positions. Finally, calculating the selection quantity M = ρ * |K|, where |K| is a set of k. From all position indices, the top M position indices are selected in descending order of column importance scores to form the first set.

[0048] The first set represents the global anchor points that consistently receive high attention weights during long-context reasoning. This application determines the first set by performing column-wise aggregation on the attention probability matrix. This requires only one column-wise reduction and one sorting operation on the existing attention probability matrix, achieving accurate, lightweight, and configurable identification of global key anchor points during long-context reasoning. This method not only has low computational complexity but also does not depend on a specific model architecture, requiring no retraining or fine-tuning. It can be directly applied to any deployed Transformer-type large language model, and by adjusting the ρ parameter, it can flexibly adapt to different model sizes, context lengths, and business requirements.

[0049] S2: Based on the position index of each word in the input word sequence and the preset window, determine the second set. The first set and the second set are composed of position indices; wherein, the position index is used to identify the storage location of the key-value pair corresponding to each word.

[0050] The position index of each word in the input word sequence is used as the current query position; the starting point of the interval is determined by the current query position, the preset window length, and the sequence start boundary; the current query position is used as the end point of the interval, and a continuous position index interval is determined based on the start point and the end point of the interval; the number of position indices contained in the position index interval does not exceed the preset window length, and the position indices are non-negative integers; the position indices contained in the position index interval are used as the second set corresponding to the word at the current query position.

[0051] The input lexical sequence refers to the sequence of lexical terms corresponding to the complete input text received during the pre-filling stage, including user questions, system instructions, search documents, and multi-turn dialogue history. Its length provides a baseline for the starting position index of subsequent decoding rounds. The position index refers to the absolute sequence number of a lexical term in the input or output sequence, starting from 0 and increasing sequentially. The position index range of the input lexical sequence is [0, seq_len-1], and the position indices of newly generated lexical terms in the decoding stage increase sequentially starting from seq_len.

[0052] For each word in the input word sequence, its position index in the sequence is used as the current query position. The starting point of the interval is determined by the current query position, the preset window length, and the sequence's initial boundary; the ending point of the interval is the current query position, and a continuous interval of position indices is determined based on the starting and ending points. The number of position indices within each interval does not exceed the preset window length, and all position indices are non-negative integers. The position indices contained within this interval are used as the second set corresponding to that word at the current query position.

[0053] The preset window length, denoted by W, is a fixed hyperparameter that controls the size of the interval in the second set. W is a positive integer, typically 128 or 256, indicating that the model provides high-precision protection for local dependencies on the nearest W terms. The sequence start boundary can be the starting position index 0 of the sequence, used to constrain the left endpoint of the interval to not be less than 0, avoiding illegal memory accesses such as negative indexes. The interval start point refers to the left endpoint of the position index interval, and this value is determined by the current query position t, the preset window length W, and the sequence start boundary, i.e., max(0, t-W+1). The interval end point refers to the right endpoint of the position index interval, i.e., the current query position t itself.

[0054] After determining the start and end points of the interval, the closed interval [t-W+1, t], consisting of all integer indices from the start to the end, is the actual effective range of the interval after boundary constraints. This is the second set corresponding to the current query position. For example, the second set is defined as the logical index range of the local attention window corresponding to the current query position t, i.e., the continuous closed interval [max(0, t-W+1), t]. This set dynamically slides as the query position t increases, ensuring the model's local dependency on the nearest context.

[0055] Ideally, the window should be W consecutive positions from t - W + 1 to t. However, when t < W - 1 (i.e., at the beginning stage of the sequence), t - W + 1 is negative, and in this case, the start point of the interval is truncated to 0. Therefore, the complete expression for the start point of the interval is max(0, t - W + 1). In this way, it is ensured that the window length never exceeds the preset window W; all position indices are non-negative integers, meeting the basic requirements of memory addressing, and the head token of the sequence obtains the maximum possible local protection when the window is not full. It can be understood that the position indices actually participating in the calculation are the intersection of this logical window and the valid physical positions in the current KV cache. However, when there is no explicit eviction or sparsification strategy, the default second set is this continuous closed interval.

[0056] Please refer to Figure 2 , during the long-context reasoning process of large language models, attention heatmaps generally exhibit two stable structural features. The first type of structural feature is vertical stripes, and the second type of structural feature is diagonal bands.

[0057] In the attention heatmap, the horizontal axis represents the positions of historical tokens (KV Cache indices), the vertical axis represents Q at different time steps, and the color bar on the right represents the magnitude of the attention score, that is, the degree to which each historical token is attended to by the current query token.

[0058] When the KV Cache index positions in a certain column are strongly attended to at multiple time steps, a vertical bright line (vertical stripe) will be formed. The vertical stripe represents that a specific historical token position (K vector index) continuously obtains a high attention score in multiple generation steps (multiple Qs).

[0059] That is, when a certain column (fixed abscissa) in the attention heatmap is always highlighted, it means that starting from a certain time point, all subsequent generated tokens are attending to this historical position. This historical token contains information crucial for subsequent generation, that is, there is an attention structure dominated by "vertical stripes" in real reasoning. In the long-chain of thought (CoT) reasoning trajectory, it is often possible to observe that the attention heatmap exhibits an obvious "vertical lines" structure, that is, there are very few fixed positions in the position dimension of Key / Value that continuously attract significant attention quality from a large number of Queries over a relatively long span. These positions usually correspond to the "anchor information" repeatedly cited or relied on during the reasoning process, such as key entities, constraints, intermediate conclusions, and evidence fragments read multiple times, and will be continuously accessed in subsequent multiple reasoning steps.

[0060] An attention heatmap is essentially a two-dimensional matrix. The horizontal axis (X) represents the position index (k) of the Key, and the vertical axis (Y) represents the position index (t) of the Query. The color intensity of each pixel (t, k) in the graph represents the attention probability value for the k-th historical word (as the Key) when the model generates the t-th word (as the Query). It is a number between 0 and 1, and the sum of all attention probability values ​​(for a fixed t, iterating through all k) is 1. Here, t is the query position or position index, representing the position of the word currently being generated / processed.

[0061] Please see Figure 3 As shown in the heatmap, the direct feature is the percentage score of most vertical lines. Based on the vertical line features, multiple tokens with the highest cumulative attention scores are selected. The vertical stripe score S(k) is the sum or integral of the heats of all elements along a given vertical line. For a given key position k, the attention probabilities of all considered query steps (set T) are summed, expressed as: This is equivalent to reading the heat value of each pixel from top to bottom along the vertical line k=c (a fixed value) on the heatmap, and then adding them all together. The sum obtained is S(k).

[0062] For example, assuming that in the current attention probability matrix, T includes three query steps t1, t2, and t3, and K includes four key positions k1, k2, k3, and k4, the attention matrix can be specifically shown below:

[0063] k1 k2 k3 k4;

[0064] t1 [ 0.1 0.8 0.05 0.05 ] ->Query t1 for the attention probability distribution of the four keys;

[0065] t2 [0.7 0.1 0.1 0.1] ->Query t2;

[0066] t3 [0.6 0.2 0.1 0.1] ->Query t3;

[0067] Calculate S(k):

[0068] S(k1)=a_{t1,k1}+a_{t2,k1}+a_{t3,k1}=0.1+0.7+0.6=1.4;

[0069] S(k2) = 0.8 + 0.1 + 0.2 = 1.1;

[0070] S(k3) = 0.05 + 0.1 + 0.1 = 0.25;

[0071] S(k4) = 0.05 + 0.1 + 0.1 = 0.25;

[0072] As can be seen, k1 has the highest S(k) value of 1.4. At this point, the k1 column is very hot at t2 and t3 (0.7, 0.6), equivalent to a vertical stripe in the attention heatmap. That is, the k1 column belongs to the potential vertical stripe in the attention probability matrix. k2 has the second highest S(k) value of 1.1, but its main heat is concentrated in step t1 (0.8), and it is not high in other steps. It may be a locally important key, but it is not a stripe that runs through multiple steps. If we set ρ = 25% here, then the first high-scoring key (ρ*|K|=1) is k1. At this time, k1 is added to the first set.

[0073] It can be seen that there is a direct numerical correspondence between the vertical bars in the attention heatmap and the column summation result of the attention probability matrix: the significance of the vertical bars is essentially the cumulative attention weight of the key position corresponding to that column over all query steps.

[0074] In this application, an importance score S(k) is maintained for each Key position k, which is dynamically updated during the inference process. This score quantifies the cumulative attention weight gained by that position in the sequence direction (i.e., across multiple query steps), and is referred to as column quality. Specifically: Let a t,k ∈ [0, 1] represents the attention probability assigned to the k-th historical word (as the Key) when the model generates the t-th word (as the Query) (i.e., the output after Softmax in the standard attention mechanism). For a set T of Query steps to be considered (e.g., all currently generated steps, or steps within a sliding window), the column quality score at the Key position is defined as: ;

[0075] Here, S(k) is the summation of the attention probability matrix along the k-th column (row direction) of the Query axis. The higher the value of S(k), the more frequently and strongly the model relies on the k-th historical information in the process of generating all Query tokens in T, and the more likely that position is to form a "vertical stripe" on the attention heatmap.

[0076] Based on a pre-defined, extremely small percentage parameter ρ (e.g., ρ = 1%), the top ρ*|K| key positions with the highest column quality scores S(k) can be selected to form a vertical stripe key set KVL: KVL = TopK(S, ρ*|K|). The update process of S(k) has extremely low computational overhead. During the autoregressive generation process, whenever the model calculates a new attention row vector, this row vector corresponds to the query term in the current decoding round and contains the attention probability of this query term for all historical key terms. It is only necessary to accumulate each attention probability value in this row vector to the accumulated S(k) value of the corresponding historical key term.

[0077] Specifically, for each historical key position k, the attention probability value obtained in the current decoding round is added to the accumulated attention probability value for that position in all previous decoding rounds to obtain the updated S(k). This operation is essentially performing element-wise accumulation on a vector, involving only one vector addition operation, and the additional computational cost introduced in the entire autoregressive generation process is negligible. The above mathematical process is equivalent to vertically accumulating the attention probability of each vertical column on the attention heatmap. The column with the highest accumulated sum corresponds to the most prominent vertical stripe. This recognition is achieved through programmatic column quality accumulation, without the need to actually generate a visualization chart.

[0078] In the visualization of the attention probability matrix, a finite-width band of highlighted area exists near the main diagonal, shifting downwards and to the right in tandem with the query position. This phenomenon reveals a crucial technical fact: when generating each lexical unit, the model maintains a persistent and significant dependency on its neighboring, recently generated contextual lexical units. This dependency is fundamental to ensuring local text coherence, grammatical correctness, and short-term information aggregation. If the numerical fidelity of lexical units within the diagonal band is compromised during quantization compression, it will directly lead to quality issues in the generated text, such as incoherent sentences, confused referencing, and logical breaks.

[0079] The second set in this application is a direct and efficient engineering capture of the aforementioned diagonal band pattern. For a given current query position t, its local window set K_Win(t) is simply defined as [max(0, t-W+1), t]. Here, W (window size) is a preset fixed parameter (e.g., 128, 256). This window covers the current token and the previous W-1 most recent historical tokens, precisely covering the diagonal band region of width W in the attention heatmap, ending at the current position t. By protecting all tokens within this window to full precision, this application unconditionally guarantees the model's access fidelity to recent contexts, thereby maintaining the local coherence and fluency of the generated text.

[0080] In the attention heatmap, a strip-shaped area near the main diagonal (the diagonal band) is typically highlighted. This reflects the model's strong dependence on recent context (nearest tokens), which is fundamental to ensuring linguistic coherence and short-term information aggregation. As generation progresses, this highlighted diagonal band slides synchronously to the lower right. The attention heatmap also contains a few vertical highlighted bars that run through multiple query rows. These represent specific historical tokens (key positions) that are repeatedly and continuously referenced and focused on when generating long answers or performing multi-step reasoning. These positions are typically key entities, constraints, intermediate conclusions, or core evidence in the question, serving as "anchor points" for reasoning. Vertical bars carry long-range, multi-step semantic dependencies; the diagonal band carries short-range, local syntactic and semantic coherence. Together, they constitute the attention support for effective reasoning.

[0081] This application uses a first set to ensure global key anchors (vertical stripe function) and a second set to ensure local fluency (diagonal stripe function). The two sets have clearly defined functions, do not overlap, and together cover the main sources of attention quality. Understandably, the first and second sets may overlap in sequence position (i.e., a key anchor token may fall within the current window), but they are designed to be independent. Even if a vertical stripe key token slides out of the fixed window due to sequence growth, it will still remain in the first set due to its high S(k) score, continuing to be protected by full precision. This solves the problem that relying solely on sliding windows leads to the forgetting of important early information.

[0082] The first set is dynamically identified and updated during the inference process; the second set is statically defined with simple rules and zero overhead. This combination of dynamic global anchors and static local windows achieves comprehensive adaptation to the attention structure at extremely low cost, ensuring that the most critical information is preserved with the highest precision, whether it is complex inference relying on long-term anchors or smooth generation relying on recent context. This allows the system to maintain extremely high output quality even when aggressively quantizing most of the remaining tokens.

[0083] Existing sparsity techniques attempt to dynamically filter historical lexical units that need to be retained through complex online importance assessments, but they face two major dilemmas: first, importance determinations frequently drift with the reasoning process; and second, the assessment algorithm itself introduces non-negligible additional overhead.

[0084] This application breaks through the above paradigm and proposes a minimalist and deterministic local protection strategy: instead of trying to determine which terms within the window are important, it assumes that all terms within the window are important and applies indiscriminate full-precision protection to them. The core insight of this design is that, regardless of model size or task type, dependence on the nearest context is an inherent property of the Transformer autoregressive architecture, rather than a flexible requirement that can be optimized away by sparsity.

[0085] For details, please refer to [link / reference]. Figure 4 Using MSE as a metric, 4-bit quantization will produce a huge MSE error. By storing a small amount of 1% tokens, the MSE can be greatly reduced, achieving 8-bit performance with only half the storage resources.

[0086] Local windows and a very small number of high-scoring positions can cover most of the attention quality. Furthermore, the phenomenon of vertical stripes dominating can be quantitatively characterized: under different context lengths (seq_len), the proportion of attention quality covered by the following sets is statistically analyzed: (1) local windows of fixed size (e.g., W=128, denoted as Win); (2) a very small number of key positions with the highest global scores (e.g., Top-1%, denoted as Top-1%); (3) the union of the two (Total). When seq_len increases from 1K to 32K, the coverage ratio of Win∪Top-1% of the total attention quality still remains stable at over half (approximately 53.24%–59.60%), indicating that attention quality remains highly concentrated over a long span.

[0087] This leads to the following engineering implications: KV compression is better suited to a non-uniform strategy, that is, to protect a small number of high-attention anchor points with higher fidelity, while applying more aggressive compression to the majority of other locations to improve overall efficiency. Based on the aforementioned structural attention phenomenon and the amplification of errors at high-weight locations, a clear engineering requirement emerges: within a limited full-precision budget, prioritize the protection of a small number of persistent critical locations (including vertical anchor points and local window locations), and implement a higher proportion of low-bit compression for the majority of other locations. This will simultaneously reduce the capacity pressure of KV and the bandwidth pressure during the decoding stage, while avoiding end-to-end quality degradation caused by critical location errors.

[0088] To meet this requirement, subsequent technical solutions need to further address: lightweight identification and updating of key locations, mixed-precision attention calculation in the Prefill stage, mixed-precision KV storage and dequantization calculation in the Decode stage, and fusion implementation combining with efficient attention operators (such as block scanning and online softmax), so as to achieve deployable long context acceleration and resource optimization while ensuring inference quality.

[0089] Please see Figure 5 The method of this application is two-stage. The first stage includes saving the token generated by the residual. The second stage performs a lightweight score statistics and classifies the tokens based on the statistical scores to quantify their accuracy.

[0090] In practical inference, only a very small number of tokens (usually less than 5%) form persistent, vertically high-weighted positions that carry disproportionate attention quality. Furthermore, these positions are more likely to dominate quantization errors and end-to-end quality degradation under aggressively low-bit conditions. To address this phenomenon, this application adopts a non-uniform compression approach: full-precision protection for critical positions and low-bit compression for the remaining positions. Error-sensitive positions (vertical stripe positions and local window positions) are explicitly maintained in full-precision representation, while the majority of remaining positions are quantized in low-bit form. A fusion operator is used to coordinate the full-precision path and the quantization path within the same computational kernel, thereby alleviating capacity and bandwidth pressures and improving inference efficiency and stability without significantly increasing overhead.

[0091] S3: For key-value pairs corresponding to the input word sequence, determine the attention calculation method and storage precision based on whether their corresponding position index belongs to the first set or the second set. Based on the attention calculation method and storage precision of the key-value pairs, establish a mixed precision cache.

[0092] Obtain the position index corresponding to each word in the input word sequence; use the key-value pairs corresponding to the position indices belonging to the first or second set in first-precision floating-point format for the calculation of the current attention layer, and write the key-value pairs to the cache in first precision after the calculation is completed; convert the key-value pairs corresponding to the position indices not belonging to the first set and not belonging to the second set to second-precision fixed-point format, and perform real-time dequantization on the second-precision fixed-point format data based on the quantization parameters within the calculation kernel of the current attention layer, use the approximate floating-point value recovered after dequantization for attention calculation, and write the second-precision fixed-point format data of the key-value pairs and the quantization parameters to the cache after the attention calculation is completed, and construct a mixed-precision cache containing first-precision and second-precision key-value pairs; the quantization parameters include at least a scaling factor and a zero point, and the number of bits corresponding to the second precision is lower than the number of bits corresponding to the first precision.

[0093] The construction of a mixed-precision cache containing key-value pairs of first precision and second precision includes: storing key-value pairs cached with first precision consecutively in a first cache area and establishing a direct mapping from the position index to the first cache area; packaging and storing the quantized data and quantization parameters compressed with key-value pairs cached with second precision in a second cache area and establishing an index table from the position index to the cache address of the quantized data packet; the first cache area and the second cache area together constitute a mixed-precision cache for subsequent decoding stage reading and access.

[0094] In response to being in the model decoding phase, S4-S6 are executed iteratively.

[0095] S4: Based on whether the position index corresponding to the historical key-value pair belongs to the first set or the second set, determine the loading method for reading historical key-value pairs from the mixed precision cache. Based on the query vector of the current decoding round and the historical key-value pairs obtained based on the loading method, generate the current word and the new key-value pair corresponding to the current word.

[0096] Traverse the historical key-value pairs stored in the mixed-precision cache; obtain the position index corresponding to each historical key-value pair, and determine whether the position index of the historical key-value pair belongs to the first set or the second set; the loading method for the historical key-value pair corresponding to the position index of the historical key-value pair belonging to the first set or the second set is: directly read the complete key-value data cached in the first precision from the first cache area according to the position index of the historical key-value pair; the loading method for the historical key-value pair corresponding to the position index of the historical key-value pair not belonging to the first set and not belonging to the second set is: locate the corresponding quantization data packet from the index table of the second cache area according to the position index of the historical key-value pair, read the compressed quantization data and the corresponding quantization parameters from the quantization data packet, and perform dequantization operation in the register or shared memory of the computing core to obtain an approximate floating-point value.

[0097] The query vector of the current decoding round is multiplied by the key vector in the loaded historical key-value pairs to obtain the original attention score. The original attention score is normalized to obtain the attention weight corresponding to each historical position index. The attention weight corresponding to each historical position index is weighted and summed with the value vector to obtain the attention layer output vector of the current decoding round. Based on the attention layer output vector, the output word of the current decoding round and the new key-value pair corresponding to the word are generated.

[0098] This application establishes a dynamic loading mechanism for mixed-precision key-value pairs based on set membership relationships, aiming to significantly reduce the memory usage of non-critical locations while ensuring the calculation accuracy of key historical positions. First, it iterates through historical key-value pairs already stored in the mixed-precision cache. The mixed-precision cache is a heterogeneous precision key-value cache system composed of a first cache region and a second cache region, simultaneously storing key-value pair data with different precisions within the same cache. For each historical key-value pair, its corresponding position index is obtained, and it is determined whether the position index belongs to the first set or the updated second set. The key-value pair is a vector pair obtained by multiplying the hidden state of a word by the key projection matrix and the value projection matrix, respectively, and is used for similarity measurement and weighted summation in subsequent attention calculations.

[0099] If the position index corresponding to the historical key-value pair belongs to the first set or the updated second set, the first loading method is adopted: the complete key-value data cached in the first precision is directly read from the first cache area according to the position index. The first cache area is configured as a contiguous memory space in the mixed precision cache dedicated to storing the first precision key-value pairs. It is stored in the full precision native format and supports addressing access based on the position index through direct mapping. Direct mapping refers to establishing a one-to-one correspondence from the position index to the storage address of the first cache area, and achieving access with O(1) time complexity by adding an offset to the base address. The first precision is configured as a high-precision floating-point format, including unquantized primitive data types such as FP32, FP16, and BF16, to ensure the numerical fidelity of key-value pairs at critical positions after loading.

[0100] If the position index corresponding to the historical key-value pair does not belong to either the first set or the updated second set, the second loading method is adopted: based on the position index corresponding to the historical key-value pair, the corresponding quantization data packet is located from the index table of the second cache area, the compressed quantization data and corresponding quantization parameters are read from the quantization data packet, and the dequantization operation is performed in the register or shared memory of the computing core to obtain an approximate floating-point value for attention calculation.

[0101] The second cache area is configured as a memory space in the mixed-precision cache dedicated to storing second-precision key-value pair quantization data packets. The stored content is compressed fixed-point data and its corresponding quantization parameters. The index table is configured as a lookup table that records the storage location of each quantization data packet in the second cache area and its corresponding position index. It is used to map the position index to the storage address of the quantization data packet after receiving it. The quantization data packet is configured as a compact storage unit that encapsulates the quantization data of the second-precision key-value pair and its quantization parameters. After encapsulation, it contains the compressed key vector, value vector, and corresponding scaling factor and zero point.

[0102] The second precision is configured as a low-bit fixed-point format, including compressed data types such as INT8, INT4, and FP8, whose corresponding bit count is lower than that of the first precision. It is used to compress the storage volume of key-value pairs in non-critical positions during the storage stage. Low-bit quantization refers to the compression process of mapping high-precision floating-point data to low-bit integer space. The original numerical range is mapped to the fixed-point representation range through scaling factors and zeros. Real-time dequantization refers to the instantaneous operation within the computing core to restore the quantized second-precision fixed-point data to floating-point data. The dequantized value is an approximate representation of the original floating-point value. Quantization parameters refer to the calibration parameters required for quantization and dequantization, including at least a scaling factor (scale) for determining the step size and zeros for aligning the zeros. The computing core refers to the arithmetic logic unit in the GPU streaming multiprocessor, NPU computing unit, or CPU core. After performing the dequantization operation, it is responsible for performing operations such as matrix multiplication and vector dot product.

[0103] After dynamically loading historical key-value pairs, attention calculation and lexical generation are performed for the current decoding round: First, the query vector of the current decoding round is multiplied by the key vectors in the loaded historical key-value pairs to obtain the raw attention score. This score represents the similarity between the current query and the key vectors at each historical position. Second, the raw attention score is normalized to obtain the attention weights corresponding to each historical position index. Typically, the Softmax function is used to map the scores to a probability distribution that sums to 1.

[0104] Subsequently, the attention weights and value vectors corresponding to each historical position index are weighted and summed to obtain the attention layer output vector for the current decoding round. This output vector incorporates historical context information, representing the aggregated representation of the current query after focusing on each historical position. Finally, based on the attention layer output vector, the output tokens for the current decoding round and the corresponding new key-value pairs are generated. The output tokens are the prediction results for the current decoding round, and the new key-value pairs are obtained by multiplying the hidden state of the token by the key projection matrix and the value projection matrix, respectively. These new key-value pairs will be stored in a mixed-precision cache in subsequent decoding rounds for future query access.

[0105] In this way, historical key-value pairs belonging to the first or second set (i.e., globally significant positions or positions within the current local window) are stored and loaded in full with first precision, ensuring the core numerical accuracy of attention calculations. Historical key-value pairs belonging to neither the first nor the second set are quantized and stored with second precision. Through a link loading process of index table location, quantized data packet reading, and real-time dequantization, the memory usage is significantly reduced. The first cache region uses direct mapping addressing to achieve constant-time access complexity. The second cache region uses indirect addressing through an index table, achieving a balance between compression ratio and access overhead. Dequantization operations are performed immediately in the registers or shared memory of the computation core, avoiding the premature decompression of compressed data to main memory, further reducing memory bandwidth pressure. This allows the model to maintain a context length several times that of traditional KV caches within limited GPU memory capacity in extremely long sequence inference scenarios, while keeping the impact on core computational accuracy within an acceptable range.

[0106] S5: Write the new key-value pair to the mixed-precision cache.

[0107] Using the target position index of the current word in the inference result sequence as a storage identifier, the new key-value pair corresponding to the current word is written to the first cache area of ​​the mixed precision cache with first precision, and the target position index is added to the second set of the current decoding round; when the target position index no longer belongs to the second set due to window sliding in subsequent decoding rounds and is not added to the first set, the key-value pair corresponding to the target position index is migrated from the first cache area to the second cache area; the migration includes: converting the data format of the key-value pair from first precision to second precision quantization representation, and associating the corresponding quantization parameters with the converted quantization representation.

[0108] The storage identifier is used to uniquely identify the index key of the key-value pair's storage location in the mixed-precision cache. In this application, the absolute position index of the source lexicon in the output sequence is used as the storage identifier. The target position index is the absolute position index of the newly generated lexicon in the inference result sequence. This index is counted starting from the length of the input lexicon sequence and increments by 1 for each generated lexicon. The first cache area, a contiguous memory space in the mixed-precision cache, is dedicated to storing first-precision key-value pairs. It is stored in full-precision native format and supports direct addressing via position index. The second cache area, a mixed-precision cache, is a memory space dedicated to storing second-precision key-value pair quantization data packets. The stored content is compressed fixed-point data and its corresponding quantization parameters. Adding the second set of the current decoding round incorporates the position index of the newly generated lexicon into the local window set of the current decoding round, allowing it to enjoy full-precision protection of the first precision in subsequent decoding rounds. As the decoding rounds progress, the position index t of the currently generated lexicon increments, the end point of the interval in the second set shifts to the right, and the start point of the interval shifts to the right accordingly, causing some historical position indices to exit the set. Migration is the process of moving key-value pairs from a first cache area to a second cache area, including data format conversion, storage location change, and metadata update. Data format conversion is the quantization process that converts the key-value pair's data representation from a first-precision floating-point format to a second-precision fixed-point format. Quantization represents the second-precision fixed-point data formed after low-bit quantization of the key-value pair; its storage volume is smaller than the original first-precision representation. Associative storage establishes a correspondence between the quantization parameters and the quantized key-value pair data and stores them together, ensuring that subsequent dequantization operations can correctly recover approximate floating-point values. Quantization parameters are the calibration parameters required for quantization and dequantization, including at least a scaling factor and zero point, used to describe the mapping relationship from the original floating-point data range to the fixed-point representation space.

[0109] This application constructs a dynamic state maintenance closed loop for a hybrid precision caching system, achieving integrated collaborative management of new word writing and old word degradation. During the writing phase, the absolute position index of the current word in the inference result sequence is used as a storage identifier. The corresponding new key-value pair is directly written to the first cache area in first-precision floating-point format. Simultaneously, the position index is added to the second set of the current decoding round, allowing it to immediately enjoy full-precision storage and access protection. This design, based on the triple technical considerations of temporal proximity, future uncertainty, and write-as-you-go protection, ensures that newly generated words enter the caching system with their highest value state. During the migration phase, when the position index of a historical word no longer belongs to the second set due to window sliding and is not absorbed by the first set, the system triggers a progressive degradation operation: the data format of the key-value pair is converted from first-precision floating-point representation to second-precision quantized representation; the scaling factor and zero point are determined by analyzing the data distribution; fixed-point mapping compression is completed; the quantized data and quantization parameters are encapsulated into a quantized data packet, which is then written to the second cache area and the index table is updated, while simultaneously releasing the space in the original first cache area. This double negation mechanism ensures that only lexical terms that are neither important in the short term nor in the long term are downgraded, minimizing accuracy loss due to erroneous migration. This application achieves fundamental differences such as default high-precision writing and automatic windowing of new lexical terms, gradual downgrading of old lexical terms while retaining quantized copies, determination based on window affiliation and first set affiliation, information retention with only precision reduction, proactive and predictable resource reclamation, and continuity of semantic information before and after migration. Furthermore, this mechanism supports flexible configuration of window margin cooldown, batch migration, and dynamic thresholds, allowing for adaptive optimization based on business scenarios and resource pressure.

[0110] S6: Generate the position index of the word based on the current decoding round, and update the second set.

[0111] Obtain the absolute position index of the tokens generated in the current decoding round in the inference result sequence; use the absolute position index as the interval endpoint, and determine the interval start point based on the absolute position index, the preset window length, and the sequence start boundary; determine a continuous position index interval based on the interval start point and the interval endpoint, as the second set of the current decoding round; the number of position indices contained in the interval does not exceed the preset window length, and all position indices are non-negative integers.

[0112] In the autoregressive decoding process, as the decoding rounds progress, the model's accessibility to historical context needs to be dynamically constrained to balance computational efficiency and local dependency modeling capabilities. Therefore, this embodiment proposes a specific implementation method based on a sliding window mechanism to dynamically update the second set.

[0113] First, obtain the absolute position index of the generated token in the inference result sequence at the current decoding round, denoted as g. This absolute position index starts counting from 0. For example, for the first generated token, g = 0, and for each subsequent generated token, g is incremented by 1.

[0114] Second, taking the absolute position index g as the interval end point, and jointly determine the interval start point according to the preset window length W and the sequence start boundary. The preset window length is a hyperparameter used to limit the maximum number of historical tokens that can be focused on at the current query position. The sequence start boundary usually takes the value of 0, representing the start position of the sequence. The calculation formula for the interval start point s is: s = max(0, g - W + 1);

[0115] Among them, the max() function is used to ensure that the interval start point is not less than the sequence start boundary (i.e., 0), thus avoiding negative indexes.

[0116] Subsequently, based on the interval start point s and the interval end point g, determine a continuous position index interval, that is: Second set = [s, g] = {s, s + 1,..., g};

[0117] This interval is a continuous closed interval, and all integer indexes included in the interval represent the positions of historical key-value pairs that are allowed to be accessed at the current decoding round g. The number of position indexes included in the interval |Second set| = g - s + 1, this value does not exceed the preset window length W, and all position indexes are non-negative integers.

[0118] In the above way, the Second set slides dynamically as the current decoding round g increases: when g < W, the interval range is [0, g], and the window expands gradually as the sequence grows; when g ≥ W, the interval is fixed at [g - W + 1, g], and the window slides stably with a length of W. This mechanism ensures that the model only focuses on the most recent up to W historical tokens at any decoding round, thereby effectively controlling the computational complexity and strengthening the ability to model local context.

[0119] In this way, by taking the absolute position index as the interval end point and combining boundary constraints and window length to calculate the interval start point, an automatic update mechanism for the dynamic sliding of the Second set with the decoding round is realized. There is no need to explicitly maintain the cache state, and the attention range of the current query can be accurately defined only through pure position index calculation. It has the advantages of extremely low computational overhead, simple implementation, and clear boundary conditions, and is suitable for attention window control in large-scale autoregressive generation tasks.

[0120] S7: In response to meeting the termination condition, output the tokens already generated in the current model decoding stage as the inference result sequence.

[0121] After writing the new key-value pair corresponding to the current word into the mixed precision cache, it is determined whether the termination condition is met. The termination condition is that the current word is a preset sequence end marker, or the length of the generated sequence reaches a preset maximum length threshold. When the termination condition is met, the decoding iteration is terminated, and all currently generated words are combined into an inference result sequence in the order of generation, and output as a response to the inference request.

[0122] The system determines whether the currently generated lexical unit is a predefined sequence end marker. This marker is used to identify the end point of the generation task. When the model output matches the EOS (Equation of Order of Presence), it indicates that the model has autonomously determined that the generation task is complete. The sequence end marker here can be a predefined special lexical unit in the large language model that indicates the completion of sequence generation, typically... <eos>The maximum length threshold can be a pre-defined upper limit for the length of the inference result sequence, used to prevent the model from generating indefinitely or from looping without limit due to generation failure. If any termination condition is met, the decoding iteration is terminated, the calculation and generation process of all subsequent decoding rounds is stopped, and all currently generated lexical units are combined into an inference result sequence in the order of generation. Finally, this sequence is used as the response output to the inference request. If the termination condition is not met, the currently generated lexical units are used as the input of the next decoding round. The decoding round count is incremented and the steps of query vector calculation, key-value pair loading, attention operation, and new lexical unit generation are continued until the termination condition is met.

[0123] In one embodiment, this application includes performing a first set update operation in response to a preset trigger condition. The first set update operation includes: obtaining a preset number of attention weights generated during the decoding process by the generated lexical units during the attention calculation process as target attention weights; the target attention weights include the attention scores calculated for each historical key-value pair of each generated lexical unit during the decoding process; constructing a local attention probability matrix for the position indexes within a local range based on the target attention weights; performing column-wise aggregation on the local attention probability matrix and recalculating the column importance score of each historical position index within the current local range; fusing the recalculated column importance score with the historically aggregated column importance score to obtain an updated column importance score; determining the current selection quantity based on the product of a preset key position ratio parameter and the total number of position indexes of the current attention head in the current layer; selecting the current selection quantity of position indexes from the position indexes in descending order of the updated column importance score; and updating the first set.

[0124] This application responds to the fulfillment of preset triggering conditions by performing a first set update operation to dynamically adjust the recognition results of global key anchors, adapting to the dynamic shift of semantic importance in long context reasoning. The preset triggering conditions may be that the number of decoding rounds reaches a preset update cycle (e.g., once every N tokens generated), or that a significant change in attention distribution is detected (e.g., by calculating the KL divergence of attention distribution in adjacent decoding rounds), or triggered according to a specific strategy configured by the user.

[0125] When the triggering condition is met, a preset number of attention weights generated during the decoding process can be retrieved as target attention weights. The target attention weights include the attention scores calculated for all historical key-value pair positions when each generated word is used as a query in the most recent b decoding rounds. These attention scores form a two-dimensional matrix, with rows corresponding to b query positions and columns corresponding to all historical key-value pair positions. Based on the target attention weights, a local attention probability matrix is ​​constructed for the position indexes within a local scope. The local scope can be the position within the current sliding window or the global scope, depending on the configuration. This local attention probability matrix reflects the distribution of attention intensity for each historical position from each query position in recent decoding rounds. Subsequently, the local attention probability matrix is ​​accumulated and aggregated in the column direction, and the column importance score of each historical position index in the current local scope is recalculated. The recalculated column importance score is then merged with the historically accumulated and aggregated column importance score to obtain the updated column importance score. The fusion method can employ strategies such as weighted averaging (e.g., historical scores account for 0.7, and new scores account for 0.3), moving average, or maximum value fusion (taking the larger value of the old and new scores) to balance long-term statistical stability with short-term dynamic sensitivity. Next, the current selection quantity is determined by multiplying the preset key position ratio parameter by the total number of position indices of the current attention head in the current layer. The key position ratio parameter can be configured independently for different layers and different attention heads to accommodate the varying sensitivity of different layers to key positions.

[0126] Finally, from the position indexes, the current number of selected position indices are selected in descending order of the updated column importance score, and the first set is updated. The updated first set is the new global key anchor set. In subsequent decoding rounds, the key-value pairs corresponding to these positions will be stored in the first cache area with first precision (high-precision floating-point format) to ensure their numerical fidelity. In this way, the first set can be adjusted in real time during the decoding process, so that anchors identified early are removed from the protection range after their importance decays, while key evidence emerging later is promptly included in the protection, achieving adaptive identification of global key positions, thereby continuously ensuring the generation quality in long sequence inference.

[0127] In one embodiment, the mixed-precision cache of this application is organized using a paging memory management structure. The method further includes: dividing the first cache region and the second cache region in the mixed-precision cache into physical page frames of fixed size; allocating contiguous virtual address spaces for key-value pairs to be written, mapping the virtual address spaces to several physical page frames, and establishing a mapping from the location index to the virtual address space; when the length of the inference result sequence dynamically increases, resulting in insufficient physical page frames, allocating new physical page frames through a memory allocator and updating the page table entries; when a key-value pair migrates from the first cache region to the second cache region due to window sliding, performing a quantization conversion from the first precision floating-point format to the second precision fixed-point format in the original physical page frame, or reallocating a new physical page frame to store the quantized second precision fixed-point format data and quantization parameters and releasing the original physical page frame.

[0128] To support the dynamic growth of the number of key-value pairs and the migration of key-value pairs between different precision storage areas during ultra-long sequence inference, the hybrid precision cache of this application adopts a paging memory management structure to improve memory utilization and reduce fragmentation.

[0129] First, the first and second cache regions in the mixed-precision cache are divided into fixed-size physical page frames. A physical page frame is the smallest unit of memory allocation, and its size is configurable, such as 4KB, 16KB, or 64KB, to accommodate different scales of key-value pair storage needs. Each physical page frame has a unique physical address and cannot be used across regions; that is, page frames in the first cache region store only first-precision data, and page frames in the second cache region store only second-precision data.

[0130] Secondly, a contiguous virtual address space is allocated for the key-value pairs to be written. The virtual address space is a linear range of addresses visible to the application, independent of the actual distribution of the underlying physical memory. Through virtualization, applications can access key-value pairs as if they were in contiguous memory, without needing to know their physical storage location.

[0131] Subsequently, the virtual address space is mapped to several physical page frames, and page table entries are established to record the correspondence between virtual addresses and physical page frames. The page table is a data structure maintained by the operating system or runtime system to perform the translation from virtual to physical addresses. Simultaneously, a mapping from location indices to the virtual address space is established, allowing a given location index to quickly locate the corresponding virtual address, and then access the actual stored key-value pair data through the page table. This mapping relationship is typically implemented using a base address plus an offset: each location index corresponds to a fixed virtual address base address plus an offset equal to the index multiplied by the key-value pair size.

[0132] When the length of the inference result sequence dynamically increases, causing the allocated physical page frames to be insufficient to accommodate newly generated key-value pairs, new physical page frames are allocated through the memory allocator. The memory allocator is responsible for managing the allocation and reclamation of free physical page frames. When it receives an allocation request, it retrieves one or more contiguous or non-contiguous physical page frames from the free list and allocates them to the requester. After allocation, the corresponding page table entries are updated to map the newly added virtual address range to the newly allocated physical page frames, thereby achieving dynamic expansion of memory capacity.

[0133] When key-value pairs migrate from the first cache area to the second cache area due to window sliding—that is, when a key-value pair no longer belongs to the first set but remains in the second set or within the sliding window—it needs to be converted from high-precision storage to low-precision storage to save space. This can be done within the original physical page frame. Specifically, the first-precision floating-point data is directly converted to second-precision fixed-point data at the original storage location, and the corresponding quantization parameters are stored. This method avoids memory reallocation and data copying, resulting in higher efficiency, but requires the original page frame to support in-place format conversion.

[0134] Alternatively, a new physical page frame can be allocated to store the quantized second-precision fixed-point format data and quantization parameters, while the original physical page frame is released. This involves allocating a new physical page frame from the second cache area, writing the quantized data and corresponding scaling factors, zero points, and other quantization parameters into the new page frame, updating the page table entries to map the virtual address to the new page frame, and finally marking the original physical page frame as free and reclaiming it. This method is suitable for scenarios requiring complete release of high-precision storage space, or when in-place conversion is not feasible.

[0135] Thus, as the sequence grows, new page frames can be allocated in real time to expand capacity, supporting inference for ultra-long sequences; fixed-size page frames reduce external fragmentation, and virtualization technology eliminates the impact of internal fragmentation; when key-value pairs are migrated from the first cache area to the second cache area, they can be flexibly handled through in-situ conversion or allocation of new page frames, avoiding full data copying; this enables the mixed-precision cache to support the storage of key-value pairs for extremely long sequences and dynamic precision adjustment under limited physical memory.

[0136] In one embodiment, in response to the completion of the current inference request processing and the generation of a complete inference result sequence, the mixed-precision cache state corresponding to the current session is saved to the storage medium; the cache state includes at least all key-value pair data and corresponding quantization parameters stored in the first cache region and the second cache region, the current composition information of the first set, and the memory organization information of the mixed-precision cache; in response to receiving a new inference request associated with the current session, the saved mixed-precision cache state is loaded from the storage medium; based on the loaded cache state, the initial state of the first set and the decoding stage is restored, and the model pre-filling stage is skipped, and the iterative generation of the decoding stage continues directly from the last lexical position corresponding to the loaded cache state.

[0137] To support scenarios that require maintaining context memory across requests, such as multi-turn dialogues and long document interactions, this application provides a persistence and session recovery mechanism for hybrid precision caching, enabling different inference requests to share historical context information and avoid redundant calculations of the pre-filling stage.

[0138] First, in response to the completion of the current inference request processing and the generation of a complete inference result sequence—that is, when the decoding termination condition is met and all output tokens of the current request have been generated—a cache persistence operation is performed. The mixed-precision cache state corresponding to the current session is saved to a storage medium. This storage medium can be a persistent storage device such as a local disk, distributed file system, object storage, or memory-mapped file, used to retain context data between sessions. The saved cache state includes at least the following three core types of information:

[0139] Key-value pair data and quantization parameters: This includes all first-precision key-value pair data stored in the first buffer area, and all second-precision quantized data packets and their corresponding quantization parameters (such as scaling factors and zero points) stored in the second buffer area. This data completely records the key-value pair information corresponding to all tokens generated in the current session.

[0140] The first set contains current composition information: the set of global key anchor point location indices identified in the current session. This information determines which key-value pairs should be stored and accessed with first-precision precision in subsequent decoding.

[0141] Mixed-precision cache memory organization information includes virtual address space layout, physical page frame allocation, page table mapping relationships, and location index-to-virtual address mapping tables—all state data required for paging memory management. This information ensures that the original memory access semantics are maintained after cache recovery.

[0142] When a user or system initiates a new inference request associated with the same session (e.g., continuing a previous conversation in a multi-turn dialogue, or adding a question while reading a long document), that is, in response to receiving a new inference request associated with the current session, the system identifies the session identifier (such as session ID) carried in the request and locates the saved cached state corresponding to that session. The saved mixed-precision cached state is loaded from the storage medium. The loading process includes: reading key-value pair data and quantization parameters into memory and refilling them into the first and second cache areas; restoring the composition information of the first set; and reconstructing the virtual address space, page table mapping, and location index mapping based on memory organization information. The first set and the initial state of the decoding stage are restored based on the loaded cached state. After the first set is restored, the key anchor point location indices it contains can continue to enjoy first-precision protection in subsequent decoding; after the initial state of the decoding stage is restored, the model can accurately know the length of the currently generated sequence, the position of the last term, and other information.

[0143] Finally, skipping the model pre-filling stage, the iterative generation of the decoding stage continues directly from the last lexical position corresponding to the loaded cached state. This means that new inference requests do not need to reprocess the historical context (such as repeatedly running the attention calculation in the pre-filling stage), but instead generate new lexicals directly from the end of the historical sequence, greatly reducing the response latency and computational overhead of multi-round interactions.

[0144] In one specific implementation, such as Figure 6 As shown, the model inference optimization method of this application can be as follows:

[0145] Step S1: Parameter and runtime initialization.

[0146] Before inference begins, configure or obtain at least the following parameters: window size W, vertical stripe ratio ρ, quantization bit width (e.g., Int8 / Int4 / FP8), quantization granularity (token / channel / group), update cycle (step or segment), and block size / thread organization method that matches the hardware backend.

[0147] Simultaneously initialize the running-state structure: vertical stripe score container S(k) (which can be maintained by layer / head), and vertical stripe set K. VL Local window set K Win(t) Quantization set K Q And the storage handles and index structures of the hybrid KV Cache.

[0148] Step S2: Key Location Awareness and Set Partitioning in the Prefill Stage. In the prefill stage, column importance scores are calculated and local window sets are constructed to complete key location awareness and set partitioning.

[0149] Step S3: Prefill stage: Fusion mixed precision quantization attention calculation. Fusion mixed precision operators are used in the attention calculation of Prefill.

[0150] Step S4: Construct a prefilled hybrid KV cache. Write the prefilled historical KV data into the hybrid cache by set. Store the key sets in high precision and the rest in low bit and quantization parameters.

[0151] Step S5: During the decoding stage, when a new word is generated, the local window is updated sequentially, the vertical stripe set is refreshed as needed, the full-precision or low-bit key-value pairs are loaded through the fusion kernel and dequantized in real time to participate in attention calculation, and finally the new key-value pairs are written to the hybrid cache.

[0152] Step S6: Terminate the loop when the maximum length is reached or the stop token set is hit, and output the final result.

[0153] The model inference optimization method of this application consists of three parts: (1) Vertical-line awareness, which is used to identify and continuously update the key locations that need to be protected; (2) Mixed precision quantization attention calculation in the prefill stage, which is used to reduce computational costs and control errors in the computationally intensive stage; (3) Low-bit KV caching and fused decoding attention calculation in the decoding stage, which is used to reduce KV storage and access overhead in the bandwidth / capacity-limited stage and avoid the additional memory traffic caused by dequantization.

[0154] The following describes the implementation process of the model inference optimization method of this application using two specific application scenarios. These embodiments are for illustrative purposes only and do not constitute a limitation on the scope of protection; the parameters (such as the critical ratio ρ, window W, quantization bits, etc.) can be adjusted according to business requirements, hardware conditions, and model characteristics.

[0155] Example 1: Real-time reasoning in long online intelligent customer service conversations.

[0156] In this embodiment, the service is an online intelligent customer service / work order system. Users continuously ask follow-up questions, supplement information, and upload historical order / work order summaries within the same session. The system needs to perform multiple rounds of reasoning and information aggregation within a long context (e.g., 8K~32K or even longer) and return answers with low end-to-end latency. Traditional solutions using full-precision attention require repeatedly reading a large number of historical key-value pairs at each step of the Decode stage, causing the output latency to increase rapidly with the increase of context. If unified low-bit key-value quantization or low-bit attention is directly adopted, errors accumulate in the later stages of long CoT inference, leading to problems such as unstable responses and omissions of key facts.

[0157] The end-to-end acceleration process for this application in this scenario is as follows:

[0158] (1) Request reception and session context concatenation.

[0159] The customer service gateway receives requests containing: session ID, current user input, historical dialogue context (up to tens of thousands of tokens), and necessary metadata from the business side (such as order number, work order number, user profile tags, etc.). The inference service concatenates the historical context with the current input to form a prefix sequence input_ids and then proceeds to the Prefill stage.

[0160] (2) Online recognition of key positions of vertical stripes.

[0161] During the Prefill computation (by layer / by head), the inference service maintains a column importance score S(k) to characterize the quality of attention received cumulatively at key position k in the vertical direction.

[0162] ;

[0163] Where a t,k Let T be the attention probability of queryt to key k, and T be the set of queries in the current statistics window (which can be a whole segment or a sliding buffer).

[0164] Then, a set of vertical stripes is constructed by selecting the Top position according to the ratio ρ:

[0165] K VL =Top K (S, ρ*|K|);

[0166] Simultaneously construct a local sliding window set (covering the most recent W tokens):

[0167] K Win(t) =[max(0,t-W+1),t];

[0168] And obtain the set partition:

[0169] K= K VL ∪K Win(t) ∪K Q K Q =K(K VL ∪K Win(t) );

[0170] ρ can be 5%, and W can be 128 (this is just an example and can be configured).

[0171] (3) Prefill: Fusion-based hybrid precision quantization attention calculation.

[0172] For K VL ∪K Win(t) The position uses a full-precision path (such as FP16 / BF16) to participate in online softmax and output accumulation;

[0173] For K Q The position is represented using low bits (e.g., Key-4bit, Value-2bit), and K is dequantized on demand within the same attention kernel to obtain K. ^ Q V ^ Q Subsequently, it participates in the same online softmax normalization and block accumulation together with the full-precision part, avoiding the additional HBM read / write and kernel overhead of dequantization and secondary calculation.

[0174] (4) Hybrid KV Cache establishment and maintenance.

[0175] After prefilling, historical key-value pairs are written to the hybrid cache by set: key set K VL ∪K Win The corresponding key-value pairs are stored in full precision; the remaining key-value pairs are stored in low-bit form with quantization parameters (scale / zero-point or equivalent parameters). This cache structure is directly used for repeated access during the subsequent decoding stage.

[0176] (5) Decode: Fusion dequantization decoding attention and token generation.

[0177] The system enters a token generation loop. During each generation step:

[0178] a) Update K Win(t) (The window slides with time steps);

[0179] b) Optionally, update S(k) and refresh K periodically. VL To accommodate the non-stationarity of "key evidence / constraints" shifting with reasoning in customer service conversations;

[0180] c) Decoding the attention kernel block scan history KV: Hitting K VL ∪K Win(t) The blocks are loaded with full-precision key-value pairs directly; the remaining blocks are loaded with low-bit key-value pairs and dequantized as needed in registers / shared memory before immediately participating in score calculation and value accumulation; the softmax state is updated online throughout the process to avoid the dequantization results being stored as large tensors.

[0181] Compared to the effect of full quantization KV caching, retaining a small number of vertical lines significantly reduces the illusion effect. Tested on a general GSM8K dataset, the accuracy was not reduced compared to full quantization, while the direct quantization strategy reduced the accuracy by 3%. In terms of latency, a high-performance fusion operator is provided, which significantly reduces the latency of the Decode stage. Compared to the two attention schemes, the token generation speed is increased by 3.52 times, meeting the real-time requirements of dialogue. In terms of storage, it is equivalent to reducing the storage space by approximately 5 times compared to full storage, which significantly improves the upper limit of the text length that the model can handle within a limited space.

[0182] Example 2: Multi-document question and answer.

[0183] In this embodiment, the business is an enterprise compliance review and multi-document question-and-answer system: the system retrieves multiple document fragments (RAG) from the enterprise knowledge base / legal base / contract base, concatenates "multi-document evidence + user questions + review rules" into a very long prefix (e.g., 16K~128K), requires the model to output a review conclusion (CoT) containing multi-step reasoning, and performs consistency checks on key clauses. Common problems in traditional solutions in this scenario include:

[0184] (1) Full-precision KV results in excessive KV cache usage under long prefix, limiting concurrency and the maximum serviceable context;

[0185] (2) Although unified low-bit KV quantization saves capacity, when multiple documents are cross-referenced and numerical / clause constraints are repeatedly read back, errors in key positions are more likely to dominate the output, resulting in instability of sensitive fields such as "clause number / amount / term".

[0186] (3) Separate dequantization introduces additional large-scale intermediate tensor reads and writes in the Decode stage, which aggravates the bandwidth bottleneck and causes queuing and tail delay jitter during peak review requests.

[0187] The end-to-end process of this application in this scenario is as follows:

[0188] (1) Retrieval and prefix construction.

[0189] The compliance service receives user review tasks (e.g., "check whether a contract complies with a certain legal provision and point out risk points"), and the retrieval module returns several evidence fragments (multiple documents, multiple paragraphs). The system concatenates the evidence fragments, citation identifiers, rule templates, and user questions into a long prefix input_ids according to a preset format, and then proceeds to Prefill.

[0190] (2) Vertical stripe anchor point identification (for "repeatedly cited evidence").

[0191] During the prefill process, the importance S(k) of this application is statistically analyzed online. In compliance / RAG scenarios, key positions of vertical bars often correspond to anchor tokens that are repeatedly read back, such as document titles, clause numbers, amounts / dates, constraints, and the beginning of evidentiary paragraphs. The system selects K based on ρ. VL And combined with local window K Win(t) Form a protective set.

[0192] (3) Prefill: Mixed precision quantification attention (to ensure high fidelity of evidence anchors).

[0193] For the protection set K VL ∪K Win A full-precision path is adopted to ensure that the anchor points of multi-document evidence maintain a stable numerical representation even in low-bit environments;

[0194] For the remaining large number of background tokens, a low-bit path (e.g., K4 / V2 or equivalent configuration) is used, and mixed-precision calculations are performed in the same online softmax using a fusion kernel, controlling the additional overhead of prefill.

[0195] (4) Decode: Hybrid KV+ fusion dequantization (reduces bandwidth and stabilizes tail latency).

[0196] During the Decode phase, which outputs the review conclusions token by token, the system repeatedly accesses historical evidence key-value pairs (KV). This application maintains full precision for anchor and window KV, stores the remaining KV in low-bit increments, and integrates dequantization and accumulation in the decoding attention kernel to avoid the additional HBM traffic from dequantization, thereby achieving more significant decoding acceleration and more stable tail latency under long prefixes (e.g., 32K and above).

[0197] In one feasible implementation, the model inference optimization method further includes at least one optimization operation among multi-level caching operation, prefetching operation, and scene adaptation operation.

[0198] Multi-level caching operation: The mixed-precision cache also includes a third storage area for storing key-value pairs represented in third precision, where the number of bits corresponding to the third precision is lower than that of the second precision; in response to the position index corresponding to a historical key-value pair not belonging to the first set and not belonging to the second set in a consecutive preset number of decoding rounds, the key-value pair is migrated from the second cache area to the third storage area.

[0199] To further compress the cache space occupied by historical tokens accessed very infrequently, this operation introduces a third-level storage area on top of the original two-level storage architecture, employing a more aggressive third precision (such as 2-bit or 1-bit) for quantization compression. When a historical token is neither identified as a global anchor nor falls within a local window in multiple consecutive decoding rounds, it can be determined as long-term cold data and migrated from the second cache area to the third storage area. Through multi-level storage and a progressive degradation strategy, cache space utilization is maximized while avoiding the impact of "one-size-fits-all" compression on model performance.

[0200] Prefetching operation: Based on the similarity between the query vector of the current decoding round and the key vectors corresponding to each position index in the first set, predict the range of key positions to be accessed in the current decoding round, and preload the corresponding key-value pairs within the key position range from the first cache area to the high-speed cache of the computing core.

[0201] Memory access latency during the decoding phase is one of the key factors limiting the speed of single token generation. This operation leverages the inherent characteristics of the attention mechanism, noting that the current query vector often has a high similarity to the key vectors of some key elements in the first set. By calculating the average similarity between the current query vector and each key vector in the first set, the subset of key locations most likely to be accessed in the current decoding round can be predicted and pre-loaded from GPU memory into the shared memory or registers of the computation core. This prefetching mechanism works in conjunction with a differentiated loading method, effectively hiding memory access latency through "computation and memory access overlap," further improving decoding throughput.

[0202] Scenario adaptation operation: In response to the application scenario of the inference result sequence being a real-time interactive task, the preset window length and preset key position ratio parameters are reduced, and the number of bits corresponding to the second precision is increased; in response to the application scenario being an offline batch processing task, the preset window length and preset key position ratio parameters are increased, and the number of bits corresponding to the second precision is reduced.

[0203] The embodiments of this application provide a model inference optimization apparatus, which is specifically as follows: Figure 7 As shown, the model inference optimization device includes: a determination module 20, an iteration module 21, and an output module 22.

[0204] The determination module 20 is used to determine the attention probability matrix based on the input word sequence corresponding to the inference request in response to the model pre-filling stage, perform column-wise aggregation calculation on the attention probability matrix to determine the first set; and determine the second set based on the position index of each word in the input word sequence and the preset window. The first set and the second set are composed of position indexes. The position index is used to identify the storage location of the key-value pair corresponding to each word. For the key-value pair corresponding to the input word sequence, the attention calculation method and storage precision are determined according to whether its corresponding position index belongs to the first set or the second set. Based on the attention calculation method and storage precision of the key-value pair, a mixed precision cache is established.

[0205] Iteration module 21, in response to being in the model decoding stage, iteratively executes the following: determining the loading method for reading historical key-value pairs from the mixed-precision cache based on whether the position index corresponding to the historical key-value pair belongs to the first set or the second set; generating the current word and the new key-value pair corresponding to the current word based on the query vector of the current decoding round and the historical key-value pairs obtained based on the loading method; writing the new key-value pair into the mixed-precision cache; and updating the second set based on the position index of the word generated in the current decoding round.

[0206] Output module 22 is used to output the tokens generated in the current model decoding stage as a sequence of inference results in response to the satisfaction of the termination condition.

[0207] like Figure 8 As shown, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described model inference optimization method embodiments.

[0208] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the model inference optimization method at runtime.

[0209] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.< / eos>

Claims

1. A model reasoning optimization method, characterized in that, The method includes: In response to being in the model pre-filling stage, the attention probability matrix is ​​determined based on the input word sequence corresponding to the inference request, and the column-direction aggregation calculation is performed on the attention probability matrix to determine the first set; Based on the position index of each word in the input word sequence and a preset window, a second set is determined. The first set and the second set are composed of position indexes. The position index is used to identify the storage location of the key-value pair corresponding to each word. Obtain the position index of each word in the input word sequence; The key-value pair corresponding to the position index belonging to the first set or the second set is directly used in the calculation of the current attention layer in the first precision floating-point format, and after the calculation is completed, the key-value pair is written to the cache in the first precision. The key-value pairs corresponding to the position indices that do not belong to the first set and the second set are converted into the second precision fixed-point format. In the current attention layer's computation kernel, the data in the second precision fixed-point format is dequantized in real time based on the quantization parameters. The approximate floating-point value recovered after dequantization is used for attention computation. At the same time, after the attention computation is completed, the second precision fixed-point format data of the key-value pairs and the quantization parameters are written to the cache. The key-value pairs cached with the first precision are stored continuously in the first cache area, and a direct mapping from the position index to the first cache area is established. The quantized data and quantization parameters compressed using key-value pairs in the second precision cache are packaged and stored in the second cache area, and an index table is established from the position index to the cache address of the quantized data packet. The first cache region and the second cache region together constitute a mixed-precision cache for subsequent decoding stage reading and access; The quantization parameters include at least a scaling factor and a zero point, and the number of bits corresponding to the second precision is lower than the number of bits corresponding to the first precision. In response to being in the model decoding phase, iterative execution continues: Based on whether the position index corresponding to the historical key-value pair belongs to the first set or the second set, determine the loading method for reading historical key-value pairs from the mixed precision cache. Based on the query vector of the current decoding round and the historical key-value pairs obtained based on the loading method, generate the current word element and the new key-value pair corresponding to the current word element. Write the new key-value pair into the mixed-precision cache; Update the second set based on the position index of the generated lexical unit in the current decoding round; In response to the termination condition being met, the tokens generated in the current model decoding phase are output as the inference result sequence.

2. The model inference optimization method according to claim 1, characterized in that, Determining the attention probability matrix based on the input lexical sequence corresponding to the inference request includes: performing multi-head attention calculation on the input lexical sequence. The multi-head attention computation performed on the input lexical sequence includes: Perform a dot product operation on the query vector and key vector corresponding to each attention head, divide the dot product result by a scaling factor, and input it into a normalized exponential function to obtain the single-head attention probability matrix corresponding to each attention head. Then, fuse the single-head attention probability matrices corresponding to each attention head to form an attention probability matrix.

3. The model inference optimization method according to claim 1, characterized in that, The aggregation calculation of the attention probability matrix along the column direction to determine the first set includes: The attention weights contained in each column of the attention probability matrix are summed, and the summed value is used as the column importance score of the position index corresponding to each column. The product of the preset key position ratio parameter and the total number of current attention head position indices is used as the selection quantity; From the position indexes, the selected number of position indexes are chosen in descending order of the column importance score to form the first set.

4. The model inference optimization method according to claim 1, characterized in that, The determination of the second set based on the position index of each word in the input word sequence and a preset window includes: Use the index of each word in the input word sequence as the current query position; The starting point of the interval is determined by combining the current query position, the preset window length, and the sequence start boundary. Using the current query position as the interval endpoint, a continuous location index interval is determined based on the interval start and end points; Wherein, the number of position indices contained in the position index interval does not exceed the preset window length, and the position index is a non-negative integer; The location indices contained within the location index range are used as the second set corresponding to the term at the current query position.

5. The model inference optimization method according to claim 1, characterized in that, The step of determining the loading method for reading historical key-value pairs from the mixed-precision cache based on whether the position index corresponding to the historical key-value pair belongs to the first set or the second set includes: Traverse the historical key-value pairs already stored in the mixed-precision cache; Obtain the position index corresponding to each historical key-value pair, and determine whether the position index corresponding to the historical key-value pair belongs to the first set or the second set. The loading method for determining the historical key-value pairs corresponding to the position index belonging to the first set or the second set is as follows: based on the position index corresponding to the historical key-value pairs, the complete key-value data cached with the first precision is directly read from the first cache area. The loading method for the historical key-value pair corresponding to the position index that does not belong to the first set and does not belong to the second set is as follows: locate the corresponding quantization data packet from the index table of the second cache area according to the position index corresponding to the historical key-value pair, read the compressed quantization data and the corresponding quantization parameters from the quantization data packet, and perform dequantization operation in the register or shared memory of the computing core to obtain an approximate floating-point value.

6. The model inference optimization method according to claim 1, characterized in that, The process of generating the current lexical unit and its corresponding new key-value pair based on the query vector of the current decoding round and the historical key-value pairs obtained based on the loading method includes: The original attention score is obtained by performing a dot product operation between the query vector of the current decoding round and the key vector in the loaded historical key-value pairs. The original attention scores are normalized to obtain the attention weights corresponding to each historical position index; The attention weights and value vectors corresponding to each historical position index are weighted and summed to obtain the attention layer output vector for the current decoding round. Based on the output vector of the attention layer, the output tokens of the current decoding round and the new key-value pairs corresponding to the tokens are generated.

7. The model inference optimization method according to claim 1, characterized in that, The step of writing the new key-value pair into the mixed-precision cache includes: The target position index of the current word in the inference result sequence is used as the storage identifier. The new key-value pair corresponding to the current word is written into the first cache area of ​​the mixed precision cache with the first precision, and the target position index is added to the second set of the current decoding round. When the target position index no longer belongs to the second set due to window sliding in subsequent decoding rounds and is not added to the first set, the key-value pair corresponding to the target position index is migrated from the first cache area to the second cache area of ​​the mixed precision cache. The migration includes: converting the data format of key-value pairs from the first precision to a second precision quantization representation, and associating and storing the corresponding quantization parameters with the converted quantization representation.

8. The model inference optimization method according to claim 1, characterized in that, The step of generating the position index of the lexical unit based on the current decoding round and updating the second set includes: Obtain the absolute position index of the word generated in the current decoding round in the inference result sequence; The absolute position index is used as the interval endpoint, and the interval start point is determined by the absolute position index, the preset window length, and the sequence start boundary. A continuous set of position index intervals is determined based on the start and end points of the interval, and used as the second set for the current decoding round. The number of position indices contained within the interval does not exceed the preset window length, and all position indices are non-negative integers.

9. The model inference optimization method according to claim 1, characterized in that, The step of outputting the generated lexical units in the current model decoding phase as a sequence of inference results in response to the satisfaction of the termination condition includes: After writing the new key-value pairs corresponding to the tokens generated in the current decoding round into the mixed precision cache, it is determined whether the termination condition is met. The termination condition is that the tokens generated in the current decoding round match the preset sequence end marker, or the length of the sequence formed by the currently generated tokens reaches the preset maximum length threshold. When the termination condition is met, the iteration terminates, and the currently generated lexical units are combined in the order of generation into a sequence of inference results, which is then output as a response to the inference request.

10. The model inference optimization method according to claim 1, characterized in that, The method further includes: In response to the fulfillment of a preset trigger condition, a first set update operation is executed, the first set update operation including: The target attention weight is obtained by acquiring the attention weights generated during the attention calculation of a preset number of tokens generated during the decoding process. The target attention weights include the attention scores calculated for each generated lexical pair and each historical key-value pair during the decoding process; Based on the target attention weights, a local attention probability matrix is ​​constructed for the location indexes within a local range; The attention probability matrix of the local area is accumulated and aggregated in the column direction, and the column importance score of each historical position index in the current local range is recalculated. The recalculated column importance score is merged with the historically accumulated and aggregated column importance score to obtain the updated column importance score; The current selection quantity is determined by multiplying the preset key position ratio parameter with the total number of position indices of the current attention head in the current layer. The position indices of the current selection quantity are selected from the position indices in descending order of the updated column importance score, and the first set is updated.

11. The model inference optimization method according to claim 1, characterized in that, The method further includes: The first and second cache regions in the mixed precision cache are divided into physical page frames of fixed size; Allocate a contiguous virtual address space for the key-value pairs to be written, map the virtual address space to several physical page frames, and establish a mapping from the location index to the virtual address space; When the length of the inference result sequence dynamically increases, resulting in insufficient physical page frames, a new physical page frame is allocated through the memory allocator and the page table entries are updated. When a key-value pair migrates from the first cache area to the second cache area due to window sliding, a quantization conversion from the first precision floating-point format to the second precision fixed-point format is performed in the original physical page frame, or a new physical page frame is reallocated to store the quantized second precision fixed-point format data and quantization parameters, and the original physical page frame is released.

12. The model inference optimization method according to claim 1, characterized in that, The method further includes: In response to the completion of the current inference request and the generation of a complete inference result sequence, the mixed precision cache state corresponding to the current model decoding stage is saved to the storage medium. The cache state includes at least: all key-value pair data and corresponding quantization parameters stored in the first cache region and the second cache region, the current position index set of the first set, and the physical page frame allocation and page table mapping information of the mixed precision cache; In response to receiving a subsequent inference request for the same inference task, the stored mixed-precision cache state is loaded from the storage medium; Based on the loaded cache state, the absolute position index of the first set and the current decoding round is restored, and the initial pre-filling stage is skipped. Decoding is then performed starting from the last position index corresponding to the loaded cache state.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the model inference optimization method as described in any one of claims 1 to 12 when executing the computer program.

Images