A similarity-aware important prefix kv speculation prefetching method and system

Abstract

The application discloses a similarity-aware important prefix KV speculative prefetching method and system, and relates to the technical field of large language model inference acceleration. In view of the fact that in the existing large language model inference system, important prefix KV loading has strict read-after-computation dependence, leading to the problem that I / O delay cannot be concealed, the application proposes to use the high similarity of the important token index set between adjacent Transformer layers for speculative prefetching. The method of the application uses the important token index of the current layer to predict the important token of the next layer and asynchronously loads the KV of the important token while calculating the current layer; meanwhile, the system realizes reactive truncation by accurately monitoring the GPU calculation state to prevent invalid prefetching; before the calculation of the next layer, the system performs a lightweight check and only synchronously loads the important KV that has not been prefetched. The application successfully breaks the serial bottleneck of calculation and I / O, and significantly reduces the first word delay of large model inference.

Description

Technical Field

[0001] This invention relates to the field of computer science and artificial intelligence systems, specifically to a method and system for speculative prefetching of key prefixes (KV) for similarity perception. Background Technology

[0002] Large language models based on Transformers typically include a pre-filling stage and a decoding stage during inference. For requests with long contexts, the pre-filling stage needs to compute the key and value vectors for each layer of the input prefix, i.e., a KV cache. In applications such as retrieval enhancement generation, multi-turn dialogues, tool calls, and few-shot hints, different requests often share the same or partially the same prefixes. Therefore, storing the KV cache of shared prefixes and reusing it in subsequent requests can reduce redundant computations and lower the latency of the first token generation.

[0003] As prefix length, concurrent requests, and model size increase, GPU memory and CPU memory struggle to accommodate all reusable prefix key-value caches. To expand cache capacity, existing systems can store prefix key-value caches in external memory and load the corresponding layer of key-value cache into GPU memory when a request arrives. However, the data transfer bandwidth and latency from external memory to GPU memory become new bottlenecks, especially when long prefix key-value caches need to be loaded layer by layer; I / O wait may account for a major portion of the initial token generation latency.

[0004] To reduce loading overhead, existing techniques can load only the prefix key-value cache that is most important to the current request. This type of method typically determines important tokens based on the attention weight between the current layer's query vector and the prefix key vector, and then reads the corresponding key-value pairs. While this strategy reduces data transfer, it introduces a new dependency: the set of important tokens at layer i+1 can only be determined after layer i has been computed and the input for layer i+1 has been obtained. Therefore, the key-value cache for the next layer cannot be prefetched during the current layer's computation, making it difficult for selective loading and prefetching to work simultaneously. Furthermore, the selective loading of important key-value pairs exposes them to the critical path of GPU inference, leading to GPU idleness and low utilization.

[0005] Therefore, a technical solution is needed that can utilize inter-layer features of the model to prefetch important prefix KV caches of the next layer without compromising inference accuracy, so as to reduce I / O wait on the critical path and improve the latency of long prefix inference in large language models. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for speculative prefetching of key prefixes (KV) for similarity perception.

[0007] The objective of this invention is achieved through the following technical solution: a similarity-aware key-value (KV) speculative prefetching method, comprising the following steps: Step 1: Receive the current request, match the reusable prefix corresponding to the current request in the prefix index structure, and obtain the KV metadata and storage location of the reusable prefix in each Transformer layer; Step 2: For the i-th layer in the Transformer layer, obtain the set of important prefix token indices S_{i} corresponding to the current request in that layer, and load the important key-value pairs required for the i-th layer to calculate; Step 3: After the calculation starts at the i-th layer, based on the similarity of the important prefix token index sets of adjacent Transformer layers, the candidate important prefix token index set S'{i+1} for the (i+1)-th layer is generated according to the important prefix token index set S_{i} obtained in Step 2, and the candidate important key-value pairs corresponding to S'_{i+1} for the (i+1)-th layer are asynchronously prefetched during the calculation of the i-th layer. Step 4: When the calculation of the i-th layer ends, stop initiating new candidate important KV prefetch requests for the (i+1)-th layer to avoid continuing to load invalid data outside the current layer calculation window; Step 5: Before the calculation of layer i+1 begins, obtain the actual important prefix token index set S_{i+1} of layer i+1, and compare and verify S_{i+1} with the pre-fetched candidate important KVs; among them, candidate important KVs belonging to S_{i+1} are used directly as important KVs; candidate important KVs not belonging to S_{i+1} are discarded; important KVs belonging to S_{i+1} but not pre-fetched are synchronously filled and loaded. Step six: Use the verified important KV cache to perform the calculation of the (i+1)th layer, and increment the layer index. Repeat steps three to five until the pre-filling stage ends or the last Transformer layer is reached. The above steps utilize the current layer's computation time to hide the I / O loading delay of important key-value pairs in the next layer, and correct speculative prefetching errors through verification and completion mechanisms, thereby reducing the first-word delay while maintaining the stability of the inference results.

[0008] Furthermore, the KV metadata includes layer index, token index, attention header index, cache block index, and the storage layer where the corresponding KV data resides.

[0009] Further, in step two, the important prefix token index set S_{i} is obtained in the following way: for each attention head in the i-th layer, the attention weight matrix between the query vector of the attention head and the reusable prefix key vector is calculated, and the attention weight matrix is ​​summed column by column to obtain the importance score of each prefix token under the attention head; the importance scores of each attention head are added together according to the corresponding prefix tokens to obtain the summary importance score of each prefix token shared by all attention heads; according to the summary importance score from high to low, a preset proportion or a preset number of prefix tokens are selected to form the important token index set S_{i} of the i-th layer.

[0010] Furthermore, in step three, generating the candidate important prefix token index set S'{i+1} for the (i+1)th layer includes: sorting S_{i} according to the importance score of the token, the historical prefetch hit rate, the prefetch buffer size, or the current I / O bandwidth, and then truncating or adjusting S_{i} to generate S'_{i+1}.

[0011] Furthermore, in step three, the asynchronous prefetching of the candidate important key-value pairs corresponding to S'_{i+1} at the (i+1)th layer during the computation of the i-th layer includes: prefetching candidate important key-value pairs in descending order of importance score; wherein, if the candidate important key-value pairs are located in GPU memory, their addresses are directly recorded or written to the prefetch buffer; if they are located in CPU memory, they are transferred to GPU memory; if they are located in external storage or disk, they are first read into CPU memory and then transferred to GPU memory.

[0012] This invention also provides a similarity-aware key-value speculative prefetching system, comprising: The important prefix KV identification module is used to match the reusable prefix corresponding to the current request in the prefix index structure, and obtain the KV metadata and storage location of the reusable prefix in each Transformer layer; for the i-th layer in the Transformer layer, obtain the important prefix token index set S_{i} corresponding to the current request in that layer, and load the important KV required for the i-th layer to calculate; The similarity-aware speculative prefetching module is configured to generate a candidate important prefix token index set S'{i+1} for the (i+1)th layer based on the important prefix token index set S_{i} after the calculation begins at the i-th layer, and asynchronously prefetch the candidate important key-value pairs corresponding to S'_{i+1} for the (i+1)th layer during the calculation of the i-th layer; continuously monitor the GPU status, and stop initiating new candidate important key-value prefetching requests for the (i+1)th layer when the GPU completes the calculation of the i-th layer; The verification and recovery module is configured to obtain the actual important prefix token index set S_{i+1} of the i+1 layer before the calculation of the i+1 layer begins, and compare S_{i+1} with the pre-fetched candidate important KV. Among them, the candidate important KV belonging to S_{i+1} is used directly as important KV; the candidate important KV not belonging to S_{i+1} is discarded; and the important KV belonging to S_{i+1} but not pre-fetched is synchronously filled and loaded. The iterative execution module is configured to trigger the calculation of the i+1th layer using the verified important KV cache after the verification and recovery module completes the comparison and verification of the candidate important KV of the i+1th layer. After the calculation of the i+1th layer is completed, the layer index is incremented, and the similarity-aware speculative prefetching module and the verification and recovery module are called again until the prefilling stage ends or the last Transformer layer is reached.

[0013] Furthermore, the KV metadata includes layer index, token index, attention header index, cache block index, and the storage layer where the corresponding KV data resides.

[0014] Furthermore, the important prefix KV identification module includes: for each attention head in the i-th layer, calculating the attention weight matrix between the query vector of the attention head and the reusable prefix key vector, and summing the attention weight matrix column by column to obtain the importance score of each prefix token under the attention head; adding the importance scores of each attention head according to the corresponding prefix tokens to obtain the aggregate importance score of each prefix token shared by all attention heads; sorting the aggregate importance scores from high to low, selecting a preset proportion or a preset number of prefix tokens to form the important token index set S_{i} of the i-th layer.

[0015] Furthermore, the similarity-aware speculative prefetching module includes: truncating or adjusting S_{i} to generate S'_{i+1} based on the importance score of the token, the historical prefetch hit rate, the size of the prefetch buffer, or the current I / O bandwidth.

[0016] Furthermore, the similarity-aware speculative prefetching module also includes: prefetching candidate important key-value pairs in descending order of importance score; wherein, if the candidate important key-value pair is located in GPU memory, its address is directly recorded or written to the prefetch buffer; if it is located in CPU memory, it is transferred to GPU memory; if it is located in external storage or disk, it is first read into CPU memory and then transferred to GPU memory.

[0017] The beneficial effects of this invention are as follows: First, this invention advances the loading of the next layer's important KV cache to the current layer's computation period, using GPU computation time to hide the data transfer latency from external storage or CPU memory to GPU video memory, thereby reducing critical path I / O wait in the first token generation latency.

[0018] Second, the present invention uses a post-speculation verification method to handle cross-layer prediction errors. The KV cache that actually participates in the attention calculation of the next layer is still determined by the set of real important prefix tokens of that layer. Therefore, the impact of prefetching errors on the inference results can be reduced.

[0019] Third, this invention only prefetches the KV cache corresponding to the candidate important prefix tokens, without needing to prefetch the complete prefix KV cache. Therefore, it can simultaneously utilize importance sparsity and cross-layer prefetching, making it suitable for multi-level KV cache systems composed of GPU memory, CPU memory, and external memory.

[0020] Fourth, this invention does not require modification of model weights, nor does it rely on additional training. It can be used as a runtime scheduling mechanism in inference systems in combination with existing prefix KV reuse, KV compression, cache replacement, and hierarchical storage strategies. Attached Figure Description

[0021] Figure 1 A schematic diagram of a key prefix KV speculative prefetching system for similarity perception; Figure 2 An example diagram of a key prefix KV speculative prefetching method for similarity perception; Figure 3 The diagram illustrates the similarity of important token sets between adjacent layers. (a) shows the similarity of adjacent layer sets when the top 10% of important tokens are selected, (b) shows the similarity of adjacent layer sets when the top 25% of important tokens are selected, and (c) shows the similarity of adjacent layer sets when the top 40% of important tokens are selected. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0023] Tests revealed that the sets of important tokens are similar in adjacent Transformer layers of the large model, such as... Figure 3As shown in Figures (a)-(c), in one embodiment, the important token index sets of adjacent Transformer layers in the Qwen3-8B model exhibit significant similarity. When the top 10%, top 25%, and top 40% of important tokens are selected, the average similarity of the adjacent layer sets is 0.553, 0.611, and 0.703, respectively. This result indicates that the important token index set of the current layer can be used to generate the candidate important token index set of the next layer, thus providing a basis for cross-layer speculative prefetching in this invention.

[0024] like Figure 1 As shown in the figure, a similarity-aware prefix key-value speculative prefetching system provided by an embodiment of the present invention includes the following modules: Important prefix KV identification module: used to analyze attention weights during the large model inference computation at each layer, output the set of truly important token indices for the current layer, and match reusable prefixes before the request begins computation.

[0025] Similarity-aware speculative prefetching module: used to break the dependency of post-read computation, it predicts the important KV of the next layer based on the important token index set of the current layer output by the important prefix KV identification module; while the GPU performs computation in the current layer, it asynchronously reads the predicted KV data into the video memory; at the same time, this module includes a precise time control mechanism, which stops initiating new I / O fetching requests the moment the GPU completes the computation of the current layer by continuously monitoring the GPU status.

[0026] Verification and recovery module: It is used to ensure the accuracy of data before the next layer of calculation; it receives the real important token index set of the next layer, compares and verifies it with the KV data extracted by the similarity-aware speculative prefetching module, and triggers the synchronous loading operation of missing important KV.

[0027] Iterative execution module: After the verification and recovery module completes the verification, completion and discarding of candidate important KV, it triggers the next layer calculation, updates the layer index after the calculation of this layer is completed, and schedules the similarity-aware speculative prefetching module and the verification and recovery module to enter the next round of processing, until the pre-filling stage ends or the last Transformer layer is reached.

[0028] In the data flow direction of the prefix KV, the prefix KV is read from external storage or SSD to CPU memory, and then transferred to GPU memory; in the control flow direction, the important prefix KV identification module outputs the important token index set of the current layer to the similarity-aware speculative prefetching module for predicting the next layer, and outputs the real important token index set of each layer to the verification and recovery module for comparison. The similarity-aware speculative prefetching module monitors the GPU's computing status to control the initiation and cessation of prefetching.

[0029] Based on the system described above, embodiments of the present invention provide a similarity-aware key-value speculative prefetching method, comprising the following steps: Step 1, prefix matching: Receive the current request, match the reusable prefix corresponding to the current request in the prefix index structure, and obtain the KV metadata and storage location of the reusable prefix in each Transformer layer.

[0030] Preferably, the metadata may include layer index, token index, attention header index, cache block index, and the storage layer where the corresponding KV data is located.

[0031] Step 2, Obtain the important set of the current layer: For the i-th Transformer layer, the important prefix token index set S_{i} corresponding to the current request at this layer is obtained through the important prefix KV identification module, and the important KV required for the calculation of the i-th layer is loaded. The important prefix token index set is determined by attention weights.

[0032] Specifically, the important prefix key-value recognition module calculates the attention weight matrix between the current layer query vector and the reusable prefix key vector in each attention head, and sums the attention weight matrix column by column to obtain the importance score of each prefix token under that attention head; then, the importance scores of each attention head are added together according to the corresponding prefix tokens to obtain the aggregate importance score of each prefix token shared by all attention heads; according to the aggregate importance score from high to low, a preset proportion or a preset number of prefix tokens are selected to form the current layer important prefix token index set S_{i}.

[0033] Step 3, Speculative prefetching based on similarity perception: Based on the similarity of important token index sets in adjacent Transformer layers, the similarity-aware speculative prefetching module generates a predicted candidate important token index set S'_{i+1} for the (i+1)th layer according to S_{i} obtained in step two.

[0034] In one implementation, S'{i+1} can be directly formed from S_{i}, that is, let S'_{i+1}=S_{i}.

[0035] In another implementation, the similarity-aware speculative prefetching module can truncate or adjust S_{i} to generate S'_{i+1} based on the importance score of the token, the historical prefetch hit rate, the size of the prefetch buffer, or the current I / O bandwidth.

[0036] While GPU computation is performed at layer i, the similarity-aware speculative prefetching module asynchronously initiates candidate important key-value prefetching requests based on S'_{i+1}, and prefetches candidate important key-values ​​in descending order of importance score.

[0037] Specifically, if the candidate important key-value pairs are located in GPU memory, their addresses are directly recorded or written to the prefetch buffer; if they are located in CPU memory, they are transferred to GPU memory; if they are located in external storage or disk, they are first read into CPU memory and then transferred to GPU memory, thus forming a prefix key-value data stream from external storage or disk to CPU memory and then to GPU memory. The prefetched candidate important key-value pairs are written to the prefetch buffer of the similarity-aware speculative prefetch module. For example... Figure 2 As shown, compared to existing methods that expose I / O serialization to the critical path due to read-after computation dependencies, resulting in GPU idleness, this step overlaps the loading of the next layer's KV with the computation of the current layer, thereby shortening the TTFT.

[0038] Step 4, Prefetch Truncation: The similarity-aware speculative prefetching module continuously submits prefetch requests while monitoring the current GPU computation status of layer i. When it detects that the computation of layer i has ended, it stops initiating new candidate important key-value prefetch requests for layer i+1 to avoid continuing to load outside the current layer's computation window, thus exposing useless data to the critical path and increasing latency. For key-value pairs that have been read but did not hit the actual important token index set, they can be released or retained by a caching strategy for use in subsequent requests. Figure 2 As shown, the time when prefetching stops corresponds to the time when the current layer's computation ends, and no new prefetching requests will be initiated after this time.

[0039] Step 5, Verification and Completion: When the model reaches layer i+1, the important prefix key-value identification module obtains the true important token index set S_{i+1} of layer i+1. The verification and recovery module compares S_{i+1} with the candidate important key-value pairs that have been prefetched in the prefetch buffer and processes them in three ways: for the matched key-value pairs, they are allowed to directly participate in the attention calculation of layer i+1; for the missing key-value pairs that belong to S_{i+1} but have not yet been prefetched, synchronous loading is triggered to fill them in; for the misprefetched key-value pairs that have been prefetched but do not belong to S_{i+1}, they are discarded, marked as invalid, or prohibited from participating in the attention calculation of layer i+1.

[0040] Since S'{i+1} is a prediction based on S_{i} and is not exactly the same as the true set S_{i+1}, the missing parts of S_{i+1} relative to S'_{i+1} always need to be filled in; however, these missing parts usually only account for a small portion of the required data, thus significantly reducing the amount of synchronous loading on the critical path. The verification only involves comparing the index set, and its overhead is far less than the time order of disk I / O.

[0041] Step 6, iterative execution: The iterative execution module uses the verified KV cache to perform the calculation of the (i+1)th layer, increments the layer index, and repeats steps three through five until the pre-filling phase ends or the last Transformer layer is reached.

[0042] To further illustrate the similarity-aware key-value (KV) speculative prefetching method provided in this embodiment of the invention, a specific embodiment is used to explain the process, such as... Figure 2 As shown: Assume the model consists of two Transformer layers, denoted as layer 0 and layer 1, and the request is a sequence of 6 tokens {a, b, c, d, e, f}. After prefix matching, the prefix part is a sequence of 5 tokens {a, b, c, d, e}, and token f is the current query.

[0043] Before execution begins at layer 0, the important prefix KV recognition module obtains the important token set S_0={a, b, d} for layer 0. At this time, the prefetch buffer is empty, so the KV of a, b, and d are directly loaded into the GPU memory. At the same time, the similarity-aware speculative prefetch module predicts the candidate important token set S'_1=S_0 for layer 1 based on S_0, which is {a, b, d}.

[0044] The calculation begins at layer 0. Simultaneously, the similarity-aware speculative prefetching module asynchronously prefetches the candidate important key-value pairs (KV) of layer 1 corresponding to S'_1. Since the prefetching is truncated when the calculation of layer 0 ends, only the KV of two candidate important tokens, {a, b}, has been prefetched at this point and stored in the prefetch buffer. d in S'_1 has not yet been retrieved.

[0045] Before the first-level computation begins, the important prefix key-value identification module obtains the true important token set S_1 = {a, c, d} for the first level. The verification and recovery module compares S_1 with {a, b} in the prefetch buffer: a matches and can be used directly; b does not belong to S_1 and is a false prefetch; c and d belong to S_1 but are not in the buffer and are missing. Therefore, the verification and recovery module synchronously loads the key-value pairs of the missing tokens {c, d} and discards the key-value pairs of the falsely prefetched token b, obtaining all the important key-value pairs required for the first-level execution, and then begins the first-level computation.

[0046] To further illustrate the effects of the present invention, in one embodiment, the technical solution of the present invention is implemented on FlexGen and tested on three different scales of open source OPT models (OPT-6.7B, OPT-13B, and OPT-30B). The experiment was conducted on a server equipped with two Intel(R) Xeon(R) Gold 5318Y CPUs @ 2.10GHz, 512GB CPU DRAM, an NVIDIA A100 GPU with 80GB HBM video memory, and a 2TB Intel SSD with a read throughput of approximately 5GB / s. The GPU and CPU are connected via a PCIe 4.0 x16 interface.

[0047] The tests used the RTE dataset, appending two to ten few-sample examples as system hints before each query. These system hints were shared across different queries, and their reuse frequency followed a normal distribution. The prefix of the OPT-30B model was extended to a maximum length of 4K, while the remaining OPT models were extended to 10K. To prevent runtime memory overflow and ensure that only a portion of the prefix key-value pairs were cached (the remaining key-value pairs were stored on SSD), 10GB of GPU cache and 32GB of CPU cache were allocated to the prefix key-value pairs. The remaining GPU and CPU memory were used to store model weights, key-value caches during the decoding phase, and input data; each block contained keys or values ​​from 64 tokens.

[0048] Table 1 compares the average TTFT per request before and after using this invention under different models. "Before using this invention" refers to the case where, under otherwise identical configuration conditions, layer-aware speculative prefetching (i.e., loading of the next layer's important key values ​​is performed serially with the current layer's computation) is not enabled. It can be seen that using this invention speeds up TTFT by up to approximately 1.35 times.

[0049] Table 1: Comparison of TTFT time (seconds) for large model inference under different models It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

Claims

1. A similarity-aware key-value speculative prefetching method, characterized in that, Includes the following steps: Step 1: Receive the current request, match the reusable prefix corresponding to the current request in the prefix index structure, and obtain the KV metadata and storage location of the reusable prefix in each Transformer layer; Step 2: For the i-th layer in the Transformer layer, obtain the set of important prefix token indices S_{i} corresponding to the current request in that layer, and load the important key-value pairs required for the i-th layer to calculate; Step 3: After the calculation starts at the i-th layer, generate the candidate important prefix token index set S'{i+1} for the (i+1)-th layer based on the important prefix token index set S_{i} obtained in Step 2, and asynchronously prefetch the candidate important key-value pairs corresponding to S'_{i+1} for the (i+1)-th layer during the calculation of the i-th layer. Step 4: When the calculation of the i-th layer is completed, stop initiating new candidate important KV prefetch requests for the (i+1)-th layer. Step 5: Before the calculation of layer i+1 begins, obtain the actual important prefix token index set S_{i+1} of layer i+1. Compare and verify S_{i+1} with the pre-fetched candidate important KVs. For candidate important KVs belonging to S_{i+1}, use them directly as important KVs; for candidate important KVs not belonging to S_{i+1}, discard them directly; for important KVs belonging to S_{i+1} but not pre-fetched, perform synchronous padding and loading. Step six: Use the verified important KV cache to perform the calculation of the (i+1)th layer, and increment the layer index. Repeat steps three to five until the pre-filling phase ends or the last Transformer layer is reached.

2. The method according to claim 1, characterized in that, The key-value metadata includes layer index, token index, attention header index, cache block index, and the storage layer where the corresponding key-value data resides.

3. The method according to claim 1, characterized in that, In step two, the important prefix token index set S_{i} is obtained in the following way: for each attention head in the i-th layer, the attention weight matrix between the query vector of the attention head and the reusable prefix key vector is calculated, and the attention weight matrix is ​​summed column by column to obtain the importance score of each prefix token under the attention head; The importance scores of each attention head are summed according to the prefix tokens to obtain the summation importance score of each prefix token shared by all attention heads; the prefix tokens are sorted from high to low according to the summation importance score, and a preset proportion or a preset number of prefix tokens are selected to form the important token index set S_{i} of the i-th layer.

4. The method according to claim 1, characterized in that, In step three, generating the candidate important prefix token index set S'{i+1} for the (i+1)th layer includes: sorting S_{i} according to the importance score of the token, the historical prefetch hit rate, the prefetch buffer size, or the current I / O bandwidth, and then truncating or adjusting S_{i} to generate S'_{i+1}.

5. The method according to claim 4, characterized in that, In step three, the asynchronous prefetching of candidate important key-value pairs corresponding to S'_{i+1} at the (i+1)th layer during the computation of the i-th layer includes: prefetching candidate important key-value pairs in descending order of importance score; wherein, if the candidate important key-value pair is located in GPU memory, its address is directly recorded or written to the prefetch buffer; if it is located in CPU memory, it is transferred to GPU memory; if it is located in external storage or disk, it is first read into CPU memory and then transferred to GPU memory.

6. A similarity-aware key-value speculative prefetching system, characterized in that, include: The important prefix KV identification module is used to match the reusable prefix corresponding to the current request in the prefix index structure, and obtain the KV metadata and storage location of the reusable prefix in each Transformer layer; for the i-th layer in the Transformer layer, obtain the important prefix token index set S_{i} corresponding to the current request in that layer, and load the important KV required for the i-th layer to calculate; The similarity-aware speculative prefetching module is configured to generate a candidate important prefix token index set S'{i+1} for the (i+1)th layer based on the important prefix token index set S_{i} after the calculation starts at the i-th layer, and asynchronously prefetch the candidate important key-value pairs corresponding to S'_{i+1} for the (i+1)th layer during the calculation of the i-th layer. Continuously monitor the GPU status, and stop initiating new candidate important KV prefetch requests for the (i+1)th layer when the GPU has completed the calculation of the i-th layer. The verification and recovery module is configured to obtain the actual important prefix token index set S_{i+1} of the i+1 layer before the calculation of the i+1 layer begins, and compare S_{i+1} with the pre-fetched candidate important KV. Among them, the candidate important KV belonging to S_{i+1} is used directly as important KV; the candidate important KV not belonging to S_{i+1} is discarded; and the important KV belonging to S_{i+1} but not pre-fetched is synchronously filled and loaded. The iterative execution module is configured to trigger the calculation of the i+1th layer using the verified important KV cache after the verification and recovery module completes the comparison and verification of the candidate important KV of the i+1th layer. After the calculation of the i+1th layer is completed, the layer index is incremented, and the similarity-aware speculative prefetching module and the verification and recovery module are called again until the prefilling stage ends or the last Transformer layer is reached.

7. The system according to claim 6, characterized in that, The important prefix key-value recognition module includes: for each attention head in the i-th layer, calculating the attention weight matrix between the query vector of the attention head and the reusable prefix key vector, and summing the attention weight matrix column by column to obtain the importance score of each prefix token under the attention head; adding the importance scores of each attention head according to the corresponding prefix tokens to obtain the aggregate importance score of each prefix token shared by all attention heads; sorting the aggregate importance scores from high to low, selecting a preset proportion or a preset number of prefix tokens to form the important token index set S_{i} of the i-th layer.

8. The system according to claim 6, characterized in that, The similarity-aware speculative prefetching module includes: truncating or adjusting S_{i} to generate S'_{i+1} based on the importance score of the token, the historical prefetch hit rate, the size of the prefetch buffer, or the current I / O bandwidth.

9. The system according to claim 8, characterized in that, The similarity-aware speculative prefetching module further includes: prefetching candidate important key-value pairs in descending order of importance score; wherein, if the candidate important key-value pair is located in GPU memory, its address is directly recorded or written to the prefetch buffer; if it is located in CPU memory, it is transferred to GPU memory; if it is located in external storage or disk, it is first read into CPU memory and then transferred to GPU memory.

Images