Large model KV cache fast storage and import acceleration system

Abstract

This application discloses a large model KV cache fast storage and import acceleration system, which involves the field of FPGA. The system includes a CPU, an accelerator, an FPGA, a memory, an external memory, and a memory; in the query phase, the CPU determines the data query task based on the input word element and controls the FPGA to start the data query task; the FPGA queries and imports the model cache data from the external memory into the memory; in the inference phase, the accelerator performs inference based on the model cache data and outputs the inference result; the FPGA reads the updated model cache data from the memory according to the control instruction of the CPU and exports it to the external memory. This solution uses KV Cache technology to reduce the number of repetitions of large model inference, and uses hash calculation, table lookup, and query access technology to make data import and export faster, further accelerating the speed of result output. It not only broadens the application scenarios, but also facilitates data backup and transfer.

Description

Technical Field

[0001] This application relates to the field of FPGA, and in particular to a fast storage and import acceleration system for large model KV cache. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have become stars in the field of natural language processing. However, while these models bring powerful capabilities, they also come with huge computational demands and inference latency.

[0003] In the inference process of LLM, the model generates each token of the text one by one. This process can be slow because the model can only generate one token at a time, and each new prediction depends on the previous context. This means that to predict the 1000th token, you need information from the first 999 tokens, which usually involves a series of matrix multiplications on the representations of these tokens. To predict the 1001st token, you need not only information from the first 999 tokens but also information from the 1000th token. Each time a new token is generated, the attention weights of all previous tokens usually need to be recalculated, resulting in a lot of redundant computation. The data generated step by step in the inference process and stored in memory raises two issues: 1. The data in memory is not permanently stored; it is erased when a session ends or the device loses power; 2. The session cannot be interrupted. Once interrupted, the session can only start inference again from the first token and cannot continue from the previous session. Summary of the Invention

[0004] This application provides a fast storage and import acceleration system for large model KV cache, which solves the problems of repeated calculations, data flushing, and session interruption affecting the inference speed of traditional large model inference.

[0005] On the one hand, this application discloses a system for fast storage and import acceleration of large-scale KV cache:

[0006] The system includes a CPU for data processing and task scheduling, an accelerator for running large models, an FPGA for executing KV cache and data querying, a memory for providing data exchange between the FPGA and the accelerator, an external memory for storing model cache data, and memory for storing conversational terms and linked lists; the linked lists record the mapping addresses of the model cache data.

[0007] During the query phase, the CPU determines the data query task based on the input tokens generated in the current session and controls the FPGA to start the data query task; the FPGA queries and imports model cache data from external memory into memory based on the input tokens, and loads the input tokens of the current session into memory;

[0008] During the inference phase, the accelerator running the large model performs model inference by extracting model cache data from the memory, outputs the inference results, and exports new model cache data; the FPGA reads the updated model cache data from the memory according to the control instructions of the CPU and exports it to the external memory.

[0009] Specifically, the CPU determines the session state based on the session number to which the current input word belongs;

[0010] When the session number indicates that the input word is the first word of the current session, or the cascaded input of the previous historical word in the current session, the CPU controls the accelerator to perform the model inference task and controls the FPGA to store the current input word.

[0011] When the session number indicates that the input term is not the first term of the current session, or is not a cascaded input of the previous historical term in the current session, the CPU allocates and controls the FPGA to perform the data query task.

[0012] Specifically, when the FPGA starts the data query task, it reads the first n-1 calculated words generated by the current session from memory based on the nth word of the current session, performs hash calculation on the first n-1 calculated words, and queries the linked list and matching model cache data according to the hash value.

[0013] Specifically, an address mapping table is also stored in memory, which stores hash values ​​and corresponding hash addresses; the linked list stores the mapping relationship between hash addresses and the physical addresses of the storage model cache data.

[0014] When searching the linked list, the target linked list is matched based on the mapping and the determined target hash address, and the physical address of the target model cache data located in external memory is determined.

[0015] Specifically, the linked list corresponds one-to-one with the hash value, and each linked list contains several sets of linked rings. Each set of linked rings stores the number of calculated tokens in the current session, the hash value of the calculated tokens, the physical address of the model cache data, the pointer to the previous linked ring, and the pointer to the next linked ring.

[0016] When searching the linked list, the target linked ring is determined from the target linked list based on the number of computed tokens in the current session and the hash value of the computed tokens, and the physical address of the target model cache data is extracted.

[0017] Specifically, after determining the physical address of the target model cache data, the target model cache data is located and matched from external memory. When the target model cache data is successfully matched from external memory, it is imported into the large model for inference. When the target model cache data is not matched from external memory, all words of the current session are extracted from memory and sent to the accelerator for model inference.

[0018] Specifically, when the FPGA exports the updated model cache data according to the control instructions of the CPU, it performs a hash calculation based on all the calculated tokens of the current session, updates the old linked list according to the hash value, and stores the physical address of the updated model cache data into the newly inserted linked list.

[0019] Specifically, the preceding link pointer and the following link pointer are the addresses of adjacent links in the linked list, respectively. When updating the linked list, links are deleted, traversed, and added based on the preceding link pointer and the following link pointer.

[0020] Specifically, the model cache data is stored in external memory in units of session tasks. Each session task stores the calculated token value, K value, and V value of the current session. The K value is stored hierarchically according to the token value order, and the V value is stored hierarchically and column-wise according to the token value order.

[0021] Specifically, a first address FIFO and a second address FIFO are also provided; the first address FIFO stores the addresses of free areas in the external memory that have not stored data within a preset time, and the second address FIFO stores the blank addresses after deleting historical model cache data within a preset time.

[0022] When exporting model cache data, the target export address is determined based on the address margins stored in the first address FIFO and the second address FIFO.

[0023] The beneficial effects of the technical solutions provided in this application include at least the following:

[0024] 1. Reduced computational load and increased reasoning speed:

[0025] KV Cache technology reduces the number of redundant calculations during large model inference, thus improving inference speed. The storage + inference acceleration approach only infers the last set of tokens each time it's used, while the results for previous tokens are directly read from external storage. This represents a significant speed improvement compared to the traditional method of inferring for every single token. Furthermore, this solution uses FPGA to accelerate hash calculations, table lookups, and memory access, enabling faster data import and export, further accelerating result output.

[0026] 2. Expanded application scenarios

[0027] The use of KV Cache for storage and import enables large models to not only engage in continuous dialogue but also to perform multiple dialogues alternately without interference. Furthermore, dialogues can be interrupted at any time, saved, and resumed at an appropriate time. Moreover, when transferring a dialogue to another device, there's no need to start reasoning from scratch; the previous dialogue can be resumed directly.

[0028] 3. Facilitates data backup and transfer

[0029] KV Cache's storage and import system was designed with the data storage structure in mind, so no additional operations are needed for data backup. Furthermore, when data is transferred to other devices for use, it can be used directly as long as the configuration parameters are the same as the current ones. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the large model KV cache fast storage and import system provided in the embodiments of this application;

[0031] Figure 2 The diagram illustrates the process of rapid import and export based on input words during system operation.

[0032] Figure 3 This is a flowchart illustrating the process of querying and importing cached data from the target model.

[0033] Figure 4 This is a schematic diagram of the address mapping table and the linked list;

[0034] Figure 5 This is a diagram illustrating the updating of the linked list;

[0035] Figure 6 A publicly available diagram illustrating the data storage structure in an SSD;

[0036] Figure 7 A schematic diagram illustrating the principle of determining the target's derived address based on FIFO has been published. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0038] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0039] Figure 1 This is a schematic diagram of a large-model KV cache fast storage and import system provided in an embodiment of this application. The system as a whole includes a CPU, an accelerator, and an FPGA. A memory is provided between the FPGA and the accelerator, connecting the FPGA's external memory and main memory. In this application, the accelerator can be a GPU and / or a DPU, the memory is usually a Memory memory, the external memory is usually a large-capacity storage device such as an SSD, and the main memory is a RAM memory.

[0040] The CPU primarily handles data processing and preprocessing, memory allocation, and task scheduling. The preprocessing stage of large models is performed on the CPU, including operations such as data vectorization and data embedding. After preprocessing, the CPU schedules whether the data is sent to the GPU or FPGA. Furthermore, the CPU typically manages system memory allocation and the location of model cache data within memory.

[0041] Accelerators (GPUs / DPUs) primarily handle the computation of large models. LLM runs on GPUs, whose main characteristics are high concurrency and high bandwidth. DPUs provide a large number of data computation units. The core of large models lies in matrix multiplication and addition, and both GPUs and DPUs can accelerate the computation process. This solution does not focus on the specific steps of large model computation but rather embeds it as a processing component within the overall system.

[0042] FPGAs primarily handle data querying and control tasks, including loop hash calculations, linked list lookups, fast SSD access, and importing / exporting model cache data. Hash values ​​can be calculated using hardware algorithms on FPGAs, resulting in high computation speed. Furthermore, FPGAs offer significant advantages over CPUs in table lookup operations. During SSD access, FPGAs can directly generate access instructions without requiring processing by the software and kernel layers, thus improving access efficiency.

[0043] The memory, either DDR or HBM, primarily provides data storage and exchange functions. DDR features high bandwidth and large capacity. High bandwidth ensures efficient data transfer between different devices. HBM has a higher bit width than DDR, enabling faster data read and write operations. In the design, large model calculations using KV Cache technology initially store the model data in memory.

[0044] SSDs provide high-capacity storage media for storing model cache data. SSDs offer advantages such as large capacity, fast storage speed, and data retention even after power failure. This satisfies the requirements for fast read / write operations in KV cache while ensuring data security. Furthermore, their large storage capacity allows for the storage of a sufficient amount of model cache data.

[0045] Memory is the storage unit in the system that stores data and instructions, mainly including RAM and ROM. This application mainly uses memory to implement a linked list for fast access and storage. The linked list records the mapping address of the model cache data, which makes it convenient for the FPGA to quickly import the model cache data according to the address, as well as store the words generated by each session task.

[0046] Based on the system described above, during the query phase, the CPU can determine the data query task based on the input tokens generated in the current session and control the FPGA to initiate the data query task. The FPGA then queries and imports model cache data from external memory into memory based on the input tokens, and loads the input tokens of the current session into memory. Upon entering the inference phase, the accelerator running the large model performs model inference by retrieving model cache data from memory, outputting the inference results and exporting new model cache data.

[0047] In some embodiments, after the large model is output, further execution is required, namely, the model cache data needs to be saved. At this time, the CPU needs to schedule and allocate tasks, control the FPGA to read the updated model cache data from the memory, and export it to the SSD.

[0048] Figure 2 The diagram illustrates the process of rapid import and export based on input tokens during system operation. When a user inputs a token, the CPU first needs to determine whether the KV Cache mechanism needs to be activated, i.e., whether model cache data needs to be imported. The CPU determines the session state based on the session number to which the current input token belongs. Determining the session state is to determine whether the KV Cache mechanism needs to be activated. The rapid import and export in this application mainly focuses on the scenario when the KV Cache mechanism is activated.

[0049] KV Cache is a caching mechanism that improves inference speed by storing key-value pairs and model parameters to reduce redundant calculations. Specifically, it caches the key and value of previous tokens, allowing new token generation to simply read these parameters from the cache without recalculation. This process continues for subsequent tokens, meaning you only need to import the cached parameters after each inference iteration and perform model inference based on these parameters and the new input token. There are several situations where re-importing is unnecessary:

[0050] 1. This token is the first token of the session; no previous tokens were entered.

[0051] 2. Within the same session, this token is the subsequent input of the previous token.

[0052] When the session number indicates that the input token (token n) is the first token of the current session, or a cascaded input of a previous historical token in the current session, the CPU can directly control the accelerator to perform model inference tasks and control the FPGA to store the current input token (token n). When the session number indicates that the input token is not the first token of the current session, or is not a cascaded input of a previous historical token in the current session, the CPU allocates and controls the FPGA to perform data query tasks.

[0053] For scenarios where a KV cache mechanism is not required, token n can be directly input into the LLM for calculation on the GPU / DPU side. For scenarios requiring a KV cache, the 0th to (n-1)th token of the current session needs to be re-entered. The input token is used to determine the storage location of the target model cache data on the SSD. After the query is complete, the retrieved target model cache data is imported into the LLM, at which point the input token n can be calculated using the LLM. Optionally, after the LLM calculation is complete, the result is returned to the user, and the model cache data can be exported to the SSD as needed.

[0054] The following details the FPGA's role in executing data query tasks and import / export phases.

[0055] Figure 3 This is a flowchart illustrating the process of querying and importing target model cache data. The specific steps performed by the FPGA include the following:

[0056] S1. When the FPGA starts the data query task, it reads all the calculated words generated in the current session from memory based on the input words of the current session.

[0057] S2. Perform hash calculation on all computed tokens in the current session;

[0058] S3. Query the linked list based on the hash value of the calculated token;

[0059] S4. Access the SSD based on the linked list query results and perform data matching;

[0060] S5. When the data match is successful, import the target model cache data into Memory and send it into LLM for inference.

[0061] S6. If data matching fails, import all terms from the current session.

[0062] In this application, token n represents the currently input uncomputed token, and the first n-1 tokens are the computed tokens. This step uses the hash calculation result to distinguish different sessions. The advantage of using hash is its simplicity and ability to distribute different sessions relatively evenly. The hash function used here needs to be able to iterate through different numbers of tokens. After the token input is complete, the linked list is queried based on the final hash value. The linked list stores the mapping relationship between hash addresses and physical addresses in the database. By looking up the table, we can quickly locate and retrieve the model cache data stored in the SSD. When the model cache data for this session exists in the SSD, it is retrieved and imported into the LLM; if the model cache data for this session cannot be found in the SSD, the tokens for this session, starting from 0 and going up to token n, are sequentially input into the accelerator's LLM.

[0063] Specifically, in the S3 process, the FPGA mainly implements this through linked lists and address mapping tables in memory. Because there is a one-to-one correspondence between hash values ​​and linked lists, the number of linked lists corresponds to the number of hashes. Therefore, it is necessary to first determine the address of the target linked list corresponding to the hash value being calculated before the linked list can be searched.

[0064] Figure 4 This is a diagram illustrating the mapping between an address mapping table and a linked list. The address mapping table stores hash values ​​and their corresponding hash addresses, and this hash address is the head address of the linked list. For example... Figure 4 In the middle, hash 1 corresponds to the hash address list 1addr, and list 1addr maps to the first linked list.

[0065] For linked lists, each linked list contains several sets of linked loops. Each linked loop stores the number of computed tokens for the current session (token_num), the hash value of the computed tokens (last_hash), the physical address of the model cache data (data_addr), the pointer to the previous linked loop (last_ptr), and the pointer to the next linked loop (next_ptr). Figure 4 The linked lists of hash 1 and hash 2 both show two cascaded links, and null indicates the end of the linked list.

[0066] When searching the linked list, the FPGA determines the target loop from the target linked list based on the number of computed tokens in the current session (assuming the current execution is tokenn, this number is n-1) and the hash value of the computed tokens, and extracts the physical address data_addr of the target model cache data. By using the number of hash values ​​of the first n-1 computed tokens and the total number of computed tokens, most erroneous sessions can be quickly filtered out. A portion of erroneous sessions will be judged after the data is read from the SSD.

[0067] In some other embodiments, for scenarios where LLM inference post-inference model cache data needs to be exported (mainly for inference scenarios where tokens continue to be input), the CPU will also allocate and schedule tasks to control the CPU / GPU to output the updated model cache data to memory, and then the FPGA will export it to the SSD. For this process, the FPGA will perform a hash calculation based on all computed tokens (the first n tokens) in the current session, update the old linked list according to the hash value, and store the physical address of the updated model cache data into the newly inserted linked list.

[0068] Figure 5 This is a diagram illustrating the updating of the linked list. During storage, the hash values ​​of all tokens need to be recalculated. However, since the hashes of the first n-1 tokens have already been calculated during the linked list lookup, they can be used directly. That is, the input token n and the hash values ​​of the first n-1 tokens are used to continue the calculation. Once the hash value is calculated, the corresponding link in the old hash linked list is deleted, and a new link is added to the end of the corresponding linked list based on the new hash. The physical address of the updated model cache data is stored in the newly inserted link. This step is implemented using `last_ptr` and `next_ptr`, because the pointers to the previous and next links are the addresses of adjacent links in the linked list, respectively. When updating the linked list, deletion, traversal, and addition of links can be achieved based on these pointers.

[0069] In some embodiments, both exporting to SSD and importing to memory require reading or storing model cache data in the SSD. Related technologies typically use direct KV pair storage for this data, but this doesn't consider the fact that the (Key) K and (Value) V values ​​are matrix data calculated based on historical input tokens. The K matrix itself needs to be transposed via the FPGA during execution, while the V matrix does not. Therefore, related technologies store the data directly in bit order. For software-executed model calculations, after extracting K and V values ​​from the SSD using a CPU algorithm, the K value data needs to be transposed, and the V value data needs to be restored in matrix form before subsequent operations can be performed. For hardware (such as FPGA), after extracting K and V values ​​from the SSD, the K value data needs to be transposed, and the V value data also needs to be restored in matrix form. This leads to layout issues during subsequent import into memory, causing latency in the input LLM. This application makes the following improvements to address this problem:

[0070] Store model cache data on SSDs in units of session tasks, see [link / reference]. Figure 6 The publicly available diagram illustrates the data storage structure in an SSD. Each session task stores the computed token value, the K value, and the V value for the current session. The storage of K and V values ​​is hierarchical, determined by the principles of the larger model. Figure 6 The K and V values ​​are stored in the order of layer 0, layer 1, layer 2, and so on. The K values ​​are further stored hierarchically according to the historical token values; for example, K[token 0] represents the K value calculated with token 0, K[token 1] represents the K value calculated with token 1, and so on. The V values ​​are stored differently from the K values. For ease of calculation, V values ​​are stored column-wise; that is, V values ​​are hierarchically layered and stored column-wise according to the token values. For example... Figure 6The diagram shows how the V values ​​of the first column (clumn 0) of all layers are stored together, followed by the second column (clumn 1), the third column (clumn 2), and so on. Within each column, V values ​​are stored in token order. For example, column 0 is stored as V[clumn 0 token 0], V[clumn 0 token 1], V[clumn 0 token 2], etc., representing the V value calculated with token 0, token 1, and token 2 respectively. The tokens, K, and V values ​​are not stored in a tightly packed arrangement; space is reserved based on the largest token value. The advantage of this approach is that the matrix elements are directly extracted in their sequential form during FPGA import, saving data layout time and reducing latency in data transmission to the LLM. For the storage process, since the data is exported after calculation, storing it in this specific format into the SSD before subsequent token input does not affect the LLM's operation, thus eliminating storage latency issues.

[0071] In some embodiments, SSD storage follows a sequential incrementing physical address sequence. When data is deleted, that physical address becomes free. If the amount of data is small, these freed addresses can be ignored. However, when the data volume is large, the free space must be fully utilized. Therefore, when generating SSD addresses, the deleted addresses need to be added back in. To this end, this application additionally provides an integrated FIFO memory for SSD storage addresses.

[0072] Figure 7 A schematic diagram illustrating the principle of determining the target export address based on FIFOs is disclosed. This integrated FIFO memory includes at least a first address FIFO (inc_fifo) and a second address FIFO (del_fifo). These two FIFOs can be controlled by CPU or FPGA instructions. inc_fifo stores the addresses of free areas in external memory that have not stored data within a preset time period, and del_fifo stores the blank addresses after deleting historical model cache data within the preset time period. When exporting model cache data, the target export address is determined based on the address space stored in the first and second address FIFOs. Optionally, a third address FIFO (addr_fifo) can be integrated to temporarily store the target export address. addr_fifo periodically outputs this address to the FPGA, which then executes the export steps.

[0073] In summary, the fast storage and import system designed using the advantages of KV Cache technology in this application can achieve the following technical effects:

[0074] 1. Reduced computational load and increased reasoning speed:

[0075] KV Cache technology reduces the number of redundant calculations during large model inference, thus improving inference speed. The storage + inference acceleration approach only infers the last set of tokens each time it's used, while the results for previous tokens are directly read from external storage. This represents a significant speed improvement compared to the traditional method of inferring for every single token. Furthermore, this solution uses FPGA to accelerate hash calculations, table lookups, and memory access, enabling faster data import and export, further accelerating result output.

[0076] 2. Expanded application scenarios

[0077] The use of KV Cache for storage and import enables large models to not only engage in continuous dialogue but also to perform multiple dialogues alternately without interference. Furthermore, dialogues can be interrupted at any time, saved, and resumed at an appropriate time. Moreover, when transferring a dialogue to another device, there's no need to start reasoning from scratch; the previous dialogue can be resumed directly.

[0078] 3. Facilitates data backup and transfer

[0079] KV Cache's storage and import system was designed with the data storage structure in mind, so no additional operations are needed for data backup. Furthermore, when data is transferred to other devices for use, it can be used directly as long as the configuration parameters are the same as the current ones.

[0080] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A system for fast storage and import of large-scale KV cache models, characterized in that, The system includes a CPU for data processing and task scheduling, an accelerator for running large models, an FPGA for executing KV cache and data querying, a memory for providing data exchange between the FPGA and the accelerator, an external memory for storing model cache data, and memory for storing conversational terms and linked lists; the linked lists record the mapping addresses of the model cache data. During the query phase, the CPU determines the data query task based on the input tokens generated in the current session and controls the FPGA to start the data query task; the FPGA queries and imports model cache data from external memory into memory based on the input tokens, and also imports the input tokens of the current session into memory; the linked list contains several sets of linked loops, and each set of linked loops stores the number of calculated tokens in the current session, the hash value of the calculated tokens, and the physical address of the model cache data; When searching the linked list, the target linked list is determined from the target linked list based on the number of computed tokens in the current session and the hash value of the computed tokens, and the physical address of the target model cache data is extracted. During the inference phase, the accelerator running the large model performs model inference by extracting model cache data from the memory, outputs the inference results, and exports new model cache data; the FPGA reads the updated model cache data from the memory according to the control instructions of the CPU and exports it to the external memory.

2. The large model KV cache fast storage and import acceleration system according to claim 1, characterized in that, The CPU determines the session state based on the session number to which the current input word belongs; When the session number indicates that the input word is the first word of the current session, or the cascaded input of the previous historical word in the current session, the CPU controls the accelerator to perform the model inference task and controls the FPGA to store the current input word. When the session number indicates that the input term is not the first term of the current session, or is not a cascaded input of the previous historical term in the current session, the CPU allocates and controls the FPGA to perform the data query task.

3. The large model KV cache fast storage and import acceleration system according to claim 1, characterized in that, When the FPGA starts the data query task, it reads the first n-1 calculated words generated by the current session from memory based on the nth word of the current session, performs hash calculation on the first n-1 calculated words, and queries the linked list and matching model cache data according to the hash value.

4. The large model KV cache fast storage and import acceleration system according to claim 3, characterized in that, An address mapping table is also stored in memory, which stores hash values ​​and corresponding hash addresses; the linked list stores the mapping relationship between hash addresses and the physical addresses of the storage model cache data; When searching the linked list, the target linked list is matched based on the mapping and the determined target hash address, and the physical address of the target model cache data located in external memory is determined.

5. The large model KV cache fast storage and import acceleration system according to claim 1, characterized in that, After determining the physical address of the target model cache data, the target model cache data is located and matched from external memory. When the target model cache data is successfully matched from external memory, it is imported into the large model for inference. When the target model cache data is not matched from external memory, all words of the current session are extracted from memory and sent to the accelerator for model inference.

6. The large model KV cache fast storage and import acceleration system according to claim 1, characterized in that, When the FPGA exports the updated model cache data according to the control instructions of the CPU, it performs a hash calculation based on all the computed tokens in the current session, updates the old linked list according to the hash value, and stores the physical address of the updated model cache data into the newly inserted linked list.

7. The large model KV cache fast storage and import acceleration system according to claim 6, characterized in that, The linked list also stores pointers to the preceding and following linked lists of the current session. The preceding and following linked list pointers are the addresses of adjacent linked lists in the linked list. When updating the linked list, linked lists are deleted, traversed, and added based on the preceding and following linked list pointers.

8. The large model KV cache fast storage and import acceleration system according to any one of claims 1-7, characterized in that, The model cache data is stored in external memory in units of session tasks. Each session task stores the calculated token value, the K value, and the V value of the current session. The K value is stored hierarchically according to the token value order; the V value is stored hierarchically and column-wise according to the token value order.

9. The large model KV cache fast storage and import acceleration system according to claim 8, characterized in that, It also includes a first address FIFO and a second address FIFO; the first address FIFO stores the addresses of free areas in the external memory that have not stored data within a preset time, and the second address FIFO stores the blank addresses after the historical model cache data has been deleted within the preset time. When exporting model cache data, the target export address is determined based on the address margins stored in the first address FIFO and the second address FIFO.

Images