Large model KV cache rapid storage and import acceleration architecture

Abstract

The invention discloses a large model KV cache rapid storage and import acceleration architecture, and relates to the field of FPGA. The architecture comprises a CPU, an accelerator, an FPGA, a memory, an external memory and a memory. In the query stage, the CPU determines a data query task according to the input lexical elements and controls the FPGA to start the data query task; the FPGA queries and imports model cache data into the memory according to the external memory; in the reasoning stage, the accelerator conducts reasoning and outputs a reasoning result based on the model cache data; and the FPGA reads the updated model cache data from the memory according to a control instruction of the CPU, and exports the updated model cache data to an external memory. According to the scheme, the KV Cache technology is used for reducing the number of times of reasoning repetition of a large model, the hash calculation, table lookup and query access technologies are used for enabling data to be more rapidly imported and exported, and the result output speed is further increased. Not only is the application scene widened, but also data backup and transfer are facilitated.

Description

Technical Field

[0001] The present application relates to the field of FPGA, and in particular to a large model KV cache fast storage and import acceleration architecture. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have become the stars in the field of natural language processing. However, while these models bring powerful capabilities, they are also accompanied by huge computational workloads and inference delays.

[0003] During the reasoning process of LLM, the model generates each word (token) of the text one by one. This process may be slow because the model can only generate one token at a time, and each new prediction depends on the previous context. This means that to predict the 1000th token, you need to use the information of the previous 999 tokens, which usually involves a series of matrix multiplication operations on the representations of these tokens. To predict the 1001th token, not only the information of the previous 999 tokens is needed, but also the information of the 1000th token. Every time a new token is generated, it is usually necessary to recalculate the attention weights of all previous tokens, which causes a lot of redundant calculations. The data generated step by step by the previous reasoning process and saved in memory creates two problems: 1. The data in the memory is not stored for a long time. When a session ends or the device loses power, the data in the memory will be flushed; 2. The session cannot be interrupted. Once interrupted, the session can only be re-reasoned from the first token and cannot continue the last session. Summary of the invention

[0004] The present application provides a large model KV cache fast storage and import acceleration architecture to solve the problem of repeated calculation of model data reasoning, data flushing and session interruption affecting the reasoning speed of traditional large models.

[0005] On the one hand, the present application discloses a large model KV cache fast storage and import acceleration architecture:

[0006] The architecture includes a CPU for data processing and task scheduling, an accelerator for running large models, an FPGA for executing KV Cache and data query, a memory for providing data exchange between the FPGA and the accelerator, an external memory for storing model cache data, and a memory for storing session words and linked lists; the linked lists record mapping addresses of model cache data;

[0007] In the query phase, the CPU determines the data query task based on the input word generated by the current session, and controls the FPGA to start the data query task; the FPGA queries and imports the model cache data from the external memory into the memory based on the input word, and stores the input word of the current session into the memory;

[0008] During the inference phase, the accelerator running the large model performs model inference by extracting the model cache data in the memory, outputs the inference results and exports the new model cache data; the FPGA reads the updated model cache data from the memory according to the control instructions of the CPU and exports it to the external memory.

[0009] Specifically, the CPU determines the session state based on the session number to which the current input word belongs;

[0010] When the session number indicates that the input word is the first word of the current session, or is a cascade input of a previous historical word in the current session, the CPU controls the accelerator to perform the model reasoning task, and controls the FPGA to store the current input word;

[0011] When the session number indicates that the input word is not the first word of the current session, or is not the cascade input of the previous historical word in the current session, the CPU allocates and controls the FPGA to execute the data query task.

[0012] Specifically, when the FPGA starts a data query task, the first n-1 calculated word elements generated by the current session are read from the memory based on the nth word element of the current session, and the first n-1 calculated word elements and hash calculations are performed, and the linked list and matching model cache data are queried according to the hash value.

[0013] Specifically, an address mapping table is also stored in the memory, wherein the address mapping table stores hash values ​​and one-to-one corresponding hash addresses; the linked list stores a mapping relationship between hash addresses and physical addresses storing model cache data;

[0014] When searching the linked list, the target linked list is matched according to the mapping and the determined target hash address and the physical address of the target model cache data located in the external memory is determined.

[0015] Specifically, the linked lists correspond to hash values ​​one by one, and each linked list contains several groups of links, each group of links stores the number of calculated word units of the current session, the hash value of the calculated word unit, the physical address of the model cache data, the previous link pointer and the next link pointer;

[0016] When searching the linked list, the target link is determined from the target linked list according to the number of calculated tokens of the current session and the hash value of the calculated token, and the physical address of the target model cache data is extracted.

[0017] Specifically, after determining the physical address of the target model cache data, locate and match the target model cache data from the external memory; when the target model cache data is successfully matched from the external memory, import it into the large model for reasoning; when the target model cache data is not matched from the external memory, extract all word elements of the current session from the memory and send them to the accelerator for model reasoning.

[0018] Specifically, when the FPGA exports the updated model cache data according to the control instruction of the CPU, it performs hash calculation based on all calculated words in the current session, updates the old linked list according to the hash value, and stores the physical address of the updated model cache data into the newly inserted link.

[0019] Specifically, the front-stage pointer and the back-stage pointer are addresses of adjacent links in the linked list, respectively. When updating the linked list, the links are deleted, traversed and added based on the front-stage pointer and the back-stage pointer.

[0020] Specifically, the model cache data is stored in an external memory in units of session tasks, and each session task corresponds to storing the calculated word unit token value of the current session, the K value and the V value of the current session, wherein the K value is stored in layers according to the order of the token value; the V value is stored in layers according to the order of the token value and in columns.

[0021] Specifically, a first address FIFO and a second address FIFO are also provided; the first address FIFO stores the address of the idle area of ​​the external memory that has not stored data within a preset time, and the second address FIFO stores the blank address after deleting the historical model cache data within the preset time;

[0022] When exporting the model cache data, a target export address is determined according to the address margins stored in the first address FIFO and the second address FIFO.

[0023] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0024] 1. Reduced computational effort and improved reasoning speed:

[0025] Executing large model reasoning through KV Cache technology reduces the number of repeated calculations and improves the speed of reasoning. The storage + reasoning acceleration method only performs reasoning on the last set of tokens each time it is used, and the results of the previous tokens are directly read from the external storage. Compared with the traditional reasoning of each token, the speed is greatly improved. In addition, this solution accelerates the process of hash calculation, table lookup and memory access through FPGA, making data import and export faster, further accelerating the speed of result output.

[0026] 2. Expanded application scenarios

[0027] Through the storage and import of KV Cache, the large model can not only directly conduct continuous conversations, but also conduct multiple conversations alternately without interfering with each other. In addition, the conversation can be interrupted at any time, saved, and continued at the appropriate time. Moreover, when we want to transfer the conversation to another device, we do not need to infer from the beginning, but can directly continue the previous conversation.

[0028] 3. Convenient data backup and transfer

[0029] The storage and import architecture of KV Cache has taken the data storage structure into consideration when it was designed, so no additional operations are required for data backup. At the same time, when data is transferred to other devices for use, it can be used directly as long as the configuration parameters are the same as the current parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of a large model KV cache fast storage and import architecture provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of the process of rapid import and export according to input word units when the system is running is shown;

[0032] Figure 3 It is a flowchart of querying and importing cache data of the target model;

[0033] Figure 4 It is a mapping diagram of the address mapping table and the linked list;

[0034] Figure 5 This is a schematic diagram of updating the linked list;

[0035] Figure 6 A schematic diagram of the data storage structure in a public SSD;

[0036] Figure 7 A schematic diagram of the principle of determining a target export address according to a FIFO is disclosed. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0038] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0039] Figure 1 It is a schematic diagram of the large model KV cache fast storage and import architecture provided by the embodiment of the present application. The architecture as a whole includes a CPU, an accelerator, and an FPGA. A memory is provided between the FPGA and the accelerator, and the external memory and internal memory of the FPGA are connected. The accelerator in the present application can be a GPU and / or a DPU, the memory is usually a Memory memory, the external memory is usually a large-capacity storage device such as an SSD, and the internal memory is a RAM memory.

[0040] The CPU mainly completes tasks such as data processing and data preprocessing, memory allocation, and task scheduling. The preprocessing stage of large models is completed on the CPU, and preprocessing includes operations such as data vectorization and data embedding. After preprocessing is completed, the data is sent to the GPU or FPGA for scheduling by the CPU. In addition, how the system memory is allocated and where the model cache data is stored in the memory are usually managed by the CPU.

[0041] The accelerator (GPU / DPU) mainly completes the calculation of large models. LLM runs on the GPU, and the main features of the GPU are high concurrency and high bandwidth. The DPU can provide a large number of data computing units. The core of the large model lies in the multiplication and addition of matrices. The calculation process of the large model can be accelerated by the GPU or DPU. This solution does not focus on the specific steps of the large model calculation, but is embedded in the overall architecture as a link of the processing.

[0042] FPGA mainly completes tasks related to data query and control, including loop hash calculation, query linked list, SSD fast access, import and export model cache data, etc. Hash value calculation can be realized on FPGA through hardware algorithm, and the calculation speed is fast; at the same time, FPGA also has great advantages over CPU in operations such as table lookup. When accessing SSD, FPGA can directly generate access instructions without the need for processing at the software layer and kernel layer, so the access efficiency is also improved.

[0043] Memory is DDR or HBM, which is mainly used to provide data storage and exchange functions. DDR has the characteristics of high bandwidth and large capacity. High bandwidth ensures the efficiency of data transmission between different devices. HBM has a higher bit width than DDR and can complete data reading and writing faster. The model data cache used by KV Cache technology in large model calculations in the design is first saved in Memory.

[0044] SSD provides a large-capacity storage medium for storing model cache data. SSD has the characteristics of large capacity, fast storage speed, and no data loss after power failure. It not only meets the needs of fast reading and writing of KV Cache, but also ensures data security. In addition, since the storage capacity is large enough, it can store enough model cache data.

[0045] Memory is a storage unit for storing data and instructions in the system, mainly including RAM and ROM. This application mainly uses memory to implement fast access and storage of linked lists. The linked list records the mapping address of the model cache data, which facilitates FPGA to quickly import model cache data according to the address, and store the words generated by each session task.

[0046] Based on the above system architecture, in the query phase, the CPU can determine the data query task based on the input word generated by the current session, and control the FPGA to start the data query task. The FPGA queries and imports the model cache data from the external memory into the memory based on the input word, and puts the input word of the current session into the memory. After entering the inference phase, the accelerator running the large model extracts the model cache data in the memory to perform model inference, output the inference results, and export the new model cache data.

[0047] In some embodiments, the large model needs to be further executed after being output, that is, the model cache data needs to be saved. At this time, the CPU needs to schedule and allocate tasks, control the FPGA to read the updated model cache data from the Memory, and export it to the SSD.

[0048] Figure 2 The schematic diagram of the process of fast import and export based on input word when the system is running is shown. When the user enters a token, the CPU first needs to determine whether the KV Cache mechanism needs to be started, that is, whether the model cache data needs to be imported. The CPU determines the session state based on the session number to which the current input word belongs. The purpose of determining the session state is to determine whether the KV Cache mechanism needs to be started. The fast import and export of this application is mainly for the scenario when the KV Cache mechanism is started.

[0049] KV Cache is a caching mechanism that reduces repeated calculations by storing key-value pairs and model parameters, thereby improving the inference speed. Specifically, it caches the model cache parameters such as the key and value of the previous token, so that the generation of a new token only needs to read the model cache parameters from the cache without recalculation, and so on. In this way, you only need to import the model cache parameters after each inference, and perform model inference based on the model cache parameters and the new input token. There are several cases where re-import is not required:

[0050] 1. This token is the first token of the session and there is no previous input;

[0051] 2. In the same session, this token is the subsequent input of the previous token.

[0052] When the session number indicates that the input token (token n) is the first token of the current session, or the cascade input of the previous historical token in the current session, the CPU can directly control the accelerator to perform the model inference task, and control the FPGA to store the current input token (token n). When the session number indicates that the input token is not the first token of the current session, or the cascade input of the previous historical token in the current session, the CPU allocates and controls the FPGA to perform the data query task.

[0053] If the KV cache mechanism does not need to be started, token n can be directly input into LLM for calculation on the GPU / DPU side. If KV Cache needs to be started, you need to re-enter the 0th to (n-1th)th tokens of the current session. The input token is used to determine the storage location of the target model cache data in the SSD. After the query is completed, the queried target model cache data is imported into LLM, and the input token n can be calculated through LLM. Optionally, after the LLM calculation is completed, the results are fed back to the user, and the model cache data is exported to the SSD as required.

[0054] The following is a detailed introduction to the details of FPGA in executing data query tasks and import and export stages.

[0055] Figure 3 This is a flowchart of querying and importing target model cache data. The specific execution of FPGA includes the following steps:

[0056] S1. When the FPGA starts the data query task, all calculated words generated by the current session are read from the memory based on the input words of the current session;

[0057] S2. Perform hash calculation on all calculated words in the current session;

[0058] S3, query the linked list according to the hash value of the calculated word;

[0059] S4, access the SSD according to the linked list query result to perform data matching;

[0060] S5. When the data is matched successfully, the target model cache data is imported into Memory and sent to LLM for reasoning;

[0061] S6. When data matching fails, import all words in the current session.

[0062] In this application, token n represents the uncalculated word element of the current input, and the first n-1 tokens are the calculated word elements. This step uses the result of hash calculation to distinguish different sessions. The advantage of using hash is that the calculation is simple and different sessions can be distributed relatively evenly. The hash function used here needs to satisfy the requirement of cyclic input of different numbers of tokens. After the token input is completed, the linked list is queried according to the final hash value. The linked list stores the mapping relationship between the hash address and the physical address in the database. After looking up the table, we can quickly locate the model cache data stored in the SSD and read it out. When the model cache data of the session exists in the SSD, it is read out and imported into the LLM; if the model cache data of the session cannot be found in the SSD, the tokens of the session are input into the LLM of the accelerator from 0 to token n.

[0063] Specifically, in the S3 process, FPGA is mainly implemented through the linked list and address mapping table in the memory. Because the hash value and the linked list are one-to-one corresponding, there are as many linked lists as there are hashes, so it is necessary to first determine the address of the target linked list corresponding to the hash value calculated this time before searching the linked list.

[0064] Figure 4 This is a mapping diagram of the address mapping table and the linked list. The address mapping table stores hash values ​​and one-to-one corresponding hash addresses, and this hash address is the header address of the linked list. Figure 4 The hash address corresponding to hash 1 is list1addr, and list 1addr maps to the first linked list.

[0065] For linked lists, each linked list contains several groups of links. Each group of links stores the number of calculated tokens in the current session (token_num), the hash value of the calculated tokens (last_hash), the physical address of the model cache data (data_addr), the previous link pointer (last_ptr), and the next link pointer (next_ptr). Figure 4 The linked lists of hash 1 and hash 2 both show two chain links in cascade, and null indicates the end of the linked list.

[0066] When searching the linked list, FPGA will determine the target link from the target linked list based on the number of calculated tokens of the current session (assuming that tokenn is currently being executed, this value is n-1) and the hash value of the calculated token, and extract the physical address data_addr of the target model cache data. Most of the wrong sessions can be quickly screened out by the two parameters of the number of hash values ​​of the first n-1 calculated tokens and the number of calculated tokens. Some of the wrong sessions need to wait until the data is read out from the SSD before judgment.

[0067] In some other embodiments, for situations where the model cache data needs to be exported after LLM reasoning (mainly for reasoning scenarios where tokens continue to be input), the CPU will also allocate and schedule tasks to control the CPU / GPU to output the updated model cache data to the Memory, which will be exported to the SSD by the FPGA. For this process, the FPGA will perform hash calculations based on all calculated words (the first n tokens) of the current session, and update the old linked list based on the hash value, and store the physical address of the updated model cache data in the newly inserted link.

[0068] Figure 5 This is a schematic diagram of updating the linked list. When storing, the hash values ​​of all tokens need to be recalculated. However, since the hash values ​​of the first n-1 tokens have been calculated when searching the linked list, they can be used directly at this time. That is, the input token n and the hash values ​​of the first n-1 tokens are used to continue the calculation. When the hash value is calculated, the corresponding link in the old hash linked list is deleted, and a new link is added at the end of the corresponding linked list according to the new hash. The physical address of the updated model cache data is stored in the newly inserted link. This step is implemented using last_ptr and next_ptr, because the previous pointer and the next pointer are the addresses of adjacent links in the linked list, respectively. When updating the linked list, deletion, traversal and addition of links can be achieved based on the previous pointer and the next pointer.

[0069] In some embodiments, for the steps of exporting SSD and importing Memory, it is necessary to read the model cache data in SSD, or store the model cache data. In the related art, the KV pair is usually used for direct storage when storing the data, but one problem is not considered here, that is, the (Key) K value and the (Value) V value themselves are matrix data obtained after calculation based on the historical input token. The K matrix itself needs to be transposed by FPGA during execution, while the V matrix does not have a transposition operation. Therefore, the related art directly stores the data in the order of bits. For the model operation executed by software, it is necessary to transpose the K value data after extracting the K and V values ​​from the SSD through the CPU algorithm, and restore the V value data in matrix form before subsequent operations can be performed; for hardware (such as FPGA), it is necessary to transpose the K value data after extracting the K and V values ​​from the SSD, and restore the V value data in matrix form, which leads to typesetting when it is subsequently imported into Memory, causing a delay in the input LLM. The present application makes the following improvements on this issue:

[0070] Store the model cache data in SSD in units of session tasks, see Figure 6 The diagram of the data storage structure in the public SSD. Each session task stores the calculated word-unit token value, the K value and the V value of the current session. The storage of K and V values ​​is hierarchical, which is determined by the principle of the large model. Figure 6 The K and V values ​​in the data are stored in the order of layer 0, layer 1, layer 2, etc. The K value is further stored in layers according to the order of historical token values. For example, K[token 0] represents the K value calculated with token 0, K[token 1] represents the K value calculated with token 1, and so on. The storage of V value is different from that of K value. For the convenience of calculation, V value is stored in columns, that is, V value is layered and stored in columns according to the order of token values. Figure 6As shown in the figure, the V values ​​of the first column of the matrix in all layers, clum 0, are stored together, followed by the second column, clum 1, the third column, clum 2, and so on. The V values ​​are stored in the order of tokens in each column. For example, the column of clum 0 is stored in the order of V[clumn 0token 0], V[clumn 0token 1], V[clumn 0token 2], etc., which means the V value calculated by token 0, the V value calculated by token 1, and the V value calculated by token 2 in the 0th column. The storage of tokens, K, and V values ​​is not closely arranged, and space is reserved according to the maximum token value. The advantage of this operation is that the matrix element sequence is directly extracted when the FPGA is imported, which saves the time of data typesetting and reduces the delay of data transmission to LLM. For the storage process, because it is exported after the calculation is completed, before there is no subsequent token input, the event stored in the SSD in this specific form will not affect the operation of LLM, so there is no storage delay problem.

[0071] In some embodiments, the physical address is incremented in sequence when the SSD is stored. When a certain data is deleted, the physical address becomes free. If the data is small, you can ignore these free addresses, but when the data scale is large, the free space must be fully utilized. Therefore, when we generate the address of the SSD, we need to re-add the deleted address. For this reason, this application sets up an integrated FIFO memory for the storage address of the SSD.

[0072] Figure 7 The invention discloses a schematic diagram of the principle of determining the target export address according to FIFO. The integrated FIFO memory at least includes a first address FIFO (inc_fifo) and a second address FIFO (del_fifo). The two FIFOs can be controlled by CPU or FPGA instructions. The inc_fifo stores the address of the idle area of ​​the external memory that does not store data within a preset time, and the del_fifo stores the blank address after deleting the historical model cache data within a preset time. When exporting the model cache data, the target export address is determined according to the address margin stored in the first address FIFO and the second address FIFO. Optionally, a third address FIFO (addr_fifo) can also be integrated to temporarily store the target export address. Addr_fifo periodically outputs the address to the FPGA, and the FPGA performs the export step.

[0073] In summary, the fast storage and import architecture designed by this application using the advantages of KV Cache technology can achieve the following technical effects:

[0074] 1. Reduced computational effort and improved reasoning speed:

[0075] Executing large model reasoning through KV Cache technology reduces the number of repeated calculations and improves the speed of reasoning. The storage + reasoning acceleration method only performs reasoning on the last set of tokens each time it is used, and the results of the previous tokens are directly read from the external storage. Compared with the traditional reasoning of each token, the speed is greatly improved. In addition, this solution accelerates the process of hash calculation, table lookup and memory access through FPGA, making data import and export faster, further accelerating the speed of result output.

[0076] 2. Expanded application scenarios

[0077] Through the storage and import of KV Cache, the large model can not only directly conduct continuous conversations, but also conduct multiple conversations alternately without interfering with each other. In addition, the conversation can be interrupted at any time, saved, and continued at the appropriate time. Moreover, when we want to transfer the conversation to another device, we do not need to infer from the beginning, but can directly continue the previous conversation.

[0078] 3. Convenient data backup and transfer

[0079] The storage and import architecture of KV Cache has taken the data storage structure into consideration when it was designed, so no additional operations are required for data backup. At the same time, when data is transferred to other devices for use, it can be used directly as long as the configuration parameters are the same as the current parameters.

[0080] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A large model KV cache fast storage and import architecture, characterized in that: The architecture includes a CPU for data processing and task scheduling, an accelerator for running large models, an FPGA for executing KV Cache and data query, a memory for providing data exchange between the FPGA and the accelerator, an external memory for storing model cache data, and a memory for storing session words and linked lists; the linked lists record mapping addresses of model cache data; In the query phase, the CPU determines the data query task based on the input word generated by the current session, and controls the FPGA to start the data query task; the FPGA queries and imports the model cache data from the external memory into the memory based on the input word, and stores the input word of the current session into the memory; During the inference phase, the accelerator running the large model performs model inference by extracting the model cache data in the memory, outputs the inference results and exports the new model cache data; the FPGA reads the updated model cache data from the memory according to the control instructions of the CPU and exports it to the external memory.

2. The large model KV cache fast storage and import acceleration architecture according to claim 1 is characterized in that: The CPU determines the session state based on the session number to which the current input word belongs; When the session number indicates that the input word is the first word of the current session, or is a cascade input of a previous historical word in the current session, the CPU controls the accelerator to perform the model reasoning task, and controls the FPGA to store the current input word; When the session number indicates that the input word is not the first word of the current session, or is not the cascade input of the previous historical word in the current session, the CPU allocates and controls the FPGA to execute the data query task.

3. The large model KV cache fast storage and import acceleration architecture according to claim 1 is characterized in that: When the FPGA starts a data query task, the first n-1 calculated word elements generated in the current session are read from the memory based on the nth word element in the current session, and hash calculation is performed on the first n-1 calculated word elements, and the linked list and matching model cache data are queried according to the hash value.

4. The large model KV cache fast storage and import acceleration architecture according to claim 3 is characterized in that: An address mapping table is also stored in the memory, wherein the address mapping table stores hash values ​​and one-to-one corresponding hash addresses; the linked list stores a mapping relationship between hash addresses and physical addresses storing model cache data; When searching the linked list, the target linked list is matched according to the mapping and the determined target hash address and the physical address of the target model cache data located in the external memory is determined.

5. The large model KV cache fast storage and import acceleration architecture according to claim 4 is characterized in that: The linked lists correspond to the hash values ​​one by one, and each linked list contains a number of groups of links, each group of links stores the number of calculated word units of the current session, the hash value of the calculated word unit, the physical address of the model cache data, the previous link pointer and the next link pointer; When searching the linked list, the target link is determined from the target linked list according to the number of calculated tokens of the current session and the hash value of the calculated token, and the physical address of the target model cache data is extracted.

6. The large model KV cache fast storage and import acceleration architecture according to claim 5, characterized in that: After determining the physical address of the target model cache data, locate and match the target model cache data from the external memory; when the target model cache data is successfully matched from the external memory, import it into the large model for reasoning; when the target model cache data is not matched from the external memory, extract all the words of the current session from the memory and send them to the accelerator for model reasoning.

7. The large model KV cache fast storage and import acceleration architecture according to claim 5, characterized in that: When the FPGA exports the updated model cache data according to the control instruction of the CPU, it performs hash calculation based on all calculated words in the current session, updates the old linked list according to the hash value, and stores the physical address of the updated model cache data into the newly inserted link.

8. The large model KV cache fast storage and import acceleration architecture according to claim 7, characterized in that: The front-stage pointer and the back-stage pointer are the addresses of adjacent links in the linked list, respectively. When updating the linked list, the links are deleted, traversed and added based on the front-stage pointer and the back-stage pointer.

9. The large model KV cache fast storage and import acceleration architecture according to any one of claims 1 to 8, characterized in that: The model cache data is stored in the external memory in units of session tasks. Each session task stores the calculated word unit token value of the current session, the K value and the V value of the current session, where the K value is stored in layers according to the order of token values; the V value is stored in layers according to the order of token values ​​and in columns.

10. The large model KV cache fast storage and import acceleration architecture according to claim 9, characterized in that: A first address FIFO and a second address FIFO are also provided; the first address FIFO stores the address of a free area of ​​the external memory that has not stored data within a preset time, and the second address FIFO stores the blank address after deleting the historical model cache data within the preset time; When exporting the model cache data, a target export address is determined according to the address margins stored in the first address FIFO and the second address FIFO.

Images