Key value cache multiplexing method and device based on end-side large model

Abstract

The invention provides a key value cache multiplexing method and device based on an end-side large model, and relates to the technical field of artificial intelligence. According to the method, cross-request multiplexing of key value vectors can be achieved by introducing a target cache table, it is avoided that the end-side large model re-executes calculation of a pre-filling stage for each new request, and the calculation efficiency is improved. The calculation burden and the calculation complexity of the end-side equipment are remarkably reduced, the first word delay is shortened, the overall reasoning efficiency of the end-side large model is improved, and the user experience is improved. In addition, the method can also reduce performance requirements of computation-intensive tasks on the end-side equipment, can be introduced to flexibly adapt to hardware configuration of different end-side equipment, and has high universality and expandability. According to the method, an end-side large model is adopted, the method is completely operated on end-side equipment and does not need to depend on a cloud server, privacy protection and offline operation can be considered, the efficiency of end-side reasoning is guaranteed, meanwhile, a completely localized reasoning scheme is provided, and the requirements of a user for privacy protection and offline operation are met.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a key-value cache reuse method and apparatus based on a large edge model. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have been widely applied in various scenarios such as chatbots, machine translation, and text generation due to their superior performance in natural language processing tasks. Large language models are typically deployed on cloud computing platforms with abundant computing resources, but this cloud deployment method cannot meet users' growing demands for privacy protection and offline operation. Against this backdrop, edge-based large language models have emerged, providing users with a new generation of human-computer interaction solutions that combine cost-effectiveness and privacy protection by deploying lightweight large language models on local computing devices.

[0003] The large-scale model on the device performs inference using word vectors, mainly divided into two stages: prefill and decoding. The prefill stage is responsible for calculating the contextual relationships of all input token sequences in parallel at once, generating key-value vectors, and storing them in pre-allocated device memory to prepare for subsequent decoding. This stage has high computational complexity and is a computationally intensive task. The decoding stage generates new tokens one by one based on the key-value vectors in an autoregressive manner. This stage only needs to calculate the semantic relationship between the current token and the preceding text each time, with a smaller computational load, but it requires frequent access to the key-value vectors in device memory, making it a memory-intensive task.

[0004] However, in the above scheme, the key-value vector generated in the pre-filling stage is only used for pre-filling and decoding within a single inference request. This will cause the computationally intensive task of the pre-filling stage to be re-executed for each new request. In interactive scenarios with a large number of overlapping prefixes, such as multi-turn dialogues and document analysis, repeatedly calculating the key-value cache with the same prefix will increase the computational load on the edge device. Summary of the Invention

[0005] This invention provides a key-value cache reuse method and apparatus based on a large end-side model to address the deficiencies in related technologies.

[0006] This invention provides a key-value cache reuse method based on a large edge model, comprising: Obtain the current request, and determine the current word sequence of the current request based on the cache block size in the device's first-level cache pool; Calculate the hash value of each current word sequence, and based on the hash value of each current word sequence, calculate the block hash value of the cache block corresponding to each current word sequence; For the first lexical sequence whose block hash value matches the target cache table, the target cache location corresponding to the first lexical sequence is determined based on the target cache table, and the key-value vector in the target cache location is extracted; the target cache table is used to cache the mapping relationship between the block hash value and the cache location identifier of the cache block corresponding to each historical lexical sequence of the historical request. For the second lexical sequence whose hash value does not match the target cache table, based on the end-side large model, during the pre-filling stage, the key-value vector of the second lexical sequence is calculated, a target cache block is requested in the first-level cache pool, and the key-value vector of the second lexical sequence is cached in the target cache block.

[0007] According to the key-value cache reuse method based on a large edge model provided by the present invention, the step of calculating the block hash value of the cache block corresponding to each current lexical sequence based on the hash value of each current lexical sequence includes: For any lexical sequence in each current lexical sequence, the block hash value of the cache block corresponding to the lexical sequence is calculated based on the hash value of the lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence in each current lexical sequence.

[0008] According to the key-value cache reuse method based on a large edge model provided by the present invention, the step of calculating the block hash value of the cache block corresponding to the arbitrary lexical sequence based on the hash value of the arbitrary lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence of the arbitrary lexical sequence in each current lexical sequence includes: If the cache block corresponding to any word sequence is not filled, then the cache block corresponding to any word sequence is filled based on the decoded words obtained by the end-side large model in the decoding stage. If the cache block corresponding to any given lexical sequence is already full, then the block hash value of the cache block corresponding to any given lexical sequence is calculated based on the hash value of the given lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence.

[0009] According to the present invention, a key-value cache reuse method based on a large end-side model is provided, wherein the target cache table includes a first-level cache table and a second-level cache table; The first-level cache table is used to cache the mapping relationship between the block hash value of the cache block corresponding to the historical lexical sequence and the cache block identifier in the first-level cache pool; The second-level cache table is used to cache the mapping relationship between the block hash value of the cache block corresponding to the historical lexical sequence and the cache address identifier in the second-level cache pool of the solid-state drive; the cache address corresponding to the second-level cache table persists the key-value vector of the historical lexical sequence.

[0010] According to the key-value cache reuse method based on a large edge model provided by the present invention, the step of requesting a target cache block in the first-level cache pool includes: If the first-level cache pool is full, the key-value vector in the first cache block that was allocated earliest in the first-level cache pool and is not currently used by other requests is cached in the cache address of the second-level cache pool, and the first cache block is used as the target cache block to update the first-level cache table and the second-level cache table. If the first-level cache pool is not full, then the unused second cache block in the first-level cache pool is used as the target cache block, and the first-level cache table is updated.

[0011] According to the key-value cache reuse method based on a large edge model provided by the present invention, the step of extracting the key-value vector in the target cache location includes: If the target cache location is a cache block in the first-level cache pool, then the key-value vector in the target cache location is copied to the pre-fill context of the pre-fill stage; If the target cache location is a cache address in the secondary cache pool, then the key-value vector in the third cache block that is the earliest allocated and used in the primary cache pool and is not currently used by other requests is cached in the cache address of the secondary cache pool, and the key-value vector in the target cache location is uploaded to the third cache block.

[0012] According to the key-value cache reuse method based on a large edge model provided by the present invention, the step of requesting a target cache block in the first-level cache pool further includes: Based on the idle cache queue, request the target cache block in the first-level cache pool; The idle cache queue is used to manage the idle cache blocks in the first-level cache pool, and the data structure of the idle cache queue is a least recently used queue implemented with a doubly linked list.

[0013] The present invention also provides a key-value cache reuse device based on a large edge model, comprising: The word segmentation module is used to obtain the current request and determine the current word sequence of the current request based on the size of the cache block in the first-level cache pool of the device memory. The calculation module is used to calculate the hash value of each current word sequence, and based on the hash value of each current word sequence, calculate the block hash value of the cache block corresponding to each current word sequence; The reuse module is used to determine the target cache location corresponding to the first lexical sequence that hits the target cache table based on the target cache table, and extract the key-value vector in the target cache location; the target cache table is used to store the mapping relationship between the block hash value and the cache location identifier of the cache block corresponding to each historical lexical sequence of the historical request. The caching module is used to calculate the key-value vector of the second word sequence in the pre-filling stage based on the end-side large model for a second word sequence whose hash value does not match the target cache table, apply for a target cache block in the first-level cache pool, and cache the key-value vector of the second word sequence in the target cache block.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the key-value cache reuse method based on the large end-side model as described above.

[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the key-value cache reuse method based on the large end-side model as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the key-value cache reuse method based on a large edge model as described above.

[0017] This invention provides a key-value caching reuse method and apparatus based on a large edge-side model. By introducing a target cache table, this method enables cross-request reuse of key-value vectors, avoiding the need for the large edge-side model to re-execute the pre-filling stage calculation for each new request. This significantly reduces the computational burden and complexity of the edge-side device, shortens the first-word latency, improves the overall inference efficiency of the large edge-side model, and enhances the user experience. Furthermore, this method reduces the performance requirements of computationally intensive tasks on the edge-side device. Its introduction allows for flexible adaptation to different edge-side device hardware configurations, exhibiting high versatility and scalability. Utilizing a large edge-side model, this method runs entirely on the edge-side device without relying on cloud servers, balancing privacy protection and offline operation. While ensuring high efficiency in edge-side inference, it provides a fully localized inference solution, meeting users' needs for privacy protection and offline operation, and broadening application scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the inference computation process of LLM in existing edge scenarios.

[0020] Figure 2 This is one of the flowcharts illustrating the key-value cache reuse method based on a large end-side model provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the end-side device in the key-value cache reuse method based on the end-side large model provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the architecture used in the key-value cache reuse method based on the large end-side model provided by the present invention.

[0023] Figure 5 This is the second flowchart of the key-value cache reuse method based on a large end-side model provided by the present invention.

[0024] Figure 6 This is a flowchart illustrating the cache allocation and usage process in the key-value cache reuse method based on a large client-side model provided by the present invention.

[0025] Figure 7 This is a flowchart illustrating the request and usage of cache in the key-value cache reuse method based on the large end-side model provided by the present invention.

[0026] Figure 8 This is a flowchart illustrating the request for and use of cache in the key-value cache reuse method based on a large client-side model provided by the present invention.

[0027] Figure 9 This is a schematic diagram of the key-value cache reuse device based on a large end-side model provided by the present invention.

[0028] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] Figure 1 This demonstrates the inference computation process of LLM in a device-side scenario. The user (Actor) inputs the string "abc". Before the actual computation, a tokenizer first segments the input into words and converts them into tokens. A token ID represents the position of each token in the vocabulary. After computation by the embedding layer, the token ID is mapped to the word embedding space to obtain word vectors. LLM performs inference using these word vectors. In the pre-filling stage, the contextual relationships of all input token sequences are computed in parallel, generating key-value vectors that are stored in pre-allocated device memory. In the decoding stage, new tokens, such as d and e, are generated sequentially using the cached key-value vectors in device memory. Here, d and e only represent the first and second tokens of the output corresponding to the input, not the correlation between English letters.

[0031] Existing LLMs generate key-value vectors during the pre-filling stage, but these vectors are only used for pre-filling and decoding within a single inference request. This results in the computationally intensive pre-filling stage needing to be re-executed for each new request. In multi-turn dialogues, document analysis, and other interactive scenarios with significant prefix overlap, repeatedly calculating key-value caches with the same prefixes further burdens the computational load on the device. Moreover, the pre-filling stage requires calculating the key-value vectors of all input lexical sequences at once using an attention mechanism, leading to high computational complexity. On resource-constrained devices such as mobile devices or embedded systems, the high computational demands of the pre-filling stage often result in high Time To First Token (TTFT) latency, severely impacting user experience. Therefore, this invention provides a key-value cache reuse method based on a large-scale device model.

[0032] Figure 2 This is a flowchart illustrating a key-value cache reuse method based on a large end-side model provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the method includes: S1, obtain the current request, and determine the current word sequence of the current request based on the cache block size in the first-level cache pool of the device memory; S2, calculate the hash value of each current word sequence, and based on the hash value of each current word sequence, calculate the block hash value of the cache block corresponding to each current word sequence; S3, for the first lexical sequence whose block hash value hits the target cache table, based on the target cache table, determine the target cache location corresponding to the first lexical sequence, and extract the key-value vector in the target cache location; the target cache table is used to store the mapping relationship between the block hash value and the cache location identifier of the cache block corresponding to each historical lexical sequence of the historical request; S4. For the second word sequence whose hash value does not match the target cache table, based on the end-side large model, in the pre-filling stage, calculate the key-value vector of the second word sequence, apply for a target cache block in the first-level cache pool, and cache the key-value vector of the second word sequence into the target cache block.

[0033] Specifically, the key-value cache reuse method based on a large edge model provided in this embodiment of the invention is executed by a key-value cache reuse device based on a large edge model. This device can be configured within a cache manager, which can be embedded in the central processing unit (CPU) of the edge device or serve as a shared module in the multi-core / heterogeneous architecture of the edge device. The edge device can be a mobile device, embedded system, or other device that has a built-in large language model based on a transformer architecture. This large language model is the edge-side large model, serving as the inference engine of the edge device.

[0034] like Figure 3 As shown, the edge device may include a computing unit and a storage unit. The computing unit includes a CPU and a computing power card. The computing power card includes a Tensor Processing Unit (TPU). The TPU is dedicated to inference computing for LLM. For 16-bit precision, its corresponding computing power is about 4 TFLOPS, which is much lower than that of service-grade computing chips.

[0035] The storage unit includes RAM, device memory, and solid-state drives (SSDs). The RAM can be approximately 16GB of DDR5 memory. The device memory, as on-chip memory for the edge device, is located within the computing card and can be approximately 16GB of DDR4 memory. The SSD is a high-speed hard drive with an input / output (IO) bandwidth of approximately 500MB / s, and its size can be 1TB, with a possible quantity of two.

[0036] First, execute step S1 to obtain the current request. The current request refers to the request input by the user at the current moment that needs to be processed by the large model on the client side and a result needs to be given, such as a user question.

[0037] The device-side memory is configured with a cache pool, named the first-level cache pool. The first-level cache pool is divided into multiple cache blocks, each of the same size (e.g., 2), meaning it can cache key-value vectors of two terms. Each cache block can be allocated a fixed-size contiguous storage space based on its size to improve spatial locality and reuse efficiency.

[0038] The current request is divided into multiple current token sequences using the cache block size in the device's primary cache pool. Here, a current token sequence refers to the token sequence within the current request. The number of current token sequences in the current request can be calculated based on the length of the current request and the cache block size. For example, the ratio of the current request length to the cache block size can be calculated, and the ratio can be rounded up to obtain the number of current token sequences, which can be expressed as: Where N is the number of the current word sequence, Ceil represents rounding up, and S is the length of the current request.

[0039] For example, the current request is "ABCDE", and its length is L =5, and N=3 is calculated using the formula, meaning that according to the size of the cache block, three current word sequences can be obtained: “AB”, “CD”, and “E”.

[0040] Next, step S2 is executed, which maps each word in each current word sequence to a hash key. The hash key is then hashed using a standard library hash function to obtain the hash value of each word in each current word sequence. The hash value of each current word sequence includes the hash values ​​of each word within that sequence. Using the hash values ​​of each current word sequence, the block hash value of the corresponding cache block can be calculated.

[0041] It is understandable that the cache block corresponding to each current word sequence refers to the cache block used to cache the key-value vectors of each word in the current word sequence. Each cache block that caches the current word sequence has a block hash value, and the block hash value of each cache block can be determined using the hash value of the current word sequence it caches.

[0042] After executing step S2, the block hash value of the cache block corresponding to each current lexical sequence is used to search the target cache table. This involves matching the block hash value of the cache block corresponding to each current lexical sequence with the block hash value in the target cache table. If a match is found, the target cache table is hit; otherwise, it is not. The target cache table stores the mapping relationship between the block hash values ​​of the cache blocks corresponding to each historical lexical sequence of historical requests and the cache location identifiers.

[0043] Historical requests are requests that precede the current request. There can be one or more historical requests. Historical requests are divided into multiple historical token sequences, where a historical token sequence refers to the sequence of tokens within a historical request. Each historical token sequence corresponds to a cache block used to cache the key-value vectors of each token in that historical token sequence. The block hash value of each cache block is determined in the same way.

[0044] The cache location identifier is used to mark the cache location, which refers to the position of the key-value vector of the cached lexical sequence. This location can be a cache block within the first-level cache pool or storage space on other storage media; no specific limitation is made here. Here, the key-value vector of the lexical sequence includes the key-value vector of each lexical in the sequence.

[0045] The target cache table allows you to bind the block hash value of an allocated cache block to the cache location identifier, thus obtaining the mapping relationship between the two.

[0046] For each current word sequence, the current word sequence whose block hash value hits the target cache table is taken as the first word sequence and step S3 is executed; the current word sequence whose block hash value does not hit the target cache block is taken as the second word sequence and step S4 is executed.

[0047] When performing step S3, the target cache location identifier is directly searched in the target cache table. The target cache location identifier is a cache location identifier in the target cache table that has a mapping relationship with the block hash value of the cache block corresponding to the first word sequence, and is used to mark the target cache location.

[0048] Once the target cache location identifier is found, the target cache location can be determined. Then, the key-value vector can be extracted from the target cache location as the result obtained by the end-side large model in the pre-filling stage for the decoding stage. There is no need to recalculate the key-value vector of the first word sequence, which reduces the computational load of the end-side large model in the pre-filling stage.

[0049] During step S4, based on the large-scale model on the client side, in the pre-filling stage, the key-value vector of the second word sequence is calculated, and a target cache block is requested in the first-level cache pool. The key-value vector of the second word sequence is then cached in the target cache block. It can be understood that when the first-level cache pool is not full, a corresponding target cache block can be allocated for the second word sequence based on the remaining total capacity of the first-level cache pool. When the first-level cache pool is full, the cache blocks already occupied in the first-level cache pool are reused.

[0050] The key-value cache reuse method based on a large edge model provided in this embodiment of the invention first obtains the current request and determines each current word sequence of the current request based on the cache block size in the first-level cache pool of the device memory. Then, it calculates the hash value of each current word sequence and, based on the hash value of each current word sequence, calculates the block hash value of the cache block corresponding to each current word sequence. Subsequently, for the first word sequence whose block hash value matches the target cache table, the target cache table is used to determine the target cache location corresponding to the first word sequence, and the key-value vector in the target cache location is extracted. For the second word sequence whose hash value does not match the target cache table, based on the large edge model, in the pre-filling stage, the key-value vector of the second word sequence is calculated, a target cache block is requested in the first-level cache pool, and the key-value vector of the second word sequence is cached in the target cache block. This method, by introducing a target cache table, enables the reuse of key-value vectors across requests, avoiding the need for the large client-side model to re-execute the pre-filling stage calculation for each new request. This significantly reduces the computational burden and complexity of the client-side device, shortens the first-word latency, improves the overall inference efficiency of the large client-side model, and enhances the user experience.

[0051] Furthermore, this method reduces the performance requirements of computationally intensive tasks on edge devices. Its introduction allows for flexible adaptation to different edge device hardware configurations, demonstrating high versatility and scalability. Employing a large edge model, this method runs entirely on edge devices without relying on cloud servers, balancing privacy protection and offline operation. While ensuring high efficiency in edge inference, it provides a fully localized inference solution, meeting users' needs for privacy protection and offline operation, and broadening application scenarios.

[0052] Based on the above embodiments, the step of calculating the block hash value of the cache block corresponding to each current lexical sequence based on the hash value of each current lexical sequence includes: For any lexical sequence in each current lexical sequence, the block hash value of the cache block corresponding to the lexical sequence is calculated based on the hash value of the lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence in each current lexical sequence.

[0053] Specifically, since the computation of Transformer involves the contextual interaction of lexical units in the latent space, the corresponding key-value vector cache can only be reused when the preorder lexical sequence is consistent. For example, the word "Beijing" in "I love to eat Beijing braised pork offal" and "I am a Beijinger" obviously have different semantics in different contexts, and their corresponding key-value vector cache data also differ. This difference is reflected in the calculation process of the block hash value.

[0054] In this embodiment of the invention, the calculation process for the block hash value of different cache blocks is the same. Taking any lexical sequence in each current lexical sequence as an example, when calculating the block hash value of the cache block corresponding to any lexical sequence, it can be calculated using the hash value of any lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence of any lexical sequence in each current lexical sequence. The preceding lexical sequence refers to all lexical sequences preceding any lexical sequence. Since the first current lexical sequence in each current lexical sequence does not have a preceding lexical sequence, the hash value of the first current lexical sequence is directly used as the block hash value of the cache block corresponding to the first current lexical sequence.

[0055] If any word sequence is the i-th current word sequence, then the block hash value of the cache block corresponding to the i-th current word sequence can be calculated using the following formula: ; ; in, Let be the block hash value of the cache block corresponding to the i-th current word sequence. For the standard library's hash function, This indicates the XOR operation. Let represent the hash value of the i-th current word sequence, N be the number of current word sequences in the current request, and S be the length of the current request. This refers to the j-th word in the current request.

[0056] In this embodiment of the invention, the block hash value of the cache block corresponding to the current lexical sequence is calculated using the block hash value of the cache block corresponding to the preceding lexical sequence. This ensures that the same lexical sequence in different positions in different requests corresponds to an accurate key-value vector, thereby improving the accuracy of key-value vector reuse.

[0057] Based on the above embodiments, the step of calculating the block hash value of the cache block corresponding to the arbitrary lexical sequence based on the hash value of the arbitrary lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence of the arbitrary lexical sequence in each current lexical sequence includes: If the cache block corresponding to any word sequence is not filled, then the cache block corresponding to any word sequence is filled based on the decoded words obtained by the end-side large model in the decoding stage. If the cache block corresponding to any given lexical sequence is already full, then the block hash value of the cache block corresponding to any given lexical sequence is calculated based on the hash value of the given lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence.

[0058] Specifically, when calculating the block hash value of the cache block corresponding to any given word sequence, different operations can be selected based on whether the cache block corresponding to the given word sequence is full. Since each current word sequence is obtained by partitioning the cache block according to its size, if the cache block corresponding to any given word sequence is not full, it means that the given word sequence must be the last current word sequence in the current request. If the cache block corresponding to any given word sequence is full, then the given word sequence could be any current word sequence in the current request.

[0059] To ensure the accuracy of the block hash value of the cache block, if the cache block corresponding to any word sequence is not filled, the block hash value of the cache block is not calculated first. Instead, the decoded words obtained by the large model on the end side during the decoding stage are used to fill the cache block corresponding to any word sequence.

[0060] For any cache block corresponding to a fully filled lexical sequence, the hash value of the lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence are used to... The calculation formula calculates the block hash value of the cache block corresponding to any byte sequence.

[0061] In this embodiment of the invention, when the cache block corresponding to the current lexical sequence is already full, calculating the block hash value of the cache block can further improve the accuracy of the block hash value.

[0062] Based on the above embodiments, the target cache table includes a first-level cache table and a second-level cache table; the first-level cache table is used to cache the mapping relationship between the block hash value of the cache block corresponding to the historical word sequence and the cache block identifier in the first-level cache pool; The second-level cache table is used to cache the mapping relationship between the block hash value of the cache block corresponding to the historical lexical sequence and the cache address identifier in the second-level cache pool of the solid-state drive; the cache address corresponding to the second-level cache table persists the key-value vector of the historical lexical sequence.

[0063] Specifically, the target cache table used in this embodiment of the invention may include a first-level cache table and a second-level cache table. The first-level cache table is used to manage cache blocks within the first-level cache pool, and the second-level cache table is used to manage cache addresses within the second-level cache pool of the solid-state drive. The solid-state drive is a high-bandwidth, high-speed hard drive.

[0064] The first-level cache table can cache the mapping relationship between the block hash value of the cache block corresponding to each historical lexical sequence of a historical request and the cache block identifier of the first-level cache pool. The second-level cache table can cache the mapping relationship between the block hash value of the cache block corresponding to each historical lexical sequence of a historical request and the cache address identifier of the second-level cache pool.

[0065] In this embodiment of the invention, a two-level cache architecture that coordinates device memory and solid-state drive can optimize the caching strategy of key-value vectors, effectively alleviate the capacity and bandwidth limitations of device memory, and further improve the inference efficiency of large models on the edge.

[0066] Based on the above embodiments, requesting a target cache block within the primary cache pool includes: Based on the idle cache queue, request the target cache block in the first-level cache pool; The idle cache queue is used to manage the idle cache blocks in the first-level cache pool, and the data structure of the idle cache queue is a least recently used queue implemented with a doubly linked list.

[0067] Specifically, in this embodiment of the invention, the first-in-first-out characteristic of the queue can be utilized to apply for the target cache block in the first-level cache pool by means of the idle cache queue, so as to realize the orderly application of the cache block in the first-level cache pool.

[0068] The free cache queue can be used to manage free cache blocks within the first-level cache pool. These free cache blocks can include unallocated cache blocks within the first-level cache pool, as well as cache blocks that have been allocated but are not currently being used by other requests. Cache blocks in the first-level cache pool can be managed by inserting cache block pointers into the free cache queue.

[0069] The free cache queue is structured as a Least Recently Used (LRU) queue implemented using a doubly linked list. This data structure allows for quick lookup of head and tail nodes, traversal from the beginning, and deletion of intermediate nodes, enabling the rapid identification of the earliest allocated cache block.

[0070] It is understood that in this embodiment of the invention, the running memory of the terminal device is not used for caching the key-value vector because the running memory capacity is limited, so as to reserve enough running space for other applications.

[0071] Based on the above embodiments, the step of requesting a target cache block in the first-level cache pool includes: If the first-level cache pool is full, the key-value vector in the first cache block that was allocated earliest in the first-level cache pool and is not currently used by other requests is cached in the cache address of the second-level cache pool, and the first cache block is used as the target cache block to update the first-level cache table and the second-level cache table. If the first-level cache pool is not full, then the unused second cache block in the first-level cache pool is used as the target cache block, and the first-level cache table is updated.

[0072] Specifically, during the process of allocating a target cache block within the first-level cache pool, different operations can be performed depending on whether the first-level cache pool is full. If the first-level cache pool is full, cache blocks within it need to be reused. The reused cache block can be the earliest allocated cache block in the first-level cache pool that is not currently being used by other requests. The key-value vector in the first cache block is cached at the cache address of the second-level cache pool, and the first cache block is used as the target cache block to update both the first-level and second-level cache tables.

[0073] In other words, when reusing the first cache block, the key-value vector cached therein needs to be transferred to the cache address of the second-level cache pool, and then the first cache block is used as the target cache block to cache the key-value vector of the second word sequence.

[0074] Updating the first-level cache table involves replacing the block hash value of the first cache block in the first-level cache table with the block hash value of the cache block corresponding to the second word sequence. Updating the second-level cache table involves adding a mapping relationship between the block hash value of the first cache block and its cache address identifier in the second-level cache table.

[0075] If the first-level cache pool is not full, the unused second cache block in the first-level cache pool can be directly used as the target cache block to cache the key-value vector of the second word sequence, and the first-level cache table can be updated. Here, updating the first-level cache table means adding a mapping relationship between the block hash value of the second cache block and the cache block identifier of the second cache block in the first-level cache table.

[0076] In this embodiment of the invention, the reuse caching of key-value vectors is achieved by intelligently allocating and scheduling between device memory and solid-state drive, thereby significantly reducing redundant calculations in the large model inference process on the edge and improving response speed and overall user experience.

[0077] Based on the above embodiments, extracting the key-value vector from the target cache location includes: If the target cache location is a cache block in the first-level cache pool, then the key-value vector in the target cache location is copied to the pre-fill context of the pre-fill stage; If the target cache location is a cache address in the secondary cache pool, then the key-value vector in the third cache block that is the earliest allocated and used in the primary cache pool and is not currently used by other requests is cached in the cache address of the secondary cache pool, and the key-value vector in the target cache location is uploaded to the third cache block.

[0078] Specifically, when extracting the key-value vector from the target cache location, different operations can be performed depending on the target cache location.

[0079] If the target cache location is a cache block in the first-level cache pool, the key-value vector in the target cache location can be directly copied to the pre-fill context in the pre-fill stage without having to recalculate the key-value vector.

[0080] If the target cache location is a cache address in the second-level cache pool, the key-value vector at that cache address needs to be uploaded to the first-level cache pool before it can be added to the pre-fill context during the pre-filling phase. In this case, the third cache block, which is the earliest allocated and currently not used by other requests in the first-level cache pool, can be selected as the free cache block to receive the key-value vector from the target cache location. This means the key-value vector in the third cache block is cached at the cache address in the second-level cache pool, and then the key-value vector from the target cache location is uploaded to the third cache block, effectively swapping the positions of the key-value vectors.

[0081] In this embodiment of the invention, by optimizing strategies such as cache block access and solid-state drive uploading and unloading during the key-value vector reuse process, the capacity and bandwidth limitations of device memory are effectively alleviated.

[0082] like Figure 4 The diagram shown is an architectural schematic of the key-value cache reuse method based on a large end-side model provided in this embodiment of the invention. Figure 4 Compared to Figure 1 In this system, solid-state drives (SSDs) are introduced on top of device memory. Through the primary cache pool of device memory and the secondary cache pool of SSDs, key-value vector caching reuse can be achieved in a coordinated manner.

[0083] It is understandable that each term, as a hash key, and its hash value constitute a hash table. The mapping relationship between the block hash value of the cache block corresponding to each historical term and the cache block ID constitutes a first-level cache table. The mapping relationship between the cache address and the cache address identifier in the second-level cache pool corresponding to each historical term constitutes a second-level cache table.

[0084] like Figure 5 The diagram shown is a flowchart illustrating the key-value cache reuse method based on a large end-side model provided in an embodiment of the present invention. Figure 5 In this method, the method includes: The current request has arrived at the cache manager; The cache manager divides the current request into chunks and calculates the hash value of each current token sequence and the block hash value of the cache block corresponding to each current token sequence; Query whether the block hash table corresponding to each current word sequence hits the target cache table; For the first word sequence that hits the target cache table with the block hash value, the target cache table is used to determine the target cache position corresponding to the first word sequence, and the key-value vector in the target cache position is extracted and copied to the pre-filling context of the pre-filling stage. For the second word sequence whose hash value does not match the target cache table, based on the large-scale model on the edge, during the pre-filling stage, the key-value vector of the second word sequence is calculated, the target cache block is requested in the first-level cache pool, the key-value vector of the second word sequence is cached in the target cache block, and the hash table is updated synchronously. Check if the first-level cache pool is full. If it is full, determine the first cache block that was allocated and used earliest in the first-level cache pool and is not currently used by other requests. Persist the first cache block to the second-level cache pool, that is, cache the key-value vector in the first cache block to the cache address of the second-level cache pool. Use the first cache block as the target cache block and update the first-level cache table and the second-level cache table.

[0085] The following example illustrates the workflow of the cache manager, assuming that the first-level cache pool contains four cache blocks with cache block IDs (0-3) and a cache block size of 2. The cache block pointers in the free cache queue are initialized to 0123, meaning that the four cache blocks are queued in the free cache queue in ascending order of their cache block IDs.

[0086] like Figure 6 The diagram shown illustrates the process of requesting and using cache for request 1.

[0087] exist Figure 6 In this context, request 1 can be "ABCDE", which is the first request received by the client-side large model. At this point, request 1 is considered the current request. Since the length of request 1 is 5, it can be divided into three word sequences: "AB", "CD", and "E" according to the cache block size. That is, three cache blocks are needed to store one word sequence each.

[0088] Calculate the hash value of each word in each word sequence, and calculate the block hash value of the cache block corresponding to each word sequence.

[0089] Query the block hash value in the target cache table, which contains a first-level cache table and a second-level cache table.

[0090] Since request 1 is the first request, after querying, both the first-level and second-level cache tables are empty, indicating a cache miss. A complete pre-filling phase is required to generate key-value vectors for all token sequences. Three cache blocks are then sequentially retrieved from the head of the free cache queue, and the key-value vectors are copied into the cache space of each block. Note that cache block 2 is not completely filled; it needs to be filled using the first decoded token "F" determined during the decoding phase. To maximize cache block utilization, decoded tokens obtained during the decoding phase only fill in partially filled cache blocks; no additional cache blocks are allocated for storing decoded tokens. After filling, the three cache block IDs are inserted into the tail of the free cache queue in the allocation order, and a cached flag is added. At this point, the free cache queue is updated to 3012.

[0091] Before the decoding stage, the first-level cache table only includes the mapping relationship between the cache block IDs and their block hash values ​​of cache block 0 and cache block 1. Specifically, the block hash value of cache block 0 is Hash: (AB), which is calculated using the hash value of the word sequence "AB". The block hash value of cache block 1 is Hash: (AD), which is calculated using the hash values ​​of the word sequences "AB" and "CD".

[0092] After obtaining the first decoded word "E" during the decoding phase, cache block 2 is filled. At this point, the first-level cache table also includes the mapping relationship between the cache block ID of cache block 2 and its block hash value. The block hash value of cache block 2 is Hash: (AF), which is calculated using the hash values ​​of the word sequences "AB", "CD", and "EF".

[0093] like Figure 7 The diagram shown illustrates the process of requesting and using cache for request 2.

[0094] exist Figure 7 In this context, request 2 can be "ABCFGH", which could be the second request received by the client-side large model. At this point, request 2 is considered the current request. Since request 2 has a length of 6, it can be divided into three word sequences: "AB", "CF", and "GH" according to the cache block size. This means that three cache blocks are needed to store one word sequence each.

[0095] Calculate the hash value of each word in each word sequence, and calculate the block hash value of the cache block corresponding to each word sequence.

[0096] Query the block hash value in the target cache table, which contains a first-level cache table and a second-level cache table.

[0097] After querying, the word sequence "AB" hit the first-level cache table, corresponding to cache block 0, while the word sequences "CF" and "GH" did not hit. Therefore, the key-value vectors in cache block 0 can be directly copied to the pre-filling context in the pre-filling stage. Only the key-value vectors of word sequences "CF" and "GH" need to be calculated, which can save 33% of the computation.

[0098] After calculating the key-value vectors of the word sequences "CF" and "GH", two cache blocks need to be allocated for the word sequences "CF" and "GH". The pointer of cache block 3 is moved from the head to the tail of the free cache queue to cache the key-value vector of the word sequence "CF". The block hash value of cache block 3 is Hash: (ABCF), which means that it is calculated by the hash values ​​of the word sequences "AB" and "CF".

[0099] At this point, traversing the free cache queue reveals that all cache blocks are marked as cached. The earliest allocated cache block, which hasn't been used by other requests, needs to be evicted to the secondary cache pool. Starting from the head of the free cache queue, the cache blocks are traversed again. The first cache block is now the first cache block, i.e., cache block 1. According to the LRU eviction principle, the pointer to cache block 1 is moved to the tail of the free cache queue, indicating that it needs to be reused to cache the key-value vector of the term sequence "GH". At this point, the key-value vector in cache block 1 needs to be persisted to the secondary cache pool, and the secondary cache table is updated synchronously. The secondary cache table stores the mapping between the block hash value of cache block 1 and its cache address (i.e., SSD address) in the secondary cache pool, where the block hash value of cache block 1 is Hash: (AD).

[0100] Then, the key-value vector of the word sequence "GH" is overwritten into cache block 1, and decoding continues. The block hash value of cache block 1 is updated to Hash: (ABCFGH), which represents the hash value calculated from the word sequences "AB", "CF", and "GH". The free cache queue is updated to 0231.

[0101] like Figure 8 The diagram shown illustrates the process of requesting and using cache for request 3.

[0102] exist Figure 8 In this context, request 3 can be "ABCDGH", which can be the third or any subsequent request received by the client-side large model. In this case, request 3 is considered the current request. Since request 3 has a length of 6, request 2 can be divided into three word sequences: "AB", "CD", and "GH" according to the cache block size. This means that three cache blocks are needed to store one word sequence each.

[0103] Calculate the hash value of each word in each word sequence, and calculate the block hash value of the cache block corresponding to each word sequence.

[0104] Query the block hash value in the target cache table, which contains a first-level cache table and a second-level cache table.

[0105] After querying, the word sequence "AB" hits the first-level cache table, corresponding to cache block 0, while the word sequence "CD" hits the second-level cache table. Therefore, the key-value vector in cache block 0 can be directly copied to the pre-filling context of the pre-filling stage, and the key-value vector in the cache address in the second-level cache pool can be uploaded to the first-level cache pool for use. The cache blocks are traversed again from the head of the free cache queue. The second cache block becomes the third cache block, i.e., cache block 2. According to the LRU cache eviction principle, the pointer of cache block 2 is moved to the tail of the free cache queue, indicating that it needs to be reused to cache the key-value vector of the word sequence "CD". The free cache queue is updated to 0312. At this point, the key-value vector in cache block 2 needs to be cached and persisted to the second-level cache pool, and the second-level cache table is updated synchronously. The second-level cache table stores the mapping relationship between the block hash value of cache block 2 and its cache address (i.e., SSD address) in the second-level cache pool, where the block hash value of cache block 2 is Hash: (AF).

[0106] Although cache block 1 currently stores the key-value vector of the lexical sequence "GH", the preceding lexical sequence of "GH" is "ABCF", which is inconsistent with the preceding lexical sequence "ABCD" of the lexical sequence "GH" in request 3. Therefore, a new cache block needs to be requested.

[0107] Starting from the head of the free cache queue, the cache blocks are traversed again. The third cache block is the third cache block, i.e., cache block 1. According to the LRU cache eviction principle, the pointer of cache block 1 is moved to the tail of the free cache queue, indicating that it needs to be reused to cache the key-value vector of the word sequence "CD". The free cache queue is updated to 0321. At this time, the key-value vector in cache block 1 needs to be persisted to the secondary cache pool, and the secondary cache table is updated synchronously. The secondary cache table stores the mapping relationship between the block hash value of cache block 1 and its cache address (i.e., SSD address) in the secondary cache pool, where the block hash value of cache block 1 is Hash: (ABCFGH).

[0108] like Figure 9 As shown, based on the above embodiments, this embodiment of the invention provides a key-value cache reuse device based on a large end-side model, comprising: The word segmentation module 91 is used to obtain the current request and determine the current word sequence of the current request based on the size of the cache block in the first-level cache pool of the device memory. The calculation module 92 is used to calculate the hash value of each current word sequence, and based on the hash value of each current word sequence, calculate the block hash value of the cache block corresponding to each current word sequence; The reuse module 93 is used to determine the target cache location corresponding to the first lexical sequence that hits the target cache table based on the target cache table, and extract the key-value vector in the target cache location; the target cache table is used to store the mapping relationship between the block hash value and the cache location identifier of the cache block corresponding to each historical lexical sequence of the historical request. The caching module 94 is used to calculate the key-value vector of the second word sequence in the pre-filling stage based on the end-side large model for the second word sequence whose hash value does not match the target cache table, apply for a target cache block in the first-level cache pool, and cache the key-value vector of the second word sequence in the target cache block.

[0109] Based on the above embodiments, the key-value cache reuse device based on a large end-side model provided in this embodiment of the invention, wherein the calculation module is specifically used for: For any lexical sequence in each current lexical sequence, the block hash value of the cache block corresponding to the lexical sequence is calculated based on the hash value of the lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence in each current lexical sequence.

[0110] Based on the above embodiments, the key-value cache reuse device based on a large end-side model provided in this embodiment of the invention, wherein the calculation module is specifically used for: If the cache block corresponding to any word sequence is not filled, then the cache block corresponding to any word sequence is filled based on the decoded words obtained by the end-side large model in the decoding stage. If the cache block corresponding to any given lexical sequence is already full, then the block hash value of the cache block corresponding to any given lexical sequence is calculated based on the hash value of the given lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence.

[0111] Based on the above embodiments, the key-value cache reuse device based on a large end-side model provided in this embodiment of the invention includes a first-level cache table and a second-level cache table in the target cache table; The first-level cache table is used to cache the mapping relationship between the block hash value of the cache block corresponding to the historical lexical sequence and the cache block identifier in the first-level cache pool; The second-level cache table is used to cache the mapping relationship between the block hash value of the cache block corresponding to the historical lexical sequence and the cache address identifier in the second-level cache pool of the solid-state drive; the cache address corresponding to the second-level cache table persists the key-value vector of the historical lexical sequence.

[0112] Based on the above embodiments, the key-value cache reuse device based on a large end-side model provided in this embodiment of the invention, wherein the cache module is specifically used for: If the first-level cache pool is full, the key-value vector in the first cache block that was allocated earliest in the first-level cache pool and is not currently used by other requests is cached in the cache address of the second-level cache pool, and the first cache block is used as the target cache block to update the first-level cache table and the second-level cache table. If the first-level cache pool is not full, then the unused second cache block in the first-level cache pool is used as the target cache block, and the first-level cache table is updated.

[0113] Based on the above embodiments, the key-value cache reuse device based on a large end-side model provided in this embodiment of the invention, wherein the reuse module is specifically used for: If the target cache location is a cache block in the first-level cache pool, then the key-value vector in the target cache location is copied to the pre-fill context of the pre-fill stage; If the target cache location is a cache address in the secondary cache pool, then the key-value vector in the third cache block that is the earliest allocated and used in the primary cache pool and is not currently used by other requests is cached in the cache address of the secondary cache pool, and the key-value vector in the target cache location is uploaded to the third cache block.

[0114] Based on the above embodiments, the key-value cache reuse device based on a large end-side model provided in this embodiment of the invention further includes a cache module specifically used for: Based on the idle cache queue, request the target cache block in the first-level cache pool; The idle cache queue is used to manage the idle cache blocks in the first-level cache pool, and the data structure of the idle cache queue is a least recently used queue implemented with a doubly linked list.

[0115] Specifically, the functions of each module in the key-value cache reuse device based on the large end-side model provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0116] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the key-value cache reuse method based on the large edge model provided in the above embodiments.

[0117] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the key-value cache reuse method based on the large end-side model provided in the above embodiments.

[0119] In another aspect, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the key-value cache reuse method based on the large edge model provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium, and is not specifically limited herein.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A key-value cache reuse method based on a large edge model, characterized in that, include: Obtain the current request, and determine the current word sequence of the current request based on the cache block size in the device's first-level cache pool; Calculate the hash value of each current word sequence, and based on the hash value of each current word sequence, calculate the block hash value of the cache block corresponding to each current word sequence; For the first lexical sequence whose block hash value matches the target cache table, the target cache location corresponding to the first lexical sequence is determined based on the target cache table, and the key-value vector in the target cache location is extracted; the target cache table is used to cache the mapping relationship between the block hash value and the cache location identifier of the cache block corresponding to each historical lexical sequence of the historical request. For the second lexical sequence whose hash value does not match the target cache table, based on the end-side large model, during the pre-filling stage, the key-value vector of the second lexical sequence is calculated, a target cache block is requested in the first-level cache pool, and the key-value vector of the second lexical sequence is cached in the target cache block.

2. The key-value cache reuse method based on a large end-side model according to claim 1, characterized in that, The step of calculating the block hash value of the cache block corresponding to each current lexical sequence based on the hash value of each current lexical sequence includes: For any lexical sequence in each current lexical sequence, the block hash value of the cache block corresponding to the lexical sequence is calculated based on the hash value of the lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence in each current lexical sequence.

3. The key-value cache reuse method based on a large end-side model according to claim 2, characterized in that, The calculation of the block hash value of the cache block corresponding to any given lexical sequence, based on the hash value of any given lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence of each given lexical sequence in the current lexical sequence, includes: If the cache block corresponding to any word sequence is not filled, then the cache block corresponding to any word sequence is filled based on the decoded words obtained by the end-side large model in the decoding stage. If the cache block corresponding to any given lexical sequence is already full, then the block hash value of the cache block corresponding to any given lexical sequence is calculated based on the hash value of the given lexical sequence and the block hash value of the cache block corresponding to the preceding lexical sequence.

4. The key-value cache reuse method based on a large end-side model according to claim 1, characterized in that, The target cache table includes a first-level cache table and a second-level cache table; The first-level cache table is used to cache the mapping relationship between the block hash value of the cache block corresponding to the historical lexical sequence and the cache block identifier in the first-level cache pool; The second-level cache table is used to cache the mapping relationship between the block hash value of the cache block corresponding to the historical lexical sequence and the cache address identifier in the second-level cache pool of the solid-state drive; the cache address corresponding to the second-level cache table persists the key-value vector of the historical lexical sequence.

5. The key-value cache reuse method based on a large end-side model according to claim 4, characterized in that, The step of requesting the target cache block in the first-level cache pool includes: If the first-level cache pool is full, the key-value vector in the first cache block that was allocated earliest in the first-level cache pool and is not currently used by other requests is cached in the cache address of the second-level cache pool, and the first cache block is used as the target cache block to update the first-level cache table and the second-level cache table. If the first-level cache pool is not full, then the unused second cache block in the first-level cache pool is used as the target cache block, and the first-level cache table is updated.

6. The key-value cache reuse method based on a large end-side model according to claim 4, characterized in that, The step of extracting the key-value vector from the target cache location includes: If the target cache location is a cache block in the first-level cache pool, then the key-value vector in the target cache location is copied to the pre-fill context of the pre-fill stage; If the target cache location is a cache address in the secondary cache pool, then the key-value vector in the third cache block that is the earliest allocated and used in the primary cache pool and is not currently used by other requests is cached in the cache address of the secondary cache pool, and the key-value vector in the target cache location is uploaded to the third cache block.

7. The key-value cache reuse method based on a large end-side model according to any one of claims 1-6, characterized in that, The step of requesting the target cache block in the first-level cache pool also includes: Based on the idle cache queue, request the target cache block in the first-level cache pool; The idle cache queue is used to manage the idle cache blocks in the first-level cache pool, and the data structure of the idle cache queue is a least recently used queue implemented with a doubly linked list.

8. A key-value cache reuse device based on a large edge model, characterized in that, include: The word segmentation module is used to obtain the current request and determine the current word sequence of the current request based on the size of the cache block in the first-level cache pool of the device memory. The calculation module is used to calculate the hash value of each current word sequence, and based on the hash value of each current word sequence, calculate the block hash value of the cache block corresponding to each current word sequence; The reuse module is used to determine the target cache location corresponding to the first lexical sequence that hits the target cache table based on the target cache table, and extract the key-value vector in the target cache location; the target cache table is used to store the mapping relationship between the block hash value and the cache location identifier of the cache block corresponding to each historical lexical sequence of the historical request. The caching module is used to calculate the key-value vector of the second word sequence in the pre-filling stage based on the end-side large model for a second word sequence whose hash value does not match the target cache table, apply for a target cache block in the first-level cache pool, and cache the key-value vector of the second word sequence in the target cache block.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the key-value cache reuse method based on the large end-side model as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the key-value cache reuse method based on the large end-side model as described in any one of claims 1-7.

Images