Cache management method, cache management scheme determination method, large language model inference system and method

Abstract

The application relates to a cache management method, a cache management scheme determination method, a large language model inference system and a method. The cache management method is applied to a memory manager, and the memory manager is used for managing key-value memory resources of a large language model and comprises the following steps: acquiring key-value vectors of input tokens carried in inference requests; and storing the key-value vectors of the input tokens into a shared memory resource in the key-value memory resources; wherein each inference request corresponds to at least two inference sequences, and the key-value vectors of the input tokens corresponding to each inference request in the shared memory resource are stored only once; acquiring key-value vectors of output tokens respectively generated by each inference sequence corresponding to an inference request in an inference process; and storing the key-value vectors of the output tokens in each inference sequence corresponding to the inference request into an independent memory resource in the key-value memory resources. By using the method, the memory resource utilization rate of the KV cache can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a cache management method, a method for determining a cache management scheme, and a large language model inference system and method. Background Technology

[0002] With the development of Artificial Intelligence (AI) and computer technology, AI-based human-computer dialogue, knowledge-based question answering, document processing, and content creation have become major trends in technological development. Large Language Models (LLMs), as the foundation for realizing these intelligent functions, have attracted much attention regarding efficient operation and resource utilization. Among these, the effective utilization of memory resources is a current research hotspot.

[0003] In related technologies, LLM models require frequent memory copying, reorganization, and splicing of key-value (KV) caches between sequences during inference, which requires a large amount of memory resources and results in severe memory fragmentation, leading to low memory resource utilization.

[0004] Therefore, how to reduce memory resource usage and improve the memory resource utilization of KV cache is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] Based on this, it is necessary to provide a cache management method, a cache management scheme determination method, a large language model inference system, a large language model inference method, an apparatus, a computer device, a computer-readable storage medium, and a computer program product that can reduce memory resource usage and improve the memory resource utilization of KV cache, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a cache management method applied to a memory manager, which manages key-value memory resources of a large language model. The method includes:

[0007] Obtain the key-value vectors of each input word carried in the inference request;

[0008] The key-value vectors of each input word are stored in the shared memory resource of the key-value memory resource; wherein, each inference request corresponds to at least 2 inference sequences, and only one copy of the key-value vector of the input word corresponding to each inference request is stored in the shared memory resource;

[0009] Obtain the key-value vector of the output word generated for each inference sequence corresponding to the inference request during the inference process; the output word in each inference sequence is generated by autoregression based on the key-value vector of each input word in the corresponding inference request in the shared memory resource and the key-value vector of each output word in the inference sequence.

[0010] Store the key-value vector of the output term in each inference sequence corresponding to the inference request into an independent memory resource in the key-value memory resource; the independent memory resource is at least a portion of the memory resource in the key-value memory resource other than the shared memory resource.

[0011] Secondly, this application provides a method for determining a cache management scheme, the method comprising:

[0012] Different cache management schemes are deployed on multiple computing nodes. The different cache management schemes include at least two of the following: hierarchical shared cache, radix tree storage, and memory paging-based shared storage; wherein, the hierarchical shared cache is the cache management method of the first aspect.

[0013] Using a preset batch size, and based on the data distribution of the target inference task, small-batch inference experiments are executed on multiple computing nodes.

[0014] Collect the actual key-value memory utilization efficiency indicators of each computing node in the small-batch inference experiment. The actual key-value memory utilization efficiency indicators include at least one of memory usage, memory fragmentation rate, memory copying times, and memory overflow times.

[0015] The target cache management scheme is determined based on the actual key-value memory utilization efficiency of each cache management scheme.

[0016] Thirdly, this application provides a large language model inference system, which includes multiple computing nodes and a request routing module, wherein:

[0017] Each compute node deploys a different cache management scheme, which includes at least two of the following: hierarchical shared cache, radix tree storage, and memory paging-based shared storage. Each compute node manages key-value memory resources according to its deployed cache management scheme; wherein, the hierarchical shared cache is the cache management method in the first aspect.

[0018] The request routing module is used to receive inference requests, extract the inference task type and data characteristics of the inference request, and route the inference request to the target computing node that best matches the inference task type and data characteristics.

[0019] Fourthly, this application provides an inference method for a large language model, which employs the cache management method described in the first aspect to store and manage key-value vectors during the inference process. This inference method includes:

[0020] Read the key-value vectors of each input word carried in the reasoning request corresponding to each reasoning sequence from the shared memory resources of the key-value memory resources;

[0021] Read the key-value vectors of the output words generated in each round of reasoning for each reasoning sequence from the independent memory resources of the key-value memory resources;

[0022] The key value vectors of each input word corresponding to each reasoning sequence are concatenated with the key value vectors of each output word in each round of reasoning according to the word sequence to obtain the key value sequence.

[0023] The key-value sequence is input into the attention layer of the large language model for inference, generating the key-value vector of the output word for each inference sequence in this round of inference.

[0024] Fifthly, this application provides an inference method for a large language model, which uses the cache management method in the first aspect to store and manage the key-value vectors during the inference process. When the number of input lexical units carried in the inference request is greater than a first preset number and the number of output lexical units corresponding to the inference sequence is greater than a second preset number, or the sum of the number of input lexical units and the number of output lexical units is greater than a third preset number, the inference method further includes:

[0025] For the input lexical key-value vector in the shared memory resource corresponding to the inference request and the query vector of the current inference step, perform the first stage attention score calculation to obtain the first attention score matrix and the first intermediate attention value;

[0026] For the output lexical key vector and query vector in the independent memory resources corresponding to each inference sequence, perform the second stage attention score calculation to obtain the second attention score matrix and the second intermediate attention value;

[0027] The first attention score matrix and the second attention score matrix are normalized, and the first intermediate attention value and the second intermediate attention value are weighted and summed according to the preset normalization weights of each stage to obtain the target attention score matrix and the target intermediate attention value.

[0028] Based on the target attention score matrix and the target intermediate attention value, the key-value vector of the output word for each inference sequence in this round of inference is generated.

[0029] Sixthly, this application also provides a cache management device applied to a memory manager, the memory manager being used to manage key-value memory resources of a large language model, including:

[0030] The first acquisition module is used to acquire the key-value vectors of each input word carried in the inference request;

[0031] The first storage module is used to store the key-value vectors of each input word into the shared memory resource in the key-value memory resource; wherein, each inference request corresponds to at least 2 inference sequences, and the key-value vector of the input word corresponding to each inference request is stored only once in the shared memory resource;

[0032] The second acquisition module is used to acquire the key value vectors of the output words generated by each inference sequence corresponding to the inference request during the inference process; the output words in each inference sequence are generated by autoregression based on the key value vectors of each input word in the corresponding inference request in the shared memory resource and the key value vectors of each output word in the inference sequence.

[0033] The second storage module is used to store the key-value vector of the output word in each inference sequence corresponding to the inference request into an independent memory resource in the key-value memory resource; the independent memory resource is at least a portion of the memory resource in the key-value memory resource other than the shared memory resource.

[0034] Seventhly, this application also provides an apparatus for determining a cache management scheme, the apparatus comprising:

[0035] The solution deployment module is used to deploy different cache management schemes on multiple computing nodes. The different cache management schemes include at least two of the following: hierarchical shared cache, radix tree storage, and memory paging-based shared storage; wherein, the hierarchical shared cache adopts the cache management method of the first aspect.

[0036] The inference experiment module is used to execute small-batch inference experiments on multiple computing nodes based on the data distribution of the target inference task and using a preset batch size.

[0037] The metrics collection module is used to collect the actual key-value memory utilization efficiency metrics of each computing node in the small-batch inference experiment. The actual key-value memory utilization efficiency metrics include at least one of memory usage, memory fragmentation rate, memory copying times, and memory overflow times.

[0038] The scheme determination module is used to determine the target cache management scheme based on the actual key-value memory utilization efficiency index of each cache management scheme.

[0039] Eighthly, this application also provides an inference apparatus for a large language model, which employs the cache management method described in the first aspect to store and manage key-value vectors during the inference process. The apparatus includes:

[0040] The first reading module is used to read the key-value vectors of each input word carried in the reasoning request corresponding to each reasoning sequence from the shared memory resources of the key-value memory resources;

[0041] The second reading module is used to read the key-value vectors of the output words generated in each round of reasoning for each reasoning sequence from the independent memory resources of the key-value memory resources;

[0042] The concatenation module is used to concatenate the key-value vectors of each input word corresponding to each reasoning sequence with the key-value vectors of each output word in each round of reasoning according to the word sequence to obtain the key-value sequence.

[0043] The inference module is used to input the key-value sequence into the attention layer of the large language model for inference, generating the key-value vector of the output word corresponding to each inference sequence in this round of inference.

[0044] Ninthly, this application also provides an inference apparatus for a large language model, which employs the cache management method in the first aspect to store and manage key-value vectors during the inference process. When the number of input lexical units carried in the inference request is greater than a first preset number and the number of output lexical units corresponding to the inference sequence is greater than a second preset number, or the sum of the number of input lexical units and the number of output lexical units is greater than a third preset number, the apparatus includes:

[0045] The first attention calculation module is used to perform the first stage attention score calculation on the input lexical key value vector in the shared memory resource corresponding to the inference request and the query vector of the current inference step, so as to obtain the first attention score matrix and the first intermediate attention value.

[0046] The second attention calculation module is used to perform second-stage attention score calculation on the output lexical key value vector and query vector in the independent memory resources corresponding to each inference sequence, so as to obtain the second attention score matrix and the second intermediate attention value.

[0047] The fusion module is used to normalize the first attention score matrix and the second attention score matrix, and to perform a weighted summation of the first intermediate attention value and the second intermediate attention value according to the preset normalization weights of each stage to obtain the target attention score matrix and the target intermediate attention value.

[0048] The inference module is used to generate the key-value vector of the output word for each inference sequence in this round of inference, based on the target attention score matrix and the target intermediate attention value.

[0049] In a tenth aspect, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the cache management method in the first aspect, the cache management scheme determination method in the second aspect, the large language model reasoning method in the fourth aspect, and the large language model reasoning method in the fifth aspect.

[0050] In the eleventh aspect, this application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the cache management method in the first aspect, the cache management scheme determination method in the second aspect, the large language model inference method in the fourth aspect, and the large language model inference method in the fifth aspect.

[0051] In a twelfth aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the cache management method in the first aspect, the cache management scheme determination method in the second aspect, the large language model inference method in the fourth aspect, and the large language model inference method in the fifth aspect.

[0052] The aforementioned cache management method, cache management scheme determination method, large language model inference system, large language model inference method, device, computer equipment, storage medium, and computer program product, wherein the cache management method is applied to a memory manager, which manages the key-value memory resources of the large language model. The memory manager obtains the key-value vectors of each input lexical carried in the inference request and stores the key-value vectors of each input lexical in the shared memory resources of the key-value memory resources. Each inference request corresponds to at least two inference sequences, and only one copy of the key-value vector of the input lexical corresponding to each inference request is stored in the shared memory resources. Furthermore, the method obtains the key-value vectors of the output lexical generated by each inference sequence corresponding to the inference request during the inference process and stores the key-value vectors of the output lexical in each inference sequence corresponding to the inference request in the independent memory resources of the key-value memory resources. The output lexical in each inference sequence is generated autoregressively based on the key-value vectors of each input lexical in the corresponding inference request and the key-value vectors of each output lexical in the inference sequence in the shared memory resources. The independent memory resources are at least a portion of the memory resources other than the shared memory resources in the key-value memory resources. In other words, the method proposed in this application configures the key-value memory resources into two parts: shared memory resources and independent memory resources. The key-value vectors of the input words shared by each inference sequence corresponding to the inference request are stored in the shared memory resources, while the key-value vectors of the output words corresponding to each inference sequence are stored in the independent memory resources. For each inference sequence, during the inference process, there is no need to frequently copy, reassemble, concatenate, allocate memory, or copy the shared parts, nor is there a need to reserve a large cache space in advance. This not only reduces cache usage resources but also solves the memory fragmentation problem, reduces memory waste, and improves the utilization rate of memory resources. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating a cache management method in one embodiment;

[0055] Figure 2 This is a flowchart illustrating the cache management method in another embodiment;

[0056] Figure 3 This is a flowchart illustrating the cache management method in another embodiment;

[0057] Figure 4This is a flowchart illustrating the cache management method in another embodiment;

[0058] Figure 5 This is a flowchart illustrating the cache management method in another embodiment;

[0059] Figure 6 This is a flowchart illustrating a method for determining a cache management scheme in one embodiment;

[0060] Figure 7 This is a flowchart illustrating the method for determining a cache management scheme in another embodiment;

[0061] Figure 8 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0062] Figure 9 This is a flowchart illustrating the reasoning method of a large language model in another embodiment;

[0063] Figure 10 This is a structural block diagram of a cache management device in one embodiment;

[0064] Figure 11 This is a structural block diagram of a device for determining a cache management scheme in one embodiment;

[0065] Figure 12 This is a structural block diagram of the reasoning device for a large language model in one embodiment;

[0066] Figure 13 This is a structural block diagram of the reasoning device for a large language model in another embodiment;

[0067] Figure 14 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] Traditional LLM services require frequent memory copying, reorganization, and splicing of key-value (KV) caches between candidate sequences (each inference request corresponds to multiple candidate sequences) during beam search. This necessitates allocating a sufficiently large cache space (accommodating Beam_Size * Max_Length) for subsequent KV cache copying of the candidate beam stream during beam search, and then avoiding frequent memory allocation and copying by maintaining pointers or indexes. Beam search, within a single task, aims to find the optimal generated sequence, retaining the k highest-scoring candidate sequences (Beams) at each step. The challenge of beam search lies in the fact that after each step, only the k retained sequences are discarded. This means the KV cache must dynamically keep up with the "generation" and "death" of candidate sequences, exacerbating memory fragmentation and further reducing memory resource utilization. Furthermore, because the KV cache for each candidate sequence needs to be stored contiguously in existing LLM inference, there is also a serious problem of KV cache redundancy among multiple candidate sequences.

[0070] Initial state:

[0071] Assuming Beam_Size = 2, i.e. k=2, initially there is only one input prompt. In the first round of generation, starting from the prompt, the probability distribution of the next token is calculated, and the two tokens with the highest probabilities are selected to form two candidate sequences.

[0072] Copying the cache:

[0073] In the first step, the initial key-value cache (corresponding to the Prompt section) needs to be copied twice. This is because there are now two different branches (e.g., "Prompt + A" and "Prompt + B") that share the key-value cache of the Prompt section, but subsequent calculations need to be performed independently.

[0074] Cache extensions:

[0075] For these two candidate sequences, calculate the KV values ​​of their newly generated tokens (A and B) and append them to their respective copied Prompt KV caches.

[0076] Cache reordering and pruning:

[0077] Upon moving to the next step, each Beam generates V (vocabulary size) new candidates, for a total of 2*V candidates. Each Beam then needs to select the two highest-scoring candidates. The score for each candidate sequence is typically calculated as the sum of log probabilities (i.e., the cumulative log probability of the sequence). These candidates are sorted from highest to lowest score, and only the k highest-scoring candidate sequences are retained as new Beams (here, 2). These two newly selected optimal candidates may come from the same parent Beam or from different parent Beams. Example explanation:

[0078] Suppose the input prompt is "I love", and the vocabulary is simplified to only a few words, k=2.

[0079] Step 1: Model predicts the probability of the next token:

[0080] "cats" (0.4), "dogs" (0.3), "reading" (0.2), "to" (0.1).

[0081] Keep top-2: Beam1: "I love cats" (score=log0.4), Beam2: "I love dogs" (score=log0.3)

[0082] Step 2: Expand the two beams respectively:

[0083] Expanding from Beam1: we might get "I love cats and" (0.5), "I love cats ." (0.3), ...; expanding from Beam2: "I love dogs and" (0.4), "I love dogs !" (0.3), ...

[0084] Calculate the scores (cumulative log probabilities) of all candidates and select the top-2:

[0085] Assume that the final choices are Beam1's "I love cats and" (score -1.2) and Beam2's "I love dogs and" (score -1.4).

[0086] ... Continue until the end.

[0087] Finally, assuming the generation is complete, the complete sequences and scores of the two beams are as follows:

[0088] Beam A: “I love cats and they are cute.” (score -5.2)

[0089] Beam B: “I love dogs and they are loyal.” (score -4.8)

[0090] The final output will then be the sequence of Beam B, because it has a higher score.

[0091] Therefore, in beam search, the key-value cache needs to support dynamic copying and reorganization. After each step of selecting the Top-K Beams, the key-value cache needs to be reorganized to ensure that each Beam has its complete and correct historical key-value cache for the next step of calculation.

[0092] To address the issue of low memory resource utilization in KV caches caused by the need for memory copying, reorganization, and splicing of each output sequence during beam search or parallel sampling output, this application proposes a cache management method and a method for determining a cache management scheme, which can largely avoid redundant storage and improve memory resource utilization.

[0093] The cache management method, cache management scheme determination method, and large language model inference method provided in this application can all be applied to single-machine or Internet server application environments. The server can serve as a computing node in the inference service, possessing characteristics such as high computing power, multiple GPUs, and large memory. It is commonly used to execute multi-path, distributed inference tasks and can be implemented using a standalone server or a server cluster composed of multiple servers.

[0094] For example, the server may include a memory manager for managing the key-value memory resources of a large language model, that is, storing the key-value (KV) memory generated by each inference request during the inference process in the key-value memory resources, thereby improving the utilization of key-value memory resources.

[0095] In one exemplary embodiment, such as Figure 1 As shown, a cache management method is provided. Taking the application of this method to a server, such as a memory manager in a server, as an example, the method includes the following steps 101 to 104. Wherein:

[0096] Step 101: Obtain the key-value vector of each input word carried in the inference request.

[0097] For example, the server can process a single inference request or process multiple inference requests in parallel using a batching approach. For each inference request, each inference request can correspond to at least two inference sequences, meaning that each inference step can retain multiple candidate sequences with the highest scores, or sample and output multiple output sequences in parallel.

[0098] Taking a single inference request as an example, when the server responds to an inference request, it can parse the input content carried in the inference request and convert the input content into at least one input token. Then, each input token is input into a large language model for analysis and processing. The key-value vector of each input token can be obtained through the large language model.

[0099] For example, if the key-value vectors of each input word carried in the inference request are obtained, the key-value vectors of each input word in the inference request can be sent to the memory manager so that the memory manager can store the key-value vectors of each input word in the inference request, that is, store the key-value vectors of each input word in the inference request in the key-value memory resource.

[0100] Step 102: Store the key-value vector of each input word into the shared memory resource in the key-value memory resource.

[0101] The key-value memory resources include shared memory resources and independent memory resources. Shared memory resources can be used to store the key-value vectors of the input words for each inference request, while independent memory resources can be used to store the key-value vectors of the output words generated by each inference sequence during the inference process.

[0102] For example, when an inference request corresponds to multiple inference sequences, the key-value vector of the input word corresponding to each inference request is stored only once in the shared memory resource. That is, multiple inference sequences corresponding to an inference request share the same key-value vector of the input word corresponding to that inference request.

[0103] It should be noted that shared memory resources can be static memory resources of a preset size or dynamic memory resources. Each inference request can correspond to its own shared memory resource, or multiple inference requests can correspond to one shared memory resource. For example, multiple inference requests in a batch can correspond to one shared memory resource. In addition, it should be noted that shared memory resources can be contiguous or non-contiguous memory resources.

[0104] In other words, during beam search or parallel sampling output, all inference sequences (beams) for the same inference request share the same input prompt, thus the key-value cache for the prompt portion is completely identical. In this example, a hierarchical shared cache scheme is used to manage key-value memory resources. This scheme includes a shared layer and independent layers. The shared layer stores the key-value cache for the input prompt (including multiple input tokens) of at least one inference request, maintaining only one copy of the input key-value cache for all request data corresponding to the inference request. The independent layer maintains a key-value cache for each inference sequence (beam) and parallel sampling sequence for each inference request, containing its own generated output tokens. The difference lies in the frequent copy-on-write operations due to output forking inference sequences and cache clearing operations for inference sequence eviction.

[0105] Based on this, once the key-value vectors of each input word in the inference request are obtained, the key-value vectors of each input word in the inference request can be stored in the shared memory resource of the key-value memory resource.

[0106] Step 103: Obtain the key-value vector of the output word generated for each inference sequence corresponding to the inference request during the inference process.

[0107] In each inference sequence, the output lexical units are generated by autoregression based on the key-value vectors of the input lexical units in the corresponding inference request in the shared memory resource and the key-value vectors of the output lexical units in the inference sequence.

[0108] Continuing with the example of a single inference request, during inference, the large language model first obtains the key-value vectors of each input word corresponding to the inference request from the shared memory resource, and performs the first round of inference based on the key-value vectors of each input word to obtain multiple candidate sequences. Then, it determines the top-k candidate sequences from the multiple candidate sequences as the inference sequences for the subsequent inference process. Each inference sequence includes at least one key-value vector of an output word, and these key-value vectors of output words will be stored in the independent memory resource of the key-value memory resource.

[0109] For each inference sequence in the intermediate rounds of inference, excluding the first round, the large language model needs to obtain the key-value vectors of each input word corresponding to the inference request from the shared memory resources, and the key-value vectors of the output words already generated in the inference sequence from the independent memory resources. Then, based on the key-value vectors of each input word corresponding to the inference request and the key-value vectors of the generated output words, autoregressive generation is performed to obtain the key-value vectors of the output words of each inference sequence corresponding to the intermediate rounds of inference.

[0110] For example, when storing the key-value vectors of each input word corresponding to the inference request, reference information of the key-value vectors of each input word can be further determined; the reference information is used to read the key-value vectors of each input word from the shared memory resource, and to perform the inference process to generate the key-value vectors of the output word.

[0111] In other words, during inference, the key-value caches in the shared and independent layers can be retrieved and concatenated into complete context input computation units for attention computation based on the physical address encoding in the linked list corresponding to each inference sequence, thereby largely avoiding redundant storage. This is especially beneficial for inference tasks with long context inputs, as it can further reduce memory redundancy and improve memory utilization.

[0112] Next, the key-value vectors of the output tokens of each inference sequence are sent to the memory manager so that the memory manager can store the key-value vectors of the output tokens of each inference sequence of the inference request into independent memory resources.

[0113] Step 104: Store the key-value vector of the output word in each inference sequence corresponding to the inference request into an independent memory resource in the key-value memory resource.

[0114] Independent memory resources are at least a portion of key-value memory resources other than shared memory resources.

[0115] For example, independent memory resources can be stored in at least one of the following ways: contiguous storage, paging storage, and radix tree storage. Contiguous storage refers to storing a single block of memory sequentially, such as storing the key-value vectors of the output terms of each inference sequence in a contiguous memory region. Paging storage involves dividing the independent memory resource into pages of the same or different sizes, associating them with physical addresses of different pages through contiguous logical addresses, so that the key-value vectors of the output terms of each inference sequence are stored in non-contiguous memory pages, significantly reducing memory waste and improving memory utilization. Radix tree storage uses a tree structure, storing candidate sequences according to the relationships between them. All candidate sequences share their prefixes, eliminating the need for additional reference counting and copy-on-write mechanisms.

[0116] The aforementioned cache management method is applied to a memory manager, which manages the key-value memory resources of a large language model. The memory manager obtains the key-value vectors of each input word carried in the inference request and stores them in the shared memory resources of the key-value memory resources. Each inference request corresponds to at least two inference sequences, and only one copy of the key-value vector of the input word corresponding to each inference request is stored in the shared memory resources. Next, the key-value vectors of the output words generated by each inference sequence corresponding to the inference request during the inference process are obtained, and the key-value vectors of the output words in each inference sequence corresponding to the inference request are stored in the independent memory resources of the key-value memory resources. The output words in each inference sequence are generated by autoregression based on the key-value vectors of each input word in the corresponding inference request and the key-value vectors of each output word in the inference sequence in the shared memory resources. The independent memory resources are at least a portion of the memory resources in the key-value memory resources other than the shared memory resources. In other words, the method proposed in this application configures the key-value memory resources into two parts: shared memory resources and independent memory resources. The key-value vectors of the input words shared by each inference sequence corresponding to the inference request are stored in the shared memory resources, while the key-value vectors of the output words corresponding to each inference sequence are stored in the independent memory resources. For each inference sequence, during the inference process, there is no need to frequently copy, reassemble, concatenate, allocate memory, or copy the shared parts, nor is there a need to reserve a large cache space in advance. This not only reduces cache usage resources but also solves the memory fragmentation problem, reduces memory waste, and improves the utilization rate of memory resources.

[0117] In one exemplary embodiment, the memory manager can allocate dynamic memory resources for each inference request as a shared memory resource for each inference request. Based on this, and building upon the above embodiments, as... Figure 2 As shown, steps 201 to 203 may be included before step 102 above. Wherein:

[0118] Step 201: Obtain the number of input tokens carried in the inference request.

[0119] Step 202: Determine dynamic memory resources from the key-value memory resources that match the number of input tokens.

[0120] Step 203: Treat the dynamic memory resource as a shared memory resource.

[0121] For example, before storing the key-value vectors of each input lexical into the shared memory resource in the key-value memory resource, the shared memory resource corresponding to the inference request can be determined first, that is, the shared memory resource matching the inference request can be allocated from the key-value memory resource.

[0122] In one alternative implementation, the memory manager can allocate a certain amount of dynamic memory resources to the inference request based on the number of input tokens corresponding to the inference request, as a shared memory resource corresponding to the inference request, wherein the memory size of the shared memory resource corresponding to the inference request is greater than or equal to the memory size required for the number of input tokens corresponding to the inference request.

[0123] For example, the memory manager can first obtain the memory size required for a single lexical, and then determine the minimum memory size corresponding to the inference request based on the number of input lexicals in the inference request and the memory size required for a single lexical. This minimum memory size can satisfy the key-value vector storage of each input lexical in the inference request. Based on this, the memory manager can determine a memory region with contiguous physical addresses that at least meets the minimum memory size from the key-value memory resources, and allocate this memory region to the inference request as the dynamic memory resource of the inference request, that is, as the shared memory resource of the inference request. Specifically, a contiguous memory region that is the minimum memory size can be used as the shared memory resource of the inference request; alternatively, a contiguous memory region whose size is greater than the minimum memory size and is a power of 2 can be used as the shared memory resource of the inference request; or, a contiguous memory region whose size is greater than the minimum memory size and is an integer can be used as the shared memory resource of the inference request.

[0124] The memory required for a single word can be related to the model structure of the large language model and the storage format of the key-value vector. The model structure of the large language model can include the number of neural network layers, the number of attention heads in each layer, and the dimension of each attention head. For example, if a language model has 9 neural network layers, each layer has 2 attention heads, and each attention head includes a 712-dimensional vector, the K and V element values ​​corresponding to each dimension of the vector are stored in BF16 format, each requiring 2 bytes of storage space. Therefore, storing the entire KV cache for a single word requires 9*2*712*2*2=51264 bytes ≈ 50.1KB.

[0125] In one optional implementation, the shared memory resource can also employ any of several storage methods, such as contiguous storage, paged storage, or radix tree storage. The memory manager can determine the storage method of the shared memory resource for the inference request based on the number of input tokens corresponding to the inference request. For example, the memory manager can determine whether the number of input tokens corresponding to the inference request is greater than or equal to a preset number of tokens. If the number of input tokens is greater than or equal to the preset number of tokens, a dynamic memory resource matching the number of input tokens is determined from the key-value memory resource using contiguous storage, and then used as the shared memory resource for the inference request. Conversely, if the number of input tokens is less than the preset number of tokens, a dynamic memory resource matching the number of input tokens is determined from the key-value memory resource using either paged storage or radix tree storage.

[0126] In this embodiment, shared memory resources are dynamically allocated to inference requests based on the number of input tokens, so that the allocated shared memory resources can meet and match the storage requirements of the inference requests. This not only avoids the problem of memory waste caused by excessive allocation of shared memory, but also avoids the problem of storage discontinuity and storage resource dispersion caused by insufficient allocation of shared memory. Reasonable allocation of shared memory resources can maximize the utilization rate of memory resources.

[0127] Furthermore, the storage method of shared memory for inference requests is determined based on the number of input tokens. That is, the shared memory storage method is reasonably determined according to the needs of different inference scenarios. For example, when the input content is long and the number of input tokens is large, a contiguous storage method is adopted. Contiguous shared memory can reduce the number of memory splicing and improve memory management efficiency. Using paging memory or radix trees can eliminate memory fragmentation and dynamic allocation overhead, which is suitable for short input content, can reduce memory fragmentation, and further improve memory utilization. In other words, in this example, the input shared memory method is determined based on the length of the input data, which can improve the flexibility of memory management.

[0128] In an exemplary embodiment, before performing inference services or storing key-value vectors generated in inference services, the target cache management method corresponding to the inference service can be determined first. The target cache management method may include the hierarchical shared cache method, radix tree storage method, and memory paging-based shared storage method described above. The hierarchical shared cache method divides key-value memory resources into a shared layer and an independent layer, i.e., the cache management method in the above embodiments. Based on this, and building upon the above embodiments, as follows... Figure 3 As shown, steps 301 to 302 may be included before step 101 above. Wherein:

[0129] Step 301: Obtain the reasoning task type and data characteristics of the reasoning request.

[0130] For example, the inference task type may include, but is not limited to, at least one of parallel sampling output type, beam search output type, and combined output type, wherein the combined output type may be a combination of parallel sampling output type and beam search output type. Furthermore, the data characteristics may include, but are not limited to, at least one of the following data features: number of candidate sequences, input sequence length, output sequence length, beam width, etc.

[0131] For example, the target cache management method corresponding to the inference request can be determined based on the inference task type and data characteristics of the inference request. Before that, the memory manager needs to obtain the inference task type and data characteristics of the inference request. In an optional implementation, the memory manager can obtain the inference task type and data characteristics corresponding to the inference request from the computing unit, management unit, or control unit of the server, or the memory manager can obtain the inference task type and data characteristics corresponding to the inference request from different units of the server respectively.

[0132] Step 302: Based on the inference task type and data characteristics, determine the target cache management method from the hierarchical shared cache method, radix tree storage method and memory paging-based shared storage method, and store and manage the key-value vector of the inference request according to the target cache management method.

[0133] For example, if the inference task type corresponding to the inference request is a parallel sampling output type, the target cache management method can be a hierarchical shared cache method or a shared storage method based on memory paging; further, if the inference task type corresponding to the inference request is a parallel sampling output type, and the data characteristics corresponding to the inference request satisfy the first data characteristic, the target cache management method can be a hierarchical shared cache method; wherein, the first data characteristic may include, but is not limited to, the number of parallel sampling outputs being greater than a first preset value, and / or, the average length of the input data being greater than a second preset value, and / or, the length of the predicted output data being less than a third preset value.

[0134] For situations meeting the first data characteristic, the memory-page-based shared storage method shares less of the last unfilled physical block of the input prompt compared to the hierarchical shared caching method. Therefore, as the number of parallel samples increases, its memory sharing rate advantage becomes less significant than that of the hierarchical shared caching method. Furthermore, the longer the input length, the more blocks the memory-page-based shared storage method shares, leading to higher overhead during concatenation, making it less efficient than the layered design of the hierarchical shared caching method. Additionally, the near-fragmented advantage of the memory-page-based shared storage method becomes less pronounced when the output length is short. For example, if the task type is a multiple-choice question, with the input being the question and the output being only A, B, C, D, then the memory-page-based shared storage method, where each output occupies a physical block, is less efficient than reserving contiguous physical space based on the output length, making it less cost-effective than the simple layering of the hierarchical shared caching method. In other cases, the memory-page-based shared storage method can be used.

[0135] For example, if the inference task type corresponding to the inference request is a bundle search output type, the target cache management method can be a radix tree storage method or a shared storage method based on memory paging; further, if the inference task type corresponding to the inference request is a bundle search output type, and the data characteristics corresponding to the inference request satisfy the second data characteristic, the target cache management method can be a radix tree storage method; wherein, the second data characteristic may include, but is not limited to, the bundle search width being greater than a fourth preset value, and / or, the length of the predicted output data being greater than a fifth preset value.

[0136] For cases that meet the second data characteristics, the hierarchical shared cache approach can handle the beam search situation, and compared with related technologies that individually copy and retain the KV cache of each candidate sequence, including the input prompt, it can share the KV cache of the prompt input part, which improves memory utilization efficiency to a certain extent. However, when the input is short, the beam search width is wide, and the output length is long, there is still a problem that the candidate output sequence is frequently copied and recycled, causing internal and external memory fragmentation that cannot be utilized. Therefore, the hierarchical shared cache approach has little advantage over other solutions.

[0137] When facing bundle search, the wider the bundle search width, the longer the output. The advantage of radix tree storage is more obvious than that of memory-paged shared storage. This is because radix tree storage stores each word's KV cache in units when storing bundle search, with almost no redundant storage. In contrast, memory-paged shared storage, because each physical block includes multiple word KV caches (e.g., 16), even if the first 15 words are the same, if the last word is different and belongs to two different candidate sequences, the entire contents of the physical block must be copied to a new physical block, resulting in a certain degree of redundancy.

[0138] Furthermore, radix tree storage is more complex to control than shared storage based on memory paging. It incurs additional overhead for the metadata of the storage structure corresponding to each node. The structure of each node is used to record information such as which subsequent sequences each node belongs to, the parent-child relationship, and the physical storage location. Therefore, radix tree storage is not cost-effective when the output length is short.

[0139] For example, if the inference task type corresponding to the inference request is a combined output type, that is, the inference request is a parallel sampling output, and each parallel sampling output uses beam search, then the target cache management method can be a shared storage method based on memory paging, which can reduce the complexity of the control system, reduce redundancy, and improve memory utilization.

[0140] It should be noted that, if the target cache management method corresponding to the inference request is determined to be a hierarchical shared cache method, the step of obtaining the key-value vector of each input word carried in the inference request in step 201 above can be performed. That is, the hierarchical shared cache method is used to manage the key-value vectors of the input words and the key-value vectors of the output words of the inference request in a hierarchical manner.

[0141] In this embodiment, based on the inference task type and data characteristics of the inference request, the target cache management method corresponding to the inference request is determined from hierarchical shared cache, radix tree storage, and memory paging-based shared storage. That is, this example proposes a multi-scheme adaptive selection mechanism, a complete decision-making mechanism that dynamically selects the optimal cache management scheme based on the inference task type and data characteristics. This mechanism systematizes the applicable boundaries and selection conditions of the three different sharing schemes, which not only ensures the feasibility of the multi-scheme adaptive selection mechanism, but also improves the flexibility and adjustability of memory management schemes, and further improves the memory resource utilization under different scenario requirements.

[0142] In an exemplary embodiment, the shared memory resource corresponding to the inference request can be released and reclaimed under certain conditions. The release and reclamation of shared memory resources will be described in detail below, based on the above embodiment, such as... Figure 4 As shown, the above cache management method may further include steps 401 to 403. Wherein:

[0143] Step 401: Maintain a reference count for the key-value vector of the input lexical corresponding to each inference request in the shared memory resource. The initial value of the reference count is equal to the total number of inference sequences corresponding to the inference request.

[0144] The total number of inference sequences corresponding to different inference requests can be the same or different. For different inference requests, the reference count value corresponding to each inference request in the shared memory resource can be set according to the total number of inference sequences corresponding to each inference request.

[0145] For example, an array can be used to record the reference count value corresponding to each inference request in the shared memory resource. In subsequent processes, if the reference count value corresponding to the inference request changes, the reference count value recorded in the array can be updated in real time.

[0146] Step 402: When the inference sequence is eliminated in the beam search or the output ends in the parallel sampling process, the independent memory resource corresponding to the inference sequence is marked as reclaimable and released, and the reference count corresponding to the inference request in the shared memory resource is decremented by one.

[0147] During the beam search process, inference sequences outside of TOP-k will be eliminated or pruned. For the eliminated inference sequence corresponding to the inference request, the memory resource corresponding to the inference sequence in the independent memory resource can be marked as reclaimable and released. For example, it can be released synchronously or after a certain period of time.

[0148] In addition, in parallel sampling output, if the output of the inference sequence corresponding to the inference request ends, the memory resource corresponding to the inference sequence in the independent memory resource can also be marked as reclaimable and the memory resource can be released.

[0149] For example, if there is an eliminated or terminated reasoning sequence, the reference count corresponding to the reasoning request in the shared memory resource will be decremented by one until all reasoning sequences corresponding to the reasoning request have ended, that is, until the reference count corresponding to the reasoning request is 0.

[0150] Step 403: When the reference count of the shared memory resource drops to zero, the release operation of the corresponding shared memory resource is triggered, and the memory resource corresponding to the inference request in the shared memory resource is returned to the key-value memory resource.

[0151] For example, when the shared memory resource includes a memory resource corresponding to a reasoning request, when the reference count of the reasoning request drops to zero, a release operation of the shared memory resource corresponding to the reasoning request is triggered, that is, the shared memory resource is returned to the key-value memory resource.

[0152] For example, when the shared memory resource includes memory resources corresponding to multiple inference requests, when the reference count of a certain inference request drops to zero, a release operation is triggered on the memory resource corresponding to that inference request within the shared memory resource, that is, the memory resource corresponding to that inference request within the shared memory resource is returned to the key-value memory resource. When the reference counts of all inference requests within the shared memory resource drop to zero, that is, when the reference count of the last inference request drops to zero, a release operation is triggered on the shared memory resource, returning the shared memory resource to the key-value memory resource.

[0153] In other words, when the reference count of an inference request drops to zero, the shared memory resource corresponding to that inference request is released and returned to the key-value memory resource. It should be noted that the released shared memory resource can be used as a shared memory resource or as an independent memory resource in subsequent memory allocations. This application embodiment does not specifically limit this. For example, if the released shared memory resource and the free memory resources before and after it can meet the storage requirements of the input tokens for the new inference request, the shared memory resource can be allocated to the new inference request based on the number of input tokens; otherwise, it can be used as an independent memory resource.

[0154] In this embodiment, by setting a reference count for the shared memory resource, after all inference sequences corresponding to the inference request have ended, a release operation of the shared memory resource corresponding to the inference request is triggered, returning the shared memory resource to the key-value memory resource. By timely reclaiming the shared memory of the completed inference requests, it is possible to avoid the completed inference requests occupying a large amount of key-value memory, which would lead to insufficient video memory for other inference requests, resulting in request response failure or slow response speed, thus ensuring the parallel processing efficiency of multiple requests.

[0155] In addition, after the inference sequence corresponding to the inference request ends, the release operation of the independent memory resources corresponding to the inference sequence is triggered, and the independent memory resources corresponding to the inference sequence are returned to the key-value memory resources. By timely reclaiming the independent memory of the ended inference sequence, the peak usage of GPU memory during the long sequence generation process can be significantly reduced, avoiding insufficient active sequence memory due to the memory occupied by the ended inference sequence, thereby avoiding interruption or failure of long sequence inference and improving the inference reliability of long sequence.

[0156] In an exemplary embodiment, the aforementioned shared memory resources can also be applied to memory sharing across requests. If, during the process of a previous request or within a preset time after the previous request ends, the new request shares all or part of the input tokens of the previous request, then the new request can share all or part of the shared memory resources of the previous request. Alternatively, in batch inference scenarios, if multiple requests (at least two requests) in a batch have the same system prompt or the same dialogue history prefix, the multiple requests can reuse the same computed and cached shared memory resources, completely eliminating the overhead of redundant computation and storage. Based on this, and building upon the above embodiments, as... Figure 5 As shown, prior to step 102 above, the cache management method may further include steps 501 to 502. Wherein:

[0157] Step 501: Retrieve cached shared memory resources in the key-value memory resources and determine whether there are cached shared memory resources that match the input lexical sequence of the inference request.

[0158] Step 502: If it exists, establish a reference association between the inference request and the cached shared memory resource, increment the reference count of the cached shared memory resource, and skip the calculation and storage process of the key-value vector of each input word in the inference request.

[0159] For example, before storing the key-value vectors of each input term carried in the inference request into the shared memory resource in the key-value memory resource, the memory manager can first determine whether there is a cached shared memory resource in the cached shared memory resource that is consistent with the input term sequence of the inference request. If it exists, there is no need to store the key-value vectors of each input term in the inference request into the shared memory resource in the key-value memory resource. It is only necessary to establish a reference association between the inference request and the cached shared memory resource.

[0160] For example, before calculating the key-value vectors of each input term carried in the inference request, the server can also search for cached shared memory resources in the key-value memory resource to determine whether there is a cached shared memory resource that matches the input term sequence of the inference request. If it exists, there is no need to perform the step of calculating and obtaining the key-value vectors of each input term carried in the inference request. The memory manager in the server also does not need to perform the step of obtaining the key-value vectors of each input term carried in the inference request. It only needs to establish a reference association between the inference request and the cached shared memory resource according to the association instruction sent by the server.

[0161] If it does not exist, calculate the key-value vector of each input word carried in the inference request, and create a new shared memory resource corresponding to the inference request to store the key-value vector of each input word carried in the inference request into the corresponding shared memory resource.

[0162] Furthermore, after establishing a reference association between the inference request and the cached shared memory resource, the memory manager can increment the reference count of the cached shared memory resource by one, or by a specified number, or by a preset number, and skip the calculation and storage process of the key-value vectors of each input lexical term in the inference request. The increment in the reference count is related to the number of output inference sequences of the associated inference request. For example, the cached shared memory resource can also include a first-level reference count and a second-level reference count. The first-level reference count can be used to represent the number of inference requests corresponding to the shared memory resource, and the second-level reference count can be used to represent the number of inference sequences corresponding to the inference request. Based on this, after establishing a reference association between the inference request and the cached shared memory resource, the first-level reference count of the cached shared memory resource can be incremented by one, and a second-level reference count corresponding to the inference request can be created based on the total number of inference sequences corresponding to the inference request.

[0163] The method proposed in this embodiment enables memory resource sharing across requests. That is, when there are the same input prompts or the same historical prefixes across requests, the shared memory resources corresponding to the input tokens can be shared across requests, which improves the implementation dimension of memory resource sharing. In other words, the memory resource sharing concept can not only be applied between multiple inference sequences of a single inference request, but also to memory resource sharing between different inference requests. In this example, the scope of memory reuse is extended to different inference requests to achieve a higher level of memory sharing mechanism, which can avoid repetitive and redundant computation and storage overhead, and further improve the utilization rate of memory resources.

[0164] In one exemplary embodiment, such as Figure 6As shown, a method for determining a cache management scheme is also provided. Taking a server, such as a memory manager in a server, as an example, this method includes the following steps 601 to 604. Wherein:

[0165] Step 601: Deploy different cache management schemes on multiple computing nodes. The different cache management schemes include at least two of the following: hierarchical shared cache, radix tree storage, and memory paging-based shared storage.

[0166] The following sections will describe each cache management scheme in detail, including:

[0167] The first approach is a hierarchical shared caching method, which is the caching management method described in the above embodiments. This involves storing the key-value vectors of the input and output terms of the inference request through shared and independent layers, respectively. For example, the hierarchical shared caching method can maintain a shared cache object (such as SharedCache) for each inference request, containing a key-value tensor (of shape [num_layers, num_heads, prompt_len, head_dim]) for the input prompt. Each inference sequence beam under the inference request maintains a sequence cache object (such as BeamCache), containing a KV tensor (dynamically growing) for generating its token, and holding a reference to the SharedCache object of the request. Alternatively, a corresponding linked list can be maintained for each candidate sequence and each independent sampled output of each inference request, used to concatenate and generate a complete continuous KV cache for generating the next term. The memory manager maintains the shared cache for all inference requests and the independent cache for active sequence beams.

[0168] When a new request arrives, the key-value cache of the input prompt is calculated and stored in the shared layer. Each beam is initialized with an empty independent layer; each time a token is generated, the new key-value pair is appended to the independent cache of that beam (using contiguous memory or a block linked list).

[0169] In a hierarchical shared cache approach, the attention implementation needs to be modified to accept two key-value sequences simultaneously (shared and independent). For example, the two sequences can be concatenated along their length before computation, or a segmented attention kernel can be used.

[0170] The advantages of hierarchical shared caching are: it can significantly reduce memory usage, especially since the prompt portion is usually quite long (e.g., hundreds of tokens), and sharing avoids redundancy caused by multiplying the number of beams by the prompt length. Furthermore, it is relatively simple to implement, requiring only a modification to the attention calculation method.

[0171] The second method, radix tree storage, stores all candidate sequences for each bundle in a shared radix tree (Radix Trie). A radix tree is a variant of a prefix tree, its core feature being the ability to automatically identify and merge paths with a common prefix. A radix tree can be viewed as a trie tree with binary bit strings as keys; it's a multi-branch tree structure, similar to a multi-level index table. Each intermediate node contains an array of pointers to multiple child nodes, and leaf nodes contain pointers to the actual objects (here, the beam of the candidate sequence for each requested bundle search). In bundle search scenarios, this means all bundles automatically share their common prefix without requiring additional reference counting and copy-on-write mechanisms.

[0172] The core advantage of radix tree storage lies in its memory footprint, which approaches O(L) rather than O(K×L), where L is the effective length of the generated sequence and K is the bundle width. This is because, regardless of how many bundles share a prefix, that prefix is ​​stored only once in physical memory.

[0173] In radix tree storage, the core data structures include: the RadixNode structure, which is the basic unit of the radix tree, representing a token and its corresponding key-value vector; the BeamPath structure, which tracks the complete path of a single bundle in the tree for fast access; and RadixTreeManager, the core class that manages the entire tree structure. Furthermore, the core operation of radix trees is to correctly expand the tree structure in each decoding step and automatically handle prefix sharing. When the bundle search algorithm decides to expand the path of a bundle, the system needs to determine if other bundles have already generated the same prefix. If so, the subtree corresponding to that prefix is ​​directly shared, specifically by incrementing the node count of each shared node by 1. In addition, memory reclamation for pruning operations is achieved by reducing the shared node count.

[0174] The memory efficiency of the radix tree scheme is its most significant advantage. Through automated prefix sharing, the system can reduce memory usage from the traditional O(K×L) to near O(L). This efficiency improvement is particularly noticeable when the bundle width is large: assuming a bundle width K=8 and a sequence length L=100, the traditional scheme requires storing KV data for 800 tokens, while the radix tree scheme may only require 120-150 nodes under typical distributions, resulting in memory savings of up to 80-85%. Furthermore, the KV cache corresponding to each node in the radix tree does not need to be stored contiguously in memory, thus eliminating internal and external fragmentation issues and improving memory utilization efficiency.

[0175] Radix tree schemes are applicable to the following scenarios: high-beamwidth scenarios (K≥8), where prefix sharing yields the greatest benefit; prefix-dominant tasks, such as the system prompt in a dialogue system or the context in code completion, where the common prefix is ​​particularly long; and constrained GPU environments requiring high memory efficiency, where radix trees can provide the highest memory utilization when GPU memory is particularly tight.

[0176] The third method is a shared storage approach based on memory paging, also known as a virtual memory-based memory management method. This method divides the physical address into multiple equal-sized blocks, each storing a fixed number of key-value (KV) cache entries. After uniquely identifying each block, a unified memory pool is formed. Upon receiving an inference request, at least one virtual memory table containing at least one virtual space block is created, and a one-to-one mapping relationship between virtual space blocks and physical space blocks is established through a memory mapping table. This allows for the management of the target memory resources based on the unified memory pool, virtual memory table, and memory mapping table.

[0177] The virtual memory table can be a single, unified table, where the size of its blocks matches that of the physical memory blocks. The number of blocks in the virtual memory table can be the same as or greater than the number of physical memory blocks to allow for external fragmentation. Alternatively, the virtual memory table can be a temporary table corresponding to each inference request, used for temporary virtual storage of the key-value cache for each request. Similarly, memory-mapped tables can also be set as one or more, just like the virtual memory table.

[0178] Taking dual-output parallel decoding as an example, this section describes in detail the shared storage method based on memory paging. Since the two outputs share the same cue, only one copy of the cue state is retained during the cue phase. The cue logic blocks of the two sequences are mapped to the same physical blocks: logic blocks 0 and 1 of the two sequences are mapped to physical blocks 7 and 1, respectively. Since a single physical block can be mapped to multiple logic blocks, a reference count is introduced for each physical block. In this scenario, the reference counts of physical blocks 7 and 1 are both 2. During the generation phase, the two outputs sample different output tokens, requiring independent storage space to be allocated for the KV cache. Therefore, a block-level copy-on-write mechanism is implemented on the physical blocks that need to be modified by multiple sequences, similar to the copy-on-write technique in the operating system's virtual memory (e.g., during process forking).

[0179] Specifically, when sampleA1 needs to write to its last logical block (logical block 1), it recognizes that the reference count of the corresponding physical block (physical block 1) is 2, which is greater than 1. It allocates a new physical block (physical block 3), instructs the block engine to copy information from physical block 1, and decrements the reference count to 1. Subsequently, when sampleA2 writes to physical block 1, the reference count has dropped to 1; therefore, sampleA2 can directly write the newly generated KV cache to physical block 1.

[0180] In other words, the key-value (KV) buffers calculated for the common input portion of multiple outputs sampled in parallel are stored together at the same physical address, and this same physical address is referenced in the logical address. Simultaneously, a reference count is incremented for each physical block. During the decoding output stage, for each output word's KV buffer, the reference count of the physical block being written is checked when storing it. If the reference count is greater than 1, the content to be written to the physical block is copied to a new physical block using a copy-on-write mechanism. The new physical block is then written to, and the reference count of the original physical block is subtracted from the number of copies to obtain a new reference count. If the reference count is not greater than 1, the KV buffer to be written is written to the physical block during the write operation. Once the block is full, a new free physical block is allocated.

[0181] In summary, by employing a copy-on-write mechanism to manage the last logical block, cross-sample sharing of the prompt key-value cache space is achieved. This cross-sample physical block sharing mechanism can significantly reduce memory usage, and its effect is particularly noticeable when processing long input prompts.

[0182] Unlike hierarchical shared caching, memory-paged shared storage does not manage memory hierarchically. Instead, it manages all memory uniformly. Utilizing physical block partitioning and copy-on-write mechanisms, it can share most of the prompt input during parallel processing. Sharing only begins from the last incomplete physical block in the key-value cache storing the prompt input and ends with a one-to-one mapping. That is, the copy-on-write mechanism is only activated when newly generated tokens are located within old shared data blocks (such as in parallel decoding scenarios), in which case only a single data block needs to be copied. Therefore, memory-paged shared storage eliminates fragmentation and dynamic allocation overhead, making it suitable for scenarios with a large number of beams and varying lengths.

[0183] Furthermore, unlike parallel sampling decoding, beam search not only shares the initial cue block but also shares other blocks between different candidate sequences, and this sharing mode changes dynamically as the decoding process progresses. Specifically, in the case of beam search, a reference count greater than 1 is allowed when writing physical blocks; the content of a physical block is copied only if inconsistencies are detected during writing. Simultaneously, the sharing and reclamation of each physical block are managed through the reference count corresponding to each physical block.

[0184] Based on this, in application scenarios involving multiple computing nodes, each computing node can adopt the different cache management schemes mentioned above. Then, using the same batch size, the target inference task is executed on each computing node to achieve small-batch inference verification, and then the target cache management scheme of the computing node is selected based on the verification results.

[0185] It is important to note that when testing multi-compute node cache management solutions, it is crucial to ensure that the hardware configurations (such as CPU, memory, and bandwidth) of each compute node are consistent. This prevents hardware differences from interfering with test results and ensures the fairness and accuracy of the test. Subsequently, different cache management solutions should be deployed on multiple compute nodes. The deployment process must strictly adhere to the technical specifications of each solution to ensure complete deployment, reasonable parameter configuration, and that the cache service on each node can start and run normally.

[0186] Step 602: Using a preset batch size, based on the data distribution of the target inference task, small-batch inference experiments are performed on multiple computing nodes respectively.

[0187] After deploying the cache management scheme on each computing node and confirming the normal operation of the cache service, a small-batch inference experiment should be initiated. The experiment process requires strict control of variables to ensure the scientific validity and comparability of the results. For example, the preset batch size can be determined by considering the actual application scenario of the target inference task (such as inference latency requirements, data throughput, and hardware capacity). The batch size setting must consider two core requirements: first, ensuring that the batch data can fully utilize cache resources, avoiding excessively small batches leading to low cache hit rates and insufficient memory utilization; second, avoiding excessively large batches leading to high memory consumption, increased inference latency, or even memory overflow. For example, multiple candidate batch sizes can be determined through pre-testing, and the preset batch size that best meets the requirements of the target inference task can be selected as the experimental standard.

[0188] Next, based on the actual data distribution characteristics of the target inference task (such as data type, number of key-value pairs, data access frequency distribution, data size distribution, etc.), prepare the experimental inference data. It is important to ensure that the experimental data matches the actual data distribution of the target inference task to avoid test results that fail to reflect the actual application effect due to differences in data distribution.

[0189] Furthermore, the prepared experimental data was evenly distributed to the computing nodes deployed with different caching management schemes according to the preset batch size, and a small-batch inference experiment was launched. During the experiment, each computing node independently executed the inference task, and the caching management scheme automatically performed operations such as caching, reading, and eviction of key-value data during the inference process without manual intervention.

[0190] To ensure the stability of the experimental results, each computing node needs to repeat the small-batch inference experiment multiple times (at least 3 times is recommended). The batch data and initial cache state (with the cache cleared) should be consistent in each experiment to avoid the influence of experimental randomness on the results. During the experiment, the running status of each node should be monitored in real time to ensure that the inference task is executed normally and there are no problems such as program crashes or cache service abnormalities. If any abnormality occurs, the experiment should be terminated in time, the problem should be investigated, and the experiment should be re-executed.

[0191] Step 603: Collect the actual key-value memory utilization efficiency index of each computing node in the small-batch inference experiment.

[0192] The actual utilization efficiency indicators for key-value memory include at least one of memory usage, memory fragmentation rate, memory copy count, and memory overflow count. Memory usage refers to the actual amount of memory used by the caching management scheme to store key-value data during inference experiments, including the memory occupied by the cached data itself and the memory occupied by the cache structure (such as radix tree nodes, page table entries, etc.). Memory usage data for each computing node is collected in real time during the experiment using operating system memory monitoring tools (such as the `top`, `free`, and `ps` commands in Linux systems), recording the peak and average memory usage in each small-batch inference experiment to ensure data accuracy. During collection, it is necessary to distinguish between cache memory usage and the memory usage of the inference task itself to avoid including the computational memory of the inference task in the cache memory usage, ensuring the relevance of the indicator collection.

[0193] Memory fragmentation rate refers to the proportion of underutilized, fragmented memory blocks to the total cache memory during cache management. A higher fragmentation rate indicates more severe memory resource waste and lower cache utilization efficiency. Memory fragmentation data is collected from each compute node during the experiment using memory fragmentation monitoring tools (such as the slabtop command on Linux systems and memory fragmentation analysis tools), and the memory fragmentation rate is calculated as (Memory fragmentation rate = Size of unused fragmented memory blocks / Total cache memory size × 100%). During the collection process, the trend of memory fragmentation rate changes in each experiment should be recorded, with particular attention paid to fluctuations in fragmentation rate during cache data insertion and eviction, reflecting the ability of different cache management schemes to control memory fragmentation.

[0194] Memory copy count refers to the number of times key-value data is copied between the cache and the inference process, and between caches on different nodes, during inference. The more memory copies, the higher the memory bandwidth usage and the greater the performance loss in inference. By using code tracking and system call monitoring, we can count the number of cache data copies on each compute node in each small-batch inference experiment. This includes copies between the local cache and the inference process, copies between the global shared cache and the local cache, and copies between shared caches on different nodes.

[0195] The number of memory overflows refers to the number of times during inference experiments that cached data cannot be stored normally due to reasons such as reaching the cache memory limit or an unreasonable eviction policy. Memory overflows can cause inference tasks to be interrupted and data to be lost, and are an important indicator for evaluating the stability and reliability of a cache management scheme. By monitoring the logs and system error messages of the cache service, the number of memory overflows on each computing node during the experiment is counted. If no memory overflow occurs, it is recorded as 0.

[0196] For example, each compute node runs a different cache management scheme. In the small-batch inference experiment, the actual key-value memory utilization efficiency index of each compute node after running the corresponding cache management scheme is obtained. The actual key-value memory utilization efficiency index is used to characterize the memory utilization efficiency of the compute node after running the corresponding cache management scheme. The higher the memory utilization efficiency, the better the running efficiency of the compute node after running the cache management scheme.

[0197] Step 604: Determine the target cache management scheme based on the actual key-value memory utilization efficiency index of each cache management scheme.

[0198] In this example, the optimal target cache management scheme can be determined by using a multi-dimensional comparative analysis based on the actual key-value memory utilization efficiency of each cache management scheme, combined with the inference requirements of the target inference task (such as prioritizing memory utilization efficiency, inference latency, and stability).

[0199] For example, for each cache management scheme, the overall utilization efficiency can be determined by weighting multiple key-value memory utilization efficiency metrics corresponding to the cache management scheme. Then, the cache management scheme with the highest overall utilization efficiency is selected as the target cache management scheme for the compute node, that is, the target cache management scheme adopted by the compute node when running the target inference task or other inference tasks, which is also the optimal cache management scheme adapted to the target inference task or other inference tasks. In one optional implementation, reasonable weights can be assigned to the collected key-value memory utilization efficiency metrics according to the actual needs of the target inference task. Based on the metric weights, the various metrics of each cache management scheme are quantitatively scored, and then the overall score of each cache management scheme, i.e., the overall utilization efficiency, is calculated.

[0200] In this embodiment, different cache management schemes can be adopted on multiple computing nodes, with the same batch size for small-batch inference. The target memory management scheme for large-batch inference can be determined based on the actual utilization efficiency of each memory management scheme under the requirements of the inference task and the characteristics of the data. That is, experimental evaluation is carried out to achieve dynamic decision-making. Through a series of operations such as multi-node concurrent deployment, small-batch trials of the same size, and horizontal comparison of actual utilization, the selection of large-batch cache management scheme can be guided, which can improve the adaptability of cache management scheme selection. It is especially suitable for non-standard data distribution scenarios (such as special question-and-answer in vertical fields) that are difficult to accurately predict based on rule thresholds. By making more accurate scheme selection decisions through actual testing rather than estimation, the utilization of memory resources can be maximized.

[0201] In one exemplary embodiment, such as Figure 7 As shown, the method for determining the above cache management scheme may further include steps 701 to 704. Wherein:

[0202] Step 701: Collect task feature data of historical reasoning tasks. The task feature data includes at least two of the following: reasoning task type, number of input lexical units, number of predicted output lexical units, number of sequences, and bundle search width.

[0203] Step 702: Obtain the actual key-value memory utilization efficiency of each historical reasoning task under the hierarchical shared cache method, radix tree storage method, and memory paging-based shared storage method.

[0204] Step 703: Using task feature data as input features and the category identifier of the cache management scheme with the best actual key-value memory utilization efficiency under each historical inference task as training label, train the memory management decision model.

[0205] Step 704: Receive a new inference request, extract the task feature data of the new inference request, input the task feature data of the new inference request into the memory management decision model, and output the target cache management scheme from the memory management decision model.

[0206] In this example, a memory management decision model can be trained based on the optimal memory management scheme under each task requirement and data characteristic. Subsequently, the input inference task requirements and data characteristics are input into the trained memory management decision model to determine the target cache management scheme corresponding to the new inference request.

[0207] For example, in practical applications, different cache management schemes can be deployed on multiple computing nodes. After determining the target cache management scheme corresponding to a new inference request based on the memory management decision model, the target computing node for deploying the target cache management scheme is determined. Then, the new inference request is allocated to the target computing node for inference analysis to achieve efficient utilization of memory management.

[0208] In this embodiment, a learning-based soft decision-making model is adopted. The memory-managed decision model can automatically learn the optimal decision boundary from historical data, and can handle complex data distributions that were not anticipated in the rule design. Furthermore, by training a reusable online decision model, the model's decision accuracy will continue to improve as historical data accumulates, forming a "data flywheel" effect, which is a technical effect that rule-based systems cannot achieve.

[0209] In one exemplary embodiment, a large language model inference system is also provided, which includes multiple computing nodes and a request routing module, wherein:

[0210] Each computing node deploys a different cache management scheme, which includes at least two of the following: hierarchical shared cache, radix tree storage, and memory paging-based shared storage. Each computing node manages key-value memory resources according to its deployed cache management scheme; wherein, the hierarchical shared cache is the cache management method in the above embodiments.

[0211] The request routing module is used to receive inference requests, extract the inference task type and data characteristics of the inference request, and route the inference request to the target computing node that best matches the inference task type and data characteristics.

[0212] The process of matching the target computing node can be referred to the above. Figure 6 The method for determining the cache management scheme shown in the example will not be repeated here.

[0213] In this embodiment, a large language model inference system is constructed, comprising multiple computing nodes and a request routing module. This system achieves efficient memory management by deploying different memory management schemes on each computing node and allocating requests to different computing nodes for processing based on the input task requirements and data characteristics. For example, computing nodes with a hierarchical shared cache are routed to inference requests with a large number of parallel sampling outputs and input lexical units; computing nodes with radix tree storage are routed to inference requests with large beam search widths and long prediction outputs; computing nodes with shared storage based on memory paging are routed to inference requests involving a combination of parallel sampling and beam search, or other inference requests that do not meet the priority conditions of the first two types of computing nodes.

[0214] In this example, the solution is specialized through physical isolation deployment in a multi-node cluster, which is a higher-dimensional system architecture innovation. In addition, the heterogeneous deployment of multiple nodes can avoid the memory reorganization overhead of switching between multiple solutions on a single node, and each node is highly optimized for specific scenarios. At the same time, the scenario-aware distribution of load through request routing can not only improve the efficiency of request allocation, but also improve the efficiency of request processing and the utilization of memory resources.

[0215] In one exemplary embodiment, such as Figure 8 As shown, a reasoning method for a large language model is also provided. This reasoning method uses the cache management method in the above embodiments to store and manage the key-value vectors during the reasoning process. Taking the application of this method to a server as an example, it includes the following steps 801 to 804. Wherein:

[0216] Step 801: Read the key-value vectors of each input word carried in the reasoning request corresponding to each reasoning sequence from the shared memory resources of the key-value memory resources.

[0217] Step 802: Read the key-value vector of the output word generated in each round of reasoning for each reasoning sequence from the independent memory resources of the key-value memory resources.

[0218] Step 803: Concatenate the key value vectors of each input word corresponding to each reasoning sequence with the key value vectors of each output word in each round of reasoning according to the word sequence to obtain the key value sequence.

[0219] Step 804: Input the key-value sequence into the attention layer of the large language model for inference, generating the key-value vector of the output word for each inference sequence in this round of inference.

[0220] In this embodiment, the attention mechanism is modified to enable simultaneous reception and processing of two key-value sequences (shared and independent). Specifically, before computation, the two sequences are concatenated along the sequence length dimension to obtain the complete key-value sequences corresponding to each inference sequence in this round of inference. Then, inference is performed based on the complete key-value sequences corresponding to each inference sequence, and the key-value vector of the next output word or the next output word sequence corresponding to each inference sequence in this round of inference is output.

[0221] In one exemplary embodiment, such as Figure 9 As shown, another reasoning method for large language models is provided. This reasoning method uses the cache management method in the above embodiments to store and manage the key-value vectors during the reasoning process. Taking the application of this method to a server as an example, it includes the following steps 901 to 904. Wherein:

[0222] Step 901: Perform the first-stage attention score calculation on the input lexical key-value vector in the shared memory resource corresponding to the inference request and the query vector of the current inference step to obtain the first attention score matrix and the first intermediate attention value.

[0223] Step 902: Perform the second-stage attention score calculation on the output lexical key vector and query vector in the independent memory resources corresponding to each inference sequence to obtain the second attention score matrix and the second intermediate attention value.

[0224] Step 903: Normalize the first attention score matrix and the second attention score matrix, and perform a weighted sum of the first intermediate attention value and the second intermediate attention value according to the preset normalization weights of each stage to obtain the target attention score matrix and the target intermediate attention value.

[0225] Step 904: Based on the target attention score matrix and the target intermediate attention value, generate the key-value vector of the output word for each inference sequence in this round of inference process.

[0226] In this embodiment, the attention mechanism is modified to enable simultaneous reception and processing of two key-value sequences (shared and independent). Specifically, a segmented attention kernel is used to calculate the key-value vectors of the input words in the shared memory and the key-value vectors of the output words in the independent memory. Finally, the calculation results based on the key-value vectors of the input words and the calculation results based on the key-value vectors of the output words are merged to obtain the inference result corresponding to this round of inference.

[0227] The above embodiments provide two implementation methods for the attention layer under the hierarchical shared caching approach. By using vector concatenation or segmented attention, the input lexical key-value vectors in the shared layer and the output lexical key-value vectors in the independent layer are processed to achieve the purpose of inference analysis. It can be seen that the hierarchical shared caching approach, by dividing memory resources into shared memory resources and independent memory resources and adaptively modifying the attention calculation method, can not only significantly reduce memory usage, avoid copying, concatenating, and storing duplicate data, and reduce server computing resources, but also has a simple implementation, optimize memory management strategies at a lower cost, and improve memory resource utilization.

[0228] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0229] Based on the same inventive concept, embodiments of this application also provide a cache management device for implementing the cache management method described above, a cache management scheme determination device for the cache management scheme determination method, and a large language model inference device for the large language model inference method. The solution provided by this device is similar to the implementation scheme described in the above methods. Therefore, the specific limitations of one or more cache management device embodiments, cache management scheme determination device embodiments, and large language model inference device embodiments provided below can be found in the limitations of the cache management method, cache management scheme determination method, and large language model inference method described above, and will not be repeated here.

[0230] In one exemplary embodiment, such as Figure 10 As shown, a cache management device is provided, applied to a memory manager. The memory manager is used to manage the key-value memory resources of a large language model, including: a first acquisition module 1001, a first storage module 1002, a second acquisition module 1003, and a second storage module 1004, wherein:

[0231] The first acquisition module 1001 is used to acquire the key-value vectors of each input word carried in the inference request.

[0232] The first storage module 1002 is used to store the key-value vectors of each input word into the shared memory resource in the key-value memory resource; wherein, each inference request corresponds to at least 2 inference sequences, and the key-value vector of the input word corresponding to each inference request is stored only once in the shared memory resource.

[0233] The second acquisition module 1003 is used to acquire the key-value vectors of the output words generated by each inference sequence corresponding to the inference request during the inference process; the output words in each inference sequence are generated by autoregression based on the key-value vectors of each input word in the corresponding inference request in the shared memory resource and the key-value vectors of each output word in the inference sequence.

[0234] The second storage module 1004 is used to store the key-value vector of the output word in each inference sequence corresponding to the inference request into an independent memory resource in the key-value memory resource; the independent memory resource is at least a portion of the memory resource in the key-value memory resource other than the shared memory resource.

[0235] In one embodiment, before the first storage module 1001 stores the key-value vectors of each input word into the shared memory resource in the key-value memory resource, the apparatus further includes:

[0236] The third acquisition module is used to acquire the number of input tokens carried in the inference request.

[0237] The shared resource allocation module is used to determine the dynamic memory resources that match the number of input tokens from the key-value memory resources, and to use the dynamic memory resources as shared memory resources.

[0238] In one embodiment, the shared resource allocation module is specifically used to determine whether the number of input tokens is greater than or equal to the preset number of tokens; if so, it determines a dynamic memory resource matching the number of input tokens from the key-value memory resource in a continuous storage manner; if not, it determines a dynamic memory resource matching the number of input tokens from the key-value memory resource in a paginated storage manner or a radix tree storage manner.

[0239] In one embodiment, the independent memory resources are stored using a paging method or a radix tree method.

[0240] In one embodiment, before the first acquisition module 1001 acquires the key-value vectors of each input lexical term carried in the inference request, the apparatus further includes:

[0241] The fourth acquisition module is used to acquire the reasoning task type and data characteristics of the reasoning request;

[0242] The management scheme confirmation module is used to determine the target cache management method from among hierarchical shared cache method, radix tree storage method and memory paging-based shared storage method based on the inference task type and data characteristics, and to store and manage the key-value vector of the inference request according to the target cache management method.

[0243] In one embodiment, the device further includes:

[0244] The reference counting configuration module is used to maintain a reference count for the key-value vector of the input lexical corresponding to each inference request in the shared memory resource. The initial value of the reference count is equal to the total number of inference sequences corresponding to the inference request.

[0245] The resource reclamation module is used to mark the independent memory resources corresponding to the inference sequence as reclaimable and release them when the inference sequence is eliminated in the beam search or when the output of parallel sampling ends during the inference process. At the same time, it decrements the reference count of the inference request in the shared memory resources by one.

[0246] The resource reclamation module is also used to trigger the release operation of the corresponding shared memory resource when the reference count of the shared memory resource drops to zero, and return the memory resource corresponding to the inference request in the shared memory resource to the key-value memory resource.

[0247] In one embodiment, before the first storage module 1002 stores the key-value vectors of each input word into the shared memory resource in the key-value memory resource, the apparatus further includes:

[0248] The resource retrieval module is used to retrieve cached shared memory resources in the key-value memory resources and determine whether there are cached shared memory resources that match the input word sequence of the inference request.

[0249] The resource association module is used to establish a reference association between the inference request and the cached shared memory resource when there is a cached shared memory resource that matches the input lexical sequence of the inference request. It increments the reference count of the cached shared memory resource and skips the calculation and storage process of the key-value vector of each input lexical of the inference request.

[0250] In one exemplary embodiment, such as Figure 11 As shown, a device for determining a cache management scheme is provided, comprising: a scheme deployment module 1101, an inference testing module 1102, an indicator collection module 1103, and a scheme determination module 1104, wherein:

[0251] The solution deployment module 1101 is used to deploy different cache management schemes on multiple computing nodes. The different cache management schemes include at least two of the following: hierarchical shared cache, radix tree storage, and memory paging-based shared storage; wherein, the hierarchical shared cache adopts the cache management method of the first aspect.

[0252] The inference experiment module 1102 is used to execute small-batch inference experiments on multiple computing nodes based on the data distribution of the target inference task and using a preset batch size.

[0253] The indicator collection module 1103 is used to collect the actual key-value memory utilization efficiency indicators of each computing node in the small-batch inference experiment. The actual key-value memory utilization efficiency indicators include at least one of memory usage, memory fragmentation rate, memory copying times, and memory overflow times.

[0254] The scheme determination module 1104 is used to determine the target cache management scheme based on the actual key-value memory utilization efficiency index of each cache management scheme.

[0255] In one embodiment, the device further includes:

[0256] The first data collection module is used to collect task feature data of historical reasoning tasks. The task feature data includes at least two of the following: reasoning task type, number of input lexical units, number of predicted output lexical units, number of sequences, and bundle search width.

[0257] The second data collection module is used to obtain the actual key-value memory utilization efficiency of each historical reasoning task under the hierarchical shared cache method, radix tree storage method and memory paging-based shared storage method.

[0258] The model training module is used to train a memory management decision model by taking task feature data as input features and using the category identifier of the cache management scheme with the best actual key-value memory utilization efficiency under each historical inference task as training labels.

[0259] The scheme determination module is used to receive new inference requests, extract the task feature data of the new inference requests, input the task feature data of the new inference requests into the memory management decision model, and output the target cache management scheme from the memory management decision model.

[0260] In one exemplary embodiment, such as Figure 12 As shown, a reasoning device for a large language model is provided, employing... Figure 10 The cache management device shown stores and manages the key-value vectors during the inference process, including: a first reading module 1201, a second reading module 1202, a splicing module 1203, and an inference module 1204, wherein:

[0261] The first reading module 1201 is used to read the key-value vectors of each input word carried in the reasoning request corresponding to each reasoning sequence from the shared memory resources of the key-value memory resources.

[0262] The second reading module 1202 is used to read the key-value vectors of the output words generated in each round of reasoning corresponding to each reasoning sequence from the independent memory resources of the key-value memory resources.

[0263] The concatenation module 1203 is used to concatenate the key value vectors of each input word corresponding to each reasoning sequence with the key value vectors of each output word in each round of reasoning according to the word sequence to obtain the key value sequence.

[0264] The inference module 1204 is used to input the key-value sequence into the attention layer of the large language model for inference, and generate the key-value vector of the output word corresponding to each inference sequence in this round of inference process.

[0265] In one exemplary embodiment, such as Figure 13 As shown, a reasoning device for a large language model is provided, employing... Figure 10 The cache management device shown stores and manages the key-value vectors during the inference process. When the number of input terms carried in the inference request is greater than a first preset number and the number of output terms corresponding to the inference sequence is greater than a second preset number, or the sum of the number of input terms and the number of output terms is greater than a third preset number, it includes: a first attention calculation module 1301, a second attention calculation module 1302, a fusion module 1303, and an inference module 1304, wherein:

[0266] The first attention calculation module 1301 is used to perform first-stage attention score calculation on the input lexical key value vector in the shared memory resource corresponding to the inference request and the query vector of the current inference step, so as to obtain the first attention score matrix and the first intermediate attention value.

[0267] The second attention calculation module 1302 is used to perform second-stage attention score calculation on the output lexical key value vector and query vector in the independent memory resources corresponding to each inference sequence, so as to obtain the second attention score matrix and the second intermediate attention value.

[0268] The fusion module 1303 is used to normalize the first attention score matrix and the second attention score matrix, and to perform a weighted summation of the first intermediate attention value and the second intermediate attention value according to the preset normalization weights of each stage to obtain the target attention score matrix and the target intermediate attention value.

[0269] The reasoning module 1304 is used to generate the key-value vector of the output word for each reasoning sequence in this round of reasoning based on the target attention score matrix and the target intermediate attention value.

[0270] Each module in the aforementioned cache management device, cache management scheme determination device, and large language model inference device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0271] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores key-value vectors and other related data generated during inference. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a cache management method, a method for determining a cache management scheme, and a large language model inference method.

[0272] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0273] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the cache management method, the cache management scheme determination method, and the large language model inference method in any of the above embodiments.

[0274] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the cache management method, the cache management scheme determination method, and the large language model inference method in any of the above embodiments.

[0275] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the cache management method, the cache management scheme determination method, and the large language model inference method in any of the above embodiments.

[0276] It should be noted that the data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data that have been fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0277] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0278] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0279] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A cache management method, characterized in that, Applied to a memory manager, the memory manager is used to manage the key-value memory resources of the large language model, the method includes: Obtain the key-value vectors of each input word carried in the inference request; The key-value vectors of each input word are stored in the shared memory resource of the key-value memory resource; wherein, each inference request corresponds to at least 2 inference sequences, and only one copy of the key-value vector of the input word corresponding to each inference request is stored in the shared memory resource; Obtain the key-value vector of the output word generated for each inference sequence corresponding to the inference request during the inference process; the output word in each inference sequence is generated by autoregression based on the key-value vector of each input word in the corresponding inference request in the shared memory resource and the key-value vector of each output word in the inference sequence; The key-value vector of the output term in each inference sequence corresponding to the inference request is stored in an independent memory resource in the key-value memory resource; the independent memory resource is at least a portion of the memory resource in the key-value memory resource other than the shared memory resource.

2. The method according to claim 1, characterized in that, Before storing the key-value vectors of each input lexical unit into the shared memory resource of the key-value memory resource, the method further includes: Obtain the number of each input lexical carried in the inference request; Determine dynamic memory resources from the key-value memory resources that match the number of input tokens; The dynamic memory resource is used as the shared memory resource.

3. The method according to claim 2, characterized in that, The step of determining the dynamic memory resource matching the number of input tokens from the key-value memory resource includes: Determine whether the number of input tokens is greater than or equal to the preset number of tokens; If so, then a dynamic memory resource matching the number of input tokens is determined from the key-value memory resource in a continuous storage manner; If not, then using either paginated storage or radix tree storage, determine a dynamic memory resource from the key-value memory resource that matches the number of input tokens.

4. The method according to any one of claims 1-3, characterized in that, The independent memory resources are stored using either paging or radix tree methods.

5. The method according to any one of claims 1-3, characterized in that, Before obtaining the key-value vectors of each input lexical carried in the inference request, the method further includes: Obtain the reasoning task type and data characteristics of the reasoning request; Based on the inference task type and data characteristics, a target cache management method is determined from hierarchical shared cache, radix tree storage, and memory paging-based shared storage, and the key-value vector of the inference request is stored and managed according to the target cache management method.

6. The method according to any one of claims 1-3, characterized in that, The method further includes: Maintain a reference count for the key-value vector of the input lexical corresponding to each inference request in the shared memory resource. The initial value of the reference count is equal to the total number of inference sequences corresponding to the inference request. When the inference sequence is eliminated in the beam search or the output ends in the parallel sampling process, the independent memory resource corresponding to the inference sequence is marked as reclaimable and released, and the reference count corresponding to the inference request in the shared memory resource is decremented by one. When the reference count of the shared memory resource drops to zero, a release operation is triggered for the corresponding shared memory resource, returning the memory resource corresponding to the inference request in the shared memory resource to the key-value memory resource.

7. The method according to any one of claims 1-3, characterized in that, Before storing the key-value vectors of each input lexical unit into the shared memory resource of the key-value memory resource, the method further includes: Retrieve the cached shared memory resources in the key-value memory resources, and determine whether there are cached shared memory resources that are consistent with the input word sequence of the inference request; If it exists, then establish a reference association between the inference request and the cached shared memory resource, increment the reference count of the cached shared memory resource, and skip the calculation and storage process of the key-value vectors of each input lexical in the inference request.

8. A method for determining a cache management scheme, characterized in that, The method includes: Different cache management schemes are deployed on multiple computing nodes, and the different cache management schemes include at least two of the following: hierarchical shared cache, radix tree storage, and memory paging-based shared storage; wherein the hierarchical shared cache is a cache management method as described in any one of claims 1-7. Using a preset batch size and based on the data distribution of the target inference task, small-batch inference experiments are executed on the multiple computing nodes respectively; Collect the actual key-value memory utilization efficiency index of each computing node in the small-batch inference experiment. The actual key-value memory utilization efficiency index includes at least one of memory usage, memory fragmentation rate, memory copying times, and memory overflow times. The target cache management scheme is determined based on the actual key-value memory utilization efficiency of each cache management scheme.

9. The method according to claim 8, characterized in that, The method includes: Collect task feature data of historical reasoning tasks, wherein the task feature data includes at least two of the following: reasoning task type, number of input lexical units, number of predicted output lexical units, number of sequences, and bundle search width; Obtain the actual key-value memory utilization efficiency of each of the aforementioned historical reasoning tasks under the hierarchical shared cache method, radix tree storage method, and memory paging-based shared storage method; Using the task feature data as input features and the category identifier of the cache management scheme with the best actual key-value memory utilization efficiency under each historical reasoning task as training label, a memory management decision model is trained. A new inference request is received, the task feature data of the new inference request is extracted, the task feature data of the new inference request is input into the memory management decision model, and the memory management decision model outputs the target cache management scheme.

10. A large language model reasoning system, characterized in that, The inference system includes multiple computing nodes and a request routing module, wherein: Each computing node deploys a different cache management scheme, which includes at least two of the following: hierarchical shared cache, radix tree storage, and memory paging-based shared storage. Each computing node manages key-value memory resources according to its deployed cache management scheme; wherein, the hierarchical shared cache is the cache management method according to any one of claims 1-7. The request routing module is used to receive inference requests, extract the inference task type and data characteristics of the inference requests, and route the inference requests to the target computing node that best matches the inference task type and data characteristics based on the inference task type and data characteristics.

11. A reasoning method for a large language model, characterized in that, The cache management method described in any one of claims 1-7 is used to store and manage the key-value vectors during the inference process, wherein the inference method includes: Read the key-value vectors of each input word carried in the reasoning request corresponding to each reasoning sequence from the shared memory resources of the key-value memory resources; From the independent memory resources of the key-value memory resources, read the key-value vectors of the output words generated in each round of reasoning corresponding to each reasoning sequence; The key value vectors of each input word corresponding to each reasoning sequence are concatenated with the key value vectors of each output word in each round of reasoning according to the word sequence to obtain the key value sequence. The key-value sequence is input into the attention layer of the large language model for inference, generating the key-value vector of the output word for each inference sequence in this round of inference.

12. A reasoning method for a large language model, characterized in that, The cache management method described in any one of claims 1-7 is used to store and manage the key-value vectors during the inference process. When the number of input tokens carried in the inference request is greater than a first preset number and the number of output tokens corresponding to the inference sequence is greater than a second preset number, or the sum of the number of input tokens and the number of output tokens is greater than a third preset number, the inference method further includes: For the input lexical key vector in the shared memory resource corresponding to the inference request and the query vector of the current inference step, perform the first stage attention score calculation to obtain the first attention score matrix and the first intermediate attention value; For the output lexical key vector and the query vector in the independent memory resources corresponding to each inference sequence, perform the second-stage attention score calculation to obtain the second attention score matrix and the second intermediate attention value; The first attention score matrix and the second attention score matrix are normalized, and the first intermediate attention value and the second intermediate attention value are weighted and summed according to the preset normalization weights of each stage to obtain the target attention score matrix and the target intermediate attention value. Based on the target attention score matrix and the target intermediate attention value, the key value vector of the output word for each inference sequence in this round of inference is generated.

Images