KV virtual address determination method and electronic equipment

Abstract

The invention provides a KV virtual address determination method, which comprises the steps of obtaining a target request which comprises a lexical element ID; a first hash value is determined according to the lexical element ID and the size of a physical block, the size of the physical block is determined according to parameters of a target model and the hardware page granularity of a target processor, and the target model is configured in the target processor; determining the number of target physical blocks from the KV tensor pool according to the first hash value; determining a target virtual address space and a target KV virtual address offset according to the number of the target physical blocks; wherein the KV tensor pool is determined according to the preset lexical element number and the size of the physical block.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and more specifically, to a method for determining a KV virtual address and an electronic device. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have achieved remarkable results in fields such as natural language processing. During model inference, to avoid redundant computation, key-value (KV) data of historical lexical units is typically cached. However, as the length of input prompts and the generated sequences increase, the GPU memory occupied by the KV cache grows dramatically, becoming a major bottleneck restricting inference performance and concurrency. Current GPU memory management technologies typically employ pre-allocated contiguous memory or simple paging mechanisms to store KV data. When dealing with the mapping between non-contiguous physical storage and logical access, they often lack efficient address translation and management mechanisms, thus reducing GPU memory utilization and inference efficiency. Summary of the Invention

[0003] In view of this, this disclosure provides a method for determining a KV virtual address and an electronic device.

[0004] One aspect of this disclosure provides a method for determining a key-value (KV) virtual address, comprising: obtaining a target request, the target request including a term ID; determining a first hash value based on the term ID and the size of a physical block, the size of the physical block being determined based on parameters of a target model and the hardware page granularity of a target processor, the target model being configured on a target processor; determining the number of target physical blocks from a KV tensor pool based on the first hash value; and determining a target virtual address space and a target KV virtual address offset based on the number of target physical blocks; wherein the KV tensor pool is determined based on a preset number of terms and the size of the physical blocks.

[0005] According to embodiments of this disclosure, a word ID corresponds one-to-one with a word, and determining the first hash value includes: determining the number of words that can be stored in a physical block based on the size of the physical block; selecting word IDs based on the number of words to be stored to obtain at least one set of word IDs; and determining the first hash value corresponding to each set of word IDs based on the word IDs contained in each set of word IDs.

[0006] According to an embodiment of this disclosure, determining the number of target physical blocks from the KV tensor pool based on a first hash value includes: obtaining a second hash value corresponding to each mapped physical block; matching based on the first hash value and the second hash value; and determining the target number of free physical blocks in the KV tensor pool in response to the absence of a first hash value that matches the second hash value.

[0007] According to embodiments of this disclosure, determining the number of target physical blocks includes: determining the number of first tokens scheduled in this operation based on the target request; obtaining the number of second tokens corresponding to the saved tokens and the number of mapped physical blocks; and determining the number of target physical blocks based on the number of first tokens, the number of second tokens, and the number of mapped physical blocks.

[0008] According to an embodiment of this disclosure, the KV virtual address determination method further includes: in response to the existence of a first hash value that matches the second hash value, determining the first physical block corresponding to the successfully matched second hash value; and determining the first virtual address space corresponding to the first physical block and the first KV virtual address offset.

[0009] According to embodiments of this disclosure, the KV virtual address determination method further includes: in response to the existence of a first hash value that fails to match, determining a third term quantity, wherein the third term quantity is the total number of terms in the term ID set corresponding to each first hash value that fails to match; determining the number of target physical blocks from the KV tensor pool, including: determining the number of target physical blocks from the KV tensor pool based on the third term quantity; determining the target virtual address space and the target KV virtual address offset, including: determining a second virtual address space and a second KV virtual address offset based on the number of target physical blocks; and determining the target virtual address space and the target KV virtual address offset based on the first virtual address space and the first virtual address offset, the second virtual address space and the second KV virtual address offset.

[0010] According to embodiments of this disclosure, the KV virtual address determination method further includes: in response to the absence of a first hash value that fails to match, using the first virtual address space and the first KV virtual address offset as the target virtual address space and the target KV virtual address offset.

[0011] According to embodiments of this disclosure, the KV tensor pool includes a set of free physical blocks, a set of mapped physical blocks, and a set of virtual address spaces.

[0012] According to embodiments of this disclosure, the KV virtual address determination method further includes: in response to the number of free physical blocks being less than or equal to a first threshold, creating new free physical blocks to supplement the free physical block set based on the current remaining storage space of the target processor.

[0013] According to embodiments of this disclosure, the first threshold is determined based on a preset number of lexical units and the size of the physical block.

[0014] According to embodiments of this disclosure, the KV virtual address determination method further includes: in response to the target processor's available storage resources not meeting the load requirements, releasing the storage space corresponding to at least a portion of the physical blocks in the KV tensor pool.

[0015] According to embodiments of this disclosure, releasing the storage space corresponding to at least a portion of the physical blocks in the KV tensor pool includes: releasing the storage space corresponding to at least a portion of the free physical blocks in the free physical block set according to load requirements.

[0016] According to embodiments of this disclosure, releasing storage space corresponding to at least a portion of physical blocks in a physical block pool includes: in response to the fact that the storage space corresponding to each physical block in the set of free physical blocks does not meet the load demand, obtaining the citation count of each physical block in the set of mapped physical blocks, and / or the number of processed tokens; determining at least one second physical block from the set of mapped physical blocks based on the citation count and / or the number of processed tokens; and releasing the storage space corresponding to each physical block in the set of free physical blocks, as well as the storage space corresponding to the second physical block.

[0017] Another aspect of this disclosure provides a KV virtual address determination apparatus, comprising: a first acquisition module for acquiring a target request, the target request including a term ID; a first determination module for determining a first hash value based on the term ID and the size of a physical block, the size of the physical block being determined based on parameters of a target model and the hardware page granularity of a target processor, the target model being configured on a target processor; a second determination module for determining the number of target physical blocks from a KV tensor pool based on the first hash value; and a third determination module for determining a target virtual address space and a target KV virtual address offset based on the number of target physical blocks; wherein the KV tensor pool is determined based on a preset number of terms and the size of the physical blocks.

[0018] Another aspect of this disclosure provides an electronic device, including: a memory, wherein at least a portion of the memory's storage space is divided into multiple physical blocks, the size of which is determined based on parameters of a target model and the hardware page granularity of a target processor, and the multiple physical blocks constitute a KV tensor pool, the KV tensor pool being determined based on a preset number of tokens and the size of the physical blocks;

[0019] The scheduler performs the following operations: obtains a target request, which includes a token ID; determines a first hash value based on the token ID and the size of the physical block; determines the number of target physical blocks from multiple physical blocks based on the first hash value; and determines the target virtual address space and the target KV virtual address offset based on the number of target physical blocks.

[0020] Another aspect of this disclosure provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform a KV virtual address determination method according to any of the foregoing embodiments.

[0021] Another aspect of this disclosure provides a computer program product, including a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the operation of the KV virtual address determination method of any of the foregoing embodiments. Attached Figure Description

[0022] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0023] Figure 1 A flowchart illustrating a KV virtual address determination method according to an embodiment of the present disclosure is shown schematically.

[0024] Figure 2 A flowchart illustrating the determination of the number of target physical blocks in a KV virtual address determination method according to an embodiment of the present disclosure is shown.

[0025] Figure 3 Another flowchart illustrating the determination of the number of target physical blocks in the KV virtual address determination method according to an embodiment of the present disclosure is shown schematically.

[0026] Figure 4 Another flowchart illustrating a KV virtual address determination method according to an embodiment of the present disclosure is shown schematically.

[0027] Figure 5 Another flowchart illustrating a KV virtual address determination method according to an embodiment of the present disclosure is shown schematically.

[0028] Figure 6 This schematically illustrates the structure of the KV tensor pool in the KV virtual address determination method according to an embodiment of the present disclosure;

[0029] Figure 7 A block diagram of a KV virtual address determination apparatus according to an embodiment of the present disclosure is illustrated schematically; and

[0030] Figure 8 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Detailed Implementation

[0031] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0033] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0034] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0035] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0036] This disclosure provides a method for determining a key-value (KV) virtual address, comprising: obtaining a target request, the target request including a term ID; determining a first hash value based on the term ID and the size of a physical block, the size of the physical block being determined based on parameters of a target model and the hardware page granularity of a target processor, the target model being configured on a target processor; determining the number of target physical blocks from a KV tensor pool based on the first hash value; and determining a target virtual address space and a target KV virtual address offset based on the number of target physical blocks; wherein the KV tensor pool is determined based on a preset number of terms and the size of the physical blocks.

[0037] Figure 1 A flowchart illustrating a KV virtual address determination method according to an embodiment of the present disclosure is shown schematically.

[0038] like Figure 1 As shown, the KV virtual address determination method may include at least operations S110~S150.

[0039] In operation S110, a target request is obtained, which includes token IDs. A target request refers to a request for an inference task to be processed, typically containing user-input prompts or context information generated in the previous iteration. A token ID is a unique numerical identifier obtained after processing text data through a tokenizer, representing the smallest semantic unit in the input sequence. Obtaining a target request can be done by reading a task to be scheduled from a request queue or by receiving inference instructions transmitted in real time. For example, the request queue manager receives a new text generation request containing a series of token IDs representing the input text and assigns a unique request identifier to the request.

[0040] In operation S120, the first hash value is determined based on the term ID and the physical block size. The physical block size is determined based on the parameters of the target model and the hardware page granularity of the target processor. The target model is configured on the target processor. The physical block size refers to the capacity of the smallest storage unit allocated in the physical memory space. Determining this size must simultaneously satisfy two constraints: first, the model dimension, which is the ability to accommodate KV data corresponding to an integer number of terms, avoiding the storage of single term data across blocks; second, the hardware dimension, which is alignment with the hardware page size of the target processor (such as a base page or big page) to reduce the overhead of virtual address to physical address translation. The first hash value is a feature fingerprint generated based on the term ID content, used to identify specific term sequence content for subsequent lookup and matching in the cache.

[0041] In operation S130, the number of target physical blocks is determined from the KV tensor pool based on the first hash value. The KV tensor pool is a set of pre-allocated or dynamically managed physical memory blocks used to store and reuse KV data. Determining the number of target physical blocks means evaluating the physical storage resources required for the current request in the KV tensor pool based on the calculated hash value.

[0042] In operation S140, the target virtual address space and target KV virtual address offset are determined based on the number of target physical blocks. The KV tensor pool is determined based on the preset number of terms and the size of the physical blocks. The target virtual address space is a contiguous virtual memory region reserved for this request. The target KV virtual address offset refers to the displacement of the starting position of the KV data to be written or read relative to the base address within the aforementioned contiguous virtual address space. Since the virtual address space is contiguous, this offset directly depends on the number of physical blocks.

[0043] For example, based on the determined number of target physical blocks (e.g., the number represents the cumulative sequence number N of the target physical blocks in the contiguous virtual address space), the number is calculated with the physical block size (e.g., (N-1) * physical block size), and the result of the calculation is determined as the target KV virtual address offset.

[0044] According to the embodiments of this disclosure, by combining model parameters and hardware page granularity to determine the physical block size, and calculating the hash value based on the selected lexical ID, a precise mapping between physical resources and logical data is achieved. The flexibility of physical blocks is utilized to reduce memory fragmentation. At the same time, through hardware page alignment and contiguous virtual address mapping, the continuity of memory access during computation is ensured, avoiding frequent address translation and context switching overhead. Thus, while improving memory utilization, inference efficiency is significantly improved.

[0045] According to embodiments of this disclosure, an initialization operation is included before operation S110. The initialization operation includes: component initialization; determining the physical block size; and KV tensor pool initialization. The initialization operation refers to the process of pre-configuring the software component state, hardware resource parameters, and memory space during the inference engine startup phase or before receiving the first request. Through the initialization operation, a basic environment for dynamic memory management is established, ensuring a rapid response to memory allocation requirements during subsequent scheduling.

[0046] The component initialization process initializes the request queue manager, request scheduler, key-value tensor manager, and dynamic key-value pool manager. These components together constitute the control plane of the inference engine. Specifically, the request queue manager maintains the order of pending tasks; the request scheduler determines which tasks enter the current execution batch; the key-value tensor manager maintains page tables or mapping tables from logical addresses to physical addresses; and the dynamic key-value pool manager monitors physical memory usage and performs actual memory allocation and deallocation.

[0047] The physical block size is determined by calculating the size of a single token (KV) based on the model parameters, and then determining the physical block size based on the size of the single token (KV) and the hardware page granularity. The size of a single token (KV) refers to the GPU memory space required to process the key and value vectors generated for each token during inference. This size is determined by the network architecture parameters of the target model, including the number of model layers, the dimension of hidden layers, the number of key-value heads, the dimension of attention heads, and the data storage precision. Hardware page granularity refers to the smallest or specific memory mapping unit supported by the target processor's (e.g., GPU) memory management unit, commonly including basic pages (e.g., 4KB) and large pages (e.g., 64KB, 2MB, etc.).

[0048] The logic for determining the physical block size lies in achieving "double alignment": on the one hand, the size of the physical block should be an integer multiple of the hardware page granularity to take advantage of the hardware's big page mapping feature and reduce address translation overhead; on the other hand, the physical block should be able to accommodate an integer number of KV data to avoid the data of a single word being split and stored in different physical blocks, thereby ensuring the continuity of memory access.

[0049] For example, assuming the target model configuration, the calculated KV data size for a single term is 64KB based on the number of layers, dimensions, and precision. If the target processor supports multiple page granularities such as 4KB, 64KB, and 2MB, to achieve efficient mapping, the physical block size can be set to a multiple of 64KB (e.g., 64KB). In this case, each physical block can exactly hold the KV data of one term, and it is perfectly aligned with the large page granularity of 64KB.

[0050] The KV tensor pool initialization process begins by calculating the lexical budget required for a single inference iteration. The lexical budget is determined based on the user-configured maximum batch size of lexical units, or the product of the maximum number of requests and the maximum context length of the model, representing the maximum load the system may face in a single scheduling iteration. Secondly, based on this lexical budget and the previously determined physical block size, the baseline number of physical blocks required to satisfy the budget is calculated. Finally, according to a preset redundancy coefficient (e.g., N, N≥1), a certain number of physical blocks (e.g., N times the baseline number) are pre-allocated. The handles, sizes, and other information of these pre-allocated physical blocks are recorded and stored in the free physical block set to construct the initial KV tensor pool. Preferably, N=2.

[0051] According to embodiments of this disclosure, by accurately calculating the physical block size during the initialization phase and aligning it with the hardware page granularity, the performance loss caused by unaligned access during inference is eliminated. Simultaneously, by pre-allocating a redundant KV tensor pool based on lexical budget, latency caused by frequent memory allocation during inference is avoided, and a dynamic pooling mechanism ensures elastic supply of memory resources during load fluctuations, effectively reducing the risk of memory overflow and improving system throughput.

[0052] According to an embodiment of this disclosure, there is a one-to-one correspondence between a lexical ID and a lexical. Operation S120 includes: determining the number of lexical units that can be stored in a physical block based on the size of the physical block; selecting lexical IDs based on the number of lexical units to obtain at least one set of lexical IDs; and determining a first hash value corresponding to each set of lexical IDs based on the lexical IDs contained in each set of lexical IDs.

[0053] Based on the size of the physical block, the number of tokens that a physical block can store is determined. The number of tokens that can be stored refers to the maximum integer number of token key-value (KV) data that a single physical memory block can completely store without overflow. For example, if the model parameters calculate that a single token KV data occupies 8KB of video memory, and the physical block size is set to 64KB according to the hardware page granularity, then the calculated number of tokens that a physical block can store is 8.

[0054] Based on the number of items to be saved, at least one set of lexical IDs is obtained. Each lexical ID set is a logical unit obtained by segmenting or grouping the lexical sequence in the target request based on the number of items to be saved. Each set contains no more than the number of lexical IDs to be saved, representing the data content that will be stored in the same physical block in the future. The process of selecting lexical IDs is essentially a sequence segmentation or sliding window process, dividing the long sequence of lexical IDs into multiple subsequences according to the step size of the number of items to be saved. Each subsequence constitutes a lexical ID set.

[0055] For example, assuming the number of items to be saved is 8 and the target request contains 20 token IDs, the first 8 token IDs are selected in sequence to form the first set, the 9th to 16th token IDs are selected to form the second set, and the remaining 4 token IDs are selected to form the third set.

[0056] Based on the term IDs contained in each of the term ID sets, a first hash value is determined for each term ID set. The first hash value is a feature fingerprint used to uniquely identify the content of the term ID set. Since this set corresponds to the storage content of a future physical block, the hash value effectively becomes the content index of that "potential physical block." The determination of the first hash value is based on the specific values ​​and arrangement order of all term IDs within the set. For example, for a set containing 8 term IDs, the values ​​of these 8 IDs are used as input parameters, and a unique string or value is calculated using a hash algorithm (such as MD5, SHA-256, or a custom hash function), which serves as the first hash value corresponding to that set.

[0057] According to embodiments of this disclosure, by determining the number of items to be stored based on the physical block size and grouping the token IDs accordingly, pre-alignment of logical data granularity and physical storage granularity is achieved. This "group first, then hash" mechanism ensures that each hash value accurately represents the data payload of a standard physical block, providing a standardized indexing basis for subsequent precise cache lookup, reuse, and management at the "block" level, avoiding the huge indexing overhead caused by managing individual tokens.

[0058] Figure 2 A flowchart illustrating the determination of the number of target physical blocks in a KV virtual address determination method according to an embodiment of the present disclosure is shown.

[0059] like Figure 2 As shown, based on the aforementioned embodiments, operation S130 may include operations S210 to S230.

[0060] In operation S210, the second hash value corresponding to each mapped physical block is obtained. The second hash value refers to the characteristic fingerprint of the data stored in the physical blocks in the current KV tensor pool that are in the "mapped" or "occupied" state. These physical blocks store KV data generated by previous inference tasks, and the second hash value is used to identify the specific content of this historical data so as to determine whether it can be reused by the current request. For example, the block management table of the KV tensor pool is accessed to read the list of hash values ​​corresponding to all physical blocks currently marked as "active" or "cached".

[0061] In operation S220, matching is performed based on the first hash value and the second hash value. Matching refers to comparing the features of the term ID set generated by the current request (the first hash value) with the features of the existing data in the KV tensor pool (the second hash value) to identify whether the KV data required by the current request already exists in the video memory. For example, all the first hash values ​​calculated by the current request are traversed, and the list of read second hash values ​​is searched sequentially to determine whether there is a record with the same hash value.

[0062] In operation S230, in response to the absence of a first hash value matching the second hash value, the target number of free physical blocks in the KV tensor pool is determined. The absence of a matching first hash value means that part or all of the data sequence of the current request cannot be found in the existing KV cache, i.e., a cache miss has occurred. Determining the target number means calculating how many new physical blocks need to be requested from the free resource pool to store the KV data to be calculated for these misses. For example, if the current request generates 3 first hash values, and comparison reveals that 3 of these hash values ​​do not exist in the second hash value list, then it is determined that the corresponding number (i.e., 3) of free physical blocks need to be obtained from the free set of the KV tensor pool as the target number.

[0063] According to the embodiments of this disclosure, by comparing the first hash value and the second hash value, and determining the target number of free physical blocks in the case of a failed match, a "demand allocation" video memory management strategy is realized. This accurately identifies which data is brand new, and thus only requests physical blocks for data segments that truly need new storage space, avoiding blind full pre-allocation and providing an accurate basis for subsequent dynamic video memory scheduling.

[0064] Figure 3 Another flowchart illustrating the determination of the number of target physical blocks in the KV virtual address determination method according to an embodiment of the present disclosure is shown.

[0065] like Figure 3 As shown, based on the aforementioned embodiments, operation S130 may include operations S310 to S330.

[0066] In operation S310, the number of first tokens in this scheduling is determined based on the target request. The number of first tokens refers to the number of tokens planned to be fed into the model for computation and to generate KV data in the current inference iteration step (whether in the pre-filling stage or the decoding stage). Since the target request contains a specific token ID sequence, this number is the length of the token ID sequence involved in this scheduling. For example, parsing the scheduling information of the target request, if the current stage is pre-filling and the prompt word length is 128, then the number of first tokens is determined to be 128; if the current stage is decoding and one new token is generated, then the number of first tokens is determined to be 1.

[0067] In operation S320, the number of second tokens corresponding to the saved tokens and the number of mapped physical blocks are obtained. The number of second tokens refers to the cumulative total number of tokens for which the request has been computed in previous inference iterations and whose KV data has been successfully stored in physical memory. The number of mapped physical blocks refers to the total number of physical blocks that have been allocated to the request and bound to its virtual address space at the current moment. For example, by accessing the KV tensor management table through the request ID index, the number of second tokens is obtained by reading the "Number of Saved Tokens" field in the request record, and the number of mapped physical blocks is obtained by reading the length of the "Physical Block List" field.

[0068] In operation S330, the number of target physical blocks is determined based on the number of first and second tokens, and the number of mapped physical blocks. Determining the number of target physical blocks essentially involves calculating the number of incremental physical blocks needed to accommodate the new data. The calculation logic is typically as follows: First, the number of first and second tokens is added to obtain the total number of tokens; second, the total number of physical blocks required to store this total number of tokens is calculated based on the capacity of the physical blocks (i.e., the number of tokens a single physical block can hold); finally, the total number of physical blocks is subtracted from the number of mapped physical blocks, and the difference is the number of new target physical blocks that need to be requested from the free pool.

[0069] For example, suppose the first word element has 10 (newly added), the second word element has 6 (already saved), and the physical block capacity is 8. Then the total number of words is 16, and theoretically 2 physical blocks (16 / 8) are needed. If the number of mapped physical blocks is 1 (only covering the first 8 words), then the calculated number of target physical blocks is 2-1=1.

[0070] Figure 4 Another flowchart illustrating a KV virtual address determination method according to an embodiment of the present disclosure is shown schematically.

[0071] like Figure 4 As shown, based on the aforementioned embodiments, the KV virtual address determination method may further include operations S410~S420.

[0072] In operation S410, in response to the existence of a first hash value that matches the second hash value, the first physical block corresponding to the successfully matched second hash value is determined. "The existence of a match with the second hash value" indicates a cache hit, meaning that a portion of the KV data required by the current request has been computed and stored in a previous inference task and has not been cleared. The first physical block refers to the physical memory unit in the KV tensor pool that actually stores this reusable data.

[0073] For example, if the first hash value calculated from a certain set of terms in the current request matches the second hash value recorded in the KV tensor pool, then the physical block I associated with that record is read, and that physical block is identified as the first physical block. Simultaneously, the reference count of that physical block is typically incremented to prevent it from being accidentally released.

[0074] In operation S420, the first virtual address space corresponding to the first physical block and the first KV virtual address offset are determined. The first virtual address space refers to the logical address range within the contiguous virtual memory reserved in the target request that is specifically used to map the aforementioned first physical block. The first KV virtual address offset is the displacement of the starting position of this range relative to the base address of the entire virtual address space. Determining these two parameters is to "attach" the multiplexed physical block to the contiguous logical view of the current request.

[0075] For example, assuming the first successfully matched physical block corresponds to the second logical block in the request sequence (i.e., the (N+1)th block, where N=1), and the physical block size is 64KB, then the first KV virtual address offset is determined to be 64KB (1*64KB), and the first virtual address space is an address range with a length of 64KB starting from the offset of 64KB. Subsequently, the first physical block is mapped to this virtual address range.

[0076] According to embodiments of this disclosure, by identifying and reusing existing physical blocks (first physical blocks) and accurately mapping them to the currently requested virtual address space, repeated calculation and storage of the same lexical sequences are avoided, reducing computational overhead and memory usage. Furthermore, by directly reusing existing data, inference latency is significantly shortened, achieving efficient shared pre-inference acceleration.

[0077] Figure 5 Another flowchart illustrating a KV virtual address determination method according to an embodiment of the present disclosure is shown schematically.

[0078] like Figure 5 As shown, based on the aforementioned embodiments, the KV virtual address determination method may further include operation S510, operation S130 may include operation S520, and operation S140 may include operations S530~S540.

[0079] In operation S510, in response to the existence of a first hash value that failed to match, the number of third transcribed words is determined. The number of third transcribed words is the total number of transcribed words in the set of transcribed word IDs corresponding to each first hash value that failed to match. "The existence of a failed match" means that some transcribed word sequences in the request do not have corresponding physical blocks in the current KV cache, i.e., a cache miss has occurred. The number of third transcribed words is essentially the scale of incremental data in this request that cannot reuse historical data and must be recalculated and stored. Determining the number of third transcribed words is a statistical analysis of the missed parts, usually by summing the sizes of the transcribed word ID sets represented by all the hash values ​​that failed to match. For example, suppose the request contains 3 transcribed word groups (8 transcribed words in each group), where the hash of the first group matches successfully, and the hashes of the second and third groups fail to match. Then, the total number of transcribed words in the second and third groups is counted (8+8=16), and the number of third transcribed words is determined to be 16.

[0080] In operation S520, the number of target physical blocks is determined from the KV tensor pool based on the number of third lexical terms. The number of target physical blocks specifically refers to the number of new physical blocks that need to be requested from the free set of the KV tensor pool to store the incremental data of the aforementioned third lexical term count. Typically, this is done by dividing the third lexical term count by the capacity of a single physical block and rounding up to ensure that the allocated physical space is sufficient to accommodate all the new data. For example, if the third lexical term count is 16 and the capacity of a single physical block is 8, then it is calculated that 2 physical blocks need to be obtained from the free pool as the number of target physical blocks.

[0081] During operation S530, the second virtual address space and the second KV virtual address offset are determined based on the number of target physical blocks. The second virtual address space is a virtual memory region specifically reserved for the newly added physical blocks. The second KV virtual address offset is the relative starting position of this region within the entire contiguous virtual address space of the request. The offset is determined based on the logical order of unmatched data in the original request sequence. Typically, unmatched data is located after matched data. For example, assuming that the matched prefix data occupies the first 16KB (0~16KB) of the virtual address space, and the number of newly allocated target physical blocks is 1 with a size of 8KB, then the second KV virtual address offset is determined to be 16KB, and the second virtual address space is the range of 16KB to 24KB.

[0082] During operation S540, the target virtual address space and target KV virtual address offset are determined based on the first virtual address space and first virtual address offset, the second virtual address space and second KV virtual address offset. The target virtual address space is the union of the first virtual address space (corresponding to the multiplexed block) and the second virtual address space (corresponding to the new block), forming a complete and contiguous virtual memory mapping view. The target KV virtual address offset contains the correct displacement information of all data segments (whether multiplexed or newly added) at their respective locations. For example, the first virtual address mapping relationship (0~16KB) pointing to the multiplexed physical block and the second virtual address mapping relationship (16KB~24KB) pointing to the newly allocated physical block are merged to construct the complete page table mapping for the request, enabling the GPU to seamlessly read KV data distributed in different physical locations by accessing the contiguous virtual addresses from 0 to 24KB.

[0083] According to the embodiments of this disclosure, by distinguishing between successfully matched and unsuccessfully matched data segments and performing "reuse mapping" and "new allocation mapping" respectively, the historical cache is fully utilized to reduce computation and video memory overhead, while also flexibly allocating resources for new data. Finally, the fragmentation and source differences of the underlying physical memory are shielded through a unified virtual address space, thereby maximizing the utilization of video memory resources while ensuring inference efficiency.

[0084] Continue to refer to Figure 5 As shown, based on the aforementioned embodiments, the KV virtual address determination method may further include operation S550.

[0085] In operation S550, in response to the absence of a first hash value that failed to match, the first virtual address space and the first KV virtual address offset are used as the target virtual address space and the target KV virtual address offset. "No first hash value that failed to match" indicates a full cache hit. This means that the KV data of all token sequences contained in the current target request have been calculated and fully stored in the KV tensor pool during previous inference processes, and have not been evicted. In this case, no new physical memory resources need to be allocated for the current request, nor is it necessary to calculate the address space of incremental data. The virtual address mapping relationship (i.e., the first virtual address space and the first KV virtual address offset) corresponding to all successfully matched first physical blocks is directly confirmed as the final address configuration of the current request. This means that the virtual address space of the current request will be completely mapped to the existing physical blocks.

[0086] For example, suppose the target request is a fixed system prompt word that has just been processed in a previous task. Through hash comparison, all block hash values ​​of this prompt word are found in the KV tensor pool. In this case, the mapping information of these existing physical blocks can be directly reused, pointing the virtual address of the current request to these physical blocks, without performing any new physical block requests or data write operations.

[0087] According to the embodiments of this disclosure, by directly reusing the existing virtual address mapping relationship when the full cache hit occurs, new physical memory allocation and redundant KV data calculation are completely avoided, memory usage is reduced to a minimum, and inference response speed is greatly improved. Especially when dealing with high-frequency repetitive requests or fixed prefix scenarios, near-instantaneous inference startup can be achieved.

[0088] Figure 6 The diagram illustrates the structure of the KV tensor pool in the KV virtual address determination method according to an embodiment of the present disclosure.

[0089] According to embodiments of this disclosure, the KV tensor pool includes a set of free physical blocks, a set of mapped physical blocks, and a set of virtual address spaces. The KV tensor pool is an intermediate management layer located between GPU memory and inference applications. By maintaining these three sets, it achieves pooled management and logical mapping of physical memory resources.

[0090] The virtual address space set (corresponding to the upper layer in the attached diagram) is used to record the logical view information of each inference request. It contains one or more virtual address objects, each object corresponding to an inference sequence (Seq), and records the KV tensor identifier (KV tensor) associated with the sequence, the current number of tokens (Num_tokens), and the physical block list (PHBlist). The physical block list defines the binding relationship between the virtual address space and the underlying physical blocks.

[0091] For example, such as Figure 6 As shown, the virtual address space set contains an object identified as "Seq:0", which records the number of tokens currently being processed as 32 and points to physical blocks Ph1 and Ph2 through the physical block list; it also contains an object identified as "Seq:1", which points to physical blocks Ph1, Ph3, and Ph4. The dashed arrows indicate the mapping from logical virtual addresses to specific physical blocks.

[0092] The set of mapped physical blocks (corresponding to the layers in the attached diagram) is used to store physical blocks that are currently being referenced or are in a cached state. Each physical block object (Ph) in the set maintains two key attributes: a hash value and a reference count. The hash value is used to identify the data content stored in the block, and the reference count is used to record how many virtual address space objects are currently sharing the block.

[0093] For example, such as Figure 6 As shown, physical block "Ph:1" is located in the set of mapped physical blocks, with a recorded hash value of 3 and a reference count of 2. This indicates that the data stored in this physical block is shared by two requests (such as Seq:0 and Seq:1), achieving memory sharing. Meanwhile, physical block "Ph:2" has a reference count of 1, indicating that it is exclusively accessed by only one request.

[0094] The set of free physical blocks (corresponding to the lower layer in the attached diagram) is used to maintain currently unused physical block resources, serving as a buffer pool for allocation. Physical block objects in this set are typically in an initialized state, with their hash values ​​and reference counts usually set to zero (Hash:0, Ref:0), awaiting allocation by the scheduler. For example, physical blocks "Ph:5" to "Ph:8" are located in the free physical block set; they directly correspond to the underlying GPU memory space and are currently in a free state (Ref:0), ready to be used to respond to new inference requests.

[0095] Based on the foregoing embodiments, the KV virtual address determination method may further include a physical block addition operation.

[0096] In the physical block addition operation, in response to the number of free physical blocks being less than or equal to a first threshold, new free physical blocks are created and added to the free physical block set based on the target processor's currently remaining storage space. When the available resources in the free physical block set fall below the first threshold, more physical memory resources are actively requested from the underlying hardware. Creating these resources based on the target processor's currently remaining storage space means that the expansion operation is constrained by hardware physical boundaries; it must ensure that the requested amount does not exceed the GPU's remaining memory capacity to prevent memory overflow.

[0097] Creating a new free physical block refers to calling the memory allocation interface of the computing device, allocating new physical memory pages in the video memory heap, encapsulating them as physical block objects (initializing handles, clearing hash values, etc.), and finally registering these objects to the free physical block set for subsequent scheduling.

[0098] For example, real-time monitoring might detect that there are only 5 physical blocks remaining in the current set of free physical blocks, which is below the set first threshold (e.g., 20). At this point, checking the GPU memory reveals that there is still 10GB of free space. Therefore, based on a preset step size or difference (e.g., to supplement to 20 or more), the GPU requests a corresponding amount of memory space, instantiates it into a new physical block, and adds it to the set.

[0099] According to embodiments of this disclosure, the first threshold is determined based on a preset number of tokens and the size of a physical block. The first threshold represents the minimum amount of free resources the system expects to retain to maintain normal inference without blocking, and is often referred to as the "water level." The preset number of tokens (i.e., the token budget) is typically set based on the maximum batch size of tokens, representing the maximum amount of resources that may be consumed in a single inference iteration. The logic for determining the first threshold is typically as follows: first, calculate how many physical blocks are needed to store the preset number of tokens (i.e., preset number of tokens / number of tokens accommodated in a single physical block), then multiply this result by a safety factor (e.g., N, N≥1) to finally obtain the first threshold.

[0100] For example, assuming a preset maximum word budget of 1024 per iteration and a single physical block size that can hold 64 words, the basic requirement for a single iteration is 16 physical blocks. If a safety factor N=2 is set, then the first threshold is determined to be 32 physical blocks. When there are fewer than 32 free blocks, expansion is triggered.

[0101] According to the embodiments of this disclosure, by setting a first threshold based on the lexical budget and monitoring the number of free blocks in real time, the "on-demand expansion" and "early warning replenishment" of video memory resources are realized. While ensuring that the inference task has sufficient resource reserves and avoiding inference interruption due to resource exhaustion, the security of expansion is ensured by checking the remaining storage space, thus achieving a dynamic balance between video memory utilization and system stability.

[0102] Based on the foregoing embodiments, the KV virtual address determination method may further include a physical block release operation.

[0103] In the physical block release operation, in response to the target processor's available storage resources not meeting load demands, at least a portion of the storage space corresponding to physical blocks in the KV tensor pool is released. "Available storage resources not meeting load demands" typically refers to a situation where, during model inference execution (e.g., forward propagation to calculate activation values), the target processor's (e.g., GPU) video memory usage reaches a critical value, triggering an out-of-memory warning, or there is no way to allocate necessary temporary video memory for new computational tasks. Releasing storage space means releasing the physical blocks from their video memory, returning it to the operating system or underlying driver, so that it can be reallocated to currently urgent computational tasks (e.g., activation value calculation).

[0104] For example, when the model executor requests video memory to store an intermediate activation tensor, it returns a failure signal. The dynamic KV pool manager captures this signal, determines that the current load demand exceeds the remaining video memory capacity, and thus triggers the release process.

[0105] According to embodiments of this disclosure, a physical block release operation may include: releasing the storage space corresponding to at least a portion of the free physical blocks in the free physical block set, based on load requirements. Although the physical blocks in the free physical block set do not store valid data, they still occupy physical video memory. Releasing these blocks incurs minimal cost because it does not involve discarding already computed key-value data. For example, if 500MB of video memory is urgently needed, and the free physical block set contains a total of 1GB of physical blocks, then 500MB (or all) of these 1GB of free blocks will be released first to meet the demand, while avoiding impacting ongoing inference tasks.

[0106] According to embodiments of this disclosure, releasing storage space corresponding to at least a portion of physical blocks in a physical block pool includes: in response to the fact that the storage space corresponding to each physical block in the set of free physical blocks does not meet the load demand, obtaining the citation count of each physical block in the set of mapped physical blocks, and / or the number of processed tokens; determining at least one second physical block from the set of mapped physical blocks based on the citation count and / or the number of processed tokens; and releasing the storage space corresponding to each physical block in the set of free physical blocks, as well as the storage space corresponding to the second physical block.

[0107] When releasing all free blocks fails to alleviate memory pressure, some computed cached data must be sacrificed, meaning some physical blocks containing stored data must be released. The reference count reflects the degree to which a physical block is shared (fewer references mean fewer affected requests), and the number of processed terms reflects the computational cost of the data corresponding to that block (fewer terms mean lower recomputation costs). Determining the second physical block involves selecting the block with the lowest sacrifice cost based on these two dimensions. Typically, the block with the lowest reference count (e.g., 1) and the fewest terms is preferred. For example, if a 200MB gap remains after releasing all free blocks, the set of mapped physical blocks is traversed, a subset of physical blocks with a reference count of 1 is selected, and within this subset, several blocks with the fewest processed terms are further selected as the second physical blocks until the released space meets the 200MB requirement. Subsequently, these blocks are released, and the inference requests that originally occupied these blocks are reset and moved back to the end of the request queue.

[0108] According to embodiments of this disclosure, by constructing a hierarchical resource release mechanism, when facing memory pressure, the crisis is alleviated first by releasing free blocks at zero cost; when cached data must be sacrificed, intelligent filtering is performed based on reference count and computational cost (number of lexicals), minimizing the impact on overall system throughput and computational resource waste.

[0109] Based on the foregoing embodiments, this disclosure also provides a model processing method, comprising: obtaining the target virtual address space corresponding to the target request and the current term length from the KV tensor management table, and accessing continuous KV data based on the target virtual address space; reading historical key-value data based on the target virtual address space, and performing attention calculation in conjunction with the query vector of the current term; determining the target key-value data of the current term, and writing the target key-value data to the position indicated by the target KV virtual address offset; updating the current term length and the target KV virtual address offset; and in response to the completion of target request processing, removing the mapping relationship between the target virtual address space and the physical block, and updating the reference count of the physical block.

[0110] The target virtual address space and current token length corresponding to the target request are obtained from the KV tensor management table, and continuous KV data is accessed based on the target virtual address space. The KV tensor management table is a metadata table used to record the logical view of the request and the physical resource status. By reading this table, the model executor can obtain the starting position (base address) of the current request in virtual memory and the amount of data processed (current token length).

[0111] Accessing contiguous key-value (KV) data based on the target virtual address space refers to using the virtual memory management mechanism of the target processor (such as a GPU) to directly read historical KV data, which may actually be scattered across different physical blocks, through contiguous virtual addresses. This mechanism masks the discreteness of the underlying physical blocks, allowing the computing core to load data as efficiently as accessing contiguous memory.

[0112] The process involves reading historical key-value data from the target virtual address space and performing attention calculations in conjunction with the query vector of the current term. The target key-value data for the current term is then determined and written to the location indicated by the target KV virtual address offset. The current term length and the target KV virtual address offset are then updated. First, historical key-value data of length "current term length × single term KV size" is read from the starting position of the virtual address. Then, combined with the query vector of the current term, attention mechanism calculations are performed using the target processor's computing unit. The write operation is an incremental update process. After calculating the target key-value data corresponding to the current term, it is directly written to the memory location pointed to by "virtual base address + KV virtual address offset". This write method avoids data movement, directly appending new data to the end of the logic. The update operation refers to synchronously modifying the records in the KV tensor management table after the data is written, incrementing the number of saved terms by 1, and increasing the virtual address offset by the size of one term KV, preparing for the next iteration.

[0113] In response to the completion of the target request processing, the mapping relationship between the target virtual address space and the physical block is released, and the reference count of the physical block is updated. When the inference task ends (e.g., a terminator is generated or the maximum length is reached), resource cleanup is required. Releasing the mapping relationship means disconnecting the request from the physical block at the logical level. Updating the reference count of the physical block usually means performing a decrement operation (reference count - 1). This is because a physical block may be shared by multiple requests (e.g., in a shared prefix scenario), and only when the reference count drops to 0 will the physical block be truly released back to the free pool; otherwise, it is only invisible to the currently completed request but remains in video memory for other requests to use. For example, after request A is processed, the system recognizes that it occupies physical blocks Ph1 and Ph2. The reference counts of Ph1 and Ph2 are each decremented by 1. If the count of Ph1 becomes 0, it is marked as reclaimable; if the count of Ph2 is still 1 (referenced by request B), it remains in the mapped state.

[0114] Figure 7 A block diagram of a KV virtual address determination apparatus according to an embodiment of the present disclosure is shown schematically.

[0115] like Figure 7 As shown, the KV virtual address determination device 700 may include a first acquisition module 710, a first determination module 720, a second determination module 730, and a third determination module 750.

[0116] The first acquisition module 710 is used to acquire a target request, which includes a token ID. In some embodiments, the first acquisition module 710 can be used to perform operation S110 in the KV virtual address determination method described above, which will not be elaborated here.

[0117] The first determining module 720 is used to determine a first hash value based on the token ID and the size of the physical block. The size of the physical block is determined based on the parameters of the target model and the hardware page granularity of the target processor. The target model is configured on the target processor. In some embodiments, the first determining module 720 can be used to perform operation S120 in the KV virtual address determination method described above, which will not be elaborated here.

[0118] The second determining module 730 is used to determine the number of target physical blocks from the KV tensor pool based on the first hash value. In some embodiments, the second determining module 730 can be used to perform operation S130 in the KV virtual address determination method described above, which will not be elaborated here.

[0119] The third determining module 740 is used to determine the target virtual address space and the target KV virtual address offset based on the number of target physical blocks; wherein, the KV tensor pool is determined based on the preset number of tokens and the size of the physical blocks. In some embodiments, the third determining module 740 can be used to perform operation S140 in the above-described KV virtual address determination method, which will not be elaborated here.

[0120] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0121] For example, any plurality of the first acquisition module 710, the first determination module 720, the second determination module 730, and the third determination module 750 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the first acquisition module 710, the first determination module 720, the second determination module 730, and the third determination module 750 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the first acquisition module 710, the first determination module 720, the second determination module 730, and the third determination module 750 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0122] It should be noted that the data processing system part in the embodiments of this disclosure corresponds to the data processing method part in the embodiments of this disclosure. The specific description of the data processing system part is referred to in the data processing method part, and will not be repeated here.

[0123] Figure 8 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0124] The memory 810 is used to store data and instructions. According to embodiments of this disclosure, such as... Figure 8 As shown, at least a portion of the storage space of memory 810 is divided into multiple physical blocks. The size of each physical block is determined based on parameters of the target model (e.g., number of layers, hidden layer dimension, number of attention heads, etc.) and the hardware page granularity of the target processor (e.g., page size of a GPU). These multiple physical blocks constitute a KV tensor pool 811, which is determined based on a preset number of tokens (e.g., the maximum batch processing token budget allowed by the system) and the size of the physical blocks. Memory 810 may include read-only memory (ROM), random access memory (RAM), high-bandwidth memory (HBM), or video memory, etc.

[0125] Scheduler 820 is used to control and manage the operation of electronic device 800. Scheduler 820 may include, for example, a general-purpose microprocessor (e.g., CPU), a graphics processing unit (GPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Scheduler 820 may also include onboard memory for caching purposes. Scheduler 820 performs various operations of the method flow according to embodiments of the present disclosure by executing programs stored in memory 810.

[0126] Specifically, the scheduler 820 is used to perform the following operations: obtain a target request, the target request including a token ID; determine a first hash value based on the token ID and the size of the physical block; determine the number of target physical blocks from the plurality of physical blocks based on the first hash value; and determine the target virtual address space and the target KV virtual address offset based on the number of target physical blocks.

[0127] like Figure 8 As shown, the electronic device 800 can also be connected to the model executor 900 (or, in some embodiments, the model executor 900 is integrated within the electronic device 800 as part of the processor). The scheduler 820 is connected to the memory 810 and the model executor 900 via a bus or internal interconnect mechanism. The model executor 900 accesses the corresponding physical block of the KV tensor pool 811 in the memory 810 to perform model inference calculations based on the target virtual address space and target KV virtual address offset determined by the scheduler 820.

[0128] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface, which is also connected to a bus. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface: an input section including a keyboard, mouse, etc.; an output section including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section including a hard disk, etc.; and a communication section including a network interface card such as a LAN card, modem, etc. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the input / output (I / O) interface as needed. Removable media, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on the drive as needed so that computer programs read from them can be installed into the memory 810 or the storage section as needed.

[0129] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by scheduler 820, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented using computer program modules.

[0130] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0131] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0132] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the KV virtual address determination method provided in the embodiments of this disclosure.

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0134] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for determining KV virtual address offset, comprising: Obtain the target request, which includes a term ID; A first hash value is determined based on the term ID and the size of the physical block, wherein the size of the physical block is determined based on the parameters of the target model and the hardware page granularity of the target processor, and the target model is configured on the target processor; Based on the first hash value, determine the number of target physical blocks from the KV tensor pool; Based on the number of target physical blocks, determine the target virtual address space and the target KV virtual address offset; The KV tensor pool is determined based on the preset number of tokens and the size of the physical block.

2. The method according to claim 1, wherein the lexical ID corresponds one-to-one with a lexical, and determining the first hash value includes: Based on the size of the physical block, determine the number of tokens that the physical block can store; Select word IDs based on the number of stored words to obtain at least one set of word IDs; Based on the lexical IDs contained in each of the lexical ID sets, determine the first hash value corresponding to each of the lexical ID sets.

3. The method according to claim 1, wherein determining the number of target physical blocks from the KV tensor pool based on the first hash value comprises: Obtain the second hash value corresponding to each mapped physical block; Matching is performed based on the first hash value and the second hash value; In response to the absence of a first hash value that matches the second hash value, the target number of free physical blocks in the KV tensor pool is determined.

4. The method according to claim 3, wherein determining the number of target physical blocks includes: The number of first tokens in this scheduling is determined based on the target request; Get the number of second tokens corresponding to the saved tokens and the number of mapped physical blocks; The number of target physical blocks is determined based on the number of the first term, the number of the second term, and the number of mapped physical blocks.

5. The method according to claim 3, further comprising: In response to the existence of a first hash value that matches the second hash value, the first physical block corresponding to the successfully matched second hash value is determined; Determine the first virtual address space and the first KV virtual address offset corresponding to the first physical block.

6. The method according to claim 1, wherein the KV tensor pool includes a set of free physical blocks, a set of mapped physical blocks, and a set of virtual address spaces.

7. The method according to claim 6, further comprising: In response to the number of free physical blocks being less than or equal to a first threshold, new free physical blocks are created and added to the set of free physical blocks based on the current remaining storage space of the target processor.

8. The method according to claim 7, wherein the first threshold is determined based on a preset number of lexical units and the size of the physical block.

9. The method according to claim 6, further comprising: In response to the fact that the available storage resources of the target processor do not meet the load requirements, at least a portion of the storage space corresponding to the physical blocks in the KV tensor pool is released.

10. An electronic device, comprising: The memory, at least a portion of which is divided into multiple physical blocks, the size of which is determined based on the parameters of the target model and the hardware page granularity of the target processor. The multiple physical blocks constitute a KV tensor pool, which is determined based on a preset number of tokens and the size of the physical blocks. The scheduler is configured to perform the following operations: obtain a target request, the target request including a token ID; determine a first hash value based on the token ID and the size of the physical block; determine the number of target physical blocks from the plurality of physical blocks based on the first hash value; and determine the target virtual address space and the target KV virtual address offset based on the number of target physical blocks.

Images