Data processing method and system, computing device, computer-readable storage medium

Abstract

The embodiment of the present specification provides a data processing method and system, a computing device, and a computer readable storage medium. The method comprises: a scheduling module receiving an inference request, performing cache query on a word sequence corresponding to the inference request, and generating and submitting a data loading task to an asynchronous task pool in a case where cache data corresponding to the word sequence exists in a data backup unit; a cache management module obtaining the data loading task from the asynchronous task pool, obtaining the cache data from the data backup unit according to a data identifier and a data storage location of the cache data carried by the data loading task, writing the cache data into a display memory, and writing a data loading completion notification into a loading confirmation queue; the scheduling module obtains the data loading completion notification from the loading confirmation queue, determines the cache data in the display memory according to the data loading completion notification, and generates an inference result according to the cache data. Through asynchronous decoupling of the scheduling module and the cache management module, inference blocking is reduced.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of model reasoning technology, and in particular to data processing methods and systems, computing devices, and computer-readable storage media. Background Technology

[0002] In autoregressive inference of large language models, while the tiered storage of KV Cache (GPU memory, RAM, and disk) can expand capacity, existing solutions suffer from significant blocking issues in concurrent execution. Specifically, loading operations of KV Cache across GPU memory, RAM, and disk are often accompanied by frequent thread synchronization mechanisms. Especially in multi-tensor parallelism (TP) scenarios, lock contention and wait amplification can easily occur between loading threads, scheduling threads, and inference threads, making it difficult for GPU (Graphics Processing Unit) computation and CPU (Central Processing Unit) data preparation to effectively overlap. This serialized thread cooperation mode often causes inference threads to be blocked while waiting for cached data to be ready, thus limiting end-to-end throughput and becoming a key bottleneck restricting the performance of high-concurrency, long-context inference. Summary of the Invention

[0003] In view of the above, the embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing system, a computing device, a computer-readable storage medium, and a computer program product, in order to solve the technical defects existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a data processing method is provided, applied to a data processing system, the data processing system including a scheduling module and a cache management module, the method comprising: The scheduling module receives an inference request, performs a cache query on the lexical sequence corresponding to the inference request, and generates a data loading task when it is determined that the cached data corresponding to the lexical sequence exists in the data backup unit, and submits the data loading task to the asynchronous task pool. The cache management module obtains the data loading task from the asynchronous task pool, retrieves the cached data from the data backup unit according to the data identifier and data storage location carried by the data loading task, writes the cached data into the video memory, generates a data loading completion notification corresponding to the data loading task, and writes the data loading completion notification into the loading confirmation queue. The scheduling module obtains the data loading completion notification from the loading confirmation queue, determines the cached data in the video memory based on the data loading completion notification, and executes the inference request to generate inference results based on the cached data.

[0005] According to a second aspect of the embodiments of this specification, a data processing system is provided, including a scheduling module and a cache management module, wherein: The scheduling module is used to receive inference requests, perform cache queries on the lexical sequence corresponding to the inference request, and generate a data loading task when it is determined that the cached data corresponding to the lexical sequence exists in the data backup unit, and submit the data loading task to the asynchronous task pool. The cache management module is used to obtain the data loading task from the asynchronous task pool, obtain the cached data from the data backup unit according to the data identifier and data storage location carried by the data loading task, write the cached data into the video memory, generate a data loading completion notification corresponding to the data loading task, and write the data loading completion notification into the loading confirmation queue. The scheduling module is further configured to obtain the data loading completion notification from the loading confirmation queue, determine the cached data in the video memory based on the data loading completion notification, and execute the inference request to generate inference results based on the cached data.

[0006] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which implement the steps of the above data processing method when executed by the processor.

[0007] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0008] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0009] This specification implements a data processing method in one embodiment. By separating the scheduling module and the cache management module, and introducing an asynchronous task pool and a load confirmation queue, a decoupled closed loop for task submission and completion notification is formed. Specifically, after receiving an inference request and determining that the cached data corresponding to the lexical sequence of the inference request exists in the data backup unit, the scheduling module can submit the data loading task to the asynchronous task pool non-blocking after generating the data loading task. It can continue to process other inference requests without waiting for the cached data to be ready. The cache management module independently obtains the data loading task from the asynchronous task pool, asynchronously executes the cached data loading, and writes the data loading completion notification to the load confirmation queue. The scheduling module can actively obtain the data loading completion notification from the load confirmation queue, retrieve the cached data from the video memory, execute the inference request based on the cached data, and generate the inference result. This design fundamentally avoids lock contention and wait amplification between the scheduling thread and the loading thread, allowing GPU computing (execution of inference requests triggered by the scheduling module) and CPU data preparation (retrieval of cached data by the cache management module) to fully overlap. Therefore, even in high-concurrency, multi-TP scenarios, the inference thread is no longer blocked while waiting for cached data, significantly reducing end-to-end inference latency and effectively improving the overall throughput of the system, thus solving the key bottleneck of concurrent execution blocking. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating a data processing method provided in one embodiment of this specification; Figure 2 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification; Figure 3 This is a system architecture diagram of a data processing system provided in one embodiment of this specification; Figure 4 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0011] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0012] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0013] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0014] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0015] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0016] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0017] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0018] KV Cache: A data structure used in autoregressive inference to cache the Key / Value tensors calculated from historical tokens at the attention layer. Its purpose is to avoid recalculating historical context in subsequent decoding stages, thereby reducing latency and improving throughput.

[0019] Longest Prefix Match: Finds the longest prefix path in the reusable cache that matches the current inference request, maximizing the KV Cache reuse depth and reducing additional computation.

[0020] RadixTree: A tree-like index structure that organizes the key space by prefix. In this scheme, it is used to store KV cache metadata, supporting prefix sharing, path-level state management, and efficient matching.

[0021] Hierarchical Cache: A mechanism that distributes KV Cache across different tiers (such as video memory, memory, and disk) according to media capabilities and manages them collaboratively, in order to balance access performance, capacity limits, and deployment costs.

[0022] TP (Tensor Parallelism): Tensor parallel inference mode, which divides model computation into multiple parallel processes / devices for collaborative execution. This solution focuses on the issues of KV cache consistency and synchronization overhead in multi-TP scenarios.

[0023] MHA (Multi-HeadAttention): A multi-head attention computation model where KV Cache is typically organized and accessed in a multi-head dimension.

[0024] MLA (Multi-head LatentAttention): An attention computation method tailored to specific model architectures. The KVCache organization method is different from MHA, but it also requires hierarchical caching and efficient transfer support.

[0025] TTFT (TimeToFirstToken): The time from when the inference request enters the system to when the first token is output, reflecting the first packet response capability and prefix reuse efficiency.

[0026] TPOT (TimePerOutputToken): The average time to generate each output token, reflecting the steady-state efficiency of the decoding phase.

[0027] E2E (End-to-EndLatency): The total delay from the input of a request to the completion of a full output, comprehensively reflecting the efficiency of system scheduling, computation, and data migration.

[0028] Throughput: The number of tokens or requests completed by the system per unit of time, used to measure overall processing capacity.

[0029] In autoregressive inference of large language models, KV Cache is one of the core acceleration mechanisms. Its principle is that when generating the t-th token, the model caches the Key / Value tensors generated by previous tokens in each attention layer, so that subsequent steps do not need to repeatedly calculate the Key / Value tensors of historical tokens, only performing incremental calculations on new tokens. Thus, in multi-turn dialogues, if the prefix cache can be hit, the overhead of repeated calculations can be significantly reduced, end-to-end inference latency can be shortened, and overall throughput can be improved. As applications expand from short dialogues to long contexts and high-concurrency services, the size of the KV Cache grows rapidly with sequence length, number of layers, and number of concurrent requests, gradually becoming the most significant bottleneck in GPU memory consumption in inference systems. However, due to the high cost of GPU memory, it is necessary to expand the KV Cache capacity and control costs through a layered collaborative approach of GPU memory, RAM, and disk, which is currently a key technical focus in building high-performance inference infrastructure.

[0030] However, existing hierarchical KV Cache solutions still face several core challenges in engineering implementation. First, at the metadata organization and hit mechanism level, vLLM (an open-source library for large language model inference and services) / Sglang (a large language model programming and inference framework) multi-level metadata storage implementation is primarily hash-based, making it difficult to express prefix sharing relationships. This results in longest prefix matching, path-level reuse, and eviction often relying on additional control logic, increasing scheduling complexity and reducing hit efficiency. Second, at the concurrent execution and cross-layer migration level, KV Cache backup, eviction, and loading between GPU memory, RAM, and disk are often accompanied by frequent synchronization. In multi-TP scenarios, lock contention and wait amplification easily occur between loading threads, scheduling threads, and inference threads, limiting the overlap between GPU computation and CPU data transfer, thus affecting end-to-end throughput. Finally, at the storage scalability level, after the disk layer expands from a single disk to multiple disks, without effective routing and load balancing mechanisms, problems such as hot disks, insufficient parallelism, and increased coordination overhead often arise, making it difficult to effectively translate capacity expansion into performance expansion.

[0031] To address the aforementioned technical problems, this specification provides a data processing method in its embodiments. One or more embodiments of this specification also relate to a data processing system, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0032] In view of the shortcomings of the aforementioned hierarchical KV Cache in terms of metadata organization, concurrent synchronization, and disk expansion, the data processing method provided in the embodiments of this specification proposes the following systematic solution.

[0033] First, addressing the technical problem that hash indexes struggle to express prefix sharing, the data processing method provided in this specification uses Radix Tree as a unified metadata structure. It incorporates longest prefix matching, cache reuse, and path-level eviction into the same semantic framework, avoiding the additional control logic of traditional point-to-point indexes, improving prefix hit efficiency and scheduling consistency, and is particularly suitable for long context scenarios.

[0034] To address the thread contention and synchronization blocking issues in cross-layer migration, the data processing method provided in the embodiments of this specification constructs an asynchronous closed-loop mechanism of "scheduling submission - cross-layer execution - status feedback": the scheduling module is only responsible for initiating load / write-back tasks and does not submit them to the asynchronous task pool in a blocking manner; the cache management module independently performs data transfer and notifies the scheduling module through the load / write-back confirmation queue after completion; at the same time, request-level fine-grained synchronization is adopted in multi-TP scenarios to effectively alleviate lock contention and wait amplification, so that GPU computing and CPU data transfer can fully overlap, thereby reducing end-to-end latency and improving throughput.

[0035] To address the performance bottleneck of single-disk expansion, the data processing method provided in the embodiments of this manual establishes a multi-level storage system of "video memory main cache - memory first-level backup - disk second-level backup", and introduces a multi-disk routing and load balancing mechanism at the disk layer. KV Cache shards are written in parallel to multiple NVMe (non-volatile memory fast channel) devices to avoid hot disks, achieve near-linear capacity expansion, and at the same time be compatible with unified management of different attention patterns such as MHA / MLA.

[0036] Actual tests show that, under TP=8 deployment, the data processing method provided in the embodiments of this specification reduces the first token latency by 31.05%–60.51%, the average token latency by 10.96%–34.05%, and the end-to-end latency by 12.28%–33.89%, significantly solving the concurrency blocking and scalability defects in the prior art and possessing high engineering implementation value.

[0037] See Figure 1 , Figure 1 This is a flowchart of a data processing method provided in one embodiment of this specification. The data processing method is applied to a data processing system, which includes a scheduling module and a cache management module. The method specifically includes the following steps.

[0038] Step 102: The scheduling module receives the inference request, performs a cache query on the lexical sequence corresponding to the inference request, and if it is determined that the cached data corresponding to the lexical sequence exists in the data backup unit, generates a data loading task and submits the data loading task to the asynchronous task pool.

[0039] The data processing method provided in one or more embodiments of this specification can be applied to various application scenarios, including large-scale multi-turn dialogue systems, long document analysis and summarization, code generation and completion, high-concurrency inference services, and model services compatible with multiple attention mechanisms. For large-scale multi-turn dialogue systems, such as intelligent customer service and virtual assistants, users interact with the inference system for extended periods and multiple turns, resulting in frequent reuse of the KV cache for historical dialogues. This data processing method can efficiently manage the prefix cache using Radix Trees, significantly reducing redundant computation overhead, and utilizes an asynchronous closed-loop mechanism to avoid I / O blocking, ensuring low-latency response under high concurrency. For long document analysis and summarization applications, such as in finance, law, and scientific research, users need to input documents containing hundreds of thousands of tokens at once for parsing, question answering, or summary generation. The hierarchical storage (GPU memory-RAM-disk) of this data processing method can accommodate extremely long contexts, and multi-disk parallel reading accelerates disk cache loading, significantly shortening the latency of first token generation. For code generation and completion applications, AI programming assistants in development tools need to process large codebase contexts in real time. This data processing method's prefix reuse capability can quickly match common code snippets (such as function headers and import statements), and its asynchronous I / O design avoids stuttering during editing, improving the interactive experience. For high-concurrency inference service applications, such as LLM services in the form of API (Application Programming Interface) gateways, it handles requests from tens of thousands of users simultaneously. The scheduling module and cache management module are decoupled, and with request-level fine-grained synchronization, lock contention under multiple process nodes (TPs) is effectively reduced, maximizing GPU utilization and thus improving overall system throughput. For model services compatible with multiple attention mechanisms, such as supporting KV cache organization and migration for different attention patterns like MHA and MLA, it can seamlessly adapt to various mainstream large models and provide a unified cache management backend.

[0040] Specifically, in a data processing system employing a tensor parallel (TP) strategy, a large language model is segmented and deployed across multiple GPUs. Each GPU runs an independent TP process, and each TP process contains a scheduling module and a cache management module. When an inference request arrives, the scheduling modules of each TP process simultaneously receive the request and obtain its lexical sequence. They then perform longest prefix matching (longest initial part matching) in their respective cache radix trees to determine whether the KV cache required by the inference request already exists in a data backup unit (such as memory or disk). In other words, each TP process independently executes the data processing method provided in the embodiments of this specification.

[0041] The data processing system can be understood as the overall software environment for running large language model inference services, including multiple GPU cards, CPU memory, disks, and program components that coordinate these resources. For example, the data processing system could be a server that has deployed a large model, containing a scheduling module and a cache management module.

[0042] The scheduling module can be understood as a software component responsible for receiving inference requests sent by users, querying cache status, deciding whether to load the cache, and advancing the request execution process. This scheduling module does not directly handle data; instead, it makes decisions and distributes tasks.

[0043] The cache management module can be understood as the software component that actually performs the movement of KV cache between different storage locations, including reading KV cache from disk to memory, copying it from memory to video memory, and writing it back. This cache management module is the actual executor of data movement.

[0044] An inference request can be understood as a complete input sent by the user to the large language model. This can be a piece of natural language text, which the large language model needs to generate an answer for. For example, an inference request could be the user inputting the natural language text "Please introduce the attractions in City B" into the large language model.

[0045] A lexical sequence can be understood as a list of the smallest semantic units after segmenting the natural language text (inference request) input by the user. It is a form that large language models can directly process. Continuing with the previous example, the lexical sequence corresponding to the inference request "Please introduce the attractions in City B" can be ["please", "introduce", "introduce", "City B", "of", "attractions"].

[0046] Cache lookup can be understood as the process by which the scheduling module searches the cache index structure for a pre-calculated key-value cache that matches the beginning of the inference request's token sequence. Using the previous example, cache lookup could be to check if ["please", "introduce", "let me explain"] already exists in the cache.

[0047] Cached data can be understood as the key-value tensors generated by the large language model when processing certain lexical units, stored in video memory, RAM, or disk for reuse in subsequent requests for the same lexical sequence. Continuing with the previous example, cached data can be a KV cache stored in video memory, RAM, or disk after historical inference requests have processed ["please", "introduce", "a little"].

[0048] The data backup unit can be understood as a secondary or tertiary storage unit relative to GPU memory, including host memory (a primary memory backup unit) and disk (a secondary disk backup unit). When GPU memory is insufficient or data is not frequently used, the KV Cache will be migrated to the data backup unit for storage.

[0049] A data loading task can be understood as a loading instruction for cached data generated by the scheduling module. It includes information such as the data identifier of the cached data to be loaded and the source storage location (data storage location: memory or disk), and is used to notify the cache management module to perform specific cached data movement. For example, a data loading task could be "load 64KB of data from disk file / nvme0 / cache.bin at offset 0000 to GPU memory address 0x0f0a0000".

[0050] The asynchronous task pool can be understood as a shared task queue between the scheduling module and the cache management module. The scheduling module adds a task to the pool and returns immediately, while the cache management module retrieves the task from the other end of the pool and executes it. This asynchronous task pool is lock-free to avoid thread blocking.

[0051] In practice, upon receiving an inference request, the scheduling module first converts the natural language text of the request into a sequence of terms. Then, based on this sequence, the scheduling module performs a longest starting part match in the cache index (such as a Radix Tree) to determine if the KV cache required by the inference request already exists in video memory or a data backup unit (memory or disk). If the query results show that the required cached data does not exist in video memory but does exist in a data backup unit (e.g., previous inference requests have processed terms with the same starting part), the scheduling module constructs a data loading task. This task explicitly specifies the unique identifier of the cached data to be loaded and its specific storage location in the data backup unit. Next, the scheduling module non-blockslyly places this data loading task into an asynchronous task pool and immediately returns to continue processing other inference requests without waiting for the cached data to be actually loaded. Therefore, the scheduling thread is not blocked by I / O operations, thus ensuring the system's responsiveness under high concurrency.

[0052] Specifically, the step of performing a cache query on the lexical sequence corresponding to the inference request, and generating a data loading task if it is determined that the cached data corresponding to the lexical sequence exists in the data backup unit, includes: Performing longest prefix matching on the token sequence corresponding to the inference request according to the cached radix tree, to obtain a target node that shares the longest prefix with the token sequence in the cached radix tree, wherein the metadata of cached data corresponding to the shared longest prefix is stored on the target node, and the metadata includes the data identifier and the data storage position; generating the data loading task according to the data identifier and the data storage position when it is determined that the cached data corresponding to the shared longest prefix exists in the data backup unit based on the data storage position.

[0053] Wherein, the cached radix tree can be understood as a tree-shaped data structure. Each node in the cached radix tree represents a token sequence (formed by sequential combination of all tokens from the root node to the current node), and each node directly records the metadata of the KV Cache corresponding to the token sequence. For example, the next node after the root node in the cached radix tree corresponds to the first token "you", and the next child node corresponds to the second token sequence "hello", and the node stores the position of the KV Cache of the token sequence "hello" in a memory or a disk.

[0054] Longest prefix matching can be understood as a process of starting from the first token of the token sequence corresponding to the inference request, performing word-by-word comparison downward along the cached radix tree, and finding the longest consecutive token sequence that can be matched in the cached radix tree. For example, the token sequence corresponding to the inference request is ["you", "hello", "how are you"], and nodes "you" and "hello" already exist in the cached radix tree, so the target node finally reached through longest prefix matching is "hello".

[0055] The target node can be understood as a node of the cached radix tree finally reached through longest prefix matching, and represents the starting token sequence shared with historical cached data in the token sequence corresponding to the inference request. The target node stores the metadata of the complete KV Cache corresponding to the starting token sequence. For example, the matched target node represents "hello", then the target node stores the metadata of the KV Cache corresponding to the two tokens "hello".

[0056] Metadata can be understood as attribute information describing cached data itself, including the unique data identifier of the cached data in a storage medium (video memory, memory or disk); data storage position refers to the specific address of the cached data in video memory, memory or disk (such as device identifier, offset, etc.), hierarchical status (which storage levels the cached data currently exists in), and the data storage position can also include data length / size (number of bytes occupied by the cached data), etc. For example, the metadata may be "cache block ID=1001, located at offset 0000 of disk file / nvme0 / cache.bin, size 64KB".

[0057] A data identifier can be understood as an ID used to uniquely identify a segment of KV cache. It can be an auto-incrementing ID, a hash value, or an encoding of a word sequence. The cache management module can use this data identifier to find the corresponding actual cached data.

[0058] Data storage location can be understood as the specific address of cached data within the storage medium. For memory, the data storage location may include a virtual address pointer, data length, etc.; for disk, the data storage location may include the device number, starting sector, and data length, etc. For example, the data storage location could be "mem_addr=0x0f0a0000,size=00000" or "disk=nvme0,offset=0000,size=00000".

[0059] In practice, upon receiving an inference request, the scheduling module first obtains the corresponding lexical sequence. Then, using the cache radix tree as an index, starting from the root node, the scheduling module compares each lexical in the sequence with a node in the cache radix tree, aiming to match the longest possible consecutive lexical sequence until no further matching is possible, thus finding the target node. This target node represents the longest starting lexical segment shared by the lexical sequence corresponding to the inference request and the historical cache. The target node stores metadata for the KV Cache corresponding to this longest starting lexical segment, including the data identifier and the specific data storage location. The scheduling module reads the data storage location field from this metadata to determine whether the cached data is currently located in the data backup unit (i.e., memory or disk). If the result is yes, the scheduling module constructs a data loading task based on the data identifier and data storage location in the metadata. This data loading task explicitly specifies the identity of the cached data to be loaded and where it will be read from. Finally, the scheduling module places this data loading task into the asynchronous task pool, and it returns immediately without waiting for the actual cached data to be loaded.

[0060] The data processing method provided in one or more embodiments of this specification, through longest prefix matching using a cache radix tree, can accurately locate the longest starting part of the token sequence corresponding to the inference request that is shared with the historical cache. This avoids prefix truncation or additional control logic under the hash point lookup method, significantly improving cache hit efficiency. The target node directly stores metadata, allowing the scheduling module to obtain the data identifier and data storage location at once without traversing multiple nodes or relying on external mapping tables, reducing query overhead. After determining that the cached data is located in the data backup unit, the scheduling module only generates a data loading task and submits it asynchronously to the asynchronous task pool, without blocking subsequent inference request processing. This fundamentally decouples cache location from data movement, enabling the scheduling thread to make decisions lightweight and quickly, laying the foundation for subsequent computation and I / O overlap, thereby effectively reducing thread waiting in high-concurrency scenarios and lowering the latency of the first token.

[0061] Specifically, the step of performing longest prefix matching on the lexical sequence corresponding to the inference request based on the cache cardinality tree to obtain the target node in the cache cardinality tree that shares the longest prefix with the lexical sequence includes: The root node of the cache radix tree is taken as the current node, and the first word of the word sequence is taken as the current word. Determine whether the current node has child nodes, and whether the word element corresponding to the child node matches the current word element. If so, the child node is taken as the current node, the child node is recorded, and the next word of the current word is taken as the current word. The steps of determining whether the current node has a child node and whether the word corresponding to the child node matches the current word are continued until the current node has no child node or the word sequence has been completely matched, and the loop stops. If not, then stop the loop; The child node that was last successfully matched is taken as the target node in the cache radix tree that shares the longest prefix with the lexical sequence.

[0062] The root node can be understood as the starting node of the cache radix tree. It does not carry any tokens, and the matching of the token sequence corresponding to each inference request starts from the root node.

[0063] The current node can be understood as the tree node of the cache radix tree that is being processed during the matching process. At the initial matching, the current node is the root node of the cache radix tree. As the matching moves down the tree, the current node will also be updated.

[0064] A child node can be understood as a next-level node directly connected to the current node. Each child node is connected to its parent node via an edge, and the edge is marked with at least one token. For example, the parent node represents the token "hello", and the edge of one of its child nodes is marked with the token "hello world", then the child node represents the token "hello world".

[0065] The current token can be understood as the token in the token sequence corresponding to the inference request that is being compared with the token on the tree node edge of the cached radix tree during the matching process.

[0066] Matching can be understood as checking whether there exists a child node among the child nodes of the current node whose token sequence carried on the edge can be completely aligned with the token sequence to be matched currently (starting from the current token). If the number of tokens on the edge is 1, only a single token needs to be compared; if the number of tokens on the edge is greater than 1 (compressed path), multiple tokens need to be compared one by one.

[0067] In specific implementation, the specific implementation steps for the scheduling module to perform the longest prefix matching are as follows: first, set the root node of the cached radix tree as the current node, and set the first token of the token sequence corresponding to the inference request as the current token. Then enter the circular matching process. If each child node among the child nodes of the current node corresponds to only one token (that is, the edge length is 1), the matching process will be relatively simple: check whether there is a child node of the current node whose token on the edge is the same as the current token. If it exists, move to the child node, set it as the new current node, record the new current node, and move the current token pointer to the next token, that is, take the next token as the current token; then repeat the above check. If it does not exist, stop the loop. If the cached radix tree adopts path compression (that is, the edge of a child node may contain multiple consecutive tokens), it is necessary to handle the comparison of multiple tokens on the edge during matching. Specifically: when a child node is found whose first token on the edge is the same as the current token, it cannot be directly considered that the matching is successful. Instead, subsequent tokens on the edge need to be compared one by one with the subsequent tokens in the token sequence corresponding to the inference request. For example, the current node has a child node, and the edge is marked as "hello world" (two tokens), the current token is "hello", and there is a subsequent token "world" in the token sequence corresponding to the inference request. In this case, it is necessary to compare "world" on the edge with the next token "world" in the token sequence corresponding to the inference request, and the child node can only be entered after complete matching. If there is an inconsistency during the comparison (for example, the token sequence corresponding to the inference request is "hello bad"), the matching fails and stops here.

[0068] In practical applications, regardless of whether it's a one-sided or multi-sided loop, the loop stops when one of the following conditions occurs: the current node has no child nodes that match the current token; the multi-token matching on the edge fails; all tokens in the token sequence corresponding to the inference request have been matched (in this case, the current node is the target node). After stopping, the last successfully matched child node (i.e., the last child node that was moved to and successfully recorded) is the target node.

[0069] For example: Suppose the cache radix tree already has the path root → "you" → "good" → "ma" (each edge contains one word), and the word sequence corresponding to the inference request is "hello ma". The matching process is as follows: If the root node has a child node "you", move to that node; if that node has a child node "good", move to that node; if that node has a child node "ma", move to that node; the word sequence corresponding to the inference request has ended, and the target node is the "ma" node. If the cache radix tree uses compressed paths, and the root node directly has a child node with an edge containing "hello ma" (three words), then during matching, the "hello ma" on the edge is compared one by one with the words in the word sequence corresponding to the inference request. After all matches are found, the target node is directly reached from that child node.

[0070] The data processing method provided in one or more embodiments of this specification employs a cache radix tree for longest prefix matching, explicitly expressing the prefix sharing relationship within the cache radix tree. This allows each query to simply traverse downwards along the tree, with complexity proportional to the matching length. This avoids the overhead of additional logic required to determine prefix inclusion relationships as in hash point lookup methods. Furthermore, since tree nodes directly carry metadata of the cached data corresponding to the prefix, the data storage location of the required cached data's hierarchical state can be obtained immediately after matching, eliminating the need for secondary lookups. This not only improves cache hit efficiency but also simplifies the implementation logic of prefix reuse and eviction. It is particularly suitable for scenarios with a large number of requests sharing a common prefix in long-context environments, thereby significantly reducing first-token latency and improving system throughput.

[0071] In practical applications, besides the scenario where all cached data resides in the data backup unit, another situation may exist: part of the cached data corresponding to the longest shared prefix exists in video memory, and the other part exists in the data backup unit. In this case, the scheduling module can generate a data loading task only for the portion of cached data located in the data backup unit. The specific implementation method is as follows: When it is determined, based on the data storage location, that the cached data corresponding to the longest shared prefix exists in the data backup unit, the data loading task is generated based on the data identifier and the data storage location, including: If, based on the data storage location, it is determined that some cached data in the cached data corresponding to the longest shared prefix exists in the data backup unit, a data loading task for the partial cached data is generated based on the data identifier and data storage location of the partial cached data.

[0072] Partial cached data can be understood as a portion of the complete cached data corresponding to the longest shared prefix, located in the data backup unit while the rest is in video memory. For example, for a prefix of 32 tokens, the cached data of the first 16 tokens is in video memory, while the cached data of the last 16 tokens is in disk. The cached data corresponding to the last 16 tokens is "partial cached data".

[0073] In practice, after completing the longest prefix matching and obtaining the target node, the scheduling module reads the metadata stored on that target node. This metadata includes the data blocks into which the cached data corresponding to the complete prefix is ​​divided (each block typically corresponds to a continuous string of tokens, such as 16 or 32 tokens) and the current data storage location of each data block. The scheduling module iterates through these data blocks, checking if each data block exists in GPU memory. For data blocks already in GPU memory, the scheduling module does not perform any processing and directly marks them as reusable. For data blocks not in GPU memory but existing in a data backup unit (memory or disk), the scheduling module records their data identifier and specific data storage location. Then, the scheduling module aggregates the information of all missing data blocks and constructs a data loading task for these partially cached data. This data loading task may contain multiple subtasks (one for each missing data block) or a batch task. The scheduling module submits the data loading task to the asynchronous task pool non-blocking and returns immediately to continue processing other inference requests. In this way, partially cached data already existing in GPU memory does not need to be reloaded; only the missing partially cached data triggers I / O operations, thus saving data transfer time and bandwidth.

[0074] For example: Suppose the longest shared prefix of the lexical sequence corresponding to the inference request corresponds to four data blocks (blocks 0-3). Metadata shows that blocks 0 and 1 are in video memory, block 2 is in RAM, and block 3 is on disk. The scheduling module only generates data loading tasks for blocks 2 and 3, carrying their respective data storage locations in RAM and disk, as well as their data identifiers. Blocks 0 and 1 are directly marked as available and do not participate in data loading.

[0075] The data processing method provided in one or more embodiments of this specification identifies the location of partially cached data with fine granularity and generates data loading tasks only for cached data missing in video memory. This avoids repeatedly loading cached data already in video memory and significantly reduces unnecessary data movement. On the one hand, it reduces the I / O load of the cache management module and the video memory bandwidth usage, allowing limited data movement resources to be concentrated on truly missing data. On the other hand, since the data loading task only contains the missing part, the task size is smaller, and the queuing latency of the asynchronous task pool and the processing time of the cache management module are shortened, thereby accelerating the transition speed of inference requests from "waiting to load" to "ready". In high-concurrency scenarios, multiple inference requests may share the same long prefix but each has some cache missing data. This mechanism allows each inference request to load only its own small amount of missing data, greatly reducing the overall I / O of the system and effectively alleviating resource competition between multiple TP threads. Ultimately, the first token latency (TTFT) and end-to-end latency are further reduced, the system throughput is improved, and the data consistency between video memory and the data backup unit is easier to maintain.

[0076] Step 104: The cache management module obtains the data loading task from the asynchronous task pool, retrieves the cached data from the data backup unit according to the data identifier and data storage location carried by the data loading task, writes the cached data into the video memory, generates a data loading completion notification corresponding to the data loading task, and writes the data loading completion notification into the loading confirmation queue.

[0077] In this context, video memory (VRAM) can be understood as the high-speed memory built into the GPU, used to store model parameters, intermediate activations, and key-value caches (KVCache). During model inference, only the KVCache located in VRAM can be directly used by the model.

[0078] A data loading completion notification can be understood as a confirmation message generated by the cache management module after completing a data loading task. This notification may include the task identifier of the data loading task, the data identifier of the loaded cached data, and the data loading status (success or failure). This notification is written to the loading confirmation queue for the scheduling module to read and update the request status.

[0079] The load confirmation queue can be understood as another queue between the cache management module and the scheduling module, used to send back notifications that data loading is complete. The scheduling module can obtain the data loading completion notification from this load confirmation queue through polling or events, thus knowing that the cached data for a certain inference request is ready, and thus proceeding with the execution of the inference request.

[0080] In practice, the cache management module runs continuously in the background, constantly retrieving data loading tasks from the asynchronous task pool. For each data loading task retrieved, the cache management module first parses the data identifier and data storage location carried within the task. If the data storage location points to memory, the cache management module directly reads the cached data from that memory address into a temporary buffer via a CPU memory copy operation; if the data storage location points to disk, the cache management module calls the underlying I / O interface to read the cached data from the specified location on the disk. In disk-based scenarios, if multi-disk parallel processing is involved, the cache management module first uses a multi-disk routing mechanism to split a large task into multiple subtasks, reads these subtasks in parallel from different disks, and then aggregates the subtasks.

[0081] After the cached data is read, the cache management module obtains the complete KV cache and then writes it to the GPU memory. The write operation can transfer the cached data from host memory to a pre-allocated GPU memory address via CUDA (a parallel computing platform and programming model) memory copy. This GPU memory address may be specified by the scheduling module in the data loading task, or it may be dynamically allocated by the cache management module based on the data identifier. After the cached data is written, the cache management module ensures that the cached data is visible to the model in GPU memory. Finally, the cache management module constructs a data loading completion notification, which may include the task identifier of the original data loading task, the data identifier, and the write success status (or an error code if it failed). This data loading completion notification is placed in the load confirmation queue. The scheduling module periodically polls or retrieves the data loading completion notifications in this queue through an event mechanism, thus knowing that the cached data for a certain inference request is ready and that the execution of the inference request can continue.

[0082] Throughout the process, the cache management module operates completely independently of the scheduling module and does not block the scheduling thread. Even if multiple data loading tasks arrive simultaneously, the cache management module can utilize its internal I / O thread pool to process them in parallel, further improving efficiency.

[0083] Specifically, the data backup unit includes a primary memory backup unit or a secondary disk backup unit; The step of retrieving the cached data from the data backup unit based on the data identifier and data storage location carried by the data loading task includes: If the cached data is determined to be stored in the memory-level backup unit based on the data storage location carried by the data loading task, the cached data is retrieved from the memory-level backup unit according to the data identifier. or If the cached data is determined to be stored in the secondary backup unit of the disk based on the data storage location carried by the data loading task, the cached data is obtained from the secondary backup unit of the disk through a multi-disk routing mechanism based on the data identifier and the data storage location.

[0084] The primary memory backup unit can be understood as a portion of the host memory (RAM, Random Access Memory) in the data processing system, used to store key-value caches evicted from or actively written back from video memory. This primary memory backup unit has a faster access speed than disk but slower than video memory, serving as the first-level backup storage for video memory. For example, when GPU video memory is insufficient, a portion of the key-value cache is copied to a fixed buffer in the primary memory backup unit, which can then be quickly read back into video memory when needed.

[0085] The disk-based secondary backup unit can be understood as a storage pool composed of multiple solid-state drives (SSDs), serving as a second-level backup storage for the KV cache. This disk-based secondary backup unit has a larger capacity than memory but higher access latency and is used to store infrequently used or rarely accessed cached data. For example, the historical KV cache of a long-running session might be migrated to disk to free up video memory and system memory.

[0086] Multi-disk routing can be understood as a scheduling method that distributes key-value cache data across multiple disks and allows for parallel retrieval or writing during reads or writes. This mechanism maintains a routing table that records the disk identifier and offset of each cached data block. When a segment of cached data needs to be read, the mechanism splits the cache data retrieval request into multiple sub-requests, sends them to different disks simultaneously, and finally merges the results (the cached data retrieved from each sub-request). For example, a 64KB cached data segment can be divided into four 16KB pieces, stored on four different disks. During reads, 16KB of cached data can be retrieved from each of the four disks in parallel, and then reassembled into the complete cached data.

[0087] In practical applications, the cached data is obtained from the secondary backup unit of the disk through a multi-disk routing mechanism. This can be understood as follows: the cache management module first determines the target disk in the secondary backup unit corresponding to the cached data through the multi-disk routing mechanism, and locates the slot or page address of the cached data in the pre-allocated large file (i.e., mmap memory-mapped pooled file) of the target disk (each slot or page address corresponds to a fixed-size contiguous storage space); then, using memory mapping (directly mapping the disk file to the virtual address space of the process, making reading and writing files as direct as accessing memory), the cached data is read from the slot or page address, and the cached data is obtained in units of pages or slots.

[0088] In practice, after the cache management module retrieves a data loading task from the asynchronous task pool, it first parses the data storage location field carried in the data loading task to determine which level of backup unit the cached data is located in.

[0089] If the data loading task indicates that the cached data is stored in a primary memory backup unit, the cache management module can directly use the host memory address (e.g., a pointer or offset) given in the data loading task, based on the data identifier, to read the cached data from that memory address into the cache management module's temporary buffer using standard CPU memory copy operations. The data identifier can be used for verification. After reading the cached data from the memory address into the temporary buffer, the cache management module can use a preset hash algorithm to calculate the cached data in the temporary buffer and compare the result with the data identifier (i.e., the standard hash value of the cached data) carried in the data loading task. If they match, the verification passes, and the cached data is written to video memory. If they do not match, the cached data is deemed corrupted, discarded, and a loading failure status is returned. The scheduling module then decides whether to retry or report an error.

[0090] If the data loading task indicates that the cached data is stored in the secondary backup unit of the disk, the cache management module will invoke the multi-disk routing mechanism and combine it with the mmap-based disk-layer pooled KV Cache storage mechanism to complete the cached data reading: First, routing and location: Based on the data storage location in the data loading task, the routing module in the cache management module queries the internal routing table (pre-defined sharding routing table) to determine the target disk corresponding to the cached data, as well as the specific slot or page address in the mmap pooled file pre-allocated on the target disk (each slot / page corresponds to a fixed-size contiguous storage space); Second, memory mapping access: Since the disk layer uses a pre-allocated, contiguously laid-out large file (i.e., the mmap pooled file), and this file has been mapped to the process virtual address space of the cache management module through the mmap system call, the cache management module does not need to execute system calls such as open, read, lseek, and close, nor does it need to copy data between the kernel page cache and the user buffer. It can directly access the KV Cache data in the target slot or page using the mapped memory address, approximating memory access. Finally, multi-disk parallel processing and merging: if the routing table indicates that the cached data is distributed across multiple target disks (e.g., due to fragmented storage), the cache management module starts multiple I / O threads to read the corresponding sub-cache data from the mmap mapping area of ​​each target disk in parallel. After all sub-cache data is returned, it is merged into a complete KV Cache in its original order. The merged complete KV Cache is temporarily stored in the host memory buffer, or it can be verified based on the data identifier. Regardless of whether the cached data comes from memory or disk, after obtaining the complete KV Cache, the cache management module can write it to the GPU memory through operations such as CUDA memory copy. After writing, a data loading completion notification (including task identifier, data identifier, memory address, etc.) is constructed and pushed into the loading confirmation queue to notify the scheduling module that the cached data is ready. Through the above mmap pooling mechanism, the disk read path avoids frequent system calls and kernel-user space data copying, while improving I / O throughput by utilizing multi-disk parallel processing and contiguous space layout.

[0091] The data processing method provided in one or more embodiments of this specification achieves efficient data acquisition under tiered storage by distinguishing between primary memory backup and secondary disk backup, and introducing a multi-disk routing mechanism for disks. On the one hand, for cached data that hits memory, the low latency of memory is utilized for fast reading, avoiding unnecessary disk I / O; on the other hand, for cached data that hits disk, the task of acquiring large blocks of cached data is split into multiple subtasks and executed simultaneously through parallel reading of multiple disks, significantly shortening the disk I / O time and avoiding the bandwidth bottleneck caused by sequential reading of a single disk. This tiered awareness reading strategy enables the cache management module to select a better reading path based on the tier where the cached data is located, utilizing the high speed of memory and improving disk read throughput through multi-disk parallelism. In high-concurrency scenarios with multiple TPs, multiple inference requests may need to load different cache blocks from the disk simultaneously. The multi-disk routing mechanism can balance the load across multiple devices, reducing I / O queue waiting time. Ultimately, this effectively reduced the loading latency of cached data from the data backup unit to the video memory, thereby shortening the first token latency and end-to-end latency of requests, while improving the overall I / O parallelism and throughput of the system.

[0092] Specifically, the step of obtaining the cached data from the secondary backup unit of the disk through a multi-disk routing mechanism based on the data identifier and the data storage location includes: Based on the data identifier and the data storage location, a data acquisition instruction is generated and sent to the disk secondary backup unit. In response to the data acquisition instruction, the disk secondary backup unit splits the data acquisition instruction into multiple sub-instructions according to a preset sharding routing table, sends the multiple sub-instructions in parallel to the corresponding multiple disks, and receives multiple sub-cache data returned by the multiple disks after executing the multiple sub-instructions. The system receives the cached data returned by the disk secondary backup unit, wherein the cached data is determined by the disk secondary backup unit based on the plurality of sub-cache data.

[0093] The data acquisition instruction can be understood as a command sent by the cache management module to the disk secondary backup unit for acquiring cached data. This data acquisition instruction may include information such as the data identifier, logical address range, and data length of the cached data to be read.

[0094] The preset fragmentation routing table can be understood as a mapping table maintained internally by the secondary backup unit of the disk. It records how many blocks each segment of KV cache data is divided into, and which physical disk and offset each block is stored at. This preset fragmentation routing table determines how logical data retrieval instructions are split into multiple physical sub-instructions.

[0095] Sub-instructions can be understood as small commands obtained by splitting data retrieval instructions according to a preset sharding routing table. Each sub-instruction corresponds to a specific cached data read request on a physical disk, including disk identifier, starting offset, read length, etc.

[0096] Multiple disks can be understood as multiple hardware devices that form a secondary backup unit for disks. For example, multiple disks could be four independent SSDs (Solid State Drives). Multiple disks can work simultaneously and process different sub-instructions in parallel.

[0097] Sub-cache data can be understood as a fragment of cached data returned after each physical disk executes a sub-instruction. For example, the original 64KB cache data is divided into four 16KB segments, and each segment is a sub-cache data.

[0098] In practice, when the cache management module needs to read the KV Cache from the disk secondary backup unit, it first constructs a data retrieval instruction based on the data identifier and data storage location in the data loading task. This data retrieval instruction clearly specifies the data range of the cached data to be read (such as the starting logical offset and data length). Then, it sends the data retrieval instruction to the disk secondary backup unit.

[0099] Upon receiving the data retrieval command, the disk secondary backup unit queries its internally maintained preset fragmentation routing table. This table records the physical distribution corresponding to each logical address range: for example, a 64KB cached data segment might be divided into four 16KB fragments, stored at offset 0x1000 on disk A, offset 0x2000 on disk B, offset 0x3000 on disk C, and offset 0x4000 on disk D, respectively. Based on this preset fragmentation routing table, the disk secondary backup unit breaks down the original data retrieval command into multiple sub-commands, each sub-command corresponding to a read request on a physical disk (disk identifier, offset, data length, etc.).

[0100] After the partitioning is complete, the secondary disk backup unit sends these sub-instructions in parallel to the corresponding multiple disks. Each disk independently performs cached data read operations without waiting for each other. After the disk controller reads the cached data from the physical media, it returns the sub-cache data to the secondary disk backup unit.

[0101] The disk secondary backup unit waits for all sub-instructions to complete and collects the returned sub-cache data. Then, according to the sharding order recorded in the preset sharding routing table, it concatenates these sub-cache data into a complete cache data in the correct sharding order. For example, it combines the 16KB data returned from disks A, B, C, and D into 64KB of cache data. Finally, the disk secondary backup unit returns the reconstructed complete cache data to the cache management module.

[0102] The data processing method provided in one or more embodiments of this specification significantly improves the bandwidth utilization and throughput of loading KV Cache from disk through instruction splitting, parallel reading, and data reorganization mechanisms within the disk secondary backup unit. Multi-disk parallelism aggregates the bandwidth of multiple disks, reducing loading time by approximately proportionally. A pre-defined sharding routing table makes data distribution transparent; the cache management module does not need to concern itself with physical details, only sending logical data retrieval commands, reducing upper-layer complexity. In high-concurrency multi-TP scenarios, multiple inference requests may trigger disk reads simultaneously. Multi-disk parallelism can distribute the load across different devices, preventing a single disk from becoming a hotspot and reducing I / O queuing latency. This mechanism also supports dynamically expanding the number of disks; adding a disk and adjusting the pre-defined sharding routing table achieves linear expansion of capacity and bandwidth, significantly shortening the loading time from disk to video memory, thereby reducing first-to-first token latency and end-to-end latency, and improving the overall inference throughput of the system. It is particularly suitable for scenarios requiring frequent loading of long context caches from disk.

[0103] Step 106: The scheduling module obtains the data loading completion notification from the loading confirmation queue, determines the cached data in the video memory based on the data loading completion notification, and executes the inference request to generate inference results based on the cached data.

[0104] The inference result can be understood as the output content generated by the large language model through forward computation based on the input lexical sequence and the loaded cached data. The inference result can be a piece of text or a token sequence.

[0105] In practice, the scheduling module can start a background loop during initialization to continuously monitor the load confirmation queue. When the cache management module completes a data loading task, it pushes a data loading completion notification into the load confirmation queue. The scheduling module retrieves this notification from the queue via polling or event-driven methods. After retrieving the notification, the scheduling module first parses the task and data identifiers carried in the notification to locate the inference request previously in the "waiting to load" state. Then, based on the memory address (i.e., the location of the loaded cached data in memory) and data length provided in the notification, the scheduling module determines that the cached data required for the inference request is fully ready. Next, the scheduling module switches the inference request's state from "waiting to load" to "ready" and places it in the executable queue. In the next scheduling loop, the scheduling module retrieves the inference request from the executable queue, binds it to the forward computation engine of the large language model, passes in the cached data pointer in memory, and triggers the GPU to perform attention calculations and subsequent token generation. The large language model performs forward inference based on the input lexical sequence and the loaded KV cache, generating output lexical units and gradually constructing a complete inference result (such as the generated text). Finally, the scheduling module returns the inference result to the user.

[0106] In practical applications, while the scheduling module generates data loading tasks for cached data missing from video memory, for the cached data already in video memory, the scheduling module directly retrieves the video memory address of this cached data and uses it for inference once all cached data is complete. The specific implementation method is as follows: After performing a cached query on the lexical sequence corresponding to the inference request, the method further includes: The scheduling module, based on the data storage location, determines the video memory address of the other part of the cached data in the video memory, whereby the other part of the cached data and the other part of the cached data constitute the cached data.

[0107] The other part of the cached data can be understood as the remaining cached data already located in video memory, excluding the part of the cached data that needs to be loaded from the data backup unit, within the complete cached data corresponding to the longest shared prefix.

[0108] A memory address can be understood as the starting position of a key-value cache data segment in the GPU's memory, typically represented by a pointer or handle. The scheduling module can locate cached data using its memory address without actually copying the cached data content.

[0109] In practice, after completing the longest prefix matching and obtaining the target node, the scheduling module iterates through the cached data of that target node. For cached data, the scheduling module checks the storage hierarchy information in its metadata. If it finds that the data storage location of a certain part of the cached data points to GPU memory, the scheduling module reads the GPU memory address of that part of the cached data from the metadata. This process does not involve any actual copying of the cached data; it only obtains the address information of the cached data. The scheduling module refers to this cached data already existing in GPU memory as "another part of cached data" and records its GPU memory address and corresponding lexical range. Simultaneously, for cached data not in GPU memory (i.e., "partial cached data"), the scheduling module generates a data loading task and submits it to the cache management module. When the cache management module completes data loading, the scheduling module receives a data loading completion notification, thus obtaining the GPU memory address of the newly loaded cached data. At this point, the scheduling module merges the GPU memory addresses of "another part of cached data" and "partial cached data" in lexical order to form a complete cached data address list. Subsequently, the scheduling module can pass this address list to the model inference engine to perform forward computation.

[0110] The data processing method provided in one or more embodiments of this specification achieves zero-overhead reuse of cached data already in video memory by directly determining the video memory address instead of copying cached data, avoiding unnecessary data movement and repeated queries. The scheduling module records the video memory address in advance, and can immediately construct a complete address list and deliver it to the inference engine after the missing cached data is loaded, without having to traverse the index again, thus reducing scheduling latency. In high-concurrency scenarios, multiple inference requests sharing a common block address can reduce memory usage and metadata maintenance costs. Ultimately, this mechanism shortens the transition time from cached data readiness to inference startup, reduces the first token latency, and improves the overall system throughput efficiency.

[0111] In practical applications, when some cached data corresponding to the longest shared prefix already exists in video memory while another portion needs to be loaded from the data backup unit, the scheduling module will wait for the cache management module to load the missing cached data. Only after all cached data is ready in video memory will the forward computation of the inference request be executed. The specific implementation method is as follows: The step of determining the cached data in the video memory based on the data loading completion notification includes: Based on the data loading completion notification, determine the video memory address of the portion of cached data in the video memory, and based on the video memory address of the portion of cached data and the video memory address of the other portion of cached data, determine the cached data in the video memory.

[0112] In practice, after receiving the data loading completion notification from the loading confirmation queue, the scheduling module first parses the content of the notification to extract the GPU memory address of the newly loaded "partial cached data." Simultaneously, the scheduling module has already recorded the GPU memory address of the "other portion of cached data." Following the lexical order, the scheduling module concatenates these GPU memory addresses into an ordered address list, thus comprehensively describing the distribution of the entire cached data in GPU memory. For example, the address list might be [addr_block0, addr_block1, addr_block2, addr_block3], where the first two are the GPU memory addresses of the other portion of cached data, and the last two are the GPU memory addresses of the newly loaded partial cached data. After constructing the address list, the scheduling module considers the complete KV Cache required for the inference request to be ready in GPU memory. Subsequently, the scheduling module passes this address list to the model inference engine, which directly reads the KV Cache based on the address list to perform attention calculations on the inference request and generate subsequent tokens.

[0113] The data processing method provided in one or more embodiments of this specification achieves seamless merging of some cached data with other cached data by concatenating address lists instead of copying data. The scheduling module does not need to move any cached data in the video memory; it only needs to record the addresses to complete the construction of the complete cached data, avoiding wasted video memory bandwidth and additional latency. Simultaneously, the determinism of the address list allows the model inference engine to access each cached data with zero overhead. Under high concurrency, different inference requests can reuse the video memory addresses of common blocks, reducing metadata redundancy. This mechanism significantly shortens the transition time from cache readiness to inference startup, reduces the first token latency, and improves the overall system throughput.

[0114] Specifically, the step of generating an inference result by executing the inference request based on the cached data includes: The inference request is executed based on the cached data, and output words in the output sequence are generated one by one. The currently generated output words are used as the input for the next execution of the inference request. The inference steps are executed in a loop until the inference termination condition is met, and the inference result of the inference request is obtained. The inference result is composed of output words in the output sequence.

[0115] The output sequence can be understood as a sequence of output words generated step by step during the autoregressive generation process of the model. It is initially empty, and a new word is added at each step to form a complete answer text.

[0116] Output lexical units can be understood as the smallest semantic unit generated by a single forward computation of the model, such as a Chinese character, a word, or a punctuation mark. Multiple output lexical units are arranged in sequence to form the output sequence.

[0117] The input for the next execution of the inference request can be understood as follows: in autoregressive generation, the output tokens generated in the current inference step are concatenated with historical input tokens and used as input for the next model calculation. The model continues to generate the next output token based on the existing KVCache and the new input tokens.

[0118] The inference termination condition can be understood as the condition for terminating the autoregressive loop, including but not limited to: generating a special termination symbol, the output sequence length reaching a preset length threshold, or triggering other stopping rules (such as time limit exceeding the limit).

[0119] In practice, the scheduling module initiates an autoregressive generation loop after confirming that the complete KV cache required for the inference request is fully located in GPU memory. First, the model, with the input lexical sequence (the sequence of lexical terms in the inference request) and cached data as its initial state, performs a forward computation to generate the first output lexical. The scheduling module records this lexical in the output sequence and uses it as input for the next round of inference. Then, based on the existing KV cache (without recalculating historical lexical terms) and the new input lexical, the model performs another forward computation to generate the next output lexical. Each time a lexical is generated, the scheduling module appends it to the output sequence and checks if the inference termination condition is met. If not, the loop continues: using the newly generated output lexical as input and inferring again using existing cached data; if the inference terminates, the loop stops. Finally, the scheduling module concatenates all the lexical terms in the output sequence into a complete text string and returns it to the user as the inference result. Throughout this process, since the KV cache is already ready, each computation only needs to process a single newly added lexical, resulting in high generation efficiency.

[0120] The data processing method provided in one or more embodiments of this specification achieves efficient word-by-word generation of output sequences by reusing the loaded complete KV cache through an autoregressive loop. Each inference step only requires calculating the attention increment corresponding to the new lexical unit; the KV cache of historical lexical units is directly reused, avoiding redundant calculations and significantly reducing the computational load per step. The loop structure allows the model to flexibly generate outputs of variable length until the termination condition is met, adapting to various inference tasks such as question answering and summarizing. Simultaneously, since the cached data is already ready in GPU memory, each inference step requires no additional I / O waiting, ensuring low latency in the generation process. In high-concurrency scenarios, the autoregressive loops for different inference requests are independent of each other and can be fully parallelized, improving overall throughput, ensuring the quality and response speed of inference results, and providing users with a smooth interactive experience.

[0121] Specifically, after executing the inference request based on the cached data and generating the output tokens in the output sequence one by one, the process further includes: The scheduling module retrieves the inference cache data corresponding to the output lexical from the video memory, generates a data write-back task based on the inference cache data, and submits the data write-back task to the asynchronous task pool. The data write-back task carries the newly added data identifier of the inference cache data. The cache management module obtains the data write-back task from the asynchronous task pool, writes the inference cache data into the data backup unit according to the new data identifier and the preset write strategy, generates a data write-back completion notification corresponding to the data write-back task, and writes the data write-back completion notification into the write-back confirmation queue. The scheduling module obtains the data write-back completion notification from the write-back confirmation queue, and updates the cache radix tree based on the newly added data identifier of the inference cache data and the newly added data storage location of the inference cache data carried in the data write-back completion notification.

[0122] The inference cache data can be understood as a new key-value cache generated when the model generates a certain output term. This inference cache data is currently located in video memory and needs to be asynchronously written back to memory or disk so that it can be reused in subsequent requests.

[0123] A data write-back task can be understood as a write-back instruction constructed by the scheduling module for inference cache data, used to notify the cache management module to move newly added KV Cache data from the video memory to the data backup unit. This data write-back task may include a new data identifier for the newly added cached data, so that the cache management module can identify and store it.

[0124] A new data identifier can be understood as a unique ID that identifies a newly generated KV cache segment. For example, a new data identifier could be the hash value of the corresponding output lexical sequence or a cache block number. It is used to locate the correct inference cache data when writing back and updating metadata.

[0125] A preset write strategy can be understood as a set of write rules within the cache management module, determining which backup level (such as memory or disk) the KV cache should be written to, whether to use direct write or write-back, and whether to compress it. In practical applications, the preset write strategy can be dynamically adjusted based on factors such as GPU memory pressure and data usage frequency.

[0126] The write-back confirmation queue can be understood as the queue where the cache management module sends write-back completion notifications to the scheduling module. After the cache management module completes the inference cache data transfer, it pushes the write-back completion notification into this write-back confirmation queue. The scheduling module then reads this write-back completion notification from the queue to update the cache index.

[0127] The newly added data storage location can be understood as the specific address recorded by the cache management module after writing the new KV Cache (inference cache data) into memory or disk. For memory, the newly added data storage location can be a host memory pointer; for disk, the newly added data storage location can be a device path and offset.

[0128] In practice, during each inference step of the autoregressive generation process, when the model generates a new output term, the corresponding inference cache data is stored in GPU memory. The scheduling module retrieves this newly generated inference cache data (actually, it only retrieves its GPU memory address and size), and then constructs a data write-back task. This data write-back task carries the newly added data identifier of the inference cache data (e.g., the hash value of the output term sequence). The scheduling module submits this data write-back task to the asynchronous task pool non-blockingly, and then immediately returns to continue processing the next inference step without waiting for the write-back to complete.

[0129] The cache management module continuously retrieves data write-back tasks from the asynchronous task pool. Based on the new data identifier in the data write-back task and the preset write strategy (e.g., selective direct write: writing the cache with important prefixes to disk; or write-back: writing to memory first, and then writing to disk in batches when memory pressure is high), the cache management module determines which backup level to write the inference cache data to. Then, it copies the KV Cache from the video memory to the host memory (memory-level backup unit), and decides whether to further write the inference cache data to disk (disk-level backup unit) according to the preset write strategy. After the inference cache data is written, the cache management module records the new data storage location (memory address or disk path + offset) of the inference cache data in the data backup unit, and constructs a data write-back completion notification, which may include the new data identifier and the new data storage location. Then, the data write-back completion notification is pushed into the write-back confirmation queue.

[0130] In each round of scheduling, the scheduling module polls the write-back confirmation queue and retrieves the data write-back completion notification. Based on the new data identifier and new data storage location in the write-back completion notification, the scheduling module locates the corresponding node in the cache radix tree (i.e., the prefix node to which the output token belongs) and then updates the node's metadata: adding the new backup level and new data storage location to the node (e.g., if it was originally IN_GPU, now it's IN_HOST or IN_DISK). This way, subsequent inference requests querying the same prefix will know that the new cached data is in memory or on disk, allowing it to be loaded on demand.

[0131] The data processing method provided in one or more embodiments of this specification completely decouples the storage operation of adding KVCache from inference computation through an asynchronous write-back mechanism, avoiding the blocking of the generation process by synchronous disk writes. After the scheduling module submits the data write-back task, it immediately continues to generate the next lexical unit, while the cache management module independently performs data transfer in the background, achieving overlap between computation and I / O. A preset write strategy allows the system to flexibly select the backup level (memory or disk) based on data importance and resource status, balancing performance and capacity. After the data write-back is completed, the cache cardinality tree is updated, ensuring real-time consistency of metadata and enabling subsequent inference requests to be aware of the newly added cached data. In high-concurrency, long-context scenarios, this mechanism effectively prevents memory overflow and, through multi-level backups, retains hot data, reducing redundant computations.

[0132] Specifically, the data backup unit includes a primary memory backup unit and a secondary disk backup unit; The step of writing the inference cache data into the data backup unit according to the newly added data identifier and the preset write strategy includes: The cache management module retrieves the inference cache data from the video memory according to the newly added data identifier, and writes the inference cache data into the memory-level backup unit. If the preset write policy determines that the utilization rate of the memory primary backup unit is greater than or equal to the preset utilization rate threshold, the inference cache data is written from the memory primary backup unit to the disk secondary backup unit through a multi-disk routing mechanism.

[0133] In practical applications, the inference cache data is written from the memory-level backup unit to the disk-level backup unit through a multi-disk routing mechanism. The cache management module determines the target disk (a physical NVMe device dynamically selected from the disk pool according to the multi-disk routing mechanism for this write operation) and the corresponding slot or page address of the target disk in the disk-level backup unit through the multi-disk routing mechanism. The inference cache data is then written directly from the memory-level backup unit to the pre-allocated, contiguous KV pool of the disk-level backup unit through mmap mapping. The allocation and migration of data are completed using slots or pages as management units.

[0134] In practice, after the cache management module obtains the data write-back task from the asynchronous task pool, it first reads the corresponding inference cache data (i.e., the newly generated KV Cache) from the video memory based on the newly added data identifier in the data write-back task. This inference cache data is then written to a designated area in the primary memory backup unit via a CPU memory copy operation, and the memory address and data length are recorded.

[0135] Next, the cache management module checks the preset write policy. If the current write policy specifies: monitor the current utilization rate of the memory-level backup unit; if the utilization rate is less than the preset utilization rate threshold (e.g., 80%), the write-back process ends; if the utilization rate reaches or exceeds the preset utilization rate threshold, a data sinking operation is triggered. At this time, the cache management module calls the multi-disk routing mechanism, determines the target disk according to the preset sharding routing table (such as consistent hashing or range sharding), and allocates a free slot or page address in the mmap pooling file of the target disk (each slot / page corresponds to a fixed-size contiguous storage space), obtaining the starting address and length of the slot / page address in the mapped memory. Subsequently, the cache management module uses the direct mapping path between memory and the disk pool (e.g., mapping the disk pool file to the process address space via mmap) to directly copy the inference cache data in the memory-level backup unit to the mapped memory area corresponding to the slot / page address, thereby completing the writing of the inference cache data to the disk-level backup unit KV pool. After the write operation is complete, the metadata in the preset shard routing table is updated (recording the disk where the inference cache data segment is located, the offset, and the data length). Optionally, the data copy in the memory-level backup unit can be released to reduce memory usage. The entire write process avoids the overhead of traditional file system open / read / write / close system calls, reduces user-mode and kernel-mode switching and data copying, and improves sinking efficiency by utilizing multi-disk parallel write capabilities.

[0136] The data processing method provided in one or more embodiments of this specification achieves a balance between performance and capacity through a two-level write strategy: "write to memory first, then migrate to disk when memory pressure is high." The memory-level backup unit has low latency characteristics, and frequently reused hot data can be quickly loaded from memory back to video memory, avoiding disk I / O overhead. When memory utilization exceeds a preset utilization threshold, a multi-disk routing mechanism writes cold data (the part of the inference cache data that is accessed less frequently and is preferentially migrated to free up memory space) to disk in parallel, which not only frees up memory space but also shortens the write time by utilizing the aggregated bandwidth of multiple disks. Compared with writing directly to disk, this strategy significantly reduces the latency of write operations; compared with writing only to memory, this mechanism prevents memory overflow by automatically migrating data, expanding the effective cache capacity. In high-concurrency scenarios with multiple process terminals (TPs), this mechanism enables the system to adaptively manage storage resources and maintain stable inference performance.

[0137] The data processing method provided in one or more embodiments of this specification can be understood as a multi-level asynchronous storage and scheduling scheme for KV Cache based on RadixTree metadata indexing and mmap disk pooling storage, achieving significant beneficial effects in long context and high-concurrency inference scenarios. Firstly, regarding performance improvement: this data processing method consistently reduces latency and improves throughput under different concurrency conditions. Under single concurrency, the first token latency (TTFT) is reduced by an average of 31.05%, the per-token output latency (TPOT) is reduced by an average of 34.05%, the end-to-end latency is reduced by an average of 33.89%, and the overall throughput is improved by an average of 50.72%. Under 2-4 concurrency, TTFT is reduced by an average of 49.98%, TPOT by an average of 10.96%, the end-to-end latency is reduced by an average of 12.28%, and the overall throughput is improved by an average of 12.35%. Under 8 concurrency, TTFT is reduced by an average of 60.51%, TPOT by an average of 12.36%, the end-to-end latency is reduced by an average of 14.19%, and the overall throughput is improved by an average of 13.05%. The above results demonstrate that this data processing method can stably reduce the first token and overall latency, and improve system throughput under high load and large request differences, thus possessing high engineering application value. Secondly, regarding metadata management: this data processing method is based on RadixTree's prefix semantic metadata management, using RadixTree as the main index for KV Cache metadata. It natively supports longest prefix matching and prefix sharing reuse, and can maintain cache location and status at the path dimension, achieving integrated "path-level governance" of hit determination, reuse, and eviction. This is superior to hash-based lookup-style metadata organization, significantly improving prefix hit efficiency and scheduling consistency. Thirdly, regarding disk storage, this data processing method can adopt a disk-layer pooled KV Cache storage mechanism based on mmap, upgrading the traditional "one file per key" object storage mode to a file-mapped KV pool based on pre-allocated large files and contiguous space layout. It uses slots or pages as the basic management unit to complete data allocation, reading, writing, migration, and recycling, thereby significantly reducing file system metadata maintenance costs and addressing issues such as inode (index node) occupation, directory bloat, path lookup, and frequent creation and deletion caused by massive small files.Compared to traditional file read / write methods, the mmap mechanism has significant advantages in the I / O path: Traditional file read / write typically relies on system calls such as open, read, write, lseek (file pointer positioning), and close. Each access requires switching between user mode and kernel mode, and data often needs to first enter the kernel page cache and then be copied from the kernel to the user buffer. When writing, it needs to be copied from the user buffer to the kernel buffer before being flushed to disk. This approach leads to high system call overhead and additional data copying costs in high-frequency, small-block, random access scenarios like KV Cache. In contrast, mmap directly maps file content to the process's virtual address space, allowing applications to read and write file data directly in a manner similar to memory access. After initial mapping, a large number of fine-grained accesses no longer rely on frequent read / write system calls, thereby effectively reducing the number of user mode and kernel mode switches and reducing the overhead of repeated data copying between the kernel buffer and the user buffer. Meanwhile, because the pooled KV Cache adopts a pre-allocated, contiguous layout for organizing large files, multiple KV data blocks can be arranged more compactly in physical space. The system can directly locate the target data based on slot offset, eliminating the lengthy path of searching for files by key, opening files, parsing metadata, and then performing read / write operations. This makes batch moving, sequential prefetching, page-level write-back, and parallel access more efficient, and also makes it easier to leverage the operating system's page caching, sequential I / O optimization, and virtual memory management mechanisms to improve overall throughput. In summary, the mmap-based pooled disk KV Cache not only optimizes the data path for single read / write operations, but also transforms the disk-layer KV Cache from an object-oriented storage model to a more efficient cache extension layer that is closer to the memory page management model by reducing metadata operations, mitigating the risk of small file fragmentation, improving spatial locality, and batch processing capabilities. Finally, regarding layered collaboration and expansion: This data processing method constructs a three-level collaborative caching system of video memory, main memory, and disk, with main memory serving as a backup for video memory and disk serving as a backup for main memory; the disk layer introduces a multi-disk routing and load balancing mechanism, enabling capacity expansion to be transformed into parallel I / O expansion, achieving near-linear storage and throughput expansion capabilities under the premise of stable performance, and being compatible with the unified implementation of different attention patterns (such as MHA / MLA), effectively solving the problems of unstable disk expansion, hot disks, and insufficient parallelism in existing solutions.

[0138] Corresponding to the above method embodiments, this specification also provides data processing system embodiments. Figure 2 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification. Figure 2 As shown, the system includes a scheduling module and a cache management module, wherein: The scheduling module 202 is used to receive inference requests, perform cache queries on the word sequence corresponding to the inference request, and generate a data loading task when it is determined that the cache data corresponding to the word sequence exists in the data backup unit, and submit the data loading task to the asynchronous task pool. The cache management module 204 is used to obtain the data loading task from the asynchronous task pool, obtain the cached data from the data backup unit according to the data identifier and data storage location carried by the data loading task, write the cached data into the video memory, generate a data loading completion notification corresponding to the data loading task, and write the data loading completion notification into the loading confirmation queue. The scheduling module 202 is further configured to obtain the data loading completion notification from the loading confirmation queue, determine the cached data in the video memory based on the data loading completion notification, and execute the inference request to generate inference results based on the cached data.

[0139] Optionally, the scheduling module 202 is further configured to perform longest prefix matching on the lexical sequence corresponding to the inference request according to the cache radix tree, and obtain the target node in the cache radix tree that shares the longest prefix with the lexical sequence, wherein the target node stores metadata of the cache data corresponding to the shared longest prefix, and the metadata includes the data identifier and the data storage location; If, based on the data storage location, it is determined that the cached data corresponding to the longest shared prefix exists in the data backup unit, the data loading task is generated based on the data identifier and the data storage location.

[0140] Optionally, the scheduling module 202 is further configured to take the root node of the cache radix tree as the current node and the first word of the word sequence as the current word; Determine whether the current node has child nodes, and whether the word element corresponding to the child node matches the current word element. If so, the child node is taken as the current node, the child node is recorded, and the next word of the current word is taken as the current word. The steps of determining whether the current node has a child node and whether the word corresponding to the child node matches the current word are continued until the current node has no child node or the word sequence has been completely matched, and the loop stops. If not, then stop the loop; The child node that was last successfully matched is taken as the target node in the cache radix tree that shares the longest prefix with the lexical sequence.

[0141] Optionally, the scheduling module 202 is further configured to, when determining that some cached data in the cached data corresponding to the longest shared prefix exists in the data backup unit based on the data storage location, generate a data loading task for the partial cached data based on the data identifier and data storage location of the partial cached data.

[0142] Optionally, the scheduling module 202 is further configured to determine the video memory address of the other part of the cached data in the video memory when it is determined that the other part of the cached data corresponding to the longest shared prefix exists in the video memory based on the data storage location, wherein the part of the cached data and the other part of the cached data constitute the cached data.

[0143] Optionally, the scheduling module 202 is further configured to determine the video memory address of the portion of cached data in the video memory according to the data loading completion notification, and determine the cached data in the video memory according to the video memory address of the portion of cached data and the video memory address of the other portion of cached data.

[0144] Optionally, the data backup unit includes a memory-based primary backup unit or a disk-based secondary backup unit; The cache management module 204 is further configured to, when determining that the cache data is stored in the memory-level backup unit based on the data storage location carried by the data loading task, retrieve the cache data from the memory-level backup unit according to the data identifier; or If the cached data is determined to be stored in the secondary backup unit of the disk based on the data storage location carried by the data loading task, the cached data is obtained from the secondary backup unit of the disk through a multi-disk routing mechanism based on the data identifier and the data storage location.

[0145] Optionally, the cache management module 204 is further configured to generate and send a data acquisition instruction to the disk secondary backup unit according to the data identifier and the data storage location, wherein the disk secondary backup unit responds to the data acquisition instruction by splitting the data acquisition instruction into multiple sub-instructions according to a preset sharding routing table, and sends the multiple sub-instructions to the corresponding multiple disks in parallel, and receives multiple sub-cache data returned by the multiple disks after executing the multiple sub-instructions; The system receives the cached data returned by the disk secondary backup unit, wherein the cached data is determined by the disk secondary backup unit based on the plurality of sub-cache data.

[0146] Optionally, the scheduling module 202 is further configured to execute the inference request according to the cached data, generate output words in the output sequence one by one, and use the currently generated output words as the input for the next execution of the inference request, and execute the inference steps in a loop until the inference termination condition is met, and obtain the inference result of the inference request, wherein the inference result is composed of the output words in the output sequence.

[0147] Optionally, the scheduling module 202 is further configured to obtain the inference cache data corresponding to the output lexical from the video memory, generate a data write-back task based on the inference cache data, and submit the data write-back task to the asynchronous task pool, wherein the data write-back task carries the newly added data identifier of the inference cache data. The cache management module obtains the data write-back task from the asynchronous task pool, writes the inference cache data into the data backup unit according to the new data identifier and the preset write strategy, generates a data write-back completion notification corresponding to the data write-back task, and writes the data write-back completion notification into the write-back confirmation queue. The scheduling module obtains the data write-back completion notification from the write-back confirmation queue, and updates the cache radix tree based on the newly added data identifier of the inference cache data and the newly added data storage location of the inference cache data carried in the data write-back completion notification.

[0148] Optionally, the data backup unit includes a primary memory backup unit and a secondary disk backup unit; The cache management module 204 is further configured to obtain the inference cache data from the video memory according to the newly added data identifier, and write the inference cache data into the memory-level backup unit; If the preset write policy determines that the utilization rate of the memory primary backup unit is greater than or equal to the preset utilization rate threshold, the inference cache data is written from the memory primary backup unit to the disk secondary backup unit through a multi-disk routing mechanism.

[0149] The above is an illustrative scheme of a data processing system according to this embodiment. It should be noted that the technical solution of this data processing system and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing system, please refer to the description of the technical solution of the data processing method described above.

[0150] See Figure 3 , Figure 3 This is a system architecture diagram of a data processing system provided in one embodiment of this specification.

[0151] Figure 3The system architecture diagram of the data processing system uses the Splang P2P-TP_1 process and the Splang P2P-TP_N process as the core deployment units. Each TP process contains two core components: a scheduling thread (scheduling module) and a cache manager (cache management module). At the same time, it relies on the hierarchical storage system and multi-disk routing module to realize cross-level and cross-process cache data scheduling.

[0152] First: The two core components within the TP process are as follows: The scheduling thread is responsible for advancing the execution of inference requests and initiating key-value cache related operation instructions. It does not directly perform data movement. Internally, it integrates scheduling, write confirmation queue, write key-value cache, load confirmation queue, load key-value cache, and forward inference. It forms an asynchronous interactive closed loop with the write thread and hierarchical loading thread of the cache manager through the write confirmation queue, load confirmation queue, and the cache manager's write thread and hierarchical loading thread.

[0153] Cache Manager: Responsible for cross-layer migration of KV Cache and execution of write strategies. Internally configured with write task pool and load task pool, it executes cache storage in the write task pool through write threads and performs cache movement in the load task pool through layered load threads. Based on write strategies such as selective direct write and write-back, it realizes the layered sinking and management of cache data, and also connects to the read, write and scheduling of KV cache.

[0154] Second: The three-tiered storage system is as follows: The system constructs a layered storage system consisting of a main GPU memory cache, a first-level memory backup, and a second-level disk backup. Each layer of cache and storage unit collaborates to carry KV cache data. The main GPU memory cache layer serves as the core layer for inference computation, directly supporting real-time inference access. It includes device-side KV cache and host-side KV cache corresponding to two types of attention structures. The device-side KV cache is divided into multi-head attention device-side KV cache and low-rank attention device-side KV cache, stored in GPU memory, and is the main cache directly reused during inference. The host-side KV cache is divided into multi-head attention host-side KV cache and low-rank attention host-side KV cache, serving as a nearest-neighbor cache on the GPU memory side for temporary buffering and fast scheduling, and is bidirectionally synchronized with the device-side cache. This layer is uniformly managed by the device-side KV pool manager, which achieves dynamic allocation of GPU memory space, cache page lifecycle maintenance, and eviction management through the device-side KV pool and GPU redundant storage.

[0155] The primary memory backup layer serves as a high-speed backup layer for video memory, providing cache copies that can be quickly refilled. It supports memory-level caches with various attention structures: multi-disk KV cache: a general cache pool in the memory layer used to uniformly manage KV cache fragments awaiting sinking / refilling; multi-head attention multi-disk KV cache: stores memory-level backups of multi-head attention (MHA) structures; low-rank attention multi-disk read-only KV cache: stores read-only backups of low-rank attention (MLA) structures, reducing write conflicts and improving access efficiency. This layer is uniformly managed by the host-side KV pool manager, which uses the host-side KV pool and host redundant storage to achieve memory space scheduling, hierarchical migration of cached data, and rapid refilling.

[0156] As a large-capacity cold data backup layer, the disk secondary backup layer overcomes capacity bottlenecks through multi-disk pooling and routing mechanisms. It abstracts KV Cache storage into an allocatable disk pool, which is uniformly scheduled by the multi-disk KV pool manager. Storage resources are integrated through the multi-disk KV pool and connected to the underlying NVMe storage devices 1 to N via the multi-disk routing module. Different NVMe devices are accessed in parallel during writing and loading, avoiding single-disk hotspots and achieving simultaneous improvement in capacity and throughput expansion. At the same time, it is compatible with different attention-based KV Cache organization formats such as MHA and MLA, ensuring unified management of the layered system.

[0157] Third: Metadata management and multi-process collaboration are described below: The system uses a Radix Tree (metadata management) as the unified metadata structure for the KV Cache, which is stored in the cache manager module of each TP process. This structure represents the prefix sharing relationship of the token sequence and maintains information such as cache page mapping, hierarchical location (GPU memory / RAM / disk), and validity status. Meanwhile, the KV pools of each TP process (device-side KV pool, host-side KV pool, and multi-disk KV pool) are respectively connected to GPU redundant storage and host redundant storage. The Radix Tree structure is maintained through operations such as inserting / removing nodes / splitting / merging nodes. The prefix matching module implements longest prefix matching based on the Radix Tree, providing semantic support for the location and loading of cached data.

[0158] The data processing system provided in one or more embodiments of this specification achieves comprehensive benefits of "high hit rate, high parallelism, and scalability" in long context and high-concurrency inference through an integrated design of "Radix Tree prefix semantic metadata + asynchronous task closed-loop scheduling + hierarchical multi-disk pooled storage". Specific technical effects include: improving the probability of deep prefix hits and reducing redundant calculations through longest prefix matching and path-level reuse of Radix Trees, significantly reducing TTFT and end-to-end latency; reducing the blocking of the main inference process by cross-layer migration and improving computational overlap through asynchronous task closed-loop and request-level fine-grained synchronization, stabilizing and improving TPOT and overall throughput; and reducing file system metadata overhead and improving parallel I / O capabilities through disk-layer pooled storage and multi-disk routing load balancing, enabling better linear scalability and tail latency stability for disk capacity expansion.

[0159] With a large language model 'a' deployed on a PPU (Parallel Processing Unit) machine (TP=8), and input lengths of 256 / 512 / 1024 / 2048 / 3072, and a fixed output length of 1024, the following results were achieved under 1 / 2 / 4 / 8 concurrency: Under single-concurrency, TTFT decreased by an average of 31.05%, TPOT decreased by an average of 34.05%, end-to-end latency decreased by an average of 33.89%, and total throughput increased by an average of 50.72%; under 2-4 concurrency, TTFT decreased by an average of 49.98%, TPOT decreased by an average of 10.96%, end-to-end latency decreased by an average of 12.28%, and total throughput increased by an average of 12.35%; under 8 concurrency, TTFT decreased by an average of 60.51%, TPOT decreased by an average of 12.36%, end-to-end latency decreased by an average of 14.19%, and total throughput increased by an average of 13.05%. The above results show that the data processing system can stably reduce the initial token and overall latency and improve system throughput under high load and large request differences, and has high engineering application value.

[0160] See Figure 4 , Figure 4 This is a structural block diagram of a computing device provided in one embodiment of this specification.

[0161] The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0162] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0163] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0164] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 400 can also be a mobile or stationary server.

[0165] The processor 420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0166] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above data processing method belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above data processing method.

[0167] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0168] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0169] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0170] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above data processing method belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above data processing method.

[0171] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0172] The computer program / instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0173] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0174] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0175] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method applied to a data processing system, the data processing system comprising a scheduling module and a cache management module, the method comprising: The scheduling module receives an inference request, performs a cache query on the lexical sequence corresponding to the inference request, and generates a data loading task when it is determined that the cached data corresponding to the lexical sequence exists in the data backup unit. The data loading task is submitted to the asynchronous task pool in a non-blocking manner. The cache management module operates independently of the scheduling module. It obtains the data loading task from the asynchronous task pool, retrieves the cached data from the data backup unit according to the data identifier and data storage location carried by the data loading task, writes the cached data into the video memory, generates a data loading completion notification corresponding to the data loading task, and writes the data loading completion notification into the loading confirmation queue. The scheduling module obtains the data loading completion notification from the loading confirmation queue, determines the cached data in the video memory based on the data loading completion notification, and executes the inference request to generate inference results based on the cached data.

2. The data processing method according to claim 1, wherein performing a cache query on the lexical sequence corresponding to the inference request, and generating a data loading task when it is determined that the cached data corresponding to the lexical sequence exists in the data backup unit, includes: Based on the cache cardinality tree, the longest prefix matching is performed on the lexical sequence corresponding to the inference request to obtain the target node in the cache cardinality tree that shares the longest prefix with the lexical sequence. The target node stores metadata of the cached data corresponding to the shared longest prefix, and the metadata includes the data identifier and the data storage location. If, based on the data storage location, it is determined that the cached data corresponding to the longest shared prefix exists in the data backup unit, the data loading task is generated based on the data identifier and the data storage location.

3. The data processing method according to claim 2, wherein the step of performing longest prefix matching on the lexical sequence corresponding to the inference request based on the cache cardinality tree to obtain the target node in the cache cardinality tree that shares the longest prefix with the lexical sequence includes: The root node of the cache radix tree is taken as the current node, and the first word of the word sequence is taken as the current word. Determine whether the current node has child nodes, and whether the word element corresponding to the child node matches the current word element. If so, the child node is taken as the current node, the child node is recorded, and the next word of the current word is taken as the current word. The steps of determining whether the current node has a child node and whether the word corresponding to the child node matches the current word are continued until the current node has no child node or the word sequence has been completely matched, and the loop stops. If not, then stop the loop; The child node that was last successfully matched is taken as the target node in the cache radix tree that shares the longest prefix with the lexical sequence.

4. The data processing method according to claim 2, wherein, when it is determined that the cached data corresponding to the longest shared prefix exists in the data backup unit based on the data storage location, generating the data loading task based on the data identifier and the data storage location includes: If, based on the data storage location, it is determined that some cached data in the cached data corresponding to the longest shared prefix exists in the data backup unit, a data loading task for the partial cached data is generated based on the data identifier and data storage location of the partial cached data.

5. The data processing method according to claim 4, after performing a cache query on the lexical sequence corresponding to the inference request, further includes: The scheduling module, based on the data storage location, determines the video memory address of the other part of the cached data in the video memory, whereby the other part of the cached data and the other part of the cached data constitute the cached data.

6. The data processing method according to claim 5, wherein determining the cached data in the video memory based on the data loading completion notification includes: Based on the data loading completion notification, determine the video memory address of the portion of cached data in the video memory, and based on the video memory address of the portion of cached data and the video memory address of the other portion of cached data, determine the cached data in the video memory.

7. The data processing method according to any one of claims 1-6, wherein the data backup unit comprises a primary memory backup unit or a secondary disk backup unit; The step of retrieving the cached data from the data backup unit based on the data identifier and data storage location of the cached data carried by the data loading task includes: If the cached data is determined to be stored in the memory-level backup unit based on the data storage location carried by the data loading task, the cached data is retrieved from the memory-level backup unit according to the data identifier. or If the cached data is determined to be stored in the secondary backup unit of the disk based on the data storage location carried by the data loading task, the cached data is obtained from the secondary backup unit of the disk through a multi-disk routing mechanism based on the data identifier and the data storage location.

8. The data processing method according to claim 7, wherein obtaining the cached data from the secondary backup unit of the disk through a multi-disk routing mechanism based on the data identifier and the data storage location includes: Based on the data identifier and the data storage location, a data acquisition instruction is generated and sent to the disk secondary backup unit. In response to the data acquisition instruction, the disk secondary backup unit splits the data acquisition instruction into multiple sub-instructions according to a preset sharding routing table, sends the multiple sub-instructions in parallel to the corresponding multiple disks, and receives multiple sub-cache data returned by the multiple disks after executing the multiple sub-instructions. The system receives the cached data returned by the disk secondary backup unit, wherein the cached data is determined by the disk secondary backup unit based on the plurality of sub-cache data.

9. The data processing method according to any one of claims 1-6, wherein the step of generating an inference result by executing the inference request based on the cached data comprises: The inference request is executed based on the cached data, and output words in the output sequence are generated one by one. The currently generated output words are used as the input for the next execution of the inference request. The inference steps are executed in a loop until the inference termination condition is met, and the inference result of the inference request is obtained. The inference result is composed of output words in the output sequence.

10. The data processing method according to claim 9, further comprising, after executing the inference request based on the cached data and generating output tokens in the output sequence one by one: The scheduling module retrieves the inference cache data corresponding to the output lexical from the video memory, generates a data write-back task based on the inference cache data, and submits the data write-back task to the asynchronous task pool. The data write-back task carries the newly added data identifier of the inference cache data. The cache management module obtains the data write-back task from the asynchronous task pool, writes the inference cache data into the data backup unit according to the new data identifier and the preset write strategy, generates a data write-back completion notification corresponding to the data write-back task, and writes the data write-back completion notification into the write-back confirmation queue. The scheduling module obtains the data write-back completion notification from the write-back confirmation queue, and updates the cache radix tree based on the newly added data identifier of the inference cache data and the newly added data storage location of the inference cache data carried in the data write-back completion notification.

11. The data processing method according to claim 10, wherein the data backup unit comprises a primary memory backup unit and a secondary disk backup unit; The step of writing the inference cache data into the data backup unit according to the newly added data identifier and the preset write strategy includes: According to the newly added data identifier, the inference cache data is obtained from the video memory, and the inference cache data is written to the memory-level backup unit; If the preset write policy determines that the utilization rate of the memory primary backup unit is greater than or equal to the preset utilization rate threshold, the inference cache data is written from the memory primary backup unit to the disk secondary backup unit through a multi-disk routing mechanism.

12. A data processing system, comprising a scheduling module and a cache management module, wherein: The scheduling module is used to receive inference requests, perform cache queries on the lexical sequence corresponding to the inference request, and generate a data loading task when it is determined that the cached data corresponding to the lexical sequence exists in the data backup unit, and submit the data loading task to the asynchronous task pool in a non-blocking manner. The cache management module operates independently of the scheduling module. It is used to obtain the data loading task from the asynchronous task pool, obtain the cached data from the data backup unit according to the data identifier and data storage location carried by the data loading task, write the cached data into the video memory, generate a data loading completion notification corresponding to the data loading task, and write the data loading completion notification into the loading confirmation queue. The scheduling module is further configured to obtain the data loading completion notification from the loading confirmation queue, determine the cached data in the video memory based on the data loading completion notification, and execute the inference request to generate inference results based on the cached data.

13. A computing device, comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1-11.

14. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1-11.

15. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1-11.

Images