Data processing method and related device

Abstract

This application discloses a data processing method and related equipment. Upon receiving a key-value caching request for a target word sequence block, this application maps the sequence block identifier of the target word sequence block to the hash value space corresponding to the current node cluster to obtain the target mapping location of the target word sequence block. Then, it filters out the target storage area corresponding to the target mapping location from the storage areas and determines the target virtual node corresponding to the target storage area. Based on the target virtual node, it filters out the target storage node corresponding to the target word sequence block from multiple storage nodes. Finally, it processes the target word sequence block based on the request type of the key-value caching request and the target storage node. This solution, by constructing a masterless distributed key-value caching management system, can achieve deterministic mapping from target word sequence blocks to storage nodes, avoiding single points of failure and expansion bottlenecks caused by centralized metadata services.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a data processing method and related equipment, including data processing devices, electronic devices, computer program products, and computer-readable storage media. Background Technology

[0002] During the inference process of a Large Language Model (LLM), the attention mechanism needs to access the key and value vectors of all historical tokens. The key-value cache (KV Cache) mechanism avoids redundant computation by caching these vectors, reducing the time complexity of the decoding phase from O(n^2) to O(n^2). 2 The memory consumption of the KV cache is reduced to O(n). However, the memory consumption of the KV cache increases linearly with the context length, forming a "working memory wall". Since the video memory of the graphics processing unit (GPU) cannot hold all the KV cache required for a large number of context inferences, the GPU needs to repeatedly recalculate the KV cache, resulting in low computational efficiency in the inference process.

[0003] The methods used in related technical solutions to improve KV Cache reuse mainly rely on a centralized scheduler to maintain global KV Cache metadata. This method has the risk of a single point of failure, and the centralized scheduler itself can become a bottleneck for expansion. Once a failure occurs, the entire cache layer will become unavailable, further reducing inference efficiency. Summary of the Invention

[0004] This application provides a data processing method and related equipment, which can achieve deterministic mapping from target lexical sequence blocks to storage nodes by constructing a masterless distributed key-value cache management system, thereby avoiding single points of failure and expansion bottlenecks caused by centralized metadata services.

[0005] In a first aspect, embodiments of this application provide a data processing method, including: Receive a key-value cache request for a target lexical sequence block, the key-value cache request carrying the sequence block identifier of the target lexical sequence block; The sequence block identifier is mapped to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes the storage intervals corresponding to multiple virtual nodes of the storage nodes. Filter out the target storage interval corresponding to the target mapping location in the storage interval, and determine the target virtual node corresponding to the target storage interval; Based on the target virtual node, the target storage node corresponding to the target word sequence block is selected from the plurality of storage nodes; Based on the request type of the key-value cache request and the target storage node, the target lexical sequence block is processed.

[0006] Accordingly, embodiments of this application provide a data processing apparatus, including: A request receiving unit is configured to receive a key-value cache request for a target lexical sequence block, wherein the key-value cache request carries the sequence block identifier of the target lexical sequence block; A hash mapping unit is used to map the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes storage intervals corresponding to multiple virtual nodes of the storage nodes. An interval filtering unit is used to filter out the target storage interval corresponding to the target mapping position in the storage interval, and determine the target virtual node corresponding to the target storage interval; A node positioning unit is used to filter out the target storage node corresponding to the target word sequence block from the plurality of storage nodes based on the target virtual node; The data processing unit is used to process the target lexical sequence block based on the request type of the key-value cache request and the target storage node.

[0007] Furthermore, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the data processing method provided in embodiments of this application.

[0008] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the data processing method provided in embodiments of this application.

[0009] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the data processing methods provided in embodiments of this application.

[0010] This application embodiment receives a key-value caching request for a target lexical sequence block, the key-value caching request carrying the sequence block identifier of the target lexical sequence block. Then, the sequence block identifier is mapped to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target lexical sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes storage intervals corresponding to multiple virtual nodes of the storage nodes. Then, the target storage interval corresponding to the target mapping position is selected from the storage intervals, and the target virtual node corresponding to the target storage interval is determined. Then, based on the target virtual node, the target storage node corresponding to the target lexical sequence block is selected from the multiple storage nodes. Finally, the target lexical sequence block is processed based on the request type of the key-value caching request and the target storage node.

[0011] Because this solution can build a distributed key-value cache management system without centralized metadata services based on the current node cluster (i.e., a masterless distributed key-value cache management system), in a masterless distributed key-value cache management system, each storage node is an equal "peer point", and each storage node has the dual role of service provider and acquirer, realizing resource sharing and load balancing.

[0012] Each storage node can maintain a local list of cluster members for building the consistent hashing ring. Furthermore, each storage node can be mapped to at least one virtual node, which is evenly distributed across the consistent hashing ring, dividing the hash value space corresponding to the consistent hashing ring into multiple storage regions.

[0013] When a key-value cache request for a target lexical sequence block is received, decentralized addressing can be quickly achieved through the aforementioned masterless distributed key-value cache management system. This means that in a masterless network environment, the target lexical sequence block's target mapping position in the hash value space can be quickly and accurately located using the sequence block identifier. Subsequently, the target virtual node can be determined based on the target mapping position, and then the target storage node corresponding to the target lexical sequence block can be determined based on the correspondence between the virtual node and the storage node, thus achieving a deterministic mapping from the target lexical sequence block to the storage node. Furthermore, the target lexical sequence block can be processed accordingly based on the request type of the key-value cache request.

[0014] Therefore, the above solution eliminates centralized metadata services, avoiding single points of failure and scalability bottlenecks. With the support of a masterless distributed key-value cache management system, the target storage node corresponding to the target lexical sequence block can be accurately and quickly located, enabling efficient utilization of the target key-value cache data corresponding to the target lexical sequence block, thereby improving the computational efficiency of large model inference processes. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1A This is a schematic diagram of an application scenario of the data processing method provided in the embodiments of this application; Figure 1B This is a schematic diagram illustrating another application scenario of the data processing method provided in the embodiments of this application; Figure 1C This is a schematic diagram illustrating another application scenario of the data processing method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the masterless distributed architecture of the current node cluster provided in the embodiments of this application; Figure 4A This is a schematic diagram of a consistent hash ring provided in an embodiment of this application; Figure 4B This is a schematic diagram of the hash value space partitioning of the consistent hash ring provided in an embodiment of this application; Figure 5 This is a schematic diagram of a data processing process corresponding to a key-value cache read request provided in an embodiment of this application; Figure 6A This is a schematic diagram of data flow after the target key-value cache data is hit in the node memory, provided in an embodiment of this application. Figure 6B This is a schematic diagram illustrating the data flow of target key-value cache data in the memory layer of a dual-queue node, as provided in an embodiment of this application. Figure 6C This is another schematic diagram illustrating the data flow of target key-value cache data in the memory layer of a dual-queue node, as provided in the embodiments of this application. Figure 7A This is a schematic diagram of data flow after the target key-value cache data is hit on the local storage hard disk, as provided in the embodiments of this application; Figure 7B This is a schematic diagram illustrating the data flow of target key-value cache data in a dual-queue local storage disk layer, as provided in an embodiment of this application. Figure 7C This is another schematic diagram illustrating the data flow of target key-value cache data in the dual-queue local storage disk layer provided in the embodiments of this application; Figure 8 This is a schematic diagram of a data processing process corresponding to a key-value cache write request provided in an embodiment of this application; Figure 9 This is a schematic diagram illustrating the eviction and warm-up of key-value cache data between different storage layers of a storage node, provided in an embodiment of this application. Figure 10A This is a schematic diagram illustrating the changes in the consistent hash ring provided in this application when a node exits; Figure 10B This is a schematic diagram illustrating the change in the spatial distribution of hash values ​​in the consistent hash ring provided in this application embodiment when a node exits; Figure 11A This is a schematic diagram illustrating the changes in the consistent hash ring provided in this application embodiment when a node is added; Figure 11B This is a schematic diagram illustrating the change in the spatial distribution of hash values ​​in the consistent hash ring provided in this application embodiment when a node is added; Figure 12 This is a flowchart illustrating the read-on-copy process; Figure 13 This is a schematic diagram of a product-side application scenario for the data processing method provided in the embodiments of this application; Figure 14 This is another schematic flowchart of the data processing method provided in the embodiments of this application; Figure 15 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application; Figure 16 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] The technical solutions described below, with reference to the accompanying drawings, will be clearly and completely described. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0018] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0019] Graphics Processing Unit (GPU): The core component of a graphics card, it is a chip specifically responsible for processing images, graphics operations, and large-scale parallel computing tasks. It is not only used for displaying images on computers and mobile phones, but also is the core hardware for artificial intelligence and scientific computing.

[0020] Central Processing Unit (CPU): As the core of a computer system for computation and control, it is the final execution unit for information processing and program execution.

[0021] Video memory (VRAM) is a dedicated memory on the graphics card responsible for temporarily storing rendering data processed by the graphics chip. The size of the VRAM directly determines how much graphics data the graphics card can process. The larger the capacity, the less likely it is to experience stuttering or errors when handling high-resolution games, 3D rendering, or artificial intelligence (AI) calculations.

[0022] High-bandwidth memory (HBM): This is the super memory of the GPU, designed specifically for artificial intelligence and high-performance computing. It uses 3D stacking technology to vertically stack multiple layers of storage chips, enabling the GPU to read and write data faster.

[0023] Solid State Drive (SSD): A hard drive made using an array of solid-state electronic storage chips. SSDs use two types of storage media: flash memory and dynamic random access memory (DRAM).

[0024] Non-volatile Memory Host Controller Interface (NVMe): This is a high-speed communication protocol designed specifically for solid-state drives (SSDs). Its purpose is to reduce latency, increase speed, and free SSDs from the limitations of older mechanical hard drive rules.

[0025] Key-Value Cache (KV Cache) is a key optimization technique used to accelerate the autoregressive inference process of large language models such as Transformer. Its core idea is to cache the calculated key and value vectors during the autoregressive generation process for reuse in subsequent generation steps, thereby avoiding redundant computation. The data processing method provided in this application can utilize the key-value caching mechanism to read or write key-value cached data.

[0026] Key-value cache offload: The process of moving the key-value cache from a high-cost, high-speed storage layer (such as GPUHBM) to a low-cost, high-capacity storage layer (such as CPU memory, solid-state drives, etc.). In this embodiment, key-value cache offload refers to the processing method corresponding to key-value cache write requests.

[0027] Key-value cache reuse: Sharing computed key-value caches across requests or sessions avoids repeatedly performing prefill computations during large language model inference processes. In this embodiment, key-value cache reuse refers to the processing method corresponding to key-value cache read requests.

[0028] Masterless architecture: The system lacks centralized metadata services or scheduling nodes; all nodes are equal and collaborate through distributed algorithms. In this embodiment, a masterless network architecture is used to achieve decentralized addressing, i.e., determining the target mapping location of the target lexical sequence block within a masterless network environment.

[0029] Peer-to-peer (P2P) networking is a distributed network architecture where participants share hardware resources and interact directly without relying on a central server. In P2P networking, all devices (nodes) in the network are equal "peers," with each node acting as both a service provider and a service recipient, achieving resource sharing and load balancing. In this embodiment, storage nodes in the current node cluster interact by building a P2P network.

[0030] Consistent Hashing: A distributed hashing algorithm that maps data to nodes in a ring space based on hash values, supporting minimal data migration when nodes are dynamically added or removed. In this embodiment, each storage node in the current node cluster locally maintains a list of cluster members for constructing the consistent hashing ring.

[0031] The Gossip Protocol is a decentralized, distributed communication protocol based on random interaction, used to propagate information synchronization between nodes to achieve eventual consistency. In this embodiment, storage nodes propagate member change events through the Gossip Protocol.

[0032] Prefill and Decode Separation Architecture (PD Separation Architecture): This is a large-scale language model inference optimization technique. Its core principle is to decouple the prefill and decode stages of the inference process, distributing them across different computational units. This architecture aims to solve problems such as memory explosion, low resource utilization, and service interruptions in large model inference. By physically isolating computationally intensive tasks (prefill) from memory-intensive tasks (decode), the system can independently configure hardware resources for the characteristics of different stages, significantly improving inference throughput and reducing latency.

[0033] The pre-filling stage includes the following tasks: processing all prompts from the user input at once, computing the attention of all input tokens in parallel, and generating an initial key-value cache. The pre-filling stage requires powerful parallel computing capabilities (GPU power) but has relatively low GPU memory requirements.

[0034] The decoding phase involves generating new tokens one by one based on the existing key-value cache, which is an autoregressive process. The decoding phase requires repeatedly and rapidly loading huge amounts of model weights and key-value cache from the existing database, placing extremely high demands on memory bandwidth but low computational power requirements.

[0035] In the process of large language model inference, the GPU's video memory cannot accommodate all the KV Cache required for a large number of context inferences, which leads to the problem that the GPU needs to repeatedly recalculate the KV Cache. Among the relevant technical solutions, the technical solution to improve the KV Cache reuse rate is Mooncake (a PD separation architecture centered on KV Cache). Its core components include: (1) a centralized scheduler (also known as a global scheduler), which maintains the metadata and location information of all KV Cache blocks; (2) a KV Cache pool, a KV Cache cache pool based on CPU DRAM; and (3) a transfer engine, a cross-node KV Cache high-speed transfer engine based on remote direct memory access (RDMA).

[0036] In the Mooncake solution, the centralized scheduler routes requests to nodes with corresponding cached key-value pairs based on the matching of request prompt prefixes, achieving cluster-level reuse. However, it has the following drawbacks: (1) Mooncake relies on a centralized scheduler to maintain global KV Cache metadata, which poses a single point of failure risk, and the scheduler itself will become a bottleneck for expansion (metadata query pressure of millions of KV blocks). (2) A failure of the centralized scheduler will render the entire cache layer unusable.

[0037] In this regard, embodiments of this application provide a data processing method and related equipment. The related equipment may include a data processing device, an electronic device, a computer program product, and a computer-readable storage medium. The data processing device may be integrated into an electronic device, which may be a server or a user terminal, etc.

[0038] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart TV, in-vehicle terminal, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0039] Figure 1AThis illustration shows an application scenario diagram of the data processing method provided in an embodiment of this application. For example... Figure 1A As shown, this example illustrates a data processing device integrated into an electronic device, where the electronic device functions as a server. Multiple servers can form a peer-to-peer network, and the servers within a peer-to-peer network collectively constitute the current node cluster of that network. For example, the current node cluster might include... Figure 1A Servers A, B, C, D, and E, as well as other servers not shown, are illustrated. Each server can be understood as a storage node in the current node cluster.

[0040] for Figure 1A For any server in the system, a large language model can be run. During the inference process of the large language model, the server can process existing key-value cache (KV Cache) data. Specifically, it can receive key-value cache requests for target lexical sequence blocks, which carry the sequence block identifier of the target lexical sequence block; map the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target lexical sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes storage intervals corresponding to multiple virtual nodes of the storage nodes; filter out the target storage interval corresponding to the target mapping position from the storage intervals, and determine the target virtual node corresponding to the target storage interval; based on the target virtual node, filter out the target storage node corresponding to the target lexical sequence block from the multiple storage nodes; and process the target lexical sequence block based on the request type of the key-value cache request and the target storage node.

[0041] Figure 1B This illustration shows another application scenario of the data processing method provided in the embodiments of this application. For example... Figure 1B As shown, an example is taken where the data processing device is integrated into an electronic device, and the electronic device serves as the terminal. Multiple terminals can also build a peer-to-peer network, and the terminals included in a peer-to-peer network collectively form the current node cluster of that network. For example, the current node cluster may include... Figure 1B The diagram shows terminals A, B, C, D, and E, as well as other terminals not shown. Each terminal can be understood as a storage node in the current node cluster.

[0042] for Figure 1BFor any terminal in the system, a large language model can be run. During the inference process of the large language model, the terminal can process existing key-value cache (KV Cache) data. Specifically, it can receive key-value cache requests for target word sequence blocks, which carry the sequence block identifier of the target word sequence block; map the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes storage intervals corresponding to multiple virtual nodes of the storage nodes; filter out the target storage interval corresponding to the target mapping position in the storage intervals, and determine the target virtual node corresponding to the target storage interval; based on the target virtual node, filter out the target storage node corresponding to the target word sequence block from multiple storage nodes; and process the target word sequence block based on the request type of the key-value cache request and the target storage node.

[0043] Figure 1C This illustration shows another application scenario of the data processing method provided in the embodiments of this application. For example... Figure 1C As shown, an example is taken where the data processing device is integrated into an electronic device, and the electronic device is a server or terminal. A peer-to-peer network can also be built between multiple terminals and multiple servers. The servers and terminals included in a peer-to-peer network together form the current node cluster of that peer-to-peer network. For example, the current node cluster may include... Figure 1C The diagram shows server A, server B, terminal A, terminal B, and terminal C, as well as other servers or terminals not shown. Each server and each terminal can be understood as a storage node in the current node cluster.

[0044] for Figure 1C For any server or terminal in the system, a large language model can be run. During the inference process of the large language model, the server or terminal can process existing key-value cache (KV Cache) data. Specifically, it can receive key-value cache requests for target word sequence blocks, the key-value cache request carrying the sequence block identifier of the target word sequence block; map the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes storage intervals corresponding to multiple virtual nodes of the storage nodes; filter out the target storage interval corresponding to the target mapping position in the storage interval, and determine the target virtual node corresponding to the target storage interval; based on the target virtual node, filter out the target storage node corresponding to the target word sequence block from multiple storage nodes; and process the target word sequence block based on the request type of the key-value cache request and the target storage node.

[0045] It should be noted that, Figures 1A-1CThere can be multiple servers or terminals.

[0046] It is understood that, in the specific implementation of this application, target word sequence blocks, request prompt words and other related data are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0047] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0048] This embodiment will be described from the perspective of a data processing device, which can be integrated into an electronic device, such as a server or a terminal. The terminal can include tablet computers, laptops, personal computers (PCs), wearable devices, virtual reality devices, or other smart devices that can generate image files.

[0049] A data processing method includes: receiving a key-value caching request for a target lexical sequence block, the key-value caching request carrying a sequence block identifier of the target lexical sequence block; mapping the sequence block identifier to a hash value space corresponding to the current node cluster to obtain the target mapping position of the target lexical sequence block, the current node cluster including multiple storage nodes, the hash value space including storage intervals corresponding to multiple virtual nodes of the storage nodes; filtering out the target storage interval corresponding to the target mapping position from the storage intervals, and determining the target virtual node corresponding to the target storage interval; filtering out the target storage node corresponding to the target lexical sequence block from the multiple storage nodes based on the target virtual node; and processing the target lexical sequence block based on the request type of the key-value caching request and the target storage node.

[0050] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application is shown. Figure 2 As shown, the specific process of this data processing method is as follows: 101. Receive a key-value cache request for the target lexical sequence block, the key-value cache request carrying the sequence block identifier of the target lexical sequence block.

[0051] In this context, the target word sequence block can be understood as the object of data processing by the data processing device during the reasoning process of a large language model. The target word sequence block can be a sequence block consisting of at least one word, and the words contained within the target word sequence block are arranged in a sequential order.

[0052] For example, the target word sequence block may include 4 words, 6 words, or other numbers of words. The number of words contained in the target word sequence block can be divided or set according to the actual situation, and this application embodiment does not limit this.

[0053] For any word in the target word sequence block, a corresponding key vector and value vector will be generated during the inference process of the large language model. The key vector and value vector constitute the key-value pair data of that word. The key-value pair data that has already been generated can be cached to obtain the key-value cache data of that word, so that it can be called at any time when needed, avoiding repeated calculations.

[0054] A target lexical sequence block may include at least one lexical. For ease of explanation, the key-value cache data corresponding to all lexicals contained in the target lexical sequence block is referred to as the target key-value cache data. A key-value cache request can be understood as a data processing request for the target key-value cache data corresponding to the target lexical sequence block.

[0055] Among them, key-value cache requests can be key-value cache read requests. The purpose of key-value cache read requests is to retrieve the cached target key-value cache data corresponding to the target lexical sequence block from a low-cost, high-capacity storage layer to avoid redundant calculations.

[0056] Key-value cache requests can also be key-value cache write requests. The purpose of a key-value cache write request is to persist the target key-value cache data corresponding to the newly calculated target lexical sequence block to a low-cost, high-capacity storage layer for reuse in future requests.

[0057] The sequence block identifier is the identity identifier of the target lexical sequence block, and different target lexical sequence blocks correspond to different sequence block identifiers. The sequence block identifier can be generated by the hash value of the lexical sequence prefix contained in the target lexical sequence block, that is, the sequence block identifier of the target lexical sequence block = hash(lexical sequence prefix).

[0058] Specifically, the sequence block identifier of the target word sequence block can be generated as follows: obtain the sequence prefix corresponding to the word sequence contained in the target word sequence block; perform hash calculation on the sequence prefix to obtain the target hash value of the target word sequence block; and use the target hash value as the sequence block identifier of the target word sequence block.

[0059] Taking request A as “The capital of France” as an example, if request A is split into two word sequence blocks, namely word sequence block 0 and word sequence block 1, both word sequence block 0 and word sequence block 1 can be used as target word sequence blocks. Among them, word sequence block 0 = [“The”, “capital”], and word sequence block 1 = [“of”, “France”].

[0060] For word sequence block 0, its corresponding sequence prefix is ​​the sequence block identifier corresponding to ["The", "capital"]; for word sequence block 1, its corresponding sequence prefix is ​​the sequence block identifier corresponding to ["The", "capital", "of", "France"], and not just the sequence block identifier corresponding to ["of", "France"].

[0061] Determining the sequence block identifier of the target lexical sequence block using the above method means that each target lexical sequence block is identified not only by a few lexical elements within the block, but by the complete history from the beginning of the sequence to the end of the target lexical sequence block. This ensures that only requests with completely identical prefixes can share the cache, avoiding erroneous reuse.

[0062] 102. Map the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes the storage range corresponding to multiple virtual nodes of the storage nodes.

[0063] The current node cluster can be understood as a network cluster system using a masterless distributed architecture. For example... Figures 1A-1C As shown, the current node cluster can include multiple storage nodes. The electronic device where the data processing device is located can be any one of the storage nodes in the current node cluster, or any one of the storage nodes in the current node cluster can be regarded as the electronic device where the data processing device is located.

[0064] Figure 3 This diagram illustrates a masterless distributed architecture for the current node cluster provided in an embodiment of this application. For example... Figure 3 As shown, the current node cluster can include multiple storage nodes ( Figure 3 The diagram shows three storage nodes (the exact number of storage nodes will be determined based on actual conditions). These storage nodes transmit data through a peer-to-peer network. All storage nodes are equal in status and work collaboratively using a distributed algorithm.

[0065] like Figure 3As shown, each storage node can include multiple functional modules (layers), namely, the global node identifier (ID) service module, the consistent hash management & key-value cache migration module, the peer routing module, the dual-queue node memory layer, the dual-queue local storage hard disk layer, and the global shared storage software development kit (i.e., the global shared storage SDK).

[0066] The global node identification service module assigns a globally unique identifier to each newly added storage node in the current node cluster and maintains a lightweight liveness status. Its key design features include: (1) A decentralized lease mechanism is adopted, in which each storage node registers a unique identity through a distributed lock when it starts up. Alternatively, a more lightweight solution can be adopted, which generates a globally unique identity based on (IP + Port + startup timestamp) without external dependencies.

[0067] (2) Member change events are propagated between storage nodes via the Gossip protocol, and the entire cluster converges in O(log N) time. (3) Each storage node maintains a local list of cluster members for the construction of a consistent hash ring.

[0068] In the masterless distributed architecture provided in this application embodiment, each storage node in the current node cluster stores and maintains its own cluster member list. Storage nodes can synchronize member changes through the Gossip protocol, so each storage node has an eventually consistent cluster member list. Each storage node can independently build a consistent hash ring based on its locally maintained cluster member list, thus eliminating the need for a centralized scheduler (such as a service discovery component).

[0069] The Consistent Hash Management & Key-Value Cache Migration module manages the mapping from virtual nodes to physical nodes (storage nodes) and performs minimal data migration when nodes are added or removed. The construction process of the consistent hashing ring is as follows: (1) Map each storage node to at least one virtual node, and generate a hash value for each virtual node. The hash value is used to represent the subsequent position of the virtual node on the hash ring to ensure that the data is evenly distributed. (2) Sort the virtual nodes according to their corresponding hash values ​​to form a ring structure with the beginning and end connected. This ring structure is called the hash ring, so that the virtual nodes are evenly distributed on the hash ring.

[0070] It's understandable that the number of virtual nodes mapped to each storage node can be the same or different. Although each storage node can construct a hash ring based on its maintained cluster member list, as long as the cluster member lists maintained by each storage node eventually become consistent, the ring structures they construct will be exactly the same (because the hash algorithm and sorting rules are deterministic). Therefore, for the same sequence block identifier, any storage node in the current node cluster will calculate the exact same target mapping location, without the need for central arbitration.

[0071] Figure 4A A schematic diagram of a consistent hash ring provided in an embodiment of this application is shown. Figure 4A As shown, taking a current node cluster with 4 storage nodes as an example, for ease of distinction, they will be referred to as storage node A, storage node B, storage node C, and storage node D. If each storage node is mapped to 3 virtual nodes, for example, the 3 virtual nodes corresponding to storage node A can include virtual node A0, virtual node A1, and virtual node A2. The above 4 storage nodes can be mapped to a total of 12 virtual nodes, which are evenly distributed on the hash ring.

[0072] Figure 4B This diagram illustrates the hash value space partitioning of a consistent hash ring provided in an embodiment of this application. Figure 4B As shown, in Figure 4A Based on the consistent hash ring shown, 12 virtual nodes divide the hash ring into 12 storage intervals (i.e., hash intervals), with each virtual node corresponding to one storage interval. In this embodiment, the hash value space can be understood as the total storage intervals corresponding to the consistent hash ring constructed based on the current node cluster, that is, all hash intervals covered by the consistent hash ring.

[0073] Based on this, the sequence block identifier can be mapped to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. Specifically, this step may include: determining the number of virtual nodes, where the number of virtual nodes indicates the number of storage intervals in the hash value space; The sequence block identifier is moduloed based on the number of virtual nodes to obtain the modulo operation result; based on the modulo operation result, the target mapping position of the target word sequence block in the hash value space is determined.

[0074] The sequence block identifier of the target word sequence block is a fixed-length hash value, for example, hash = crc32("user:123:profile") → 0x4A2F3B6E. The hash value represents a large integer space, which needs to be mapped to a finite number of N virtual nodes. This can be achieved by performing a modulo operation on the sequence block identifier based on the number of virtual nodes, obtaining the modulo result.

[0075] The modulo operation involves calculating the remainder after dividing the sequence block identifier by the number of virtual nodes. In the modulo operation, the sequence block identifier is the dividend, and the number of virtual nodes is the divisor.

[0076] 103. Filter out the target storage interval corresponding to the target mapping location in the storage interval, and determine the target virtual node corresponding to the target storage interval.

[0077] After determining the target mapping position of the target word sequence block in the hash value space, the target storage interval corresponding to that target mapping position can be selected from the multiple storage intervals contained in the hash value space. For example, N virtual nodes can divide the hash value space into N storage intervals. After performing a modulo operation on the sequence block identifier according to the number of virtual nodes (N), the result of the modulo operation can indicate one of the storage intervals (target storage intervals) in which the sequence block identifier finally falls. The target storage interval corresponds to the target mapping position of the target word sequence block in the hash value space.

[0078] Specifically, such as Figure 4B As shown, if the modulo operation result is 2, it means that the sequence block identifier is mapped to the second storage interval in the consistent hash ring, that is, the interval identifier of the target storage interval is 2.

[0079] After determining the target storage range, the first virtual node can be found by searching clockwise along the consistent hash ring, and this first virtual node will be used as the target virtual node for that storage range. For example... Figure 4B As shown, the first virtual node in the clockwise direction of the second storage interval is virtual node C0. Therefore, virtual node C0 is the target virtual node corresponding to the target storage interval.

[0080] 104. Based on the target virtual node, select the target storage node corresponding to the target word sequence block from multiple storage nodes.

[0081] As mentioned earlier, during the construction phase of the consistent hashing ring, each storage node can be mapped to at least one virtual node. This means that there is a correspondence between storage nodes and virtual nodes. After determining the target virtual node, the target storage node corresponding to the target word sequence block can be selected from multiple storage nodes based on the correspondence between the two.

[0082] For example, for Figure 4B The target virtual node (virtual node C0) in the sequence is located at storage node C. Therefore, the target storage node corresponding to the target word sequence block can be determined to be storage node C.

[0083] In this embodiment of the application, the target storage node is the physical node where the target word sequence block should be stored.

[0084] Therefore, the peer routing module in the storage node is used to receive key-value caching requests for target lexical sequence blocks issued by the upper-layer inference engine, and to determine the target storage node based on the sequence block identifier of the target lexical sequence block. Its workflow may include: (1) Map the sequence block identifier in the key-value cache request to the consistent hash ring to determine the target mapping location; (2) Find the first virtual node corresponding to the target mapping position in the clockwise direction along the consistent hash ring to obtain the target virtual node; (3) Based on the target virtual node, select the target storage node corresponding to the target word sequence block from multiple storage nodes.

[0085] Furthermore, during the operation of the peer routing module, if the target storage node is detected to be unreachable (timeout / failure), it can automatically find the next virtual node as the target virtual node by clockwise following the consistent hash ring, and determine the target storage node based on the newly determined target virtual node, until an available target storage node is found as a replacement.

[0086] The peer routing module can also support the nearest neighbor affinity strategy, which prioritizes routing to the local node (i.e., the GPU node that generates the key-value cache request) to reduce network transmission.

[0087] In this embodiment, a distributed key-value cache management system that eliminates the need for a centralized metadata service is constructed. This masterless architecture corresponds to a current node cluster comprising multiple storage nodes, all of which are equal in status and collaborate through a distributed algorithm. Based on this, consistent hashing can be used to automatically address target lexical sequence blocks and determine their corresponding target storage nodes. This method eliminates centralized metadata services (such as the centralized scheduler in Mooncake), avoiding single points of failure and scalability bottlenecks. Furthermore, deterministic mapping from target lexical sequence blocks to storage nodes can be achieved based on consistent hashing and virtual nodes.

[0088] 105. Process the target word sequence block based on the request type and target storage node of the key-value cache request.

[0089] Once the target storage node corresponding to the target lexical sequence block is determined, the target lexical sequence block can be processed based on the request type of the key-value cache request and the target storage node.

[0090] As mentioned earlier, the request type can be either a key-value cache read request or a key-value cache write request. Different request types correspond to different processing methods, which will be explained in detail later.

[0091] Furthermore, when processing target word sequence blocks, it is also necessary to determine whether the target storage node is a local storage node. Specifically, based on the request type of the key-value cache request and the target storage node, the processing of target word sequence blocks can be divided into the following two cases: The first scenario: When the target storage node is a local storage node, the target word sequence block is processed through the target storage node according to the request type.

[0092] The local storage node is the storage node that receives key-value cache requests, and it is also the storage node where the data processing device is currently located. Figure 4A Taking the four storage nodes corresponding to the consistent hash ring shown as an example, if the local storage node is storage node C, and the target storage node calculated above is also storage node C, then the target storage node is the same as the local storage node. In this case, the target word sequence block can be processed directly through the target storage node (storage node C) according to the request type, without needing to process the target word sequence block across nodes.

[0093] The second scenario: When the target storage node is not a local storage node, the key-value cache request is sent to the target storage node so that the target storage node can process the target lexical sequence block based on the request type.

[0094] Similarly Figure 4A Taking the four storage nodes corresponding to the consistent hash ring shown as an example, if the local storage node is storage node A, and the target storage node calculated above is storage node C, the two are inconsistent. Therefore, the target storage node is a non-local storage node. In this case, the local storage node (storage node A) needs to send the key-value cache request it receives to the target storage node (storage node C) so that the target storage node (storage node C) can process the target token sequence block according to the request type carried in the key-value cache request.

[0095] It is understandable that when the target storage node is not a local storage node, after the local storage node forwards the key-value cache request to the target storage node, the target storage node does not need to repeatedly calculate the target mapping position and other information based on the sequence block identifier in the key-value cache request. It can directly process the target word sequence block according to the request type carried in the key-value cache request.

[0096] like Figure 3 As shown, each storage node internally includes three layers of storage control: a dual-queue node memory layer, a dual-queue local storage disk layer, and a global shared storage SDK. Key-value cached data can automatically flow between these three storage layers.

[0097] In this embodiment, the dual-queue local storage hard disk layer and global shared storage SDK can overcome the capacity limitation of single-machine dynamic random access memory (DRAM), enabling the key-value cache data that the storage node can manage to expand from the TB (terabyte) level to the PB (petabyte) level.

[0098] It is understandable that the key-value cache data can be the target key-value cache data corresponding to the target word sequence block, or it can be the key-value cache data corresponding to other word sequence blocks.

[0099] As mentioned earlier, key-value cache requests can include key-value cache read requests. The purpose of a key-value cache read request is to retrieve the cached key-value cache data (i.e., the target key-value cache data) corresponding to the target lexical sequence block, thus avoiding redundant calculations.

[0100] The inference engine can initiate a key-value cache read request in the following situations: (1) Prefill stage: prefix cache lookup.

[0101] When a new prompt arrives, the inference engine breaks it down into several lexical sequence blocks, each of which can contain at least one lexical. For each sequence block, the inference engine calculates its sequence block identifier and checks if the corresponding key-value cache data exists. If it exists, the inference engine directly obtains the key-value cache data for that lexical sequence block; otherwise, it uses that lexical sequence block as the target lexical sequence block and initiates a key-value cache read request for that target lexical sequence block.

[0102] (2) Decode stage: Read the existing key-value cache data and perform incremental calculation.

[0103] During the autoregressive generation process, for each new term generated, the inference engine needs to read all existing key-value cache data (from the first term to the current term) of that sequence to calculate the probability distribution of the next term. The existing key-value cache data may all be stored in GPU memory; in this case, the inference engine can directly utilize the key-value cache data in GPU memory for calculation.

[0104] Existing key-value cache data may be partially stored in GPU memory, while the remainder may be distributed across low-cost, high-capacity storage layers (such as CPU memory, solid-state drives, etc.) on the same storage node, or distributed across different storage nodes. In this case, it is necessary to treat the missing word sequence blocks in GPU memory as target word sequence blocks and initiate key-value cache read requests for the target word sequence blocks.

[0105] In this embodiment of the application, when the request type is a key-value cache read request, the target word sequence block is processed through the target storage node according to the request type, which may include: querying the target key-value cache data corresponding to the target word sequence block in the node memory of the target storage node; and reading the target key-value cache data from the node memory when the target key-value cache data exists in the node memory.

[0106] Figure 5 This illustration shows a data processing procedure corresponding to a key-value cache read request provided in an embodiment of this application. For example... Figure 5 As shown, for a received key-value cache read request, the system first queries the node memory of the target storage node to determine whether the target key-value cache data corresponding to the target word sequence block exists in the node memory.

[0107] When target key-value cache data exists in the node's memory, it can be read directly from the node's memory. This process of reading target key-value cache data from the node's memory can be understood as returning the target key-value cache data stored in the node's memory to the GPU memory corresponding to the inference engine.

[0108] Specifically, when the target storage node is a local storage node, the target key-value cache data stored in the node's memory can be directly returned to the GPU memory of that local storage node. When the target storage node is not a local storage node, the target key-value cache data stored in the target storage node can be transferred to the local storage node via a peer-to-peer network. After the target key-value cache data enters the local storage node, the return path to the GPU memory of the local storage node is determined based on the current location of the target key-value cache data.

[0109] like Figure 3 As shown, the node memory is designed as a dual-queue node memory layer. This dual-queue layer identifies data that only happens to pass through the node memory and is not truly hot data, thus protecting the truly hot data in the node memory. Specifically, the node memory can include a memory history queue and a memory cache queue.

[0110] The memory history queue is used to store key-value cached data that enters the node's memory. It is a first-in, first-out (FIFO) queue; all new data entering the node's memory first enters this queue, serving as an "observation period" for assessing data potential.

[0111] The memory cache queue is used to store key-value cached data that has been accessed in the memory history queue. The memory cache queue is also a first-in, first-out (FIFO) queue; when data in the memory history queue is accessed a second time, this accessed data is "promoted" to the memory cache queue. Therefore, the data in the memory cache queue represents the truly hot data in the node's memory.

[0112] In this embodiment, the memory quota of a node can be divided to obtain the respective memory quota percentages for the memory history queue and the memory cache queue. The memory quota percentages for the memory history queue and the memory cache queue can be set and adjusted according to actual conditions. For example, 25% of the memory quota can be allocated to the memory history queue, and 75% of the memory quota can be allocated to the memory cache queue.

[0113] In this embodiment, when target key-value cache data exists in the node memory, it can not only be read from the node memory, but also, based on the aforementioned dual-queue node memory layer design, the target key-value cache data can be transferred between the node memory's memory history queue and memory cache queue. Compared to reading target key-value cache data from the node memory, transferring target key-value cache data between the node memory's memory history queue and memory cache queue can be an asynchronous process, with relatively lower timeliness requirements.

[0114] Specifically, Figure 6A This illustration shows a data flow diagram after the target key-value cache data is hit in the node memory, as provided in an embodiment of this application. For example... Figure 6A As shown, after reading the target key-value cache data from the node's memory, the following two scenarios may occur: Case (1): When the target key-value cache data is in the memory cache queue, the target key-value cache data is moved to the tail of the memory cache queue.

[0115] Figure 6B This diagram illustrates the data flow of target key-value cache data in the dual-queue node memory layer, as provided in an embodiment of this application. Figure 6B As shown, for ease of explanation, the position where the arrow enters is called the tail of the queue, and the position where the arrow exits is called the head of the queue (the same applies to the following diagrams).

[0116] When the target key-value cache data is located in the memory cache queue ( Figure 6B When the shaded area marks the storage location of the target key-value cache data, the target key-value cache data can be moved to the tail of the memory cache queue. Following the first-in, first-out (FIFO) rule, moving the target key-value cache data to the tail of the memory cache queue can extend its residence time in the memory cache queue.

[0117] Case (2): When the target key-value cache data is in the memory history queue, load the target key-value cache data from the memory history queue to the tail of the memory cache queue.

[0118] Figure 6C This illustration shows another schematic diagram of the target key-value cache data flowing through the dual-queue node memory layer, as provided in the embodiments of this application. For example... Figure 6C As shown, when the target key-value cache data is located in the memory history queue ( Figure 6C When the shaded area marks the storage location of the target key-value cache data, the target key-value cache data can be loaded from its original location in the memory history queue to the tail of the memory cache queue.

[0119] The process of loading the target key-value cache data from its original location in the memory history queue to the tail of the memory cache queue is also the process of "promoting" the target key-value cache data from its original location in the memory history queue to the tail of the memory cache queue.

[0120] like Figure 5 As shown, after querying the target key-value cache data corresponding to the target word sequence block in the node memory of the target storage node, it may further include: when the target key-value cache data does not exist in the node memory, querying the target key-value cache data in the local storage hard disk of the target storage node; when the target key-value cache data exists in the local storage hard disk, reading the target key-value cache data from the local storage hard disk.

[0121] In this embodiment of the application, when the node memory does not find the target key-value cache data (i.e., the target key-value cache data does not exist in the node memory), the node can continue to query the local storage hard disk of the target storage node to determine whether the target key-value cache data corresponding to the target word sequence block exists in the local storage hard disk.

[0122] When target key-value cache data exists on the local storage hard drive, it can be read from the local storage hard drive. This process can be understood as returning the target key-value cache data stored on the local storage hard drive to the GPU memory corresponding to the inference engine.

[0123] Specifically, when the target storage node is a local storage node, the target key-value cache data stored in the local storage node's local hard drive can be directly returned to the GPU memory of that local storage node. When the target storage node is not a local storage node, the target key-value cache data stored in the target storage node can be transferred to the local storage node via a peer-to-peer network. After the target key-value cache data enters the local storage node, the return path to the GPU memory of the local storage node is determined based on the current location of the target key-value cache data.

[0124] like Figure 3 As shown, the local storage hard drive is designed as a dual-queue local storage hard drive layer. This dual-queue approach identifies data that only happens to pass through the local storage hard drive layer and is not truly hot data, thus protecting the truly hot data within the local storage hard drive layer. Specifically, the local storage hard drive can include a hard drive history queue and a hard drive cache queue.

[0125] The hard disk history queue is used to store key-value cache data entering the local storage hard disk. It is a first-in, first-out (FIFO) queue; all new data entering the local storage hard disk first enters this queue, serving as an "observation period" to assess data potential.

[0126] The disk cache queue stores key-value cached data that has been accessed in the disk history queue. The disk cache queue is also a first-in, first-out (FIFO) queue; when data in the disk history queue is accessed a second time, this accessed data is "promoted" to the disk cache queue. Therefore, the data in the disk cache queue represents the truly frequently accessed data on the local storage disk.

[0127] In this embodiment, the storage quota can also be allocated according to the local storage hard drive, resulting in the respective storage quota percentages for the hard drive history queue and the hard drive cache queue. The storage quota percentages for the hard drive history queue and the hard drive cache queue can be set and adjusted according to actual conditions. For example, 25% of the storage quota can be allocated to the hard drive history queue, and 75% to the hard drive cache queue.

[0128] In this embodiment, when target key-value cache data exists in the local storage hard disk, it can not only be read from the local storage hard disk, but also, based on the aforementioned dual-queue local storage hard disk layer design, the target key-value cache data can be transferred between the hard disk history queue and the hard disk cache queue of the local storage hard disk. Compared to reading target key-value cache data from the local storage hard disk, transferring target key-value cache data between the hard disk history queue and the hard disk cache queue of the local storage hard disk can be an asynchronous process, with relatively lower timeliness requirements.

[0129] Specifically, Figure 7A This illustration shows a data flow diagram after the target key-value cache data is hit on the local storage hard disk, as provided in an embodiment of this application. For example... Figure 7A As shown, after reading the target key-value cache data from the local storage hard drive, the following two situations may occur: Case (1): When the target key-value cache data is in the disk cache queue, the target key-value cache data is loaded to the tail of the memory history queue in the node memory.

[0130] Figure 7B This illustration shows a schematic diagram of the data flow of target key-value cache data in a dual-queue local storage disk layer, as provided in an embodiment of this application. Figure 7B As shown, when the target key-value cache data is located in the disk cache queue ( Figure 7B When the shaded area indicates the storage location of the target key-value cache data, the target key-value cache data can be loaded across levels into the node memory and loaded to the tail of the memory history queue in the node memory.

[0131] Case (2): When the target key-value cache data is in the disk history queue, load the target key-value cache data from the disk history queue to the tail of the disk cache queue.

[0132] Figure 7C This illustration shows another schematic diagram of the target key-value cache data flowing through a dual-queue local storage disk layer, as provided in an embodiment of this application. For example... Figure 7C As shown, when the target key-value cache data is located in the disk history queue ( Figure 7C When the shaded area indicates the storage location of the target key-value cache data, the target key-value cache data can be loaded from its original location in the hard disk history queue to the tail of the hard disk cache queue.

[0133] The process of loading the target key-value cache data from its original location in the hard disk history queue to the tail of the hard disk cache queue is also the process of "promoting" the target key-value cache data from its original location in the hard disk history queue to the tail of the hard disk cache queue.

[0134] like Figure 5 As shown, after querying the target key-value cache data in the local storage hard disk of the target storage node, the process may further include: when the target key-value cache data does not exist in the local storage hard disk, querying the target key-value cache data from the global shared storage space corresponding to the current node cluster through the global shared query interface of the target storage node; when the target key-value cache data exists in the global shared storage space, reading the target key-value cache data from the global shared storage space; and loading the target key-value cache data to the tail of the hard disk history queue of the local storage hard disk.

[0135] In this embodiment of the application, when the target key-value cache data is not found in the local storage hard disk (i.e., the target key-value cache data does not exist in either the node memory or the local storage hard disk), the query can continue to be performed in the global shared storage space corresponding to the current node cluster to determine whether the target key-value cache data exists in the global shared storage space.

[0136] The global shared storage space can be understood as the underlying storage space shared by all storage nodes in the current node cluster. For example... Figure 3 As shown, for any storage node, a Global Shared Storage Software Development Kit (i.e., Global Shared Storage SDK) can be installed. The Global Shared Storage SDK provides a global shared query interface for accessing the global shared storage space. Through the global shared query interface, not only can the global shared storage space be accessed for data reading, but the key-value cache data of the storage node (such as key-value cache data that has been evicted from the storage node) can also be persisted to the global shared storage space, ensuring that data is not lost and fault recovery is possible.

[0137] Specifically, the global shared storage SDK provided in this application embodiment supports the following functions: (1) Connect to general distributed storage systems (such as CephFS, HDFS or object storage). (2) Asynchronous batch writing: Multiple evicted data blocks are aggregated and written in batches to reduce input / output (IO) frequency; (3) Supports configurable replication strategies (single replica / dual replica), balancing storage costs and reliability; (4) When a storage node fails and recovers, the key-value cache data (blocks) responsible for the sharding of that storage node are pulled from the global storage as needed to rebuild the local cache; (5) Provide data source for the preheating scheduler: respond to preheating requests and return key-value cached data (blocks) according to priority.

[0138] like Figure 5 As shown, when target key-value cache data exists in the global shared storage space, it can be read from the global shared storage space. Furthermore, the target key-value cache data can be asynchronously loaded to the tail of the local storage hard drive's history queue, thereby enabling cross-level flow of target key-value cache data.

[0139] like Figure 5As shown, after querying the target key-value cache data from the global shared storage space corresponding to the current node cluster through the global shared query interface of the target storage node, it may further include: when the target key-value cache data does not exist in the global shared storage space, calculating the key-value data corresponding to the target word sequence block; writing the key-value data to the tail of the memory history queue of the target storage node, and copying the key-value data to obtain backup key-value data; storing the backup key-value data in the global shared storage space.

[0140] In this embodiment of the application, if the target key-value cache data is not found in the global shared storage space, it is necessary to perform pre-filling calculation on the target word sequence block to obtain the key-value data corresponding to the target word sequence block.

[0141] The calculated key-value data can be written to the tail of the memory history queue of the target storage node. Simultaneously, the key-value data can be copied to obtain a backup, which is then persisted to the global shared storage space as a backup for persistence.

[0142] As mentioned earlier, key-value cache requests can include key-value cache write requests. The purpose of a key-value cache write request is to persist the target key-value cache data corresponding to the newly calculated target lexical sequence block to a low-cost, high-capacity storage layer for reuse in future requests. Since a key-value cache write request writes the newly calculated target key-value cache data, if the key-value cache request is a key-value cache write request, then the key-value cache request already includes the target key-value cache data corresponding to the target lexical sequence block.

[0143] The inference engine can initiate a key-value cache write request in the following situations: (1) Prefill stage: The key-value cache data corresponding to the newly calculated complete lexical sequence block is obtained.

[0144] When pre-filling a missed lexical sequence block, the inference engine actually performs attention computation, generating key-value cache data for that lexical sequence block. After computation, the inference engine assembles the key-value cache data for that lexical sequence block and writes the key-value cache data into the cache.

[0145] (2) Decode stage: the moment when a block of lexical sequences is filled.

[0146] During word-by-word generation, each new word is appended to the end of the current sequence. When the last incomplete word sequence block is just filled (i.e., the sequence length becomes an integer multiple of the sequence block size), this newly completed block can be cached.

[0147] For example, if the sequence block size is 4, and the current sequence already contains 7 tokens (token sequence block 0 is full, and token sequence block 1 has only 3 tokens), then after the 8th token is generated, token sequence block 1 is filled. At this point, the inference engine calculates the prefix hash corresponding to token sequence block 1 (i.e., the cumulative hash of the first 8 tokens, which is also the sequence block identifier of token sequence block 1); and writes the key-value cache data of token sequence block 1 into the cache according to the sequence block identifier of token sequence block 1.

[0148] In this embodiment of the application, when the key-value cache request is a key-value cache write request, the target lexical sequence block is processed by the target storage node according to the request type. This may include: determining the memory history queue load information of the memory history queue in the node memory of the target storage node at the current moment; when the memory history queue load information indicates that the memory history queue has reached its capacity limit, migrating the key-value cache data located at the head of the memory history queue to the local storage hard disk of the target storage node, and writing the target key-value cache data to the tail of the memory history queue; when the memory history queue load information indicates that the memory history queue has not reached its capacity limit, writing the target key-value cache data to the tail of the memory history queue.

[0149] Figure 8 This illustration shows a data processing procedure corresponding to a key-value cache write request provided in an embodiment of this application. For example... Figure 8 As shown, for the received key-value cache write request, the target key-value cache data of the newly calculated target lexical sequence block needs to be written into the node memory of the target storage node. Since the target key-value cache data enters the node memory, it needs to be written into the memory history queue of the target storage node.

[0150] To address this, it is necessary to determine the memory history queue load information in the target storage node's memory at the current moment. The memory history queue load information indicates whether the memory history queue has reached its capacity limit. The capacity limit of the memory history queue can be set and adjusted according to actual conditions. For example, the capacity limit of the memory history queue can be 90%, 95%, or other percentage values ​​of the memory quota allocated to the memory history queue.

[0151] like Figure 8 As shown, if the memory history queue load information indicates that the memory history queue has not reached its capacity limit, it means that the memory history queue can still accommodate other data at the current moment. Accordingly, the target key-value cache data of the target word sequence block can be directly written to the tail of the memory history queue.

[0152] It's understandable that if the target key-value cache data is large, the process of writing it from the tail of the queue to the memory history queue might cause the memory history queue to reach its capacity limit. Therefore, it's necessary to first move the key-value cache data at the head of the memory history queue out of the queue to free up sufficient space before continuing to write the remaining target key-value cache data.

[0153] like Figure 8 As shown, if the memory history queue load information indicates that the memory history queue has reached its capacity limit, it means that the target key-value cache data of the target word sequence block cannot be directly written to the memory history queue at the current moment. In this case, the key-value cache data at the head of the memory history queue needs to be moved out of the memory history queue first to make room for writing the target key-value cache data.

[0154] like Figure 3 As shown, since the local storage hard disk is the next level of storage space after the node memory, key-value cache data that has been evicted from the memory history queue can be migrated to the local storage hard disk. Specifically, when the memory history queue load information indicates that the memory history queue has reached its capacity limit, the key-value cache data located at the head of the memory history queue can be migrated to the local storage hard disk of the target storage node, and the target key-value cache data can be written to the tail of the memory history queue.

[0155] The process of migrating key-value cache data at the head of the memory history queue to the local storage hard disk of the target storage node may include: determining the hard disk history queue load information of the hard disk history queue in the local storage hard disk of the target storage node at the current moment; when the hard disk history queue load information indicates that the hard disk history queue has reached its capacity limit, migrating the key-value cache data at the head of the hard disk history queue to the global shared storage space, and migrating the key-value cache data at the head of the memory history queue to the tail of the hard disk history queue; when the hard disk history queue load information indicates that the hard disk history queue has not reached its capacity limit, migrating the key-value cache data at the head of the memory history queue to the tail of the hard disk history queue.

[0156] In this embodiment, key-value cache data located at the head of the memory history queue is migrated to the local storage hard disk of the target storage node. The purpose is to migrate the key-value cache data located at the head of the memory history queue to the hard disk history queue of the local storage hard disk. This is because the hard disk history queue is used to store key-value cache data entering the local storage hard disk.

[0157] Based on this, such as Figure 8As shown, it is necessary to first determine whether the disk history queue in the target storage node's local storage disk has reached its capacity limit at the current moment. The capacity limit of the disk history queue can be set and adjusted according to actual conditions. For example, the capacity limit of the disk history queue can be 90%, 95%, or other percentage values ​​of the storage quota allocated to the disk history queue.

[0158] If the hard disk history queue load information indicates that the hard disk history queue has not reached its capacity limit, it means that the hard disk history queue can still accommodate other data at the current moment. Accordingly, the key-value cache data at the head of the memory history queue can be moved to the tail of the hard disk history queue.

[0159] If the disk history queue load information indicates that the disk history queue has reached its capacity limit, it means that the key-value cache data at the head of the memory history queue cannot be directly migrated to the disk history queue at the current moment. In this case, it is necessary to first migrate the key-value cache data at the head of the disk history queue out of the disk history queue in order to leave enough space to accommodate the key-value cache data at the head of the memory history queue.

[0160] like Figure 3 As shown, since the global shared storage space is the next level of storage space after the local storage hard disk, key-value cache data that has been evicted from the hard disk history queue can be migrated to the global shared storage space. Specifically, when the hard disk history queue load information indicates that the hard disk history queue has reached its capacity limit, the key-value cache data located at the head of the hard disk history queue can be migrated to the global shared storage space, and then the key-value cache data located at the head of the memory history queue can be migrated to the tail of the hard disk history queue.

[0161] In this embodiment, even if no key-value cache write request is received, for any storage node's node memory, when the memory water level exceeds the high watermark (e.g., 90% or other percentage values), data at the head of the memory history queue (e.g., key-value cache data) is preferentially evicted. Data evicted from the head of the memory history queue is downgraded and written to the storage node's local storage hard drive, specifically to the hard drive's history queue. When the memory water level exceeds the high watermark (e.g., 90% or other percentage values) and the memory history queue is empty, data at the head of the memory cache queue (e.g., key-value cache data) can be evicted. Data evicted from the head of the memory cache queue can be written to the tail of the memory history queue so that it can be subsequently downgraded and written to the storage node's local storage hard drive through the memory history queue's eviction mechanism.

[0162] As described above, the dual-queue node memory layer in the storage node provided in this application embodiment can manage key-value cached hot data in the node memory and achieve intelligent eviction against scan contamination. Its data structure and working mechanism are as follows: (1) Memory history queue: Newly written key-value cache data first enters the memory history queue and is evicted in the Least Recently Used (LRU) order. Key-value cache data generated during one-time pre-filling is naturally evicted here; (2) Memory cache queue: When key-value cache data in the memory history queue is accessed a second time, it is promoted to the tail of the memory cache queue (latest position) and evicted in the Least Recently Used (LRU) order.

[0163] (3) Eviction trigger: When the memory water level exceeds the high water level (e.g., 90%), the key-value cache data at the head of the memory history queue is evicted first; when the memory history queue is empty, the key-value cache data at the head of the memory cache queue is evicted.

[0164] (4) Elimination target: Eliminated key-value cache data is downgraded and written to the local storage hard disk for warm storage.

[0165] Furthermore, the dual-queue local storage hard disk layer in the storage node provided in this application embodiment can manage warm data in the local storage hard disk. Warm data is data that is accessed infrequently but can be retrieved quickly when needed, with performance requirements between hot and cold data. Warm storage is an intermediate layer in a data hierarchical storage system, used to store warm data. The local storage hard disk acts as a buffer between the node memory and the global shared storage space, and also undertakes the responsibility of intelligent preheating scheduling. Through the local storage hard disk, key-value cache data stored in the global shared storage space can be preheated to the local storage hard disk. Alternatively, through the local storage hard disk, key-value cache data stored in the local storage hard disk can be pre-preset to the node memory. The basic storage design of the local storage hard disk is as follows: (1) The same dual-queue structure is used to manage the key-value cache data on the local storage hard disk; (2) The discarded key-value cache data is written to the global shared storage space for cold storage as a backup for persistence.

[0166] Unlike existing solutions (such as Mooncake), which only support a one-way replacement migration mechanism from hot to warm to cold, this application introduces a two-way replacement-preheating mechanism on each storage node. This not only enables a replacement mechanism in the hot → warm → cold direction but also a preheating mechanism in the cold → warm → hot direction. In the preheating mechanism, the preheat scheduler can proactively load key-value cache data from lower-level storage layers into each storage layer in advance.

[0167] Accordingly, the data processing method provided in this application embodiment may further include: receiving a request prompt word corresponding to a target inference request, and determining a prompt word identifier corresponding to the request prompt word; calculating the identifier overlap length of the historical sequence block identifier corresponding to at least one historical lexical sequence block of a local storage node in the prompt word identifier, wherein the historical lexical sequence block is stored in a global shared storage space or a local storage hard disk of the local storage node; based on the identifier overlap length, selecting a target historical lexical sequence block that overlaps with the prompt word identifier from at least one historical lexical sequence block; selecting target historical key-value cache data corresponding to the target historical lexical sequence block from historical key-value cache data, and updating the storage location of the target historical key-value cache data.

[0168] In this embodiment, a prefix matching prediction method can be used to determine whether a preheating process needs to be initiated. Specifically, upon receiving a new request prompt (i.e., a new request prompt arriving at the peer routing module of the local storage node), the prompt word identifier of the request prompt can be extracted. The prompt word identifier can be understood as the prefix hash value of the request prompt.

[0169] Next, the prompt word identifier can be matched with the historical sequence block identifier corresponding to at least one existing historical lexical sequence block on the local storage node, and the identifier overlap length (i.e., prefix overlap length) between the two can be calculated. Then, based on the identifier overlap length, target historical lexical sequence blocks that overlap with the prompt word identifier are selected from these historical lexical sequence blocks. The number of target historical lexical sequence blocks may be one or more.

[0170] It should be noted that the aforementioned historical word sequence blocks are stored in the global shared storage space or the local storage hard disk of the local storage node. Since the historical word sequence blocks stored in the node memory of the local storage node can be quickly accessed, in this embodiment of the application, it is not necessary to include the word sequence blocks already stored in the node memory within the warm-up scope.

[0171] It is understandable that storing historical word sequence blocks in the global shared storage space or the local storage hard drive of the local storage node means that the historical key-value cache data corresponding to these historical word sequence blocks has also been synchronously stored in the global shared storage space or the local storage hard drive of the local storage node.

[0172] After identifying the target historical lexical sequence block, the target historical key-value cache data corresponding to the target historical lexical sequence block can be filtered out from the historical key-value cache data. Based on this, an asynchronous warm-up process is triggered to preload the target historical key-value cache data into the corresponding storage location on the local storage node.

[0173] The process of preloading the target historical key-value cache data to the corresponding storage location of the local storage node is also the process of updating the storage location of the target historical key-value cache data.

[0174] In this embodiment of the application, selecting target historical word sequence blocks that overlap with the prompt word identifier from at least one historical word sequence block based on the identifier overlap length may include: determining the total identifier length of the prompt word identifier and calculating the ratio of the identifier overlap length to the total identifier length to obtain the overlap length ratio; when the overlap length ratio is greater than a preset length ratio threshold, selecting target historical word sequence blocks that overlap with the prompt word identifier from at least one historical word sequence block.

[0175] The preset length ratio threshold can be set according to actual conditions. For example, the preset length ratio threshold can be 80%, 85%, or other ratios.

[0176] It's important to note that reusing historical key-value cache data requires completely identical prefixes (including spaces, newlines, word boundaries, etc.), not just similarities. For example, with a preset length ratio threshold of 80%, 80% overlap does not mean the last 20% can be reused. For instance, if the prefixes are completely identical up to the 800th historical word sequence block, but different up to the 801st, then the first 800 historical word sequence blocks can be reused.

[0177] Figure 9 This illustration shows the eviction and warm-up process of key-value cached data between different storage layers of a storage node, as provided in an embodiment of this application. Figure 9 As shown, the preheating direction includes: (1) Global shared storage space (cold) → local storage hard disk (warm): The target historical key-value cache data that is about to be accessed is pulled from the remote storage to the local storage hard disk. Specifically, the hard disk history queue is loaded onto the local storage hard disk.

[0178] (2) Local storage hard disk (warm) → node memory (hot): The historical key-value cache data of the target that is predicted to be hit in the local storage hard disk is asynchronously loaded into the node memory. Specifically, the historical data is loaded into the memory queue of the node memory.

[0179] Accordingly, updating the storage location of the target historical key-value cache data can include: when the storage location is the local storage hard disk of the target storage node, loading the target historical key-value cache data into the node memory of the target storage node; when the storage location is the global shared storage space, reading the target historical key-value cache data from the global shared storage space and loading the target historical key-value cache data into the local storage hard disk.

[0180] This application's embodiments leverage the characteristic that a large number of requests share common prefixes (such as system prompts + tool descriptions) in large language model scenarios. By preloading potentially useful target historical key-value cache data when the request prompt arrives, the first token latency (TTFT) is significantly reduced. Furthermore, preloading potentially useful target historical key-value cache data improves the hit rate of subsequent key-value cache data and reduces redundant pre-filling calculations, thereby greatly enhancing the overall inference throughput of the current node cluster.

[0181] As mentioned above, Figure 3 The consistent hashing management & key-value cache migration module for storage nodes is used to manage the mapping from virtual nodes to physical nodes (storage nodes) and to perform minimal data migration when nodes are added or removed.

[0182] Storage nodes in the current node cluster may change at any time. For example, a new storage node can join the current node cluster, or an existing storage node can leave the cluster. After a storage node changes, the hash space is updated accordingly, and similarly, the consistent hash ring is updated accordingly.

[0183] In this regard, the data processing method provided in the embodiments of this application may further include: when receiving change information published by a storage node, taking the storage node that published the change information as the change node, and determining the change type corresponding to the change information; updating the current node cluster according to the change node and the change type to obtain the updated node cluster and the updated hash value space corresponding to the updated node cluster; taking the updated node cluster as the current node cluster, and taking the updated hash value space as the hash value space of the current node cluster.

[0184] Each storage node maintains a local list of cluster members. When any storage node undergoes a change, it can publish the change information to its peer storage nodes. When a local storage node receives the change information, it can identify the publishing storage node as the changed node and determine the change type. The change type can be node exit (node ​​failure) or node joining (new node joining).

[0185] Subsequently, the local storage node can update the current node cluster based on the changed node and change type, resulting in the updated node cluster. Since the current node cluster corresponds to the hash value space, the hash value space is also updated after the current node cluster is updated. Therefore, updating the current node cluster yields the updated node cluster and its corresponding updated hash value space.

[0186] After obtaining the updated node cluster and the updated hash value space, the updated node cluster can be used as the current node cluster, and the updated hash value space can be used as the hash value space of the current node cluster.

[0187] The process involves updating the current node cluster based on the changed node and change type, resulting in an updated node cluster and a corresponding updated hash value space. This can include: when the change type is node exit, identifying the successor node corresponding to the changed node among multiple storage nodes in the current node cluster based on the mapping position of the changed node's node identifier in the hash value space; identifying the first storage interval corresponding to the changed node and the second storage interval corresponding to the successor node in the storage interval of the hash value space; updating the second storage interval based on the first storage interval to obtain the updated second storage interval; deleting the changed node from the current node cluster to obtain the updated node cluster, and using the updated second storage interval as the storage interval corresponding to the successor node to obtain the updated hash value space.

[0188] Figure 10A This diagram illustrates the changes in the consistent hash ring provided in an embodiment of this application when a node exits. Figure 10A As shown, when the change type is node exit, the virtual node data managed by the changed node is taken over by the successor virtual node on the consistent hash ring. If replicas are stored in the global shared storage space, they can be directly marked for reallocation without migration.

[0189] Figure 10B This diagram illustrates the change in the spatial distribution of hash values ​​in the consistent hash ring provided in this embodiment of the application when a node exits. Figure 10B As shown, the example of changing the node to storage node B is used for illustration. Storage node B corresponds to 3 virtual nodes on the consistent hash ring, namely virtual node B0, virtual node B1, and virtual node B2. Among them, the successor node of virtual node B0 is virtual node C0, the successor node of virtual node B1 is virtual node A1, and the successor node of virtual node B2 is virtual node D2.

[0190] The three virtual nodes of storage node B correspond to the following first storage intervals in the hash value space: storage interval 1 (corresponding to virtual node B0), storage interval 4 (corresponding to virtual node B1), and storage interval 10 (corresponding to virtual node B2). The successor node corresponds to the following second storage intervals in the hash value space: storage interval 2 (corresponding to virtual node C0), storage interval 5 (corresponding to virtual node A1), and storage interval 11 (corresponding to virtual node D2).

[0191] Since the change type of the modified node (storage node B) is node exit, storage node B can be deleted from the current node cluster. After storage node B exits the current node cluster, the second storage intervals of the three successor nodes are updated, resulting in their respective updated second storage intervals. For example, the updated second storage interval of virtual node C0 includes storage interval 1 and storage interval 2, the updated second storage interval of virtual node A1 includes storage interval 4 and storage interval 5, and the updated second storage interval of virtual node D2 includes storage interval 10 and storage interval 11.

[0192] After completing the above updates, you will obtain the updated node cluster. For example... Figure 10B As shown, the updated node cluster includes storage node A, storage node C, and storage node D. The storage ranges of the relevant storage nodes (successor nodes) in the updated node cluster are updated, so the updated second storage range can be used as the storage range corresponding to the successor node, thus obtaining the updated hash value space.

[0193] The process involves updating the current node cluster based on the changed node and change type, resulting in an updated node cluster and a corresponding updated hash value space. This can include: when the change type is "node addition," obtaining the node identifier of the changed node and identifying the third storage interval corresponding to the changed node in the hash value space based on the mapping position of the node identifier in the hash value space; selecting at least one fourth storage interval containing the third storage interval from among the multiple storage intervals corresponding to the hash value space, and using the storage node corresponding to the fourth storage interval as the neighbor node of the changed node; adding the changed node to the current node cluster to obtain the updated node cluster; and using the remaining storage intervals in the fourth storage interval (excluding the third storage interval) as the updated storage intervals of the neighbor nodes to obtain the updated hash value space.

[0194] Figure 11A This diagram illustrates the changes in the consistent hash ring provided in this embodiment of the application as a node is added. Figure 11A As shown, when the change type is node joining, the affected 1 / N data shards can be asynchronously migrated from adjacent nodes, during which the old nodes can still continue to provide cached data reading services.

[0195] Figure 11B This diagram illustrates the change in the spatial distribution of hash values ​​in the consistent hash ring provided in this embodiment of the application when a node is added. Figure 11BAs shown, the explanation will be based on the example of changing the node to storage node E. Storage node E includes three virtual nodes on the consistent hash ring: virtual node E0, virtual node E1, and virtual node E2. The third storage interval corresponding to storage node E in the hash value space includes storage interval 1a (corresponding to virtual node E0), storage interval 3a (corresponding to virtual node E1), and storage interval 8a (corresponding to virtual node E2).

[0196] Before storage node E joins the current node cluster, the hash space comprises nine storage intervals. Storage interval 1a falls within storage interval 1, storage interval 3a falls within storage interval 3, and storage interval 8a falls within storage interval 8. Therefore, the fourth storage interval, containing the third storage interval, includes storage intervals 1, 3, and 8. Storage interval 1 corresponds to virtual node C0, storage interval 3 corresponds to virtual node A1, and storage interval 8 corresponds to virtual node D2. Correspondingly, the neighboring nodes of the changed node (storage node E) include virtual nodes C0, A1, and D2.

[0197] Adding storage node E to the current node cluster will result in an updated node cluster. The updated node cluster includes storage node A, storage node C, storage node D, and storage node E.

[0198] Furthermore, the storage ranges of related neighboring nodes in the updated node cluster are updated, resulting in the updated hash value space. For example, the storage range of virtual node C0 changes from storage range 1 to storage range 1b (storage range 1b is the remaining storage range in storage range 1 excluding storage range 1a). Similarly, the storage range of virtual node A1 changes from storage range 3 to storage range 3b, and the storage range of virtual node D2 changes from storage range 8 to storage range 8b.

[0199] In this embodiment, a copy-on-read strategy can be used to implement data migration. Copy-on-read is a strategy used in distributed caching systems to ensure data consistency, especially suitable for concurrent access scenarios to cached objects in multi-threaded or distributed environments. Its core idea is: when reading a cached object, a copy of the object is automatically created and returned to the caller, rather than directly returning the original reference, thereby preventing accidental modification of the original data in the cache by external code.

[0200] Figure 12 A schematic diagram of the read-time copy process is shown. Figure 12As shown, when reading key-value cache data that has not yet been migrated, it can be fetched from the original storage node and cached on the new storage node for the first time, avoiding blocking during the full migration. Subsequent cache data read requests with the same hash will directly hit the new storage node, without needing to fetch it from the original storage node again.

[0201] The data processing method provided in this application supports the dynamic addition and removal of storage nodes, and can linearly expand cache capacity. When adding storage nodes, only 1 / N of the data is redistributed, and the expansion has no impact on the service. Furthermore, the migration process is performed on demand, without the need for a full migration that blocks traffic, and is transparent to online requests.

[0202] As can be seen from the foregoing, the data processing method provided in this application embodiment can solve the problems of "where to store" and "how to share" key-value cache data, and achieve the technical effects of capacity expansion and high availability. Figure 13 This illustration shows a product-side application scenario diagram of the data processing method provided in an embodiment of this application. For example... Figure 13 As shown, the data processing method provided in this application embodiment can be applied to the following scenarios: (1) Multi-turn dialogue service: Users engage in multi-turn dialogues with the large language model, generating a large amount of common context between sessions. Key-value cache data (KV Cache) can be persisted in three-level storage, and session recovery does not require re-population.

[0203] (2) Retrieval Enhanced Generation (RAG) Knowledge Q&A: In the retrieval enhanced generation scenario, the same document block is referenced by multiple requests. Key-value cache data is automatically shared and reused in the cluster.

[0204] (3) Agent tool call: The AI ​​agent repeatedly calls the same tool description. The key-value cache data described by the tool can reside in the memory cache queue of the node memory or the hard disk cache queue of the local storage hard disk, so that hot data is not eliminated.

[0205] (4) Prefill and Decode Separation Architecture: Prefill and Decode are deployed on different nodes respectively. The data processing method provided in this application embodiment is responsible for the storage and transmission of key-value cache data as an intermediate layer.

[0206] (5) Flexible scaling: Dynamically add or remove inference nodes (storage nodes) during peak / off-peak periods. The masterless architecture supports storage nodes to be added or removed at any time without downtime.

[0207] (6) Disaster recovery and fault tolerance: After the storage node fails, the service is restored. The global shared storage space ensures the persistence of key-value cache data and automatically recovers from other replicas / storage.

[0208] As can be seen from the above, this embodiment of the application receives a key-value caching request for a target word sequence block. The key-value caching request carries the sequence block identifier of the target word sequence block. Then, the sequence block identifier is mapped to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes storage intervals corresponding to multiple virtual nodes of the storage nodes. Then, the target storage interval corresponding to the target mapping position is selected from the storage intervals, and the target virtual node corresponding to the target storage interval is determined. Then, based on the target virtual node, the target storage node corresponding to the target word sequence block is selected from the multiple storage nodes. Finally, the target word sequence block is processed based on the request type of the key-value caching request and the target storage node.

[0209] Because this solution can build a distributed key-value cache management system without centralized metadata services based on the current node cluster (i.e., a masterless distributed key-value cache management system), in a masterless distributed key-value cache management system, each storage node is an equal "peer point", and each storage node has the dual role of service provider and acquirer, realizing resource sharing and load balancing.

[0210] Each storage node can maintain a local list of cluster members for building the consistent hashing ring. Furthermore, each storage node can be mapped to at least one virtual node, which is evenly distributed across the consistent hashing ring, dividing the hash value space corresponding to the consistent hashing ring into multiple storage regions.

[0211] When a key-value cache request for a target lexical sequence block is received, decentralized addressing can be quickly achieved through the aforementioned masterless distributed key-value cache management system. This means that in a masterless network environment, the target lexical sequence block's target mapping position in the hash value space can be quickly and accurately located using the sequence block identifier. Subsequently, the target virtual node can be determined based on the target mapping position, and then the target storage node corresponding to the target lexical sequence block can be determined based on the correspondence between the virtual node and the storage node, thus achieving a deterministic mapping from the target lexical sequence block to the storage node. Furthermore, the target lexical sequence block can be processed accordingly based on the request type of the key-value cache request.

[0212] Therefore, the above solution eliminates centralized metadata services, avoiding single points of failure and scalability bottlenecks. With the support of a masterless distributed key-value cache management system, the target storage node corresponding to the target lexical sequence block can be accurately and quickly located, enabling efficient utilization of the target key-value cache data corresponding to the target lexical sequence block, thereby improving the computational efficiency of large model inference processes.

[0213] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0214] In this embodiment, the data processing device will be specifically integrated into an electronic device, which will be a server, as an example for explanation. Figure 14 Another flowchart illustrating the data processing method provided in an embodiment of this application is shown. Figure 14 As shown, a data processing method has the following specific steps: 201. The server receives a key-value cache request for the target lexical sequence block, and the key-value cache request carries the sequence block identifier of the target lexical sequence block.

[0215] For example, the server can obtain the sequence prefix corresponding to the lexical sequence contained in the target lexical sequence block; perform hash calculation on the sequence prefix to obtain the target hash value of the target lexical sequence block; and use the target hash value as the sequence block identifier of the target lexical sequence block.

[0216] 202. The server maps the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes the storage range corresponding to multiple virtual nodes of the storage nodes.

[0217] For example, the server can determine the number of virtual nodes, which indicates the number of storage intervals in the hash value space; perform a modulo operation on the sequence block identifier based on the number of virtual nodes to obtain the modulo operation result; and determine the target mapping position of the target word sequence block in the hash value space based on the modulo operation result.

[0218] For example, when receiving change information published by a storage node, the server can take the storage node that published the change information as the change node and determine the change type corresponding to the change information; based on the change node and the change type, the current node cluster is updated to obtain the updated node cluster and the updated hash value space corresponding to the updated node cluster; the updated node cluster is taken as the current node cluster, and the updated hash value space is taken as the hash value space of the current node cluster.

[0219] For example, when the change type is node exit, the server can identify the successor node corresponding to the changed node among multiple storage nodes in the current node cluster based on the mapping position of the changed node's node identifier in the hash value space; identify the first storage interval corresponding to the changed node and the second storage interval corresponding to the successor node in the storage interval of the hash value space; update the second storage interval based on the first storage interval to obtain the updated second storage interval; delete the changed node in the current node cluster to obtain the updated node cluster, and use the updated second storage interval as the storage interval corresponding to the successor node to obtain the updated hash value space.

[0220] For example, when the change type is node addition, the server can obtain the node identifier of the changed node and identify the third storage interval corresponding to the changed node in the hash value space based on the mapping position of the node identifier in the hash value space; filter out at least one fourth storage interval containing the third storage interval from the multiple storage intervals corresponding to the hash value space, and take the storage node corresponding to the fourth storage interval as the neighbor node of the changed node; add the changed node to the current node cluster to obtain the updated node cluster; take the remaining storage intervals in the fourth storage interval other than the third storage interval as the updated storage intervals of the neighbor nodes to obtain the updated hash value space.

[0221] 203. The server filters out the target storage range corresponding to the target mapping location in the storage range and determines the target virtual node corresponding to the target storage range.

[0222] 204. Based on the target virtual node, the server selects the target storage node corresponding to the target word sequence block from multiple storage nodes.

[0223] 205. The server processes the target word sequence block based on the request type and target storage node of the key-value cache request.

[0224] For example, when the target storage node is a local storage node, the server can process the target word sequence block through the target storage node according to the request type; when the target storage node is not a local storage node, the server can send the key-value cache request to the target storage node so that the target storage node can process the target word sequence block according to the request type.

[0225] For example, the request type could include a key-value cache read request. The server can query the target key-value cache data corresponding to the target lexical sequence block in the node memory of the target storage node; if the target key-value cache data exists in the node memory, it can read the target key-value cache data from the node memory.

[0226] For example, node memory can include a memory history queue and a memory cache queue. The memory history queue is used to store key-value cache data that enters the node memory, and the memory cache queue is used to store key-value cache data that is called in the memory history queue.

[0227] When the target key-value cache data is in the memory cache queue, the server can move the target key-value cache data to the tail of the memory cache queue; when the target key-value cache data is in the memory history queue, the server can load the target key-value cache data from the memory history queue to the tail of the memory cache queue.

[0228] For example, if the target key-value cache data does not exist in the node's memory, the server can query the target key-value cache data in the target storage node's local storage hard drive; if the target key-value cache data exists in the local storage hard drive, the server can read the target key-value cache data from the local storage hard drive.

[0229] For example, a local storage hard drive includes a hard drive history queue and a hard drive cache queue. The hard drive history queue is used to store key-value cache data that enters the local storage hard drive, and the hard drive cache queue is used to store key-value cache data that is accessed in the hard drive history queue.

[0230] When the target key-value cache data is in the disk cache queue, the server can load the target key-value cache data to the tail of the memory history queue in the node memory; when the target key-value cache data is in the disk history queue, the server can load the target key-value cache data from the disk history queue to the tail of the disk cache queue.

[0231] For example, when the target key-value cache data does not exist in the local storage hard drive, the server can query the target key-value cache data from the global shared storage space corresponding to the current node cluster through the global shared query interface of the target storage node; when the target key-value cache data exists in the global shared storage space, the target key-value cache data is read from the global shared storage space; and the target key-value cache data is loaded into the tail of the hard drive history queue of the local storage hard drive.

[0232] For example, when the target key-value cache data does not exist in the global shared storage space, the server can calculate the key-value data corresponding to the target lexical sequence block; write the key-value data to the tail of the memory history queue of the target storage node, and copy the key-value data to obtain backup key-value data; and store the backup key-value data in the global shared storage space.

[0233] For example, the request type may include a key-value cache write request, which includes the target key-value cache data corresponding to the target lexical sequence block.

[0234] The server can determine the memory history queue load information in the memory memory of the target storage node at the current moment; when the memory history queue load information indicates that the memory history queue has reached its capacity limit, the key-value cache data located at the head of the memory history queue is migrated to the local storage hard disk of the target storage node, and the target key-value cache data is written to the tail of the memory history queue; when the memory history queue load information indicates that the memory history queue has not reached its capacity limit, the target key-value cache data is written to the tail of the memory history queue.

[0235] For example, the server can determine the disk history queue load information of the local storage disk of the target storage node at the current moment; when the disk history queue load information indicates that the disk history queue has reached its capacity limit, the key-value cache data at the head of the disk history queue is migrated to the global shared storage space, and the key-value cache data at the head of the memory history queue is migrated to the tail of the disk history queue; when the disk history queue load information indicates that the disk history queue has not reached its capacity limit, the key-value cache data at the head of the memory history queue is migrated to the tail of the disk history queue.

[0236] 206. The server receives the request prompt word corresponding to the target inference request and determines the prompt word identifier corresponding to the request prompt word.

[0237] 207. The server calculates the overlap length of the historical sequence block identifier corresponding to at least one historical word sequence block of the local storage node in the prompt word identifier. The historical word sequence block is stored in the global shared storage space or the local storage hard disk of the local storage node.

[0238] 208. Based on the overlap length of the identifier, the server filters out the target historical word sequence block that overlaps with the prompt word identifier from at least one historical word sequence block.

[0239] For example, the server can determine the total length of the prompt word identifier and calculate the ratio of the overlap length of the identifier to the total length of the identifier to obtain the overlap length ratio; when the overlap length ratio is greater than a preset length ratio threshold, the server can filter out the target historical word sequence block that overlaps with the prompt word identifier from at least one historical word sequence block.

[0240] 209. The server filters out the target historical key-value cache data corresponding to the target historical word sequence block from the historical key-value cache data, and updates the storage location of the target historical key-value cache data.

[0241] For example, when the storage location is the local storage hard drive of the target storage node, the server can load the target historical key-value cache data into the node memory of the target storage node; when the storage location is the global shared storage space, the server can read the target historical key-value cache data from the global shared storage space and load the target historical key-value cache data into the local storage hard drive.

[0242] As can be seen from the above, the embodiments of this application can receive key-value caching requests for target lexical sequence blocks, the key-value caching request carrying the sequence block identifier of the target lexical sequence block; map the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target lexical sequence block, the current node cluster includes multiple storage nodes, and the hash value space includes storage intervals corresponding to multiple virtual nodes of the storage nodes; filter out the target storage interval corresponding to the target mapping position in the storage interval, and determine the target virtual node corresponding to the target storage interval; based on the target virtual node, filter out the target storage node corresponding to the target lexical sequence block from multiple storage nodes; based on the request type and target of the key-value caching request... The storage node processes the target lexical sequence block; receives the request prompt word corresponding to the target inference request and determines the prompt word identifier corresponding to the request prompt word; calculates the identifier overlap length of the historical sequence block identifier corresponding to at least one historical lexical sequence block of the local storage node in the prompt word identifier, and the historical lexical sequence block is stored in the global shared storage space or the local storage hard disk of the local storage node; based on the identifier overlap length, it filters out the target historical lexical sequence block that overlaps with the prompt word identifier from at least one historical lexical sequence block; it filters out the target historical key-value cache data corresponding to the target historical lexical sequence block from the historical key-value cache data, and updates the storage location of the target historical key-value cache data.

[0243] This solution can build a distributed key-value cache management system (i.e., a masterless distributed key-value cache management system) based on the current node cluster, eliminating the need for a centralized metadata service. In this system, each storage node is an equal "peer," acting as both a service provider and a service seeker, thus achieving resource sharing and load balancing. Each storage node can maintain a local list of cluster members for building the consistent hashing ring. Furthermore, each storage node can be mapped to at least one virtual node, which is evenly distributed across the consistent hashing ring, dividing the hash value space corresponding to the ring into multiple storage regions.

[0244] When a key-value cache request for a target lexical sequence block is received, decentralized addressing can be quickly achieved through the aforementioned masterless distributed key-value cache management system. This means that in a masterless network environment, the target lexical sequence block's target mapping position in the hash value space can be quickly and accurately located using the sequence block identifier. Subsequently, the target virtual node can be determined based on the target mapping position, and then the target storage node corresponding to the target lexical sequence block can be determined based on the correspondence between the virtual node and the storage node, thus achieving a deterministic mapping from the target lexical sequence block to the storage node. Furthermore, the target lexical sequence block can be processed accordingly based on the request type of the key-value cache request.

[0245] Therefore, the above solution eliminates centralized metadata services, avoiding single points of failure and scalability bottlenecks. With the support of a masterless distributed key-value cache management system, the target storage node corresponding to the target lexical sequence block can be accurately and quickly located, enabling efficient utilization of the target key-value cache data corresponding to the target lexical sequence block, thereby improving the computational efficiency of large model inference processes.

[0246] Furthermore, this solution leverages the characteristic of a large number of requests sharing common prefixes (such as system prompts + tool descriptions) in large language model scenarios. It preloads potentially needed target historical key-value cache data as soon as the request prompt arrives, significantly reducing the initial token latency. By preloading potentially needed target historical key-value cache data, the hit rate of subsequent key-value cache data is improved, and redundant pre-filling calculations are reduced, thereby greatly enhancing the overall inference throughput of the current node cluster.

[0247] To better implement the above methods, this application also provides a data processing device that can be integrated into a network device, such as a server or terminal. The terminal may include a tablet computer, a laptop computer, and / or a personal computer.

[0248] Figure 15 A schematic diagram of the structure of the data processing apparatus provided in an embodiment of this application is shown. Figure 15 As shown, the data processing device may include a request receiving unit 301, a hash mapping unit 302, an interval filtering unit 303, a node positioning unit 304, and a data processing unit 305, as follows: (1) Request receiving unit 301; The request receiving unit 301 is used to receive a key-value cache request for a target lexical sequence block, wherein the key-value cache request carries the sequence block identifier of the target lexical sequence block.

[0249] For example, the request receiving unit 301 can be used to obtain the sequence prefix corresponding to the lexical sequence contained in the target lexical sequence block; perform hash calculation on the sequence prefix to obtain the target hash value of the target lexical sequence block; and use the target hash value as the sequence block identifier of the target lexical sequence block.

[0250] (2) Hash mapping unit 302; The hash mapping unit 302 is used to map the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes the storage intervals corresponding to multiple virtual nodes of the storage nodes.

[0251] For example, the hash mapping unit 302 can be used to determine the number of virtual nodes of the virtual node, the number of virtual nodes indicating the number of storage intervals in the hash value space; perform a modulo operation on the sequence block identifier according to the number of virtual nodes to obtain the modulo operation result; and determine the target mapping position of the target word sequence block in the hash value space based on the modulo operation result.

[0252] For example, the hash mapping unit 302 can be used to, when receiving change information published by a storage node, take the storage node that published the change information as the change node and determine the change type corresponding to the change information; update the current node cluster according to the change node and the change type to obtain the updated node cluster and the updated hash value space corresponding to the updated node cluster; take the updated node cluster as the current node cluster and the updated hash value space as the hash value space of the current node cluster.

[0253] For example, the hash mapping unit 302 can be used to identify the successor node corresponding to the changed node among multiple storage nodes in the current node cluster based on the mapping position of the node identifier of the changed node in the hash value space when the change type is node exit; identify the first storage interval corresponding to the changed node and the second storage interval corresponding to the successor node in the storage interval of the hash value space; update the second storage interval according to the first storage interval to obtain the updated second storage interval; delete the changed node in the current node cluster to obtain the updated node cluster, and use the updated second storage interval as the storage interval corresponding to the successor node to obtain the updated hash value space.

[0254] For example, the hash mapping unit 302 can be used to obtain the node identifier of the changed node when the change type is node addition, and identify the third storage interval corresponding to the changed node in the hash value space according to the mapping position of the node identifier in the hash value space; filter at least one fourth storage interval where the third storage interval is located from multiple storage intervals corresponding to the hash value space, and take the storage node corresponding to the fourth storage interval as the neighbor node of the changed node; add the changed node to the current node cluster to obtain the updated node cluster; and take the remaining storage intervals in the fourth storage interval other than the third storage interval as the updated storage intervals of the neighbor nodes to obtain the updated hash value space.

[0255] (3) Interval filtering unit 303; The interval filtering unit 303 is used to filter out the target storage interval corresponding to the target mapping position in the storage interval, and determine the target virtual node corresponding to the target storage interval.

[0256] (4) Node positioning unit 304; The node positioning unit 304 is used to select the target storage node corresponding to the target word sequence block from multiple storage nodes based on the target virtual node.

[0257] (5) Data processing unit 305.

[0258] The data processing unit 305 is used to process the target word sequence block based on the request type and target storage node of the key-value cache request.

[0259] For example, the data processing unit 305 can be used to process the target word sequence block through the target storage node according to the request type when the target storage node is a local storage node; and to send the key-value cache request to the target storage node when the target storage node is a non-local storage node, so that the target storage node can process the target word sequence block according to the request type.

[0260] For example, the request type may include a key-value cache read request. Based on this, the data processing unit 305 can specifically be used to query the target key-value cache data corresponding to the target lexical sequence block in the node memory of the target storage node; when the target key-value cache data exists in the node memory, it reads the target key-value cache data from the node memory.

[0261] For example, the node memory includes a memory history queue and a memory cache queue. The memory history queue stores key-value cache data entering the node memory, and the memory cache queue stores key-value cache data that is accessed in the memory history queue. Based on this, the data processing unit 305 can specifically be used to: when the target key-value cache data is in the memory cache queue, move the target key-value cache data to the tail of the memory cache queue; and when the target key-value cache data is in the memory history queue, load the target key-value cache data from the memory history queue to the tail of the memory cache queue.

[0262] For example, the data processing unit 305 can be used to query the target key-value cache data in the local storage hard disk of the target storage node when the target key-value cache data does not exist in the node memory; and to read the target key-value cache data from the local storage hard disk when the target key-value cache data exists in the local storage hard disk.

[0263] For example, the local storage hard disk includes a hard disk history queue and a hard disk cache queue. The hard disk history queue stores key-value cache data entering the local storage hard disk, and the hard disk cache queue stores key-value cache data that is called from the hard disk history queue. Based on this, the data processing unit 305 can specifically be used to load the target key-value cache data to the tail of the memory history queue in the node memory when the target key-value cache data is in the hard disk cache queue; and to load the target key-value cache data from the hard disk history queue to the tail of the hard disk cache queue when the target key-value cache data is in the hard disk history queue.

[0264] For example, the data processing unit 305 can be used to query the target key-value cache data from the global shared storage space corresponding to the current node cluster through the global shared query interface of the target storage node when the target key-value cache data does not exist in the local storage hard disk; when the target key-value cache data exists in the global shared storage space, it can read the target key-value cache data from the global shared storage space; and load the target key-value cache data to the tail of the hard disk history queue of the local storage hard disk.

[0265] For example, the data processing unit 305 can be used to calculate the key value data corresponding to the target word sequence block when there is no target key value cache data in the global shared storage space; write the key value data to the tail of the memory history queue of the target storage node, and copy the key value data to obtain backup key value data; and store the backup key value data in the global shared storage space.

[0266] In addition, the data processing unit 305 can also be used to receive the request prompt word corresponding to the target inference request and determine the prompt word identifier corresponding to the request prompt word; calculate the identifier overlap length of the historical sequence block identifier corresponding to at least one historical word sequence block of the local storage node in the prompt word identifier, wherein the historical word sequence block is stored in the global shared storage space or the local storage hard disk of the local storage node; based on the identifier overlap length, select the target historical word sequence block that overlaps with the prompt word identifier from at least one historical word sequence block; select the target historical key-value cache data corresponding to the target historical word sequence block from the historical key-value cache data, and update the storage location of the target historical key-value cache data.

[0267] For example, the data processing unit 305 can be used to determine the total length of the prompt word identifier and calculate the ratio of the overlap length of the identifier to the total length of the identifier to obtain the overlap length ratio; when the overlap length ratio is greater than the preset length ratio threshold, the target historical word sequence block that overlaps with the prompt word identifier is selected from at least one historical word sequence block.

[0268] For example, the data processing unit 305 can be used to load the target historical key-value cache data into the node memory of the target storage node when the storage location is the local storage hard disk of the target storage node; and to read the target historical key-value cache data from the global shared storage space and load the target historical key-value cache data into the local storage hard disk when the storage location is the global shared storage space.

[0269] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0270] In this embodiment, the request receiving unit 301 receives a key-value cache request for a target word sequence block. The key-value cache request carries the sequence block identifier of the target word sequence block. Then, the hash mapping unit 302 maps the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes storage intervals corresponding to multiple virtual nodes of the storage nodes. After that, the interval filtering unit 303 filters out the target storage interval corresponding to the target mapping position in the storage interval and determines the target virtual node corresponding to the target storage interval. The node positioning unit 304 then filters out the target storage node corresponding to the target word sequence block from multiple storage nodes based on the target virtual node. Finally, the data processing unit 305 processes the target word sequence block based on the request type of the key-value cache request and the target storage node.

[0271] Because this solution can build a distributed key-value cache management system without centralized metadata services based on the current node cluster (i.e., a masterless distributed key-value cache management system), in a masterless distributed key-value cache management system, each storage node is an equal "peer point", and each storage node has the dual role of service provider and acquirer, realizing resource sharing and load balancing.

[0272] Each storage node can maintain a local list of cluster members for building the consistent hashing ring. Furthermore, each storage node can be mapped to at least one virtual node, which is evenly distributed across the consistent hashing ring, dividing the hash value space corresponding to the consistent hashing ring into multiple storage regions.

[0273] When a key-value cache request for a target lexical sequence block is received, decentralized addressing can be quickly achieved through the aforementioned masterless distributed key-value cache management system. This means that in a masterless network environment, the target lexical sequence block's target mapping position in the hash value space can be quickly and accurately located using the sequence block identifier. Subsequently, the target virtual node can be determined based on the target mapping position, and then the target storage node corresponding to the target lexical sequence block can be determined based on the correspondence between the virtual node and the storage node, thus achieving a deterministic mapping from the target lexical sequence block to the storage node. Furthermore, the target lexical sequence block can be processed accordingly based on the request type of the key-value cache request.

[0274] Therefore, the above solution eliminates centralized metadata services, avoiding single points of failure and scalability bottlenecks. With the support of a masterless distributed key-value cache management system, the target storage node corresponding to the target lexical sequence block can be accurately and quickly located, enabling efficient utilization of the target key-value cache data corresponding to the target lexical sequence block, thereby improving the computational efficiency of large model inference processes.

[0275] This application also provides an electronic device, such as... Figure 16 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 16 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0276] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Correspondingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0277] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0278] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0279] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows: The system receives a key-value cache request for a target lexical sequence block, which carries the sequence block identifier of the target lexical sequence block. It maps the sequence block identifier to the hash space corresponding to the current node cluster to obtain the target mapping location of the target lexical sequence block. The current node cluster includes multiple storage nodes, and the hash space includes storage intervals corresponding to multiple virtual nodes of the storage nodes. It then filters out the target storage interval corresponding to the target mapping location from the storage intervals and determines the target virtual node corresponding to the target storage interval. Based on the target virtual node, it filters out the target storage node corresponding to the target lexical sequence block from the multiple storage nodes. Finally, it processes the target lexical sequence block based on the request type of the key-value cache request and the target storage node.

[0280] For example, an electronic device can receive a key-value cache request for a target lexical sequence block, the key-value cache request carrying the sequence block identifier of the target lexical sequence block; map the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping location of the target lexical sequence block, the current node cluster including multiple storage nodes, the hash value space including storage intervals corresponding to multiple virtual nodes of the storage nodes; filter out the target storage interval corresponding to the target mapping location from the storage intervals, and determine the target virtual node corresponding to the target storage interval; based on the target virtual node, filter out the target storage node corresponding to the target lexical sequence block from multiple storage nodes; and based on the request type of the key-value cache request and the target storage node... The system processes the target word sequence block; receives the request prompt word corresponding to the target inference request and determines the prompt word identifier corresponding to the request prompt word; calculates the identifier overlap length of the historical sequence block identifier corresponding to at least one historical word sequence block of the local storage node in the prompt word identifier, and stores the historical word sequence block in the global shared storage space or the local storage hard disk of the local storage node; based on the identifier overlap length, it filters out the target historical word sequence block that overlaps with the prompt word identifier from at least one historical word sequence block; it filters out the target historical key-value cache data corresponding to the target historical word sequence block from the historical key-value cache data and updates the storage location of the target historical key-value cache data, etc.

[0281] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0282] As can be seen from the above, this embodiment of the application receives a key-value caching request for a target word sequence block. The key-value caching request carries the sequence block identifier of the target word sequence block. Then, the sequence block identifier is mapped to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes storage intervals corresponding to multiple virtual nodes of the storage nodes. Then, the target storage interval corresponding to the target mapping position is selected from the storage intervals, and the target virtual node corresponding to the target storage interval is determined. Then, based on the target virtual node, the target storage node corresponding to the target word sequence block is selected from the multiple storage nodes. Finally, the target word sequence block is processed based on the request type of the key-value caching request and the target storage node.

[0283] Because this solution can build a distributed key-value cache management system without centralized metadata services based on the current node cluster (i.e., a masterless distributed key-value cache management system), in a masterless distributed key-value cache management system, each storage node is an equal "peer point", and each storage node has the dual role of service provider and acquirer, realizing resource sharing and load balancing.

[0284] Each storage node can maintain a local list of cluster members for building the consistent hashing ring. Furthermore, each storage node can be mapped to at least one virtual node, which is evenly distributed across the consistent hashing ring, dividing the hash value space corresponding to the consistent hashing ring into multiple storage regions.

[0285] When a key-value cache request for a target lexical sequence block is received, decentralized addressing can be quickly achieved through the aforementioned masterless distributed key-value cache management system. This means that in a masterless network environment, the target lexical sequence block's target mapping position in the hash value space can be quickly and accurately located using the sequence block identifier. Subsequently, the target virtual node can be determined based on the target mapping position, and then the target storage node corresponding to the target lexical sequence block can be determined based on the correspondence between the virtual node and the storage node, thus achieving a deterministic mapping from the target lexical sequence block to the storage node. Furthermore, the target lexical sequence block can be processed accordingly based on the request type of the key-value cache request.

[0286] Therefore, the above solution eliminates centralized metadata services, avoiding single points of failure and scalability bottlenecks. With the support of a masterless distributed key-value cache management system, the target storage node corresponding to the target lexical sequence block can be accurately and quickly located, enabling efficient utilization of the target key-value cache data corresponding to the target lexical sequence block, thereby improving the computational efficiency of large model inference processes.

[0287] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0288] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the data processing methods provided in embodiments of this application. For example, the instructions can execute the following steps: The system receives a key-value cache request for a target lexical sequence block, which carries the sequence block identifier of the target lexical sequence block. It maps the sequence block identifier to the hash space corresponding to the current node cluster to obtain the target mapping location of the target lexical sequence block. The current node cluster includes multiple storage nodes, and the hash space includes storage intervals corresponding to multiple virtual nodes of the storage nodes. It then filters out the target storage interval corresponding to the target mapping location from the storage intervals and determines the target virtual node corresponding to the target storage interval. Based on the target virtual node, it filters out the target storage node corresponding to the target lexical sequence block from the multiple storage nodes. Finally, it processes the target lexical sequence block based on the request type of the key-value cache request and the target storage node.

[0289] For example, a key-value cache request for a target lexical sequence block is received, carrying the sequence block identifier of the target lexical sequence block; the sequence block identifier is mapped to the hash space corresponding to the current node cluster to obtain the target mapping location of the target lexical sequence block. The current node cluster includes multiple storage nodes, and the hash space includes storage intervals corresponding to multiple virtual nodes of the storage nodes; the target storage interval corresponding to the target mapping location is selected from the storage intervals, and the target virtual node corresponding to the target storage interval is determined; based on the target virtual node, the target storage node corresponding to the target lexical sequence block is selected from the multiple storage nodes; based on the request type of the key-value cache request and the target storage node, the target lexical sequence block is cached. The process involves: processing token sequence blocks; receiving the request prompt word corresponding to the target inference request and determining the prompt word identifier corresponding to the request prompt word; calculating the identifier overlap length of the historical sequence block identifier corresponding to at least one historical token sequence block of the local storage node in the prompt word identifier, with the historical token sequence blocks stored in the global shared storage space or the local storage hard disk of the local storage node; filtering out target historical token sequence blocks that overlap with the prompt word identifier from at least one historical token sequence block based on the identifier overlap length; filtering out target historical key-value cache data corresponding to the target historical token sequence block from the historical key-value cache data and updating the storage location of the target historical key-value cache data, etc.

[0290] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0291] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0292] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0293] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in various alternative implementations of the data processing method described above.

[0294] The data processing method and related equipment provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing method, characterized in that, include: Receive a key-value cache request for a target lexical sequence block, the key-value cache request carrying the sequence block identifier of the target lexical sequence block, and the request type of the key-value cache request includes a key-value cache read request; The sequence block identifier is mapped to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes the storage intervals corresponding to multiple virtual nodes of the storage nodes. Filter out the target storage interval corresponding to the target mapping location in the storage interval, and determine the target virtual node corresponding to the target storage interval; Based on the target virtual node, the target storage node corresponding to the target word sequence block is selected from the plurality of storage nodes; When the target storage node is a local storage node, the target key value cache data corresponding to the target word sequence block is queried in the node memory of the target storage node; When the target key-value cache data exists in the node's memory, the target key-value cache data is read from the node's memory.

2. The data processing method according to claim 1, characterized in that, After selecting the target storage node corresponding to the target word sequence block from the plurality of storage nodes based on the target virtual node, the method further includes: When the target storage node is not a local storage node, the key-value cache request is sent to the target storage node so that the target storage node can process the target lexical sequence block based on the request type.

3. The data processing method according to claim 1, characterized in that, The node memory includes a memory history queue and a memory cache queue. The memory history queue stores key-value cache data that enters the node memory, and the memory cache queue stores key-value cache data that is accessed from the memory history queue. After reading the target key-value cache data from the node memory, the method further includes: When the target key-value cache data is located in the memory cache queue, the target key-value cache data is moved to the tail of the memory cache queue; When the target key-value cache data is located in the memory history queue, the target key-value cache data is loaded from the memory history queue to the tail of the memory cache queue.

4. The data processing method according to claim 1, characterized in that, After querying the target key-value cache data corresponding to the target lexical sequence block in the node memory of the target storage node, the method further includes: When the target key-value cache data does not exist in the node's memory, the target key-value cache data is queried from the local storage hard disk of the target storage node; When the target key-value cache data exists in the local storage hard disk, the target key-value cache data is read from the local storage hard disk.

5. The data processing method according to claim 4, characterized in that, The local storage hard disk includes a hard disk history queue and a hard disk cache queue. The hard disk history queue is used to store key-value cache data entering the local storage hard disk, and the hard disk cache queue is used to store key-value cache data that is accessed in the hard disk history queue. After reading the target key-value cache data from the local storage hard disk, the method further includes: When the target key-value cache data is located in the disk cache queue, the target key-value cache data is loaded into the tail of the memory history queue of the node memory; When the target key-value cache data is located in the hard disk history queue, the target key-value cache data is loaded from the hard disk history queue to the tail of the hard disk cache queue.

6. The data processing method according to claim 4, characterized in that, After querying the target key-value cache data in the local storage hard disk of the target storage node, the process further includes: When the target key-value cache data is not present in the local storage hard disk, the target key-value cache data is queried from the global shared storage space corresponding to the current node cluster through the global shared query interface of the target storage node. When the target key-value cache data exists in the global shared storage space, the target key-value cache data is read from the global shared storage space; The target key-value cache data is loaded into the tail of the hard disk history queue of the local storage hard disk.

7. The data processing method according to claim 6, characterized in that, After querying the target key-value cache data from the global shared storage space corresponding to the current node cluster through the global shared query interface of the target storage node, the method further includes: When the target key-value cache data does not exist in the global shared storage space, calculate the key-value data corresponding to the target word sequence block; The key-value data is written to the tail of the memory history queue of the target storage node, and the key-value data is copied to obtain backup key-value data. The backup key-value data is stored in the global shared storage space.

8. The data processing method according to claim 1, characterized in that, The request type includes a key-value cache write request, and the key-value cache request includes the target key-value cache data corresponding to the target lexical sequence block. After selecting the target storage node corresponding to the target word sequence block from the plurality of storage nodes based on the target virtual node, the method further includes: Determine the memory history queue load information of the target storage node at the current moment. When the memory history queue load information indicates that the memory history queue has reached its capacity limit, the key-value cache data located at the head of the memory history queue is migrated to the local storage hard disk of the target storage node, and the target key-value cache data is written to the tail of the memory history queue. When the memory history queue load information indicates that the memory history queue has not reached its capacity limit, the target key-value cache data is written to the tail of the memory history queue.

9. The data processing method according to claim 8, characterized in that, The step of migrating the key-value cache data located at the head of the memory history queue to the local storage hard disk of the target storage node includes: Determine the disk history queue load information of the local storage disk of the target storage node at the current moment; When the hard disk history queue load information indicates that the hard disk history queue has reached its capacity limit, the key-value cache data located at the head of the hard disk history queue is migrated to the global shared storage space, and the key-value cache data located at the head of the memory history queue is migrated to the tail of the hard disk history queue. When the hard disk history queue load information indicates that the hard disk history queue has not reached its capacity limit, the key-value cache data at the head of the memory history queue is migrated to the tail of the hard disk history queue.

10. The data processing method according to claim 1, characterized in that, The step of mapping the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block includes: Determine the number of virtual nodes, where the number of virtual nodes indicates the number of intervals in the storage interval within the hash value space; The sequence block identifier is moduloed based on the number of virtual nodes to obtain the modulo operation result. Based on the modulo operation result, the target mapping position of the target word sequence block in the hash value space is determined.

11. The data processing method according to claim 1, characterized in that, Also includes: Receive the request prompt word corresponding to the target inference request, and determine the prompt word identifier corresponding to the request prompt word; Calculate the overlap length of the identifier of the historical sequence block corresponding to at least one historical lexical sequence block of the local storage node in the prompt word identifier, wherein the historical lexical sequence block is stored in the global shared storage space or the local storage hard disk of the local storage node; Based on the overlap length of the identifier, target historical word sequence blocks that overlap with the prompt word identifier are selected from the at least one historical word sequence block; The target historical key-value cache data corresponding to the target historical word sequence block is selected from the historical key-value cache data, and the storage location of the target historical key-value cache data is updated.

12. The data processing method according to claim 11, characterized in that, The step of filtering out target historical word sequence blocks that overlap with the prompt word identifier from the at least one historical word sequence block based on the overlap length of the identifier includes: Determine the total length of the prompt word identifier and calculate the ratio of the overlap length of the identifier to the total length of the identifier to obtain the overlap length ratio; When the overlap length ratio is greater than a preset length ratio threshold, a target historical word sequence block that overlaps with the prompt word identifier is selected from the at least one historical word sequence block.

13. The data processing method according to claim 11, characterized in that, Updating the storage location of the target historical key-value cache data includes: When the storage location is the local storage hard disk of the target storage node, the target historical key-value cache data is loaded into the node memory of the target storage node; When the storage location is the global shared storage space, the target historical key-value cache data is read from the global shared storage space and loaded into the local storage hard disk.

14. The data processing method according to claim 1, characterized in that, Also includes: When a change information is received from a storage node, the storage node that published the change information is identified as the change node, and the change type corresponding to the change information is determined. Based on the changed node and the changed type, the current node cluster is updated to obtain the updated node cluster and the updated hash value space corresponding to the updated node cluster. The updated node cluster is used as the current node cluster, and the updated hash value space is used as the hash value space of the current node cluster.

15. The data processing method according to claim 14, characterized in that, The step of updating the current node cluster based on the changed node and the change type to obtain the updated node cluster and the updated hash value space corresponding to the updated node cluster includes: When the change type is node exit, the successor node corresponding to the change node is identified among multiple storage nodes in the current node cluster according to the mapping position of the node identifier of the changed node in the hash value space. In the storage space of the hash value, identify the first storage space corresponding to the changed node and the second storage space corresponding to the successor node; The second storage interval is updated based on the first storage interval to obtain the updated second storage interval; The changed node is deleted from the current node cluster to obtain the updated node cluster, and the updated second storage area is used as the storage area corresponding to the successor node to obtain the updated hash value space.

16. The data processing method according to claim 14, characterized in that, The step of updating the current node cluster based on the changed node and the change type to obtain the updated node cluster and the updated hash value space corresponding to the updated node cluster includes: When the change type is node addition, the node identifier of the changed node is obtained, and the third storage area corresponding to the changed node is identified in the hash value space according to the mapping position of the node identifier in the hash value space. Select at least one fourth storage interval from the multiple storage intervals corresponding to the hash value space, and take the storage node corresponding to the fourth storage interval as the neighbor node of the change node; Add the changed node to the current node cluster to obtain the updated node cluster; The remaining storage intervals in the fourth storage interval, excluding the third storage interval, are used as the updated storage intervals for the neighboring nodes to obtain the updated hash value space.

17. A data processing apparatus, characterized in that, include: A request receiving unit is configured to receive a key-value cache request for a target lexical sequence block, wherein the key-value cache request carries the sequence block identifier of the target lexical sequence block, and the request type of the key-value cache request includes a key-value cache read request. A hash mapping unit is used to map the sequence block identifier to the hash value space corresponding to the current node cluster to obtain the target mapping position of the target word sequence block. The current node cluster includes multiple storage nodes, and the hash value space includes storage intervals corresponding to multiple virtual nodes of the storage nodes. An interval filtering unit is used to filter out the target storage interval corresponding to the target mapping position in the storage interval, and determine the target virtual node corresponding to the target storage interval; A node positioning unit is used to filter out the target storage node corresponding to the target word sequence block from the plurality of storage nodes based on the target virtual node; The data processing unit is configured to, when the target storage node is a local storage node, query the target key-value cache data corresponding to the target word sequence block in the node memory of the target storage node, and when the target key-value cache data exists in the node memory, read the target key-value cache data from the node memory.

18. An electronic device, characterized in that, It includes a processor and a memory, the memory storing an application program, and the processor running the application program within the memory to perform the steps of the data processing method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the data processing method according to any one of claims 1 to 16.

Images