Server system, data processing method and system

Abstract

The application discloses a server system and a data processing method and system, relates to the technical field of data processing, and comprises an inference unit, at least one storage unit and a data processing unit. The inference unit is used for executing an inference task corresponding to an inference request and generating key-value cache data when the inference task is executed. The at least one storage unit is used for storing the key-value cache data. The data processing unit is connected with the inference unit and the storage unit respectively and is used for receiving the inference request, generating the inference task according to the inference request and sending the inference task to the inference unit for execution, and obtaining the key-value cache data generated by the inference unit after the inference unit generates the key-value cache data. The key-value cache data is stored into the at least one storage unit, and efficient processing of the key-value cache data can be realized.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a server system, data processing method and system. Background Technology

[0002] With the rapid development of Large Language Models (LLMs) in the field of natural language processing, they have demonstrated powerful capabilities in knowledge-intensive tasks, especially in inference acceleration scenarios. In distributed computing scenarios, the inference of LLMs relies on server clusters, which frequently transfer key-value cache (KVCache) data between multiple servers. Related technologies depend on the Central Processing Unit (CPU) for data scheduling, transmission, and cache management. However, these technologies suffer from low storage resource utilization and high transmission latency, resulting in low data processing efficiency. Therefore, how to quickly process key-value cache data has become a pressing technical problem. Summary of the Invention

[0003] This application provides a server system, a data processing method, and a system to at least solve the problems of low data processing efficiency and how to quickly process key-value cached data in related technologies.

[0004] This application provides a server system, including:

[0005] The inference unit is used to execute the inference task corresponding to the inference request and generate key-value cache data when executing the inference task;

[0006] At least one storage unit is used to store key-value cache data;

[0007] The data processing unit, connected to both the inference unit and the storage unit, is used for:

[0008] Receive inference request;

[0009] Based on the inference request, generate an inference task and send the inference task to the inference unit for execution;

[0010] After the inference unit generates key-value cache data, retrieve the key-value cache data generated by the inference unit;

[0011] Store key-value cache data in at least one storage unit.

[0012] This application also provides a data processing method, including:

[0013] Receive inference request;

[0014] Based on the inference request, generate an inference task and send the inference task to the inference unit for execution;

[0015] After the inference unit generates key-value cache data, retrieve the key-value cache data generated by the inference unit;

[0016] Store key-value cache data in at least one storage unit.

[0017] This application also provides a data processing system, including the server system described above, and a first remote server and a second remote server connected to the server system.

[0018] The server system, data processing method, and system provided in this application establish a connection with the inference unit and at least one storage unit through a data processing unit (DPU). The data processing unit is responsible for receiving inference requests and triggering inference tasks to incite the inference unit to execute inference tasks. Furthermore, the data processing unit can directly obtain key-value cache data generated by the inference unit during the inference task processing and store the key-value cache data in the storage unit. Based on the aforementioned server system, the DPU can handle the scheduling and control of key-value cache data throughout the entire process of receiving, generating, transmitting, and storing. The CPU does not need to participate in the data path of the key-value cache data, shortening the transmission path and reducing data transmission latency. Simultaneously, it frees up CPU computing resources to execute other computing tasks, improving the overall resource utilization of the server system and thus increasing data processing efficiency. Attached Figure Description

[0019] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 Schematic diagram of the server system provided in the embodiments of this application Figure 1 ;

[0021] Figure 2 Schematic diagram of the server system provided in the embodiments of this application Figure 2 ;

[0022] Figure 3 This is a schematic diagram of the structure of a data processing unit provided in an embodiment of this application;

[0023] Figure 4A schematic diagram of a DPU-based KV cache data acceleration device provided in this application embodiment;

[0024] Figure 5 A schematic diagram of a DPU-based KV cache data storage device is provided for embodiments of this application;

[0025] Figure 6 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0027] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0028] Currently, AI servers based on LLM inference acceleration are becoming increasingly mainstream. Data needs to be distributed across AI server clusters, resulting in a large amount of data transfer between servers, with a significant proportion being cached data. Related technologies involve CPUs, DPUs, and memory during data processing, leading to relatively high transmission latency, which is further impacted by the heavy computational load on the CPU. Distributed computing completion time can be significantly reduced. For example, the traditional distributed computing process between two servers is as follows: The CPU on server 1 receives all user AI inference requests, such as questions in a question-and-answer system. These requests are processed by the CPU, including accepting requests, queuing, scheduling, and prioritizing them; preprocessing the requests by converting text into tokens, performing word segmentation, encoding, and construction. A large amount of request data is received from the network, processed by the CPU, and then stored in memory. In Server 1, the CPU notifies the system to use the GPU for LLM inference. Based on the inference model requirements and actual request characteristics, a large amount of KV cache data is generated. This cache data is stored in layers across the GPU's memory, Server 1's memory, and the server's SSD. When the inference task is complex or requires coordination across multiple servers, the CPU controls the relevant data to be sent to Server 2 via the DPU or network card. The cached data is read from memory or the SSD into the DPU and then sent to Server 2. Upon receiving the data, Server 2's DPU caches it in its memory and, depending on the data format (e.g., file format), saves it to Server 2's SSD. After completing these steps, the DPU notifies the CPU that data saving is complete. The CPU then notifies the GPU to read the cached data according to the AI ​​inference task requirements. In this process, the technology utilizes the CPU, GPU, and DPU, and data transmission and usage are serial. This usage method not only consumes a large amount of computing resources, but also incurs significant time consumption due to its serial operation. Related technologies suffer from low storage resource utilization and high transmission latency, resulting in low data processing efficiency. Therefore, how to quickly process key-value cached data has become an urgent technical problem to be solved.

[0029] To address the aforementioned issues, the server system, data processing method, and system provided in this application employ a Data Processing Unit (DPU) to schedule and control the key-value cache data throughout the entire process of receiving, generating, transmitting, and storing it. The DPU establishes connections with the inference unit and at least one storage unit. The DPU is responsible for receiving inference requests and triggering inference tasks to incite the inference unit to execute inference tasks. Furthermore, the DPU can directly acquire the key-value cache data generated by the inference unit during the processing of inference tasks and store the key-value cache data in the storage unit.

[0030] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Optionally, Figure 1 Schematic diagram of the server system provided in the embodiments of this application Figure 1 ,like Figure 1 As shown, the server system 10 provided in this application embodiment includes an inference unit 11, at least one storage unit 12, and a data processing unit 13.

[0032] The inference unit 11 is used to execute the inference task corresponding to the inference request and generate key-value cache data when executing the inference task.

[0033] Optionally, the inference unit 11 can be a GPU.

[0034] Optionally, the inference unit 11 can also be an artificial intelligence acceleration chip with parallel computing capabilities, such as a neural processing unit (NPU) or a tensor processing unit (TPU).

[0035] In one possible implementation, after receiving the inference task from the data processing unit 13, the inference unit 11 performs LLM inference computation based on the input data carried in the inference task. During the inference computation, the inference unit 11 generates a corresponding key vector and value vector for each word in the input sequence, and caches the key vector and value vector as key-value pairs as key-value cache data for reuse in subsequent inference steps.

[0036] At least one storage unit 12 is used to store key-value cache data.

[0037] Optionally, the storage unit 12 may include at least one of Dynamic Random Access Memory (DRAM) and SSD. DRAM is characterized by its fast read and write speed and is suitable for storing frequently accessed hot data; SSD is characterized by its large capacity and non-volatility and is suitable for storing infrequently accessed cold data.

[0038] Alternatively, the storage unit 12 can be an internal storage unit of the server system, such as onboard memory (e.g., DRAM) or a local solid-state drive.

[0039] Optionally, storage unit 12 can also be a storage unit 12 in another server connected to the server system, such as onboard memory or solid-state drive of the remote server. Data processing unit 13 connects to the remote server through network interface 14 to obtain storage information of storage unit 12 in the remote server, thereby realizing unified management and scheduling of cross-node storage resources.

[0040] Optionally, the storage unit 12 can also be a combination of the storage unit 12 inside the server system and the storage unit 12 in other servers. The data processing unit 13 can realize the pooling of distributed storage resources across nodes by connecting with at least one storage unit 12, that is, integrating local storage resources and remote storage resources into a unified multi-layer storage structure, so that the storage capacity is no longer limited by the physical storage limit of a single server.

[0041] The data processing unit 13 is connected to both the inference unit 11 and the storage unit 12. The data processing unit 13 is used to: receive an inference request; generate an inference task based on the inference request and send the inference task to the inference unit 11 for execution; obtain the key-value cache data generated by the inference unit 11 after it generates key-value cache data; and store the key-value cache data in at least one storage unit 12.

[0042] Optionally, the data processing unit 13 integrates a processor core and a hardware acceleration module, enabling it to run an operating system and perform data processing tasks independently without relying on CPU intervention.

[0043] Alternatively, the processor core can be an Advanced Reduced Instruction Set Machine (ARM).

[0044] Optionally, the data processing unit 13 receives AI inference requests from users via the network interface 14, such as question text from a question-answering system, text generation requests, or semantic query requests. As the data entry point for the server system, the data processing unit 13 is responsible for receiving and initially processing all inference requests.

[0045] Optionally, the data processing unit 13 is configured to: preprocess the inference request to generate an inference task; wherein the preprocessing includes at least one of queuing, scheduling and priority management of the inference request; and / or convert the natural language text in the inference request into tokens and perform at least one of word segmentation, encoding and construction operations on the tokens.

[0046] In one possible implementation, the data processing unit 13 receives all user AI inference requests (e.g., questions, text generation requests, or semantic query requests in a question-answering system), and performs request reception, queuing, scheduling, and priority management. The data processing unit 13 queues and schedules the received inference requests according to priority and / or arrival time order based on a preset scheduling strategy (e.g., first-come, first-served, priority preemption, or weighted fair queue), ensuring that high-priority requests are processed promptly. When preprocessing the AI ​​inference requests, the data processing unit 13 converts the natural language text in the request into tokens, performs word segmentation, encoding, and construction operations on the tokens, and sends the preprocessed data as input to the inference task to the inference unit 11. A large amount of request data received from the network interface 14 is processed by the data processing unit 13 and stored in local memory 101 or onboard memory.

[0047] In this embodiment, the data processing unit 13 preprocesses the inference requests to generate inference tasks, enabling it to independently complete the entire process from receiving the inference request to generating the task. The CPU does not need to participate in the preprocessing and task generation of the inference requests, further freeing up the CPU's computing resources. Simultaneously, the data processing unit 13 performs queuing, scheduling, and priority management on the inference requests. Based on the priority of the inference requests and the current load of the server system, it can rationally arrange the processing order of inference tasks, avoiding system performance degradation due to inference request congestion and improving the task processing efficiency and stability of the server system in high-concurrency scenarios.

[0048] Furthermore, the data processing unit 13 converts the natural language text in the inference request into tokens and performs word segmentation, encoding, and construction operations on the tokens. This converts the user-inputted raw natural language text into a standardized data format that the inference unit 11 can recognize and process. This ensures that the inference unit 11 can directly perform inference tasks based on the preprocessed data without having to perform data format conversion operations itself. This reduces the computational overhead of the inference unit 11, allowing its computational resources to focus on the inference computation itself, and further improves the inference efficiency of the inference unit 11.

[0049] Optionally, the data processing unit 13 generates an inference task after preprocessing the received inference request. This preprocessing includes at least one of queuing, scheduling, and priority management of the inference request; and / or, converting the natural language text in the inference request into tokens, and performing at least one of word segmentation, encoding, and construction operations on the tokens. The data processing unit 13 sends the preprocessed input data to the inference unit 11 along with the inference task to trigger the inference unit 11 to execute inference.

[0050] Optionally, a direct memory mapping relationship is pre-established between the data processing unit 13 and the inference unit 11. Specifically, after the server system starts, the data processing unit 13 requests a contiguous memory region in the operating system kernel driver layer and maps the high-bandwidth memory (HBM) of the inference unit 11 to this contiguous memory region. Based on this mapping relationship, the data processing unit 13 controls the inference unit 11 to directly write the generated key-value cache data into the contiguous memory region specified by the data processing unit 13, and then the data processing unit 13 reads the key-value cache data from this contiguous memory region. In this process, the key-value cache data does not need to pass through the CPU and system bus to travel from the HBM of the inference unit 11 to the memory region accessible by the data processing unit 13, realizing zero-copy data transmission.

[0051] Optionally, after reading key-value cache data from a contiguous memory region, the data processing unit 13 stores the key-value cache data in at least one storage unit 12 according to a preset storage strategy.

[0052] In some embodiments, the storage strategy may be determined based on the popularity level of key-value cache data and / or the remaining capacity of each storage unit 12 in the multi-layer storage structure, for example, storing frequently accessed hot data in high-speed storage unit 12 (such as DRAM) and storing infrequently accessed cold data in large-capacity storage unit 12 (such as SSD).

[0053] Optionally, the server system also includes a kernel driver layer, and the inference unit 11 includes a high-bandwidth memory; the data processing unit 13 is further configured to: register a character device in the kernel driver layer in response to the server system's startup operation. The character device is used to establish a control channel between the data processing unit 13 and the inference unit 11.

[0054] The kernel driver layer requests a contiguous memory region and maps high-bandwidth memory to the contiguous memory region to establish a data channel between the data processing unit 13 and the inference unit 11.

[0055] Optionally, the data processing unit 13 can obtain all the key-value cache data generated by the inference unit 11. Alternatively, the data processing unit 13 can obtain a portion of the key-value cache data generated by the inference unit 11.

[0056] In one possible implementation, the data processing unit 13 controls the inference unit 11 to write all key-value cache data into the contiguous memory region based on the mapping relationship between the contiguous memory region and the high-bandwidth memory. The data processing unit 13 then reads all key-value cache data from the contiguous memory region to ensure that subsequent inference tasks can obtain the complete cache state and avoid inference errors caused by missing data.

[0057] In another possible implementation, the data processing unit 13, based on the mapping relationship between contiguous memory regions and high-bandwidth memory, controls the inference unit 11 to write a portion of the key-value cache data into the contiguous memory region, and the data processing unit 13 to read a portion of the key-value cache data from the contiguous memory region. For example, during intermediate computation stages of the inference task or when cache space is insufficient, the data processing unit 13 can retrieve only the key-value cache data required for the current inference step according to actual needs, thereby reducing data transfer volume, lowering data transmission latency, and reducing memory bandwidth usage.

[0058] In the above implementation, the data processing unit 13 can flexibly choose to acquire all or part of the key-value cache data according to the actual application scenario. When the complete inference state is required, the data processing unit 13 acquires all key-value cache data to ensure the integrity and correctness of the inference task; when only a partial cache state is needed or system resources are scarce, the data processing unit 13 acquires partial key-value cache data to reduce unnecessary data handling, reduce data transmission latency and memory bandwidth pressure, thereby optimizing system resource utilization and data processing efficiency while ensuring the correctness of inference.

[0059] In some embodiments, the size of the contiguous memory region is determined by the product of the number of inference units 11 and the capacity of the high-bandwidth memory, ensuring that the contiguous memory region can fully map the high-bandwidth memory address space of all inference units 11, avoiding mapping failure or data loss due to insufficient address space, and ensuring the reliable operation of the dual-channel architecture in large-scale inference scenarios.

[0060] In some embodiments, the character device, acting as a control channel, is also used by the inference unit 11 to send a completion notification message to the data processing unit 13. After directly writing the key-value cache data into a contiguous memory region, the inference unit 11 sends a completion notification message to the data processing unit 13 via the character device. In response to this completion notification message, the data processing unit 13 reads the key-value cache data from the contiguous memory region. Through this completion notification mechanism, the data processing unit 13 does not need to continuously poll and wait for the completion status of the inference unit 11. Instead, it responds to the completion notification of the inference unit 11 in an event-driven manner, avoiding the resource waste and latency overhead caused by polling and waiting. This enables asynchronous parallel operation between the data processing unit 13 and the inference unit 11, further improving the overall processing efficiency of the server system.

[0061] In one possible implementation, when the data processing unit 13 sends an inference task to the inference unit 11 through a character device, it writes the task descriptor corresponding to the inference task into the kernel buffer corresponding to the character device. After the inference unit 11 detects that a new task descriptor has arrived in the character device through polling or interrupt, it reads the task descriptor from the kernel buffer and starts executing the inference task.

[0062] Optionally, the task descriptor includes at least one of the following: a task identifier, the storage address and length of the input data in the memory of the data processing unit 13, the storage address of the output key-value cache data in a contiguous memory region, and a model identifier. By carrying the above information through the task descriptor, the inference unit 11 can directly obtain the input data according to the task descriptor and write the output key-value cache data to the specified location, without the need for additional information interaction between the data processing unit 13 and the inference unit 11, further reducing the number of control message interactions and communication overhead.

[0063] In this embodiment, the data processing unit 13 establishes an independent control channel with the inference unit 11 by registering a character device in the kernel driver layer when the server system starts. It also maps the high-bandwidth memory of the inference unit 11 to a contiguous memory region, thereby constructing a dual-channel architecture between the data processing unit 13 and the inference unit 11, separating the control and data channels. Through this dual-channel architecture, control messages (such as inference task issuance and completion notifications) are transmitted through an independent character device channel, while key-value cache data is transmitted through the mapped channel of the high-bandwidth memory. The control messages and key-value cache data are physically isolated from each other, avoiding competition for bandwidth resources between control messages and large data blocks on a shared channel. This eliminates transmission delays caused by channel congestion and improves the response speed of control messages and the transmission efficiency of key-value cache data.

[0064] Optionally, the data processing unit 13 is used to: control the inference unit 11 to directly write key-value cache data into the contiguous memory region according to the mapping relationship between the contiguous memory region and the high-bandwidth memory; and read key-value cache data from the contiguous memory region.

[0065] In one possible implementation, the data processing unit 13 requests a contiguous memory region with physically contiguous reserved memory regions when the server system starts up. The inference unit 11, based on the Input / Output Memory Management Unit (IOMMU) mapping relationship between the high-bandwidth memory and this contiguous memory region, writes key-value cache data into this contiguous memory region via Direct Memory Access (DMA). Through IOMMU mapping and DMA, the inference unit 11 can directly access the contiguous memory region specified by the data processing unit 13 without address translation by the CPU's Memory Management Unit (MMU) or data copying instructions from the CPU, achieving data transfer with zero CPU involvement.

[0066] In this embodiment, the data processing unit 13, based on the mapping relationship between contiguous memory regions and high-bandwidth memory, controls the inference unit 11 to directly write key-value cache data into and read key-value cache data from the contiguous memory region. This allows the key-value cache data to travel from the high-bandwidth memory of the inference unit 11 to the memory region accessible by the data processing unit 13 without passing through the central processing unit and system bus, achieving zero-copy data transmission. Compared to the serial transmission path of key-value cache data in related technologies, this shortens the transmission path, reduces the number of data copies and bus transmissions between different levels of storage units 12, and lowers data transmission latency. Simultaneously, since the CPU does not need to participate in the copying and moving of key-value cache data, the CPU's data handling load is released, and its computing resources can be used to execute other business processing tasks, further improving the overall resource utilization and data processing efficiency of the server system.

[0067] In one possible implementation, after preprocessing the inference request, the data processing unit 13 sends an inference task to the inference unit 11 via a character device, notifying the inference unit 11 to use the GPU for large language model inference. Upon receiving the inference task, the inference unit 11 performs LLM inference computation based on the preprocessed input data, generating a large amount of key-value cache data during the inference process. Based on the actual needs of the inference model and the characteristics of the inference request, the inference unit 11 stores the generated KV Cache data hierarchically in the high-bandwidth memory of the inference unit 11 and the local memory 101 of the server system. Frequently accessed hot KV Cache data resides in the high-bandwidth memory to ensure extremely low latency access, while KV Cache data with moderate access frequency resides in the local memory 101.

[0068] In one possible implementation, after the server system (which can also be referred to as the first server in this embodiment) boots up, the DPU registers a character device at the operating system kernel driver layer for control message passing between the GPU and the DPU. The DPU kernel driver allocates a contiguous memory region and maps the GPU's HBM to this contiguous memory region. The memory region is determined according to the product of the actual number of GPUs in the first server and the HBM capacity.

[0069] In this embodiment, a connection is established between the DPU and the inference unit 11 and at least one storage unit 12. The DPU is responsible for receiving inference requests and triggering inference tasks to instruct the inference unit 11 to execute inference tasks. The DPU can directly obtain key-value cache data generated by the inference unit 11 during inference task processing and store the key-value cache data in the storage unit 12. Based on the above server system, the DPU can handle the scheduling and control of key-value cache data throughout the entire process of receiving, generating, transmitting, and storing. The CPU does not need to participate in the data path of key-value cache data, shortening the transmission path and reducing data transmission latency. Simultaneously, it frees up CPU computing resources to execute other computing tasks, improving the overall resource utilization of the server system and thus increasing data processing efficiency.

[0070] Optionally, Figure 2 Schematic diagram of the server system provided in the embodiments of this application Figure 2 To enable cross-node data transmission and control, the server system 10 in this embodiment further includes a network interface 14, through which the data processing unit 13 is connected to the first remote server 20 and the second remote server 30 respectively.

[0071] Optionally, the server system also includes local memory 101, which is the onboard memory of the server system.

[0072] Optionally, the first remote server 20 further includes a first remote data processing unit 201, a remote inference unit 202, and a remote memory 203 (for example, the remote memory 203 is the onboard memory of the first remote server 20, i.e., the first storage unit in the embodiments of this application).

[0073] Optionally, the second remote server 30 further includes a second remote data processing unit 301 and a remote storage unit 302 (for example, the remote storage unit 302 is the solid-state drive of the second remote server 30, i.e. the second storage unit in the embodiments of this application).

[0074] The data processing unit 13 is used to: obtain the storage information of the first storage unit in the first remote server 20 through the network interface 14; and / or obtain the storage information of the second storage unit in the second remote server 30 through the network interface 14; and establish a multi-layer storage structure composed of at least one storage unit 12 based on the storage information of the first storage unit and / or the storage information of the second storage unit.

[0075] Optionally, the storage information can be at least one of storage capacity, remaining storage space, or read / write speed level.

[0076] In one possible implementation, the first remote server 20 includes a first remote data processing unit 201 and a first storage unit (e.g., a solid-state drive), and the second remote server 30 includes a second remote data processing unit 301 and a second storage unit (e.g., onboard memory). The data processing unit 13 establishes a network connection with the first remote data processing unit 201 of the first remote server 20 and the second remote data processing unit 301 of the second remote server 30 via a Remote Direct Memory Access (RDMA) protocol or a Transmission Control Protocol / Internet Protocol (TCP / IP) protocol. The data processing unit 13 obtains storage information from the first and second storage units through this network connection.

[0077] In one possible implementation, the remote memory 203 in the first remote server 20 is onboard memory, used to provide high-speed remote memory access services, serving as a high-speed remote storage layer (e.g., the third storage layer) in a multi-layer storage structure, storing warm data that needs to be accessed quickly by other servers. The remote storage unit 302 in the second remote server 30 is a solid-state drive, used to provide large-capacity persistent storage services, serving as a persistent remote storage layer (e.g., the fourth storage layer) in a multi-layer storage structure, storing cold data that is accessed infrequently.

[0078] Optionally, the remote memory 203 is the onboard memory (such as DRAM) of the first remote server 20, and the remote storage unit 302 is the solid-state drive (SSD) of the second remote server 30. The data processing unit 13 establishes network connections with the first remote data processing unit 201 of the first remote server 20 and the second remote data processing unit 301 of the second remote server 30 via RDMA protocol or TCP / IP protocol, respectively, to obtain the storage information of the remote memory 203 and the remote storage unit 302.

[0079] In one possible implementation, after the server system starts, the data processing unit 13 establishes network connections with the first remote server 20 and the second remote server 30 according to the network configuration information in the configuration file. The data processing unit 13 establishes a peer-to-peer network connection with the first remote data processing unit 201 of the first remote server 20 and with the second remote data processing unit 301 of the second remote server 30. Through these peer-to-peer network connections, the data processing unit 13 can communicate directly with the data processing units 13 of the remote servers without going through the central processing unit of the remote servers, achieving end-to-end direct communication between DPUs and further reducing the latency of cross-node communication and the involvement of the central processing unit.

[0080] Optionally, the network configuration information includes at least one of IP address, port number, and communication protocol type.

[0081] Optionally, network interface 14 supports at least one of TCP protocol and RDMA protocol. When network interface 14 supports RDMA protocol, data processing unit 13 can directly read key-value cache data from the second storage unit (onboard memory) of the second remote server 30 via RDMA, or directly write key-value cache data to the second storage unit of the second remote server 30 via RDMA, realizing zero-copy data transmission across nodes, further reducing the latency of cross-node data transmission and the participation of the central processing unit.

[0082] In this embodiment, the DPU connects to two remote servers via network interface 14 and obtains their storage information. Based on this information, it establishes a multi-layered storage structure consisting of at least one storage unit 12. This allows the data processing unit 13 to integrate local and remote storage resources, forming a unified storage resource pool across nodes. Since the acquisition of storage information and the establishment of the multi-layered storage structure are both completed independently by the DPU, the CPU does not need to participate in the discovery and management of remote storage resources. The CPU completely exits the storage resource management link, further reducing the resource utilization of the central processing unit.

[0083] By connecting to remote servers and incorporating their storage units 12 into a multi-tiered storage structure, the server system's storage capacity is no longer limited by the physical storage limit of a single server. Horizontal expansion of storage capacity can be achieved by adding remote servers, meeting the storage capacity demands of the exponentially growing key-value cache data volume in large-scale distributed inference scenarios. Simultaneously, the establishment of the multi-tiered storage structure provides the hardware foundation for subsequent hierarchical storage migration based on popularity levels and / or remaining capacity. This allows frequently accessed hot data to reside in the high-speed storage layer, while infrequently accessed cold data can be moved to the remote storage layer. This ensures performance for hot data access while fully utilizing large-capacity storage resources, improving the overall storage resource utilization and data processing efficiency of the server system.

[0084] Optionally, the server system also includes onboard memory, and the inference unit 11 includes high-bandwidth memory; the multi-layer storage structure includes high-bandwidth memory and onboard memory, as well as a first storage unit and / or a second storage unit.

[0085] Optionally, the multi-layer storage structure includes a first storage layer, a second storage layer, a third storage layer, and a fourth storage layer arranged in sequence.

[0086] The high-bandwidth memory of the inference unit 11 serves as the first storage layer, used to store frequently accessed key-value cache data. The onboard memory of the server system serves as the second storage layer, used to store key-value cache data with moderate access frequency. The remote memory 203 of the first remote server 20 serves as the third storage layer, used to store key-value cache data that needs to be accessed quickly by other servers. The remote storage unit 302 of the second remote server 30 serves as the fourth storage layer, used to store infrequently accessed key-value cache data. The read / write speeds of these storage layers decrease sequentially, while their storage capacities increase sequentially, forming a complete storage hierarchy from high-speed, small-capacity to low-speed, large-capacity.

[0087] Through the aforementioned multi-layered storage structure, the server system can achieve a flexible balance between storage capacity and data access speed. This allows frequently accessed hot data to reside in high-speed storage layers such as high-bandwidth memory or onboard memory, while infrequently accessed cold data is moved to remote storage layers. This ensures the performance of hot data access while making full use of large-capacity remote storage resources, thus balancing the needs of both data access speed and storage capacity. This further improves the storage resource utilization and data processing efficiency of the server system in distributed inference scenarios.

[0088] Optionally, by incorporating the remote server's storage unit 12 into a multi-tiered storage structure, the data processing unit 13 can dynamically migrate key-value cache data between different storage tiers based on the access frequency of the key-value cache data and / or the remaining capacity of each storage unit 12 in the multi-tiered storage structure. For example, when the access frequency of a certain key-value cache data increases, the data processing unit 13 can migrate it from the fourth storage tier (remote SSD) to the third storage tier (remote RDMA memory) or the second storage tier (local onboard memory) to shorten the response time of subsequent accesses; when the access frequency of a certain key-value cache data decreases, the data processing unit 13 can migrate it from the first storage tier (HBM) or the second storage tier (local onboard memory) to the fourth storage tier (remote SSD) to free up high-speed storage resources for other hot data. Through the above dynamic migration mechanism, the server system can continuously maintain the optimal storage resource allocation state under dynamic changes in data access patterns, further improving the utilization efficiency of storage resources.

[0089] Through the aforementioned multi-layered storage structure, the server system can achieve a flexible balance between storage capacity and data access speed. This allows frequently accessed hot data to reside in high-speed storage layers such as high-bandwidth memory or onboard memory, while infrequently accessed cold data is moved to remote storage layers. This ensures the performance of hot data access while making full use of large-capacity remote storage resources, thus balancing the needs of both data access speed and storage capacity. This further improves the storage resource utilization and data processing efficiency of the server system in distributed inference scenarios.

[0090] Optionally, the data processing unit 13 is also used for:

[0091] Based on the popularity level of the key-value cache data and / or the remaining capacity of each storage unit 12 in the multi-level storage structure, the key-value cache data is stored or migrated hierarchically among the storage units 12 in the multi-level storage structure.

[0092] In one possible implementation, the data processing unit 13 manages the organization and location of key-value cache data through internally running KV cache scheduling software. Specifically, the KV cache scheduling software performs the following operations: cache data lookup: locating the current storage layer and physical storage address of the key-value cache data in the multi-level storage structure based on its unique identifier; inter-layer data transfer operation: migrating key-value cache data between different storage layers based on its popularity level and / or the remaining capacity of each storage layer; cache data eviction operation: when the used storage space of a certain storage layer reaches a preset eviction threshold, determining the key-value cache data to be evicted based on its popularity level and deleting or migrating it to the next level of storage layer. Through the aforementioned KV cache scheduling software, the data processing unit 13 achieves autonomous management of the entire lifecycle of key-value cache data in the multi-level storage structure, from lookup to transfer to eviction.

[0093] In one possible implementation, the heat level is determined by: the data processing unit 13 counting the number of accesses to each key-value cached data within a preset time window, and determining the heat level of each key-value cached data based on the number of accesses. For example, when the number of accesses to a certain key-value cached data within the preset time window is greater than or equal to a first threshold, the data processing unit 13 marks it as high-heat data; when the number of accesses is less than or equal to a second threshold, the data processing unit 13 marks it as low-heat data; when the number of accesses is between the first and second thresholds, the data processing unit 13 marks it as moderate-heat data. The data processing unit 13 determines the storage location of each key-value cached data based on the heat level. It is understood that the above thresholds can be determined according to actual conditions, and this application embodiment does not impose specific limitations.

[0094] In one possible implementation, the remaining capacity of each storage unit 12 serves as another basis for storage layout. The data processing unit 13 monitors the used and remaining storage space of each storage unit 12 in the multi-layer storage structure in real time. When the used storage space of a certain storage unit 12 reaches a preset storage upper limit threshold (e.g., the used space accounts for 85% of the total capacity), the data processing unit 13 triggers a storage migration operation, migrating some key-value cache data in that storage unit 12 to other storage units 12 with sufficient remaining capacity. Specifically, the data processing unit 13 determines the key-value cache data to be migrated from the current storage unit 12 according to the popularity level, prioritizing the migration of key-value cache data with lower popularity levels, and then migrates the key-value cache data to be migrated to the next level of storage layer storage units 12 with sufficient remaining capacity. Through the above migration triggering mechanism based on remaining capacity, the data write failure or system abnormality caused by any storage unit 12 being full is avoided, ensuring the stability and reliability of the server system during long-term operation.

[0095] In one possible implementation, the data processing unit 13 is further configured to: migrate key-value cache data from the onboard memory to the remote memory 203 of the first remote server 20 or the remote storage unit 302 of the second remote server 30 when the used storage space of the onboard memory of the server system reaches a preset threshold. Specifically, when the used storage space of the onboard memory reaches a preset threshold (e.g., the used space accounts for 80% of the total capacity), the data processing unit 13 determines the key-value cache data with the lowest popularity level from the onboard memory, and migrates the key-value cache data to the solid-state drive (fourth storage layer) of the first remote server 20 or the onboard memory (third storage layer) of the second remote server 30 via RDMA, so as to release the storage space of the onboard memory for storing newly generated or increasingly popular key-value cache data, ensuring that the high-speed storage resource of the onboard memory can always be used to store frequently accessed hot data.

[0096] Furthermore, by incorporating the remote server's storage unit 12 into a multi-layered storage structure, the data processing unit 13 can dynamically migrate key-value cache data between different storage layers based on the access frequency of the cached data. For example, when the access frequency of a certain key-value cached data increases, the data processing unit 13 can migrate it from the fourth storage layer to the third or second storage layer to shorten the response time for subsequent accesses; when the access frequency of a certain key-value cached data decreases, the data processing unit 13 can migrate it from the first or second storage layer to the fourth storage layer to free up high-speed storage resources for other frequently accessed data. Through this dynamic migration mechanism, the server system can continuously maintain the optimal storage resource allocation state even when data access patterns change dynamically, further improving the utilization efficiency of storage resources.

[0097] Based on this multi-layered storage structure, the DPU internally runs KV cache scheduling software to manage the KV cache. This includes: cache data lookup, data movement between cache layers, and cache data eviction.

[0098] In one possible implementation, the data processing unit 13 internally runs KV cache scheduling software, which performs hierarchical storage or migration operations. Responding to events such as changes in popularity level or storage capacity triggering events, the KV cache scheduling software performs a lookup operation on cached data, locating the key-value cache data to be migrated within the multi-level storage structure; it performs inter-level data transfer operations, moving the key-value cache data to be migrated from the source storage layer to the target storage layer; and it performs cache data eviction operations, deleting the key-value cache data with the lowest popularity level to free up storage space when space in any storage layer is insufficient. Through the execution of the KV cache scheduling software, the hierarchical storage and migration of key-value cache data are entirely completed autonomously by the data processing unit 13, without the need for central processing unit involvement in decision-making and execution, thus achieving independent processing of key-value cache data storage management by the data processing unit 13.

[0099] Here, the data processing unit 13 performs hierarchical storage or migration of key-value cache data among the storage units 12 in the multi-layer storage structure based on the popularity level of the key-value cache data and / or the remaining capacity of each storage unit 12 in the multi-layer storage structure. This achieves dynamic management and optimized layout of key-value cache data in the multi-layer storage structure. By introducing popularity level as the basis for storage layout, the data processing unit 13 can identify the differences in access frequency of different key-value cache data and store frequently accessed hot data in high-speed storage units 12 such as high-bandwidth memory or onboard memory, while storing infrequently accessed cold data in large-capacity storage units 12 such as remote solid-state drives. This ensures the performance of hot data access while making full use of large-capacity storage resources to store cold data, avoiding the waste of high-speed storage resources and improving the overall utilization rate of storage resources.

[0100] Optionally, the data processing unit 13 is further configured to: determine whether the inference task requires collaborative processing by the first remote server 20; if so, read key-value cache data from at least one storage unit 12; and send the key-value cache data to the first remote server 20 via the network interface 14.

[0101] In one possible implementation, the data processing unit 13 determines whether the inference task requires collaborative processing from the first remote server 20 by: determining the amount of computing resources required to complete the inference task based on the model type and / or the complexity of the inference request; when the required computing resources exceed the available computing resources of the local inference unit 11, the data processing unit 13 determines that collaborative processing from the first remote server 20 is required; or, the data processing unit 13 determines whether the current inference task involves cross-server model layer execution based on a preset distributed inference strategy (e.g., a strategy where different layers of the model are assigned to different servers for execution), and if so, determines that collaborative processing from the first remote server 20 is required.

[0102] In one possible implementation, the data processing unit 13 reads key-value cache data from at least one storage unit 12 in the following ways: the data processing unit 13 determines the target storage layer where the key-value cache data is currently located based on the storage location information of the key-value cache data; if the target storage layer is a local storage layer (such as HBM or onboard memory), the data processing unit 13 directly reads the key-value cache data from the local storage layer; if the target storage layer is a remote storage layer, the data processing unit 13 reads the key-value cache data from the remote storage layer via RDMA, and then sends the read key-value cache data to the first remote server 20. Through this method, regardless of which storage layer in the multi-layer storage structure the key-value cache data is stored in, the data processing unit 13 can read it and forward it to the first remote server 20 that needs to be processed collaboratively, realizing on-demand scheduling of key-value cache data across storage layers and nodes.

[0103] In one possible implementation, the data processing unit 13 sends the key-value cache data directly to the data processing unit 13 of the first remote server 20 via the RDMA protocol. After receiving the key-value cache data, the data processing unit 13 of the first remote server 20 stores the key-value cache data in the storage unit 12 corresponding to the first remote server 20 according to the task context information carried in the key-value cache data, and triggers the inference unit 11 of the first remote server 20 to continue to execute the subsequent inference task.

[0104] In one possible implementation, when the data processing unit 13 sends the key-value cache data to the first remote server 20, it also simultaneously sends task status information or task context information. The task status information or task context information includes at least one of the following: current inference progress, identifiers of processed lexical sequences, starting position of the lexical sequence to be processed, and task identifier corresponding to the inference task. By synchronously sending the above information, the first remote server 20 can accurately know the execution progress of the current inference task. Without additional status synchronization interaction, it can directly continue executing subsequent inference tasks based on the received key-value cache data, reducing the number of interactions between servers and further reducing the communication overhead and end-to-end completion time of distributed inference.

[0105] Through the aforementioned cross-node collaborative forwarding mechanism, when the computational complexity of the local inference task is high or requires the collaboration of multiple servers, the data processing unit 13 can proactively send the locally generated key-value cache data to the collaborative server. This allows the collaborative server to directly continue executing subsequent inference tasks based on the received key-value cache data, without needing to recalculate the generated key-value cache data. This avoids the waste of computing resources and time overhead caused by repeated calculations, thereby shortening the end-to-end completion time of the distributed inference task.

[0106] In one possible implementation, the server system provided in this application primarily uses a DPU network card, memory, and an SSD for hardware; the software includes software running on the DPU. The DPU-based KV cache data storage management method refers to a method for the DPU to manage the organization and location of data.

[0107] After the server system boots up, the DPU registers a character device at the operating system kernel driver layer for control message passing between the GPU and the DPU. The DPU kernel driver allocates a contiguous memory region and maps the GPU's HBM to this contiguous memory region. The memory region is determined by the product of the number of GPUs in the actual server system and the HBM capacity.

[0108] After startup, the server system establishes connections with the first remote server 20 and the second remote server 30 according to the network configuration information in the configuration file. During this process, the DPU in the server system establishes network connections with the DPUs in the first remote server 20 and the second remote server 30 respectively; the DPU in the server system obtains the memory information it manages from the DPU in the first remote server 20; and the DPU in the server system obtains the SSD information it manages from the DPU in the second remote server 30.

[0109] Based on the above steps, the server system establishes a multi-layered storage structure: a GPU HBM high-speed memory layer; a server system onboard memory layer; an SSD storage layer for the second remote server; and an RDMA memory layer for the first remote server.

[0110] Optionally, Figure 3 This is a schematic diagram of the structure of a data processing unit provided in an embodiment of this application, as shown below. Figure 3 As shown, the data processing unit 13 includes a retrieval accelerator 130.

[0111] Optionally, the data processing unit 13 may also include a data transceiver module 131, an advanced reduced instruction set machine 132, and on-chip memory 133.

[0112] Optionally, the data processing unit 13 further includes a first circuit board 1311 and a second circuit board 1312, which are connected together.

[0113] The retrieval accelerator 130, the data transceiver module 131, the advanced reduced instruction set machine, and the on-chip memory 133 are all configured on the first circuit board 1311.

[0114] Optionally, the first circuit board 1311 is a field programmable gate array (FPGA), and the second circuit board 1312 is a peripheral component interconnect express (PCIe).

[0115] Optionally, the FPGA board is used to implement data receiving and sending functions, supporting TCP and RDMA protocols; the ARM processor built into the FPGA is used to run the operating system and perform data scheduling and transfer control operations; the retrieval accelerator 130 is used for hardware acceleration of data index creation, searching, updating and deleting operations.

[0116] In this embodiment, the data processing unit 13 adopts a heterogeneous hardware architecture of FPGA+ARM. The FPGA is responsible for high-speed data transmission and reception and protocol processing, while the ARM processor is responsible for running the operating system and executing scheduling control. The two work together in parallel, enabling the data processing unit 13 to independently complete the entire process of data reception, scheduling, transfer, and retrieval acceleration without relying on a central processing unit. Simultaneously, through the built-in retrieval accelerator 130, operations such as index creation, searching, updating, and deletion are accelerated by hardware, further improving data retrieval efficiency and reducing retrieval latency.

[0117] Optionally, the data processing unit includes a retrieval accelerator 130; the retrieval accelerator 130 is used to perform accelerated operations on the index of key-value cache data.

[0118] The index is used to locate the physical storage address of the key-value cache data in at least one storage unit 12.

[0119] In this embodiment, the data processing unit 13 performs hardware-level accelerated computation on the index of key-value cache data through the built-in retrieval accelerator 130, so that operations such as index creation, searching, updating and deleting are all accelerated by the retrieval accelerator 130 in hardware, without the central processing unit participating in the calculation and management of the index.

[0120] The search acceleration strategy of the search accelerator 130 is described below with reference to specific embodiments.

[0121] Optionally, the data processing unit 13 is further configured to: when storing key-value cache data in at least one storage unit 12, obtain the unique identifier of the key-value cache data and the physical storage address of the key-value cache data; call the retrieval accelerator 130 to perform hash value calculation on the unique identifier and the physical storage address to establish a mapping index between the unique identifier and the physical storage address, and store the mapping index in the vector database.

[0122] The mapping index is used to perform retrieval operations on key-value cached data.

[0123] When storing key-value cache data, the DPU simultaneously obtains its unique identifier and physical storage address, calls the retrieval accelerator 130 to perform hash value calculation to build a mapping index, and stores it in the vector database. This achieves atomic operations for key-value cache data storage and index construction, ensuring that each key-value cache data has a corresponding mapping index. The hash value calculation is accelerated by the retrieval accelerator 130 in hardware, without CPU involvement. Index construction does not consume CPU resources, and the persistent storage of the vector database ensures that the index data can be quickly restored after a system restart.

[0124] In one possible implementation, this application embodiment addresses the need for efficient retrieval of massive KV cache data in large model inference scenarios. Relying on the hardware computing power of the data processing unit 13 and the built-in dedicated retrieval accelerator 130, the entire process of KV cache index construction, querying, deletion, and updating is offloaded to the data processing unit 13 for execution, freeing it from the host CPU computing power. Combined with the persistent storage of hash index mapping relationships in the vector database, the host-side retrieval latency is significantly reduced, memory bandwidth pressure is alleviated, and millisecond-level rapid location and dynamic management of KV cache data are achieved.

[0125] In one possible implementation, the core architecture includes: a DPU hardware layer, integrating a dedicated data retrieval accelerator 130 ( Figure 3 The retrieval accelerator (130) in the system undertakes hardware-accelerated computations such as hash operations, index traversal, and address matching, and independently completes data operation logic; the KV cache business layer is responsible for generating the original KV key-value cache data, issuing operation instructions, and receiving the physical storage address returned by the DPU; the vector database storage layer uniformly stores the hash index and hash value generated by the DPU and the mapping relationship between the KV cache physical address, and realizes persistent index management.

[0126] The retrieval of cached data relies on the DPU's built-in data retrieval accelerator 130, which provides operations such as creating, searching, deleting, and updating data indexes. The creation of a data index involves establishing a hash value for the data's address, and the accelerator performs the hash calculation. The hash value is stored in a vector database. The search for a data index involves finding the address of the data through the vector database. The deletion of a data index requires finding the data index through the accelerator and then updating the database to delete the data address. The update process is similar to the above operations, first performing a search and then updating the data address.

[0127] Specifically, the KV cache data index creation process includes: the host sends the generated KV cache data and corresponding physical storage address to the DPU; the DPU calls the built-in retrieval accelerator 130 to perform hardware-level hash value calculation on the unique identifier of the KV cache and the storage address; after the accelerator completes the hash value generation, it establishes a bidirectional mapping index between the hash value and the physical address of the KV cache; the DPU writes the completed hash index entries in batches to the vector database for persistent storage, thus completing the index creation.

[0128] Optionally, the data processing unit 13 is further configured to: upon receiving a retrieval request carrying a target retrieval identifier, invoke the retrieval accelerator 130 to calculate the target hash value corresponding to the target retrieval identifier; perform a matching query in the vector database based on the target hash value to obtain the corresponding physical storage address; and read key-value cache data from at least one storage unit 12 based on the physical storage address.

[0129] In one possible implementation, the KV cache data index lookup and retrieval process includes: the business side sends a KV cache retrieval request, carrying the target retrieval identifier; the request goes directly to the DPU retrieval accelerator 130, where the accelerator quickly calculates the standard hash value corresponding to the target identifier; the DPU initiates a hash value matching query to the vector database to accurately retrieve the mapped and bound KV cache physical storage address; the DPU sends the retrieved real data address back to the host, and the host directly reads the KV cache data based on the address to complete the high-speed retrieval.

[0130] Here, when the DPU receives a retrieval request, it calls the retrieval accelerator 130 to calculate the target hash value corresponding to the target retrieval identifier, and performs hardware-accelerated hash matching query through the vector database to obtain the physical storage address. Then, it directly reads the key-value cache data based on the physical storage address. The entire retrieval process is completed by the retrieval accelerator 130 hardware, without the CPU participating in hash calculation and index matching, which realizes fast location of key-value cache data and greatly shortens the retrieval response time.

[0131] Optionally, the data processing unit 13 is further configured to: upon receiving a deletion request carrying a key-value cache identifier to be deleted, call the retrieval accelerator 130 to calculate the hash value corresponding to the key-value cache identifier to be deleted; and locate and delete the corresponding mapping index in the vector database based on the hash value.

[0132] In one possible implementation, the KV cache data index deletion process includes: issuing a cache deletion command, passing in the KV cache identifier to be deleted; the DPU accelerator calculates the corresponding hash value, and iterates through the corresponding index entries in the matching vector database; after locating the target hash index and the bound data address, the DPU local cache index is cleared synchronously; the database update interface is called to delete the mapping relationship between the hash value and the physical address in the vector database, thus completing the data index deletion.

[0133] When the DPU receives a deletion request, it calls the retrieval accelerator 130 to calculate the hash value corresponding to the key-value cache identifier to be deleted, locates and deletes the corresponding mapping index in the vector database, thus achieving hardware acceleration of index deletion. The deletion operation of the mapping index is completed by the retrieval accelerator 130 hardware, without the CPU participating in index traversal and matching, freeing up the central processing unit's computing resources. At the same time, the mapping index in the vector database is deleted in a timely manner, avoiding invalid indexes occupying storage space and improving resource utilization.

[0134] Optionally, the data processing unit 13 is further configured to: upon receiving an update request carrying an old key-value cache identifier and updated key-value cache data, invoke the retrieval accelerator 130 to calculate the hash value corresponding to the old key-value cache identifier; locate and delete the corresponding old mapping index in the vector database based on the hash value corresponding to the old key-value cache identifier; obtain the unique identifier of the updated key-value cache data and the corresponding updated physical storage address; invoke the retrieval accelerator 130 to calculate the new hash value corresponding to the unique identifier of the updated key-value cache data; establish a new mapping index between the new hash value and the updated physical storage address, and store the new mapping index in the vector database.

[0135] In one possible implementation, the KV cache data index update process includes: the business side initiates a KV cache data update request, synchronously carrying new and old cache data information; the retrieval process is executed first, and the DPU accelerator matches and locates the original hash index and the old storage address; the accelerator recalculates a new hash value for the updated KV cache data to generate a new address mapping relationship; the old index entries in the vector database are deleted first, and then the new hash index and the new physical address mapping data are written to complete the full index coverage update.

[0136] When the DPU receives an update request, it first calls the retrieval accelerator 130 to calculate the hash value based on the old key-value cache identifier, locates and deletes the old mapping index in the vector database, then obtains the unique identifier of the updated key-value cache data and its corresponding updated physical storage address, calls the retrieval accelerator 130 to calculate the new hash value, establishes the new mapping index, and stores it in the vector database. This achieves full-process hardware acceleration for index updates. Throughout the update process, the location and deletion of the old index, the calculation of the new hash value, and the establishment of the new index are all completed by the retrieval accelerator 130 in hardware, without the involvement of the central processing unit. This ensures the synchronous consistency of the index data after the key-value cache data is updated, while also improving the execution efficiency of the update operation.

[0137] Optionally, Figure 4 This is a schematic diagram of a DPU-based KV cache data acceleration device provided in an embodiment of this application, applied in the aforementioned server system and the first remote server. The data acceleration device includes a hardware system and a software system.

[0138] The hardware system includes a data processing unit 13, a memory and inference unit 11, with a dedicated DPU processor and network interface card at its core, and a hardware topology architecture based on the DPU design. The software system includes software running on the DPU, specifically drivers and applications running on the DPU. This data acceleration device is a CPU-free server design that fully leverages the advantages of the highly integrated hardware and software design of the DPU.

[0139] Optionally, the physical topology of the data acceleration device is a combination of 1 to 8 DPUs and GPUs. That is, the server system may include 1 to 8 DPUs and 1 to 8 GPUs, with the DPUs and GPUs connected via a high-speed bus 40 (e.g., PCIe bus) to form a tightly coupled heterogeneous computing architecture, with the DPU serving as the core of the data plane and control plane, and the GPU serving as the core of inference computing.

[0140] Through the aforementioned data acceleration device, the server system achieves a CPU-free hardware architecture design. The DPU handles all data path operations, including data reception, request preprocessing, storage management, and cross-node forwarding, while the GPU focuses on executing inference computing tasks. The two work together efficiently through a dual-channel architecture. Since the CPU is completely removed from the data and control paths, the data transfer latency and CPU resource consumption of the server system are significantly reduced, and the overall power consumption and heat dissipation requirements of the system are also reduced accordingly. This makes it particularly suitable for deployment scenarios of large-scale distributed AI inference clusters.

[0141] It should be noted that, Figure 4 This is merely an illustrative representation of one topology connection between a DPU and a GPU. In actual deployments, the ratio of DPUs to GPUs and the connection topology can be adjusted according to specific application scenarios. For example, one DPU may connect to multiple GPUs, multiple DPUs may connect to one GPU, or multiple DPUs may form cross connections with multiple GPUs. This application does not limit this.

[0142] Optionally, Figure 5 This is a schematic diagram of a DPU-based KV cache data storage device provided in an embodiment of this application, which is applied in the aforementioned second remote server 30. The KV cache data storage device includes a hardware system and a software system.

[0143] The hardware system includes a second remote data processing unit 301 and a remote storage unit 302. The remote storage unit 302 can be a solid-state drive (SSD), and its core is a dedicated DPU processor and network interface card (NIC), based on a hardware topology architecture designed for the DPU. The software system includes software running on the DPU, specifically drivers and applications running on the DPU. This KV cache data storage is a CPU-free server design that fully leverages the advantages of the highly integrated hardware and software design of the DPU.

[0144] Optionally, the physical topology of the KV cache data storage is a combination of one DPU and multiple SSDs. That is, the second remote server 30 may include one DPU and multiple SSDs. The DPU and multiple SSDs are connected through a high-speed bus (such as a PCIe bus) to form a storage node architecture with the DPU as the core. The DPU is responsible for receiving key-value cache data sent from the remote server and distributing the data to multiple SSDs for storage, as well as managing the storage space of the SSDs and providing storage information query services.

[0145] Through the aforementioned KV cache data storage, the second remote server 30 is deployed as a pure storage node in the distributed system. It requires no CPU or GPU; the DPU, in conjunction with multiple SSDs, provides large-capacity, high-performance persistent storage services for key-value cache data. The DPU handles all storage-related operations, including network data reception, storage space management, data distribution and aggregation, and storage information reporting. No central processing unit is required to participate in any storage management tasks, significantly reducing the hardware cost and power consumption of the storage node. Simultaneously, the parallel storage of multiple SSDs improves data read / write throughput, making it particularly suitable for large-capacity persistent storage scenarios for cold data in large-scale distributed AI inference clusters.

[0146] Optionally, Figure 6 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The data processing unit is the execution entity of the data processing method, and the method includes:

[0147] S601: Receive inference request.

[0148] S602: Generate a reasoning task based on the reasoning request, and send the reasoning task to the reasoning unit for execution.

[0149] S603: After the inference unit generates key-value cache data, retrieve the key-value cache data generated by the inference unit.

[0150] S604: Store key-value cache data in at least one storage unit.

[0151] In one possible implementation, combining Figure 2The workflow of the server system provided in this application embodiment is as follows:

[0152] The data processing unit receives all AI inference requests from users (such as questions in a question-answering system, text generation requests, or semantic query requests). It handles request reception, queuing, scheduling, and priority management; and preprocesses the requests, converting natural language text into tokens and performing operations such as word segmentation, encoding, and construction. A large amount of request data received from the network interface is processed by the data processing unit and stored in memory.

[0153] The data processing unit sends inference tasks to the inference unit via a control channel (e.g., a character-based device), instructing the inference unit to use the GPU to perform large language model inference. The inference unit generates a large amount of key-value cache data based on the actual needs and request characteristics of the inference model. The inference unit stores this key-value cache data in layers, residing in its high-bandwidth memory and the server system's local memory. Frequently accessed hot key-value cache data resides in the high-bandwidth memory to ensure extremely low latency access, while less frequently accessed key-value cache data is stored in local memory.

[0154] Once the key-value cache data has been stored in the local storage of the server system for a preset threshold, the data processing unit will save the key-value cache data to the solid-state drive of the second remote server (e.g., the second remote server 30).

[0155] When the inference task is complex or requires coordination among multiple servers, and the key-value cache data needs to be sent to the first remote server for collaborative processing, the data processing unit reads the key-value cache data from local memory or the solid-state drive of the second remote server, and sends the key-value cache data to the first remote server through the network interface.

[0156] After receiving the key-value cache data, the first remote server continues to complete the AI ​​inference task based on the received cache data.

[0157] By receiving inference requests and generating inference tasks, and then directly sending them to the inference unit for execution, the distribution path of inference tasks becomes clearer, reducing interference from intermediate steps in the inference process. After key-value cache data is generated by the inference unit, it is promptly received and further stored in at least one storage unit, thus forming a data processing link that coordinates inference execution with cache storage. This ensures that key-value cache data is transferred as quickly as possible after generation. This reduces the reliance on host-side control resources continuously participating in cache processing, thereby reducing resource consumption and data transfer overhead. Therefore, it helps to shorten cache processing latency and improve the overall throughput and continuous operational stability in distributed inference scenarios.

[0158] This application also provides a data processing system (such as...) Figure 2 As shown), it includes the server system provided above, and a first remote server and a second remote server connected to the server system.

[0159] By connecting the server system to a first remote server and a second remote server, the server system can not only execute inference tasks and process key-value cache data locally, but also form a collaborative data flow and storage system with the remote servers. This allows key-value cache data to be transferred, backed up, or reused between different servers according to task status, capacity requirements, or scheduling strategies. Since the data processing unit in the aforementioned server system can directly receive and control cached data from the inference and storage units, cross-server data interaction does not need to rely excessively on CPU-level forwarding, thus shortening communication paths and reducing processing latency. Therefore, in distributed inference scenarios, this improves the scheduling and access efficiency of key-value cache data, helps alleviate local storage pressure, and enhances the system's concurrent processing capabilities, scalability, and continuous stable operation.

[0160] The technology provided in this application, when applied to a server system, can effectively reduce CPU usage through dedicated computing hardware acceleration, thereby saving the CPU from running more input / output (I / O) processing programs and indirectly improving performance. This also effectively enhances the security of network data transmission and storage, and avoids system-level performance bottlenecks.

[0161] This application embodiment utilizes a data acceleration technology based on DPU network interface card design in storage, which can achieve hardware-level I / O data retrieval, management, and transfer acceleration. This technology is widely used in various industries, supports major projects, and ensures the high availability and high performance of customer storage servers.

[0162] The data acceleration in this application embodiment will be applied to AI acceleration and storage, enabling the storage controller to accelerate data movement and computation. This solution, applied to the aforementioned storage products, employs hardware-level data movement acceleration technology to improve the reliability and availability of the storage system while meeting customer performance (low latency, high concurrency) and capacity requirements, and increasing data bandwidth. Compared to traditional solutions, it offers independent performance scalability and reduced storage controller latency.

[0163] This application also provides a data processing apparatus, including:

[0164] The first receiving module is used to receive inference requests;

[0165] The processing module is used to generate inference tasks based on inference requests and send the inference tasks to the inference unit for execution.

[0166] The second receiving module is used to obtain the key-value cache data of the inference unit after the inference unit generates key-value cache data;

[0167] A storage module is used to store key-value cache data in at least one storage unit.

[0168] This application also provides a schematic diagram of an electronic device. It includes at least one processor and a memory. Optionally, the electronic device further includes a communication component. The processor, memory, and communication component are connected via a bus.

[0169] In a specific implementation, at least one processor executes computer execution instructions stored in memory, causing at least one processor to execute the above-described data processing method embodiment.

[0170] The specific implementation process of the processor can be found in the above method embodiments, and its implementation principle and technical effect are similar, so it will not be repeated here.

[0171] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0172] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0173] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0174] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above data processing method embodiments when it is run.

[0175] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0176] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0177] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0178] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0179] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0180] The server system, data processing method, and system provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A server system, characterized in that, include: Inference unit (11) is used to execute the inference task corresponding to the inference request and generate key-value cache data when executing the inference task; At least one storage unit (12) is used to store the key-value cache data; The data processing unit (13), connected to the inference unit (11) and the storage unit (12) respectively, is used for: Receive the inference request; Based on the inference request, the inference task is generated and sent to the inference unit (11) for execution; After the inference unit (11) generates the key-value cache data, the key-value cache data generated by the inference unit (11) is obtained; The key-value cache data is stored in the at least one storage unit (12); The server system also includes a kernel driver layer, and the inference unit (11) includes a high-bandwidth memory; The data processing unit (13) is also used for: In response to the startup operation of the server system, a character device is registered in the kernel driver layer; wherein the character device is used to establish a control channel between the data processing unit (13) and the inference unit (11); The kernel driver layer requests a contiguous memory region and maps the high-bandwidth memory to the contiguous memory region to establish a data channel between the data processing unit (13) and the inference unit (11); The data processing unit (13) is used for: Based on the mapping relationship between the contiguous memory region and the high-bandwidth memory, the inference unit (11) is controlled to directly write the key-value cache data into the contiguous memory region; Read the key-value cache data from the contiguous memory region.

2. The server system according to claim 1, characterized in that, The server system also includes a network interface (14); the data processing unit (13) is connected to the first remote server (20) and the second remote server (30) through the network interface (14). The data processing unit (13) is used for: The storage information of the first storage unit in the first remote server (20) is obtained through the network interface (14); and / or, the storage information of the second storage unit in the second remote server (30) is obtained through the network interface (14); Based on the storage information of the first storage unit and / or the storage information of the second storage unit, a multi-layer storage structure composed of the at least one storage unit (12) is established.

3. The server system according to claim 2, characterized in that, The server system also includes onboard memory, and the inference unit (11) includes high-bandwidth memory; The multi-layer storage structure includes the high-bandwidth memory and the onboard memory, as well as the first storage unit and / or the second storage unit.

4. The server system according to claim 2, characterized in that, The data processing unit (13) is also used for: Based on the popularity level of the key-value cache data and / or the remaining capacity of each storage unit (12) in the multi-layer storage structure, the key-value cache data is hierarchically stored or migrated among the storage units (12) in the multi-layer storage structure.

5. The server system according to claim 2, characterized in that, The data processing unit (13) is also used for: Determine whether the reasoning task requires collaborative processing by the first remote server (20); If necessary, the key-value cache data is read from the at least one storage unit (12); The key-value cache data is sent to the first remote server (20) through the network interface (14).

6. The server system according to any one of claims 1 to 5, characterized in that, The data processing unit (13) includes a retrieval accelerator (130); The retrieval accelerator (130) is used to perform accelerated operations on the index of the key-value cache data; wherein the index is used to locate the physical storage address of the key-value cache data in the at least one storage unit (12).

7. The server system according to claim 6, characterized in that, The data processing unit (13) is also used for: When storing the key-value cache data in the at least one storage unit (12), obtain the unique identifier of the key-value cache data and the physical storage address where the key-value cache data is stored; The retrieval accelerator (130) is invoked to perform hash value calculation on the unique identifier and the physical storage address to establish a mapping index between the unique identifier and the physical storage address, and the mapping index is stored in the vector database; wherein, the mapping index is used to perform retrieval operations on the key-value cache data.

8. The server system according to claim 7, characterized in that, The data processing unit (13) is also used for: Upon receiving a retrieval request carrying a target retrieval identifier, the retrieval accelerator (130) is invoked to calculate the target hash value corresponding to the target retrieval identifier; Based on the target hash value, a matching query is performed in the vector database to obtain the corresponding physical storage address; The key-value cache data is read from the at least one storage unit (12) according to the physical storage address.

9. The server system according to claim 7, characterized in that, The data processing unit (13) is also used for: Upon receiving a deletion request carrying a key-value cache identifier to be deleted, the retrieval accelerator (130) is invoked to calculate the hash value corresponding to the key-value cache identifier to be deleted; Based on the hash value, locate and delete the corresponding mapping index in the vector database.

10. The server system according to claim 7, characterized in that, The data processing unit (13) is also used for: Upon receiving an update request carrying the old key-value cache identifier and the updated key-value cache data, the retrieval accelerator (130) is invoked to calculate the hash value corresponding to the old key-value cache identifier; Based on the hash value corresponding to the old key-value cache identifier, locate and delete the corresponding old mapping index in the vector database; Obtain the unique identifier and corresponding updated physical storage address of the updated key-value cache data; The retrieval accelerator (130) is invoked to calculate the new hash value corresponding to the unique identifier of the updated key-value cache data; A new mapping index is established between the new hash value and the updated physical storage address, and the new mapping index is stored in the vector database.

11. The server system according to any one of claims 1 to 5, characterized in that, The data processing unit (13) is used for: The inference request is preprocessed to generate the inference task; wherein the preprocessing includes: The inference request is subjected to at least one of queuing, scheduling, and priority management processing; and / or, The natural language text in the inference request is converted into tokens, and at least one of word segmentation, encoding, and construction operations is performed on the tokens.

12. A data processing method, characterized in that, include: Receive inference request; Based on the reasoning request, a reasoning task is generated and sent to the reasoning unit (11) for execution; After the inference unit (11) generates key-value cache data, the key-value cache data generated by the inference unit (11) is obtained; The key-value cache data is stored in at least one storage unit (12); The method further includes: In response to the startup operation of the server system, a character device is registered in the kernel driver layer; wherein, the character device is used to establish a control channel between the data processing unit (13) and the inference unit (11); The kernel driver layer requests a contiguous memory region and maps a high-bandwidth memory to the contiguous memory region to establish a data channel between the data processing unit (13) and the inference unit (11); The method further includes: Based on the mapping relationship between the contiguous memory region and the high-bandwidth memory, the inference unit (11) is controlled to directly write the key-value cache data into the contiguous memory region; Read the key-value cache data from the contiguous memory region.

13. A data processing system, characterized in that, It includes a server system as described in any one of claims 1 to 11, and a first remote server (20) and a second remote server (30) connected to the server system.

Images