A GPU cluster distributed inference system and method, electronic device, storage medium and computer program product

Abstract

This disclosure relates to the field of computer technology, and more particularly to a GPU cluster distributed inference system and method, electronic device, storage medium, and computer program product. The system includes: a cache management layer; the cache management layer includes: a key-value read / write interface and a cache engine; the key-value read / write interface is used to receive key-value read / write requests sent by a target compute node in the GPU cluster; the cache engine is used to respond to the key-value read / write requests, perform a lookup in the GPU cache of the target compute node, and, if a GPU cache miss is determined, write the key-value data corresponding to the key-value read / write request to the GPU cache of the target compute node using a target memory access channel, wherein the target memory access channel includes a high-speed memory direct connection bus and / or an RDMA network. Embodiments of this disclosure can effectively improve the key-value data I / O speed during the inference process, thereby improving inference efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a GPU cluster distributed inference system and method, electronic device, storage medium and computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale generative models, especially large language models based on the Transformer architecture, have been widely used in many fields such as natural language processing, content generation, and dialogue systems. The model inference process, especially when processing long context sequences or performing streaming generation, requires storing and frequently accessing large amounts of key-value caches, posing a significant challenge to computing and storage systems. Summary of the Invention

[0003] In view of this, this disclosure presents a GPU cluster distributed inference system and method, electronic device, storage medium and computer program product.

[0004] According to one aspect of this disclosure, a GPU cluster distributed inference system is provided, comprising: a cache management layer; the cache management layer includes: a key-value read / write interface and a cache engine; the key-value read / write interface is used to receive key-value read / write requests sent by a target compute node in the GPU cluster; the cache engine is used to respond to the key-value read / write request by searching in the GPU cache of the target compute node, and, if it is determined that the GPU cache is not hit, by using a target memory access channel to write the key-value data corresponding to the key-value read / write request into the GPU cache of the target compute node, wherein the target memory access channel includes a high-speed memory direct-connect bus and / or a remote direct memory access (RDMA) network.

[0005] In one possible implementation, the GPU cluster includes: multiple compute nodes and multiple storage nodes; the system further includes: a distributed multi-level storage pool; GPU caches and CPU memory of different compute nodes are interconnected via the memory high-speed direct connection bus; persistent storage of different compute nodes and storage nodes is interconnected via the RDMA network; the distributed multi-level storage pool includes: GPU caches and CPU memory of different compute nodes, and persistent storage of different compute nodes and storage nodes.

[0006] In one possible implementation, the cache management layer further includes: a block allocator and a block space manager; the block allocator is used to initialize and manage the unified virtual address space of the distributed multi-level storage pool, and to initialize the key-value metadata table of the distributed multi-level storage pool; the block space manager is used to call the block allocator to allocate physical blocks in the distributed multi-level storage pool to the inference task according to the unified virtual address space, and to create a block resource mapping table for the inference task.

[0007] In one possible implementation, the caching engine is further configured to: if it is determined that the GPU cache of the target computing node is not hit, query the block resource mapping table corresponding to the inference task to which the key-value read / write request belongs, and determine the address of the target physical block corresponding to the key-value read / write request in the multi-level storage pool; query the key-value metadata table corresponding to the multi-level storage pool, and determine the address offset of the key-value data corresponding to the key-value read / write request in the target physical block.

[0008] In one possible implementation, the cache engine further includes a memory manager; the memory manager is configured to, when determining that the target physical block is located in the CPU memory of the target computing node or another computing node other than the target computing node, use the high-speed memory direct connection bus to write the key value data corresponding to the key value read / write request from the CPU memory of the target computing node or other computing nodes to the GPU cache of the target computing node, based on the address offset of the key value data corresponding to the key value read / write request in the target physical block.

[0009] In one possible implementation, the caching engine further includes: a memory manager; the memory manager is configured to, when determining that the target physical block is located in the persistent storage of the target computing node, use memory swapping technology to write the key-value data corresponding to the key-value read / write request from the persistent storage of the target computing node to the CPU memory of the target computing node, based on the address offset of the key-value data corresponding to the key-value read / write request in the target physical block; the memory manager is also configured to use the high-speed direct memory bus to write the key-value data corresponding to the key-value read / write request from the CPU memory of the target computing node to the GPU cache of the target computing node.

[0010] In one possible implementation, the caching engine further includes a storage manager; the storage manager is configured to, when determining that the target physical block is located in the persistent storage of the target storage node, use the RDMA network to write the key-value data corresponding to the key-value read / write request from the persistent storage of the target storage node to the GPU cache of the target computing node, based on the address offset of the key-value data corresponding to the key-value read / write request in the target physical block, wherein the target storage node is one of the plurality of storage nodes.

[0011] In one possible implementation, the caching engine further includes: a prediction module; the prediction module is configured to predict the next key-value read / write request during the decoding phase of the inference task; the caching engine is further configured to, if it is determined that the next key-value read / write request is not hit in the GPU cache of the target computing node, use the target memory access channel to pre-write the key-value data corresponding to the next key-value read / write request into the GPU cache of the target computing node before receiving the next key-value request.

[0012] In one possible implementation, the cache management layer further includes: a version management module; the version management module is used to assign a snapshot version identifier to the inference task when any inference task starts; the cache engine is also used to respond to any key-value read / write request of the inference task and determine the key-value data corresponding to the key-value read / write request based on the snapshot version identifier.

[0013] According to another aspect of this disclosure, a GPU cluster distributed inference method is provided. The method is applied to a GPU cluster distributed inference system, which includes a cache management layer. The cache management layer includes a key-value read / write interface and a cache engine. The method utilizes the key-value read / write interface to receive key-value read / write requests sent by a target compute node in the GPU cluster. The cache engine responds to the key-value read / write requests by searching in the GPU cache of the target compute node. If a GPU cache miss is determined, the key-value data corresponding to the key-value read / write request is written to the GPU cache of the target compute node through a target memory access channel. The target memory access channel includes a high-speed memory direct-connect bus and / or a remote direct memory access (RDMA) network.

[0014] According to another aspect of this disclosure, an electronic device is provided, including the aforementioned GPU cluster distributed inference system.

[0015] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0016] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0018] The GPU cluster distributed inference system of this disclosure includes a cache management layer comprising a key-value read / write interface and a cache engine. Target compute nodes in the GPU cluster can send key-value read / write requests using the key-value read / write interface to achieve structured key-value data read / write. The cache engine responds to the key-value read / write requests of the target compute nodes by performing cache lookups in the GPU cache of the target compute nodes. In the event of a GPU cache miss, the engine can utilize a high-speed memory direct connection bus or RDMA network to quickly write the key-value data corresponding to the key-value read / write request into the GPU cache of the target compute nodes. Since the GPU cache is the storage unit closest to the processor executing the inference process in the target compute nodes, it can effectively improve the key-value data read / write speed during the inference process, thereby improving inference efficiency.

[0019] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0021] Figure 1 A block diagram of a GPU cluster distributed inference system according to an embodiment of the present disclosure is shown.

[0022] Figure 2 A schematic diagram of a GPU cluster according to an embodiment of the present disclosure is shown.

[0023] Figure 3 A framework diagram of a GPU cluster distributed inference system according to an embodiment of the present disclosure is shown.

[0024] Figure 4 A flowchart illustrating the execution of an inference task by a GPU cluster distributed inference system according to an embodiment of the present disclosure is shown.

[0025] Figure 5 A schematic diagram of a cache management layer according to an embodiment of the present disclosure is shown.

[0026] Figure 6 A flowchart illustrating a GPU cluster distributed inference method according to an embodiment of the present disclosure is shown.

[0027] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0028] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0029] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0030] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0031] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0032] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0033] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0034] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0035] With the rapid development of artificial intelligence technology, large-scale generative models, especially large language models based on the Transformer architecture, have been widely applied in many fields such as natural language processing, content generation, and dialogue systems. The model inference process, especially when processing long context sequences or performing streaming generation, requires storing and frequently accessing large amounts of key-value caches, posing a significant challenge to computing and storage systems. Based on research and industrial practice, inference optimization can be divided into three levels: token-level, model-level, and system-level.

[0036] 1. Token-level optimization

[0037] Sparsification: Discard tokens with low attention weights, for example, StreamingLLM (retains the initial token + sliding window); achieve dynamic sparsification by truncating the key-value (KV) cache.

[0038] Quantization: Compresses FP16 to INT8 / INT4. For example, a lossless quantization scheme can compress video memory by 50%.

[0039] Merging and Compression: Merge key-value vectors of similar tokens, for example, retain only 5% of key tokens to maintain performance.

[0040] 2. Model-level optimization

[0041] Attention mechanism improvements: Grouped Query Attention (GQA) or MoE architecture is adopted to reduce the number of key-value headers and reduce memory usage.

[0042] Location encoding decoupling: Separates location information from content, allowing historical key-value caches to be truncated without affecting location logic.

[0043] Dynamic context window: The model supports dynamic window length during training, reducing the need for fixed caching.

[0044] 3. System-level optimization

[0045] Tiered storage scheduling: DRAM+SSD tiered storage is used, and asynchronous preloading reduces access latency; a distributed memory pool is built through GPU clusters to support on-demand expansion of video memory.

[0046] Storage-compute separation architecture: Separating the model inference prefill stage, autoregressive decoding stage, and KV cache management into independent resource pools improves throughput by 5-10 times.

[0047] Hardware-aware design: Combining GPUDirect and RDMA to achieve microsecond-level KV cache access.

[0048] Existing system-level optimizations mainly rely on traditional technical approaches, such as data layering, resource grouping, and CPU bypass. They do not directly optimize the business model of key-value caching by combining hardware and software, and can only improve capabilities through localized optimizations.

[0049] To address the aforementioned issues, this disclosure provides a GPU cluster distributed inference system capable of structured key-value (KV) cache read / write operations, thereby improving inference efficiency by increasing the KV cache I / O speed during the inference process.

[0050] Figure 1 A block diagram of a GPU cluster distributed inference system according to an embodiment of the present disclosure is shown. Figure 1 As shown, the GPU cluster distributed inference system includes: a cache management layer; the cache management layer includes: a key-value read / write interface and a cache engine; the key-value read / write interface is used to receive key-value read / write requests sent by target compute nodes in the GPU cluster; the cache engine is used to respond to key-value read / write requests, search in the GPU cache of the target compute node, and, if it is determined that the GPU cache is not hit, use the target memory access channel to write the key-value data corresponding to the key-value read / write request into the GPU cache of the target compute node, wherein the target memory access channel includes a high-speed memory direct connection bus and / or a remote direct memory access (RDMA) network.

[0051] The GPU cluster distributed inference system of this disclosure includes a cache management layer comprising a key-value read / write interface and a cache engine. Target compute nodes in the GPU cluster can send key-value read / write requests using the key-value read / write interface to achieve structured key-value data read / write. The cache engine responds to the key-value read / write requests of the target compute nodes by performing cache lookups in the GPU cache of the target compute nodes. In the event of a GPU cache miss, the engine can utilize a high-speed direct memory bus or RDMA network to quickly write the key-value data corresponding to the key-value read / write request into the GPU cache of the target compute nodes. Since the GPU cache is the storage unit closest to the processor executing the inference process in the target compute nodes, it can effectively improve the key-value data I / O speed during the inference process, thereby improving inference efficiency.

[0052] In one possible implementation, the GPU cluster includes: multiple compute nodes and multiple storage nodes; the GPU cluster distributed inference system also includes: a distributed multi-level storage pool; GPU caches and CPU memory of different compute nodes are interconnected via a high-speed memory direct connection bus; persistent storage of different compute nodes and storage nodes is interconnected via an RDMA network; the distributed multi-level storage pool includes: GPU caches and CPU memory of different compute nodes, and persistent storage of different compute nodes and storage nodes.

[0053] The GPU cache (e.g., GPUHBM) is effectively expanded using CPU memory (e.g., CPU DRAM) and persistent storage (e.g., SSD). Simultaneously, high-speed direct memory access technology maximizes the low-latency characteristics of high-frequency data access across compute nodes, thereby significantly improving the overall efficiency of the GPU cluster in handling long-sequence inference tasks and the KV cache I / O throughput.

[0054] Figure 2 A schematic diagram of a GPU cluster according to an embodiment of the present disclosure is shown. Figure 2 As shown, a GPU cluster includes multiple compute nodes, multiple storage nodes, and interconnect components.

[0055] Each compute node includes: GPU cache (e.g., HBM), CPU memory (e.g., DRAM), and persistent storage (local SSD).

[0056] Each storage node includes: persistent storage (remote SSD shared storage pool).

[0057] Interconnect components include: high-speed direct memory interconnect components and RDMA interconnect components.

[0058] The high-speed direct-connect memory (HDM) component interconnects the GPU cache (HBM) and CPU memory (DRAM) of different compute nodes via the high-speed direct-connect memory bus, forming a cluster-level memory domain. The RDMA interconnect component interconnects the persistent storage (SSD) of different compute nodes and storage nodes via the RDMA network, forming a cluster-level RDMA domain. Based on the cluster-level memory domain and RDMA domain, a distributed multi-level storage pool is formed.

[0059] For any computing node, the key-value data generated during the inference process can be stored in layers: local GPU cache (high-frequency access hot data), local CPU memory (medium-frequency access warm data), local persistent storage (low-frequency access cold / historical data), and remote persistent storage (low-frequency access cold / historical data).

[0060] For any compute node, key-value data in the local persistent storage of the compute node can be written to the local CPU memory for memory access via memory swapping; key-value data in the remote persistent storage of other nodes can be accessed remotely via the RDMA network.

[0061] In one possible implementation, the cache management layer also includes: a block allocator and a block space manager; the block allocator is used to initialize and manage the unified virtual address space of the distributed multi-level storage pool, and to initialize the key-value metadata table of the distributed multi-level storage pool; the block space manager is used to call the block allocator to allocate physical blocks in the distributed multi-level storage pool to the inference task according to the unified virtual address space, and to create a block resource mapping table for the inference task.

[0062] The block space manager manages the block resources and block resource mapping table for inference tasks, and calls the block allocator to allocate resources. The block resource mapping table for inference tasks indicates the mapping relationship between each logical block ID (chunk-id) allocated to the inference task and the physical block, Chunk-ID → {path, offset, length, replica location, etc.}.

[0063] The block allocator performs the initialization of the distributed multi-level storage pool, providing unified address space management for CPU memory in the cluster-level memory domain and persistent storage in the RDMA domain, and initializing the KV metadata table of the distributed multi-level storage pool. Each node in the GPU cluster corresponds to a KV metadata table, which indicates the list of all logical block IDs stored in that node, as well as their version numbers, etc.

[0064] In the process of allocating resources for inference tasks, the block allocator selects a storage level (CPU memory level or persistent storage level) based on a strategy, then allocates physical blocks to the inference task from the selected storage level, registers them in the unified address space, and updates the KV metadata table of the node where the physical block resides. For example, the KV metadata table records the mapping relationship (logical block A - physical block X - inference task Y).

[0065] Figure 3 A framework diagram of a GPU cluster distributed inference system according to an embodiment of the present disclosure is shown. Figure 3 As shown, the GPU cluster distributed inference system includes: an inference engine (LLM Engine), a scheduler, a block space manager, a block allocator, a worker, and a cache engine.

[0066] For an inference task, first, a vllm instance is initialized on the inference engine (LLM Engine). The scheduler selects and starts a worker on the target compute node for the inference task according to the load balancing strategy.

[0067] When a GPU cluster distributed inference system executes multiple inference tasks, the scheduler manages the request queues of multiple inference requests and performs priority scheduling. For details, please refer to relevant technologies. This disclosure does not make any specific limitations on this process.

[0068] For any inference request in an inference task, the scheduler parses the request and estimates resources, then sends a block resource allocation request to the Block Space Manager. For example, based on key parameters extracted from the inference request, such as the model identifier, input context length, and preset generation length, the scheduler estimates the KV cache requirements of this inference task throughout its lifecycle and sends a corresponding block resource allocation request to the Block Space Manager.

[0069] The Block Space Manager allocates logical blocks for the inference task and calls the Block Allocator to allocate physical blocks from the distributed multi-level storage pool for the logical storage blocks. The logical blocks and physical blocks allocated for the inference task are returned to the Scheduler, which generates a block resource mapping table for the inference task.

[0070] Figure 4 A flowchart illustrating the execution of an inference task in a GPU cluster distributed inference system according to an embodiment of the present disclosure is shown. Figure 4 As shown, the scheduler starts the workers on the target computing node to execute the inference task.

[0071] In the prefill phase of the inference task, the worker performs Transformer forward propagation on the prompt text in the inference request sent by the inference task, calculates attention layer by layer, generates key-value data for that layer, and writes it to the distributed multi-level storage pool using the cache management layer; in the autoregressive decoding phase, the cache management layer reads historical key-value data from the multi-level storage pool for attention calculation, and writes it to the distributed multi-level storage pool again using the cache management layer.

[0072] In the KV data, the Key is the token location hash, and the Value is the attention vector.

[0073] Figure 5 A schematic diagram of a cache management layer according to an embodiment of the present disclosure is shown. Figure 5 As shown, the cache management layer includes: key-value read / write interface, key-value routing table (kv-router), garbage data scheduler (GC-scheduler), and version manager (version-mgr).

[0074] The kv read / write interfaces include: kv-put() for reading and kv-get() for writing.

[0075] During the prefill phase, for any KV data, the kv-put() interface is called to send a KV read / write request kv-put(Key Value). Based on the Key (Token location hash) in the KV read / write request, the kv routing table (kv-router) is queried, and then the KV data is directed to the specified node, realizing load balancing and dynamic addressing of distributed data.

[0076] In one example, the kv routing table (kv-router) stores the mapping relationship between keys and nodes. The implementation method can be a hash ring, consistent hashing, or virtual nodes, which is used to indicate which node provides services for a specific key. For specific implementation methods, please refer to the existing technology. This disclosure does not make any specific limitations on this.

[0077] In one example, routing can be based on the semantics of the request content (extracting semantic features) and load-aware routing based on the request features. Request feature load awareness involves inserting a secondary routing layer of "semantic features-weights" after the initial hashing, specifically including the following process: 1. Perform a regular hash on the token position to determine the key, and then obtain a list of candidate nodes (usually 3 virtual nodes) based on the routing table (kv-router); 2. Input the request content (or the key's prefix, tag, or even embedding vector) into a lightweight feature model, outputting: read / write ratio r:w, predicted QPS heat q, large value flag b, etc.; 3. Using {r,w,q,b} as feature vectors, query the local "node real-time load table" (CPU, memory, network card, queue length queue), and calculate the score using a simple linear weighted or LR model: score = w1 × CPU + w2 × queue + w3 × (q × r) + w4 × b; 4. Select the candidate node with the smallest score from the candidate nodes as the service node, execute the request, and simultaneously write the service node back to the client cache.

[0078] In one example, a dynamic hash rerouting mechanism is used to migrate only requests from changed or affected nodes. This gradual migration avoids business interruptions and updates the routing table and client-side routing cache in real time. Rerouting refers to the process where, when nodes go offline / online, experience hotspot imbalances, or require manual maintenance, the key-value routing table needs to migrate a specific range of keys from old nodes to new nodes. This is achieved by remapping nodes within a hash ring, generating a change range table. Gradual migration avoids distributing the entire change range table at once. Instead, it divides the change range into slots, sorts them by popularity (hot or cold), and migrates only a portion of slots in each batch, maintaining dual writes between old and new nodes throughout the migration process.

[0079] During the execution of a key-value (kv-put) read / write request, the garbage collection scheduler (GC-scheduler) periodically performs stress tests on the GPU cache of the target compute node. Based on the stress test results, it deletes key-value data with low reuse rates from the GPU cache. It also uses Bloom filters and LRU eviction algorithms to determine whether to trigger the migration of key-value data stored in the GPU cache to CPU memory (DRAM) or persistent storage (SSD).

[0080] During the autoregressive decoding phase, when generating the Nth token, the key-value (KV) data of the previous N-1 historical tokens is required. At this time, the KV read interface kv-put() is called to send a KV read / write request kv-get(Key), which includes the Key (Token position hash) of the previous N-1 historical tokens.

[0081] For key-value read / write requests, the system first performs a cache lookup in the GPU cache (HBM) of the target compute node. If the GPU cache hits, the data is read directly.

[0082] In one possible implementation, the cache engine is also used to: when it is determined that the GPU cache of the target computing node is missed, query the block resource mapping table corresponding to the inference task to which the key-value read / write request belongs, and determine the address of the target physical block corresponding to the key-value read / write request in the multi-level storage pool; query the key-value metadata table corresponding to the multi-level storage pool, and determine the address offset of the key-value data corresponding to the key-value read / write request in the target physical block.

[0083] For KV read / write requests, if the GPU cache of the target compute node misses, the block resource mapping table corresponding to the inference task and the key-value metadata table corresponding to the multi-level storage pool are queried based on the Key (Token location hash) in the KV read / write request. This allows the target physical block to be located in the unified virtual address space of the multi-level storage pool, as well as the address offset of the KV data corresponding to the KV read / write request in the target physical block.

[0084] In one possible implementation, the cache engine further includes: a memory manager; the memory manager is used to, when it is determined that the target physical block is located in the CPU memory of the target compute node or other compute nodes, write the KV data corresponding to the KV read / write request from the CPU memory of the target compute node or other compute nodes to the GPU cache of the target compute node based on the address offset of the KV data corresponding to the KV read / write request in the target physical block and using the high-speed direct memory bus.

[0085] If the KV data that the KV read / write request needs to read is currently stored in the CPU memory (DRAM) of the target compute node or other compute nodes, the KV data can be directly written from the CPU memory (DRAM) of the target compute node or other compute nodes to the GPU cache (HBM) of the target compute node using the high-speed memory direct connection bus.

[0086] In one possible implementation, the cache engine further includes: a memory manager; the memory manager, when determining that the target physical block is located in the persistent storage of the target compute node, uses memory swapping technology to write the KV data corresponding to the KV read / write request from the persistent storage of the target compute node to the CPU memory of the target compute node, based on the address offset of the KV data corresponding to the KV read / write request in the target physical block; the memory manager, also used to use a high-speed direct-access memory bus to write the KV data corresponding to the KV read / write request from the CPU memory of the target compute node to the GPU cache of the target compute node.

[0087] If the KV data that the KV read / write request needs to read is currently stored in the local persistent storage (SSD) of the target compute node, the KV data is first written from the local persistent storage (SSD) to the local CPU memory (DRAM) using memory swapping technology. Then, the KV data is written directly from the local CPU memory (DRAM) to the local GPU cache (HBM) using the high-speed memory direct connection bus.

[0088] In one possible implementation, the cache engine further includes: a storage manager; the storage manager is used to, when determining that the target physical block is located in the persistent storage of the target storage node, use an RDMA network to write the KV data corresponding to the KV read / write request from the persistent storage of the target storage node to the GPU cache of the target computing node according to the address offset of the KV data corresponding to the KV read / write request in the target physical block, wherein the target storage node is one of multiple storage nodes.

[0089] If the key-value data that the key-value read / write request needs to read is currently stored in the persistent storage (remote SSD) of a target storage node, the parallel file system is first used to locate the specific address of the key-value data in the persistent storage (remote SSD) of the target storage node, and then the RDMA network is used to write the key-value data from the persistent storage (remote SSD) of the target storage node to the GPU cache (HBM) of the target compute node.

[0090] In one possible implementation, the cache engine further includes: a prediction module; the prediction module is used to predict the next key-value read / write request during the decoding phase of the inference task; the cache engine is also used to, when it is determined that the next key-value read / write request is not hit in the GPU cache of the target compute node, use the target memory access channel to pre-write the key-value data corresponding to the next key-value read / write request into the GPU cache of the target compute node before receiving the next key-value request.

[0091] The prediction module can perform intelligent prefetching based on access patterns, predict the key-value data to be accessed, and write the key-value data to be accessed into the GPU cache of the target computing node in advance before receiving the next key-value request corresponding to the key-value data to be accessed, thereby improving the efficiency of subsequent key-value data reading.

[0092] During the autoregressive decoding phase, after generating a new token, the key-value data is calculated for the newly generated token. The key-value write interface kv-put() is called again to send a key-value read / write request kv-put(Key Value) to write the new key-value data to the distributed multi-level storage pool. This allows computation and cache I / O to overlap, optimizing the inference pipeline. Furthermore, only the new key-value data needs to be written, reducing the amount of data.

[0093] In one possible implementation, the cache management layer further includes: a version management module; the version management module is used to assign a snapshot version identifier to any inference task when it starts; and the cache engine is also used to respond to any key-value read / write request from the inference task and determine the key-value data corresponding to the key-value read / write request based on the snapshot version identifier.

[0094] The version management module ensures the atomicity, consistency, and traceability of historical versions of data operations through logical clocks (e.g., Revision) and snapshot isolation mechanisms.

[0095] In one example, the data structure carries a 64-bit header revision global logical clock (a hybrid clock, 48-bit wall-clock + 16-bit logical counter) for each key-value pair (KV), along with a monotonically increasing `snap_id` for snapshot isolation. When a worker starts an inference task, it first obtains the global `snap_id`. The write path is as follows: After the GPU calculates the new KV data, it marks the old KV data as pending-free; the KV manager allocates a new physical block, with revision = current global clock + 1; after the new KV data is written, the KV data index is atomically updated: `index[#1234] → {ptr=new_block, rev=10086}`; the reference count of the old KV data is decremented by 1, and only if it is 0 and revision < the earliest active `snap_id` is it truly reclaimed. Read path: The KV read request carries snap_id=10080; the service node finds the latest local revision=10085, but 10080 is still valid, so it returns the historical KV data pointer of revision=10080; if the KV data of revision=10080 has been recycled, the KV data of revision=10080 is reconstructed using adjacent KV data and delta-LOG.

[0096] Figure 6 A flowchart illustrating a GPU cluster distributed inference method according to an embodiment of this disclosure is shown. The method is applied to... Figures 1 to 5 The GPU cluster distributed inference system shown in the image includes a cache management layer; the cache management layer includes a key-value read / write interface and a cache engine. For example... Figure 6 As shown, the method may include:

[0097] In step S61, the key-value read / write interface is used to receive key-value read / write requests sent by the target computing node in the GPU cluster.

[0098] In step S62, the cache engine responds to key-value read / write requests by searching in the GPU cache of the target computing node, and if the GPU cache miss is determined, the key-value data corresponding to the key-value read / write request is written to the GPU cache of the target computing node through the target memory access channel, wherein the target memory access channel includes a high-speed memory direct connection bus and / or an RDMA network.

[0099] In one possible implementation, the GPU cluster includes: multiple compute nodes and multiple storage nodes; the GPU cluster distributed inference system also includes: a distributed multi-level storage pool;

[0100] GPU caches and CPU memory on different computing nodes are interconnected via a high-speed direct memory bus;

[0101] Persistent storage for different compute nodes and storage nodes is interconnected via an RDMA network;

[0102] The distributed multi-level storage pool includes: GPU cache and CPU memory on different computing nodes, and persistent storage on different computing and storage nodes.

[0103] In one possible implementation, the cache management layer also includes: a block allocator and a block space manager;

[0104] The method also includes:

[0105] The block allocator is used to initialize and manage the unified virtual address space of the distributed multi-level storage pool, as well as to initialize the key-value metadata table of the distributed multi-level storage pool.

[0106] Using the block space manager, the block allocator is invoked to allocate physical blocks from the distributed multi-level storage pool to the inference task based on the unified virtual address space, and to create a block resource mapping table for the inference task.

[0107] In one possible implementation, the method further includes:

[0108] If the GPU cache of the target computing node is determined to be missing, the cache engine is used to query the block resource mapping table corresponding to the inference task to which the key-value read and write request belongs, and to determine the address of the target physical block corresponding to the key-value read and write request in the multi-level storage pool.

[0109] By using the caching engine, the key-value metadata table corresponding to the multi-level storage pool is queried to determine the address offset of the key-value data in the target physical block corresponding to the key-value read / write request.

[0110] In one possible implementation, the cache engine also includes a memory manager;

[0111] The method also includes:

[0112] When it is determined that the target physical block is located in the CPU memory of the target compute node or other compute nodes, the memory manager is used to write the key value data corresponding to the key value read / write request from the CPU memory of the target compute node or other compute nodes to the GPU cache of the target compute node, based on the address offset of the key value data in the target physical block and using the high-speed direct memory bus.

[0113] In one possible implementation, the method further includes:

[0114] When determining that the target physical block is located in the persistent storage of the target computing node, the memory manager is used to write the key value data corresponding to the key value read / write request from the persistent storage of the target computing node to the CPU memory of the target computing node based on the address offset of the key value data in the target physical block and the key value read / write request.

[0115] By utilizing the memory manager and the high-speed direct-access memory bus, the key-value data corresponding to the key-value read / write request is written from the CPU memory of the target compute node to the GPU cache of the target compute node.

[0116] In one possible implementation, the caching engine also includes a storage manager;

[0117] The method also includes:

[0118] When determining that the target physical block is located in the persistent storage of the target storage node, the storage manager is used to write the key-value data corresponding to the key-value read / write request from the persistent storage of the target storage node to the GPU cache of the target compute node using the RDMA network, based on the address offset of the key-value data in the target physical block. The target storage node is one of multiple storage nodes.

[0119] In one possible implementation, the caching engine also includes a prediction module;

[0120] The method also includes:

[0121] During the decoding phase of the inference task, the prediction module is used to predict the next key-value read / write request.

[0122] If it is determined that the next key-value read / write request will not be hit in the GPU cache of the target compute node, the key-value data corresponding to the next key-value read / write request will be written to the GPU cache of the target compute node in advance before the next key-value request is received, using the cache engine and the target memory access channel.

[0123] In one possible implementation, the cache management layer also includes a version management module;

[0124] The method also includes:

[0125] When any inference task starts, the version management module is used to assign a snapshot version identifier to the inference task;

[0126] By utilizing the caching engine, it responds to any key-value read / write request in the inference task and determines the key-value data corresponding to the key-value read / write request based on the snapshot version identifier.

[0127] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0128] This disclosure also provides an electronic device including the aforementioned GPU cluster distributed inference system.

[0129] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0130] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0131] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0132] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. (Refer to...) Figure 7 Device 1900 can be provided as a server or terminal device. (See reference...) Figure 7 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0133] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0134] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0135] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0136] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0137] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0138] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0139] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0140] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0142] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A GPU cluster distributed inference system, characterized in that, include: Cache management layer; The cache management layer includes: a key-value read / write interface and a cache engine; The key-value read / write interface is used to receive key-value read / write requests sent by the target computing node in the GPU cluster; The caching engine is used to respond to the key-value read / write request by searching in the GPU cache of the target computing node, and, if it is determined that the GPU cache is not hit, to write the key-value data corresponding to the key-value read / write request into the GPU cache of the target computing node using the target memory access channel, wherein the target memory access channel includes a high-speed memory direct connection bus and / or a remote direct memory access (RDMA) network.

2. The system according to claim 1, characterized in that, The GPU cluster includes multiple compute nodes and multiple storage nodes; the system also includes a distributed multi-level storage pool. GPU caches and CPU memory of different computing nodes are interconnected through the high-speed direct memory bus; Persistent storage for different computing nodes and storage nodes is interconnected through the RDMA network; The distributed multi-level storage pool includes: GPU cache and CPU memory of different computing nodes, and persistent storage of different computing nodes and storage nodes.

3. The system according to claim 2, characterized in that, The cache management layer also includes: a block allocator and a block space manager; The block allocator is used to initialize and manage the unified virtual address space of the distributed multi-level storage pool, and to initialize the key-value metadata table of the distributed multi-level storage pool. The block space manager is used to call the block allocator to allocate physical blocks in the distributed multi-level storage pool to the inference task according to the unified virtual address space, and to create a block resource mapping table for the inference task.

4. The system according to claim 2, characterized in that, The caching engine is also used for: If the GPU cache of the target computing node is determined to be missed, the block resource mapping table corresponding to the inference task to which the key-value read / write request belongs is queried to determine the address of the target physical block corresponding to the key-value read / write request in the multi-level storage pool. Query the key-value metadata table corresponding to the multi-level storage pool to determine the address offset of the key-value data corresponding to the key-value read / write request in the target physical block.

5. The system according to claim 4, characterized in that, The caching engine also includes: a memory manager; The memory manager is configured to, when determining that the target physical block is located in the CPU memory of the target computing node or another computing node other than the target computing node, use the memory high-speed direct connection bus to write the key value data corresponding to the key value read / write request from the CPU memory of the target computing node or other computing nodes to the GPU cache of the target computing node, based on the address offset of the key value data corresponding to the key value read / write request in the target physical block.

6. The system according to claim 4, characterized in that, The caching engine also includes: a memory manager; The memory manager is used to, when determining that the target physical block is located in the persistent storage of the target computing node, use memory swapping technology to write the key value data corresponding to the key value read / write request from the persistent storage of the target computing node to the CPU memory of the target computing node, according to the address offset of the key value data corresponding to the key value read / write request in the target physical block. The memory manager is used to use the high-speed direct-connect bus to write the key-value data corresponding to the key-value read / write request from the CPU memory of the target computing node to the GPU cache of the target computing node.

7. The system according to claim 4, characterized in that, The caching engine also includes: a storage manager; The storage manager is configured to, when determining that the target physical block is located in the persistent storage of the target storage node, use the RDMA network to write the key-value data corresponding to the key-value read / write request from the persistent storage of the target storage node to the GPU cache of the target computing node, based on the address offset of the key-value data corresponding to the key-value read / write request in the target physical block. The target storage node is one of the plurality of storage nodes.

8. The system according to claim 1, characterized in that, The caching engine also includes: a prediction module; The prediction module is used to predict the next key-value read / write request during the decoding phase of the inference task. The caching engine is further configured to, when it is determined that the next key-value read / write request does not hit the GPU cache of the target computing node, use the target memory access channel to pre-write the key-value data corresponding to the next key-value read / write request into the GPU cache of the target computing node before receiving the next key-value request.

9. The system according to claim 1, characterized in that, The cache management layer also includes: a version management module; The version management module is used to assign a snapshot version identifier to any inference task when it starts. The caching engine is also used to respond to any key-value read / write request of the inference task and determine the key-value data corresponding to the key-value read / write request based on the snapshot version identifier.

10. A GPU cluster distributed inference method, characterized in that, The method is applied to a GPU cluster distributed inference system, which includes a cache management layer; the cache management layer includes a key-value read / write interface and a cache engine. Using the key-value read / write interface, receive key-value read / write requests sent by the target computing node in the GPU cluster; The cache engine responds to the key-value read / write request by searching in the GPU cache of the target computing node, and if the GPU cache miss is determined, the key-value data corresponding to the key-value read / write request is written to the GPU cache of the target computing node through the target memory access channel, wherein the target memory access channel includes a high-speed memory direct connection bus and / or a remote direct memory access (RDMA) network.

11. An electronic device, characterized in that, The GPU cluster distributed inference system includes any one of claims 1 to 9.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 10.

13. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 10.

14. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 10.

Images