A large language model inference system and method under a limited resource environment

Abstract

The application discloses a large language model inference system and method under a limited resource environment. The system comprises a host, a storage-computing integrated inference chip, an intelligent solid state disk controller chip and a NAND flash memory array; the intelligent solid state disk controller chip uniformly addresses and manages weight data objects and key value cache block objects stored in the NAND flash memory array, and performs predictive prefetching on the weight data objects and expert weights of the mixed expert model according to inference calculation progress. Prefix hash matching and hit judgment of the key value cache block are completed by hardware logic on the storage side, the key value cache block is actively pulled by the controller and asynchronously sinks to the flash memory, and the flash memory capacity is managed by high and low water lines. Through an end-to-end direct data path, weight data and key value cache data are directly transported between the flash memory and the local memory of the inference chip without passing through the host memory. The host only bears the control plane function of request scheduling, word segmentation and sampling output.

Description

Technical Field

[0001] This invention belongs to the field of computer technology and relates to artificial intelligence reasoning acceleration and data storage scheduling, specifically to a large language model reasoning system and method under resource-constrained environments. Background Technology

[0002] The parameter size of large language models continues to grow. The weight data for models with hundreds of billions of parameters is approximately 200GB at FP16 precision and still around 100GB at INT8 quantization precision. Simultaneously, the key-value cache (KV cache) used to cache attention key-value pairs during inference grows linearly with the context length; the key-value cache for contexts with hundreds of thousands of tokens can reach tens of GB. This dual capacity pressure from weight data and key-value cache means that the deployment of large language models has long relied on data center-grade graphics processors equipped with large amounts of video memory.

[0003] Weight offloading solutions cache model weights or key-value pairs in host memory or SSDs, loading them into the GPU layer by layer via the PCIe bus during inference. This approach requires data transfer between the host software stack and host memory, making PCIe bandwidth a bottleneck, resulting in high host processor utilization. Furthermore, it cannot run models with a total weight exceeding the available host memory in scenarios with limited host memory. Distributed key-value caching pool solutions organize idle memory and SSDs in the cluster into a shared cache pool. Host-side software background processes periodically offload evicted key-value cache objects to the local SSD, and upon a hit, transfer them between nodes via RDMA. These solutions are designed for data center clusters, relying on RDMA network cards and dedicated metadata services. Key-value cache indexing, matching, and eviction are all handled by host software, continuously consuming host processor and memory resources. They are unsuitable for single-node, constrained environments without RDMA networks or with limited host memory. Compute storage disk solutions integrate general-purpose FPGAs and other computing resources within SSDs, offloading some attention computations to the disk for execution. In this type of solution, general-purpose FPGAs are expensive and consume a lot of power. Furthermore, the on-board computing resources and external inference chips operate independently, lacking unified object scheduling and end-to-end direct data paths, which limits the overall efficiency of the system.

[0004] In-memory computing chips perform matrix multiplication and addition operations within the storage array, exhibiting high energy efficiency and low power consumption, making them suitable for inference computation in constrained environments. However, the local memory capacity of such chips is limited, unable to accommodate the weight data of models with hundreds of billions of parameters, necessitating efficient collaboration with high-capacity storage media. On the other hand, solid-state drive (SSD) controller chips possess multi-channel flash memory management, direct memory access, and a certain scale of on-chip computing power. However, existing SSD controllers only provide block-level read / write interfaces and lack object-level cache management capabilities for large language model inference.

[0005] Therefore, how to enable in-memory computing inference chips and solid-state drives to work together efficiently in a limited environment where there is no RDMA network and the host memory and chip local memory are both much smaller than the total number of model weights, to run large language models with hundreds of billions of parameters and reduce the first token latency in long context scenarios, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a large language model inference system and method for resource-constrained environments. It offloads key-value cache management and weight scheduling from the host software to the hardware logic of the solid-state drive controller chip, and establishes an end-to-end direct data path between the solid-state drive and the in-memory computing inference chip without going through the host, thereby enabling the low-power operation of large language models with hundreds of billions of parameters in resource-constrained environments.

[0007] In a first aspect, the present invention provides a large language model reasoning system under resource-constrained environments, comprising: The host computer runs the control plane program, which includes inference request scheduling, word segmentation, and sampling output. The in-memory computing inference chip includes an in-memory computing array and local memory. The in-memory computing array performs incremental computation on missed suffix tokens using in-memory computation. The local memory stores the weight data of the currently active layer and hot key-value cache blocks, and its address window is mapped to a target space that can be directly addressed by the intelligent solid-state drive controller chip.

[0008] The intelligent solid-state drive controller chip connects the host, the in-memory computing inference chip, and the NAND flash memory array, and includes a cache management engine, a prefetch scheduling engine, and a direct DMA engine.

[0009] The cache management engine is used for unified object addressing, index maintenance, and hit determination of weighted data objects and key-value cache block objects stored in the NAND flash array. It includes a prefix hash matching unit and a lease buffer management unit. The prefix hash matching unit compares chained prefix hash values ​​in hardware parallel and determines the longest reusable prefix. The lease buffer management unit manages direct-transmission data using buffer slots with lease time limits and automatically reclaims timeout slots.

[0010] Preferably, the chained prefix hash value is calculated as follows: the token sequence is divided into blocks of fixed length, and the hash value of the i-th block is calculated by combining the token content of the i-th block with the hash value of the (i-1)-th block. Thus, the hash value of any prefix block uniquely identifies the complete prefix from the beginning to the block.

[0011] Preferably, the method for determining the longest reusable prefix is ​​as follows: compare sequentially from the longest prefix block to the shortest prefix block, and the first match is the longest reusable prefix.

[0012] The prefetching scheduling engine prefetches data objects based on the inference calculation of the schedule weight data objects.

[0013] Preferably, the prefetching scheduling engine includes an expert prediction unit and an inter-layer pipeline scheduling unit, which performs predictive prefetching on the expert weights of the hybrid expert model and enables the three pipelines of computation, prefetching and sinking to run in parallel.

[0014] The direct-transfer DMA engine moves data between the NAND flash array and the local memory of the in-memory computing inference chip via an end-to-end direct data path, without passing through the host's processor and memory. The host submits object-level read and write requests only through the intelligent solid-state drive controller chip. Prefix matching, sinking, eviction, and prefetch scheduling of key-value cache blocks are all completed on the storage side by the intelligent solid-state drive controller chip.

[0015] Preferably, the end-to-end direct data path is a PCIe point-to-point path or a dedicated serial high-speed link.

[0016] Preferably, the cache management engine actively queries the host or the in-memory computing inference chip for newly generated key-value cache blocks at predetermined intervals, and pulls the key-value cache blocks from local memory via the direct transfer DMA engine and asynchronously writes them to the NAND flash memory array; the asynchronous writing is performed in parallel with the inference computation and does not occupy the host's processor resources.

[0017] Preferably, the cache management engine further includes a waterline eviction unit, which is used to select key-value cache block objects to be evicted according to the first-in-first-out or least recently used strategy when the usage of the key-value cache area of ​​the NAND flash array exceeds a preset high waterline, first update the index table to make the objects to be evicted unavailable, and then delete the corresponding flash data until the usage drops to a preset low waterline.

[0018] Preferably, the intelligent solid-state drive controller chip further includes a quantization compression unit, which performs INT8 or INT4 quantization compression before the key value cache block is written to the NAND flash memory array.

[0019] Secondly, this invention provides a large language model reasoning method under resource-constrained environments, applied to the aforementioned system, comprising the following steps: Step 1: The host receives the inference request and generates a token sequence through word segmentation.

[0020] Step 2: The host calculates the chained prefix hash value for each token block and submits a prefix matching query to the smart solid-state drive controller chip via the host interface. If a match is found, proceed to Step 3; otherwise, proceed to Step 4.

[0021] Step 3: The direct transfer DMA engine directly transfers the hit key-value cache block to the local memory of the in-memory computing inference chip. The in-memory computing array then performs incremental calculations on the missing suffix tokens, proceeding to step 5.

[0022] Step 4: The in-memory computing array performs a complete calculation on all tokens, then proceeds to Step 5.

[0023] Step 5: The in-memory computing inference chip decodes token by token to generate results and writes the newly generated key-value cache block into local memory.

[0024] Step 6: The intelligent solid-state drive controller chip actively queries and retrieves newly generated key-value cache blocks according to a predetermined cycle, and after being compressed by the quantization compression unit, it is asynchronously written to the key-value cache area of ​​the NAND flash memory array.

[0025] Step 7: When the usage of the key-value cache exceeds the high watermark, the watermark eviction unit of the cache management engine reduces the usage to the low watermark according to the eviction policy, and updates the index before deleting the data.

[0026] Step 8: Transmit the decoded result to the host for output.

[0027] The present invention has the following beneficial effects: 1. Weighted data and key-value cached data are transferred directly between flash memory and inference chip local memory via an end-to-end data path, without passing through host memory, eliminating the bandwidth bottleneck of host software stack and PCIe transfer, and significantly reducing host processor usage. 2. The indexing, prefix matching, sinking and eviction of key-value cache are completed by the hardware logic of the solid-state drive controller chip, with minimal consumption of host memory and processor resources. The system can run in a limited environment where the host memory does not exceed one-quarter of the total weight. 3. It does not rely on RDMA networks and cluster deployment; a single node can work. 4. When the prefix is ​​hit, the key-value cache block can be reused and directly passed to local memory, which greatly reduces the latency of the first token in multi-turn dialogues and long context scenarios; 5. The combination of the low power consumption of in-memory computing chips and the high capacity and low cost of solid-state drives enables models with hundreds of billions of parameters to be deployed in environments with limited power consumption and size, such as ruggedized terminals, vehicle-mounted and airborne equipment, and edge nodes. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a large language model inference system architecture under a resource-constrained environment. Figure 2 A schematic diagram of the intelligent solid-state drive controller chip provided in the embodiment; Figure 3A schematic diagram illustrating the three-stage pipeline timing of computation, prefetching, and sinking in the prefetching scheduling engine; Figure 4 This is a flowchart of a large language model reasoning method under a resource-constrained environment. Detailed Implementation

[0029] The present invention will be further explained below with reference to the accompanying drawings; To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0030] In this invention, key-value cache (KV Cache) refers to the cached data of key and value matrices generated by the attention mechanism during the inference process of a large language model; prefix reuse refers to reusing the key-value cache corresponding to the prefix when the token sequences of different inference requests have the same prefix to avoid duplicate calculations; hybrid expert model refers to a model structure in which each layer contains multiple expert sub-networks and some experts are activated by the router.

[0031] The restricted resource environment is a single-node environment, which does not have an RDMA network, and the memory capacity of the host does not exceed one-quarter of the total weighted data.

[0032] like Figure 1 As shown, a large language model inference system under a resource-constrained environment includes a host, an in-memory computing inference chip, an intelligent solid-state drive controller chip, and a NAND flash memory array.

[0033] The host only runs control plane programs, including inference request scheduling, word segmentation and sampling output, and does not handle data plane transfer.

[0034] The in-memory computing inference chip includes an in-memory computing array and local memory. The in-memory computing array performs matrix multiplication and addition operations on the model using in-memory computing. The capacity of the local memory is smaller than the total weight data of the large language model, and it is used to store the weight data of the currently active layer and hot key-value cache blocks. Its address window is mapped to a target space that can be directly addressed by the intelligent solid-state drive controller chip.

[0035] The NAND flash memory array is divided into a weighted storage area and a key-value cache area.

[0036] like Figure 2 As shown, the intelligent solid-state drive controller chip connects to the host via a host interface, receiving object-level read / write requests from the host; it establishes an end-to-end direct data path with the in-memory computing inference chip. It includes a cache management engine, a quantization and compression unit, a prefetch scheduling engine, and a direct DMA engine.

[0037] The cache management engine includes a unified object addressing and indexing table, a prefix hash matching unit, a lease buffer management unit, and a waterline eviction unit. The unified object addressing and indexing table addresses weighted data objects and key-value cache block objects in the same object namespace. Index entries include object type fields, hash identifier fields, flash memory location fields, and status fields. Read-only and preservation policies are applied to weighted data objects, while writable and eviction policies are applied to key-value cache block objects. The prefix hash matching unit compares the chained prefix hash values ​​submitted by the host with the hash identifiers in the index table in parallel, determining the longest reusable prefix. The lease buffer management unit allocates buffer slots with lease time limits for each successful read, automatically reclaiming them upon timeout. The waterline eviction unit evicts objects to the low waterline when the key-value cache usage in the NAND flash array exceeds the high waterline, using a first-in-first-out (FIFO) or least recently used (LRU) policy. It first updates the index to make the object to be evicted unavailable before deleting the flash data.

[0038] The metadata of the unified object addressing and index table is persistently stored in the NAND flash array. After the system restarts, the metadata in the NAND flash array can be scanned and the index table can be rebuilt. The key-value cache blocks that were sunk before the restart remain in a hit-able state, avoiding the complete invalidation of the cache due to power failure or maintenance.

[0039] The chained prefix hash value is calculated as follows: the token sequence is divided into blocks of fixed length. The hash value of the i-th block is calculated by combining the token content of the i-th block with the hash value of the (i-1)-th block. Thus, the hash value of any prefix block uniquely identifies the complete prefix from the beginning to that block. The prefix hash matching unit compares the longest prefix block to the shortest prefix block sequentially, and the first match is the longest reusable prefix. This comparison logic can be implemented by content-addressable memory or a parallel comparison array. A single query is completed in microseconds and does not occupy the host processor.

[0040] The prefetch scheduling engine includes an expert prediction unit and an inter-layer pipeline scheduling unit. The expert prediction unit uses a sliding window to statistically analyze the historical routing results of each layer's routers, establishing expert activation and transfer statistics. During the computation of layer n by the in-memory computing inference chip, experts with transfer probabilities exceeding a threshold are selected as prediction targets, generating prefetch instructions. The inter-layer pipeline scheduling unit enables parallel execution of three pipelines between the in-memory computing inference chip and the NAND flash array: computation, prefetching, and sinking. During the computation of layer n by the in-memory computing array, the scheduled direct-transfer DMA engine prefetches the predicted weight data objects of layer n+1 to local memory, while asynchronously sinking the key-value cache blocks of the completed layers to the NAND flash array. When the prediction hit rate is higher than the random hit rate, the decoding pause time is reduced accordingly; even if the prediction fails, the inter-layer pipeline scheduling unit can still revert to on-demand reading without affecting correctness. Prefetching is completed one layer ahead of schedule, and sinking is executed two layers behind schedule, neither of which blocks the computation path, thus masking the difference between flash read bandwidth and computation speed. Figure 3 As shown.

[0041] The quantization compression unit performs INT8 or INT4 quantization compression before the key-value cache block is written to the NAND flash memory array.

[0042] The direct transfer DMA engine performs zero-copy transfer.

[0043] By mapping the address window of the local memory in the in-memory computing inference chip to the target space that can be directly addressed by the direct transfer DMA engine through the cache management engine, the data transfer between the in-memory computing inference chip and the NAND flash array does not pass through the host processor and memory.

[0044] The inference system can be widely used in the localized deployment of large models in restricted environments such as ruggedized computers, vehicle and airborne equipment, edge computing nodes, and AI PCs, and has significant industrial applicability.

[0045] Taking a hybrid expert model with 100B parameters as an example, after INT8 quantization, the weights are approximately 100GB, all residing in the weight storage area. Each token activates approximately 12B parameters, corresponding to approximately 12GB of weight reads per token. Under conditions of sequential read bandwidth of approximately 6.5GB / s and inter-layer pipelined parallel masking, the decoding speed can reach the order of 0.5 to 1 token per second, meeting the availability requirements of scenarios such as offline document processing and intelligence analysis. The key-value cache of the 32K token context is approximately 16GB after INT8 quantization. In multi-turn dialogues, prefix hits are achieved from the second round onwards, and the latency of the first token is reduced from tens of seconds for complete recalculation to less than a second.

[0046] like Figure 4 As shown, a large language model reasoning method under resource-constrained environment includes the following steps.

[0047] Step 1: The host receives the inference request and generates a token sequence through word segmentation.

[0048] Step 2: The host calculates the chained prefix hash value for each token block and submits a prefix matching query to the smart solid-state drive controller chip via the host interface. If a match is found, proceed to Step 3; otherwise, proceed to Step 4.

[0049] Step 3: The direct transfer DMA engine directly transfers the hit key-value cache block to the local memory of the in-memory computing inference chip. The in-memory computing array then performs incremental calculations on the missing suffix tokens, proceeding to step 5.

[0050] Step 4: The in-memory computing array performs a complete calculation on all tokens, then proceeds to Step 5.

[0051] Step 5: The in-memory computing inference chip decodes token by token to generate results and writes the newly generated key-value cache block into local memory.

[0052] Step 6: The intelligent solid-state drive controller chip actively queries and retrieves newly generated key-value cache blocks according to a predetermined cycle, and after being compressed by the quantization compression unit, it is asynchronously written to the key-value cache area of ​​the NAND flash memory array.

[0053] Step 7: When the usage of the key-value cache exceeds the high watermark, the watermark eviction unit of the cache management engine reduces the usage to the low watermark according to the eviction policy, and updates the index before deleting the data.

[0054] Step 8: Transmit the decoded result to the host for output.

Claims

1. A large language model reasoning system under resource-constrained environments, characterized in that: include: The host only runs the control plane program and submits object-level read / write requests through the intelligent solid-state drive controller chip. The in-memory computing inference chip includes an in-memory computing array and local memory. The in-memory computing array is used to perform incremental computation on missing suffix tokens. The local memory is used to store the weight data of the current active layer and hot key-value cache blocks. Its address window is mapped to a target space that can be directly addressed by the intelligent solid-state drive controller chip. The intelligent solid-state drive controller chip connects the host, the in-memory computing inference chip, and the NAND flash memory array to realize prefix matching, sinking, eviction, and prefetch scheduling of key-value cache blocks, including a cache management engine, a prefetch scheduling engine, and a direct transfer DMA engine. The cache management engine is used to perform unified object addressing, index maintenance, and hit determination on weighted data objects and key-value cache block objects stored in the NAND flash array. It includes a prefix hash matching unit and a lease buffer management unit. The prefix hash matching unit compares chained prefix hash values ​​in hardware parallel and determines the longest reusable prefix. The lease buffer management unit manages direct-transmission data with buffer slots with lease time limits and automatically reclaims timeout slots. The prefetching scheduling engine prefetches data objects based on the inference calculation of the schedule weight data objects.

2. The large language model reasoning system under constrained resource environments as described in claim 1, characterized in that: The chained prefix hash value is calculated as follows: the token sequence is divided into blocks of fixed length, and the hash value of the i-th block is calculated by combining the token content of the i-th block with the hash value of the (i-1)-th block. Thus, the hash value of any prefix block uniquely identifies the complete prefix from the beginning to the block.

3. The large language model reasoning system under constrained resource environments as described in claim 2, characterized in that: The prefix hash matching unit receives the chained prefix hash value calculated per token block submitted by the host, and compares the chained prefix hash value with the hash value of the key-value cache block object stored in the index table sequentially from the longest prefix block to the shortest prefix block. When a match is found for the first time, it is determined to be the longest reusable prefix. The direct transfer DMA engine writes the corresponding key-value cache block directly to the local memory through the end-to-end direct data path. The in-memory computing inference chip only performs incremental calculations on the missing suffix tokens.

4. The large language model reasoning system under constrained resource environment as described in claim 1, characterized in that: The cache management engine actively queries the host or the in-memory computing inference chip for newly generated key-value cache blocks at predetermined intervals. The key-value cache blocks are then pulled from local memory by the direct transfer DMA engine and asynchronously written to the NAND flash memory array. The asynchronous writing is performed in parallel with the inference computation and does not occupy the host's processor resources.

5. The large language model reasoning system under constrained resource environment as described in claim 4, characterized in that: The intelligent solid-state drive controller chip also includes a quantization compression unit, which performs INT8 or INT4 quantization compression before the key value cache block is written to the NAND flash memory array.

6. The large language model reasoning system under constrained resource environment as described in claim 1, characterized in that: The cache management engine also includes a waterline eviction unit, which is used to select key-value cache block objects to be evicted according to the first-in-first-out or least recently used strategy when the usage of the key-value cache area of ​​the NAND flash array exceeds a preset high waterline. The index table is updated first to make the object to be evicted unavailable, and then the corresponding flash data is deleted until the usage drops to a preset low waterline.

7. The large language model reasoning system under constrained resource environment as described in claim 1, characterized in that: The unified object addressing and indexing table addresses weighted data objects and key-value cache block objects in the same object namespace. The index table entries include object type field, hash identifier field, flash location field and status field. Read-only and preservation policies are applied to weighted data objects, and writable and eviction policies are applied to key-value cache block objects.

8. The large language model reasoning system under a resource-constrained environment as described in claim 1, characterized in that: The metadata of the unified object addressing and index table is persistently stored in the NAND flash array. After the system restarts, the metadata in the NAND flash array can be scanned and the index table can be rebuilt. The key-value cache blocks that were sunk before the restart remain in a hit-able state, avoiding the complete invalidation of the cache due to power failure or maintenance.

9. The large language model reasoning system under constrained resource environment as described in claim 1, characterized in that: The prefetching scheduling engine includes an expert prediction unit and an inter-layer pipeline scheduling unit. The expert prediction unit uses a sliding window to statistically analyze the historical routing results of each layer router, establishes expert activation transfer statistics, and selects experts whose transfer probability exceeds the threshold as prediction objects during the calculation of the nth layer by the in-memory computing inference chip, and generates prefetching instructions. The inter-layer pipeline scheduling unit is used to realize the parallel execution of three pipelines: computation, prefetching, and sinking between the in-memory computing inference chip and the NAND flash array. During the computation of the nth layer by the in-memory computing array, the scheduled direct transfer DMA engine prefetches the predicted weight data object of the n+1th layer to local memory, and at the same time, asynchronously sinks the key-value cache block of the completed layer to the NAND flash array.

10. A method for reasoning about a large language model under resource-constrained environments, characterized in that: The system applied to any one of claims 1 to 9 includes the following steps; Step 1: The host receives the inference request and generates a token sequence through word segmentation; Step 2: The host calculates the chained prefix hash value based on the token block and submits the prefix matching query to the smart solid-state drive controller chip through the host interface; if a match is found, proceed to step 3; otherwise, proceed to step 4. Step 3: The direct transfer DMA engine directly transfers the hit key-value cache block to the local memory of the in-memory computing inference chip. The in-memory computing array then performs incremental calculations on the missing suffix tokens, proceeding to step 5. Step 4: The in-memory computing array performs a complete calculation on all tokens, then proceeds to Step 5; Step 5: The in-memory computing inference chip decodes each token to generate the result and writes the newly generated key-value cache block into local memory; Step 6: The intelligent solid-state drive controller chip actively queries and retrieves newly generated key-value cache blocks according to a predetermined cycle, and after being compressed by the quantization compression unit, it is asynchronously written to the key-value cache area of ​​the NAND flash memory array. Step 7: When the usage of the key-value cache exceeds the high watermark, the watermark eviction unit of the cache management engine reduces the usage to the low watermark according to the eviction policy, and updates the index before deleting the data. Step 8: Transmit the decoded result to the host for output.

Images