A large language model low-latency inference method based on dynamic inference graph optimization

Abstract

The application discloses a large language model low-delay inference method based on dynamic inference graph optimization, and proposes a low-delay inference method based on dynamic inference graph optimization; a rewritable and replayable template inference graph is constructed, and a template library is established according to an input shape vector; during inference, a template is matched according to a distance threshold value, and only attention / forward subgraphs are locally recaptured when the threshold value is exceeded; a forward execution graph injects a key-value cache page pointer, a numerical precision and an adapter identifier, and is replayed; pre-populated and decoded subgraphs are divided to implement a graph-level scheduling; a key operator is switched between a standard / quick kernel and different precisions during operation; a page-level rollback of a speculative branch is realized through a shadow page table and a reference count, and a batch and a template selection are adaptively adjusted based on online indicators; compared with existing schemes, the application reduces recapture and startup overhead, suppresses tail delay and jitter, and improves hardware utilization and service stability.

Description

Technical Field

[0001] This invention belongs to the field of deep learning model inference optimization technology, and specifically relates to a low-latency inference method for large language models based on dynamic inference graph optimization. Background Technology

[0002] In recent years, generative large-scale language models have made breakthroughs in fields such as dialogue, code generation, and natural language processing, with their scale and performance continuously increasing. However, large models have a huge number of parameters, and the inference process (i.e., generating output) often faces high latency bottlenecks. LLMs typically use an autoregressive approach to generate words one by one, and each token generation requires a complete forward propagation. This means that the model needs to repeatedly load huge weights and compute all layers, making the inference speed extremely slow. For example, a sentence containing 12 tokens requires the model to run the forward network 12 times, processing a massive number of parameters each time, resulting in a large amount of repetitive computation and I / O overhead. As the model size increases, this stepwise sequential inference leads to a sense of lag when using it.

[0003] Existing technologies improve LLM inference from the perspectives of cache reuse, graph computation optimization, dynamic execution, and parallel decoding, partially alleviating the high latency problem of large models. However, these methods each have their limitations: static graph optimization is difficult to handle dynamic branches and parameter changes that occur during inference; relying solely on hardware acceleration and operator fusion still leaves kernel idle time and wasted memory bandwidth; speculative decoding requires careful design of two sets of models, and the low acceptance rate of the draft model leads to invalid computation; while dynamic jump execution requires a prediction module, increasing system complexity; in addition, when users want to temporarily customize model behavior (e.g., inject domain knowledge or style preferences), existing inference frameworks have difficulty updating model parameters online and can only complete the task offline through fine-tuning training, which cannot meet real-time requirements.

[0004] Given the above shortcomings, a comprehensive low-latency inference scheme is urgently needed: it should be able to fully utilize graph optimization techniques to reduce operator-level overhead, have dynamic adaptive capabilities to adjust the computation path based on input or historical states, incorporate speculative parallelism to reduce the number of sequence dependency steps, and allow online parameter injection to achieve rapid customization of model behavior. To this end, this application proposes a low-latency inference method for large language models based on dynamic inference graph optimization, in order to further reduce the inference latency of large models, improve hardware utilization, and enhance system flexibility. Summary of the Invention

[0005] To achieve the above objectives, this invention provides a low-latency inference method for large language models based on dynamic inference graph optimization. This method constructs a dynamically adjustable model inference computation graph and combines multiple optimization strategies to achieve low-latency, high-efficiency inference for large models, including:

[0006] Template graph construction steps: Construct a template inference graph that supports rewriting and replay, where nodes represent operator instances and edges represent data dependencies and scheduling dependencies;

[0007] Template library creation steps: Store the template inference graphs according to the input shape vector v = (b, t, d) in the shape buckets corresponding to the template graph library, and reserve placeholders for injecting running parameters in each template inference graph;

[0008] Template matching and generation steps: During model inference, the distance metric is calculated in the template graph library based on the input shape vector of the current inference request, and the template inference graph with the smallest distance is obtained; if the calculated minimum distance is less than a preset threshold δ, the template inference graph is selected and an execution graph for the current input is generated accordingly; otherwise, while keeping the topology of the template inference graph unchanged, local graph capture and update are performed to generate an execution graph that matches the current input.

[0009] Parameter injection step: Before executing the execution graph, inject the corresponding running parameters into the execution graph, including the starting address of the KV memory page, the numerical calculation precision flag, and the adapter model handle, wherein the adapter model handle is used to load the pre-configured low-rank adaptive model parameters;

[0010] Graph execution steps: Execute the execution graph after injecting runtime parameters to complete model inference, and establish the binding relationship between operator nodes and KV memory pages in the execution graph; after the execution graph inference is completed, release the KV memory pages bound to each operator node according to the topology of the execution graph and the reference count of the KV memory pages.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] (1) Template matching is performed based on distance metric and threshold δ of v=(b,t,d); when no match is found, attention and MLP subgraphs are locally recaptured and replaced, keeping the backbone topology and interface unchanged, which significantly reduces the cost and jitter of whole graph reconstruction.

[0013] (2) Inject the KV page start address, precision flag and adapter handle before execution to decouple the graph structure from the runtime state, which facilitates rapid instantiation and on-demand optimization;

[0014] (3) The execution graph is divided into pre-filling / decoding and graph slicing is performed. The graph-level scheduler performs overlapping calculations and merges similar operators at each autoregressive step to improve the parallelism of the device and reduce the tail latency.

[0015] (4) Based on online performance feedback, the system dynamically switches between standard core, fast core, FP16 and FP8. For bandwidth-limited operators, high-performance and low-precision implementation is given priority, while high precision is maintained for key operators, taking into account both speed and numerical stability.

[0016] (5) Paging and page table mapping are used to manage key-value pairs. Combined with shadow page tables, constant-level rollback and precise release of speculative branches are achieved, which significantly reduces memory fragmentation and ensures data consistency. Attached Figure Description

[0017] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0018] Figure 1 This is a flowchart of the system workflow of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way; it should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention; these all fall within the protection scope of the present invention.

[0020] like Figure 1 As shown, the present invention provides a low-latency inference method for large language models based on dynamic inference graph optimization, comprising the following steps:

[0021] Template graph instantiation steps: Upon receiving a user's inference request, the system first instantiates the template graph under the control of the template graph management module (also known as the dynamic inference graph management module). Using a pre-built model template computation graph as a blueprint, an execution graph is instantiated based on the specific parameters of the current request. For example, considering the actual sequence length of the input Prompt, batch size, and currently enabled model components (such as whether LoRA fine-tuning weights are loaded), the template graph management module configures or trims the dynamic parts of the template to generate a dedicated computation graph tailored to this request. Template matching, distance measurement, and threshold determination follow the principles described in the invention. The input shape vector... Center of each shaped barrel Calculate the weighted 11 or L2 distance when Instantiation is done directly; the bucket center can be obtained from the mode or weighted mean of historical requests, and the threshold... Settings can be configured via offline grid search; among which Indicates the batch size. Indicates the sequence length or number of time steps. This represents the dimension of the feature vector.

[0022] Preferably, the instantiation phase can also incorporate operator fusion optimization based on the target hardware: While maintaining equivalence, adjacent operators are merged into a single kernel to reduce the overhead of loading and writing back intermediate data; before execution, runtime parameters are injected into the execution graph, including the starting address of the KV memory page, numerical calculation precision flags, and adapter model handles, thereby establishing the binding relationship between operator nodes and the KV pages; after instantiation, the resulting inference execution graph maintains the overall model structure and optimization strategy, and is subsequently executed by the graph-level scheduling module. To facilitate scheduling and concurrency, the execution graph can be divided into pre-filled subgraphs and decoding subgraphs without changing the external interface, and sliced ​​by layer or operator group for segmented submission and interleaved execution.

[0023] Preferably, when the features of the current request differ from those of previous requests, making it impossible for existing templates to directly match, the template graph management module will trigger a local recapture process as needed. While maintaining the main topology and external interfaces of the template inference graph, it will only re-record the actual computational operation sequence for the attention mechanism subgraph and the multilayer perceptron (MLP) subgraph, generating new subgraphs with the same interfaces. Subsequently, the subgraph will undergo necessary optimization (such as operator fusion, constant folding, etc.) and be embedded in a reserved replacement position, thereby updating the execution graph of this instantiation. Through this local recapture, an optimized graph matching the current input can be obtained promptly without reverting to a step-by-step interpretation execution mode, which is crucial for maintaining low latency. For example, in a dialogue inference, if a user request triggers a previously unused attention or MLP variant path, and the system recognizes that the template cannot be directly matched, it will recapture, optimize, and replace the corresponding subgraph, enabling subsequent computations on that path to be executed efficiently in the form of an optimized graph.

[0024] The graph-level scheduling module takes over the execution process steps as follows: The scheduling module parses the dependencies of the entire inference computation graph and puts the operator nodes into the scheduling queue in topological order; combined with the current system resource status, it groups and schedules the operators in the queue in parallel; for example, it allocates operators without data dependencies to different GPU streams or different devices (CPU / GPU) for parallel execution; for operators with high computational costs such as matrix multiplication, if there are multiple requests for the same type of operator waiting to be executed, it attempts to merge them into a larger batch for unified computation; in each token generation iteration of the autoregressive algorithm, the scheduling module performs graph-level scheduling once to make the computation required for the new token overlap with the computation of the previous token as much as possible, reducing idle waiting time; at the same time, it dynamically adjusts the execution priority and interleaving order of the pre-filled / decoding subgraph and its slices based on the system load to form a functionally consistent graph-level scheduling strategy.

[0025] In concurrent scenarios, the graph-level scheduling module maintains the computation graph of all active inference sessions and dynamically improves throughput using a continuous batch processing strategy. Specifically, when the system serves multiple requests simultaneously, and some of these requests need to generate the next token, the scheduling module merges similar computations of these requests and processes multiple sequences in a single GPU scheduling, thereby improving the parallelism of matrix operations and hardware utilization. Completed requests are removed from the queue, and new requests are inserted into the corresponding positions according to rules when they arrive. Through this elastic scheduling, GPU / CPU resources are fully utilized, achieving high-throughput service while ensuring low latency. The system collects metrics such as latency and utilization online and adaptively adjusts the continuous batch size and template selection priority accordingly to stabilize overall performance and latency targets.

[0026] Kernel / Precision Dynamic Switching Steps: During inference execution, the kernel / precision switching module works in conjunction with the scheduling module. Before operator submission, based on online performance metrics (such as previous iteration latency, operator execution time, GPU utilization) and algorithm precision requirements, it selects the appropriate implementation method for execution. It maintains and optimizes the kernel library for different hardware platforms, including standard kernel implementation, fast kernel implementation, FP16 implementation, and FP8 implementation. On NVIDIA GPUs, fusion / tensor kernel optimization can be used, and on CPUs, SIMD parallel optimization can be used, but it is not limited to these. When certain operators are found to be bottlenecks and precision tolerance allows, a higher-performance low-precision kernel is prioritized to reduce bandwidth consumption and accelerate computation; for operators with high numerical stability requirements, a high-precision kernel is consistently used to ensure stability.

[0027] KV Cache Storage and Management Steps: During the autoregressive generation process, for each new Token generated, the attention mechanism at each layer generates a new Key / Value (K / V) vector, which needs to be concatenated with the existing cache to participate in the next calculation. The KV cache management module plays a role in each iteration of the graph-level scheduling, ensuring efficient access to KV data and reasonable memory utilization. During system initialization, a certain number of KV cache page frames (fixed page size, capable of holding a vector of several Tokens) are pre-allocated to each attention layer. When processing an inference request, the required free pages are allocated and a page table is established to record the mapping relationship of KV pages at each layer for the request. As generation progresses, when the t-th Token is generated, the module writes the K / V vector of the Token to the corresponding offset position of the corresponding layer's page frame. After the execution graph inference is completed, based on the topology of the execution graph and the reference count of the KV pages, the KV pages bound to the operator nodes are released in an orderly manner, and pages not referenced by other nodes are recycled to the free list. Old Tokens are evicted when the request ends or the context window slides out. When the relevant pages are marked as free and added to the free list for reuse, memory fragmentation is avoided. To support speculative decoding and rollback, the module maintains a shadow page table: during the speculation phase, the key-value pairs of candidate tokens are written to the shadow page without modifying the main page table; when a candidate is accepted, the new mapping corresponding to the shadow page table is submitted to the main page table; if a candidate is rejected, the shadow page table and its associated pages are discarded directly without affecting the original content of the main page table; with the help of the shadow page table, rollback does not require element-by-element restoration, only the corresponding mapping needs to be canceled and the occupied pages released to efficiently restore the consistent state before speculation.

[0028] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A low-latency inference method for large language models based on dynamic inference graph optimization, characterized in that, Includes the following steps: Template graph construction steps: Construct a template inference graph that supports rewriting and replay, where nodes represent operator instances and edges represent data dependencies and scheduling dependencies; Template library creation steps: Store the template inference graphs according to the input shape vector v = (b, t, d) in the shape buckets corresponding to the template graph library, and reserve placeholders for injecting running parameters in each template inference graph; Template matching and generation steps: During model inference, the distance metric is calculated in the template library based on the input shape vector of the current inference request, and the template inference graph with the smallest distance is obtained; If the calculated minimum distance is less than the preset threshold δ, the template inference graph is selected and an execution graph for the current input is generated accordingly; otherwise, while keeping the topology of the template inference graph unchanged, its local graph capture is updated to generate an execution graph that matches the current input. Parameter injection step: Before executing the execution graph, inject the corresponding running parameters into the execution graph, including the starting address of the KV memory page, the numerical calculation precision flag, and the adapter model handle, wherein the adapter model handle is used to load the pre-configured low-rank adaptive model parameters; Graph execution steps: Execute the execution graph after injecting runtime parameters to complete model inference, and establish the binding relationship between operator nodes in the execution graph and the KV memory pages; after the execution graph inference is completed, release the KV memory pages bound to each operator node according to the topology of the execution graph and the reference count of the KV memory pages; In the input shape vector v = (b, t, d), b represents the batch size, t represents the sequence length or time step, and d represents the feature vector dimension. When no template inference graph that meets the threshold δ is matched, the local graph capture update performed on the template inference graph includes updating only the attention mechanism subgraph and the multilayer perceptron subgraph, while keeping the rest of the topology of the template inference graph unchanged.

2. The low-latency inference method for large language models based on dynamic inference graph optimization according to claim 1, characterized in that: The execution graph includes a speculative decoding branch; When the speculative decoding fails, the KV memory pages occupied by the speculative decoding branch are released according to the mapping relationship between the operator node and the KV memory page, based on the pre-maintained shadow page table and reference count, while keeping the execution graph trunk topology unchanged.

3. The low-latency inference method for large language models based on dynamic inference graph optimization according to claim 1, characterized in that: The execution graph is divided into a pre-filling stage subgraph and a decoding stage subgraph, and the execution priority and graph slicing method are selected according to the system load to implement a graph-level scheduling strategy.

4. The low-latency inference method for large language models based on dynamic inference graph optimization according to claim 1, characterized in that: At least one operator node in the execution graph has multiple implementation methods, including: standard core implementation, fast core implementation, FP16 implementation and FP8 implementation; the implementation method of the operator node is dynamically switched according to a preset latency and utilization feedback function.

5. The low-latency inference method for large language models based on dynamic inference graph optimization according to claim 1, characterized in that: The online data acquisition and inference process includes two performance metrics: latency and GPU utilization. These performance metrics are fed back to adjust the batch size, template inference graph selection, and operator node implementation method to optimize inference execution performance.

Images