High-concurrency large language model high-speed reasoning deployment method

Abstract

The invention relates to the technical field of large language models, in particular to a high-concurrency large language model high-speed reasoning deployment method, which comprises the following steps of: extracting a weight coefficient corresponding to a user identity identifier, an emergency degree numerical value mapped by a request type and a resource occupation value converted by an estimated calculation amount according to the user identity, the request type and the estimated calculation amount; according to the method, the dynamic priority is generated by performing comprehensive operation on the user identity, the request type and the estimated calculation amount, so that differentiated services for reasoning tasks are realized, interactive requests with high urgency degree can bypass batch processing tasks with long time consumption, the response delay and delay jitter of key services are reduced, and the user experience is improved. Meanwhile, according to the complexity of the request text and the geographic coordinates of the user, the task is intelligently routed to the model node with the most suitable scale in the distributed network, so that the wide area network transmission delay is greatly reduced through edge processing, and the computing power waste caused by using a super-large-scale model to process the simple task is also avoided.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to a high-concurrency, high-speed inference deployment method for large language models. Background Technology

[0002] The field of large language model technology involves building, training, and applying deep learning models with massive parameter scales. By pre-training on trillion-byte-level text data through self-supervised learning, models can master language rules, factual knowledge, and preliminary reasoning abilities.

[0003] Current large language model deployment technologies generally employ a single centralized deployment and first-in-first-out batch processing scheduling. Due to a lack of awareness of request heterogeneity, a real-time interactive request with low computational load and high response requirements can easily be blocked for a long time by a long document summarization task that arrives earlier. For example, a user's spell check request may have to wait for several seconds or even longer simply because a long report is being generated ahead of the queue. This leads to a severe disconnect in user experience and significant fluctuations in response latency. Furthermore, all requests need to be transmitted over long distances to the core data center, and the resulting wide area network latency is a significant performance bottleneck for users in geographically remote areas. Moreover, using an all-encompassing, ultra-large-scale model to handle all types of tasks, from simple greetings to complex logical reasoning, results in a waste of computing resources. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a high-concurrency, high-speed inference deployment method for large language models.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a high-concurrency, high-speed inference deployment method for large language models, comprising the following steps: Based on user identity, request type, and estimated computational load, extract the weight coefficient corresponding to the user identity identifier, the urgency value mapped to the request type, and the resource consumption value converted from the estimated computational load. Calculate the weight coefficient, urgency value, and resource consumption value, and compare the calculation results with the preset priority threshold to classify the request into the target priority queue and obtain the dynamic priority of the inference task. Based on the number of tokens, syntactic complexity, and geographic coordinates of the user's network access point in the request text, the complexity score of the text is quantified. Then, based on the complexity score and the user's geographic coordinates, a pre-set distributed node routing table is queried. Combining the dynamic priority of the inference task, the student model node of the large language model with the highest confidence is selected. Entities and intents in the request text are extracted on the node, an initial KV cache state is generated, and the target inference node and initial context state are established.

[0006] Preferably, the method further includes the following steps: Based on the target inference node and the initial context state, the dynamic priority value of the new task's inference task is compared with the priority value of the currently running task on the target inference node. If the new task's priority value is higher, the decoding loop of the current task is immediately terminated, and the key tensors and value tensors of all attention layers in the GPU memory are concatenated into a continuous memory block. The direct memory access channel is called to transfer the memory block to the CPU main memory for storage, thereby obtaining the context of the task to be executed and the swap-out cache state. Based on the context of the task to be executed and the swap-out cache state, the context data of the new task is separated from the state, the data is loaded into the GPU memory and decoding is started. After the new task generates an end symbol, the swap-out task identifier recorded in the state is retrieved, the corresponding contiguous memory block in the CPU main memory is retrieved, the original address of the GPU memory is written back through the memory access channel, and the KV cache is continued to be executed from the interrupted decoding position to generate the inference result sequence.

[0007] Preferably, the step of obtaining the dynamic priority of the inference task is as follows: Based on user identity, request type, and estimated computational load, the weight coefficient is queried from the preset user identity weight table, the corresponding urgency value is retrieved from the request type mapping table, and the resource consumption value is calculated in the segmentation function based on the estimated computational load to establish the request metadata quantification value. Based on the requested metadata quantification value, the weight coefficient and the urgency value are multiplied, and then the result is added to the resource consumption value. The sum is input into the activation function for non-linear transformation, compressing the multi-dimensional input into a single scalar value to obtain the original request priority score. Based on the original priority score of the request, it is compared one by one with the preset priority thresholds arranged from smallest to largest. If the original priority score of the request is less than a certain threshold, the comparison stops and the request is assigned to a specific priority queue of the interval to which the original priority score of the request belongs, so as to obtain the dynamic priority of the inference task.

[0008] Preferably, the steps for obtaining the target inference node and the initial context state are as follows: Based on the number of tokens in the request text, the complexity of the syntactic structure, and the geographical coordinates of the user's network access point, the tokens in the text are counted, a dependency syntax tree is constructed and its maximum depth is measured, the network topology is queried to calculate the estimated network latency between the user and each node, and the three factors are weighted and summed to obtain a comprehensive score for the text complexity. Based on the comprehensive text complexity score and combined with the dynamic priority of the inference task, the comprehensive text complexity score and geographical coordinates are used as a joint index in the pre-set distributed node routing table to retrieve all nodes that meet the conditions, and the large language model student model node that best matches the model capability and task complexity is selected from them to obtain the selected inference node. Based on the selected inference node, a lightweight feedforward network is run on the node to process the embedding vector of the input request text, identify named entities and user intent within the text, and encode the information into initial key tensors and value tensors to establish the target inference node and initial context state.

[0009] Preferably, the steps for obtaining the context of the task to be executed and the state of the swapped-out cache are as follows: Based on the target inference node and the initial context state, extract the dynamic priority value of the inference task from the metadata of the new task, and subtract it from the priority value of the currently running task on the target inference node. If the difference is positive, send a hardware interrupt signal to the GPU kernel to generate a preemptive execution instruction. According to the preemptive execution instruction, the GPU kernel immediately pauses the decoding loop of the current task and traverses the key tensors and value tensors of all attention layers in the GPU memory. It then splices them together along the sequence length dimension in the order of the layers to form a physically continuous data structure in memory, namely a continuous cache block to be unloaded.

[0010] Preferably, the step of obtaining the context of the task to be executed and the state of the swapped-out cache further includes: Based on the contiguous cache block to be unloaded, the direct memory access controller is invoked, and the starting physical address and length of the contiguous cache block to be unloaded are used as parameters to initiate a block transfer operation from GPU memory to CPU main memory, and the task identifier and memory address are recorded to obtain the context of the task to be executed and the swap-out cache status.

[0011] Preferably, the step of obtaining the inference result sequence is as follows: Based on the context of the task to be executed and the state of the swapped-out cache, the data structure of the state is parsed, the context data of the new task is separated, the context data is loaded into the specified working area of ​​the GPU memory, and the decoder is started to generate words one by one until the model outputs the preset end symbol to obtain the output of the high-priority task. Based on the high-priority task output, the previously recorded swap-out task identifier and CPU main memory address are read from the metadata fields of the task context and swap-out cache status. The corresponding contiguous memory block is located in the CPU main memory based on the CPU main memory address, and a cache data block to be restored is established.

[0012] Preferably, the step of obtaining the inference result sequence further includes: The cached data block to be recovered is written from the CPU main memory back to the original address recorded in the GPU video memory, and the internal state of the decoder is restored. The KV cache is then executed again from the interrupted decoding position to generate the inference result sequence.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are as follows: This invention generates dynamic priorities by comprehensively calculating user identity, request type, and estimated computational load, enabling differentiated services for inference tasks. This allows high-urgency interactive requests to bypass time-consuming batch processing tasks, reducing response latency and latency jitter in critical business operations. Simultaneously, based on request text complexity and user geographic coordinates, tasks are intelligently routed to the most appropriately sized model nodes in the distributed network. This not only significantly reduces WAN transmission latency through edge processing but also avoids the waste of computing power caused by using ultra-large-scale models to process simple tasks, improving overall throughput and resource utilization efficiency. When high-priority requests arrive, the KV cache state of low-priority tasks is quickly unloaded from GPU memory to CPU main memory, achieving immediate preemption of computing resources and seamless context switching. This interruption and recovery mechanism solves the task blocking problem under the first-in-first-out scheduling model, ensuring service quality stability in high-concurrency scenarios. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the steps of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0016] Please see Figure 1 This invention provides a technical solution, a high-concurrency, high-speed inference deployment method for large language models, comprising the following steps: Based on user identity, request type, and estimated computational load, extract the weight coefficient corresponding to the user identity identifier, the urgency value mapped to the request type, and the resource consumption value converted from the estimated computational load. Calculate the weight coefficient, urgency value, and resource consumption value, and compare the calculation results with the preset priority threshold to classify the request into the target priority queue and obtain the dynamic priority of the inference task. Based on the number of tokens, syntactic complexity, and geographic coordinates of the user's network access point in the request text, the complexity score of the text is quantified. Then, based on the complexity score and the user's geographic coordinates, a pre-set distributed node routing table is queried. Combined with the dynamic priority of the inference task, the student model node of the large language model with the highest confidence is selected. Entities and intents in the request text are extracted on the node, an initial KV cache state is generated, and the target inference node and initial context state are established. Based on the target inference node and the initial context state, the dynamic priority value of the new task's inference task is compared with the priority value of the currently running task on the target inference node. If the new task's value is higher, the decoding loop of the current task is immediately terminated, and the key tensors and value tensors of all attention layers in the GPU memory are concatenated into a continuous memory block. The direct memory access channel is called to transfer the memory block to the CPU main memory for storage, thus obtaining the context of the task to be executed and the swap-out cache state. Based on the context of the task to be executed and the swap-out cache state, the context data of the new task is separated from the state, the data is loaded into the GPU memory and decoding is started. After the new task generates the end symbol, the swap-out task identifier recorded in the state is retrieved, the corresponding contiguous memory block in the CPU main memory is retrieved, the original address of the GPU memory is written back through the memory access channel, and the KV cache is continued to be executed from the interrupted decoding position to generate the inference result sequence.

[0017] The steps for obtaining the dynamic priority of the inference task are as follows: Based on user identity, request type, and estimated computational load, the weight coefficient is queried from the preset user identity weight table, the corresponding urgency value is retrieved from the request type mapping table, and the resource consumption value is calculated in the segmentation function based on the estimated computational load to establish the request metadata quantification value. Based on the quantified value of the request metadata, the weight coefficient is multiplied by the urgency value, and then the result is added to the resource consumption value. The sum is input into the activation function for non-linear transformation, compressing the multi-dimensional input into a single scalar value to obtain the original score of the request priority. Based on the original priority score of the request, it is compared one by one with the preset priority thresholds arranged from smallest to largest. If the original priority score of the request is less than a certain threshold, the comparison stops and the request is assigned to a specific priority queue in the interval to which the original priority score of the request belongs, so as to obtain the dynamic priority of the inference task.

[0018] Specifically, based on user identity, request type, and estimated computational load, the system first accesses a user identity weight table pre-configured for different user categories. This table is constructed by analyzing data such as users' historical payment records, service level agreements, and platform contributions. For example, the weight coefficient for top-tier members with annual subscriptions is set to 1.5, the weight coefficient for ordinary registered users is set to 1.0, and the weight coefficient for unregistered visitor users is set to 0.8. This setting process is based on historical data regression analysis, aiming to give users with higher payments a higher base weight. If a request comes from a top-tier member, its corresponding weight coefficient of 1.5 is extracted. Next, the system queries a request type mapping table. This table maps the real-time requirements of the business scenario to specific urgency values. This mapping relationship is defined by the product operations team based on business priorities. For example, for real-time conversation requests, the urgency value is set to 1.2, and for asynchronous requests such as document summarization or translation... The urgency value for the request is set to 0.9, while for background batch analysis tasks, the urgency value is set to 0.7. Assuming the current request is a real-time dialogue, the urgency value obtained is 1.2. Subsequently, based on the estimated computational load, which is initially estimated from the number of tokens in the request text, a piecewise function is applied to calculate the resource consumption value. This piecewise function is set based on the offline performance analysis results of GPU computing power and memory consumption for inputs of different lengths. Specifically, when the estimated computational load is less than 100 tokens, the resource consumption value is 0.1; when the computational load is between 100 and 1000 tokens, the resource consumption value is 0.5; and when the computational load exceeds 1000 tokens, the resource consumption value is 0.9. Assuming the current request contains 500 tokens, the calculated resource consumption value is 0.5. Finally, the obtained weight coefficient of 1.5, urgency value of 1.2, and resource consumption value of 0.5 are integrated to establish the request metadata quantification value.

[0019] Based on the request metadata quantification value established in the previous step, which includes weight coefficients, urgency values, and resource consumption values, a series of calculations are performed to generate a single priority score. First, a multiplication operation is performed, multiplying the user identity weight coefficient by the urgency value of the request type. Using the previous example, this is equivalent to multiplying 1.5 by 1.2, resulting in an intermediate result of 1.8. This step aims to give high-priority requests from high-weight users a multiplier effect boost. Then, the result of this multiplication operation is added to the estimated resource consumption value converted from computational load, i.e., adding 1.8 to 0.5, resulting in a sum of 2.3. This step introduces resource consumption in an additive form to ensure that the priority of long text requests is not excessively suppressed. Next, to map this sum, which may be distributed over a wide range, to a standardized, bounded priority interval, the sum of 2.3 is input into a preset activation function for non-linear transformation. Here, a parameterized Sigmoid function is chosen, and its expression is: ,in, This is the original score of the output request priority. This is the maximum score, set to 1.0. This is the kurtosis coefficient of the function, set to 0.5 based on the variance of the historical data distribution, to control the sensitivity to changes in score. It is the center point of the function, set to 3.0 based on the historical average and the mean of the values, to center the scores of most requests. The input sum is 2.3. Substituting it into the calculation, the original score of the request priority is approximately 0.33. Through this nonlinear transformation, the input features of different dimensions and scales are effectively compressed and normalized into a single scalar value, thus obtaining the original score of the request priority.

[0020] Based on the calculated raw request priority score, it is compared sequentially with a set of preset priority thresholds arranged in ascending order of value to determine which specific processing queue the request should be assigned to. These priority thresholds are not fixed but dynamically set after statistical analysis of the raw scores of all requests over the past month. Specifically, all historical scores are collected and their probability density distribution is plotted. Then, specific percentiles of the distribution are selected as thresholds. For example, the 20th, 60th, and 90th percentiles of the score distribution are set as low, medium, and high thresholds, respectively, to ensure that approximately 20% of requests enter the low-priority queue, 40% enter the normal queue, 30% enter the high-priority queue, and the remaining... 10% of requests are placed into the emergency queue. Assuming the thresholds set using this method are 0.25, 0.60, and 0.85, then for the original priority score of 0.33 obtained in the previous example, it is first compared with the first threshold of 0.25. Since 0.33 is greater than 0.25, the comparison continues. Then it is compared with the second threshold of 0.60. Since 0.33 is less than 0.60, the comparison stops. Subsequently, based on the position where the comparison stops, the request is assigned to the priority interval to which the original score of 0.33 belongs, that is, the interval greater than or equal to 0.25 and less than 0.60. This interval corresponds to the system's "normal" priority queue. Finally, through this assignment action, the dynamic priority of the inference task is obtained as "normal".

[0021] The steps for obtaining the target inference node and the initial context state are as follows: Based on the number of tokens in the request text, the complexity of the syntactic structure, and the geographical coordinates of the user's network access point, the tokens in the text are counted, a dependency syntax tree is constructed and its maximum depth is measured, the network topology is queried to calculate the estimated network latency between the user and each node, and the three factors are weighted and summed to obtain a comprehensive score for the text complexity. Based on the comprehensive text complexity score and combined with the dynamic priority of the inference task, the comprehensive text complexity score and geographical coordinates are used as a joint index in the pre-set distributed node routing table to retrieve all nodes that meet the conditions, and the large language model student model node that best matches the model capability and task complexity is selected from them to obtain the selected inference node. Based on the selected inference node, a lightweight feedforward network is run on the node to process the embedding vector of the input request text, identify named entities and user intent in the text, and encode the information into initial key tensors and value tensors to establish the target inference node and initial context state.

[0022] Specifically, based on the number of tokens in the request text, the complexity of the syntactic structure, and the geographic coordinates of the user's network access point, the request text is first processed using a byte-pair-based word segmenter. This segmentation decomposes the text into a series of tokens and counts them precisely. For example, a sentence containing 150 words might be decomposed into 200 tokens, resulting in a token count of 200. Simultaneously, to quantify the complexity of the syntactic structure, a pre-trained dependency parsing model based on the Transformer architecture is invoked. This model, fine-tuned on a standard treebank corpus, generates a dependency tree for the input text. The tree structure is then traversed, and its maximum depth is calculated and recorded. For example, for a complex long sentence containing multiple clauses, the maximum depth of its dependency tree might be 15, while a simple sentence might only have a depth of 5. Assuming the current text tree depth is 12, the system then queries the locally cached GeoIP database based on the IP address of the user request, resolving it to specific geographical coordinates (latitude and longitude). Using this coordinate as a starting point, a dynamically updated network topology map is queried. This topology map is maintained by monitoring agents deployed in various data centers periodically executing ping and traceroute commands to record latency data between nodes. The Dijkstra algorithm is used to calculate the estimated network latency from the user access point to each available inference node, resulting in a latency list. For example, the latency to node A is 50ms, and to node B is 120ms. The minimum value of 50ms is taken as the estimated network latency. Finally, to obtain a comprehensive score, these three dimensions are weighted and summed. The calculation method is as follows: ,in A comprehensive score is given for text complexity. , , The weights for the number of lexical units, tree depth, and network latency were determined through multiple linear regression analysis of the inference time of historical tasks and set to 0.4, 0.5, and 0.1, respectively. , , It is a normalized value, obtained by dividing the current value by the historical maximum observation value of each dimension (e.g., the maximum number of lexical units is 4096, the maximum depth is 25, and the maximum latency is 300ms), and then substituting the value into the calculation to obtain the final comprehensive score of text complexity.

[0023] Based on the calculated comprehensive text complexity score and combined with the previously obtained dynamic priorities of inference tasks, a query and selection is performed in a pre-set distributed node routing table. This routing table is a real-time updated hash table stored in a cache. Its key is the node ID, and the value is a structure containing information such as the node's geographical location, current GPU load percentage, memory usage, the type of student model deployed, and its capability range (represented as the range of text complexity scores that can be processed). The data in this table is refreshed every 5 seconds by the monitoring daemon on each computing node. First, nodes are initially filtered using the dynamic priorities of inference tasks. For example, "urgent" and "high" priority tasks have access to all online nodes, while "normal" priority tasks can only access nodes with a current GPU load below 80%, and "low" priority tasks are restricted to nodes with a GPU load below 50%. Assuming the current task is "normal" priority... First, all nodes with a load exceeding 80% are filtered out. Then, a second round of filtering is performed using the user's geographic coordinates to calculate the physical distance between the user and the remaining nodes. Only nodes with a distance less than a preset range (e.g., 3000 kilometers) are retained to ensure a low network round-trip time. In the node candidate pool obtained after two rounds of filtering, the comprehensive text complexity score is used as the final selection criterion. The score value is matched with the capability range of the student model deployed on each candidate node. For example, a student model with 7B parameters is defined as capable of handling tasks with a complexity score between 0.3 and 0.7, while a model with 3B parameters corresponds to a range of 0.0 to 0.4. If the complexity score of the current task is 0.55, then all nodes with a capability range including 0.55 are selected. If multiple nodes meet the conditions, then the node with the lowest current GPU load is selected to determine the optimal match and obtain the selected inference node.

[0024] Based on the selected inference node information, a lightweight feedforward neural network model is executed on that node. This model is specifically designed for quickly parsing the core semantics of user requests. Its network structure includes an input layer that receives 768-dimensional word embedding vectors, two hidden layers containing 256 and 128 neurons respectively, both using the GELU activation function, and two parallel output heads: one for Named Entity Recognition (NER) and the other for Intent Classification. The NER output head is a linear layer whose output dimension equals the product of the vocabulary size and the number of entity labels, followed by a Softmax function for sequence labeling. The intent classification output head is also a linear layer with an output dimension equal to the number of predefined intent categories (e.g., 15 categories), similarly followed by a Softmax function. The tmax function is used in this model, which is trained offline on a dataset containing 500,000 manually labeled samples. The Adam optimizer is employed with a learning rate of 0.0001. The cross-entropy loss of NER and the cross-entropy loss of intent classification are weighted at 0.6 and 0.4 respectively as the total loss function. The training process lasts for 20 epochs. The model version that performs best on the validation set is then deployed. During inference, the embedding vector of the input request text is fed into this network to obtain the named entities (e.g., "Company A", "Project B") and user intent (e.g., "information query") contained in the text. Then, this identified information is encoded into a structured prefix, such as "[INTENT:Information Query][ENTITIES:Company A, Project B”, and this prefix text is computed through the first few layers (e.g., the first 3 attention layers) of the large language model on the selected inference node. This process generates the initial key tensor and value tensor corresponding to the core semantics, and pre-fills them into the KV cache of the GPU memory, thereby establishing a complete state containing the target inference node address and the pre-computed initial context information, namely the target inference node and the initial context state.

[0025] The steps to obtain the context of the task to be executed and the state of the swapped-out cache are as follows: Based on the target inference node and the initial context state, extract the dynamic priority value of the inference task from the metadata of the new task, and subtract it from the priority value of the currently running task on the target inference node. If the difference is positive, send a hardware interrupt signal to the GPU kernel to generate a preemptive execution instruction. According to the preemptive execution instruction, the GPU kernel immediately pauses the decoding loop of the current task and traverses the key tensors and value tensors of all attention layers in the GPU memory. It then splices them together along the sequence length dimension in the order of the layers to form a physically continuous data structure in memory, namely a continuous cache block to be unloaded. Based on the contiguous cache blocks to be unloaded, the direct memory access controller is invoked, taking the starting physical address and length of the contiguous cache blocks to be unloaded as parameters, to initiate a block transfer operation from GPU memory to CPU main memory, and record the task identifier and memory address to obtain the context of the task to be executed and the swap-out cache status.

[0026] Specifically, based on the target inference node and the initial context state, the dynamic priority of the inference task is first parsed from the metadata structure of the new task. This priority was mapped to a specific value in the previous steps; for example, 4.0 corresponds to the emergency queue, 3.0 to the high-priority queue, 2.0 to the normal-priority queue, and 1.0 to the low-priority queue. Assuming the new task's priority is "high," the value 3.0 is extracted. Simultaneously, a task status register maintained on the target inference node is queried. This register records the unique identifier of the task currently executing on the GPU and its corresponding priority value. Assuming the current task's priority is "normal," its value is 2.0. Then, a subtraction operation is performed: the new task's priority value of 3.0 is subtracted from the current task's priority value of 2.0, resulting in a difference of 1.0. This difference is compared with a preset preemption sensitivity threshold, which is determined based on context switching... The latency overhead introduced by operations (switch-out and swap-in) is set after offline benchmarking. The goal is to avoid excessively frequent preemption caused by small priority differences, which could lead to system jitter. For example, if a complete context switch takes an average of 5 milliseconds, the threshold is set to a priority difference that can bring a reduction of at least 10 milliseconds in expected waiting time. This threshold is 0.5 obtained by function fitting. Since the calculated difference of 1.0 is greater than this threshold of 0.5, the preemption condition is determined to be met. Then, a 32-bit preemption instruction word is constructed, where the high 8 bits encode the preemption operation type (e.g., 0x01 represents preemption), and the low 24 bits specify the ID of the target CUDA stream. Then, this instruction word is written to a specific command register in the memory-mapped I / O address space that communicates with the GPU kernel scheduler through the driver interface. This write operation will trigger a hardware interrupt, send a preemption signal to the GPU kernel, and generate a preemption execution instruction.

[0027] Upon receiving the preemptive execution instruction, the GPU's firmware-level interrupt service routine is activated. This routine first queries the opcode in the instruction; upon confirming a preemption request, it immediately sets a "pause" flag in the shared memory region of the CUDA stream associated with the preempted task. The decoding loop CUDA kernel currently executing on this stream is designed to check this flag after each term is generated. Once the flag is detected as set, the kernel stops decoding the next term and saves its current internal state, including the loop counter (i.e., the number of terms generated), random number generator state, etc., to a pre-allocated GPU memory state area, then exits safely. Next, the scheduler launches a dedicated "KV cache-packed" CUDA kernel. This kernel receives an array of pointers as input, containing the addresses of all attention layer key and value tensors of the current task in GPU memory. This array is... The inference engine's memory manager is created and maintained at the start of the task. The packaging kernel first traverses this pointer array, reads the dimension information of each tensor (batch size, number of attention heads, sequence length, head dimension), calculates the total GPU memory size occupied by all Key and Value tensors, and allocates a temporary buffer of the same size with contiguous physical addresses in GPU memory. Then, the kernel allocates each thread block to different attention layers in parallel. Each thread block is responsible for copying the Key and Value tensor data of its corresponding layer from their original scattered storage locations to the contiguous temporary buffer in the order of the layers (from layer 0 to the top layer). During the copying process, Key and Value tensors in the same layer are placed adjacent to each other, and finally a compact and contiguous data block containing the complete context state is formed in the temporary buffer, which is the contiguous cache block to be unloaded.

[0028] Based on the contiguous cache block to be unloaded, the scheduler then calls the underlying driver interface to execute an asynchronous data transfer operation from the GPU to the CPU. First, in the CPU main memory, a pinned memory block of size matching the contiguous cache block to be unloaded is allocated by calling functions such as cudaHostAlloc. Using pinned memory allows the GPU's Direct Memory Access (DMA) engine to read and write directly to it without going through the operating system's memory paging management, thereby improving the transfer rate. Subsequently, the cudaMemcpyAsync function is called, using the starting GPU physical address of the contiguous cache block to be unloaded as the source address, the starting address of the newly allocated CPU pinned memory as the destination address, and specifying the transfer direction as cudaMemcpyDeviceToHost. This call submits a transfer task to the GPU's DMA controller. This task executes on independent hardware and does not consume the computing resources of the CUDA cores, thus initiating the asynchronous transfer. Simultaneously, a new data structure is created on the CPU side to encapsulate the entire context switch state. This structure records the unique identifier of the swapped-out task, the storage address and total byte length of its KV cache data in the CPU page-locked memory, a list containing the addresses of all its original Key tensors and Value tensors in GPU video memory (for subsequent recovery), and the internal state of the decoder at the time of the interruption (such as the number of generated tokens) read back from the GPU state area. At the same time, this structure also contains the initial context state information of the new task to be executed. This complete data structure, which aggregates the swapped-out task information and the information of the task to be executed, is the context of the task to be executed and the swapped-out cache state.

[0029] The steps to obtain the inference result sequence are as follows: Based on the context of the task to be executed and the state of the swapped-out cache, the data structure of the state is parsed, the context data of the new task is separated, the context data is loaded into the specified working area of ​​the GPU memory, and the decoder is started to generate words one by one until the model outputs the preset end symbol, and the high-priority task output is obtained. Based on the output of high-priority tasks, read the previously recorded swap-out task identifier and CPU main memory address from the metadata fields of the context of the task to be executed and the swap-out cache status, and locate the corresponding contiguous memory block in the CPU main memory based on the CPU main memory address to establish the cache data block to be restored. Write the cached data block to be recovered from the CPU main memory back to the original address recorded in the GPU video memory, restore the internal state of the decoder, continue execution of the KV cache from the interrupted decoding position, and generate the inference result sequence.

[0030] Specifically, based on the context of the task to be executed and the swapped-out cache state, the context data belonging to the new task is first accessed and separated from the state data structure. This data contains the initial key and value tensors pre-calculated in previous steps. These tensors may currently be located in a temporary storage area of ​​the GPU. Next, the scheduler starts a lightweight CUDA kernel to copy these initial key and value tensors from the temporary storage area to the standard KV cache working area used by the large language model for inference. This working area is the GPU memory space that was just swapped out by the previous task. After the data is loaded, the main decoder CUDA kernel is immediately started to generate subsequent tokens for the new task. The decoder runs in a loop, performing a complete forward propagation in each loop iteration to calculate the next... The algorithm calculates the probability distribution of the most likely term and determines a term ID through sampling (such as greedy sampling or kernel sampling). Then, it appends the newly generated term ID to the output sequence and updates the key-value cache. This loop continues. At the end of each loop, the newly generated term ID is compared with a preset set of end-of-text IDs. This set is defined during model training and usually contains IDs with special markers such as <|endoftext|>. For example, if the term ID 50256 in the model vocabulary represents an end-of-text character, then each generated ID will be compared with 50256. Once the generated term ID matches any end-of-text character in the set, the decoding loop terminates, and the generated complete term ID sequence is copied from GPU memory back to CPU main memory for high-priority task output.

[0031] Based on the completion signal output by the high-priority task, the inference scheduler determines that the GPU resources are idle and then initiates the recovery process for the interrupted task. It first accesses and parses the metadata fields of the previously created pending task context and swap-out cache status data structure to accurately read information related to the swapped-out task, including the previously recorded swap-out task identifier (such as a UUID string or an integer ID) used to uniquely identify the task, and the specific storage address where the task's KV cache data was unloaded into the CPU main memory. This address is a pointer to a previously allocated page-locked memory region. Then, using this CPU main memory address pointer, it directly locates the contiguous memory block storing all the key tensors and value tensors of the task in the CPU's virtual address space. The content and structure of this memory block are completely consistent with the contiguous cache block to be unloaded formed in the GPU video memory just before the task was interrupted. Through this location operation, a direct reference to the data to be recovered, i.e., the cache data block to be recovered, is established.

[0032] The process of writing the cached data block to be recovered from CPU main memory back to GPU memory begins by calling the `cudaMemcpyAsync` function and specifying the transfer direction as `cudaMemcpyHostToDevice`. This initiates an asynchronous DMA transfer, copying the entire contiguous cached data block from the CPU's page-locked memory region to a temporary contiguous buffer in GPU memory. After the transfer is complete, a dedicated "KV cache unpacking" CUDA kernel is started. This kernel receives the GPU's temporary buffer address and a previously saved list of original addresses as input. Each thread block of the kernel is responsible for processing a portion of the data, based on the address in the original address list. The needle and the dimension information of each tensor are used to "distribute" the content in the continuous data block and write it back to the original, non-contiguous storage locations of the Key and Value tensors in the GPU memory of the interrupted task. After the data is restored, the previously saved decoder internal state is read from the context of the task to be executed and the swap-out cache state. For example, if the number of tokens generated at the time of the interruption was 128, this value is used as a parameter to restart the decoder CUDA kernel of the task. The kernel continues to execute its decoding loop from the 129th token, and uses the fully restored KV cache state to seamlessly generate subsequent tokens until a terminator is finally output, generating a complete inference result sequence.

[0033] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A high-concurrency, high-speed inference deployment method for large language models, characterized in that, Includes the following steps: Based on user identity, request type, and estimated computational load, extract the weight coefficient corresponding to the user identity identifier, the urgency value mapped to the request type, and the resource consumption value converted from the estimated computational load. Calculate the weight coefficient, urgency value, and resource consumption value, and compare the calculation results with the preset priority threshold to classify the request into the target priority queue and obtain the dynamic priority of the inference task. Based on the number of tokens, syntactic complexity, and geographic coordinates of the user's network access point in the request text, the complexity score of the text is quantified. Then, based on the complexity score and the user's geographic coordinates, a pre-set distributed node routing table is queried. Combining the dynamic priority of the inference task, the student model node of the large language model with the highest confidence is selected. Entities and intents in the request text are extracted on the node, an initial KV cache state is generated, and the target inference node and initial context state are established.

2. The high-concurrency, high-speed inference deployment method for large language models according to claim 1, characterized in that, The method further includes the following steps: Based on the target inference node and the initial context state, the dynamic priority value of the new task's inference task is compared with the priority value of the currently running task on the target inference node. If the new task's priority value is higher, the decoding loop of the current task is immediately terminated, and the key tensors and value tensors of all attention layers in the GPU memory are concatenated into a continuous memory block. The direct memory access channel is called to transfer the memory block to the CPU main memory for storage, thereby obtaining the context of the task to be executed and the swap-out cache state. Based on the context of the task to be executed and the swap-out cache state, the context data of the new task is separated from the state, the data is loaded into the GPU memory and decoding is started. After the new task generates an end symbol, the swap-out task identifier recorded in the state is retrieved, the corresponding contiguous memory block in the CPU main memory is retrieved, the original address of the GPU memory is written back through the memory access channel, and the KV cache is continued to be executed from the interrupted decoding position to generate the inference result sequence.

3. The high-concurrency, high-speed inference deployment method for large language models according to claim 1, characterized in that, The steps for obtaining the dynamic priority of the inference task are as follows: Based on user identity, request type, and estimated computational load, the weight coefficient is queried from the preset user identity weight table, the corresponding urgency value is retrieved from the request type mapping table, and the resource consumption value is calculated in the segmentation function based on the estimated computational load to establish the request metadata quantification value. Based on the requested metadata quantification value, the weight coefficient and the urgency value are multiplied, and then the result is added to the resource consumption value. The sum is input into the activation function for non-linear transformation, compressing the multi-dimensional input into a single scalar value to obtain the original request priority score. Based on the original priority score of the request, it is compared one by one with the preset priority thresholds arranged from smallest to largest. If the original priority score of the request is less than a certain threshold, the comparison stops and the request is assigned to a specific priority queue of the interval to which the original priority score of the request belongs, so as to obtain the dynamic priority of the inference task.

4. The high-concurrency, high-speed inference deployment method for large language models according to claim 1, characterized in that, The steps for obtaining the target inference node and the initial context state are as follows: Based on the number of tokens in the request text, the complexity of the syntactic structure, and the geographical coordinates of the user's network access point, the tokens in the text are counted, a dependency syntax tree is constructed and its maximum depth is measured, the network topology is queried to calculate the estimated network latency between the user and each node, and the three factors are weighted and summed to obtain a comprehensive score for the text complexity. Based on the comprehensive text complexity score and combined with the dynamic priority of the inference task, the comprehensive text complexity score and geographical coordinates are used as a joint index in the pre-set distributed node routing table to retrieve all nodes that meet the conditions, and the large language model student model node that best matches the model capability and task complexity is selected from them to obtain the selected inference node. Based on the selected inference node, a lightweight feedforward network is run on the node to process the embedding vector of the input request text, identify named entities and user intent within the text, and encode the information into initial key tensors and value tensors to establish the target inference node and initial context state.

5. The high-concurrency, high-speed inference deployment method for large language models according to claim 2, characterized in that, The steps for obtaining the context of the task to be executed and the state of the swapped-out cache are as follows: Based on the target inference node and the initial context state, extract the dynamic priority value of the inference task from the metadata of the new task, and subtract it from the priority value of the currently running task on the target inference node. If the difference is positive, send a hardware interrupt signal to the GPU kernel to generate a preemptive execution instruction. According to the preemptive execution instruction, the GPU kernel immediately pauses the decoding loop of the current task and traverses the key tensors and value tensors of all attention layers in the GPU memory. It then splices them together along the sequence length dimension in the order of the layers to form a physically continuous data structure in memory, namely a continuous cache block to be unloaded.

6. The high-concurrency, high-speed inference deployment method for large language models according to claim 5, characterized in that, The steps for obtaining the context of the task to be executed and the state of the swapped-out cache also include: Based on the contiguous cache block to be unloaded, the direct memory access controller is invoked, and the starting physical address and length of the contiguous cache block to be unloaded are used as parameters to initiate a block transfer operation from GPU memory to CPU main memory, and the task identifier and memory address are recorded to obtain the context of the task to be executed and the swap-out cache status.

7. The high-concurrency, high-speed inference deployment method for large language models according to claim 2, characterized in that, The steps for obtaining the inference result sequence are as follows: Based on the context of the task to be executed and the state of the swapped-out cache, the data structure of the state is parsed, the context data of the new task is separated, the context data is loaded into the specified working area of ​​the GPU memory, and the decoder is started to generate words one by one until the model outputs the preset end symbol to obtain the output of the high-priority task. Based on the high-priority task output, the previously recorded swap-out task identifier and CPU main memory address are read from the metadata fields of the task context and swap-out cache status. The corresponding contiguous memory block is located in the CPU main memory based on the CPU main memory address, and a cache data block to be restored is established.

8. The high-concurrency, high-speed inference deployment method for large language models according to claim 7, characterized in that, The steps for obtaining the inference result sequence also include: The cached data block to be recovered is written from the CPU main memory back to the original address recorded in the GPU video memory, and the internal state of the decoder is restored. The KV cache is then executed again from the interrupted decoding position to generate the inference result sequence.

Images