An end-side device based multi-model online inference service optimization method and system

Abstract

The application provides a kind of based on end side device's multi-model online inference service optimization method and system, the steps of the method include: receiving user inference request, parsing the target model of the user inference request;If current GPU state is running state, further determine whether the target model of user is the model currently running;If the target model of user is not the model currently running, determine whether the running time of the model currently running is less than the preset minimum time slice, if yes, determine whether there is model hunger currently, if there is model hunger, determine the model queue of current hunger, and determine whether the model queue of the model currently running by GPU is empty, if empty, the inference task in the model queue of current hunger is processed;If there is no model hunger, the inference task corresponding to user inference request is suspended to the model queue of corresponding target model, and the inference task in the model queue of the model currently running by GPU is continued to be processed.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and system for optimizing multi-model online inference services based on edge devices. Background Technology

[0002] Large Language Models (LLMs) have made groundbreaking progress in the field of natural language processing, but their inference services face significant resource challenges. A single large language model typically requires tens to hundreds of gigabytes of GPU memory. For example, the LLaMA-70B model requires approximately 140GB of GPU memory at FP16 accuracy, while the GPT-3 model requires over 350GB. This makes it impossible to load multiple models simultaneously to meet the needs of different business scenarios on resource-constrained edge devices or in single-machine multi-GPU environments.

[0003] Existing large language model inference frameworks (such as vLLM, TensorRT-LLM, and DeepSpeed-Inference) primarily optimize single-model inference, employing techniques like pagedattention and continuous batching to improve the throughput and concurrency of a single model. However, in real-world production environments, there are often requirements for multi-model concurrency: for example, intelligent customer service systems need to simultaneously support general dialogue models, domain expert models, and code generation models; edge AI devices need to dynamically switch between lightweight and high-parameter models based on different task scenarios.

[0004] When there is a need for concurrent use of multiple models, existing technologies usually require frequent model switching to synchronize model parameters from the CPU to the GPU, and each model switch consumes a large amount of computing resources. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method for optimizing multi-model online inference services based on edge devices, in order to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides a method for optimizing a multi-model online inference service based on an edge device, the method comprising the following steps: Receive user reasoning requests and parse the target model labels in the user reasoning requests to determine the user's target model; Determine the current GPU state. If the current GPU state is running, then further determine whether the user's target model is the currently running model. If the user's target model is the currently running model, then determine whether there is model starvation. If there is no model starvation, then process the inference task corresponding to the user's inference request. If there is model starvation, then determine the currently starved model queue and determine whether the model queue of the model currently running on the GPU is empty. If it is empty, then process the inference tasks in the currently starved model queue. If the user's target model is not the currently running model, then it is determined whether the runtime of the currently running model is less than the preset minimum time slice. If so, it is determined whether there is model starvation. If there is model starvation, the currently starved model queue is determined, and it is determined whether the model queue of the model currently running on the GPU is empty. If it is empty, the inference tasks in the currently starved model queue are processed. If there is no model starvation, the inference task corresponding to the user's inference request is suspended to the model queue of the corresponding target model, and the inference tasks in the model queue of the model currently running on the GPU are processed.

[0007] Using the above scheme, upon receiving a user's inference request, this scheme first determines whether the user's target model and the currently running model are consistent. If they are consistent and there is no model starvation, the inference task can be processed directly. If model starvation exists, there are timed-out inference tasks that have been waiting for too long, and these timed-out inference tasks are processed first. If the user's target model and the currently running model are inconsistent, this scheme does not need to switch directly, saving computing resources. Instead, it makes a judgment based on the runtime of the currently running model. If the runtime is short and there is no model starvation, the inference tasks in the model queue of the model currently running on the GPU can continue to be processed without switching, saving computing resources.

[0008] In some embodiments of the present invention, in the step of determining whether model starvation exists, the waiting time of the longest inference task in each model queue is obtained. If the waiting time of the longest inference task in any model queue is greater than a preset waiting threshold, then model starvation is determined to exist.

[0009] In some embodiments of the present invention, after determining whether the model queue of the model currently running on the GPU is empty, if the model queue of the model currently running on the GPU is not empty, the difference in queue length between the currently hungry model queue and the model queue of the model currently running on the GPU is calculated. If the difference in queue length is greater than a preset queue difference threshold, the inference tasks in the currently hungry model queue are processed; if the difference in queue length is not greater than the preset queue difference threshold, the inference tasks in the model queue of the model currently running on the GPU are processed.

[0010] In some embodiments of the present invention, after the step of determining the current GPU state, if the current GPU running state is not running, the inference task corresponding to the user inference request is suspended to the model queue of the corresponding target model.

[0011] In some embodiments of the present invention, if the inference task in the currently hungry model queue is to be processed, the current GPU state is switched to the drainage state and an asynchronous drainage loop is started. In the steps of the asynchronous drainage loop, it is determined whether the current GPU is idle. If it is not idle, the duration of the asynchronous drainage loop is counted and compared with the drainage threshold duration. If the duration of the asynchronous drainage loop is less than the drainage threshold duration, the current GPU is re-determined to be idle.

[0012] In some embodiments of the present invention, the drainage threshold duration includes a hard duration and a soft duration, wherein the hard duration is greater than the soft duration. In the step of comparing the duration of the asynchronous drainage loop with the drainage threshold duration, the duration of the asynchronous drainage loop is compared with the hard duration. If it is not less than the hard duration, the current GPU operation is stopped, and the asynchronous drainage loop is completed. If it is greater than the hard duration, the duration of the asynchronous drainage loop is compared with the soft duration. If it is not less than the soft duration...

[0013] In some embodiments of the present invention, after the asynchronous drainage loop is completed, the current GPU state is switched to a switching state, and the currently running model is switched to the model corresponding to the currently hungry model queue.

[0014] In some embodiments of the present invention, in the step of switching the currently running model to the model corresponding to the currently hungry model queue, the model parameters of the model corresponding to the currently hungry model queue are obtained from the CPU and synchronized to the GPU.

[0015] In some embodiments of the present invention, after the step of switching the currently running model to the model corresponding to the currently hungry model queue, the current GPU state is switched to the running state.

[0016] A second aspect of the present invention also provides a multi-model online inference service optimization system based on an edge device. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0017] A third aspect of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned multi-model online inference service optimization method based on an edge device.

[0018] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the text, or may be learned by practice of the invention. The objects and other advantages of the invention will become apparent from the description and the accompanying drawings.

[0019] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0020] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0021] Figure 1 This is a schematic diagram illustrating one embodiment of the multi-model online inference service optimization method based on edge devices according to the present invention; Figure 2 This is a schematic diagram of the architecture of this solution; Figure 3 A logical diagram illustrating the transition from operating state to drainage state; Figure 4 A logic diagram illustrating the transition from drainage state to switching state; Figure 5 A schematic diagram illustrating the logic for restoring the running state after a state switch. Figure 6 This is a schematic diagram of the hot-switching process in this solution. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0023] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0024] Introduction to existing technologies: Existing mainstream technical solutions for multi-model inference scenarios include: 1. Multi-process parallel deployment Each model is launched with an independent inference process, and each process uses its own GPU memory. When a user request arrives, the system routes the request to the corresponding inference process for processing based on the model identifier carried in the request. This strategy ensures rapid response to requests for different models while making full use of available hardware resources to achieve high-concurrency service.

[0025] 2. Request queuing and process restart On memory-constrained edge devices, only a single inference process is launched, serving only one model at a time. When requests for different models are received, these requests are temporarily stored in a request queue. After the system completes all inference tasks for the current model, it terminates the current inference process, releasing all related resources, including GPU memory, communication groups, and device context. Subsequently, the inference process restarts and loads a new model to handle the next request in the queue.

[0026] 3. Efficient parameter switching based on LoRA For multiple downstream models fine-tuned based on the same base model, this solution employs efficient parameter fine-tuning techniques such as LoRA (Low-Rank Adaptation). When switching models, only the parameters of the LoRA adapter need to be changed, while the weights of the base model remain unchanged. This approach significantly improves the efficiency and response speed of switching. Currently, inference frameworks such as vLLM support concurrent inference with multiple LoRAs, enabling efficient utilization of GPU resources while flexibly handling requests from multiple models.

[0027] Disadvantages of existing technology: Existing technologies employ a multi-process parallel deployment approach, which can support inference services for multiple models simultaneously, but suffers from extremely low GPU memory utilization. Since each model occupies GPU memory independently, even if some models are idle, resources cannot be released for other models to use. For example, on a server with 8 GPUs and 80GB of GPU memory each, after deploying 4 models, each model can only use about 160GB of GPU memory. However, actual workloads often exhibit significant time-domain imbalance, resulting in a large amount of idle GPU memory.

[0028] Meanwhile, the existing technology cannot adapt to dynamic load changes. When the number of requests for a certain model suddenly increases, it cannot dynamically allocate the GPU memory resources of other idle models, resulting in increased queuing latency or a higher request rejection rate. In addition, to ensure service quality, sufficient GPU memory and computing resources need to be reserved for each model, which leads to a significant increase in hardware costs, making it difficult to implement in resource-constrained edge devices or small- to medium-sized deployment scenarios.

[0029] On resource-constrained edge devices, model switching can be achieved through request queuing and process restart. Process restart involves operations such as operating system resource reclamation, GPU device context reinitialization, and NCCL communication group reconstruction, typically taking 20-60 seconds, which cannot meet the real-time requirements of online services. During process restart, the system is completely unresponsive to any requests, resulting in a poor user experience, especially in high-frequency switching scenarios (e.g., intelligent customer service systems need to frequently switch between general dialogue models and domain expert models), where accumulated service interruption time can account for more than 30% of the total service time. Furthermore, each switch requires repeated execution of NCCL initialization, CUDA device initialization, and other operations. This reallocation of underlying resources not only wastes computing resources but may also introduce reliability risks such as state loss, resource leaks, and connection drops.

[0030] While existing LoRA (Low-Rank Adaptation) techniques can efficiently switch parameters between multiple downstream models fine-tuned on the same pedestal model, their limitations are also apparent. First, LoRA dynamic switching requests typically rely on the same downstream task as during training. If the requested task differs significantly from the training task, LoRA's model performance may be suboptimal. Furthermore, LoRA is suitable for fine-tuning on the same pedestal model; it is ineffective when requests involve models with entirely different structures. Although LoRA offers the possibility of rapid switching, it may still fail to provide an immediate response to frequent, high-load requests, especially when dealing with complex or large models. Moreover, the performance and training quality of the pedestal model directly impact LoRA's effectiveness; if the pedestal model performs poorly, adapter optimizations are unlikely to compensate for the performance deficit.

[0031] like Figure 1 and Figure 2 As shown, this invention proposes a method for optimizing multi-model online inference services based on edge devices. The steps of the method include: Step S100: Receive user reasoning request, parse the target model label of the user reasoning request to determine the user's target model; In practice, the user inference request carries the target model label and inference task parameters.

[0032] Step S200: Determine the current GPU state. If the current GPU state is running, further determine whether the user's target model is the currently running model. In the specific implementation process, it is determined whether the current GPU state is running. If the result is "no", that is, the GPU state is in the drainage state or switching state, the request is suspended, that is, the request is suspended to the model queue of the corresponding target model, the process returns, and continues to listen for new requests.

[0033] The "Running" state indicates that the currently loaded model is being served normally, and new requests are continuously added to the batch processing queue. In this state, the scheduler monitors system metrics in real time. When it receives a request to call another model, and the system meets any of the following conditions, it triggers a transition to the "Drainage" state: Time slice exhaustion: The current model continuously occupies the GPU for more than the preset minimum time slice (Min Slice) to prevent system jitter caused by frequent switching.

[0034] Starvation trigger: In the model queues of other models, there are requests whose waiting time exceeds the waiting threshold (Max Wait). To avoid long-tail requests timeout.

[0035] Load skew: When the difference in queue length between model queues exceeds a set threshold, the system needs to prioritize processing the model queues corresponding to high-load models.

[0036] Step S300: If the user's target model is the currently running model, determine whether there is model starvation. If there is no model starvation, process the inference task corresponding to the user's inference request. If there is model starvation, determine the currently starved model queue and determine whether the model queue of the model currently running on the GPU is empty. If it is empty, process the inference task in the currently starved model queue. In practice, the user's target model and the currently running model are further compared to determine if model starvation exists. This involves checking if any inference task in the model queue has a waiting time exceeding its corresponding maximum waiting time limit. If starvation is detected, a forced switch is performed; otherwise, the request is added directly to the inference engine's task queue for normal execution.

[0037] In step S400, if the user's target model is not the currently running model, it is determined whether the runtime of the currently running model is less than the preset minimum time slice. If so, it is determined whether there is model starvation. If there is model starvation, the currently starved model queue is determined, and it is determined whether the model queue of the model currently running on the GPU is empty. If it is empty, the inference tasks in the currently starved model queue are processed. If there is no model starvation, the inference task corresponding to the user's inference request is suspended to the model queue of the corresponding target model, and the inference tasks in the model queue of the model currently running on the GPU are processed.

[0038] In the specific implementation process, in the step of determining whether the runtime of the currently running model is less than the preset minimum time slice, it is further checked whether there are starvation requests in order to avoid performance jitter caused by frequent switching.

[0039] like Figure 3 As shown, using the above scheme, after receiving a user's inference request, this scheme first determines whether the user's target model and the currently running model are consistent. If they are consistent and there is no model starvation, the inference task can be processed directly. If there is model starvation, there are timed-out inference tasks that have been waiting for too long, and these timed-out inference tasks are processed first. If the user's target model and the currently running model are inconsistent, this scheme does not need to switch directly, saving computing resources. Instead, it makes a judgment based on the runtime of the currently running model. If the runtime is short and there is no model starvation, the inference tasks in the model queue of the model currently running on the GPU can continue to be processed without switching, saving computing resources.

[0040] Specifically, the architecture of this solution is as follows: Figure 2 As shown, this invention is based on a vLLM inference engine design and includes: Client Access Layer (AsyncLLM): Provides a unified control interface to the outside world, responsible for receiving user inference requests, parsing target model labels, and forwarding requests to the core layer through a message queue.

[0041] Engine Process Management Layer (EngineCoreProc): As a resident daemon, it is responsible for maintaining the lifecycle of the inference engine and achieving process-level isolation and protection.

[0042] The Inference Control Engine (EngineCore): The core of the system, running within the engine process management layer, responsible for maintaining the native scheduler, batch queues, and executors, and generating executable inference plans in each iteration. It contains two key sub-modules: 1. Adaptive Flow Control Scheduler (DrainableScheduler): The core decision-making unit of this invention. It is responsible for maintaining multi-model request queues and executing a scheduling algorithm based on a three-state machine. The three states are: Running, Draining, and Switching. It determines whether to continue serving the current model, enter the drainage phase, or immediately perform a switch based on the real-time load. 2. Task Scheduler: Responsible for scheduling the execution order of specific requests in the currently loaded task queues that call the same model.

[0043] Executor layer: Responsible for coordinating underlying computing resources. Supports single-machine or distributed multi-machine modes, issuing SWITCH_MODEL commands to worker nodes via RPC instructions.

[0044] Worker Layer: The actual computing unit running on the GPU. It includes the ModelRunner and the memory manager. It can unload old model weights and load new model weights in a hot-swappable manner without releasing the GPU context.

[0045] In some embodiments of the present invention, in the step of determining whether model starvation exists, the waiting time of the longest inference task in each model queue is obtained. If the waiting time of the longest inference task in any model queue is greater than a preset waiting threshold, then model starvation is determined to exist.

[0046] In some embodiments of the present invention, after determining whether the model queue of the model currently running on the GPU is empty, if the model queue of the model currently running on the GPU is not empty, the difference in queue length between the currently hungry model queue and the model queue of the model currently running on the GPU is calculated. If the difference in queue length is greater than a preset queue difference threshold, the inference tasks in the currently hungry model queue are processed; if the difference in queue length is not greater than the preset queue difference threshold, the inference tasks in the model queue of the model currently running on the GPU are processed.

[0047] Using the above approach, if the model queue of the model currently running on the GPU is not empty, it is necessary to further determine whether there are other models whose queue lengths differ from the current model's queue length by a preset load skew threshold. If a significant load skew exists, the system selects the model with the longest queue as the switching target to achieve multi-model load balancing; if no such model exists, the system determines that the current model should continue to serve and does not trigger a switch.

[0048] In some embodiments of the present invention, after the step of determining the current GPU state, if the current GPU running state is not running, the inference task corresponding to the user inference request is suspended to the model queue of the corresponding target model.

[0049] like Figure 4As shown, in some embodiments of the present invention, if the inference task in the currently hungry model queue is to be processed, the current GPU state is switched to the drainage state and an asynchronous drainage loop is started. In the steps of the asynchronous drainage loop, it is determined whether the current GPU is idle. If it is not idle, the duration of the asynchronous drainage loop is counted and compared with the drainage threshold duration. If the duration of the asynchronous drainage loop is less than the drainage threshold duration, the current GPU is re-determined to be idle.

[0050] In some embodiments of the present invention, the drainage threshold duration includes a hard duration and a soft duration, wherein the hard duration is greater than the soft duration. In the step of comparing the duration of the asynchronous drainage loop with the drainage threshold duration, the duration of the asynchronous drainage loop is compared with the hard duration. If it is not less than the hard duration, the current GPU operation is stopped, and the asynchronous drainage loop is completed. If it is greater than the hard duration, the duration of the asynchronous drainage loop is compared with the soft duration. If it is not less than the soft duration...

[0051] Specifically, it determines whether the drainage duration exceeds the preset hard timeout (hard_drain_timeout, default value is 10.0 seconds) and whether it exceeds the preset soft timeout (soft_drain_timeout, default value is 5.0 seconds).

[0052] In the specific implementation process, when the soft duration is reached, a preemption instruction is sent to the Task Scheduler, and execution is resumed to continue checking the engine's idle status; when the hard duration is reached, an error level log is recorded to indicate that there is a long-running request blocking the switching process.

[0053] Using the above approach, the drainage state is a smooth intermediate transition during model switching. The system stops inputting new inference requests to the computing engine, but allows the engine to continue processing requests that have not yet been completed in the current batch.

[0054] Soft Timeout: Upon entering a soft timeout: batch expansion stops, merging new sequences stops, and only existing sequences are allowed to continue.

[0055] Hard Timeout: If the drainage time exceeds the timeout threshold (such as when generating excessively long text), the system triggers a forced preemption mechanism to preempt / terminate the excessively long request, and optionally saves the intermediate state for subsequent recovery, forcibly entering the switching state.

[0056] In some embodiments of the present invention, after the asynchronous drainage loop is completed, the current GPU state is switched to a switching state, and the currently running model is switched to the model corresponding to the currently hungry model queue.

[0057] In practice, when the model hot-swap process begins, the system switches to the switching state. During the switching period, data is not sent to the computing engine, but it can be received by the access layer / scheduling layer and temporarily stored in a waiting queue. It will be scheduled after the switching is complete. After the switching is complete, the system automatically returns to the running state.

[0058] like Figure 5 As shown, specifically, the adaptive flow control scheduler sends the SWITCH_MODEL instruction to the inference engine core (EngineCore), carrying the configuration parameters of the target model (model_path, tensor_parallel, etc.).

[0059] When the inference engine core receives the SWITCH_MODEL instruction, it synchronizes it to all worker nodes via the RPC mechanism to perform parallel model switching.

[0060] Worker nodes perform model update operations, including unloading old model components (model_runner, weight tensors, etc.), releasing related key-value caches, loading the new model, and rebuilding related components.

[0061] The inference engine core re-executes the Profile process, including recalculating the number of pre-allocated blocks for the KV Cache tensor and performing a Kernel pre-compilation for the new model.

[0062] The switchover is complete, and the inference engine core notifies the adaptive flow control scheduler that the switchover is complete. The scheduler updates the current model identifier, resets the service start time, and extracts the task queue corresponding to the new model from the pending queue, feeding it back into the inference engine in FIFO order.

[0063] The GPU state transitions from switched to running state, and it begins processing inference requests for new models.

[0064] In some embodiments of the present invention, in the step of switching the currently running model to the model corresponding to the currently hungry model queue, the model parameters of the model corresponding to the currently hungry model queue are obtained from the CPU and synchronized to the GPU.

[0065] In some embodiments of the present invention, after the step of switching the currently running model to the model corresponding to the currently hungry model queue, the current GPU state is switched to the running state.

[0066] In this architecture, the internal components of the worker node layer are divided into resident components and pluggable components. Resident components include the GPU device context, distributed communication infrastructure (NCCL communication group, inter-process RPC / IPC channel), and memory management infrastructure (memory allocator, memory pool metadata). These components are initialized when the worker process starts and remain intact throughout the lifecycle of the vLLM instance, allowing for direct reuse during model switching. Other objects, such as model weight tensors, model runner objects, KV cache tensors, model-specific attention backends, and pre-compiled kernels, are strongly bound to the specific model architecture and need to be unloaded and rebuilt during switching.

[0067] During model switching, pluggable components are selectively unloaded first, cleaned up in the order of Attention Backend, KV Cache tensor, and model weights. Then, the model configuration references within the worker node are updated according to the target model, including model architecture parameters, cache configuration, and parallel configuration. Next, the model runner object is rebuilt, and the new model weights are loaded into GPU memory. After loading, the Profile process is executed to measure the peak activation memory of the new model, and the number of KV Cache blocks is dynamically calculated based on available GPU memory, automatically adapting to the memory requirements of different models. A new KV Cache tensor is allocated based on the calculation results; its shape is determined by parameters such as the number of layers, number of attention heads, and hidden layer dimensions of the new model.

[0068] Finally, the kernel warm-up process is executed, which triggers the just-in-time compilation of the custom CUDA kernel by performing multiple batches of empty inference and captures the CUDA Graph for inference acceleration with a fixed batch size. After the warm-up is complete, the inference service for the new model is officially ready.

[0069] Throughout the switchover process, the GPU device context, CUDA streams, NCCL communication groups, and inter-process communication channels remain continuously available without requiring reinitialization. This design optimizes model switching from traditional process-level restarts to component-level replacements, significantly reducing switching latency while maintaining service continuity and system stability.

[0070] like Figure 6 As shown, this invention transforms the traditional serial initialization process into a parallel, memory-operation-based hot-switching process. The specific steps are as follows: S1 (Request Access): AsyncLLM is started in online mode, receiving inference requests via API and parsing the target model identifier carried in the request; S2 (Adaptive Decision): DrainableScheduler determines whether the target model is consistent with the currently loaded models: If they match, the request is allowed to enter EngineCore's native scheduling path; If there is a discrepancy, the scheduler issues a switching instruction, the engine core layer suspends task scheduling, temporarily stores the request in the pending queue, and triggers the engine to enter the drainage process.

[0071] S3 (Drainage): In the drainage phase, the entry point is controlled to prevent new requests from entering the execution queue of the worker node; at the same time, requests that are already being executed are allowed to continue to be generated until the tasks in the current model queue are completed or the hard timeout threshold control condition is reached. S4 (Switch): A SWITCH_MODEL instruction is sent to the executor layer, which in turn sends a model switching instruction to the inference engine core. Upon receiving the signal, the worker node performs memory cleanup operations: swapping out necessary KV cache data to CPU main memory and releasing the GPU memory space occupied by the model weights. Then, based on the new model configuration, a "component reuse" loading process is performed: this includes updating the global configuration object in memory, loading the new model's weight data from main memory or disk into GPU memory, and finally, re-initializing the Model Runner based on the new configuration.

[0072] S5 (profile): The engine core layer recalculates the number of available KV Cache blocks based on the parameter scale of the new model and reinitializes the metadata of the memory manager.

[0073] S6 (Recovery): After the switch is complete, the scheduler is reset to the running state, and DrainableScheduler will push the temporarily stored requests back to the original scheduling path to restore normal inference.

[0074] This invention achieves low-latency dynamic switching of multiple models in a single inference system through two core technologies: component reuse and adaptive scheduling.

[0075] Regarding component reuse, this invention decouples the "execution environment" and "GPU state" of the inference engine. Traditional solutions require terminating the process and reinitializing all resources during switching, while this invention keeps the process and underlying communication connections alive during switching, dynamically replacing model weights and resetting the state only in memory, thereby achieving second-level hot switching and shortening the switching time.

[0076] In terms of adaptive scheduling, this invention introduces a drainage mechanism based on intermediate state management. The system defines three states and dynamically adjusts the switching timing using time slice protection periods and load pressure to avoid performance fluctuations caused by frequent switching and achieve a smooth model transition. Simultaneously, a multi-level timeout protection mechanism is set up to prevent long-tail requests from blocking system switching, achieving a balance between response speed and system throughput. Through state machine management and adaptive decision-making, the system can automatically select the optimal switching timing based on the actual load, ensuring dual optimization of resource utilization and service quality.

[0077] The beneficial effects of this plan include: 1. This solution achieves low-latency model hot-switching, supporting dynamic switching of multiple models within a single inference process, thus solving the problem of low GPU memory resource utilization in traditional solutions. This technology maintains continuous inference process operation and uninterrupted external communication channels by dynamically reusing GPU memory resources, while significantly reducing model switching latency and meeting the real-time requirements of online services.

[0078] 2. This solution maximizes request response rate. It employs an adaptive request scheduling mechanism that tracks request arrival time and waiting threshold duration to shorten the average Time to First Token (TTFT) generation time, thus ensuring service quality. This mechanism avoids request loss and connection drops caused by traditional restart solutions during continuous inference processes, significantly improving the successful response rate and reducing the request failure rate due to model switching.

[0079] 3. This solution targets large-scale online language inference service scenarios based on edge devices. Building upon the existing vLLM inference engine, it integrates an adaptive drainage scheduling mechanism and a hot-switching mechanism based on component reuse. This supports rapid hot-switching and resource adaptation without altering the underlying inference core, significantly reducing system jitter and congestion latency caused by frequent switching and improving resource utilization stability. Regarding service continuity, flexible switching between multi-model inference within a single inference system supports dynamic load adaptation, thereby achieving more stable multi-model concurrent inference services in edge device and small-to-medium-scale deployment scenarios.

[0080] 4. This solution designs an adaptive scheduling mechanism based on a "three-state machine." Through the coordinated operation of a three-level judgment mechanism—time slice protection period, starvation detection, and load balancing—adaptive scheduling of online inference requests based on inference engine state transitions is achieved, thereby significantly improving the system's response speed and stability to dynamic loads. This mechanism can dynamically adjust the execution order and switching timing based on the current resource state in edge scenarios, and quickly clear requests through soft and hard preemption mechanisms, ensuring controllable switching latency. This results in higher throughput and lower latency fluctuations in multi-model concurrent environments. Specific operational steps are detailed in Section 3.2.2.

[0081] 5. This solution integrates a model hot-switching mechanism based on component reuse into the vLLM inference engine. By leveraging the hot-switching mechanism of reusing resident components in the worker node process and dynamically updating pluggable components, there is no need to reinitialize the underlying resources, which greatly improves the switching speed and system stability. At the same time, it reduces the risk of resource leakage and state loss, ensures the continuity of external requests during the switching process, and thus improves the service availability and user experience of edge devices.

[0082] This invention also provides a multi-model online inference service optimization system based on an edge device. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0083] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned multi-model online inference service optimization method based on an edge device. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0084] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0085] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0086] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for optimizing multi-model online inference services based on edge devices, characterized in that, The steps of the method include: Receive user reasoning requests and parse the target model labels in the user reasoning requests to determine the user's target model; Determine the current GPU state. If the current GPU state is running, then further determine whether the user's target model is the currently running model. If the user's target model is the currently running model, then determine whether there is model starvation. If there is no model starvation, then process the inference task corresponding to the user's inference request. If there is model starvation, then determine the currently starved model queue and determine whether the model queue of the model currently running on the GPU is empty. If it is empty, then process the inference tasks in the currently starved model queue. If the user's target model is not the currently running model, then it is determined whether the runtime of the currently running model is less than the preset minimum time slice. If so, it is determined whether there is model starvation. If there is model starvation, the currently starved model queue is determined, and it is determined whether the model queue of the model currently running on the GPU is empty. If it is empty, the inference tasks in the currently starved model queue are processed. If there is no model starvation, the inference task corresponding to the user's inference request is suspended to the model queue of the corresponding target model, and the inference tasks in the model queue of the model currently running on the GPU are processed.

2. The multi-model online inference service optimization method based on edge devices according to claim 1, characterized in that, In the step of determining whether model starvation exists, the waiting time of the longest inference task in each model queue is obtained. If the waiting time of the longest inference task in any model queue is greater than the preset waiting threshold, then model starvation is determined to exist.

3. The multi-model online inference service optimization method based on edge devices according to claim 1, characterized in that, After determining whether the model queue of the model currently running on the GPU is empty, if the model queue of the model currently running on the GPU is not empty, the difference in queue length between the currently hungry model queue and the model queue of the model currently running on the GPU is calculated. If the difference in queue length is greater than a preset queue difference threshold, the inference tasks in the currently hungry model queue are processed; if the difference in queue length is not greater than the preset queue difference threshold, the inference tasks in the model queue of the model currently running on the GPU are processed.

4. The multi-model online inference service optimization method based on edge devices according to claim 1, characterized in that, After determining the current GPU state, if the current GPU is not in a running state, the inference task corresponding to the user inference request is suspended to the model queue of the corresponding target model.

5. The multi-model online inference service optimization method based on edge devices according to any one of claims 1 to 4, characterized in that, If the inference tasks in the currently hungry model queue are to be processed, the current GPU state is switched to drainage state and an asynchronous drainage loop is started. In the steps of the asynchronous drainage loop, it is determined whether the current GPU is idle. If it is not idle, the duration of the asynchronous drainage loop is counted and compared with the drainage threshold duration. If the duration of the asynchronous drainage loop is less than the drainage threshold duration, the current GPU is re-determined to be idle.

6. The multi-model online inference service optimization method based on edge devices according to claim 5, characterized in that, The drainage threshold duration includes a hard duration and a soft duration, where the hard duration is greater than the soft duration. In the step of comparing the duration of the asynchronous drainage loop with the drainage threshold duration, the duration of the asynchronous drainage loop is compared with the hard duration. If it is not less than the hard duration, the current GPU operation is stopped, and drainage is completed. If it is greater than the hard duration, the duration of the asynchronous drainage loop is compared with the soft duration. If it is not less than the soft duration...

7. The multi-model online inference service optimization method based on edge devices according to claim 5, characterized in that, After the asynchronous drainage loop is completed, the current GPU state is switched to the switching state, and the currently running model is switched to the model corresponding to the currently hungry model queue.

8. The multi-model online inference service optimization method based on edge devices according to claim 7, characterized in that, In the step of switching the currently running model to the model corresponding to the currently hungry model queue, the model parameters of the model corresponding to the currently hungry model queue are obtained from the CPU and synchronized to the GPU.

9. The multi-model online inference service optimization method based on edge devices according to claim 7, characterized in that, After switching the currently running model to the model corresponding to the currently hungry model queue, the current GPU state is switched to running state.

10. A multi-model online inference service optimization system based on edge devices, characterized in that, The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1 to 9.

Images