Quick and quality-aware cloud collaborative big language model reasoning method and system
The cloud-based collaborative large language model system, which utilizes task complexity assessment and dynamic resource scheduling, solves the problems of high latency in cloud deployment and low quality in edge deployment, achieving low-latency, low-cost, and high-quality inference services.
Patent Information
- Application Number
- CN202511945862.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-22
- Publication Date
- 2026-04-17
AI Technical Summary
Existing large language models suffer from high latency and high cost when deployed purely on the cloud, while purely on the client side suffer from low quality and resource bottlenecks. Furthermore, existing collaborative reasoning solutions fail to effectively balance task complexity, latency, and cost.
Intelligent routing is achieved by using an execution selector based on task complexity, combined with runtime monitoring and priority scheduling mechanisms. A load-aware GPU-CPU dynamic offloading strategy is used to dynamically adapt to the system state, ensuring that critical tasks are processed first. The lightweight DistilBERT model is used to evaluate task complexity, enabling efficient task allocation in the cloud and locally.
While ensuring high-quality response, it reduces latency and costs, improves the achievement rate of service level targets, solves the problems of high latency in pure cloud deployment and low quality in pure edge deployment, and provides efficient, high-quality and economical inference services.
Smart Images
Figure CN121882250A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer networks and artificial intelligence, specifically relating to a fast and quality-aware cloud-based collaborative large language model reasoning method and system. Background Technology
[0002] In recent years, large language models (LLMs), such as GPT-4, have achieved great success in various applications. In typical deployment scenarios, service providers utilize large-scale LLM clusters on cloud server platforms to handle massive user inference requests. However, this purely cloud-based deployment model faces several challenges: First, a large number of concurrent requests often leads to a surge in cloud server load, resulting in high operating costs; second, user requests and model responses incur significant network latency during transmission and are susceptible to network fluctuations, making it difficult to guarantee Service Level Objectives (SLOs); furthermore, data transmission raises user concerns about privacy breaches. As a countermeasure, deploying smaller language models (SLMs) on edge devices has become a trend. Pure edge deployment can effectively reduce latency, save costs, and protect user privacy. However, limited by the computing and storage resources of edge devices, the size of SLMs is much smaller than that of cloud-based LLMs, resulting in insufficient response quality and generalization ability when handling complex tasks. Simultaneously, when multiple applications generate a large number of concurrent requests, the limited resources of edge devices may also become a bottleneck, preventing the achievement of Service Level Objectives (such as First Token Generation Time (TTFT)).
[0003] To combine the advantages of both, researchers began exploring collaborative inference frameworks for cloud and edge devices. However, existing solutions have significant shortcomings. Some task allocation methods only consider factors such as latency and cost, ignoring the complexity of the tasks themselves. This may lead to complex tasks being incorrectly assigned to edge-side SLMs, sacrificing response quality, or simple tasks being sent to cloud-based LLMs, causing unnecessary latency and cost. Other solutions, such as model segmentation or speculative decoding, while aiming to optimize performance, lack effective system-level design to minimize the additional overhead they introduce (such as segmentation overhead and multi-round communication latency), and do not fully consider the actual hardware architecture characteristics of edge devices, resulting in challenges in practical deployment. Summary of the Invention
[0004] This invention addresses the problems of high latency and high cost in pure cloud-based deployments, and low quality and resource bottlenecks in pure edge-based deployments in existing LLM inference deployment schemes. It proposes a fast and quality-aware cloud-based collaborative large language model inference system (CEC-LM). The system employs an intelligent task routing mechanism based on task complexity execution selector, a mechanism that dynamically adapts to system states by combining runtime monitoring and priority scheduling, and a load-aware GPU-CPU dynamic offloading strategy to minimize inference interference on edge devices. This ensures high-quality response while achieving low-latency and low-cost inference services, solving the key problems of high latency and high cost in pure cloud-based deployments, and low quality and slow response in pure edge-based deployments.
[0005] According to one aspect of the present invention, a fast and quality-aware cloud-based collaborative large language model inference method is provided, comprising:
[0006] The complexity prediction model is used to predict the complexity of each task in the input task pool, and the predicted complexity of each task is obtained.
[0007] Based on the predicted complexity of each task, the task is assigned to either a queue of a local small language model or a queue of a cloud-based large language model.
[0008] The local small language model queue and the cloud large language model queue are used to schedule tasks according to the priority of each task in their respective queues.
[0009] Based on the order of the scheduled tasks, reasoning is performed on the tasks in their respective queues using both a local small language model and a cloud-based large language model.
[0010] Furthermore, the complexity of each task in the input task pool is predicted using a complexity prediction model, including:
[0011] The user query portion from the public dataset is input into the local small language model and the cloud large language model respectively to obtain the corresponding inference results;
[0012] When the similarity between the inference result of the local small language model and the inference result of the cloud large language model is greater than a set threshold, the complexity of the user query is set to 0 and sent to the local small language model for processing; otherwise, it is set to 1; a fine-tuning dataset is constructed based on the processing results.
[0013] Based on the fine-tuning dataset, the complexity prediction model is fine-tuned through full fine-tuning;
[0014] Each task is input into the finely tuned complexity prediction model to obtain the corresponding task complexity.
[0015] Furthermore, the processing order of tasks is scheduled according to the priority of each task in its respective queue, including:
[0016] Assign priority to each task based on its first token generation time (TTFT);
[0017] Processing order is assigned based on the priority of each task; whereby a task's waiting time exceeds a preset score of its first token generation time, its priority will be increased by one level.
[0018] Furthermore, the processing order is assigned according to the priority of each task, including:
[0019] Retrieve tasks to be processed from the current task queue;
[0020] Determine if the current priority of the task to be processed is already the highest level in the queue: If the priority of the task to be processed is already the highest, no adjustment is needed, and the process can be returned directly.
[0021] Furthermore, if the priority of the task to be processed is not the highest, it is then determined whether the time of its first token generation has exceeded the preset score of its total expected processing time.
[0022] When the waiting time exceeds the preset score of the total expected processing time, the priority of the pending task is raised to a higher level, and the internal timer of the pending task is reset to zero so that the next wait can be recalculated.
[0023] Furthermore, the method also includes: mapping the memory addresses of the remaining blocks of the complexity prediction model from the GPU to the CPU when the local small language model is under high load or there is network latency.
[0024] Furthermore, when the local small language model load is not high, this includes: mapping the memory addresses of all blocks of the complexity prediction model from the CPU to the GPU.
[0025] According to one aspect of the present invention, a fast and quality-aware cloud-based collaborative large language model inference system is provided, comprising:
[0026] The complexity prediction module is used to predict the complexity of each task in the input task pool using a complexity prediction model, and obtain the predicted complexity of each task.
[0027] The queue allocation module is used to allocate the task to the queue of the local small language model or the queue of the cloud large language model according to the predicted complexity of each task.
[0028] The scheduling module is used for queues for local small language models and queues for cloud-based large language models, and schedules tasks according to the priority of each task in their respective queues.
[0029] The task reasoning module is used to reason about the tasks in their respective queues based on the order of the scheduled tasks, using both a local small language model and a cloud-based large language model.
[0030] According to one aspect of the present invention, an electronic device is provided, comprising a cloud server and an end device, the cloud server and the end device being used to implement the steps of the fast and quality-aware cloud-based collaborative large language model inference method.
[0031] According to one aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a cloud server or an end device, implements the steps of the fast and quality-aware cloud-based collaborative large language model inference method.
[0032] Compared with the prior art, the beneficial effects of the present invention are:
[0033] 1. This invention designs a mechanism that combines runtime task monitoring and priority scheduling to ensure that critical tasks can be prioritized in dynamic and high-load environments, effectively improving the achievement rate of service level objectives and preventing task starvation.
[0034] 2. This invention employs a task execution selector finely tuned based on a complexity prediction model (lightweight DistilBERT model), which effectively balances response quality, service latency, and cost, and avoids performance degradation caused by unreasonable task allocation.
[0035] 3. This invention achieves a significant reduction in service costs under the cloud-based collaborative inference framework through intelligent task routing based on complexity prediction models, dynamic monitoring and scheduling during runtime, and GPU-CPU collaborative optimization with end-device load awareness. It maintains service level target achievement rate and response quality comparable to pure cloud solutions, providing efficient, high-quality and economical inference services for language models in resource-constrained edge environments and complex network conditions. Attached Figure Description
[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0037] Figure 1This is a schematic diagram of a fast and quality-aware cloud-based collaborative large language model inference method provided by the present invention.
[0038] Figure 2 This is a schematic diagram of the structure of a fast and quality-aware cloud-based collaborative large language model system provided by the present invention.
[0039] Figure 3 A diagram of a GPU-CPU dynamic offloading strategy based on UMA provided for this invention. Detailed Implementation
[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0041] like Figure 1 As shown, this invention proposes a fast and quality-aware cloud-based collaborative large language model inference method and system, comprising: predicting the complexity of each task in the input task pool using a complexity prediction model to obtain the predicted complexity of each task; allocating tasks to a queue of a local small language model or a queue of a cloud-based large language model based on the predicted complexity of each task; scheduling tasks in the local small language model queue or the cloud-based large language model queue according to the priority of each task in their respective queues; and performing inference on the tasks in their respective queues using the local small language model and the cloud-based large language model based on the order of the scheduled tasks. The technologies involved in this invention include cloud-based collaborative inference, task complexity assessment based on lightweight models, priority scheduling based on service level objectives, load-aware GPU-CPU dynamic offloading, and zero-copy memory management technology. The entities included in this invention mainly fall into two categories: cloud servers and edge devices.
[0042] Specifically, this embodiment of the invention employs a task execution selector based on a lightweight model, which can evaluate task complexity in real time based on user request prompts. Complex tasks are routed to the cloud-based LLM to ensure response quality, while simple tasks are assigned to the local SLM for execution, effectively balancing inference quality, latency, and cost. To address the interference of the complexity prediction model on the end device to local SLM inference, the system implements an adaptive load-aware switching strategy. When high load is detected on the local SLM, the complexity prediction model is dynamically offloaded from the GPU to the CPU, and zero-copy parameter transfer is achieved using a unified memory architecture (UMA), significantly reducing system overhead. To further enhance the system's robustness in dynamic environments, CEC-LM introduces a runtime task monitoring and priority scheduling mechanism. By continuously monitoring network conditions and system load, it adaptively adjusts task routing and prioritizes urgent tasks, ensuring the achievement of service level objectives. Among them, the complexity prediction model can be the DistilBERT model, which is from "Sanh V, Debut L, Chaumond J, et al. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter[J]. 2019".
[0043] Specifically, such as Figure 2 As shown, the main technologies of this invention are quality-aware task routing and load-aware GPU-CPU dynamic offloading strategies. It also involves a collaborative working mechanism between cloud servers and end devices. Therefore, it will be mainly introduced from the following aspects.
[0044] First, the formal definitions of the symbols used in the embodiments of this invention are given:
[0045] Task K includes information such as prompt, service level objective (SLO), priority, task generation time, and first token generation time objective (TTFT).
[0046] Specifically, the embodiments of the present invention propose an overall workflow, as shown in Algorithm 1, which includes the following core steps:
[0047] Task generation and pooling: Inference task K generated by user input or application API call enters the task pool on the terminal device (mobile phone, in-vehicle device, etc.) to wait for processing.
[0048] Task Complexity Prediction: For each task entering the task pool, a lightweight complexity prediction model analyzes its prompts and assesses its intrinsic complexity. Specifically, this involves inputting the user query portion of a public dataset (such as ShareGPT or SQuAD) into both the edge and cloud-based models to obtain the corresponding inference results (each sample in the public dataset has two fields: user query and LLM response; here, the user query portion is input). If the similarity between the edge model's inference result and the cloud-based inference result exceeds a given threshold, the complexity of that sample is set to 0, indicating that the edge model can handle the sample; otherwise, it is set to 1. 0 and 1 are labels, and this process is repeated to construct a fine-tuning dataset. Full fine-tuning is then used to fine-tune the complexity prediction model on the fine-tuning dataset; the task is then input into the complexity prediction model to obtain the task complexity.
[0049] Initial routing based on complexity: Based on the predicted complexity, the system initially assigns tasks to either the local SLM queue (low complexity) or the cloud LLM queue (high complexity).
[0050] Priority scheduling and runtime monitoring: Tasks are scheduled in their respective queues according to priority, which is set based on the task's TTFT. Simultaneously, a runtime task monitor continuously tracks network latency and inference load at both ends. When anomalies are detected (such as network congestion or local SLM overload), the execution position of tasks is dynamically adjusted to ensure SLO is met.
[0051] Load-aware GPU-CPU dynamic offloading: On the end device, the system is aware of the inference load of the local SLM in real time. When the load is high, in order to avoid the complexity prediction model interfering with the SLM inference speed, part or all of the computation of the complexity prediction model will be dynamically offloaded from the GPU to the CPU.
[0052] Task execution and inference result return: The task is executed by the specified SLM or LLM and the generated inference results are returned to the user or application. Final storage involves storing the inference results for later evaluation of the execution performance.
[0053]
[0054] Specifically, processing is carried out according to the priority order of each task, including:
[0055] Retrieve task K from the priority queue; determine if task K's current priority is already the highest in the queue; if task K's priority is already the highest, return directly without adjustment; if task K's priority is not the highest, continue to determine if its current waiting time has exceeded one-third of its set total expected processing time. If the waiting time has exceeded the threshold, perform the following operations: raise task K's priority to a level one level higher than its current level. Reset task K's internal timer to zero to recalculate the next waiting time.
[0056] Specifically, in order to achieve the best balance between response quality, latency, and cost, embodiments of the present invention provide quality-aware task routing, namely, an intelligent task routing mechanism that includes task complexity assessment, priority scheduling, and runtime monitoring, as detailed below:
[0057] Task Complexity Assessment: Traditional collaborative reasoning methods often overlook the inherent difficulty of the task itself. This invention introduces a task execution selector, the core of which is a lightweight DistilBERT model. We fine-tune this model using a dataset containing various task types, enabling it to accurately classify the complexity of input prompts using a binary classification (0 represents low complexity, 1 represents high complexity). Low-complexity tasks are suitable for processing by edge-side SLM to leverage its low latency and zero cost; while high-complexity tasks must be routed to cloud-based LLM to utilize its powerful "emergent capabilities" to ensure response quality. The selector model itself has a small number of parameters (approximately 67M) and low evaluation overhead, providing an efficient decision-making basis for subsequent intelligent routing.
[0058] Priority Scheduling and Runtime Monitoring: Relying solely on static complexity assessment is insufficient to handle dynamically changing system environments. Therefore, this invention designs two collaborative dynamic scheduling mechanisms: First, priority-based task scheduling. Different tasks have different Scheduled Least Immediate Results (SLOs), for example, question-answering tasks typically require very low Time-To-Flight (TTFTs). The system assigns priorities based on the strictness of the task's TTFT SLO (the shorter the TTFT, the higher the priority), ensuring that high-priority tasks are processed first in the queue. To prevent low-priority tasks from "starving," the system also introduces a dynamic priority boosting mechanism: when a task's waiting time exceeds a preset score of its TTFT SLO (e.g., 1 / 3), its priority is boosted by one level. Secondly, there is the runtime task monitor, which continuously tracks the network latency from local to cloud and the inference speed of local and cloud environments. It dynamically migrates local tasks to the cloud or cloud tasks back to local to ensure that the TTFT SLO of a large number of tasks can be met. This includes: (1) measuring network latency through the "Ping-Pong" mechanism. When the latency is too high, even high-complexity tasks may be forced to remain in local processing to avoid timeouts; (2) monitoring the inference speed (prefill and decoding rates) of local SLM and cloud LLM. When the performance of either end drops sharply due to overload, subsequent tasks will be redirected to the other end. The combination of these two mechanisms enables task routing decisions to be transformed from static "complexity" assessment to dynamic, real-time comprehensive assessment of "system state + complexity".
[0059] Specifically, embodiments of this invention provide load-aware GPU-CPU co-optimization. On resource-constrained devices, simultaneously running a local SLM and complexity predictor model can lead to resource contention, especially on GPUs. This significantly reduces the inference speed of the SLM, thus impacting the SLO achievement rate. To address this issue, this invention proposes a load-aware GPU-CPU dynamic offloading strategy, such as... Figure 3 As shown.
[0060] Specifically, the core idea of the GPU-CPU dynamic offloading strategy is to prioritize GPU resources for the SLM (Simplified Modeling). The system monitors the SLM's inference load in real time (judging based on the actual situation). When the load is low, the complexity prediction model runs on the GPU for maximum speed. When the SLM load increases, the system dynamically offloads some or all of the computations of the complexity prediction model (such as Transformer blocks) from the GPU to the CPU for execution through model partitioning. Although CPU execution is slower, for small models like DistilBERT, the increased latency is within an acceptable range, and allocating valuable GPU resources entirely to the SLM yields greater overall benefits.
[0061] Specifically, traditional GPU-CPU switching requires explicitly copying model parameters between the two independent memory spaces, a process that introduces significant I / O latency. End devices typically employ a Unified Memory Architecture (UMA), where the CPU and GPU share the same physical memory space. This invention fully leverages this characteristic, designing a zero-copy parameter transfer mechanism based on UMA. Model parameters are stored in a special tensor structure (SMTensor), which internally maintains two tensor views, one for the CPU and one for the GPU, but they share the underlying physical memory. When switching from GPU to CPU is required, the system does not need to perform a physical data copy; it only needs to perform a low-cost memory address mapping to switch control from the GPU tensor view to the CPU tensor view. This mechanism completely eliminates the data transfer overhead during the switching process, making dynamic offloading extremely efficient and ensuring the smoothness of inference on the end device.
[0062] Specifically, CEC-LM employs a task execution selector finely tuned based on a lightweight DistilBERT model. This selector can accurately assess the inherent complexity of user requests based on prompts, and then assign tasks deemed highly complex to the cloud-based LLM to ensure response quality, while assigning tasks deemed less complex to the edge-side SLM to reduce network latency and service costs. This effectively balances response quality, service latency, and cost, and avoids performance degradation caused by unreasonable task allocation.
[0063] Specifically, CEC-LM designs a mechanism that combines runtime task monitoring and priority scheduling. This mechanism can continuously track the status of requests in transit, monitor network conditions and inference load on end devices and cloud servers, thereby enabling adaptive adjustment of task routing. At the same time, the mechanism adopts a priority task scheduling strategy based on service level objectives and introduces a dynamic priority boosting mechanism to ensure that critical tasks can be processed first in dynamic and high-load environments, effectively improving the achievement rate of service level objectives and preventing task starvation.
[0064] Specifically, CEC-LM implements an adaptive load-aware switching strategy to address the interference of the complexity prediction model on the local SLM inference. When the local SLM load is detected to be high, this strategy dynamically offloads the complexity prediction model from the GPU to the CPU for execution through model segmentation. It also utilizes UMA to achieve zero-copy parameter transfer between the CPU and GPU, replacing physical data copying with address mapping, which significantly reduces data transfer overhead and system latency during the dynamic offloading process, ensuring the high efficiency of inference on the end device.
[0065] The implementation of the various embodiments of this invention is based on programmed processing through a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of this invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of this invention provide a fast and quality-aware cloud-based collaborative large language model inference system. This system is used to execute a fast and quality-aware cloud-based collaborative large language model inference method from the above method embodiments.
[0066] The system includes: a complexity prediction module, used to predict the complexity of each task in the input task pool using a complexity prediction model, and obtain the predicted complexity of each task; a queue allocation module, used to allocate the tasks to a queue of a local small language model or a queue of a cloud-based large language model according to the predicted complexity of each task; a scheduling module, used to schedule tasks in the queues of the local small language model and the cloud-based large language model according to the priority of each task in their respective queues; and a task reasoning module, used to reason about the tasks in their respective queues based on the order of the scheduled tasks, using both the local small language model and the cloud-based large language model.
[0067] The fast and quality-aware cloud-based collaborative large language model inference system provided in this invention addresses the problems of high latency and high cost in pure cloud-based deployments, and low quality and resource bottlenecks in pure client-side deployments in existing LLM inference deployment schemes. It adopts several modules, including an intelligent task routing task selector based on task complexity, a mechanism that combines runtime monitoring and priority scheduling to dynamically adapt to system states, and a load-aware GPU-CPU dynamic offloading strategy to minimize inference interference on client devices. Thus, it achieves low-latency and low-cost inference services while ensuring high-quality response, solving key problems such as high latency and cost in pure cloud-based deployments, and low quality and slow response in pure client-side deployments.
[0068] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention also provides a fast and quality-aware cloud-based collaborative large language model reasoning system, including a cloud server and end devices, which are used to implement a fast and quality-aware cloud-based collaborative large language model reasoning method as proposed in the above embodiments.
[0069] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this program significantly reduces service costs within a cloud-based collaborative reasoning framework, while maintaining service level target achievement rates and response quality comparable to pure cloud solutions. This provides efficient, high-quality, and economical reasoning services for language models in resource-constrained edge environments and complex network conditions.
[0070] In summary, this invention proposes a fast and quality-aware cloud-based collaborative large language model inference system, CEC-LM. In cloud-based collaborative inference, it combines task complexity-based execution selectors with system-level dynamic resource scheduling. It can evaluate task complexity in real time through a lightweight selector model, routing complex tasks to the cloud-based large language model (LLM) for processing, while executing simple tasks on the local small language model (SLM). Simultaneously, it reduces interference with local SLM inference by dynamically unloading and switching the selector's execution between the GPU and CPU units of the end device, and utilizes a unified memory architecture (UMA) to eliminate parameter copying overhead. This solves the problems of high latency and high cost associated with single-cloud deployment, as well as the degradation of inference quality and resource bottlenecks caused by single-endpoint deployment.
[0071] Finally, it should be noted that the above specific embodiments are merely representative examples of the present invention. Obviously, the present invention is not limited to the above specific embodiments and many variations are possible. Any simple modifications, equivalent changes, and alterations made to the above specific embodiments based on the technical essence of the present invention should be considered within the protection scope of the present invention.
Claims
1. A fast and quality-aware cloud-side collaborative large language model inference method, characterized in that, include: The complexity prediction model is used to predict the complexity of each task in the input task pool, and the predicted complexity of each task is obtained. Based on the predicted complexity of each task, the task is assigned to either a queue of a local small language model or a queue of a cloud-based large language model. The queues for local small language models and cloud-based large language models are used to schedule tasks according to the priority of each task in their respective queues. Based on the order of the scheduled tasks, reasoning is performed on the tasks in their respective queues using both a local small language model and a cloud-based large language model.
2. The fast and quality-aware cloud-based collaborative large language model inference method according to claim 1, characterized in that, The complexity prediction model is used to predict the complexity of each task in the input task pool, including: The user query portion from the public dataset is input into the local small language model and the cloud large language model respectively to obtain the corresponding inference results; When the similarity between the inference result of the local small language model and the inference result of the cloud large language model is greater than a set threshold, the complexity of the user query is set to 0 and sent to the local small language model for processing; otherwise, it is set to 1; a fine-tuning dataset is constructed based on the processing results. Based on the fine-tuning dataset, the complexity prediction model is fine-tuned through full fine-tuning; Each task is input into the finely tuned complexity prediction model to obtain the corresponding task complexity.
3. The fast and quality-aware cloud-based collaborative large language model reasoning method according to claim 1, characterized in that, The processing order of tasks is scheduled according to the priority of each task in its respective queue, including: Assign priority to each task based on the time when the first token is generated; Processing order is assigned based on the priority of each task; when the waiting time of a task exceeds a preset score of its first token generation time, its priority will be increased by one level.
4. The fast and quality-aware cloud-based collaborative large language model inference method according to claim 1, characterized in that, The processing order is assigned based on the priority of each task, including: Retrieve tasks to be processed from the current task queue; Determine if the current priority of the task to be processed is already the highest level in the queue: If the priority of the task to be processed is already the highest, no adjustment is needed, and the process can be returned directly.
5. The fast and quality-aware cloud-based collaborative large language model inference method according to claim 4, characterized in that, If the priority of the task to be processed is not the highest, then it continues to determine whether the time of its first token generation has exceeded the preset score of its total expected processing time: When the waiting time exceeds the preset score of the total expected processing time, the priority of the pending task is raised to a higher level, and the internal timer of the pending task is reset to zero so that the next wait can be recalculated.
6. The fast and quality-aware cloud-based collaborative large language model inference method according to claim 1, characterized in that, The method further includes: mapping the memory addresses of the remaining blocks of the complexity prediction model from the GPU to the CPU when the local small language model is under high load or there is network latency.
7. The fast and quality-aware cloud-based collaborative large language model inference method according to claim 6, characterized in that, When the local small language model load is not high, this includes: mapping the memory addresses of all blocks of the complexity prediction model from the CPU to the GPU.
8. A fast and quality-aware cloud-based collaborative large language model reasoning system, characterized in that, include: The complexity prediction module is used to predict the complexity of each task in the input task pool using a complexity prediction model, and obtain the predicted complexity of each task. The queue allocation module is used to allocate the task to the queue of the local small language model or the queue of the cloud large language model according to the predicted complexity of each task. The scheduling module is used to schedule tasks according to the priority of each task in the local small language model queue or the cloud large language model queue. The task reasoning module is used to reason about the tasks in their respective queues based on the order of the scheduled tasks, using both a local small language model and a cloud-based large language model.
9. A fast and quality-aware cloud-based collaborative large language model inference system, comprising a cloud server and terminal devices, characterized in that, The cloud server and the terminal device are used to implement the steps of the fast and quality-aware cloud-based collaborative large language model inference method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a cloud server or end device, it implements the steps of the fast and quality-aware cloud-based collaborative large language model reasoning method as described in any one of claims 1 to 7.