Large model deployment method based on heterogeneous resource scheduling and multi-model collaborative reasoning

Abstract

The invention discloses a large model deployment method based on heterogeneous resource scheduling and multi-model collaborative reasoning, and relates to the field of artificial intelligence and server system deployment. Comprising the following steps of: 1, calculating a deployment score of a target model by utilizing a deployment scoring function according to a GPU video memory residual rate, a CPU load, network delay and a current task quantity of each node through a heterogeneous resource sensing scheduler, and selecting a node with the highest score of the target model to load or reuse the target model; a cache index key is generated, and whether key value KV pairs supporting multiplexing of the target model exist or not is inquired; step 3, if the cache is hit, skipping a forward propagation stage, and directly decoding based on a cache result, otherwise, executing complete forward reasoning and writing the result into the cache; 4, based on a plurality of user requests, performing fusion according to user priorities, model weights, reasoning costs and cue word lengths to generate execution sorting weights, and scheduling execution models in batches according to the weights; and 5, returning the model response to the user, and releasing or updating node state information.

Description

Technical Field

[0001] This invention discloses a method for deploying large models based on heterogeneous resource scheduling and multi-model collaborative reasoning, which relates to the fields of artificial intelligence and server system deployment. Background Technology

[0002] With the widespread application of large-scale language models such as GPT, BERT, LlaMA, and their derivatives in natural language processing, code generation, and intelligent question answering, the computational burden of deploying such models on the server side has significantly increased. Current technologies for large-model inference services face the following bottlenecks, such as:

[0003] It consumes a lot of resources; loading a single model requires a large amount of video memory and is slow.

[0004] Lacking resource-aware scheduling, it cannot adapt to the dynamic scheduling requirements of heterogeneous resources;

[0005] Poor multi-model coordination makes it difficult to support the collaborative deployment of multiple fine-tuning models.

[0006] The caching mechanism is imperfect, making it difficult to reuse the inference results of common prompt words. Summary of the Invention

[0007] This invention addresses the problems of existing technologies by providing a large-scale model deployment method based on heterogeneous resource scheduling and multi-model collaborative inference. It integrates resource scheduling, cache sharing, and inference optimization strategies, enabling efficient deployment and execution of multiple large-scale pre-trained model tasks in multi-user concurrent scenarios. This significantly reduces response latency and resource consumption, and improves the overall throughput and service capabilities of the system.

[0008] The specific solution proposed in this invention is as follows:

[0009] This invention provides a method for deploying large models with heterogeneous resource scheduling and multi-model collaborative inference, including:

[0010] Step 1: Using a heterogeneous resource-aware scheduler, the target model's deployment score is calculated based on each node's GPU memory availability, CPU load, network latency, and current task count using a deployment scoring function. The node with the highest target model score is then selected to load or reuse the target model.

[0011] Step 2: For the prompt words requested by the user, generate a cache index key and query whether there are key-value pairs that support the reuse of the target model;

[0012] Step 3: If the cache is hit, skip the forward propagation stage and decode directly based on the cached result; otherwise, perform full forward inference and write the result to the cache.

[0013] Step 4: Based on multiple user requests, generate execution ranking weights by fusing user priority, model weights, inference cost, and prompt word length, and schedule the models to be executed in batches according to their weights.

[0014] Step 5: Return the model response to the user and release or update the node state information.

[0015] Furthermore, the deployment scoring function in step 1 of the method is:

[0016] score = w1 * (gpu_free_mem / model_size) - w2 * cpu_usage - w3 * latency + w4 * (1 / task_count), where gpu_free_mem is the node's currently unused GPU memory, model_size is the memory requirement of model M, the ratio of gpu_free_mem / model_size indicates the sufficiency of the node's remaining GPU memory relative to the size of model M, cpu_usage is the CPU utilization rate, which measures the CPU load pressure on the node, latency is the network latency between the node and the outside world, w3 × latency measures the node's latency, w3 is a negative weight, reducing the score in high-latency scenarios, task_count is the total number of tasks currently running on the node, 1 / task_count indicates the current task congestion level of the node, and w1-w4 are adjustable preset weights.

[0017] Furthermore, step 2 of the method, generating the cache index key, includes:

[0018] The user input is digested using SHA256 (prompt) and used as the cache index key.

[0019] Ensure that the models use the same tokenizer, set consistent embedding layer parameters, and verify compatibility by comparing the hash code compat_id(model).

[0020] Furthermore, in step 4 of the method, the fusion execution ranking weight is calculated using the following formula:

[0021] score = user_priority × model_weight / (cost_estimate + α × prompt_length), where user_priority is the user priority, representing the priority level of the task or user; model_weight is the model weight, reflecting the model's performance or applicability in the task; cost_estimate is the cost estimate, reflecting the resource cost required to execute the task; α is an adjustment coefficient that controls the degree of influence of prompt_length on the denominator. The larger α is, the more significant the influence of prompt length on the score, and the smaller α is, the weaker the influence; prompt_length is the prompt length, representing the length of text or data input into the model.

[0022] This invention also provides a large-scale model deployment system for heterogeneous resource scheduling and multi-model collaborative reasoning, including a heterogeneous resource-aware scheduler module, a context cache module, and an adaptive fusion execution module.

[0023] The heterogeneous resource-aware scheduler module calculates the deployment score of the target model based on the GPU memory availability, CPU load, network latency, and current task count of each node using a deployment scoring function, and selects the node with the highest target model score to load or reuse the target model.

[0024] The context caching module generates a cache index key for the prompt words of the user request and checks whether there are key-value pairs that support the reuse of the target model;

[0025] If the cache is hit, the context cache module skips the forward propagation phase and directly decodes based on the cached result; otherwise, it performs complete forward inference and writes the result to the cache.

[0026] The adaptive fusion execution module generates execution ranking weights by fusing multiple user requests according to user priority, model weights, inference cost, and prompt word length, and then schedules the models to be executed in batches according to their weights.

[0027] The model response is returned to the user, and the node state information is released or updated.

[0028] Furthermore, the deployment scoring function utilized by the heterogeneous resource-aware scheduler module of the system is:

[0029] score = w1 * (gpu_free_mem / model_size) - w2 * cpu_usage - w3 * latency + w4 * (1 / task_count), where gpu_free_mem is the node's currently unused GPU memory, model_size is the memory requirement of model M, the ratio of gpu_free_mem / model_size indicates the sufficiency of the node's remaining GPU memory relative to the size of model M, cpu_usage is the CPU utilization rate, which measures the CPU load pressure on the node, latency is the network latency between the node and the outside world, w3 × latency measures the node's latency, w3 is a negative weight, reducing the score in high-latency scenarios, task_count is the total number of tasks currently running on the node, 1 / task_count indicates the current task congestion level of the node, and w1-w4 are adjustable preset weights.

[0030] Furthermore, the system's context caching module generates a cache index key, including:

[0031] The user input is digested using SHA256 (prompt) and used as the cache index key.

[0032] Ensure that the models use the same tokenizer, set consistent embedding layer parameters, and verify compatibility by comparing the hash code compat_id(model).

[0033] Furthermore, the adaptive fusion execution module of the system calculates the fusion execution ranking weight using the following formula:

[0034] score = user_priority × model_weight / (cost_estimate + α × prompt_length), where user_priority is the user priority, representing the priority level of the task or user; model_weight is the model weight, reflecting the model's performance or applicability in the task; cost_estimate is the cost estimate, reflecting the resource cost required to execute the task; α is an adjustment coefficient that controls the degree of influence of prompt_length on the denominator. The larger α is, the more significant the influence of prompt length on the score, and the smaller α is, the weaker the influence; prompt_length is the prompt length, representing the length of text or data input into the model.

[0035] The advantages of this invention are:

[0036] This invention features a highly modular and distributed architecture, adapting to the elastic scheduling requirements of heterogeneous computing environments. It can maintain service continuity and performance stability even when server nodes are added, deleted, or changed, making it particularly suitable for multi-tenant, high-concurrency large-model service scenarios.

[0037] Enhancing the universality and reusability of model services: The proposed caching mechanism and model compatibility verification framework provide a solid foundation for the continuous fine-tuning and version evolution of large models in the future, enabling efficient sharing of common computing results between new and old models, reducing long-term deployment costs, and showing good prospects for ecological expansion.

[0038] Meeting the real-time requirements of large-scale inference services: In scenarios with multiple concurrent user requests, this invention minimizes system response time and model switching overhead through a collaborative mechanism of heterogeneous scheduling, cache reuse, and fusion inference, meeting the deployment requirements of large-scale model applications with low latency and high concurrency. It is widely applicable to real-time inference scenarios such as intelligent question answering, semantic search, and code generation.

[0039] It has good engineering implementation and commercial feasibility: This method can be directly embedded into existing cloud model inference platforms or edge inference frameworks, with low system coupling and deployment threshold. It can be flexibly adapted to mainstream large model inference frameworks such as HuggingFace Transformers, TensorRT, ONNX Runtime, etc., and is easy to productize and deploy at scale. Attached Figure Description

[0040] Figure 1 This is the overall architecture diagram in the application of this invention.

[0041] Figure 2 This is a schematic diagram of the model scheduling and deployment process in this invention.

[0042] Figure 3 This is a schematic diagram of a shared context cache structure.

[0043] Figure 4 This is the flowchart for the execution of fusion reasoning. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0045] Example 1

[0046] This invention provides a method for deploying large models with heterogeneous resource scheduling and multi-model collaborative inference, including:

[0047] Step 1: The heterogeneous resource-aware scheduler HR-Scheduler calculates the deployment score of the target model based on the GPU memory remaining rate, CPU load, network latency and current number of tasks of each node using the deployment scoring function, and selects the node with the highest target model score to load or reuse the target model.

[0048] The deployment scoring function is as follows:

[0049] score = w1 * (gpu_free_mem / model_size) - w2 * cpu_usage - w3 * latency + w4 * (1 / task_count), where gpu_free_mem is the unused GPU memory of the node, model_size is the memory requirement of model M, and the ratio of gpu_free_mem / model_size represents the sufficiency of the node's remaining GPU memory relative to the size of model M. The larger the ratio, the more sufficient the remaining memory. For example, when the ratio is >1, the memory can fully accommodate the model; when the ratio is <1, the memory may be insufficient, and there is a risk of OOM (Out of Memory).

[0050] `task_count` represents the total number of tasks currently running on the node. `1 / task_count` measures the current task congestion level of the node. The fewer tasks, the greater the positive contribution to the score. `1 / task_count` decreases as the number of tasks increases, reflecting that nodes with fewer tasks have weaker resource competition and are more efficient at processing model M. To avoid resource contention such as GPU / CPU time slices due to too many tasks, idle nodes are prioritized.

[0051] cpu_usage is the CPU utilization rate, usually ranging from 0-100%, reflecting the CPU's busyness and measuring the CPU load pressure of a node. The higher the load, the lower the score. The operation of model M not only depends on the GPU, but may also involve CPU data preprocessing such as reading data, feature transformation, or post-processing such as result parsing. Excessive CPU load will slow down the overall process, so high-load nodes should be avoided.

[0052] Latency refers to the network latency between a node and external sources such as data sources, result receivers, or scheduling centers. w3×latency measures the network latency of a node. If model M requires network transmission, such as input data coming from remote storage, output needing to be sent back to the client, or communication with other nodes in distributed inference, high latency will increase the overall time consumption. Therefore, low-latency nodes are preferred. w1-w4 are adjustable preset weights.

[0053] The calculation process can be found in the following code:

[0054]

[0055] Step 2: For the prompt words requested by the user, generate a cache index key and query whether there are key-value pairs that support the reuse of the target model.

[0056] Generating cache index keys may include:

[0057] The user input is digested using SHA256 (prompt) and used as the cache index key. The cache item structure can be seen in the code below:

[0058]

[0059]

[0060] Ensure that the models use the same tokenizer, set consistent embedding layer parameters, and verify compatibility by comparing the hash code compat_id(model).

[0061] The pseudocode for the cache retrieval logic is as follows:

[0062]

[0063] def set(self,prompt,model_id,kv):

[0064] key = sha256(prompt)

[0065] entry=self.cache.get(key,{'model_ids':[],'kv_tensor':None})

[0066] entry['model_ids'].append(model_id)

[0067] entry['kv_tensor'] = kv

[0068] self.cache[key] = entry

[0069] Step 3: If the cache is hit, skip the forward propagation stage and decode directly based on the cached result; otherwise, perform full forward inference and write the result to the cache.

[0070] Step 4: Based on multiple user requests, an execution ranking weight is generated by fusing user priority, model weight, inference cost, and prompt length. Models are then batch-executed according to their weights. This process receives inference requests from multiple users or systems and merges them through a priority ranking mechanism to achieve batch processing and resource sharing. Each request includes the following attributes: user_priority: such as system, VIP, or regular user; model_weight: the importance score of the model used in the task; cost_estimate: the model's inference cost, such as memory usage and latency; prompt_length: the prompt length.

[0071] The formula for calculating the fusion execution ranking weight is as follows:

[0072] score = user_priority × model_weight / (cost_estimate + α × prompt_length), where user_priority is the user priority, representing the priority level of the task or user; model_weight is the model weight, reflecting the model's performance or applicability in the task; cost_estimate is the cost estimate, reflecting the resource cost required to execute the task; α is an adjustment coefficient that controls the degree of influence of prompt_length on the denominator. The larger α is, the more significant the influence of prompt length on the score, and the smaller α is, the weaker the influence; prompt_length is the prompt length, representing the length of text or data input into the model.

[0073] The entire inference service process is as follows:

[0074] User initiates request: API receives model ID and Prompt;

[0075] Call HR-Scheduler: Calculate the deployment nodes;

[0076] Query the key-value cache: if a match is found, skip the model forward pass;

[0077] Enter the fusion sorting queue: sort by weight to determine the execution order;

[0078] Inference execution: After batch execution, update the cache and return the results.

[0079] Step 5: Return the model response to the user and release or update the node state information.

[0080] Example 2

[0081] This invention also provides a large-scale model deployment system for heterogeneous resource scheduling and multi-model collaborative reasoning, including a heterogeneous resource-aware scheduler module, a context cache module, and an adaptive fusion execution module.

[0082] The heterogeneous resource-aware scheduler module calculates the deployment score of the target model based on the GPU memory availability, CPU load, network latency, and current task count of each node using a deployment scoring function, and selects the node with the highest target model score to load or reuse the target model.

[0083] The context caching module generates a cache index key for the prompt words of the user request and checks whether there are key-value pairs that support the reuse of the target model;

[0084] If the cache is hit, the context cache module skips the forward propagation phase and directly decodes based on the cached result; otherwise, it performs complete forward inference and writes the result to the cache.

[0085] The adaptive fusion execution module generates execution ranking weights by fusing multiple user requests according to user priority, model weights, inference cost, and prompt word length, and then schedules the models to be executed in batches according to their weights.

[0086] The model response is returned to the user, and the node state information is released or updated.

[0087] The information interaction and execution process between the modules in the above system are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.

[0088] Similarly, the system of the present invention has a highly modular and distributed architecture, adapts to the elastic scheduling requirements of heterogeneous computing environments, and can maintain service continuity and performance stability when server nodes are added, deleted or changed. It is particularly suitable for multi-tenant, high-concurrency large model service scenarios.

[0089] Enhancing the universality and reusability of model services: The proposed caching mechanism and model compatibility verification framework provide a solid foundation for the continuous fine-tuning and version evolution of large models in the future, enabling efficient sharing of common computing results between new and old models, reducing long-term deployment costs, and showing good prospects for ecological expansion.

[0090] Meeting the real-time requirements of large-scale inference services: In scenarios with multiple concurrent user requests, this invention minimizes system response time and model switching overhead through a collaborative mechanism of heterogeneous scheduling, cache reuse, and fusion inference, meeting the deployment requirements of large-scale model applications with low latency and high concurrency. It is widely applicable to real-time inference scenarios such as intelligent question answering, semantic search, and code generation.

[0091] It has good engineering implementation and commercial feasibility: it can be directly embedded into existing cloud model inference platforms or edge inference frameworks, with low system coupling and deployment threshold, and can be flexibly adapted to mainstream large model inference frameworks such as HuggingFace Transformers, TensorRT, ONNX Runtime, etc., making it easy to productize and deploy at scale.

[0092] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for deploying large models with heterogeneous resource scheduling and multi-model collaborative reasoning, characterized by: include: Step 1: Using a heterogeneous resource-aware scheduler, the target model's deployment score is calculated based on each node's GPU memory availability, CPU load, network latency, and current task count using a deployment scoring function. The node with the highest target model score is then selected to load or reuse the target model. Step 2: For the prompt words requested by the user, generate a cache index key and query whether there are key-value pairs that support the reuse of the target model; Step 3: If the cache is hit, skip the forward propagation stage and decode directly based on the cached result; otherwise, perform full forward inference and write the result to the cache. Step 4: Based on multiple user requests, generate execution ranking weights by fusing user priority, model weights, inference cost, and prompt word length, and schedule the models to be executed in batches according to their weights. Step 5: Return the model response to the user and release or update the node state information.

2. The method according to claim 1, characterized in that the deployment scoring function in step 1 is: score = w1 * (gpu_free_mem / model_size) - w2 * cpu_usage - w3 * latency + w4 * (1 / task_count), where gpu_free_mem is the node's currently unused GPU memory, model_size is the memory requirement of model M, the ratio of gpu_free_mem / model_size indicates the sufficiency of the node's remaining GPU memory relative to the size of model M, cpu_usage is the CPU utilization rate, which measures the CPU load pressure on the node, latency is the network latency between the node and the outside world, w3 × latency measures the node's network latency, task_count is the total number of tasks currently running on the node, 1 / task_count indicates the current task congestion level of the node, and w1-w4 are adjustable preset weights.

3. The method according to claim 1, characterized in that generating the cache index key in step 2 includes: The user input is digested using SHA256 (prompt) and used as the cache index key. Ensure that the models use the same tokenizer, set consistent embedding layer parameters, and verify compatibility by comparing the hash code compat_id(model).

4. The method according to claim 1, characterized in that the fusion execution ranking weight is calculated in step 4, using the following formula: score = user_priority × model_weight / (cost_estimate + α × prompt_length), where user_priority is the user priority, representing the priority level of the task or user; model_weight is the model weight, reflecting the model's performance or applicability in the task; cost_estimate is the cost estimate, reflecting the resource cost required to execute the task; α is an adjustment coefficient that controls the degree of influence of prompt_length on the denominator. The larger α is, the more significant the influence of prompt length on the score, and the smaller α is, the weaker the influence; prompt_length is the prompt length, representing the length of text or data input into the model.

5. A large-scale model deployment system for heterogeneous resource scheduling and multi-model collaborative reasoning, characterized by: It includes a heterogeneous resource-aware scheduler module, a context cache module, and an adaptive fusion execution module. The heterogeneous resource-aware scheduler module calculates the deployment score of the target model based on the GPU memory availability, CPU load, network latency, and current task count of each node using a deployment scoring function, and selects the node with the highest target model score to load or reuse the target model. The context caching module generates a cache index key for the prompt words of the user request and checks whether there are key-value pairs that support the reuse of the target model; If the cache is hit, the context cache module skips the forward propagation phase and directly decodes based on the cached result; otherwise, it performs complete forward inference and writes the result to the cache. The adaptive fusion execution module generates execution ranking weights by fusing multiple user requests according to user priority, model weights, inference cost, and prompt word length, and then schedules the models to be executed in batches according to their weights. The model response is returned to the user, and the node state information is released or updated.

6. The system according to claim 5, characterized in that it is heterogeneous. The deployment scoring function used by the resource-aware scheduler module is: score = w1 * (gpu_free_mem / model_size) - w2 * cpu_usage - w3 * latency + w4 * (1 / task_count), where gpu_free_mem is the node's currently unused GPU memory, model_size is the memory requirement of model M, the ratio of gpu_free_mem / model_size indicates the sufficiency of the node's remaining GPU memory relative to the size of model M, cpu_usage is the CPU utilization rate, which measures the CPU load pressure on the node, latency is the network latency between the node and the outside world, w3 × latency measures the node's latency, w3 is a negative weight, reducing the score in high-latency scenarios, task_count is the total number of tasks currently running on the node, 1 / task_count indicates the current task congestion level of the node, and w1-w4 are adjustable preset weights.

7. The system according to claim 5, characterized in that the context... The caching module generates cache index keys, including: The user input is digested using SHA256 (prompt) and used as the cache index key. Ensure that the models use the same tokenizer, set consistent embedding layer parameters, and verify compatibility by comparing the hash code compat_id(model).

8. The system according to claim 5, characterized in that it is adaptive. The fusion execution module calculates the fusion execution ranking weight using the following formula: score = user_priority × model_weight / (cost_estimate + α × prompt_length), where user_priority is the user priority, representing the priority level of the task or user; model_weight is the model weight, reflecting the model's performance or applicability in the task; cost_estimate is the cost estimate, reflecting the resource cost required to execute the task; α is an adjustment coefficient that controls the degree of influence of prompt_length on the denominator. The larger α is, the more significant the influence of prompt length on the score, and the smaller α is, the weaker the influence; prompt_length is the prompt length, representing the length of text or data input into the model.

Images