Calculation power scheduling method for large model reasoning

Abstract

The invention relates to the technical field of artificial intelligence and cloud computing, in particular to a computing power scheduling method oriented to large model reasoning, which runs in a computing power cluster composed of a plurality of heterogeneous computing nodes, and comprises the following steps: S1, resource abstraction and monitoring: performing resource abstraction and monitoring through an agent program deployed on the computing nodes; state information of bottom layer physical computing resources is monitored and collected in real time, and the physical computing resources comprise a GPU, a CPU and a memory; the method comprises the following steps: uniformly abstracting heterogeneous GPU resources, and constructing a multi-dimensional resource view comprising a computing power unit, a video memory block and an interconnection bandwidth; on the basis, the extreme resource utilization rate and cost optimization are realized: physical GPU resources are abstracted into fine-grained virtual computing power units and virtual video memory blocks, and safe sharing and reuse of the resources among a plurality of reasoning instances are supported, so that the resource waste mode of one card and one instance is fundamentally broken.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and cloud computing technology, specifically a computing power scheduling method for large-scale model inference. Background Technology

[0002] With the widespread application of large language models, characterized by hundreds of billions or even trillions of parameters, their inference services have become a core component of artificial intelligence infrastructure. Large model inference has distinct technical characteristics: computationally intensive, consumes huge amounts of memory, has strong burstiness and volatility in requests, is extremely sensitive to response latency (especially first token latency), and has significantly different computational requirements for different models and different requests (such as input length, generation length, and accuracy requirements).

[0003] Currently, computing power scheduling for large model inference mainly faces the following challenges: Resource utilization and cost dilemma: In order to ensure service quality, the method of statically allocating exclusive computing resources (such as a whole GPU) to a single model instance is usually adopted. This results in a large amount of expensive computing resources (especially video memory) being idle during request intervals or low load periods, leading to low resource utilization and high operating costs.

[0004] Fragmented resources are difficult to utilize: Fragmented computing resources (such as fragmented GPU memory and idle GPU computing units) are widely present in the cluster due to partial release, mismatched task specifications, etc. Traditional scheduling methods are difficult to effectively aggregate these fragmented resources to support tasks such as large model inference that have a strong demand for continuous large blocks of resources.

[0005] Service quality and efficiency are difficult to balance: static allocation strategies cannot dynamically adjust resources based on real-time load. Under high concurrency requests, this can easily lead to queue congestion and a surge in response latency; while under low load, it is impossible to quickly scale down to save energy and costs. At the same time, the lack of a unified abstraction and intelligent scheduling of heterogeneous computing power (such as different models of GPUs, CPUs, and dedicated AI chips) makes it difficult to achieve global optimization.

[0006] Lack of fine-grained elastic scheduling: Existing scheduling granularity is usually based on the entire GPU card or instance, which cannot support finer-grained computing power (such as GPU computing core grouping) and memory partitioning and sharing, thus limiting the flexibility of scheduling and the potential for resource reuse.

[0007] Therefore, the industry urgently needs an intelligent computing power scheduling method that can achieve high resource utilization, low response latency, low cost, and adapt to the dynamic characteristics of large model inference and heterogeneous environments. Summary of the Invention

[0008] This invention aims to overcome the shortcomings of existing technologies and provide a computing power scheduling method, system, and electronic device for large-model inference, so as to achieve fine-grained, dynamic, and intelligent computing power resource scheduling, improve the overall resource utilization of the cluster, ensure the quality of inference services, and reduce operating costs. The invention provides the following technical solutions: A computing power scheduling method for large-scale model inference, running in a computing power cluster composed of multiple heterogeneous computing nodes, the method includes: S1: Resource Abstraction and Monitoring. Through agent programs deployed on computing nodes, the system monitors and collects real-time status information of underlying physical computing resources, including GPUs, CPUs, and memory. It performs unified abstraction of heterogeneous GPU resources, constructing a multi-dimensional resource view encompassing computing units, memory blocks, and interconnect bandwidth. Specifically, it abstracts the computing core of a single GPU into multiple independently schedulable virtual computing units and GPU memory into multiple configurable virtual memory blocks. Simultaneously, it continuously collects real-time utilization data for each resource dimension and the running status of inference service instances. S2: Inference request analysis and demand modeling: Receive inference task requests submitted by users; parse the requests and extract or predict their key features; based on a preset or online learning performance prediction model, estimate the peak computing power, memory usage and expected execution time required when the request is executed according to the key features, and form a dynamic resource demand profile of the inference request. S3: Global resource scheduling decision. The scheduler periodically or upon triggering an event executes a global scheduling decision process: integrating multi-dimensional resource views of all nodes to identify idle resources, allocated resources, and various resource fragments in the current cluster; constructing a multi-objective optimization function based on the current system strategy; solving the optimization model using an optimization algorithm to generate one or more candidate scheduling schemes; the schemes include: creating new inference service instances and allocating resources for new requests, or routing them to suitable existing instances; dynamically scaling resources on existing instances; migrating instances between nodes; and releasing resources from idle instances. S4: Fine-grained resource allocation and instance deployment. Based on the final scheduling scheme generated in step S3, perform resource allocation and instance deployment operations on the target computing node: Utilize containerization technology and resource constraint mechanisms to precisely allocate a specified number of virtual computing units, a specified size and address range of virtual memory blocks, and corresponding CPU cores and system memory to the inference service instance according to the scheduling scheme; In the allocated resource environment, load the corresponding model weights, and dynamically select the model's sharding strategy, quantization level, or computation graph optimization options based on the allocated memory size and computing power specifications; Start the inference service instance and register its service endpoint to the global request dispatcher; S5: Runtime dynamic rescheduling and optimization. During the inference instance's operation, real-time data is continuously monitored; the actual resource consumption and request latency are compared with the predicted demand profile in step S2. If there is a significant deviation, compensation measures are triggered; according to the preset elastic scaling strategy, vertical or horizontal scaling of instance resources is automatically triggered; periodically or when the overall fragmentation of the system exceeds the threshold, the resource fragmentation consolidation process is initiated, releasing and merging fragmented virtual resources by migrating instances to form a larger continuous available resource block.

[0009] As a preferred embodiment of the computing power scheduling method for large model inference described in this invention, the unified abstraction of GPU resources in S1 specifically includes: controllingly intercepting and forwarding GPU kernel drivers through a combination of hardware abstraction layer and driver extension, realizing fine-grained management and isolation of computing flow and memory access, thereby logically dividing the physical GPU streaming multiprocessor (SM) cluster into virtual computing power units, and dividing the physical video memory address space into virtual video memory blocks according to a preset granularity.

[0010] As a preferred embodiment of the computing power scheduling method for large model inference described in this invention, wherein: in S2, the key features include: the requested model identifier, the length of the input sequence, the maximum length of the requested output, the accuracy requirement, and the expected latency service level agreement; the performance prediction model is a regression model trained using machine learning methods based on historical inference task execution data, with the input being the request features and resource configuration, and the output being the predicted resource consumption and execution time.

[0011] As a preferred embodiment of the computing power scheduling method for large model inference described in this invention, wherein: in S3, the construction of the multi-objective optimization function specifically involves: taking the weighted sum of one or more of the following as optimization objectives: maximizing overall resource utilization, minimizing average request latency, minimizing energy consumption, and minimizing resource migration overhead; and constructing a constrained optimization problem model with resource capacity constraints, request SLA latency constraints, and model deployment constraints as conditions.

[0012] As a preferred embodiment of the computing power scheduling method for large model inference described in this invention, the optimization algorithm used in S3 is a reinforcement learning algorithm; at this time, the scheduler acts as an intelligent agent, its state space is a cluster resource view and request queue state, its action space is a set of scheduling operations, and the reward function integrates quantitative indicators of resource utilization, request satisfaction rate, average latency and energy consumption.

[0013] As a preferred embodiment of the computing power scheduling method for large model inference described in this invention, wherein: in S4, the dynamic selection of model sharding strategy, quantization level, or computation graph optimization option specifically includes: when allocated incomplete single-card GPU resources or multiple discrete memory blocks, automatically selecting a model parallel strategy or a more aggressive quantization level for loading; when allocated virtual resources from multiple GPUs, using tensor parallelism or pipelined parallelism strategies for model sharding deployment.

[0014] As a preferred embodiment of the computing power scheduling method for large model inference described in this invention, the compensation measures in S5 include: dynamically increasing the local resource quota for the current instance; or rerouting some of the queued requests of the current instance to other instances in the cluster with lighter loads that are running the same model.

[0015] As a preferred embodiment of the computing power scheduling method for large model inference described in this invention, the resource fragmentation process in S5 specifically includes: identifying a set of key instances that are occupied by low-priority or small-sized instances and cause resource fragmentation; planning a migration path to smoothly migrate the key instances to other computing nodes with continuous idle resources; releasing all virtual computing units and virtual memory blocks occupied by the migrated instances on the source node, merging adjacent idle blocks, and updating the global resource view.

[0016] As a preferred embodiment of the computing power scheduling method for large-model inference described in this invention, the system is deployed in a computing power cluster composed of multiple heterogeneous computing nodes, and includes: The resource monitoring and abstraction module is used to monitor and collect the status information of the underlying physical computing resources in real time through the agent program deployed on the computing node, and to perform unified abstraction of GPU resources to build a multi-dimensional resource view including virtual computing power units and virtual memory blocks. The request analysis module is used to receive and parse the inference task requests submitted by users, extract key features, and form a dynamic resource demand profile based on the performance prediction model. The global scheduling decision module is used to periodically or when triggered by events to integrate the resource view, build and solve a multi-objective optimization model, generate candidate scheduling schemes that include instance creation, routing, elastic scaling, and migration, and make a final decision. The resource allocation and deployment execution module is used to perform fine-grained resource allocation, adaptive model loading, and service instance startup and registration on the target node according to the final scheduling scheme. The runtime dynamic optimization module is used to monitor performance deviations, trigger elastic scaling, and perform resource defragmentation operations during instance operation. The modules work together to implement the computing power scheduling method according to any one of claims 1 to 8.

[0017] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method as claimed in any one of claims 1 to 8.

[0018] Compared with existing technologies, the computing power scheduling method, system, and electronic device for large model inference provided by this invention have the following significant advantages: Achieving ultimate resource utilization and cost optimization: By abstracting physical GPU resources into fine-grained virtual computing units and virtual memory blocks, and supporting their secure sharing and reuse among multiple inference instances, the resource waste model of "one GPU, one instance" is fundamentally broken; combined with dynamic elastic scaling and intelligent scheduling, cluster computing resources can flow and be allocated dynamically according to the load like a fluid, significantly improving the average utilization of GPU computing cores and memory, and directly reducing the unit computing cost of large model inference services.

[0019] Efficiently utilizing fragmented resources to enhance service capacity: The resource fragmentation identification, evaluation, and organization mechanism provided by this invention can transform fragmented computing power and GPU memory, which were originally unusable for large model inference tasks, into valuable resources through intelligent aggregation or allocation to lightweight, low-configuration-requirement inference instances. This allows the cluster to handle more concurrent inference services or requests with the same hardware investment, effectively improving overall service capacity and throughput.

[0020] Intelligent service quality assurance and reduced response latency: Through request feature analysis, performance prediction, and real-time monitoring, the scheduler can more accurately match resource supply and demand. Under high load, rapid elastic scaling and intelligent request routing prevent the formation of long queues and ensure the SLA of critical requests; when the load decreases, resources are released in a timely manner by scaling down; the dynamic rescheduling mechanism can quickly respond to performance deviations, ensuring service stability and response speed.

[0021] It offers high scheduling flexibility and automation: It supports the linkage between technologies such as model sharding and quantization and scheduling strategies, enabling the same model to dynamically select the optimal operating mode based on the currently allocated resources; The entire scheduling process is highly automated, from request analysis, decision-making, deployment to operation optimization, without the need for manual intervention, which greatly reduces the complexity of operation and maintenance and can adapt to rapid changes in business requests.

[0022] Excellent heterogeneous compatibility and scalability: By unifying the abstraction of underlying heterogeneous computing power, the scheduler can manage hybrid clusters containing different models of GPUs and AI accelerator cards; the scheduling method does not depend on specific hardware, and through modular design, it can be easily extended to support new resource types and scheduling strategies, demonstrating good foresight and scalability. Attached Figure Description

[0023] Figure 1 The overall flowchart of the computing power scheduling method for large model inference provided in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the optimized scheduling process of the global scheduling decision module in this embodiment of the invention. Figure 3 This is a structural block diagram of a computing power scheduling system for large model inference provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0025] This invention provides a computational power scheduling method for large-scale model inference. Please refer to [link / reference]. Figure 1-3 This invention provides a computing power scheduling method for large model inference, running on a computing power cluster composed of multiple heterogeneous computing nodes. The method includes: Step S1: Resource abstraction and monitoring.

[0026] Through agent programs deployed on computing nodes, the status information of underlying physical computing resources, including GPUs, CPUs, and memory, is monitored and collected in real time. Heterogeneous GPU resources are uniformly abstracted to construct a multi-dimensional resource view encompassing computing units, memory blocks, and interconnect bandwidth. Specifically, the computing core of a single GPU is abstracted into multiple independently schedulable virtual computing units, and GPU memory is abstracted into multiple configurable virtual memory blocks. Simultaneously, real-time utilization, temperature, and power consumption data for each resource dimension, as well as the running status, request queue length, and latency metrics of inference service instances, are continuously collected.

[0027] Step S2: Reasoning request analysis and requirement modeling.

[0028] Receive inference task requests submitted by users. Parse the requests and extract or predict their key features, including but not limited to: the requested model identifier, input sequence length, maximum output length to be generated, accuracy requirements (FP16, INT8, etc.), and expected latency service level agreement. Based on a preset or online-learned performance prediction model, and according to the key features, estimate the peak computational load, memory usage, expected execution time, and latency performance under different resource configurations required for the execution of the request, forming a dynamic resource requirement profile for the inference request.

[0029] Step S3: Global resource scheduling decision.

[0030] The scheduler periodically or upon triggering events (such as the arrival of new requests, instance malfunctions, or resource utilization thresholds) executes a global scheduling decision process. S3.1: Resource Status Integration and Fragmentation Assessment: Summarize the multi-dimensional resource views of all nodes, identify idle resources, allocated resources, and various resource fragments (such as discontinuous virtual memory blocks and idle virtual computing units) in the current cluster. Assess the aggregability of fragmented resources.

[0031] S3.2: Scheduling Objective Optimization Modeling: Based on the current system strategy (e.g., prioritizing latency, prioritizing utilization, prioritizing cost reduction), construct a multi-objective optimization function. Optimization variables include request-to-instance allocation, instance-to-physical resource mapping, and resource start-up, shutdown, and migration. Constraints include resource capacity constraints, request SLA constraints, and model deployment constraints.

[0032] S3.3: Scheduling Scheme Generation Based on Optimization Model: Using heuristic algorithms, constraint programming, or reinforcement learning algorithms, the optimization model is solved to generate one or more candidate scheduling schemes. The schemes include: creating new inference service instances and allocating resources for new requests, or routing them to existing suitable instances with remaining processing capacity; dynamically scaling resources on existing instances (vertical scaling); migrating instances between nodes to achieve load balancing; and releasing resources from idle instances that have timed out.

[0033] Step S4: Fine-grained resource allocation and instance deployment.

[0034] Based on the final scheduling scheme generated in step S3, perform specific resource allocation and instance deployment operations on the target computing node: S4.1: Node-level resource isolation and allocation: Utilizing containerization technology and resource constraint mechanisms, or with the support of hardware virtualization, precisely allocates a specified number of virtual computing units, a specified size and address range of virtual memory blocks, and corresponding CPU cores and system memory to inference service instances according to the scheduling scheme. This enables the secure and isolated coexistence of multiple instances on the same physical GPU.

[0035] S4.2: Model Loading and Optimization: Load the corresponding model weights within the allocated resource environment. Based on the allocated GPU memory size and computing power specifications, dynamically select the model's sharding strategy, quantization level, and computation graph optimization options. For example, when allocated incomplete GPU resources, model parallelism or more aggressive quantization is used; when allocated multiple discrete memory blocks, a partitioning technique similar to a zero-redundancy optimizer is employed for management.

[0036] S4.3: Runtime Injection and Routing Configuration: Start the inference service instance and register its service endpoint with the global request dispatcher. Update the request dispatcher's routing table to route requests that match the model and meet specific characteristics (such as input length range) to this instance.

[0037] Step S5: Runtime dynamic rescheduling and optimization.

[0038] During the execution of the inference instance, the real-time data collected in step S1 is continuously monitored.

[0039] S5.1: Performance Deviation Detection and Compensation: Compare actual resource consumption and request latency with the demand profile predicted in step S2. If a significant deviation occurs (such as actual memory usage exceeding expectations or excessively high queuing latency), an alert is triggered. Possible compensation measures include: dynamically adjusting (increasing) the local resource quota for the current instance; or rerouting some queued requests to other instances.

[0040] S5.2: Elastic Scaling Trigger: Based on a preset elastic scaling strategy (such as based on average utilization or request queue length), automatically trigger vertical scaling (adjusting the number of virtual computing units or virtual memory blocks) or horizontal scaling of instance resources. Scaling down operations must be combined with a model state saving and restoring mechanism to reduce reloading overhead.

[0041] S5.3: Defragmentation and Load Balancing: Periodically or when the overall system fragmentation exceeds a threshold, initiate a background resource defragmentation process. By systematically migrating low-priority or a small number of instances, fragmented virtual memory blocks and virtual computing units are released and merged to form larger, contiguous, available resource blocks to serve subsequent requests requiring large resources. Simultaneously, based on the node load balancing, instance migration between nodes is triggered.

[0042] Preferably, the unified abstraction of GPU resources in step S1 adopts a combination of hardware abstraction layer and driver extension. By controlling the interception and forwarding of GPU kernel drivers, fine-grained management and isolation of computing flow and memory access are achieved.

[0043] Preferably, the performance prediction model in step S2 is a regression model trained using machine learning methods based on historical inference task execution data. Its inputs are request features and resource configuration, and its outputs are predicted resource consumption and execution time.

[0044] Preferably, in step S3.3, when a reinforcement learning algorithm is used, the scheduler acts as an agent, its state space is a cluster resource view and a request queue state, its action space is scheduling operations (creation, routing, scaling, migration), and the reward function integrates resource utilization, request satisfaction rate, average latency, and energy consumption.

[0045] Preferably, in step S4.2, the model loading process supports obtaining model parameters of different shards and different quantization versions from a centralized model repository as needed to adapt to the dynamically allocated resource environment.

[0046] Secondly, the present invention provides a computing power scheduling system for large-scale model inference, comprising: The resource monitoring and abstraction module is used to execute step S1; The request analysis module is used to execute step S2; The global scheduling decision module is used to execute step S3; The resource allocation and deployment execution module is used to execute step S4. The runtime dynamic optimization module is used to execute step S5.

[0047] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect.

[0048] In practical use, the specific steps are as follows: Example

[0049] This embodiment describes a computing power scheduling method for large model inference, applied to a private cloud cluster consisting of multiple servers equipped with different types of GPUs. Figure 1 As shown in the overall flowchart, this method includes the following detailed steps: Step S1: Resource abstraction and monitoring.

[0050] Deploy a lightweight monitoring agent on each server node; in addition to collecting common metrics such as CPU, memory, and disk I / O, this agent focuses on collecting the following information for each GPU through an extended GPU driver interface or management library: Computing power dimension: total number of SMs (streaming multiprocessors), current active thread bundle ratio of each SM, and computing unit utilization.

[0051] Video memory dimensions: total video memory, allocated video memory, maximum contiguous free video memory block size, and address distribution map of all free video memory blocks.

[0052] Other dimensions: GPU power consumption, temperature, and interconnect bandwidth utilization with PCIe switches or other GPUs (NVLink).

[0053] The agent program logically divides the physical GPU SM cluster into multiple virtual computing units, each representing a certain proportion (e.g., 5%) of the SM computing power. Simultaneously, it divides the physical memory address space into multiple virtual memory blocks at a preset granularity (e.g., 256MB) and marks their allocation status. This abstract information, along with real-time monitoring data, is periodically reported to the central scheduler, forming a global, multi-dimensional resource view.

[0054] Step S2: Reasoning request analysis and requirement modeling.

[0055] Users submit inference requests through the API gateway. The request body includes the model name (e.g., "llama3-70B-chat"), input text, and generation parameters (max_tokens, temperature, etc.). The request analysis module intercepts the request and executes: Parse the request to determine the model identifier and the length of the input token.

[0056] Query the model metadata database to obtain the model's basic architecture information (number of layers, number of attention heads, etc.) and benchmark performance data under different configurations.

[0057] Based on the input length and the requested max_tokens, a pre-trained Transformer inference overhead prediction model is used to estimate the peak GPU memory (including KVCache) and theoretical number of floating-point operations required for the request at FP16 precision.

[0058] Based on the available resource configuration types in the current cluster (such as different combinations of virtual blocks, such as A100-40GB and H100-80GB), predict the expected processing time (TTFT and Token latency) of the request under different resource configurations.

[0059] Output the dynamic resource requirement profile of the request, for example: {Model: llama3-70B, Estimated peak memory: 35GB, Recommended minimum computing units: 8, Expected SLA: TTFT < 2s, Latency per Token < 100ms}.

[0060] Step S3: Global resource scheduling decision.

[0061] The central scheduler runs a scheduling cycle every N seconds, or is triggered when a new request arrives and there are no suitable idle instances. The decision-making process is as follows: Figure 2 As shown: Status collection: Integrates the resource views of all nodes and the status (load, queue) of all active inference instances.

[0062] Matching and Filtering: First, the "demand profile" of the new request is matched with existing instances. Matching criteria include: the instances are running the same model, the instance's remaining resources (such as free memory blocks) can meet the estimated demand of the new request, the remaining interconnect bandwidth of the node where the instance is located is sufficient, and the SLA is predicted to be met even after the new request is added. If a matching instance is found, a candidate solution of "routing to instance X" is generated.

[0063] Instance Creation Decision: If no suitable instance is found, a decision is made as to whether to create a new instance. The global resource pool is checked to find a combination of resources that can satisfy the request's "demand profile." This may require collecting discrete virtual memory blocks and virtual computing units from multiple GPUs. The scheduler uses a heuristic knapsack algorithm to allocate resources to nodes with high fragmentation and low utilization, while satisfying resource constraints, to facilitate subsequent defragmentation. Simultaneously, the cost (resource consumption) and expected benefits (request processing capacity) of creating a new instance are evaluated.

[0064] Elastic scaling and migration decisions: Simultaneously, existing instances are examined. For instances whose utilization consistently falls below the lower threshold, candidate solutions for "reducing their resource quotas" are generated; for instances with unbalanced node loads or where a better resource combination exists, candidate solutions for "migrating instance Y from node A to node B" are generated.

[0065] Multi-option evaluation and decision-making: A weighted evaluation function is used to calculate the results of multiple candidate scheduling options (routing, creation, scaling, migration). This function integrates: the predicted global resource utilization after the option is executed, the predicted average request latency, the expected resource migration overhead, and energy consumption changes. The option with the highest comprehensive evaluation is selected as the final scheduling instruction.

[0066] Step S4: Fine-grained resource allocation and instance deployment.

[0067] The scheduler executor receives the final scheduling instruction and coordinates with the node agents to complete the operation. For example, consider the instruction to "create new instances on GPU0 and GPU1 of node 5, allocating 16 virtual compute units (from both cards) and 40 virtual memory blocks (across cards)": Resource reservation and isolation: Upon receiving the instruction, the node agent launches a container on node 5 by calling the container runtime interface and a custom GPU resource management plugin. It allocates the specified CPU cores and memory to this container. Then, using the modified GPU driver context, it binds the virtual computing units numbered 0-7 on GPU0 and a set of specified virtual memory blocks (e.g., physical addresses 0x0000-0x4000, 0x8000-0xC000...) to the container's compute context; a similar operation is performed on GPU1. The driver ensures that the compute flow and memory access of each container are strictly isolated.

[0068] Adaptive Model Loading: Within the container, the model loader pulls the corresponding model file from the model repository based on the allocated resources (spanning two GPUs, with a total of 40 memory blocks, approximately 10GB*). Since the available memory is less than the required FP16 weights for the model, the loader automatically selects to load the INT4 quantized version of the model and employs a sharding strategy to deploy different layers across the memory blocks of both GPUs. Simultaneously, the parallel computing parameters of the inference engine are adjusted according to the number of allocated virtual computing units.

[0069] Service registration: After successful instance startup, it registers its service endpoint information (such as IP:Port) with the API gateway or global load balancer, and reports metadata such as the current service model, supported concurrency, and current load. The gateway updates its routing table.

[0070] Step S5: Runtime dynamic rescheduling and optimization.

[0071] Monitoring continues during instance operation.

[0072] Performance Feedback Loop: Instances report the actual processing latency and peak resource usage for each request in real time. The request analysis module compares this data with previous predictions to continuously optimize the accuracy of the performance prediction model.

[0073] Elastic scaling execution: Assume that the request queue length of instance A has consistently exceeded the high watermark for the past minute, and its GPU computing unit utilization has reached 90%. The runtime dynamic optimization module determines that it needs to be expanded and generates an instruction to "add 4 virtual computing units to instance A". The scheduler allocates 4 virtual computing units to instance A online without interrupting service (possibly through hot migration technology similar to GPU MIG or through dynamic reconfiguration of the inference engine). Instance A's inference engine detects the increased resources, dynamically adjusts the computational parallelism, improves throughput, and the queue length decreases accordingly.

[0074] Periodic defragmentation: At a fixed time each hour, the system initiates a defragmentation process. It discovers three low-priority small instances on node 3, consuming a large number of discontinuous virtual memory blocks, preventing the deployment of a new 70B model request requiring a large contiguous block of memory. Therefore, it plans a defragmentation operation: first, smoothly migrate these three small instances to other nodes with free contiguous resources (such as node 8), then release and merge all the memory blocks they occupy on node 3, forming a large contiguous free memory area. The entire migration process has minimal impact on currently processed requests, achieving resource "fragmentation reorganization."

[0075] Through the cyclical operation of the above five steps, this embodiment achieves efficient, intelligent, and automated scheduling of computing resources for large model inference. Example

[0076] This embodiment provides a system for implementing the above method, such as... Figure 3 As shown, the system is deployed at the management layer of the computing cluster, including: Resource monitoring and abstraction module: corresponding to step S1, responsible for data collection and resource abstraction.

[0077] Request Analysis Module: Corresponding to step S2, it is responsible for request parsing and demand prediction.

[0078] Global scheduling decision module: corresponding to step S3, it is the "brain" of the system and is responsible for optimization decisions.

[0079] Resource allocation and deployment execution module: corresponding to step S4, it is the system's "executor" and is responsible for implementing scheduling instructions.

[0080] Runtime dynamic optimization module: corresponding to step S5, responsible for runtime fine-tuning and optimization.

[0081] Data storage center: stores resource status, model metadata, performance history, scheduling strategies, etc.

[0082] API Gateway and Request Distributor: Provides a unified service interface to the outside world and distributes requests internally based on the routing table.

[0083] The modules mentioned above can be deployed on the same server or in a distributed manner, communicating via message queues or RPC. Example

[0084] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the method described in Embodiment 1. This electronic device can be a management node in a cluster or a standalone control server.

[0085] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0086] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A computational power scheduling method for large-scale model inference, characterized in that, Running on a computing power cluster composed of multiple heterogeneous computing nodes, the method includes: S1: Resource Abstraction and Monitoring. Through agent programs deployed on computing nodes, the system monitors and collects real-time status information of underlying physical computing resources, including GPUs, CPUs, and memory. It performs unified abstraction of heterogeneous GPU resources, constructing a multi-dimensional resource view encompassing computing units, memory blocks, and interconnect bandwidth. Specifically, it abstracts the computing core of a single GPU into multiple independently schedulable virtual computing units and GPU memory into multiple configurable virtual memory blocks. Simultaneously, it continuously collects real-time utilization data for each resource dimension and the running status of inference service instances. S2: Inference request analysis and demand modeling: Receive inference task requests submitted by users; parse the requests and extract or predict their key features; based on a preset or online learning performance prediction model, estimate the peak computing power, memory usage and expected execution time required when the request is executed according to the key features, and form a dynamic resource demand profile of the inference request. S3: Global resource scheduling decision. The scheduler periodically or upon triggering an event executes a global scheduling decision process: integrating multi-dimensional resource views of all nodes to identify idle resources, allocated resources, and various resource fragments in the current cluster; constructing a multi-objective optimization function based on the current system strategy; solving the optimization model using an optimization algorithm to generate one or more candidate scheduling schemes; the schemes include: creating new inference service instances and allocating resources for new requests, or routing them to suitable existing instances; dynamically scaling resources on existing instances; migrating instances between nodes; and releasing resources from idle instances. S4: Fine-grained resource allocation and instance deployment. Based on the final scheduling scheme generated in step S3, perform resource allocation and instance deployment operations on the target computing node: Utilize containerization technology and resource constraint mechanisms to precisely allocate a specified number of virtual computing units, a specified size and address range of virtual memory blocks, and corresponding CPU cores and system memory to the inference service instance according to the scheduling scheme; In the allocated resource environment, load the corresponding model weights, and dynamically select the model's sharding strategy, quantization level, or computation graph optimization options based on the allocated memory size and computing power specifications; Start the inference service instance and register its service endpoint to the global request dispatcher; S5: Runtime dynamic rescheduling and optimization. During the inference instance's operation, real-time data is continuously monitored; the actual resource consumption and request latency are compared with the predicted demand profile in step S2. If there is a significant deviation, compensation measures are triggered; according to the preset elastic scaling strategy, vertical or horizontal scaling of instance resources is automatically triggered; periodically or when the overall fragmentation of the system exceeds the threshold, the resource fragmentation consolidation process is initiated, releasing and merging fragmented virtual resources by migrating instances to form a larger continuous available resource block.

2. The computing power scheduling method for large-scale model inference according to claim 1, characterized in that, The unified abstraction of GPU resources in S1 specifically includes: controlling the interception and forwarding of GPU kernel drivers through a combination of hardware abstraction layer and driver extension, realizing fine-grained management and isolation of computing flow and memory access, thereby dividing the physical GPU streaming multiprocessor (SM) cluster logically into the virtual computing power units, and dividing the physical video memory address space into the virtual video memory blocks according to a preset granularity.

3. The computing power scheduling method for large model inference according to claim 1, characterized in that, In S2, the key features include: the model identifier of the request, the length of the input sequence, the maximum output length generated by the request, the accuracy requirement, and the expected latency service level agreement; the performance prediction model is a regression model trained using machine learning methods based on historical inference task execution data, with the input being the request features and resource configuration, and the output being the predicted resource consumption and execution time.

4. The computing power scheduling method for large model inference according to claim 1, characterized in that, In S3, the construction of the multi-objective optimization function specifically involves: using the weighted sum of one or more of the following as optimization objectives: maximizing overall resource utilization, minimizing average request latency, minimizing energy consumption, and minimizing resource migration overhead; and constructing a constrained optimization problem model based on resource capacity constraints, request SLA latency constraints, and model deployment constraints.

5. The computing power scheduling method for large model inference according to claim 4, characterized in that, The optimization algorithm used in S3 is a reinforcement learning algorithm; at this time, the scheduler acts as an intelligent agent, its state space is a cluster resource view and request queue state, its action space is a set of scheduling operations, and the reward function integrates quantitative indicators of resource utilization, request satisfaction rate, average latency and energy consumption.

6. The computing power scheduling method for large model inference according to claim 1, characterized in that, In S4, the dynamic selection of model sharding strategy, quantization level, or computation graph optimization option specifically includes: when allocated incomplete single-card GPU resources or multiple discrete memory blocks, automatically selecting a model parallel strategy or a more aggressive quantization level for loading; when allocated virtual resources from multiple GPUs, using tensor parallel or pipelined parallel strategies for model sharding deployment.

7. The computing power scheduling method for large model inference according to claim 1, characterized in that, The compensation measures in S5 include: dynamically increasing the local resource quota for the current instance; or rerouting some of the queued requests of the current instance to other instances in the cluster with lighter loads that are running the same model.

8. The computing power scheduling method for large model inference according to claim 1, characterized in that, The resource fragmentation process in S5 specifically includes: identifying a set of key instances that are occupied by low-priority or small-sized instances and cause resource fragmentation; planning a migration path to smoothly migrate the key instances to other computing nodes with continuous free resources; releasing all virtual computing units and virtual memory blocks occupied by the migrated instances on the source node, merging adjacent free blocks, and updating the global resource view.

9. The computing power scheduling method for large model inference according to claim 1, characterized in that, Deployed in a computing power cluster composed of multiple heterogeneous computing nodes, the system includes: The resource monitoring and abstraction module is used to monitor and collect the status information of the underlying physical computing resources in real time through the agent program deployed on the computing node, and to perform unified abstraction of GPU resources to build a multi-dimensional resource view including virtual computing power units and virtual memory blocks. The request analysis module is used to receive and parse the inference task requests submitted by users, extract key features, and form a dynamic resource demand profile based on the performance prediction model. The global scheduling decision module is used to periodically or when triggered by events to integrate the resource view, build and solve a multi-objective optimization model, generate candidate scheduling schemes that include instance creation, routing, elastic scaling, and migration, and make a final decision. The resource allocation and deployment execution module is used to perform fine-grained resource allocation, adaptive model loading, and service instance startup and registration on the target node according to the final scheduling scheme. The runtime dynamic optimization module is used to monitor performance deviations, trigger elastic scaling, and perform resource defragmentation operations during instance operation. The modules work together to implement the computing power scheduling method according to any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 8.

Images