A computing power resource management system and method for large model applications

Abstract

This application provides a computing resource management system and method for large-scale model applications. The method includes: constructing a unified resource mirror (UCRM) for large-scale model application requests, wherein the UCRM contains a resource state vector R=(T,K,M,C,P), where T is the Token state, K is the KVCache state, M is the GPU memory state, C is the communication state, and P is the parallel strategy state; the state vector in the UCRM serves as the unified control basis for scheduling decisions, GPU memory allocation, and resource release; according to the correspondence between lifecycle levels and page pool threshold intervals, non-contiguous pages are allocated from the corresponding physical GPU memory page pool and a mapping is established; actual running results are collected, prediction model parameters are updated, weight coefficients are dynamically corrected, and lifecycle level classification thresholds are adjusted. This application reduces GPU memory fragmentation rate, reduces communication overhead, and improves the utilization rate of heterogeneous computing resources and inference throughput.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a computing resource management system and method for large-scale applications. Background Technology

[0002] With the rapid development of Large Language Models (LLMs) represented by the Transformer architecture, the demand for computing resources (especially GPU memory and communication bandwidth) for training and inference is growing exponentially. Existing cloud computing resource scheduling systems (such as Kubernetes) are mainly designed for traditional microservices or data-parallel training, with coarse scheduling granularity, and cannot adapt to the unique operating characteristics of large model applications, specifically manifested in the following technical pain points: Memory fragmentation and dynamic fluctuations: During large model inference, the size of the key-value cache (KVCache) changes dynamically with the number of input and output tokens. Traditional solutions statically allocate memory for each request according to the maximum context length of the model (e.g., 32K tokens), resulting in a large amount of idle memory ("memory wall") and easily causing memory overflow (OOM) errors.

[0003] Communication bottleneck in heterogeneous computing power: Tensor parallelism (TP) and pipelined parallelism (PP) of large models are extremely sensitive to the communication bandwidth of high-bandwidth interconnects within nodes (such as dedicated inter-chip interconnect buses) and general-purpose buses between nodes (such as peripheral component interconnect standard buses). Existing schedulers allocate tasks only according to the remaining video memory or CPU load, ignoring the real-time differences in the underlying communication topology, resulting in the "communication wall" phenomenon, which severely slows down the speed of distributed inference.

[0004] Load fluctuations and timing conflicts: Large model inference includes a computationally intensive prefill phase and a storage access-intensive decoding phase. Traditional First-Come, First-Served (FCFS) scheduling causes prefill tasks to block decoding tasks, making it impossible to utilize idle computing power in microsecond intervals, resulting in low resource utilization.

[0005] Therefore, there is an urgent need for a computing power management system that can sense the dynamic characteristics of LLM runtime (Token length, KVCache lifecycle, communication topology) and realize precise on-demand scheduling of micro-resources (memory pages, bandwidth time slots, computing units). Summary of the Invention

[0006] This application aims to provide a computing resource management system and method for large-scale model applications. By using a unified resource mirror (UCRM), business characteristics, KVCache state, memory state, communication state, and parallel strategy state are integrated into a continuously evolving state vector. Based on this state vector, cost function optimization, lifecycle page pool allocation, and closed-loop feedback are driven, solving the technical problems of memory fragmentation, communication bottlenecks, and scheduling fragmentation in the prior art.

[0007] This application provides a computing resource management system for large-scale model applications. The aforementioned modules, engines, and units reside in the host memory of the computing resource management system and are executed by at least one physical central processing unit (CPU) of the host machine.

[0008] First, the system builds a unified resource image (UCRM) for each large model application request.

[0009] Specifically, upon receiving a large model inference request, business-side features (input token length, model structure parameters) and hardware-side features (remaining node memory, communication bandwidth) are collected to generate a unique UCRM for each request. The UCRM contains a resource state vector R=(T,K,M,C,P), where: T (Token State): Distribution of the length of the input tokens currently processed and the expected length of the output tokens; K (KVCache Status): Number of currently allocated KVCache virtual blocks, expected peak number of blocks, and lifecycle level; M (Memory Status): List of mapped physical memory pages, page pool ownership, fragmentation rate; C (Communication Status): The inter-node communication bandwidth requirements and real-time topology mapping associated with this request; P (Parallel Strategy State): The currently used TP / PP strategy, slice boundaries, and node allocation.

[0010] The state vector in this UCRM serves as the unified control basis for subsequent scheduling decisions, memory allocation, and resource release, and is also a shared control data object for all modules.

[0011] Next, the system converts the business characteristic information into the initial value of the state vector in UCRM, and calculates the expected peak number of blocks and lifecycle level according to the prediction model, and writes it into the K component.

[0012] Specifically, based on business characteristic information and historical statistics, a pre-trained lightweight regression model is used to predict the peak number of virtual blocks in the KVCache and the lifespan level (short, medium, long) of the request throughout its entire lifecycle. The prediction results are directly converted into the K component (peak_blocks, life_level) in the state vector and written into UCRM.

[0013] Based on this, the system generates computational cost items, memory cost items, communication cost items, and resource reallocation cost items according to the T, K, M, and C state components in UCRM, and constructs a resource coordination cost matrix.

[0014] Specifically, the computation cost term T_c is estimated based on the token length in the T component to predict computation time; the memory cost term M_f is based on the peak number of blocks and the current fragmentation rate in the K component; the communication cost term T_n is based on the communication requirements and real-time topology bandwidth in the C component; and the resource reallocation cost term R_r is based on the policy change history in the P component. This forms the basis of a resource coordination cost matrix.

[0015] Subsequently, based on the resource coordination cost matrix, the system uses a weighted sum minimization method to calculate the cost of candidate parallel slicing strategies, selects the candidate parallel slicing strategy with the lowest cost value as the target distributed parallel slicing strategy, and determines the corresponding node mapping relationship.

[0016] To achieve this goal, the system predefines a resource coordination cost function J=α·T c +β·T n +γ·M f +δ·R r Where α, β, γ, and δ are weight coefficients, satisfying α + β + γ + δ = 1, and each weight coefficient can be dynamically corrected and normalized based on subsequent closed-loop feedback. The scheduling decision engine calculates the J value for each candidate strategy and selects the strategy with the smallest J.

[0017] Next, based on the correspondence between the lifecycle level of the K component in UCRM and the page pool threshold range, the system allocates non-contiguous physical memory pages from the corresponding physical memory page pool, establishes a mapping relationship between virtual memory blocks and physical memory pages, and stores the mapping relationship in the M component of UCRM.

[0018] Specifically, the system maintains short-lifetime page pools, medium-lifetime page pools, and long-lifetime page pools, with the capacity and lifetime thresholds of each page pool dynamically adjustable. The correspondence between lifetime levels and page pool threshold ranges is determined based on preset lifetime threshold division rules (e.g., short <1s, medium 1s~10s, long >10s). When the K component in the UCRM indicates the need to allocate KVCache virtual blocks, non-contiguous physical memory pages are allocated from the corresponding page pool, a mapping table from virtual blocks to physical pages is established, stored in the M component of the UCRM, and on-demand append allocation and atomic release are supported.

[0019] Finally, the system collects the actual running results, calculates the error with the UCRM prediction value, uses the error to update the prediction model parameters, dynamically corrects the cost function weight coefficients, and adjusts the life cycle level classification threshold; after the dynamic correction, the corrected weight coefficients are normalized to meet the preset constraint that the sum is 1.

[0020] Specifically, after the task is completed, the actual number of KVCache blocks consumed, the actual communication latency, and the actual memory fragmentation rate are collected, and the error between these values ​​and the predicted values ​​in UCRM is calculated. Feedback includes: updating the KVCache prediction model parameters (e.g., using online gradient descent); dynamically correcting the cost function weight coefficients α, β, γ, and δ, and maintaining a sum of 1 through normalization; dynamically adjusting the lifecycle level classification threshold; and storing the updated UCRM state vector in a historical statistics database for use as input to the prediction model for subsequent requests of the same type. This forms a closed-loop self-evolving system.

[0021] Compared with the prior art, the beneficial effects of the present invention are: 1. Improved memory utilization: Reduce fragmentation rate and increase concurrency through predictive dynamic paged memory management and lifecycle page pools.

[0022] 2. Communication bottleneck mitigation: The cost function explicitly considers communication costs, and the scheduling decision optimizes communication topology adaptation.

[0023] 3. Adaptive closed loop: Feedback updates the prediction model, cost weights, and lifecycle thresholds simultaneously, enabling continuous system optimization.

[0024] 4. Unified control basis: UCRM state vector runs through the entire link of perception, prediction, scheduling, allocation and feedback, avoiding decision conflicts between multiple modules. Attached Figure Description

[0025] To illustrate the technical solutions in the embodiments of the present invention or the prior art more specifically and intuitively, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0026] Figure 1This is a schematic diagram of the overall system architecture provided for an embodiment of this application.

[0027] Figure 2 This is a schematic diagram of the structure and state vector of the Unified Resource Mirror (UCRM) provided in the embodiments of this application.

[0028] Figure 3 This is a schematic diagram of lifecycle page pool management provided in an embodiment of this application.

[0029] Figure 4 This is a schematic diagram of the resource coordination cost function optimization process provided in the embodiments of this application.

[0030] Figure 5 The overall system operation flowchart provided for the embodiments of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment takes a large language model inference service with an autoregressive architecture running on a heterogeneous computing cluster (containing 8 general-purpose graphics processing units (GPUs) of the first architecture, of which 4 are fully interconnected through a high-bandwidth inter-chip interconnect bus) deployed by a cloud service provider as an example. Figures 1 to 5 The document showcases the overall system architecture, UCRM structure, page pool management, cost function optimization, and overall operation process.

[0032] Example: Cooperative scheduling of large model inference requests based on unified resource mirroring Step 1: Build a unified resource mirror UCRM (see...) Figure 1 , Figure 2 ) like Figure 1 As shown, the system first collects business and hardware features through a multi-dimensional feature perception layer. Specifically, the system intercepts the user's inference request, "Please explain the principle of quantum entanglement in detail." After processing by the token segmenter, the input token length is 15. Target model structure: 32 layers, 4096 hidden layer dimensions, 32 attention heads. Hardware topology: 600GB / s bandwidth within the high-bandwidth group, 32GB / s bandwidth across groups, 12GB of remaining video memory on each GPU, etc.

[0033] like Figure 2 As shown, the system generates a unique unified resource image (UCRM) for this request, and initializes the state vector R=(T,K,M,C,P): T:cur_len=15,pred_len=None K:alloc_blocks=0,peak_blocks=None,life_level=None M:mapped_pages=[],pool_cluster_id=None,frag_rate=0 C:req_bandwidth=None,topo_mapping=None P:tp_size=None,pp_stages=None,node_map=None This UCRM is stored in a global image repository as a unified control basis for all subsequent decisions. The state vectors in the UCRM serve as a unified control basis for scheduling decisions, memory allocation, and resource release, and are also shared control data objects among all modules.

[0034] Step 2: KVCache Prediction and State Update (see...) Figure 1 , Figure 4 ) like Figure 1 The prediction module and Figure 4 The cost term generation process shown involves inputting a token length of 15, model parameters, and historical statistics (each output token consumes approximately 2KB of GPU memory on average, with each page being 4KB) into a pre-trained lightweight regression model. The model predicts an output token length of approximately 800, requiring a peak KVCache block count of 800 × 2KB × 32 layers / 4KB = 12800 blocks, with an expected lifespan of approximately 2 seconds (marked as a medium lifespan). The prediction result is written to the K component of the UCRM: peak_blocks = 12800, life_level = "mid". This lifespan level is subsequently used for page pool selection and GPU memory fragmentation cost calculation in the cost function.

[0035] Step 3: Construct the resource coordination cost matrix (see...) Figure 4 ) like Figure 4 As shown, based on the T, K, M, and C state components in UCRM, the following are generated respectively: The calculation time for the cost term T_c is estimated to be approximately 0.8 seconds based on the token length in the T component. Memory cost term M_f: Calculated based on the peak number of blocks in the K component and the current fragmentation rate, resulting in a memory fragmentation cost of 0.05; Communication cost term T_n: Based on the communication requirements in the C component and the real-time topology bandwidth, the communication time is calculated to be 0.05 seconds. Resource reallocation cost item R_r: The reallocation cost is calculated based on the policy change history in the P component, which is 0.02.

[0036] This leads to the construction of a resource coordination cost matrix. This matrix serves as the input for subsequent scheduling decisions.

[0037] Step 4: Hybrid Parallel Scheduling Decision (see...) Figure 1 , Figure 4 ) like Figure 1 The scheduling engine in Figure 4 As shown in the cost function optimization process, the system adopts a weighted sum minimization method, with preset weight coefficients α=0.4, β=0.3, γ=0.2, and δ=0.1. The resource coordination cost function is J=α·T_c+β·T_n+γ·M_f+δ·R_r, satisfying α+β+γ+δ=1. The J values ​​for the two candidate strategies are calculated: Strategy A (pure data parallelism): J=0.54; Strategy B (TP=2, PP=2, TP within a group, PP across groups): J=0.347.

[0038] Select strategy B with the smallest J value as the target distributed parallel slicing strategy, determine the corresponding node mapping relationship, and write the decision result into the P component of UCRM.

[0039] Step 5: Lifecycle page pool allocation and video memory mapping (see...) Figure 1 , Figure 3 ) like Figure 3 As shown, based on the "medium" lifecycle level of the K component in UCRM, and according to the preset lifecycle threshold division rules (short <1s, medium 1s~10s, long >10s), the corresponding page pool is determined to be a medium lifecycle page pool. Non-contiguous physical memory pages are allocated from the medium lifecycle page pool, initially allocating 4096 pages. A mapping table from virtual blocks to physical pages is established and stored in the M component of UCRM. The mapping records virtual block 0 → physical page 0x1234, etc. During decoding, the KVCache dynamically grows, and new pages are appended and allocated from the same page pool as needed, with the mapping table updated synchronously. After the task is completed, all physical pages are atomically released and the mapping table is cleared.

[0040] Step 6: Inference Execution and Dynamic Adjustment (see...) Figure 1 , Figure 5 ) like Figure 5 As shown in the overall flowchart, the system performs large model inference according to the allocated resources. During the decoding phase, if the KVCache continues to grow, on-demand appending is automatically triggered; if an idle gap is detected (such as a decoding phase interval exceeding 50 microseconds), computing power slot preemption (an optional feature) can be initiated, parasitically scheduling the pre-filling phase of new requests to be executed in the idle stream. This process continues until inference is completed.

[0041] Step 7: Closed-loop feedback and self-evolution (see...) Figure 1 , Figure 5 ) like Figure 1Bottom feedback loop and Figure 5 As shown in step S7, after the task is completed, the actual number of KVCache blocks consumed is 13100, the actual communication latency is 0.06s, and the actual fragmentation rate is 4%. Calculate the error between the actual value and the UCRM prediction, and execute: Update the KVCache prediction model parameters (online gradient descent, learning rate 0.01). The weighting coefficients are dynamically adjusted, and α+β+γ+δ=1 is maintained through normalization (for example, β is adjusted from 0.3 to 0.31, and other coefficients are adjusted accordingly). Adjust the lifecycle level classification threshold (e.g., adjust the short / medium boundary from 1s to 0.9s); Store the UCRM state vector in a historical statistics database for use as input to the prediction model for subsequent requests of the same type.

[0042] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for managing computing resources for large-scale model applications, characterized in that, Includes the following steps: S1: Construct a unified resource image (UCRM) for large model application requests. The UCRM contains a resource state vector R=(T,K,M,C,P), where T is the Token state, K is the KVCache state, M is the video memory state, C is the communication state, and P is the parallel strategy state. The state vector in the UCRM serves as the unified control basis for scheduling decisions, video memory allocation decisions, and resource release decisions. S2: Convert the business feature information into the initial value of the state vector in UCRM, and calculate the expected peak number of blocks and life cycle level according to the prediction model, and write it into the K component; S3: Generate computation cost items, memory cost items, communication cost items, and resource reallocation cost items based on the T, K, M, and C state components in the UCRM, and construct a resource coordination cost matrix; S4: Based on the resource coordination cost matrix, the weighted sum minimization method is used to calculate the cost of the candidate parallel slicing strategies, and the candidate parallel slicing strategy with the lowest cost value is selected as the target distributed parallel slicing strategy, and the corresponding node mapping relationship is determined. S5: Based on the correspondence between the lifecycle level of the K component in UCRM and the page pool threshold range, allocate non-contiguous physical memory pages from the corresponding physical memory page pool, establish a mapping relationship between virtual memory blocks and physical memory pages, and store the mapping relationship in the M component of UCRM. S6: Collect actual running results, calculate the error with the UCRM prediction value, use the error to update the prediction model parameters, dynamically correct the cost function weight coefficients, and adjust the life cycle level classification threshold; After the dynamic correction, the corrected weight coefficients are normalized to satisfy the preset constraint that the sum is 1.

2. The method according to claim 1, characterized in that, The K component in the UCRM includes the number of currently allocated virtual blocks, the expected peak number of blocks, and the lifecycle level; the lifecycle level is output by the prediction model and includes short lifecycle, medium lifecycle, and long lifecycle.

3. The method according to claim 1, characterized in that, The resource synergy cost function used for cost calculation in step S4 is J = α·T c +β·T n +γ·M f +δ·R r , where α, β, γ, and δ are weighting coefficients, satisfying α+β+γ+δ=1, and each weighting coefficient is dynamically corrected according to the error in step S6; after the dynamic correction, normalization is performed so that the corrected α, β, γ, and δ satisfy α+β+γ+δ=1.

4. The method according to claim 1, characterized in that, The physical memory page pool in step S5 includes a short-lifetime page pool, a medium-lifetime page pool, and a long-lifetime page pool. The capacity and lifetime threshold of each page pool are dynamically adjusted according to the error in step S6. The correspondence between the lifetime level and the page pool threshold range is determined according to the preset lifetime threshold division rules.

5. The method according to claim 1, characterized in that, In step S6, the online gradient descent method is used to update the prediction model parameters; when dynamically correcting the weight coefficients of the cost function, the constraint that the sum of the weights is 1 is maintained.

6. A computing resource management system for large-scale model applications, characterized in that, include: The Unified Resource Image Building Module is used to generate a UCRM for each large model application request. The UCRM contains a resource state vector R=(T,K,M,C,P); the state vector in the UCRM serves as a shared control data object for each module. The KVCache prediction and status update module is connected to the unified resource image building module and is used to predict the peak number of KVCache blocks and lifecycle level and update the K component of UCRM. The resource coordination cost matrix and cost function calculation module is used to generate calculation cost items, memory cost items, communication cost items, and resource reallocation cost items based on the T, K, M, and C state components in UCRM, construct the cost matrix, and calculate the cost function value based on the weighted sum and minimization method. A hybrid parallel scheduling decision engine, connected to the resource coordination cost matrix and cost function calculation module, is used to select the candidate parallel slicing strategy with the lowest cost value as the target distributed parallel slicing strategy and determine the node mapping relationship. The lifecycle page pool management unit, connected to the UCRM and scheduling decision engine, is used to allocate non-contiguous physical memory pages from the corresponding page pool according to the correspondence between the lifecycle level in the K component of the UCRM and the page pool threshold range, and to establish a mapping from virtual blocks to physical pages. The mirror-driven closed-loop feedback module is used to collect actual running results, update prediction model parameters, dynamically correct cost function weight coefficients, and adjust life cycle level classification thresholds. After correction, the weight sum is maintained to a constraint of 1 through normalization.

7. The system according to claim 6, characterized in that, The lifecycle page pool management unit maintains short-lifecycle page pools, medium-lifecycle page pools, and long-lifecycle page pools, and the lifecycle threshold of each page pool can be dynamically adjusted.

8. The system according to claim 6, characterized in that, The system also includes a computing power time slot allocation and preemption unit, which is used to implement staggered execution of the pre-filling stage and the decoding stage based on the parallel strategy status in UCRM and the monitored idle gaps of the GPU computing flow.

9. The system according to claim 6, characterized in that, The cost function used in the resource coordination cost matrix and cost function calculation module is J=α·T. c +β·T n +γ·M f +δ·R r , where α, β, γ, δ satisfy α+β+γ+δ=1, and are corrected and normalized online by the mirror-driven closed-loop feedback module.

10. The method according to claim 1, characterized in that, The weighted sum minimization method in step S4 is based on the resource collaborative cost function J described in claim 3, and the candidate parallel slicing strategies include tensor parallelism (TP), pipeline parallelism (PP), and combinations thereof.

Images