Resource planning method and apparatus, electronic device, storage medium and program product

Abstract

The present disclosure relates to a resource planning method, device, electronic equipment, storage medium and program product, the method comprising: obtaining a model image, a hardware image and a scene image; generating at least two resource planning candidate schemes based on the hardware image; calculating the memory surplus of each resource planning candidate scheme based on the model image, the hardware image and the scene image; and selecting a target resource planning scheme from the at least two resource planning candidate schemes based on the calculated memory surplus of each resource planning candidate scheme. In this way, by taking the hardware image, the model image and the scene image as unified input, the feasibility determination and robust ordering of the resource planning scheme can be formed before resource deployment, avoiding the problem of long iteration cycle and high cost of resource planning scheme caused by relying only on pre-online stress testing or runtime trial and error, thereby improving the explainability, reproducibility and transferability of resource planning.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to resource planning methods, apparatus, electronic devices, storage media, and program products. Background Technology

[0002] In mainstream Transformer inference, the Prefill and Decode stages have different sensitivities to hardware resources. The Prefill stage primarily processes the input sequence, and its computation typically takes the form of a hybrid of "large matrix computation + attention computation." The Decode stage mainly generates tokens one by one, with relatively smaller single-step computations, but it requires repeatedly reading the historical KV cache and continuously appending new KV values. Therefore, the Decode stage is more sensitive to GPU memory capacity and bandwidth. When the application simultaneously pursues a longer context length (CTX) and higher concurrency, the KV cache will quickly become a core constraint of the system. This is because it brings both a linear increase in GPU memory usage and continuous pressure on read and write bandwidth, which will ultimately manifest in tail latency and stability.

[0003] In practical engineering, multiple inference backends are often supported simultaneously to cover the stability and performance differences under different model families, different quantization forms, and different CTX tiers. In this case, the deployment configuration not only includes parallel parameters, but also engine selection and key parameter combinations related to the engine. The configuration space is significantly expanded, the cost of manual trial and error increases, and reusability is poor.

[0004] In related technologies, resource planning schemes are primarily evaluated through offline load testing, which involves multiple rounds of load testing before deployment to screen different parallelism / configuration combinations. However, even if the load testing results show that the parallel parameters are feasible, issues such as communication jitter and tail latency degradation caused by cross-domain / cross-machine Tensor Parallelism (TP) may still occur. Furthermore, these risks often only become apparent after load testing or deployment, resulting in long iteration cycles and high costs for resource planning schemes. Summary of the Invention

[0005] This disclosure provides resource planning methods, apparatus, electronic devices, storage media, and program products to at least solve the problems of long iteration cycles and high costs of resource planning schemes in the aforementioned related technologies.

[0006] According to a first aspect of the present disclosure, a resource planning method is provided, comprising: acquiring a model profile, a hardware profile, and a scenario profile, wherein the model profile is used to characterize the inference attributes of each large language model in a service cluster, the hardware profile is used to characterize the configuration attributes of the service cluster, and the scenario profile is used to characterize the demand characteristics of the business scenario to be served by the service cluster; generating at least two resource planning candidate schemes based on the hardware profile, wherein the resource planning candidate schemes are schemes for allocating GPU resources for each of a variety of parallel strategies corresponding to the large language model, the parallel strategies are parallel strategies adopted for distributed inference of the large language model, and the GPUs are GPUs in the device nodes included in the service cluster; calculating the remaining GPU memory for each of the at least two resource planning candidate schemes based on the model profile, the hardware profile, and the scenario profile, wherein the remaining GPU memory is the remaining amount within the upper limit of GPU memory allowed for resource planning, and the GPU memory is the GPU memory; and selecting a target resource planning scheme from the at least two resource planning candidate schemes based on the calculated remaining GPU memory for each resource planning candidate scheme.

[0007] Optionally, the hardware profile includes the number of available GPUs, wherein the available GPUs are all or some of the GPUs included in the service cluster; generating at least two resource planning candidate schemes based on the hardware profile includes: allocating at least one GPU for each of the multiple parallel strategies based on the number of available GPUs and a preset resource allocation rule, thereby obtaining the at least two resource planning candidate schemes, wherein the preset resource allocation rule is that the number of available GPUs is the product of the number of GPUs allocated to each of the multiple parallel strategies.

[0008] Optionally, the hardware profile includes the number of usable GPUs, wherein the usable GPUs are all or some of the GPUs included in the service cluster; generating at least two resource planning candidate schemes based on the hardware profile includes: determining a first number of GPUs allocated for the prefilling stage and a second number of GPUs allocated for the decoding stage, wherein the sum of the first number and the second number is the number of usable GPUs, and the prefilling stage and the decoding stage are two stages included in the inference process of the large language model; for each stage in the prefilling stage and the decoding stage, based on the number of GPUs allocated to that stage and a preset resource allocation rule, at least one GPU is allocated for each of the multiple parallel strategies to obtain the at least two resource planning candidate schemes, wherein the preset resource allocation rule is that the product of the number of GPUs allocated to each of the multiple parallel strategies is the number of GPUs allocated in the corresponding stage.

[0009] Optionally, the model profile includes the following items: attention head configuration, key-value structure, number of bytes, number of layers, and total weights after quantization; the hardware profile includes the following items: runtime memory overhead and the upper limit of memory allowed by the resource planning; the scenario profile includes the following items: concurrency limit, skew coefficient, context length, and effective sharding factor of key-value pairs in the tensor parallel dimension.

[0010] Optionally, calculating the video memory remaining space for each of the at least two resource planning candidate schemes based on the model profile, the hardware profile, and the scene profile includes: calculating the total video memory usage for each resource planning candidate scheme based on the model profile, the hardware profile, and the scene profile, wherein the total video memory usage is used to indicate the estimated total video memory usage of a single GPU under the resource planning candidate scheme; and calculating the video memory remaining space for the resource planning candidate scheme based on the hardware profile and the total video memory usage.

[0011] Optionally, the step of calculating the total video memory usage for each resource planning candidate scheme based on the model profile, the hardware profile, and the scenario profile includes: calculating the effective concurrency upper bound per replica for each resource planning candidate scheme based on the concurrency upper limit and the skew coefficient; calculating the single-request KV usage based on the attention head configuration, the KV structure, the number of bytes, the number of layers, and the context length; calculating the peak KV usage per GPU based on the effective concurrency upper bound per replica, the single-request KV usage, and the effective sharding factor of KV in the tensor parallel dimension; calculating the weight usage per GPU based on the total quantized weights; and calculating the total video memory usage based on the weight usage per GPU, the peak KV usage per GPU, and the runtime video memory overhead.

[0012] Optionally, the step of calculating the video memory reserve of the resource planning candidate scheme based on the hardware profile and the total video memory usage includes: calculating the difference between the upper limit of video memory allowed by the resource planning and the total video memory usage as the video memory reserve of the resource planning candidate scheme.

[0013] Optionally, selecting a target resource planning scheme from the at least two resource planning candidate schemes based on the calculated memory reserve of each resource planning candidate scheme includes: selecting the resource planning candidate scheme with the largest memory reserve among the at least two resource planning candidate schemes as the target resource planning scheme.

[0014] Optionally, the multiple parallel strategies include tensor parallelism, pipeline parallelism, expert parallelism, and data parallelism; the step of selecting a target resource planning scheme from the at least two resource planning candidate schemes based on the calculated GPU memory availability of each resource planning candidate scheme includes: for each resource planning candidate scheme, calculating a GPU memory availability score based on the GPU memory availability of that resource planning candidate scheme, wherein the larger the GPU memory availability, the higher the corresponding GPU memory availability score; calculating an operational complexity score based on the number of GPUs allocated to each of the pipeline parallelism, expert parallelism, and data parallelism strategies, wherein each of the parallelism strategies... The more GPUs the row strategy is allocated to, the lower the operational complexity score. A topology risk score is calculated based on the data communication type of the GPUs allocated to the tensor parallel strategy, wherein the topology risk score for intra-machine communication is higher than that for cross-machine communication. A total score is calculated based on the memory availability score, the operational complexity score, and the topology risk score. Based on the total score, the target resource planning scheme is selected from at least two resource planning candidate schemes, wherein a higher total score increases the likelihood that the corresponding resource planning candidate scheme will be selected as the target resource planning scheme.

[0015] Optionally, the resource planning method further includes: performing structuring processing on the target resource planning scheme to obtain a structured resource planning scheme; and deploying resources based on the structured resource planning scheme.

[0016] Optionally, obtaining the model profile, hardware profile, and scenario profile includes: obtaining the model structure information of the large language model, the hardware information of the service cluster, and the business-side deployment target information; and performing standardization processing on the model structure information of the large language model, the hardware information of the service cluster, and the business-side deployment target information to obtain the model profile, the hardware profile, and the scenario profile.

[0017] According to a second aspect of the present disclosure, a resource planning apparatus is provided, comprising: a profile acquisition module configured to acquire a model profile, a hardware profile, and a scenario profile, wherein the model profile is used to characterize the inference attributes of each large language model in a service cluster, the hardware profile is used to characterize the configuration attributes of the service cluster, and the scenario profile is used to characterize the demand characteristics of the business scenario to be served by the service cluster; and a scheme generation module configured to generate at least two resource planning candidate schemes based on the hardware profile, wherein the resource planning candidate schemes are schemes for allocating GPU resources for each of the multiple parallel strategies corresponding to the large language model. The parallel strategy is the parallel strategy adopted for distributed inference of the large language model, and the GPU is the GPU in the device nodes included in the service cluster; the memory reserve calculation module is configured to calculate the memory reserve of each of the at least two resource planning candidate schemes based on the model profile, the hardware profile, and the scenario profile, wherein the memory reserve is the remaining reserve within the upper limit of memory allowed for resource planning, and the memory is the memory of the GPU; the scheme selection module is configured to select a target resource planning scheme from the at least two resource planning candidate schemes based on the calculated memory reserve of each resource planning candidate scheme.

[0018] Optionally, the hardware profile includes the number of available GPUs, wherein the available GPUs are all or some of the GPUs included in the service cluster; the scheme generation module is configured to: allocate at least one GPU for each of the multiple parallel strategies based on the number of available GPUs and a preset resource allocation rule, thereby obtaining the at least two resource planning candidate schemes, wherein the preset resource allocation rule is that the number of available GPUs is the product of the number of GPUs allocated to each of the multiple parallel strategies.

[0019] Optionally, the hardware profile includes the number of usable GPUs, wherein the usable GPUs are all or some of the GPUs included in the service cluster; the scheme generation module is configured to: determine a first number of GPUs allocated for the prefilling stage and a second number of GPUs allocated for the decoding stage, wherein the sum of the first number and the second number is the number of usable GPUs, the prefilling stage and the decoding stage are two stages included in the inference process of the large language model; for each stage in the prefilling stage and the decoding stage, based on the number of GPUs allocated to that stage and a preset resource allocation rule, allocate at least one GPU for each of the multiple parallel strategies to obtain the at least two resource planning candidate schemes, wherein the preset resource allocation rule is that the product of the number of GPUs allocated to each of the multiple parallel strategies is the number of GPUs allocated in the corresponding stage.

[0020] Optionally, the model profile includes the following items: attention head configuration, key-value structure, number of bytes, number of layers, and total weights after quantization; the hardware profile includes the following items: runtime memory overhead and the upper limit of memory allowed by the resource planning; the scenario profile includes the following items: concurrency limit, skew coefficient, context length, and effective sharding factor of key-value pairs in the tensor parallel dimension.

[0021] Optionally, the memory availability calculation module is configured to: calculate the total memory usage for each resource planning candidate scheme based on the model profile, the hardware profile, and the scene profile, wherein the total memory usage is used to indicate the estimated total memory usage of a single GPU under the resource planning candidate scheme; and calculate the memory availability of the resource planning candidate scheme based on the hardware profile and the total memory usage.

[0022] Optionally, the memory availability calculation module is configured to: for each resource planning candidate scheme, calculate the effective concurrency upper bound per replica based on the concurrency upper limit and the skew coefficient; calculate the single-request KV occupancy based on the attention head configuration, the KV structure, the number of bytes, the number of layers, and the context length; calculate the peak KV occupancy per GPU based on the effective concurrency upper bound per replica, the single-request KV occupancy, and the effective sharding factor of KV in the tensor parallel dimension; calculate the weight occupancy per GPU based on the total quantized weights; and calculate the total memory occupancy based on the weight occupancy per GPU, the peak KV occupancy per GPU, and the runtime memory overhead.

[0023] Optionally, the video memory reserve calculation module is configured to: calculate the difference between the upper limit of video memory allowed by the resource planning and the total video memory usage as the video memory reserve of the resource planning candidate scheme.

[0024] Optionally, the scheme selection module is configured to: select the resource planning candidate scheme with the largest available video memory among the at least two resource planning candidate schemes as the target resource planning scheme.

[0025] Optionally, the multiple parallel strategies include tensor parallelism, pipeline parallelism, expert parallelism, and data parallelism. The scheme selection module is configured to: for each resource planning candidate scheme, calculate a memory availability score based on the available memory for that resource planning candidate scheme, wherein a larger memory availability results in a higher memory availability score; and calculate an operational complexity score based on the number of GPUs allocated to each of the pipeline parallelism, expert parallelism, and data parallelism strategies, wherein a higher number of GPUs allocated to each parallelism strategy results in a lower operational complexity score. The lower the complexity score, the higher the topology risk score. Based on the data communication type of the GPU to which the tensor parallel strategy is assigned, a topology risk score is calculated, wherein the topology risk score for data communication type intra-machine communication is higher than that for data communication type inter-machine communication. Based on the memory availability score, the operational complexity score, and the topology risk score, a total score is calculated. Based on the total score, the target resource planning scheme is selected from the at least two resource planning candidate schemes, wherein the higher the total score, the greater the likelihood that the corresponding resource planning candidate scheme will be selected as the target resource planning scheme.

[0026] Optionally, the resource planning device further includes: a structured processing module configured to perform structured processing on the target resource planning scheme to obtain a structured resource planning scheme; and a resource deployment module configured to deploy resources based on the structured resource planning scheme.

[0027] Optionally, the profile acquisition module is configured to: acquire the model structure information of the large language model, the hardware information of the service cluster, and the business-side deployment target information; and perform standardization processing on the model structure information of the large language model, the hardware information of the service cluster, and the business-side deployment target information to obtain the model profile, the hardware profile, and the scenario profile.

[0028] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a resource planning method according to the present disclosure.

[0029] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a resource planning method according to the present disclosure.

[0030] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the resource planning method according to the present disclosure.

[0031] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0032] This disclosure unifies hardware conditions, model structure, and business scenarios into an interpretable static planning framework to form a feasibility assessment and robust ranking mechanism for resource planning schemes. Specifically, by using hardware profiles, model profiles, and scenario profiles as unified inputs, this disclosure enables the feasibility assessment and robust ranking of resource planning schemes before resource deployment. This avoids the problems of long iteration cycles and high costs associated with relying solely on pre-deployment stress testing or runtime trial and error for resource planning schemes, thereby improving the interpretability, recalculation, and transferability of resource planning.

[0033] According to exemplary embodiments of this disclosure, by using communication domain / interconnection capability as an explicit constraint, prioritizing the placement of high-frequency communication parallel groups within the same high-speed interconnection domain, and explicitly marking cross-domain boundaries, it is possible to reduce tail delay jitter and stability risks.

[0034] According to exemplary embodiments of this disclosure, by decomposing and estimating weights, KV cache, runtime overhead, and security reservations, it is possible to determine both "infeasible" (over budget) and "high-risk edge" solutions, which can then be addressed in scoring or policies, thereby reducing OOM and uncertainty.

[0035] According to an exemplary embodiment of this disclosure, multi-constraint pruning + interpretable scoring selection: by pruning with hard constraints such as memory capacity, topology rules, and inference backend availability, and scoring within the feasible set according to interpretable sub-items such as memory margin, topology risk, and operational complexity, a more robust resource planning scheme can be output.

[0036] According to an exemplary embodiment of this disclosure, a deployment configuration list that can be directly implemented can be output: the planner can output information including parallel parameters, mapping from dp replicas to node / gpu, key engine parameter suggestions, risk tags, etc., so that the planning results can be directly used for deployment and operation and maintenance.

[0037] According to exemplary embodiments of this disclosure, a unified planning expression for PD separation can be supported. That is, in long prefix scenarios, the resource allocation, parallel parameters and placement mapping summary of prefill_plan and decode_plan can be output, and risks such as KV handover can be alerted, thereby enhancing feasibility and controllability.

[0038] According to an exemplary embodiment of this disclosure, the quantization form can be used as a constraint to ensure product consistency: it can receive an externally specified quantization form / precision as an input constraint, and can record the quant_' and model_artifact (including quant_artifact_id / path) fields in the Plan, thereby ensuring that the deployment plan is consistent with the actual loaded model product, thereby reducing the risk of deployment deviation.

[0039] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0041] Figure 1 This is a schematic diagram illustrating a resource planning architecture according to exemplary embodiments of the present disclosure;

[0042] Figure 2 This is a flowchart illustrating a resource planning method according to exemplary embodiments of the present disclosure;

[0043] Figure 3 This is a block diagram illustrating a resource planning apparatus according to an exemplary embodiment of the present disclosure;

[0044] Figure 4 This is a block diagram illustrating an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0046] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0047] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0048] The performance and stability of multi-machine, multi-GPU deployments largely depend on the data exchange paths between GPUs. Within the same machine, GPUs may reside in the same NVLink interconnect domain (high-speed interconnect link between GPUs on the same machine), or they may require bridging across the Peripheral Component Interconnect Express (PCIe) bus. When crossing machines, they rely on network links such as IB / ROCE / Ethernet. Differences in bandwidth, latency, and jitter among different links directly affect the stability of aggregated communication, thereby impacting performance metrics such as Time To First Token (TTFT) and P95. Furthermore, the sensitivity of parallel strategies to communication paths varies. For example, TP involves high-frequency aggregated communication and is most sensitive to link capacity; Pipeline Parallelism (PP) concentrates communication at stage boundaries, and the frequency is usually lower; Data Parallelism (DP) improves overall throughput by splitting traffic among multiple replicas, and high-frequency aggregated GPU communication is usually not required between replicas; The MoE model can distribute expert weights after introducing Expert Parallelism (EP), but at the same time it will introduce routing-related data exchange and load unevenness risks.

[0049] In related technologies, the common implementation methods for deployment and resource planning for large model inference mainly include the following categories:

[0050] (1) Staff adjust parameters based on experience

[0051] Operations / engineering personnel select parameters such as tp / pp / dp / ep based on model size, memory capacity, and historical experience, and manually specify node mapping to GPU in a multi-machine, multi-GPU environment.

[0052] (2) Internal heuristic / default configuration of the inference engine

[0053] Some inference engines provide default suggestions for parallel parameters, pagination / blocking, batch processing limits, etc. when loading models, but their output is usually focused on running within a single engine and lacks unified planning and interpretable output across topologies, scenarios, engines, and models.

[0054] (3) General orchestration and scheduling systems (e.g., container orchestration / resource scheduling)

[0055] While general-purpose scheduling systems can handle resource allocation and process initiation, they lack dedicated modeling for the memory peaks, key-value cache risks, parallel communication modes (TP / PP / EP), and topology sensitivity of Large Language Model (LLM) inference. This makes it difficult to directly provide low-risk parallel strategies and placement mappings. In other words, general-purpose orchestration systems excel at resource allocation and process management, but lack dedicated modeling for LLM inference characteristics (e.g., key-value peaks, parallel communication modes, and differences in MoE structure), making it difficult to directly provide low-risk deployment plans for LLM.

[0056] (4) Offline stress testing / automatic trial and error selection

[0057] By conducting multiple rounds of stress testing before deployment, different combinations of parallelism and configurations are screened. This method can identify problems to some extent, but it is time-consuming, costly, and easily affected by environmental fluctuations, making it difficult to form reusable and interpretable rules.

[0058] The aforementioned technical solutions in related technologies typically have the following limitations:

[0059] (i) Lack of a unified input profile: The hardware topology, model structure and business scenarios are not expressed in a structured manner, which makes it difficult to recalculate and audit the planning results, and makes them unstable when migrating to new hardware / new models.

[0060] (ii) Topology without explicit constraints on placement mapping: Even if the parallel parameters are feasible, problems such as communication jitter and tail latency deterioration may occur due to cross-domain / cross-machine TP. The risks are often exposed only after stress testing or going live.

[0061] (iii) Difficulty in identifying memory risks in advance: There is a lack of interpretable decomposition of weights / KV / operational overhead and safety margins, making it difficult to identify "feasible but high-risk" solutions in a timely manner.

[0062] (iv) Lack of directly applicable output: Many solutions only provide suggested values ​​or conclusions, without outputting the mapping from replicas / parallel groups to nodes / GPUs and a list of key parameters. The engineering side still needs a lot of manual translation and verification.

[0063] (v) Lack of unified expression for PD separation and other forms: In long prefix scenarios, the resource allocation, parallel parameters and placement mapping of Prefill / Decode often lack a unified planning standard, making it difficult to provide clear feasibility and risk warnings during the planning stage.

[0064] (vi) Lack of constraints on consistency between quantization form and model product: There is a lack of strong consistency between the deployment parameters and the actual loaded quantization weight product, which can easily lead to the hidden problem of "inconsistency between the planned and the online product".

[0065] To address the aforementioned issues in related technologies, the resource planning method, apparatus, electronic device, storage medium, and program product provided in this disclosure can unify hardware conditions, model structure, and business scenarios into an interpretable static planning framework, thereby forming a feasibility determination and robust ranking mechanism for resource planning schemes. Specifically, by using hardware profiles, model profiles, and scenario profiles as unified inputs, this disclosure can determine the feasibility and robust ranking of resource planning schemes before resource deployment. This avoids the problems of long iteration cycles and high costs associated with relying solely on pre-deployment stress testing or runtime trial and error for resource planning schemes, thus improving the interpretability, recalculation, and portability of resource planning.

[0066] Figure 1 This is a schematic diagram illustrating a resource planning architecture according to exemplary embodiments of the present disclosure.

[0067] Reference Figure 1 This disclosure primarily focuses on hardware-driven static deployment planning. Without conducting prolonged stress tests, it generates a resource planning scheme based on hardware profiles, model profiles, and scenario profiles, and outputs a directly applicable deployment configuration list. It should be noted that the execution entity of the resource planning method provided in this disclosure can be a device node within the service cluster, or a device independent of the service cluster; this disclosure does not impose specific restrictions in this regard.

[0068] The resource planning architecture described above may include, but is not limited to: a Model ProfileBuilder, a Hardware Profile Builder, a Scenario Profile Builder, a memory budget and peak risk prediction unit, a topology constraint and communication risk prediction unit, an engine support matrix and risk library, a Static Planning Solver module, and a Deployment Configuration Generator module. Furthermore, each module / unit can exchange information using structured data, offering advantages such as interpretability, auditability, and reusability.

[0069] The characteristics and uses of each of the above modules / units will be described in detail below.

[0070] The model profiling unit is primarily responsible for standardizing the structural information of the target model. For example, a model profiling may include information such as the number of layers, hidden dimensions, attention head configuration, key-value structure (number of key-value heads and head dimension), whether it uses MoE (Modal of Excellence) and its expert configuration, and the data type / quantization form of weights and key-value pairs. Model profiling can be used not only for subsequent resource estimation but also for engine compatibility assessment and parameter recommendation.

[0071] Additionally, the model profile can include contextual capability parameters for consistency verification with the CTX level in the scene profile. For example, contextual capability parameters can include the maximum supported context length (ctx_model_max), position encoding / long context expansion strategy (e.g., Rope-related configuration), and key constraint information related to the key-value structure. The planner can then determine whether the CTX exceeds the model's supported upper limit based on the contextual capability parameters.

[0072] The hardware profiling unit is primarily responsible for collecting and standardizing cluster hardware information. Hardware profiling can include: a list of nodes and GPUs, the memory capacity per GPU, the interconnect domain division within nodes (e.g., NVLink connectivity domains), and the network level and basic capability tiers between nodes. The core value of hardware profiling lies in making the "communication domain" explicit, thereby providing a basis for parallel grouping and placement mapping.

[0073] In addition, the hardware profile can also include hardware capability information related to inference accuracy, such as, but not limited to: hardware architecture category and available operator capability classification, supported data types and quantization computing capability range (e.g., availability of BF16 / FP16 / FP8 / INT8 / INT4, and specific low-bit data types such as MXFP4 / NVFP4), and can record related runtime environment constraints (e.g., driver / CUDA / backend version, etc.). This information can be used for the generation of quantization type candidates and availability verification.

[0074] The scenario profile building unit is primarily used to receive deployment targets from the business side. Scenario profiles may include, but are not limited to, CTX levels and concurrency limits. For more refined sorting, they can also include typical input / output length levels and preference weights (stability priority, TTFT priority, throughput priority, etc.). Scenario profiles are mainly used to characterize KV peak risks and phased bottleneck tendencies.

[0075] Additionally, when further optimization is needed for long prefix scenarios, the scenario profile can also include deployment goals such as whether to enable Prefill / Decode separation (PD separation), typical input length ranges (prompt_tokens_range), and token experience preferences. This information is mainly used to drive the planner to output either monolithic deployment or PD-separated deployment schemes. It should be noted that if there are inconsistencies between different profiles, for example, if CTX exceeds the model's support limit or the target accuracy exceeds the hardware / backend capabilities, the corresponding candidate scheme can be judged as infeasible or downgraded before generation.

[0076] It should be noted that model profiles and scene profiles can be mapped to resource requirement estimates, which can then be combined with hardware profiles to form constraints. For example, in a multi-tenant or multi-model coexistence cluster, the set of GPUs used by the planner in a single planning operation can be denoted as G_alloc (i.e., the number of GPUs allocated to this model / service), and its size can be less than or equal to the total number of GPUs in the cluster. Furthermore, the integer division relationships of subsequent parallel parameters can be based on G_alloc.

[0077] For example, at the planning level, the parallel parameters can satisfy:

[0078] G_alloc = tp × pp × ep × dp

[0079] Where tp represents tensor parallelism, pp represents pipeline parallelism, ep represents expert parallelism, and dp represents the number of data parallel replicas. In practical scenarios, dp can be derived from the above divisibility relationship after determining tp / pp / ep, or it can be enumerated as a candidate variable and then used together with tp / pp / ep to satisfy the above divisibility constraint. This disclosure can cover both methods.

[0080] For memory budget and peak risk estimation units, in order to make robust judgments before deployment, the planner can break down memory usage into several interpretable parts and determine feasibility and risk levels with safety margins.

[0081] (1) Definition of budget

[0082] Mem_total_gpu: Total physical video memory;

[0083] Budget = Mem_total_gpu × safety_mem_ratio: The maximum amount of video memory allowed by resource planning;

[0084] Reserved = Mem_total_gpu - Budget: Safety reservation (for "non-accurately modelable items" such as fragmentation / jitter / peak uncertainty);

[0085] Mem_est = Mem_params + Mem_kv + Mem_overhead: The "actual video memory that will be used" estimated by resource planning;

[0086] Headroom = Budget - Mem_est: The remaining amount of video memory within the upper limit allowed by resource planning.

[0087] (2) Occupation Decomposition

[0088] Planning-level estimation: Mem_est = Mem_params + Mem_kv + Mem_overhead.

[0089] When Mem_est exceeds Budget, the candidate solution is not feasible under the current hardware and scenario constraints; when Budget - Mem_est is less than the preset threshold, the corresponding solution is feasible, but it is more susceptible to fragmentation and fluctuations, and has a higher risk level. It is suitable to deduct points in the scoring or directly eliminate it under the stability priority strategy.

[0090] It should be noted that the memory budget and peak risk prediction unit can also include weighted memory, KV cache memory, runtime overhead, and reserved alignment.

[0091] For weighted memory, at the planning level, to quickly compare candidate solutions, the parameters can be split into a Dense part and an Expert part, and the quantization form can be mapped to the number of weight bytes b_w. For example, a commonly used planning-level approximation can be: the Dense part is amortized with tp × pp, and the Expert part is amortized with tp × pp × ep in the MoE scenario.

[0092] For KV Cache video memory, the usage and concurrency of KV Cache are highly related to CTX. Therefore, the concurrency limit in scene profile S and CTX must be estimated.

[0093] (1) Effective concurrency upper limit for each copy

[0094] When using dp replicas for load balancing, the planner typically approximates the upper bound of concurrency per replica with B_eff = ceil(concurrency_max / dp). This approximation relies on a relatively even distribution of requests across replicas. Considering potential session stickiness, tenant isolation, or load skew, the planner can introduce a skew coefficient ≥ 1 to amplify the upper bound of concurrency per replica.

[0095] B_eff = ceil(concurrency_max / dp)

[0096] B_peak = min(concurrency_max, ceil(B_eff × skew)).

[0097] The skew setting can be determined by historically observed traffic imbalance, service routing strategies, or conservative configurations. When traffic distribution guarantees are lacking and stability is the primary objective, a more conservative skew setting can be chosen to avoid underestimating the memory risks associated with key-value (KV) peak values.

[0098] (2) KV fragmentation factor

[0099] The planner can use Shard_kv(engine, tp, n_heads_kv) to express the effective sharding factor of KV in the TP dimension, where n_heads_kv comes from the KV structure field of the model profile M. For example, when there is a lack of explicit information, Shard_kv can take a conservative value of 1; when the engine explicitly supports sharding KV by TP / head dimension, Shard_kv can be set according to its actual sharding rules (e.g., Shard_kv≈min(tp, n_heads_kv) or Shard_kv≈tp when n_heads_kv≥tp), thereby avoiding over-conservatism.

[0100] To align runtime overhead with reservations, runtime overhead (Mem_overhead) can be used to characterize predictable fixed or quasi-fixed memory usage, including frame resident memory, communication / operator working areas (buffers), and additional overhead related to the communication domain. Furthermore, difficult-to-model items such as memory fragmentation, allocation jitter, and peak uncertainty are not included in Mem_overhead but are uniformly covered by the reserved safety (i.e., Mem_total_gpu - Budget). The planner can also express Mem_overhead using a conservative model of "constant + scale + domain-dependent terms," ​​and can appropriately increase the domain-dependent terms or conservative coefficients in cross-domain (e.g., cross-machine) scenarios.

[0101] In addition, the inference engine can provide its own reservation / capping mechanism. The safety_mem_ratio / Reserved on the planning side needs to be aligned with the reservation standard on the engine side to avoid unnecessary conservatism caused by duplicate reservations by unifying the budget standard.

[0102] For the topology constraint and communication risk prediction unit, the planner can establish topology constraints based on the intra-node interconnect domain, inter-node network capabilities, and device affinity in the hardware profile H, and can classify the risks of bandwidth, latency, and jitter differences brought by different communication paths. For high-frequency communication parallel groups such as TP, the planner can prioritize constraining them to fall within the same high-speed interconnect domain; when candidate solutions inevitably cross PCIe or cross nodes, the corresponding communication risks can be explicitly recorded in the subsequent placement mapping and scoring stages.

[0103] Based on the engine support matrix and risk database, the planner can verify the compatibility and executability of candidate solutions on the selected inference backend. Verification dimensions can include at least: whether the model family is supported, whether the quantization format is supported, whether the CTX level is supported, and whether the candidate parallel format meets the capability boundaries of the selected inference backend. For candidate solutions that do not meet the above compatibility requirements, the planner can determine them as infeasible and prune them immediately after generation; for candidate solutions that meet basic compatibility requirements but have known stability or performance risks, these can be further reflected in subsequent risk labeling or scoring stages.

[0104] For the static programming solution module, this module can generate candidate solutions under the above constraints and output a Plan through placement mapping, feasibility judgment, and scoring optimization. The Plan can at least include: engine selection suggestions, parallel strategy parameters (tp / pp / ep / dp), topology-aware placement mapping, key operating parameter suggestions, and risk explanation information.

[0105] In addition, when generating candidate solutions, the static programming solution module can accept externally specified quantization type / precision (e.g., BF16 / FP16 / INT4) as input constraints, besides parallel parameters (tp / pp / ep / dp). It can also generate plans under inference backend availability checks and memory budget constraints. The plan can record fields such as quant_mode, weight_dtype, act_dtype, kv_dtype, and model_artifact to ensure consistency between the plan and the actual loaded model output.

[0106] It should be noted that static programming solutions can include candidate solution generation, placement mapping, pruning, and scoring.

[0107] For candidate scheme generation, the main principle can be factor decomposition that closely matches the hardware topology: tp should take the factor of NVLink domain size first; pp starts from 1 by default and is expanded when weight constraints or cross-machine splitting requirements occur; ep is only introduced in the MoE model; dp satisfies integer division relationship.

[0108] For placement mapping, the core principle can be communication domain consistency: parallel groups of high-frequency communication can be preferentially placed in the same high-speed interconnect domain; when the resource scale must be expanded across machines, placement mapping can explicitly reflect cross-domain boundaries and reflect the risks brought by the boundary in the scoring.

[0109] Regarding pruning and scoring, the pruning stage can use hard constraints such as memory capacity, topology rules, and inference backend availability to ensure the feasibility of the output solution; the scoring stage can comprehensively consider memory margin, communication risks, operational complexity, and optional performance heuristics within the feasible set to make the output solution more robust under the target preference.

[0110] It should be noted that after candidate solutions pass hard constraints such as memory and topology, the planner can score feasible candidates to select the best one. The score can consist of several interpretable sub-items, which can be normalized to 0-1 (higher values ​​indicate better performance). Then, a weighted summation can be performed according to preference weights. For example, commonly used scoring items may include:

[0111] ① Memory margin score S_mem: can be determined based on the ratio of Headroom to Budget. The larger the memory margin, the higher the score. When the memory margin is lower than the preset threshold, points can be deducted based on edge risk.

[0112] ② Communication domain score S_topo: can be determined based on the communication domain in which the TP group and PP stage are located in the Placement: the score is highest within the NVLink domain, followed by cross-PCIe communication within the same machine, and the score is lower for cross-machine communication in the stability-first scenario.

[0113] ③ Complexity score S_ops: It is related to dp (number of replicas) and pp (number of stages). The more replicas and stages there are, the higher the complexity of resource coordination and operation and maintenance, and the lower the score will be.

[0114] In summary, the total score can be expressed as: Score = w_mem·S_mem + w_topo·S_topo + w_ops·S_ops, where the weights w_mem, w_topo, and w_ops can be determined by business preferences.

[0115] The deployment configuration generation module can organize the Plan into a directly applicable deployment configuration list. This list can include necessary information such as parallel parameters, placement mapping, port / instance partitioning, as well as key parameter suggestions related to the engine, so that the Plan has the complete information required for direct implementation.

[0116] It should be noted that during the deployment configuration generation phase, the Plan output by the planner can be represented in a structured manner to ensure information integrity, reusability, and auditability. For example, to balance "direct applicability" and "interpretability / recalculability," this disclosure can divide the Plan fields into two categories: minimum required fields and enhanced fields. Minimum required fields can be used to generate a directly applicable deployment configuration list and drive the backend to load the correct model artifacts; enhanced fields can be used to further express information such as communication domain consistency, instance-level mapping, and runtime parameter suggestions, and can be output as needed.

[0117] (a) Minimum Required Field

[0118] 1. Plan identifier: plan_id (used for Top-K candidates and audit traceability).

[0119] 2. Context length CTX, maximum concurrency limit concurrency_max.

[0120] 3. Engine and Availability: engine (recommended engine / backend), backend_compatibility (pass / fail with a summary of reasons, risk level can be attached).

[0121] 4. Parallel parameters: tp, pp, ep, dp, and the corresponding resource pool size G_alloc.

[0122] 5. Deployment Mode: pd_mode (whether PD separation is enabled). When pd_mode is enabled, the Plan can provide the parallel parameters and placement mapping digests for prefill_plan and decode_plan respectively (at least including their respective tp / pp / ep / dp / G_alloc and dp_replicas).

[0123] 6. Quantization and Precision: quant_mode (quantization mode / schema identifier), weight_dtype (weight data type or quantization bit width), act_dtype (activation data type), kv_dtype (KV cache data type), and key parameters related to quantization (e.g., group_size). When quantization mode is not enabled, the above fields can be set to none / default.

[0124] 7. Model Artifact: model_artifact(base_model_id; version; quant_mode; quant_artifact_id (or quantization weight artifact path / identifier); artifact_format; layout_id), used to ensure that the Plan is consistent with the actual loaded weight artifact.

[0125] 8. Placement Mapping (Minimum): dp_replicas (a list of node_id and gpu_id corresponding to each replica).

[0126] 9. Scenario Assumptions / Recalculated Reference Values: skew, B_eff, B_peak, Shard_kv, used to verify the KV peak value and the memory estimation reference value.

[0127] 10. Resource Budget Summary: Budget per GPU (including Mem_total_gpu and safety_mem_ratio), estimated usage of Mem_est (which can at least include weight / KV / cost decomposition), and headroom.

[0128] 11. Risk and Explanation Summary: topology_tags and risk_tags (or equivalent fields).

[0129] (II) Enhanced Fields (Output on Demand)

[0130] 1. Placement mapping enhancements: tp_groups (a list of nodes / GPUs in TP groups, used to reflect communication domain consistency), pp_stages (when pp>1), and instance_mapping (the mapping relationship between parallel instances / replicas / stages / groups and node_id and GPU_id).

[0131] 2. Suggested key operating parameters: engine_params (suggested values ​​and adjustable ranges for engine-related block / pagination, batch processing limits, etc.).

[0132] 3. Scoring and pruning information: reasons for candidate pruning, score decomposition and weight, Top-K candidate list and comparison conclusions, etc.

[0133] In this disclosure, by unifying hardware conditions, model structure, and business scenarios into an interpretable static planning framework, a feasibility determination and robust ranking mechanism can be formed. This allows for the identification of memory and topology risks before deployment, reducing OOM and tail latency jitter; it enables explicit expression of communication domains and placement mappings in multi-machine, multi-GPU environments, ensuring deployment results are consistent with the hardware topology; and it covers both Dense and MoE models.

[0134] Figure 2 This is a flowchart illustrating a resource planning method according to an exemplary embodiment of the present disclosure.

[0135] Reference Figure 2 In step 201, model profiles, hardware profiles, and scenario profiles can be obtained. The model profile is mainly used to characterize the inference attributes of each large language model in the service cluster; the hardware profile is mainly used to characterize the configuration attributes of the service cluster; and the scenario profile is mainly used to characterize the requirement characteristics of the business scenarios that the service cluster will serve.

[0136] For example, as mentioned above, the model profile may include, but is not limited to: number of layers, hidden dimension, attention head configuration, key-value structure (number of key-value heads and head dimension), whether it is MoE and its expert configuration, data type / quantization form of weights and key-values, etc.; the hardware profile may include, but is not limited to: node and GPU list, memory capacity per GPU, interconnect domain division within nodes (e.g., NVLink connectivity domain), network level and basic capability tiers between nodes; the scenario profile may include, but is not limited to: CTX tiers and concurrency limits; when more refined sorting is required, it may also include typical input / output length tiers and preference weights (stability priority, TTFT priority, throughput priority, etc.).

[0137] According to an exemplary embodiment of this disclosure, model structure information of a large language model, hardware information of a service cluster, and business-side deployment target information can be obtained. Then, the model structure information of the large language model, the hardware information of the service cluster, and the business-side deployment target information can be standardized to obtain a model profile, a hardware profile, and a scenario profile.

[0138] In step 202, at least two resource planning candidate schemes can be generated based on the hardware profile. These resource planning candidate schemes can be schemes for allocating GPU resources for each of the various parallel strategies corresponding to the large language model, and the parallel strategies can be the parallel strategies used for distributed inference of the large language model. The GPUs can be GPUs in the device nodes included in the service cluster.

[0139] According to an exemplary embodiment of this disclosure, the hardware profile described above may include the number of usable GPUs, wherein the usable GPUs may be all or some of the GPUs contained in the service cluster. Based on the number of usable GPUs and a preset resource allocation rule, at least one GPU may be allocated to each of a variety of parallel strategies, resulting in at least two resource planning candidate schemes. The preset resource allocation rule may be the product of the number of GPUs allocated to each of the various parallel strategies, which represents the number of usable GPUs.

[0140] For example, the multiple parallel strategies corresponding to the large language model can be four parallel strategies: tp, pp, ep, and dp. And, assuming the number of GPUs available is G_alloc, the above-mentioned preset resource allocation rule can be expressed as:

[0141] G_alloc = tp × pp × ep × dp.

[0142] For example, in the scenario where G_alloc=16, pp=1, and ep=1, the following conditions are met:

[0143] G_alloc = tp × pp × ep × dp = tp × dp

[0144] Therefore, two typical candidate solutions can be obtained:

[0145] Candidate solution A1: tp=8 dp = 16 / 8 = 2

[0146] Candidate solution A2: tp=4 dp=16 / 4=4.

[0147] According to an exemplary embodiment of this disclosure, the hardware profile described above may include the number of usable GPUs, wherein the usable GPUs may be all or some of the GPUs included in the service cluster.

[0148] A first number of GPUs can be allocated for the prefill stage and a second number of GPUs can be allocated for the decoding stage, where the sum of the first and second numbers is the number of GPUs available. The prefill stage and the decoding stage can be two stages included in the inference process of a large language model.

[0149] Then, for each stage in the prefilling and decoding stages, at least one GPU can be allocated for each of the multiple parallel strategies based on the number of GPUs allocated to that stage and a preset resource allocation rule, resulting in at least two resource planning candidate schemes. The preset resource allocation rule can be the product of the number of GPUs allocated to each of the multiple parallel strategies, which represents the number of GPUs allocated in the corresponding stage.

[0150] For example, assuming the total resources under PD separation are still G_alloc=16, the resources can be divided as follows:

[0151] G_alloc_prefill=4: Used for prefill_plan.

[0152] G_alloc_decode=12: Used for decode_plan.

[0153] Furthermore, the divisibility relation must be satisfied separately:

[0154] G_alloc_prefill = tp_p × pp_p × ep_p × dp_p

[0155] G_alloc_decode = tp_d × pp_d × ep_d × dp_d

[0156] In this disclosure, the model can be a Dense model (ep=1), and to reduce operational complexity, pp=1 can be chosen. In this case, a set of feasible candidate solutions can be selected:

[0157] prefill_plan:tp_p=4, pp_p=1, ep_p=1, dp_p=1 → G_alloc_prefill=4

[0158] decode_plan: tp_d=4, pp_d=1, ep_d=1, dp_d=3 → G_alloc_decode=12.

[0159] In step 203, based on the aforementioned model profile, hardware profile, and scenario profile, the remaining video memory for each of the at least two resource planning candidate schemes can be calculated. This remaining video memory can be the surplus within the upper limit of video memory allowed by resource planning, and the video memory can be the video memory of the GPU.

[0160] According to an exemplary embodiment of this disclosure, for each resource planning candidate scheme, the total video memory usage (Mem_est) can be calculated based on the model profile, hardware profile, and scenario profile. This total video memory usage can be used to indicate the estimated total video memory usage of a single GPU under that resource planning candidate scheme. Furthermore, the model profile can be used to characterize the video memory required by the large language model itself; the scenario profile can be used to characterize the KV video memory required for business concurrency; and the hardware profile can be used to characterize the runtime video memory overhead of the framework resident, communication buffer, and operator working area. Then, the remaining video memory for that resource planning candidate scheme can be calculated based on the hardware profile and the total video memory usage (Mem_est).

[0161] According to an exemplary embodiment of this disclosure, the above model profile may include the following items: attention head configuration d_head, i.e., the dimension of the head in the large language model; KV structure n_heads_kv, i.e., the number of KV heads in the large language model; number of bytes b_kv, i.e., how many bytes each numerical element occupies in the KV Cache in the large language model; number of layers n_layers, i.e., the number of layers contained in the large language model; and total weights after quantization Mem_params_total.

[0162] The above hardware profile may include the following items: runtime memory overhead (Mem_overhead), which is the runtime memory overhead of framework resident, communication buffer, operator working area, etc., and the upper limit of memory allowed by resource planning (Budget).

[0163] The scenario profile described above can include the following items: concurrency limit (concurrency_max), maximum concurrency, context length (CTX), effective sharding factor (Shard_kv) for key-value pairs in the tensor parallel dimension (where Shard_kv = 1 means each card carries a complete replica of key-value pairs; if Shard_kv = 4, it means the request key-value pairs are distributed across four cards), and skew coefficient (skew primarily indicates the evenness of concurrent requests among replicas). For example, skew = 1 represents an ideal balanced distribution. However, in real-world systems, due to factors such as historical request distribution, business routing strategies, session stickiness, or hotspot loads, it is often difficult to guarantee a completely balanced distribution of requests among replicas. In such cases, skew > 1 can be used to conservatively amplify the average concurrency, thereby more reasonably estimating peak concurrency.

[0164] According to an exemplary embodiment of this disclosure, for each resource planning candidate scheme, the effective concurrency upper bound B_peak per replica can be calculated based on the aforementioned concurrency upper bound concurrency_max and skew coefficient skew. The effective concurrency upper bound B_peak per replica is mainly used to represent the upper bound of the number of requests that each replica may actually carry at peak times.

[0165] Then, based on the aforementioned attention head configuration d_head, KV structure n_heads_kv, number of bytes b_kv, number of layers n_layers, and context length CTX, the KV per req occupancy for a single request can be calculated, i.e., how much KV cache is needed for a single request.

[0166] Next, based on the effective concurrency upper bound B_peak per replica, the KV occupancy per request KV_per_req, and the effective sharding factor Shard_kv of the aforementioned KV in the tensor parallel dimension, the peak KV occupancy Mem_kv per GPU can be calculated. This peak KV occupancy Mem_kv per GPU is mainly used to represent the KV Cache memory occupancy of a single GPU under peak concurrency.

[0167] Then, based on the total quantized weights Mem_params_total, the weight usage per GPU Mem_params can be calculated. This weight usage per GPU Mem_params is mainly used to represent the memory usage of the model weights distributed to a single GPU under the current parallel scheme.

[0168] Next, the total video memory usage Mem_est can be calculated based on the weight usage Mem_params per GPU, the peak KV usage Mem_kv per GPU, and the runtime overhead Mem_overhead.

[0169] According to an exemplary embodiment of this disclosure, the difference between the upper limit of video memory Budget allowed by the aforementioned resource planning and the total video memory usage Mem_est can be calculated as the video memory margin of the candidate resource planning scheme.

[0170] Thus, in this disclosure, by uniformly modeling planning-level resource metrics such as weight occupancy (Mem_params) per GPU, peak KV occupancy (Mem_kv) per GPU, runtime overhead (Mem_overhead), the maximum allowed memory capacity (Budget) for resource planning, and memory headroom, the feasibility of candidate solutions can be determined before deployment. Furthermore, by explicitly incorporating topological factors such as intra-node interconnect domains and inter-node link capabilities into constraints, high-frequency communication parallel groups can be prioritized to fall within high-speed interconnect domains, thereby reducing tail latency jitter and stability risks caused by cross-domain communication. In addition, by combining inference backend compatibility constraints, memory constraints, and topological constraints, candidate solutions can be pruned, scored, and optimized, rather than relying solely on pre-deployment stress testing or runtime trial and error to expose problems, thereby improving the interpretability, recalculability, and portability of resource planning.

[0171] According to an exemplary embodiment of this disclosure, the effective concurrency upper bound B_peak for each replica can be positively correlated with both the concurrency upper bound concurrency_max and the skew coefficient skew.

[0172] For example, the basic concurrency upper bound B_eff per replica can be calculated using the following formula:

[0173] B_eff = ceil(concurrency_max / dp);

[0174] Where dp is the number of parallel data replicas, i.e., the number of instances, and ceil(.) is the round-up function.

[0175] Then, the effective concurrency upper bound B_peak for each replica can be calculated using the following formula:

[0176] B_peak = min(concurrency_max, ceil(B_eff × skew))

[0177] According to an exemplary embodiment of this disclosure, the KV_per_req occupied by the single request KV can be positively correlated with the attention head configuration d_head, the KV structure n_heads_kv, the number of bytes b_kv, the number of layers n_layers, and the context length CTX.

[0178] For example, the KV_per_req usage per request KV can be calculated using the following formula:

[0179] KV_per_req = CTX × n_layers × 2 × d_head × n_heads_kv × b_kv.

[0180] According to an exemplary embodiment of this disclosure, the peak KV occupancy Mem_kv per GPU can be positively correlated with the effective concurrency upper bound B_peak per replica and the KV occupancy KV per request KV_per_req, and the peak KV occupancy Mem_kv per GPU can be negatively correlated with the effective sharding factor Shard_kv of KV in the tensor parallel dimension.

[0181] For example, the peak KV usage per GPU (Mem_kv) can be calculated using the following formula:

[0182] Mem_kv = B_peak × KV_per_req / Shard_kv.

[0183] According to an exemplary embodiment of this disclosure, the aforementioned weight occupancy Mem_params per GPU can be positively correlated with the total quantized weight occupancy Mem_params_total.

[0184] For example, the weight usage (Mem_params) per GPU can be calculated using the following formula:

[0185] Mem_params = Mem_params_total / (tp×pp)

[0186] Where tp stands for tensor parallelism and pp stands for pipelined parallelism.

[0187] According to an exemplary embodiment of this disclosure, the total video memory usage Mem_est can be positively correlated with the weight usage Mem_params per GPU, the peak KV usage Mem_kv per GPU, and the runtime overhead Mem_overhead.

[0188] For example, the total video memory usage Mem_est can be calculated using the following formula:

[0189] Mem_est = Mem_params + Mem_kv + Mem_overhead.

[0190] In step 204, a target resource planning scheme can be selected from at least two resource planning candidate schemes based on the calculated available memory for each resource planning candidate scheme.

[0191] According to an exemplary embodiment of this disclosure, the resource planning candidate with the largest available video memory among at least two resource planning candidate schemes can be used as the target resource planning scheme.

[0192] According to exemplary embodiments of this disclosure, the above-mentioned parallel strategies may include tensor parallel strategies, pipeline parallel strategies, expert parallel strategies, and data parallel strategies.

[0193] For each resource planning candidate scheme, a memory reserve score S_mem can be calculated based on the memory reserve of the candidate scheme. The larger the memory reserve, the higher the corresponding memory reserve score S_mem can be.

[0194] For example, as mentioned above, the memory availability score S_mem can be calculated in the following way:

[0195] S_mem: Obtained from the Headroom mapping. For example, if H_good = 12 GiB and H_bad = 4 GiB, then:

[0196] S_mem = clamp((Headroom - H_bad) / (H_good - H_bad), 0, 1)

[0197] Here, `clamp(x, 0, 1)` means truncating `x` to the interval [0, 1]. `H_bad` and `H_good` are preset two-level margin thresholds. If `Headroom ≤ H_bad`, it can be considered as high risk of touching the edge, and the score is 0; if `Headroom ≥ H_good`, it can be considered as sufficient margin, and the score is saturated to 1; if `Headroom` is between the two, linear interpolation can be used to maintain monotonicity and interpretability.

[0198] Then, the operational complexity score S_ops can be calculated based on the number of GPUs allocated to each of the pipeline parallel strategy, expert parallel strategy, and data parallel strategy. The more GPUs allocated to each parallel strategy, the lower the operational complexity score S_ops can be.

[0199] For example, as mentioned above, the operational complexity score S_ops can be calculated in the following way:

[0200] S_ops = 1 - 0.10×(dp-1) - 0.05×(pp-1) - 0.05×(ep-1) (lower limit truncated to 0)

[0201] When dp=pp=ep=1 (the simplest parallel form), (·-1)=0, so S_ops=1; when the parallel scale increases, the increase in the number of replicas / stages / expert parallelism will bring more instance objects, more cross-domain communication and more complex fault domains and troubleshooting links. At this time, the incremental items can be deducted linearly.

[0202] Next, the topology risk score S_topo can be calculated based on the data communication type of the GPU to which the tensor parallel strategy TP is assigned. The topology risk score S_topo for intra-machine communication can be higher than that for cross-machine communication. For example, as mentioned earlier, the topology risk score S_topo can be 1 when the TP group is entirely located in a single-machine high-speed interconnect domain; the topology risk score S_topo can be downgraded when cross-machine high-frequency communication exists.

[0203] Then, the total score can be calculated based on the aforementioned memory availability score S_mem, operational complexity score S_ops, and topology risk score S_topo. For example, as mentioned earlier, under a "stability-first" preference, this disclosure can use a set of implementable score item weights: w_mem=0.40, w_topo=0.40, w_ops=0.20. The total score can be obtained by weighted summation of the above three scores based on these score item weights.

[0204] Next, based on the total score, a target resource planning scheme can be selected from at least two resource planning candidate schemes. The higher the total score, the greater the likelihood that the corresponding resource planning candidate scheme will be selected as the target resource planning scheme.

[0205] According to exemplary embodiments of this disclosure, the aforementioned target resource planning scheme can also be structured to obtain a structured resource planning scheme. Then, resource deployment can be performed based on this structured resource planning scheme. In this way, by representing the resource planning scheme in a structured manner, the integrity, reusability, and auditability of information can be guaranteed.

[0206] In this disclosure, by unifying the modeling of planning-level resource metrics such as Mem_params, Mem_kv, Mem_overhead, Budget, and Headroom, the feasibility of candidate solutions can be determined before deployment. Furthermore, by explicitly incorporating topological factors such as intra-node interconnect domains and inter-node link capabilities into constraints, high-frequency communication parallel groups are prioritized to fall within high-speed interconnect domains, thereby reducing tail latency jitter and stability risks caused by cross-domain communication. In addition, by combining inference backend compatibility constraints, memory constraints, and topological constraints, candidate solutions are pruned, scored, and optimized, rather than relying solely on pre-deployment stress testing or runtime trial and error to expose problems, thus improving the interpretability, recalculability, and portability of resource planning.

[0207] The following two examples will illustrate the resource planning process for multi-machine, multi-card systems.

[0208] First example: 2 nodes × 8 cards (16 cards in total), CTX=64K, concurrency 48; Plan generation and optimization of TP + DP on the same machine.

[0209] (I) Input Profile and Criterion Assumptions

[0210] Hardware profile H (example specification): Number of nodes: 2; Number of GPUs per node: 8; Planned GPU set G_alloc=16; Single card memory: Mem_total_gpu=80 GiB.

[0211] Intra-node interconnection: 8 cards are in the same high-speed interconnect domain (which can be understood as an NVLink domain); cross-node interconnection is a network link.

[0212] Safety budget ratio: safety_mem_ratio=0.90, therefore, single card budget = 80 × 0.90= 72 GiB.

[0213] Runtime overhead estimate: Mem_overhead = 6 GiB (including planning-level merging items such as framework resident, communication buffer, etc.).

[0214] Model portrait M (example caliber, not MoE):

[0215] Number of layers n_layers=40; KV structure: single KV header format (can be understood as MQA); n_heads_kv=1; d_head=128; KV data type: FP16, number of bytes b_kv=2.

[0216] Total amount of quantized (or equivalent) weights: Mem_params_total = 160 GiB (estimated at the planning level, including main weights and necessary metadata; constant terms that are少量不可分片常量项 are not distinguished).

[0217] Explanation of quantization caliber: In this embodiment, Mem_params_total represents "the weight occupancy of a single copy without sharding caliber (including necessary quantization metadata) under the selected quant_mode". The planner can approximately estimate Mem_params_total based on the model parameter quantity and weight_dtype, and calculate Mem_params = Mem_params_total / (tp×pp) accordingly. The selected inference backend can load the corresponding quantized weight product according to the quant_artifact_id in the model_artifact, so as to ensure that the Plan is consistent with the actually loaded model.

[0218] Scenario portrait S (example caliber):

[0219] Context length: CTX = 64K = 65536 tokens; Concurrency upper limit: concurrency_max = 48; Shunt skew coefficient: skew = 1.10 (when it is impossible to ensure completely uniform shunting, this coefficient is used to conservatively amplify the upper bound of single-copy concurrency).

[0220] KV sharding factor caliber: To ensure no underestimation, Shard_kv(engine, tp, n_heads_kv) = 1 can be taken (that is, estimate "each card needs to carry all KV of this copy" according to the most conservative caliber).

[0221] In addition, when it is known that the actual engine supports sharding KV by TP, the value can be taken according to the backend rules. Commonly, Shard_kv≈tp can be approximately taken; in scenarios such as GQA / MQA where n_heads_kv < tp, Shard_kv≈min(tp, n_heads_kv) can be taken (if the backend uses other splitting rules, the actual rules can be used as the standard), and Mem_kv / Shard_kv can be used for correction accordingly.

[0222] (2) Generation of candidate solutions and verification of inference backend availability

[0223] In this disclosure, the model can be a Dense model (not MoE), therefore, ep=1 can be chosen. In addition, considering the priority of stability and the desire to limit high-frequency communication to a single-machine high-speed interconnect domain as much as possible, in this disclosure, tp can be preferentially limited to a factor of the size of the high-speed interconnect domain within the node (8 cards), and tp=8 and tp=4 can be selected as representative candidates for comparison.

[0224] It should be noted that since pp introduces stage partitioning and scheduling complexity, and since the memory budget in this disclosure does not require weight amortization through pipeline parallelism, pp=1 can be chosen. Based on this, as mentioned earlier, the parallel parameters satisfy G_alloc = tp × pp × ep × dp, where dp can be derived from integer division relations.

[0225] For example, in a scenario where G_alloc=16, pp=1, ep=1 (as determined above), the following conditions are met:

[0226] G_alloc = tp × pp × ep × dp = tp × dp

[0227] Therefore, two typical candidate solutions can be obtained:

[0228] Candidate solution A1: tp=8 dp = 16 / 8 = 2

[0229] Candidate solution A2: tp=4 dp = 16 / 4 = 4

[0230] Before proceeding with memory estimation and scoring, the planner can first verify the availability of the selected inference backend. Verification dimensions can include at least: model family (structure and operator type), quantization type (e.g., FP16 / INT8), CTX level (e.g., 64K), and parallelism type (e.g., whether tp and dp support multiple machines and multiple GPUs). In this disclosure, if the inference backend meets the above conditions, candidate scheme A1 and candidate scheme A2 can proceed to the subsequent memory feasibility assessment and scoring selection process.

[0231] It should be noted that this disclosure assumes that the quantization product has been generated by the offline process and pre-specified by the operations / business side (e.g., quant_mode=W4A16, corresponding to a specific product quant_artifact_id). After the planner passes the availability verification on the inference backend, it can write the quantization form and the corresponding product identifier into the quant_' field and model_artifact field of the Plan, thereby ensuring that the deployment Plan is consistent with the actual loaded model.

[0232] (III) Determination of the feasibility of video memory.

[0233] ① The effective concurrency upper limit for each copy

[0234] First, we can obtain the basic upper bound of concurrency per replica by using dynamic programming (DP):

[0235] Candidate solution A1: B_eff = ceil(48 / 2) = 24

[0236] Candidate solution A2: B_eff = ceil(48 / 4) = 12

[0237] As mentioned earlier, since the concurrency limit in this disclosure is concurrency_max=48, a conservative amplification of the flow skew can be introduced, and combined with the total concurrency limit constraint, the planned peak concurrency can be obtained:

[0238] Candidate solution A1: B_peak = min(concurrency_max, ceil(B_eff × skew)) = min(48, ceil(24×1.10)) = 27;

[0239] Candidate solution A2: B_peak = min(concurrency_max, ceil(B_eff × skew)) = min(48, ceil(12×1.10)) = 14.

[0240] ② Single request KV occupancy (can be recalculated at the planning level)

[0241] The number of KV bytes per token and per layer is: 2(K+V) × d_head × n_heads_kv × b_kv. Substituting into the example, we can get: 2×128×1×2 = 512 bytes.

[0242] The total number of KV bytes for a single request when CTX=65536 is:

[0243] KV_per_req = CTX × n_layers × 512 byte= 65536 × 40 × 512 = 1,342,177,280 byte ≈ 1.25 GiB

[0244] ③ Peak KV usage per GPU (planning-level estimation)

[0245] Mem_kv = B_peak × KV_per_req / Shard_kv (for example, Shard_kv=1)

[0246] Candidate solution A1: Mem_kv ≈ 27 × 1.25 = 33.75 GiB

[0247] Candidate solution A2: Mem_kv ≈ 14 × 1.25 = 17.50 GiB

[0248] ④ Weight allocation per GPU (planning-level estimation)

[0249] The Dense model can be approximated at the planning level as: Mem_params = Mem_params_total / (tp×pp) (where pp=1).

[0250] Candidate solution A1: Mem_params = 160 / 8 = 20 GiB

[0251] Candidate solution A2: Mem_params = 160 / 4 = 40 GiB

[0252] ⑤ Total usage (Mem_est) and remaining space (Headroom)

[0253] Mem_est = Mem_params + Mem_kv + Mem_overhead

[0254] Candidate solution A1: Mem_est ≈ 20 + 33.75 + 6 = 59.75 GiB

[0255] Headroom = Budget - Mem_est = 72 - 59.75 = 12.25 GiB

[0256] Candidate solution A2: Mem_est ≈ 40 + 17.50 + 6 = 63.50 GiB

[0257] Headroom = 72 - 63.50 = 8.50 GiB

[0258] Since both candidate scheme A1 and candidate scheme A2 have a memory margin greater than zero, both candidate scheme A1 and candidate scheme A2 are feasible. Furthermore, since candidate scheme A1 has a more ample memory margin, it has a lower risk of failing to reach the edge of the memory.

[0259] (iv) Plan output example

[0260] To demonstrate the consistent, permissible output of the placement mapping / communication domain, the following are examples of the minimum fields for two candidate plans:

[0261] Plan-A1 (tp = 8, pp = 1, ep = 1, dp = 2):

[0262] - plan_id: A1

[0263] - engine: Sglang

[0264] - backend_compatibility: pass (model_family / quant / CTX / parallel pass)

[0265] - pd_mode: off

[0266] - quant_mode: W4A16

[0267] - weight_dtype: INT4 (group_size = 128)

[0268] - act_dtype: BF16

[0269] - kv_dtype: FP16

[0270] - model_artifact: base_model_id =...; version = tag; quant_mode = W4A16;

[0271] quant_artifact_id = model_xxx / W4A16 / g128; artifact_format = hf_safetensors; layout_id = auto_detect

[0272] - parallel: tp = 8, pp = 1, ep = 1, dp = 2, G_alloc = 16

[0273] - placement (replica → node → GPU):

[0274] - replica_0: node_0 → gpu[0,1,2,3,4,5,6,7]

[0275] - replica_1: node_1 → gpu[0,1,2,3,4,5,6,7]

[0276] - scenario_assumption: CTX=65536; concurrency_max=48; skew=1.10; B_eff=24; B_peak=27; Shard_kv=1

[0277] - memory_budget: Mem_total_gpu=80GiB; safety_mem_ratio=0.90; Budget=72GiB, Reserved=8GiB

[0278] - memory_breakdown: Mem_params=20GiB; Mem_kv=33.75GiB; Mem_overhead=6GiB; Mem_est=59.75GiB; Headroom=12.25GiB

[0279] - topology_tags: TP groups fall within a single-machine high-speed interconnect domain; cross-node usage is only for DP replica distribution.

[0280] - risk_tags: Sufficient video memory; moderate number of replicas; stability priority.

[0281] Plan-A2 (tp=4, pp=1, ep=1, dp=4):

[0282] - plan_id: A2

[0283] - engine: Sglang

[0284] - backend_compatibility: pass (model_family / quant / CTX / parallel passed)

[0285] - pd_mode: off

[0286] - quant_mode: W4A16

[0287] - weight_dtype: INT4 (group_size=128)

[0288] - act_dtype: BF16

[0289] - kv_dtype: FP16

[0290] - model_artifact: base_model_id=...; version=tag; quant_mode=W4A16;

[0291] quant_artifact_id=model_xxx / W4A16 / g128;artifact_format=hf_safetensors;layout_id=auto_detect

[0292] - parallel: tp=4, pp=1, ep=1, dp=4, G_alloc=16

[0293] - Placement (example: "2 replicas per node, 4 cards per replica"):

[0294] - replica_0: node_0 → gpu[0,1,2,3]

[0295] - replica_1: node_0 → gpu[4,5,6,7]

[0296] - replica_2: node_1 → gpu[0,1,2,3]

[0297] - replica_3: node_1 → gpu[4,5,6,7]

[0298] - scenario_assumption: CTX=65536; concurrency_max=48; skew=1.10; B_eff=12; B_peak=14; Shard_kv=1

[0299] - memory_budget: Mem_total_gpu=80GiB; safety_mem_ratio=0.90; Budget=72GiB; Reserved=8GiB

[0300] - memory_breakdown: Mem_params=40GiB; Mem_kv=17.50GiB; Mem_overhead=6GiB; Mem_est=63.50GiB; Headroom=8.50GiB

[0301] - topology_tags: TP group falls within a single-machine high-speed interconnect domain; replicas are more distributed.

[0302] - risk_tags: High number of replicas; low VRAM availability; caution required when prioritizing stability.

[0303] Thus, this disclosure can output a deployment configuration list that can be directly implemented: the planner can output information including parallel parameters, mapping of dp replicas to nodes / gpus, key engine parameter suggestions, risk tags, etc., so that the planning results can be directly used for deployment and operation and maintenance.

[0304] (V) Scoring Calculation Example

[0305] Under the preference of "stability first", this disclosure can adopt a set of feasible scoring item weights: w_mem=0.40, w_topo=0.40, w_ops=0.20, and the sub-scoring can be as follows:

[0306] S_mem: Obtained from the Headroom mapping. For example, if H_good = 12 GiB and H_bad = 4 GiB, then:

[0307] S_mem = clamp((Headroom - H_bad) / (H_good - H_bad), 0, 1)

[0308] Here, `clamp(x, 0, 1)` means truncating `x` to the interval [0, 1]. `H_bad` and `H_good` are preset two-level margin thresholds. If `Headroom ≤ H_bad`, it can be considered as high risk of touching the edge, and the score is 0; if `Headroom ≥ H_good`, it can be considered as sufficient margin, and the score is saturated to 1; if `Headroom` is between the two, linear interpolation can be used to maintain monotonicity and interpretability.

[0309] S_topo: Topology risk score. For example, the value can be 1 when the TP group is entirely located in a single-machine high-speed interconnect domain; it can be downgraded when there is cross-machine high-frequency communication (in this disclosure, the S_topo of candidate scheme A1 and candidate scheme A2 can both be 1).

[0310] S_ops: Operational complexity score. For example, S_ops can be represented as follows:

[0311] S_ops = 1 - 0.10×(dp-1) - 0.05×(pp-1) - 0.05×(ep-1) (lower limit truncated to 0)

[0312] When dp=pp=ep=1 (the simplest parallel form), (·-1)=0, so S_ops=1; when the parallel scale increases, the increase in the number of replicas / stages / expert parallelism will bring more instance objects, more cross-domain communication and more complex fault domains and troubleshooting links. At this time, the incremental items can be deducted linearly.

[0313] For candidate solution A1:

[0314] S_mem = clamp((12.25-4) / 8,0,1)=1.00

[0315] S_topo=1.00

[0316] S_ops = 1 - 0.10 × (2 - 1) = 0.9

[0317] Total score Score_A1 = 0.40×1 + 0.40×1 + 0.20×0.90 = 0.98.

[0318] For candidate solution A2:

[0319] S_mem = (8.50-4) / 8 = 0.56

[0320] S_topo=1.00

[0321] S_ops = 1 - 0.10 × (4 - 1) = 0.70

[0322] The total score is Score_A2 = 0.40×0.56 + 0.40×1 + 0.20×0.70≈ 0.764.

[0323] Since candidate solution A1 scores higher than candidate solution A2, the planner can output candidate solution A1 as the default plan and can keep candidate solution A2 as an alternative plan (Top-K).

[0324] Second example: 2 nodes × 8 cards (16 cards in total), CTX=64K, total concurrency 48; Generation and optimization of PD separation plan in long prefix scenario.

[0325] (a) Input profile and caliber assumptions

[0326] The hardware profile H and the model profile M can be the same as in the first example mentioned above:

[0327] Mem_total_gpu=80GiB, safety_mem_ratio=0.90, Budget=72GiB, Reserved=8GiB; Mem_overhead=6GiB.

[0328] n_layers=40, n_heads_kv=1, d_head=128, b_kv=2 (FP16); Mem_params_total=160GiB (selected quant_mode is the weight of the single copy without sharding, including necessary quantization metadata).

[0329] Shard_kv(engine, tp, n_heads_kv)=1 (e.g., n_heads_kv=1, conservative caliber).

[0330] Scene portrait S:

[0331] CTX=65536, concurrency_max=48, skew=1.10;

[0332] Long prefix level: prompt_tokens_p95=8192;

[0333] Phase concurrency upper bound (used for static programming recalculation): prefill_concurrency_max=12, decode_concurrency_max=36, and it is required that prefill_concurrency_max + decode_concurrency_max = concurrency_max).

[0334] (II) Candidate Generation and Resource Allocation (PD Separation)

[0335] Even with PD separation, the total resources remain G_alloc=16. For example, the resources can be divided as follows:

[0336] G_alloc_prefill=4: Used for prefill_plan.

[0337] G_alloc_decode=12: Used for decode_plan.

[0338] Furthermore, the divisibility relation must be satisfied separately:

[0339] G_alloc_prefill = tp_p × pp_p × ep_p × dp_p

[0340] G_alloc_decode = tp_d × pp_d × ep_d × dp_d

[0341] In this disclosure, the model is a Dense model (ep=1), and to reduce operational complexity, pp=1 can be chosen. In this case, a set of feasible candidate solutions can be selected:

[0342] prefill_plan:tp_p=4, pp_p=1, ep_p=1, dp_p=1 → G_alloc_prefill=4

[0343] decode_plan:tp_d=4, pp_d=1, ep_d=1, dp_d=3 → G_alloc_decode=12

[0344] (III) Determining the feasibility of video memory (recalculating prefill / decode separately)

[0345] B_eff_prefill: The effective concurrency upper bound for each (prefill) parallel data replica under "average / balanced distribution"; B_peak_prefill: The peak concurrency upper bound for a single replica under the planning caliber during the prefill phase (used to estimate the peak KV memory of the replica); Number of KV bytes per token and per layer: 2(K+V) × d_head × n_heads_kv × b_kv = 2 × 128 × 1 × 2 = 512 bytes.

[0346] ① Prefill side KV (estimated based on prompt_tokens_p95)

[0347] Single request KV:

[0348] KV_per_req_prefill = prompt_tokens_p95 × n_layers × 512 byte

[0349] = 8192 × 40 × 512 = 167,772,160 byte ≈ 0.15625 GiB

[0350] Concurrency (dp=1):

[0351] B_eff_prefill = ceil(prefill_concurrency_max / dp_p)=ceil(12 / 1)=12;

[0352] B_peak_prefill = min(prefill_concurrency_max, ceil(B_eff_prefill ×skew)) = min(12, ceil(12×1.10))=12.

[0353] Peak KV per GPU:

[0354] Mem_kv_prefill ≈ B_peak_prefill × KV_per_req_prefill / Shard_kv =12 × 0.15625 / 1 = 1.875 GiB;

[0355] Weight: Mem_params_prefill = 160 / (tp_p×pp_p)=160 / 4=40 GiB;

[0356] Total usage: Mem_est_prefill = 40 + 1.875 + 6 = 47.875 GiB;

[0357] Memory allowance: Headroom_prefill = 72 - 47.875 = 24.125 GiB (feasible).

[0358] ② Decode-side key-value pairs (estimated based on CTX)

[0359] Single request KV:

[0360] KV_per_req_decode = CTX × n_layers × 512 byte = 65536×40×512 ≈1.25 GiB

[0361] Concurrency (dp=3):

[0362] B_eff_decode = ceil(decode_concurrency_max / dp_d)=ceil(36 / 3)=12

[0363] B_peak_decode = min(decode_concurrency_max, ceil(B_eff_decode ×skew)) = min(36, ceil(12×1.10))=14

[0364] Peak KV per GPU:

[0365] Mem_kv_decode ≈ 14 × 1.25 / 1 = 17.50 GiB

[0366] Weight: Mem_params_decode = 160 / (tp_d×pp_d)=160 / 4=40 GiB

[0367] Total usage: Mem_est_decode = 40 + 17.50 + 6 = 63.50 GiB

[0368] Margin: Headroom_decode = 72 - 63.50 = 8.50 GiB (The solution is feasible, but the margin is too small).

[0369] (iv) Plan output example (pd_mode=on)

[0370] Plan-PD1:

[0371] - plan_id: PD1

[0372] - engine: Sglang

[0373] - backend_compatibility: pass (model_family / quant / CTX / parallel passed)

[0374] - pd_mode: on

[0375] - quant_mode: W4A16

[0376] - weight_dtype: INT4 (group_size=128)

[0377] - act_dtype: BF16

[0378] - kv_dtype: FP16

[0379] - model_artifact: base_model_id=...; version=tag; quant_mode=W4A16;

[0380] quant_artifact_id=model_xxx / W4A16 / g128;artifact_format=hf_safetensors;layout_id=auto_detect

[0381] prefill_plan:

[0382] - parallel: tp=4, pp=1, ep=1, dp=1, G_alloc=4

[0383] - placement(dp_replicas):

[0384] - replica_0:node_0 → gpu[4,5,6,7]

[0385] - scenario_assumption:prompt_tokens_p95=8192;prefill_concurrency_max=12;B_eff=12;B_peak=12;Shard_kv=1

[0386] - memory_budget:Mem_total_gpu=80GiB;safety_mem_ratio=0.90;Budget=72GiB;Reserved=8GiB

[0387] - memory_breakdown:Mem_params=40GiB;Mem_kv=1.875GiB;Mem_overhead=6GiB;Mem_est=47.875GiB;Headroom=24.125GiB

[0388] decode_plan:

[0389] - parallel: tp=4, pp=1, ep=1, dp=3, G_alloc=12

[0390] - placement(dp_replicas):

[0391] - replica_0:node_0 → gpu[0,1,2,3]

[0392] - replica_1:node_1 → gpu[0,1,2,3]

[0393] - replica_2:node_1 → gpu[4,5,6,7]

[0394] - scenario_assumption: CTX=65536; decode_concurrency_max=36; skew=1.10; B_eff=12; B_peak=14; Shard_kv=1

[0395] - memory_budget: Mem_total_gpu=80GiB; safety_mem_ratio=0.90; Budget=72GiB; Reserved=8GiB

[0396] - memory_breakdown: Mem_params=40GiB; Mem_kv=17.50GiB; Mem_overhead=6GiB; Mem_est=63.50GiB; Headroom=8.50GiB

[0397] - topology_tags: prefill copies are on the same node as some decode copies; other decode copies are across nodes.

[0398] - risk_tags: The margin on the decode side is small; there is KV handoff in PD, and the link jitter and bandwidth need to be considered when handing over across nodes.

[0399] It should be noted that this disclosure does not focus on adjusting request routing, queue priority, or resource pool allocation online based on real-time load after service operation. Instead, it focuses on the pre-deployment stage of large-scale model inference, using hardware profile H, model profile M, and scenario profile S as unified inputs to perform multi-constraint static planning on candidate deployment schemes. This static planning process can uniformly incorporate factors such as GPU memory budget, KV cache risk, topology constraints, and inference backend compatibility into the modeling, and can generate candidate parallel parameters, topology-aware placement mappings, risk explanation information, and a list of deployment configurations that can be directly implemented before long-term stress testing. Furthermore, in PD separation scenarios, this disclosure can also output sub-plans for prefill_plan and decode_plan separately. That is, in long-prefix scenarios, it can output resource allocation, parallel parameters, and placement mapping summaries for prefill_plan and decode_plan respectively, and can provide risk warnings such as KV handover, thereby enhancing the feasibility and controllability in long-prefix scenarios.

[0400] Figure 3 This is a block diagram illustrating a resource planning apparatus 300 according to an exemplary embodiment of the present disclosure.

[0401] Reference Figure 3The resource planning device 300 may include an image acquisition module 301, a scheme generation module 302, a video memory margin calculation module 303, and a scheme selection module 304.

[0402] The profile acquisition module 301 can acquire model profiles, hardware profiles, and scenario profiles. Model profiles are mainly used to characterize the inference attributes of each large language model in the service cluster; hardware profiles are mainly used to characterize the configuration attributes of the service cluster; and scenario profiles are mainly used to characterize the requirement characteristics of the business scenarios that the service cluster will serve.

[0403] According to an exemplary embodiment of this disclosure, the profile acquisition module 301 can acquire model structure information of a large language model, hardware information of a service cluster, and business-side deployment target information. Then, the profile acquisition module 301 can perform standardization processing on the model structure information of the large language model, the hardware information of the service cluster, and the business-side deployment target information to obtain a model profile, a hardware profile, and a scenario profile.

[0404] The scheme generation module 302 can generate at least two resource planning candidate schemes based on the hardware profile. These resource planning candidate schemes can be schemes for allocating GPU resources for each of the various parallel strategies corresponding to a large language model, where the parallel strategy can be the parallel strategy used for distributed inference of the large language model, and the GPU can be the GPU in the device nodes included in the service cluster.

[0405] According to an exemplary embodiment of this disclosure, the hardware profile described above may include the number of usable GPUs, wherein the usable GPUs may be all or some of the GPUs included in the service cluster. The scheme generation module 302 may allocate at least one GPU for each of a variety of parallel strategies based on the number of usable GPUs and a preset resource allocation rule, thereby obtaining at least two resource planning candidate schemes. The preset resource allocation rule may be that the number of usable GPUs is the product of the number of GPUs allocated to each of the various parallel strategies.

[0406] According to an exemplary embodiment of this disclosure, the hardware profile described above may include the number of usable GPUs, wherein the usable GPUs may be all or some of the GPUs included in the service cluster.

[0407] The scheme generation module 302 can determine a first number of GPUs allocated for the prefill stage and a second number of GPUs allocated for the decoding stage, wherein the sum of the first number and the second number is the number of GPUs that can be used, and the prefill stage and the decoding stage can be two stages included in the reasoning process of a large language model.

[0408] Then, for each stage in the pre-filling and decoding stages, the scheme generation module 302 can allocate at least one GPU for each of the multiple parallel strategies based on the number of GPUs allocated to that stage and a preset resource allocation rule, thereby obtaining at least two resource planning candidate schemes. The preset resource allocation rule can be the product of the number of GPUs allocated to each of the multiple parallel strategies, which represents the number of GPUs allocated in the corresponding stage.

[0409] The video memory reserve calculation module 303 can calculate the video memory reserve for each of at least two resource planning candidate schemes based on the aforementioned model profile, hardware profile, and scene profile. This video memory reserve can be the remaining reserve within the upper limit of video memory allowed by resource planning, and the video memory can be the video memory of the GPU.

[0410] According to an exemplary embodiment of this disclosure, for each resource planning candidate scheme, the memory reserve calculation module 303 can calculate the total memory usage Mem_est based on the model profile, hardware profile, and scene profile. This total memory usage can be used to indicate the estimated total memory usage of a single GPU under the resource planning candidate scheme. Furthermore, the model profile can be used to characterize the memory required by the large language model itself; the scene profile can be used to characterize the KV memory required for business concurrency; and the hardware profile can be used to characterize the runtime memory overhead of the framework resident, communication buffer, and operator working area. Then, the memory reserve calculation module 303 can calculate the memory reserve of the resource planning candidate scheme based on the hardware profile and the total memory usage Mem_est.

[0411] According to an exemplary embodiment of this disclosure, the above model profile may include the following items: attention head configuration d_head, i.e., the dimension of the head in the large language model; KV structure n_heads_kv, i.e., the number of KV heads in the large language model; number of bytes b_kv, i.e., how many bytes each numerical element occupies in the KV Cache in the large language model; number of layers n_layers, i.e., the number of layers contained in the large language model; and total weights after quantization Mem_params_total.

[0412] The above hardware profile may include the following items: runtime memory overhead (Mem_overhead), which is the runtime memory overhead of framework resident, communication buffer, operator working area, etc., and the upper limit of memory allowed by resource planning (Budget).

[0413] The scenario profile described above can include the following items: concurrency limit (concurrency_max), maximum concurrency, context length (CTX), effective sharding factor (Shard_kv) for key-value pairs in the tensor parallel dimension (where Shard_kv = 1 means each card carries a complete replica of key-value pairs; if Shard_kv = 4, it means the request key-value pairs are distributed across four cards), and skew coefficient (skew primarily indicates the evenness of concurrent requests among replicas). For example, skew = 1 represents an ideal balanced distribution. However, in real-world systems, due to factors such as historical request distribution, business routing strategies, session stickiness, or hotspot loads, it is often difficult to guarantee a completely balanced distribution of requests among replicas. In such cases, skew > 1 can be used to conservatively amplify the average concurrency, thereby more reasonably estimating peak concurrency.

[0414] According to an exemplary embodiment of this disclosure, for each resource planning candidate scheme, the memory margin calculation module 303 can calculate the effective concurrency upper bound B_peak per replica based on the aforementioned concurrency upper bound concurrency_max and skew coefficient skew. The effective concurrency upper bound B_peak per replica is mainly used to represent the upper bound of the number of requests that each replica may actually carry at peak times.

[0415] Then, the memory availability calculation module 303 can calculate the KV_per_req occupied by a single request KV based on the aforementioned attention head configuration d_head, KV structure n_heads_kv, number of bytes b_kv, number of layers n_layers and context length CTX, that is, how much KV Cache a single request requires in total.

[0416] Next, the memory availability calculation module 303 can calculate the peak KV memory usage Mem_kv per GPU based on the effective concurrency upper bound B_peak per copy, the KV memory usage per request KV_per_req, and the effective sharding factor Shard_kv of the aforementioned KV in the tensor parallel dimension. The peak KV memory usage Mem_kv per GPU is mainly used to represent the KV cache memory usage undertaken by a single GPU under peak concurrency.

[0417] Then, the memory margin calculation module 303 can calculate the weight occupancy Mem_params per GPU based on the total weight after quantization Mem_params_total. The weight occupancy Mem_params per GPU is mainly used to represent the memory occupancy of the model weights distributed to a single GPU under the current parallel scheme.

[0418] Next, the memory margin calculation module 303 can calculate the total memory usage Mem_est based on the weight usage Mem_params per GPU, the peak KV usage Mem_kv per GPU, and the runtime overhead Mem_overhead.

[0419] According to an exemplary embodiment of this disclosure, the memory margin calculation module 303 can calculate the difference between the upper limit of memory Budget allowed by the aforementioned resource planning and the total memory usage Mem_est as the memory margin of the resource planning candidate scheme.

[0420] The scheme selection module 304 can select a target resource planning scheme from at least two resource planning candidate schemes based on the calculated memory margin of each resource planning candidate scheme.

[0421] According to an exemplary embodiment of this disclosure, the scheme selection module 304 can select the resource planning candidate scheme with the largest available video memory from at least two resource planning candidate schemes as the target resource planning scheme.

[0422] According to exemplary embodiments of this disclosure, the above-mentioned parallel strategies may include tensor parallel strategies, pipeline parallel strategies, expert parallel strategies, and data parallel strategies.

[0423] For each resource planning candidate scheme, the scheme selection module 304 can calculate the memory reserve score S_mem based on the memory reserve of the resource planning candidate scheme. The larger the memory reserve, the higher the corresponding memory reserve score S_mem can be.

[0424] Then, the scheme selection module 304 can also calculate the operation and maintenance complexity score S_ops based on the number of GPUs allocated to each of the pipeline parallel strategy, expert parallel strategy and data parallel strategy. The more GPUs allocated to each parallel strategy, the lower the operation and maintenance complexity score S_ops can be.

[0425] Next, the scheme selection module 304 can also calculate the topology risk score S_topo based on the data communication type of the GPU to which the tensor parallel strategy TP is assigned. The topology risk score S_topo for data communication type is intra-machine communication can be higher than the topology risk score S_topo for data communication type is cross-machine communication.

[0426] Then, the scheme selection module 304 can calculate the total score based on the above memory availability score S_mem, operation and maintenance complexity score S_ops, and topology risk score S_topo.

[0427] Next, the scheme selection module 304 can select a target resource planning scheme from at least two resource planning candidate schemes based on the total score. The higher the total score, the greater the likelihood that the corresponding resource planning candidate scheme will be selected as the target resource planning scheme.

[0428] According to exemplary embodiments of the present disclosure, the resource planning apparatus 300 may further include a structured processing module and a resource deployment module.

[0429] The structured processing module can perform structured processing on the aforementioned target resource planning scheme to obtain a structured resource planning scheme. Then, the resource deployment module can deploy resources based on this structured resource planning scheme. In this way, by representing the resource planning scheme in a structured manner, the integrity, reusability, and auditability of the information can be guaranteed.

[0430] Figure 4 This is a block diagram illustrating an electronic device 400 according to an exemplary embodiment of the present disclosure.

[0431] Reference Figure 4 The electronic device 400 includes at least one memory 401 and at least one processor 402. The at least one memory 401 stores instructions that, when executed by the at least one processor 402, perform a resource planning method according to an exemplary embodiment of the present disclosure.

[0432] As an example, electronic device 400 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 400 is not necessarily a single electronic device, but may be a collection of any devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 400 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0433] In electronic device 400, processor 402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0434] The processor 402 can execute instructions or code stored in the memory 401, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0435] The memory 401 may be integrated with the processor 402, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 401 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory 401 and the processor 402 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 402 to read files stored in the memory.

[0436] In addition, the electronic device 400 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 400 can be interconnected via a bus and / or network.

[0437] According to exemplary embodiments of this disclosure, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the resource planning method described above. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0438] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including a computer program that, when executed by a processor, implements the resource planning method according to the present disclosure.

[0439] According to the resource planning method, apparatus, electronic device, storage medium, and program product disclosed herein, hardware conditions, model structure, and business scenarios can be unified into an interpretable static planning framework to form a feasibility determination and robust ranking mechanism for resource planning schemes. Specifically, by using hardware profiles, model profiles, and scenario profiles as unified inputs, this disclosure allows for the feasibility determination and robust ranking of resource planning schemes before resource deployment. This avoids the problems of long iteration cycles and high costs associated with relying solely on pre-deployment stress testing or runtime trial and error for resource planning schemes, thereby improving the interpretability, recalculation, and transferability of resource planning.

[0440] According to exemplary embodiments of this disclosure, by using communication domain / interconnection capability as an explicit constraint, prioritizing the placement of high-frequency communication parallel groups within the same high-speed interconnection domain, and explicitly marking cross-domain boundaries, it is possible to reduce tail delay jitter and stability risks.

[0441] According to exemplary embodiments of this disclosure, by decomposing and estimating weights, KV cache, runtime overhead, and security reservations, it is possible to determine both "infeasible" (over budget) and "high-risk edge" solutions, which can then be addressed in scoring or policies, thereby reducing OOM and uncertainty.

[0442] According to an exemplary embodiment of this disclosure, multi-constraint pruning + interpretable scoring selection: by pruning with hard constraints such as memory capacity, topology rules, and inference backend availability, and scoring within the feasible set according to interpretable sub-items such as memory margin, topology risk, and operational complexity, a more robust resource planning scheme can be output.

[0443] According to an exemplary embodiment of this disclosure, a deployment configuration list that can be directly implemented can be output: the planner can output information including parallel parameters, mapping from dp replicas to node / gpu, key engine parameter suggestions, risk tags, etc., so that the planning results can be directly used for deployment and operation and maintenance.

[0444] According to exemplary embodiments of this disclosure, a unified planning expression for PD separation can be supported. That is, in long prefix scenarios, the resource allocation, parallel parameters and placement mapping summary of prefill_plan and decode_plan can be output, and risks such as KV handover can be alerted, thereby enhancing feasibility and controllability.

[0445] According to an exemplary embodiment of this disclosure, the quantization form can be used as a constraint to ensure product consistency: it can receive an externally specified quantization form / precision as an input constraint, and can record the quant_' and model_artifact (including quant_artifact_id / path) fields in the Plan, thereby ensuring that the deployment plan is consistent with the actual loaded model product, thereby reducing the risk of deployment deviation.

[0446] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0447] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A resource planning method, characterized in that, include: Obtain model profiles, hardware profiles, and scenario profiles, wherein the model profiles are used to characterize the inference attributes of each large language model in the service cluster, the hardware profiles are used to characterize the configuration attributes of the service cluster, and the scenario profiles are used to characterize the requirement characteristics of the business scenarios that the service cluster will serve. Based on the hardware profile, at least two resource planning candidate schemes are generated. The resource planning candidate schemes are schemes for allocating GPU resources for each of the multiple parallel strategies corresponding to the large language model. The parallel strategies are the parallel strategies adopted for distributed inference of the large language model. The GPUs are the GPUs in the device nodes included in the service cluster. Based on the model profile, the hardware profile, and the scene profile, calculate the video memory reserve for each of the at least two resource planning candidate schemes, wherein the video memory reserve is the remaining amount within the upper limit of video memory allowed by resource planning, and the video memory is the video memory of the GPU. Based on the calculated available memory for each resource planning candidate, a target resource planning scheme is selected from the at least two resource planning candidate schemes. The hardware profile includes the maximum amount of video memory allowed by the resource planning. The step of calculating the GPU memory reserve for each of the at least two resource planning candidate schemes based on the model profile, the hardware profile, and the scenario profile includes: For each resource planning candidate scheme, the total video memory usage is calculated based on the model profile, the hardware profile, and the scene profile, wherein the total video memory usage is used to indicate the estimated total video memory usage of a single GPU under the resource planning candidate scheme; Based on the hardware profile and the total video memory usage, calculate the video memory reserve of the candidate resource planning scheme; The step of calculating the remaining video memory for the candidate resource planning scheme based on the hardware profile and the total video memory usage includes: The difference between the maximum allowed video memory in the resource planning and the total video memory usage is calculated as the video memory margin of the candidate resource planning scheme.

2. The resource planning method as described in claim 1, characterized in that, The hardware profile includes the number of usable GPUs, wherein the usable GPUs are all or some of the GPUs contained in the service cluster; Based on the hardware profile, at least two resource planning candidate schemes are generated, including: Based on the number of available GPUs and a preset resource allocation rule, at least one GPU is allocated to each of the multiple parallel strategies to obtain at least two resource planning candidate schemes. The preset resource allocation rule is that the product of the number of GPUs allocated to each of the multiple parallel strategies is the number of available GPUs.

3. The resource planning method as described in claim 1, characterized in that, The hardware profile includes the number of usable GPUs, wherein the usable GPUs are all or some of the GPUs contained in the service cluster; Based on the hardware profile, at least two resource planning candidate schemes are generated, including: A first number of GPUs allocated for the prefilling stage and a second number of GPUs allocated for the decoding stage are determined, wherein the sum of the first number and the second number is the number of GPUs available, and the prefilling stage and the decoding stage are two stages included in the inference process of the large language model; For each stage in the prefilling stage and the decoding stage, based on the number of GPUs allocated to that stage and a preset resource allocation rule, at least one GPU is allocated to each of the multiple parallel strategies to obtain the at least two resource planning candidate schemes. The preset resource allocation rule is that the product of the number of GPUs allocated to each of the multiple parallel strategies is the number of GPUs allocated in the corresponding stage.

4. The resource planning method as described in claim 1, characterized in that, The model profile includes the following items: attention head configuration, key-value structure, number of bytes, number of layers, and total weights after quantization; The hardware profile also includes the following: runtime video memory overhead; The scene profile includes the following items: concurrency limit, skew coefficient, context length, and effective sharding factor of KV in tensor parallel dimension.

5. The resource planning method as described in claim 4, characterized in that, For each resource planning candidate scheme, the total video memory usage is calculated based on the model profile, the hardware profile, and the scenario profile, including: For each resource planning candidate scheme, the effective concurrency upper bound per replica is calculated based on the concurrency upper limit and the skew coefficient; Based on the attention header configuration, the key-value structure, the number of bytes, the number of layers, and the context length, calculate the key-value occupancy for a single request; Based on the effective concurrency upper bound per replica, the single request KV occupancy, and the effective sharding factor of KV in the tensor parallel dimension, calculate the peak KV occupancy per GPU; Based on the total quantized weights, calculate the weight occupancy per GPU; The total video memory usage is calculated based on the weight usage per GPU, the peak KV usage per GPU, and the runtime video memory overhead.

6. The resource planning method as described in claim 1, characterized in that, The step of selecting a target resource planning scheme from at least two resource planning candidate schemes based on the calculated available memory for each candidate scheme includes: The resource planning candidate with the largest available video memory among the at least two resource planning candidate schemes is selected as the target resource planning scheme.

7. The resource planning method as described in claim 1, characterized in that, The various parallel strategies include tensor parallelism, pipeline parallelism, expert parallelism, and data parallelism. The step of selecting a target resource planning scheme from at least two resource planning candidate schemes based on the calculated available memory for each candidate scheme includes: For each resource planning candidate scheme, a memory reserve score is calculated based on the memory reserve of the resource planning candidate scheme, wherein the larger the memory reserve, the higher the corresponding memory reserve score. The operational complexity score is calculated based on the number of GPUs allocated to each of the pipeline parallel strategy, the expert parallel strategy, and the data parallel strategy. The more GPUs allocated to each of the parallel strategies, the lower the operational complexity score. Based on the data communication type of the GPU to which the tensor parallel strategy is assigned, a topology risk score is calculated, wherein the topology risk score for data communication type intra-machine communication is higher than the topology risk score for data communication type cross-machine communication. The total score is calculated based on the memory remaining capacity score, the operation and maintenance complexity score, and the topology risk score. Based on the total score, the target resource planning scheme is selected from the at least two resource planning candidate schemes, wherein the higher the total score, the greater the likelihood that the corresponding resource planning candidate scheme will be selected as the target resource planning scheme.

8. The resource planning method as described in claim 1, characterized in that, The resource planning method also includes: The target resource planning scheme is processed in a structured manner to obtain a structured resource planning scheme. Based on the structured resource planning scheme, resource deployment is carried out.

9. The resource planning method as described in claim 1, characterized in that, The acquisition of model profile, hardware profile, and scene profile includes: Obtain the model structure information of the large language model, the hardware information of the service cluster, and the business-side deployment target information; The model structure information of the large language model, the hardware information of the service cluster, and the deployment target information of the business side are standardized to obtain the model profile, the hardware profile, and the scenario profile.

10. A resource planning device, characterized in that, include: The profile acquisition module is configured to acquire model profiles, hardware profiles, and scenario profiles. The model profile is used to characterize the inference attributes of each large language model in the service cluster, the hardware profile is used to characterize the configuration attributes of the service cluster, and the scenario profile is used to characterize the requirement characteristics of the business scenario that the service cluster will serve. The solution generation module is configured to generate at least two resource planning candidate solutions based on the hardware profile. The resource planning candidate solutions are solutions for allocating GPU resources for each of the multiple parallel strategies corresponding to the large language model. The parallel strategies are the parallel strategies used for distributed inference of the large language model. The GPUs are the GPUs in the device nodes included in the service cluster. The video memory reserve calculation module is configured to calculate the video memory reserve of each of the at least two resource planning candidate schemes based on the model profile, the hardware profile, and the scene profile, wherein the video memory reserve is the remaining reserve within the upper limit of video memory allowed by resource planning, and the video memory is the video memory of the GPU. The scheme selection module is configured to select a target resource planning scheme from the at least two resource planning candidate schemes based on the calculated memory margin of each resource planning candidate scheme; The hardware profile includes the maximum amount of video memory allowed by the resource planning. The memory margin calculation module is configured as follows: For each resource planning candidate scheme, the total video memory usage is calculated based on the model profile, the hardware profile, and the scene profile, wherein the total video memory usage is used to indicate the estimated total video memory usage of a single GPU under the resource planning candidate scheme; Based on the hardware profile and the total video memory usage, calculate the video memory reserve of the candidate resource planning scheme; The memory margin calculation module is configured as follows: The difference between the maximum allowed video memory in the resource planning and the total video memory usage is calculated as the video memory margin of the candidate resource planning scheme.

11. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the resource planning method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the resource planning method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the resource planning method as described in any one of claims 1 to 9.

Images