Computing resource allocation method and device, storage medium and program product

Abstract

Embodiments of the present disclosure disclose a computing resource allocation method and device, a storage medium and a program product. The method comprises: collecting current running state information of computing devices in each node in a computing cluster to form a global resource view; generating a plurality of feasible resource allocation candidate schemes based on the global resource view, memory requirements of a target model and a parallel strategy, and establishing an optimization model based on the schemes; solving the optimization model to obtain an optimal resource allocation candidate scheme; based on a preset arrangement template, arranging the optimal resource allocation candidate scheme into a deployment configuration list; and based on the deployment configuration list, executing an inference task of the target model. The method can build a global resource view, automatically generate an optimal resource allocation scheme, and execute an inference task in combination with a preset arrangement template, thereby realizing rational utilization of computing resources, reducing manual configuration errors and time, improving execution efficiency, and flexibly coping with diversified task requirements.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, storage medium, and program product for allocating computing resources. Background Technology

[0002] In today's technological development, cluster resource allocation and task deployment management are crucial. Existing technical solutions mostly rely on manual configuration or simple script automation for management. With the rapid popularization of Large Models (LLM) in scenarios such as search, dialogue, and code generation, enterprise-level inference services are characterized by multiple models, multiple tenants, and long-term online operation. The current mainstream approach is to deploy services on orchestration platforms such as Kubernetes, using Device Plugins to expose heterogeneous computing resources such as GPUs and ASICs to the workload.

[0003] However, manual configuration or simple script automation methods are insufficient to handle the demands for dynamic resource allocation and efficient task deployment in complex scenarios involving multiple models, multiple tenants, and long online periods brought about by large models. While deploying services on platforms such as Kubernetes can achieve some resource management and task scheduling, its shortcomings in resource awareness, high-throughput scheduling, and optimization support for native AI workloads are becoming increasingly apparent as large models become more widespread. Summary of the Invention

[0004] In view of this, the present disclosure provides a computing resource allocation method, apparatus, storage medium, and program product, which can achieve rational utilization of computing resources, reduce manual configuration errors and time, improve execution efficiency, and flexibly meet diverse task requirements by constructing a global resource view, automatically generating the optimal resource allocation scheme, and executing inference tasks in combination with preset orchestration templates.

[0005] In a first aspect, embodiments of this disclosure provide a method for allocating computing resources, employing the following technical solution:

[0006] Collect the current operating status information of computing devices in each node of the computing power cluster, and construct a global resource view based on the current operating status information;

[0007] Based on the global resource view, the memory requirements of the target model, and the parallel strategy, several feasible resource allocation candidate schemes are generated.

[0008] An optimization model is established based on all candidate resource allocation schemes.

[0009] Solve the optimization model to obtain the optimal resource allocation candidate scheme;

[0010] Based on the preset orchestration template, the optimal resource allocation candidate schemes are organized into a deployment configuration list;

[0011] Based on the deployment configuration list, perform inference tasks for the target model.

[0012] Optionally, the process of collecting operational status information of each node and the computing devices within the computing power cluster, and constructing a global resource view based on the operational status information, includes:

[0013] Real-time monitoring of the operating status information of computing devices in each node of the computing power cluster;

[0014] When the inference task of the target model is received, a lightweight snapshot is uniformly pulled within the preset acquisition window through the control container group in the computing power cluster to obtain the current running status information of the computing devices in each node;

[0015] The current running status information is organized into a global resource view in tabular form.

[0016] Optionally, based on the global resource view, the memory requirements of the target model, and the parallel strategy, multiple feasible resource allocation candidate schemes are generated, including:

[0017] Based on the global resource view, the memory requirements of the target model, and the parallel strategy, multiple feasible resource allocation candidate schemes are enumerated, and the variable parameters of the resource allocation candidate schemes are obtained.

[0018] Configure global variable parameters to bind the variable parameters of each resource allocation candidate scheme to the global variable parameters.

[0019] Optionally, the optimization model includes an objective function and constraints;

[0020] The constraints include global consistency constraints and linear inequality constraints.

[0021] Optionally, the expression for the objective function is:

[0022]

[0023] In the formula, Indicates the weighting coefficient; Indicates the first The selection variables for each node ,when When, it indicates the first When no node is selected, When, it indicates the first One node was selected; Indicates the first Available video memory for each computing device; Indicates the first The selection variables for each computing device ,when When, it indicates the first A computing device was not selected when When, it indicates the first One computing device was selected; Indicates according to the first When deploying a model with multiple resource allocation candidate schemes, the amount of video memory usage needs to be evenly distributed across each computing device. Indicates the first The actual total number of computing devices required for each resource allocation candidate scheme; Indicates the first The selection variables for each resource allocation candidate scheme ,when When, it indicates the first When no resource allocation candidate was selected, When, it indicates the first One resource allocation candidate was selected.

[0024] Optionally, the expression for the global consistency constraint is:

[0025]

[0026] In the formula, Indicates the total number of computing devices. ; This indicates the total number of devices that need to be calculated globally;

[0027] The expression for the linear inequality constraint is:

[0028]

[0029] In the formula, This indicates the actual allocated video memory of the computing device; Represents a constant.

[0030] Optionally, solving the optimization model to obtain the optimal resource allocation candidate scheme includes:

[0031] The constraint planning-Boolean satisfiability solver is invoked to solve the optimization model and obtain the optimal resource allocation candidate scheme.

[0032] Secondly, this disclosure also provides a computing resource allocation system, which adopts the following technical solution:

[0033] The acquisition module is used to collect the current operating status information of computing devices in each node of the computing power cluster, and to construct a global resource view based on the current operating status information;

[0034] The generation module is used to generate multiple feasible resource allocation candidate schemes based on the global resource view, the memory requirements of the target model, and the parallel strategy.

[0035] A module is established to build an optimization model based on all candidate resource allocation schemes;

[0036] The acquisition module is used to solve the optimization model and obtain the optimal resource allocation candidate scheme;

[0037] The sorting module is used to sort the candidate schemes for optimal resource allocation into a deployment configuration list based on a preset orchestration template;

[0038] The execution module is used to perform inference tasks on the target model based on the deployment configuration manifest.

[0039] Thirdly, this disclosure also provides a computer device, which adopts the following technical solution:

[0040] The computer device includes:

[0041] At least one processor; and,

[0042] A memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform any of the computing resource allocation methods described above.

[0044] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions for causing a computer to execute any of the computing resource allocation methods described above.

[0045] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0046] The computing resource allocation method provided in this disclosure collects the current operating status information of computing devices within each node of the computing cluster and constructs a global resource view based on this information, providing comprehensive and accurate foundational information for subsequent resource allocation. By constructing a global resource view, the system can clearly understand the overall status of computing resources within the cluster. Combined with the memory requirements and parallel strategies of the target model, this allows for more rational utilization of computing resources in the cluster, avoiding resource waste. Only with a clear understanding of the resources can scientific allocation be made according to actual needs, preventing resource idleness or over-allocation.

[0047] Based on a global resource view, the memory requirements of the target model, and parallel strategies, multiple feasible resource allocation candidate schemes are generated, an optimization model is established, and then the optimal resource allocation candidate scheme is obtained through solving. This series of steps automates the process of generating the optimal resource allocation scheme. In traditional resource allocation, manual configuration is not only time-consuming but also prone to errors. This method, through automation, reduces the time and errors of manual configuration and improves the execution efficiency of inference tasks. The system can quickly select the optimal scheme from multiple candidate schemes according to preset rules and algorithms, making resource allocation more accurate and efficient.

[0048] Based on a pre-defined orchestration template, the optimal resource allocation candidate schemes are compiled into a deployment configuration list, and the inference task of the target model is executed based on this list. This demonstrates that the method can quickly adapt to different model memory requirements and parallel strategies, and flexibly adjust the resource allocation scheme. Different target models may have different memory requirements and parallel strategies. This method can dynamically generate a suitable deployment configuration list based on these changes, thereby better handling diverse task requirements. Whether facing simple or complex models, this method can respond quickly, allocate resources reasonably, and ensure the smooth execution of the inference task.

[0049] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating the computing resource allocation method provided in this embodiment of the disclosure;

[0052] Figure 2 A flowchart illustrating the global resource view construction method provided in this embodiment of the disclosure;

[0053] Figure 3 A flowchart illustrating the method for obtaining resource allocation candidate schemes provided in this embodiment of the disclosure;

[0054] Figure 4 A schematic diagram of the computing resource allocation system provided in the embodiments of this disclosure;

[0055] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. Detailed Implementation

[0056] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0057] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0058] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0059] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0060] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0061] Reference Figure 1 This disclosure provides a method for allocating computing resources, including the following steps:

[0062] S1: Collect the current operating status information of computing devices in each node of the computing power cluster, and construct a global resource view based on the current operating status information;

[0063] S2: Based on the global resource view, the memory requirements of the target model, and the parallel strategy, generate multiple feasible resource allocation candidate schemes;

[0064] S3: Establish an optimization model based on all candidate resource allocation schemes;

[0065] S4: Solve the optimization model to obtain the optimal resource allocation candidate scheme;

[0066] S5: Based on the preset orchestration template, the optimal resource allocation candidate schemes are compiled into a deployment configuration list;

[0067] S6: Based on the deployment configuration list, execute the inference task of the target model.

[0068] The computing resource allocation method disclosed herein provides comprehensive and accurate foundational information for subsequent resource allocation by collecting the current operating status information of computing devices within each node of the computing cluster and constructing a global resource view based on this information. By constructing a global resource view, the system can clearly understand the overall status of computing resources within the cluster. Combined with the memory requirements and parallel strategies of the target model, this allows for more rational utilization of computing resources in the cluster, avoiding resource waste. Only with a clear understanding of the resources can scientific allocation be made according to actual needs, preventing resource idleness or over-allocation.

[0069] Based on a global resource view, the memory requirements of the target model, and parallel strategies, multiple feasible resource allocation candidate schemes are generated, an optimization model is established, and then the optimal resource allocation candidate scheme is obtained through solving. This series of steps automates the process of generating the optimal resource allocation scheme. In traditional resource allocation, manual configuration is not only time-consuming but also prone to errors. This method, through automation, reduces the time and errors of manual configuration and improves the execution efficiency of inference tasks. The system can quickly select the optimal scheme from multiple candidate schemes according to preset rules and algorithms, making resource allocation more accurate and efficient.

[0070] Based on a pre-defined orchestration template, the optimal resource allocation candidate schemes are compiled into a deployment configuration list, and the inference task of the target model is executed based on this list. This demonstrates that the method can quickly adapt to different model memory requirements and parallel strategies, and flexibly adjust the resource allocation scheme. Different target models may have different memory requirements and parallel strategies. This method can dynamically generate a suitable deployment configuration list based on these changes, thereby better handling diverse task requirements. Whether facing simple or complex models, this method can respond quickly, allocate resources reasonably, and ensure the smooth execution of the inference task.

[0071] In S1, refer to Figure 2 The flowchart illustrating the global resource view construction method shows the following steps: "Collecting the current operating status information of computing devices in each node of the computing power cluster, and constructing a global resource view based on the current operating status information":

[0072] S11: Real-time monitoring of the operating status information of computing devices in each node of the computing power cluster;

[0073] S12: When the inference task of the target model is received, a lightweight snapshot is uniformly pulled once within the preset acquisition window through the control container group in the computing power cluster to obtain the current running status information of the computing devices in each node;

[0074] S13: Organize the current running status information into a global resource view in tabular form.

[0075] In the above steps, computing devices include GPUs, ASICs, TPUs, etc. This solution mainly focuses on GPU resource allocation, but it can also be applied to other computing devices such as ASICs and TPUs. NVIDIA DCGM (or dcgm-exporter issued by the computing device operator) is deployed and persistently run on each node (i.e., the computing device server, usually a GPU server). NVIDIA DCGM is a tool for managing and monitoring computing devices in a data center environment. dcgm-exporter is a component related to NVIDIA DCGM and can be issued by the computing device operator. The main function of dcgm-exporter is to expose the computing device operational status indicators collected by DCGM in Prometheus format. The main function of these tools is to continuously maintain the computing device operational status indicators, acting like a "guardian," constantly monitoring the operational status of the computing devices within the node, preparing for the subsequent acquisition of accurate resource information.

[0076] Upon receiving an inference task from a target model (such as a neural network, Bayesian network, or other intelligent model), before resource allocation, the control container group within the computing cluster will uniformly pull a lightweight snapshot within a preset collection window. The control container group refers to the control Pod, specifically the Inventory-Snapshotter control Pod, which is a basic deployable and manageable computing unit in Kubernetes (K8s, the kube container orchestration engine). The control container group obtains the UUID, total GPU memory, and available GPU memory for each computing device by accessing the HTTP metrics endpoint of dcgm-exporter, and associates this with the node name where the computing device resides. This step is performed based on continuous monitoring on the host side. Since tools are already continuously maintaining computing device status metrics, the control container group can directly obtain the necessary information from the HTTP metrics endpoints provided by these tools. Through a one-time snapshot collection, key resource information for all computing devices can be obtained at a specific moment, providing data support for the subsequent generation of a resource table. The collected data is then aggregated to generate a temporary resource table at the computing device level. The core fields of this table only include the computing device's `gpu_id`, `node`, `mem_total`, and `mem_free`, representing the computing device's unique identifier, the name of the node it resides on, the total GPU memory, and the available GPU memory, respectively. This step is based on the previously collected snapshot data. The collected data is scattered; through aggregation and organization, it is transformed into a structured temporary resource table for convenient subsequent use. This temporary resource table is the global resource view. This method adopts a minimal implementation scheme of "host-side resident monitoring + centralized one-time snapshot" to generate a temporary resource table at the computing device level. In this process, there is no need for topology inference or timing consistency control. Moreover, with this global resource view, subsequent resource allocation candidate scheme generation and linear constraint modeling can directly use the information in it, avoiding complex topology inference and timing consistency control, and simplifying the resource allocation process.

[0077] In S2, refer to Figure 3 The flowchart illustrating the resource allocation candidate solution acquisition method shows that "based on the global resource view, the memory requirements of the target model, and the parallel strategy, multiple feasible resource allocation candidate solutions are generated," including:

[0078] S21: Based on the global resource view, the memory requirements of the target model, and the parallel strategy, enumerate multiple feasible resource allocation candidate schemes and obtain the variable parameters of the resource allocation candidate schemes.

[0079] S22: Configure global variable parameters to bind the variable parameters of each resource allocation candidate scheme to the global variable parameters.

[0080] In S21, if the user interface receives a memory requirement for the target model, it is directly adopted; otherwise, the target model is analyzed in detail to determine its total memory requirements at runtime. This can be estimated through the model's architecture, number of parameters, and the scale of data during training or inference. For deep learning models, the memory occupied by intermediate computation results and gradient information also needs to be considered. Accurate memory requirements can be obtained using tools provided by the model framework or through manual calculation. Similarly, if the user interface receives a parallelism strategy for the target model, it is directly adopted; otherwise, a suitable parallelism strategy is selected based on the characteristics of the target model and system resources. Parallelism strategies include tensor parallelism and pipelined parallelism. Both fall under the category of model parallelism; they are methods of decomposing the model to achieve parallel computation. Tensor parallelism decomposes tensors in the model, while pipelined parallelism decomposes layers. The choice of parallelism strategy is determined based on factors such as model size, computational complexity, and data distribution. Based on the previously obtained global resource view, the target model's memory requirements, and parallelism strategies, all possible resource allocation combinations, i.e., resource allocation candidate schemes, are generated. For example, if a tensor parallelism strategy is adopted, resource allocation candidates could be to split large tensors in the model by rows or columns in different proportions and allocate them to different numbers and combinations of computing devices. If a pipelined parallelism strategy is adopted, resource allocation candidates could be to partition different layers of the model onto computing devices in different ways. Tensor parallelism and pipelined parallelism can be used in combination to form a hybrid parallelism strategy. During the enumeration process, it is necessary to ensure that each resource allocation candidate meets the memory requirements of the target model and the resource constraints of the system.

[0081] Based on a pre-defined symbol and variable system, variable parameters for resource allocation candidate schemes are set. These variable parameters include model deployment method, tensor parallelism, pipeline parallelism, and the total number of computing devices actually required. The model deployment method includes single-node single-computing-device, single-node multi-computing-device, and multi-node multi-computing-device. Single-node single-computing-device, also known as single-machine single-card, refers to using only one computing device on a single node for computational tasks. Single-node multi-computing-device, also known as single-machine multi-card, refers to using multiple computing devices simultaneously on a single node to collaboratively complete computational tasks. Multi-node multi-computing-device, also known as multi-machine multi-card, refers to using computing devices on multiple nodes simultaneously to collaboratively complete computational tasks. Tensor parallelism refers to the number of computing devices or partitions used when large tensors in a model are split into rows, columns, or other components and distributed to different computing devices for parallel computation in a tensor parallelism strategy. It reflects the granularity and scale of tensor parallel computation. Pipeline parallelism refers to the number of pipeline stages used when different layers of a model are divided into different computing devices to form a pipeline for parallel computation in a pipeline parallelism strategy. It reflects the degree of stage division and parallelization level of pipeline parallel computation. The variable parameters of each resource allocation candidate scheme can be represented by a quadruple, which is... ,in, Indicates the first The model deployment method for each resource allocation candidate scheme; Indicates the first Tensor parallelism of resource allocation candidate schemes; Indicates the first Pipeline parallelism of each resource allocation candidate scheme; Indicates the first The actual total number of computing devices required for each resource allocation candidate scheme; , This represents the set of candidate resource allocation schemes, and can also be regarded as the total number of candidate resource allocation schemes.

[0082] The variable parameters for each resource allocation candidate scheme must satisfy the following rules: If the model deployment method is single-node single computing device, then tensor parallelism, pipeline parallelism, and the actual total number of computing devices required are all set to 1; if the model deployment method is single-node multi-computing device, then tensor parallelism is set to 1, and the actual total number of computing devices required is equal to the value of pipeline parallelism; if the model deployment method is multi-node multi-computing device, then the actual total number of computing devices required is equal to the product of the value of tensor parallelism and the value of pipeline parallelism. This rule can be expressed by the following formula:

[0083]

[0084] In the formula, Indicates the first The deployment method for each resource allocation candidate scheme is a single node and a single computing device; Indicates the first The deployment method for each resource allocation candidate scheme is a single node with multiple computing devices; Indicates the first The deployment mode of the resource allocation candidate scheme is multi-node and multi-computing device.

[0085] In S22, global variable parameters include the total number of computing devices required globally, the actual allocated GPU memory per computing device, and deployment type switches. The deployment type switches include a first type switch, a second type switch, and a third type switch, corresponding to single-node single-computing-device, single-node multiple-computing-device, and multi-node multiple-computing-device scenarios, respectively. To facilitate binding the variable parameters of each resource allocation candidate scheme with the global variable parameters, it is also necessary to obtain the GPU memory allocation per computing device when deploying the model according to the resource allocation candidate scheme. The calculation formula is as follows:

[0086]

[0087] In the formula, Indicates according to the first When deploying a model with multiple resource allocation candidate schemes, the amount of video memory usage needs to be evenly distributed across each computing device. This represents the floor function; This represents the estimated total video memory required for the target model.

[0088] The formula for binding the variable parameters of resource allocation candidate schemes to global variable parameters is as follows:

[0089]

[0090] In the formula, This indicates the total number of devices that need to be calculated globally; Indicates the first The selection variables for each resource allocation candidate scheme ,when When, it indicates the first When no resource allocation candidate was selected, When, it indicates the first One resource allocation candidate was selected; This indicates the actual allocated video memory of the computing device; This indicates the first type of switch, which corresponds to a single node and a single computing device. This indicates the second type of switch, which corresponds to a single-node multi-computing device. This indicates a third type of switch, which corresponds to multiple nodes and multiple computing devices. , The values ​​of the three type switches are determined by the resource allocation candidate scheme. Since a resource allocation candidate scheme has only one deployment mode: single-node single-computing device, single-node multi-computing device, or multi-node multi-computing device, the values ​​for the deployment type switches are determined accordingly. Only one value is 1, and the rest are 0; Indicates the satisfaction conditional Perform summation; Indicates the satisfaction conditional Perform summation; Indicates the satisfaction conditional Perform summation.

[0091] To adapt to the solver's requirements for linear constraints, this method employs a "resource allocation candidate scheme discretization" strategy. This transforms the nonlinearity of the ratio between the estimated total video memory required by the target model and the actual number of computing devices needed into a constant parameter input, eliminating the nonlinearity caused by the "total video memory / number of cards" equation in the original problem. First, multiple candidate configurations, including deployment type, TP (tensor parallelism), PP (pipeline parallelism), and total number of cards, are pre-enumerated within a discrete parallelism and deployment paradigm space. The actual video memory allocated to each computing device is then used as a constant bound to the candidate to construct the input for the optimization model. This approach achieves a purely linear expression of problem feasibility and the objective function, laying the foundation for subsequent screening and rapid solution. For each resource allocation candidate scheme, pre-calculate the total number of computing devices actually needed and the amount of video memory allocated to each computing device when deploying the model according to the resource allocation candidate scheme. Then, in the preset solver, the unique activation of the resource allocation candidate scheme selection variable completes the linear binding with the global variable, thereby describing the total number of cards used and the video memory requirement of a single card in a pure linear / Boolean form, avoiding the solution difficulties caused by nonlinearity in the future, and ensuring seamless connection with the preset variable system.

[0092] In S3, the optimization model includes an objective function and constraints. The constraints include globally consistent constraints and linear inequality constraints. The objective function is expressed as follows:

[0093]

[0094] In the formula, Indicates the weighting coefficient; Indicates the first The selection variables for each node ,when When, it indicates the first When no node is selected, When, it indicates the first One node was selected; Indicates the first Available video memory for each computing device; Indicates the first The selection variables for each computing device ,when When, it indicates the first A computing device was not selected when When, it indicates the first One computing device was selected.

[0095] The expression for the global consistency constraint is:

[0096]

[0097] In the formula, Indicates the total number of computing devices. Regarding global consistency, it is required that the number of computing devices selected be strictly equal to the total number of cards required for the candidates. This constraint ensures that the solver will not produce unexecutable solutions due to "too many or too few selections" and serves as the benchmark for all subsequent feasibility and objective calculations.

[0098] The expression for the linear inequality constraint is:

[0099]

[0100] In the formula, This represents a constant, which is a sufficiently large constant obtained through the Big-M() method, used in... At that time, "relax" the constraint (make it automatically apply), while This requires that the remaining video memory on the computing device be no less than the video memory required per card. This is achieved using constants pre-computed during the candidate discretization stage. The entire constraint remains linear, thus enabling it to be efficiently processed by solvers such as CP-SAT.

[0101] This method employs Constraint Programming (CP) to establish an optimization model containing decision variables and linear constraints. Constraint Programming is a mathematical modeling and solution technique based on constraint satisfaction, particularly suitable for handling complex combinatorial optimization problems. This method models the problem in a "variable-constraint-objective" format and automatically searches for the optimal solution that meets the conditions. The optimization objective adopts a two-level lexicographical strategy: the first-level objective is to minimize the number of nodes occupied, which is the primary objective, prioritizing the control of cross-node communication and scheduling overhead; based on satisfying the first-level optimality, the second-level objective is to minimize memory waste (i.e., maximize memory utilization), and can be extended to introduce linear indicators such as cross-node communication overhead, specifically defined as "the difference between the total available memory of the selected GPUs and the actual memory required by the model." This strategy uniformly supports three deployment paradigms: single-machine single-GPU, single-machine multi-GPU, and multi-machine multi-GPU, and ensures the satisfaction of the "consistency of GPU usage per node" constraint through explicitly modeled node selection variables. Therefore, the objective function is a combination of the functions of the first-level objective and the second-level objective. The function expression of the first-level objective is:

[0102]

[0103] The functional expression for the second-level objective is:

[0104]

[0105] For ease of engineering implementation, lexicographical order optimization can be approximated as a weighted model with a single solution: by assigning a sufficiently large weight coefficient to the first-level objective. By making its numerical range significantly larger than the possible maximum value of the second-level target, the lexicographical order effect of "number of nodes first, then waste of video memory" can be simulated in one optimization.

[0106] This method models the consistency of the number of nodes and computing devices through a "node triggering mechanism + fixed number of cards per node" approach to uniformly support both single-machine and multi-machine deployment paradigms. If the deployment type is a single-machine paradigm (i.e....) or The constraints of the optimization model ensure that only one node is enabled, and the number of computing devices allocated on that node is equal to the total number of cards in the single-machine mode of the candidate configuration (when...). hour, Representing 1 or The number of cards used on the remaining nodes is forced to 0. The specific format is as follows:

[0107]

[0108]

[0109] In the formula, Indicates the first The node number where each computing device is located.

[0110] If the deployment type is multi-machine paradigm ( This requires that the number of nodes enabled be exactly equal to the pipeline parallelism. Furthermore, the number of computing devices allocated on each enabled node is always equal to the tensor parallelism. The specific format is as follows:

[0111]

[0112]

[0113] This method can merge multiple rounds of optimization into a single round of solution without changing the constraints, thus maintaining the optimization effect, improving the solution efficiency, and making it easier to deploy in actual systems.

[0114] Based on the above, after obtaining the computing resource snapshot and resource allocation candidate schemes, this method integrates them and transforms them into a linear model (i.e., an optimization model) that can be directly processed by the solver. It uses the decision variables and constants defined earlier and explicitly binds the attributes of each candidate combination to global variables, ensuring that "the actual total number of computing devices required," "the actual amount of GPU memory allocated to computing devices," and "deployment type" have a consistent and unique input source in the optimization model. Regarding the construction of the constraint system, the CP (Constraint Programming) model mainly follows and implements the following three principles: global consistency, GPU memory capacity feasibility, and consistency between node and computing device usage. Through this method, an optimization model containing decision variables and linear constraints can be constructed.

[0115] In S4, the Constraint Programming-Boolean Satisfiability Problem Solver (CP-SAT) is invoked to solve the optimization model and obtain candidate solutions for optimal resource allocation. CP-SAT combines techniques from both constraint programming and Boolean satisfiability problem solving to address complex constraint satisfaction and optimization problems. This CP-SAT solver efficiently handles linear constraints between Boolean and integer variables and maintains good performance even in large-scale problems thanks to a conflict-driven search strategy and cutting plane techniques.

[0116] By calling the CP-SAT solver to search for the optimal subset of devices that satisfy the global objective, it can automatically generate a deployment list and parameter configurations (such as TP, PP, and device mapping relationships) suitable for mainstream inference frameworks. Finally, the allocation results are sent to the orchestration system of the computing cluster and the resource status is updated, achieving automated and interpretable resource allocation. The results returned by the CP-SAT solver and related runtime parameters specifically cover: the selected optimal resource allocation candidate schemes (including...) ), the set of computing devices allocated , set of enabled nodes ,as well as Derived values ​​include image address, model path, service port, environment variables, number of instance replicas, and deployment parameters such as deployment strategy.

[0117] In constrained programming modeling, the system of symbols and variables includes the following three categories: indices and sets, which are used to define the search space for the solution (computing devices, nodes, and candidate resource allocation schemes, etc.). The variable parameters of these indices and sets include... , (node ​​set, ), , (Candidate set of tensor parallelism) and (Candidate set of pipeline parallelism); Input parameters are known quantities determined by inventory snapshots and parallelism strategies to limit the feasible region. These input parameters include... , , , , and ={0: Single node, single computing device; 1: Single node, multiple computing devices; 2: Multiple nodes, multiple computing devices} (deployment paradigm enumeration), where... and It can be tailored according to cluster capabilities, for example , Decision variables characterize the choices and switches (card selection, node usage, deployment paradigm, and candidate activations) that the solver needs to decide. Their values ​​will be determined under subsequent linear constraints and objectives. These decision variables include... , , , , , , and ,in, It refers to positive integers, for All decision variables have a unique activation pattern.

[0118] In S5 and S6, the final output of the CP-SAT solver is a standard deployment description file that can be directly distributed to the orchestration system. This standard deployment description file is a deployment configuration list compiled from candidate optimal resource allocation schemes based on a preset orchestration template. It includes two types of configuration information: first, inference framework-specific parameters, such as `--tensor-parallel-size`, `--pipeline-parallel-size`, and device mappings required by vLLM or SGLang; and second, a list of cluster orchestration resources, such as Kubernetes Pod descriptions or Helm Values ​​configurations. The orchestration system uses this deployment configuration list to automatically deploy large model inference services, thereby guiding the system to execute inference tasks for the target model.

[0119] With the rapid proliferation of large-scale models in search, dialogue, and code generation scenarios, enterprise-level inference services are exhibiting characteristics of multi-model, multi-tenancy, and long-term online operation. The current mainstream approach is to deploy services on orchestration platforms such as Kubernetes, exposing heterogeneous resources through Device Plugins, employing virtualization isolation methods such as "full-card passthrough" or vGPU / MIG, and completing deployment by combining inference framework configuration interfaces. The platform is equipped with a monitoring and alerting system, and service stability is maintained after deployment by adding or removing instances or stopping services. To reduce idle time, some platforms have introduced GPU sharing / isolation and scheduling strategies at the container layer, resulting in a general technology stack of "resource virtualization + scheduling algorithms." However, current resource virtualization and containerization operation and maintenance solutions have significant shortcomings in automated computing power orchestration for distributed inference of large models. First, there is a lack of a unified global optimization scheme that considers parallel strategies, cluster topology, and resource availability. Equipment selection relies on experience or heuristic scripts, which can easily lead to increased memory fragmentation and cross-node bandwidth pressure. Second, the feasibility verification methods for deployment schemes are rudimentary and lack a unified mechanism, requiring multiple trials and errors. Third, multi-objective collaborative optimization relies on fixed weights or heuristic strategies, making it difficult to optimize multiple objectives simultaneously. Fourth, the "resources are allocated once and released only when the service terminates" operation and maintenance model exacerbates global resource fragmentation and waste. Fifth, the current process lacks an end-to-end automated closed loop, resulting in low deployment and delivery efficiency and high operation and maintenance costs.

[0120] To address the aforementioned issues, this disclosure proposes an automated computing resource allocation system based on constraint programming for distributed inference of large models. The system first constructs a global resource view at the computing device level through a one-time resource snapshot, shielding redundant information and real-time fluctuation interference. Next, it discretizes candidate resource allocation schemes and pre-calculates the required GPU memory per GPU, ensuring the constraint programming model is solvable within a linear range. Finally, based on a unified CP-SAT constraint model and lexicographical order optimization strategy, it prioritizes minimizing the number of occupied nodes and then reduces GPU memory waste, automatically generating deployment parameters and orchestration templates to achieve an end-to-end automated closed loop of "data acquisition—candidate generation—constraint modeling—solving—execution backfilling." This system can achieve global automated optimization of distributed inference resources without altering the existing orchestration and inference framework, significantly reducing cross-node communication overhead, improving operational stability, suppressing GPU memory fragmentation, increasing overall resource utilization, reducing manual intervention and trial-and-error costs, shortening service launch time, and possessing good versatility and scalability. For example, in a real-world application by a telecommunications operator, the system has resulted in more stable service throughput, less cross-node communication, significantly shorter deployment time, and a marked improvement in overall GPU utilization and resource reuse in multi-model, multi-tenant concurrent scenarios.

[0121] Reference Figure 4 This disclosure provides a computing resource allocation system, including:

[0122] The acquisition module 101 is used to collect the current operating status information of computing devices in each node of the computing power cluster, and to construct a global resource view based on the current operating status information;

[0123] The generation module 102 is used to generate multiple feasible resource allocation candidate schemes based on the global resource view, the memory requirements of the target model, and the parallel strategy.

[0124] Module 103 is established to build an optimization model based on all candidate resource allocation schemes;

[0125] The acquisition module 104 is used to solve the optimization model and obtain the optimal resource allocation candidate scheme;

[0126] The sorting module 105 is used to sort the optimal resource allocation candidate schemes into a deployment configuration list based on the preset orchestration template;

[0127] Execution module 106 is used to perform inference tasks on the target model based on the deployment configuration manifest.

[0128] The various variations and specific examples of the computing resource allocation method provided above are also applicable to the computing resource allocation system provided in this disclosure. Through the foregoing detailed description of the computing resource allocation method, those skilled in the art can clearly understand the implementation method of the computing resource allocation system. For the sake of brevity, they will not be described in detail here.

[0129] A computer device according to embodiments of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0130] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the computer device to perform all or part of the steps of the computing resource allocation methods of the foregoing embodiments of this disclosure.

[0131] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.

[0132] like Figure 5 This is a schematic diagram of a computer device provided for an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the computer device in the embodiments of the present disclosure. Figure 5 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0133] like Figure 5 As shown, a computer device may include a processor (such as a central processing unit, graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from storage devices into random access memory (RAM). The RAM also stores various programs and data required for the operation of the computer device. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0134] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow the computer device to communicate wirelessly or wiredly with other devices (such as edge computing devices) to exchange data. Although Figure 5 A computer apparatus with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or included alternatively.

[0135] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the computing resource allocation method of embodiments of this disclosure are performed.

[0136] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0137] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the computing resource allocation methods of the foregoing embodiments of the present disclosure are performed.

[0138] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0139] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0140] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0141] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0142] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0143] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0144] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0145] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0146] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for allocating computing resources, characterized in that, include: Collect the current operating status information of computing devices in each node of the computing power cluster, and construct a global resource view based on the current operating status information; Based on the global resource view, the memory requirements of the target model, and the parallel strategy, several feasible resource allocation candidate schemes are generated. An optimization model is established based on all candidate resource allocation schemes. The optimization model includes an objective function; The expression for the objective function is: In the formula, Indicates the weighting coefficient; Indicates the first The selection variables for each node ,when When, it indicates the first When no node is selected, When, it indicates the first One node was selected; Indicates the first Available video memory for each computing device; Indicates the first The selection variables for each computing device ,when When, it indicates the first A computing device was not selected when When, it indicates the first One computing device was selected; Indicates according to the first When deploying a model with multiple resource allocation candidate schemes, the amount of video memory usage needs to be evenly distributed across each computing device. Indicates the first The actual total number of computing devices required for each resource allocation candidate scheme; Indicates the first The selection variables for each resource allocation candidate scheme ,when When, it indicates the first When no resource allocation candidate was selected, When, it indicates the first One resource allocation candidate was selected; Solve the optimization model to obtain the optimal resource allocation candidate scheme; Based on the preset orchestration template, the optimal resource allocation candidate schemes are organized into a deployment configuration list; Based on the deployment configuration list, perform inference tasks for the target model.

2. The computing power resource allocation method according to claim 1, characterized in that, The system collects operational status information of each node and computing device within the computing power cluster, and constructs a global resource view based on this operational status information, including: Real-time monitoring of the operating status information of computing devices in each node of the computing power cluster; When the inference task of the target model is received, a lightweight snapshot is uniformly pulled within the preset acquisition window through the control container group in the computing power cluster to obtain the current running status information of the computing devices in each node; The current running status information is organized into a global resource view in tabular form.

3. The computing resource allocation method according to claim 1, characterized in that, Based on the global resource view, the memory requirements of the target model, and the parallel strategy, several feasible resource allocation candidate schemes are generated, including: Based on the global resource view, the memory requirements of the target model, and the parallel strategy, multiple feasible resource allocation candidate schemes are enumerated, and the variable parameters of the resource allocation candidate schemes are obtained. Configure global variable parameters to bind the variable parameters of each resource allocation candidate scheme to the global variable parameters.

4. The computing resource allocation method according to claim 1, characterized in that, The optimization model also includes constraints; The constraints include global consistency constraints and linear inequality constraints.

5. The computing power resource allocation method according to claim 4, characterized in that, The expression for the global consistency constraint is: In the formula, Indicates the total number of computing devices. ; This indicates the total number of devices that need to be calculated globally; The expression for the linear inequality constraint is: In the formula, This indicates the actual allocated video memory of the computing device; Represents a constant.

6. The computing resource allocation method according to claim 1, characterized in that, Solving the optimization model to obtain the optimal resource allocation candidate scheme includes: The constraint planning-Boolean satisfiability solver is invoked to solve the optimization model and obtain the optimal resource allocation candidate scheme.

7. A computer device, characterized in that, The computer device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the computing resource allocation method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the computing resource allocation method according to any one of claims 1-6.

9. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the computing resource allocation method according to any one of claims 1-6.

Images